├── .gitignore ├── Makefile ├── any.rst ├── basic.rst ├── buildsys.rst ├── concept.rst ├── conf.py ├── depend.rst ├── diagrams └── eclass.dot ├── distutils-legacy.rst ├── distutils.rst ├── eclass.rst ├── expert-multi.rst ├── helper.rst ├── index.rst ├── interpreter-maintenance.rst ├── interpreter.rst ├── migration.rst ├── multi.rst ├── package-maintenance.rst ├── porting.rst ├── preface.rst ├── pypi.rst ├── pytest.rst ├── qawarn.rst ├── single.rst ├── test.rst └── tox.ini /.gitignore: -------------------------------------------------------------------------------- 1 | /_build/ 2 | /.tox/ 3 | /diagrams/*.svg 4 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Minimal makefile for Sphinx documentation 2 | # 3 | 4 | # You can set these variables from the command line, and also 5 | # from the environment for the first two. 6 | SPHINXOPTS ?= 7 | SPHINXBUILD ?= sphinx-build 8 | SOURCEDIR = . 9 | BUILDDIR = _build 10 | 11 | DATA = diagrams/eclass.svg 12 | 13 | # Put it first so that "make" without argument is like "make help". 14 | help: 15 | @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 16 | 17 | %.svg: %.dot 18 | rm -f $@.tmp 19 | dot -Tsvg $< > $@.tmp 20 | mv $@.tmp $@ 21 | 22 | .PHONY: help Makefile 23 | 24 | # Catch-all target: route all unknown targets to Sphinx using the new 25 | # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). 26 | .DEFAULT: Makefile 27 | +$(MAKE) $(DATA) 28 | @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) 29 | -------------------------------------------------------------------------------- /any.rst: -------------------------------------------------------------------------------- 1 | ===================================== 2 | python-any-r1 — build-time dependency 3 | ===================================== 4 | 5 | .. highlight:: bash 6 | 7 | The ``python-any-r1`` eclass is used to enable Python support 8 | in packages needing it purely at build time. 9 | 10 | Eclass reference: `python-any-r1.eclass(5)`_ 11 | 12 | 13 | Basic use for unconditional Python 14 | ================================== 15 | The defining feature of this eclass is that it defines a ``pkg_setup`` 16 | phase. It normally calls ``python_setup`` function in order to find 17 | a suitable Python interpreter, and set the global environment 18 | appropriately. 19 | 20 | This means that a most trivial package using an autotools-compatible 21 | build system that needs Python at build time could look like 22 | the following: 23 | 24 | .. code-block:: bash 25 | :emphasize-lines: 6,8,21 26 | 27 | # Copyright 1999-2024 Gentoo Authors 28 | # Distributed under the terms of the GNU General Public License v2 29 | 30 | EAPI=8 31 | 32 | PYTHON_COMPAT=( python3_{10..13} ) 33 | 34 | inherit python-any-r1 35 | 36 | DESCRIPTION="A repository of data files describing media player capabilities" 37 | HOMEPAGE="https://cgit.freedesktop.org/media-player-info/" 38 | SRC_URI="https://www.freedesktop.org/software/${PN}/${P}.tar.gz" 39 | 40 | LICENSE="BSD" 41 | SLOT="0" 42 | KEYWORDS="~alpha amd64 ~arm ~arm64 ~hppa ~ia64 ~mips ppc ppc64 ~sh ~sparc x86" 43 | 44 | RDEPEND=">=virtual/udev-208" 45 | DEPEND="${RDEPEND}" 46 | BDEPEND=" 47 | ${PYTHON_DEPS} 48 | virtual/pkgconfig 49 | " 50 | 51 | This ebuild demonstrates the absolute minimum working code. Only 52 | the three highlighted lines are specific to Python eclasses, plus 53 | the implicitly exported ``pkg_setup`` phase. 54 | 55 | 56 | .. index:: python_gen_any_dep; python-any-r1 57 | .. index:: python_check_deps; python-any-r1 58 | .. index:: PYTHON_USEDEP; python-any-r1 59 | .. index:: python_has_version 60 | 61 | Dependencies 62 | ============ 63 | When depending on other Python packages, USE dependencies need to be 64 | declared in order to ensure that the dependencies would be built against 65 | the Python implementation used for the package. When Python 66 | dependencies need to be specified, ``${PYTHON_DEPS}`` gets replaced 67 | by a call to ``python_gen_any_dep`` generator and a matching 68 | ``python_check_deps()`` function. 69 | 70 | The ``python_gen_any_dep`` function accepts a template where literal 71 | ``${PYTHON_USEDEP}`` is substituted with appropriate USE dependency. 72 | It generates an any-of (``||``) dependency that requires all 73 | the packages to use the same Python interpreter, for at least one 74 | of the supported implementations. 75 | 76 | The ``python_check_deps()`` function needs to be declared by ebuild 77 | in order to test whether the implementation in question is suitable 78 | for building the package. In particular, it needs to verify whether 79 | the particular branch of the any-of was satisfied, or whether all 80 | dependencies were installed for the current interpreter. For that 81 | purpose, the function is called with ``PYTHON_USEDEP`` variable declared 82 | to the USE dependency string for the currently tested implementation. 83 | 84 | This is best explained using an example: 85 | 86 | .. code-block:: bash 87 | :emphasize-lines: 19-22,25-28 88 | 89 | # Copyright 1999-2024 Gentoo Authors 90 | # Distributed under the terms of the GNU General Public License v2 91 | 92 | EAPI=8 93 | 94 | PYTHON_COMPAT=( python3_{10..13} ) 95 | 96 | inherit meson python-any-r1 97 | 98 | DESCRIPTION="A file manager for Cinnamon, forked from Nautilus" 99 | HOMEPAGE="http://developer.linuxmint.com/projects/cinnamon-projects.html" 100 | SRC_URI="https://github.com/linuxmint/nemo/archive/${PV}.tar.gz -> ${P}.tar.gz" 101 | 102 | LICENSE="GPL-2+ LGPL-2+ FDL-1.1" 103 | SLOT="0" 104 | KEYWORDS="amd64 x86" 105 | 106 | DEPEND=" 107 | $(python_gen_any_dep ' 108 | dev-python/polib[${PYTHON_USEDEP}] 109 | dev-python/pygobject:3[${PYTHON_USEDEP}] 110 | ') 111 | " 112 | 113 | python_check_deps() { 114 | python_has_version "dev-python/polib[${PYTHON_USEDEP}]" && 115 | python_has_version "dev-python/pygobject:3[${PYTHON_USEDEP}]" 116 | } 117 | 118 | This means that the package will work with Python 3.6, 3.7 or 3.8, 119 | provided that its both dependencies have the same implementation 120 | enabled. The generated ``||`` dep ensures that this is true for 121 | at least one of them, while ``python_check_deps()`` verifies which 122 | branch was satisfied. 123 | 124 | The eclass provides a ``python_has_version`` wrapper that helps 125 | verifying whether the dependencies are installed. The wrapper takes 126 | a single optional dependency class flag, followed by one or more package 127 | dependencies. Similarly to EAPI 7+ ``has_version``, the root flag 128 | can be ``-b`` (for packages from ``BDEPEND``), ``-d`` (for ``DEPEND``) 129 | or ``-r`` (for ``RDEPEND``, ``IDEPEND`` and ``PDEPEND``). When no flag 130 | is passed, ``-b`` is assumed. The wrapper verifies whether 131 | the specified packages are installed, verbosely printing the checks 132 | performed and their results. It returns success if all packages were 133 | found, false otherwise. 134 | 135 | Note that when multiple invocations are used, ``&&`` needs to be used 136 | to chain the results. The example above can be also written as:: 137 | 138 | python_check_deps() { 139 | python_has_version \ 140 | "dev-python/polib[${PYTHON_USEDEP}]" \ 141 | "dev-python/pygobject:3[${PYTHON_USEDEP}]" 142 | } 143 | 144 | It is important to understand that this works correctly only if 145 | ``python_gen_any_dep`` and ``python_check_deps()`` match exactly. 146 | Furthermore, for any USE flag combination ``python_gen_any_dep`` must be 147 | called at most once. In particular, it is invalid to split the above 148 | example into multiple ``python_gen_any_dep`` calls. 149 | 150 | 151 | Conditional Python use 152 | ====================== 153 | In some packages, Python is only necessary with specific USE flag 154 | combinations. This is particularly common when Python is used for 155 | the test suite. In that case, the dependencies and ``pkg_setup`` call 156 | need to be wrapped in appropriate USE conditions: 157 | 158 | .. code-block:: bash 159 | :emphasize-lines: 17,18,22-28,36 160 | 161 | # Copyright 1999-2024 Gentoo Authors 162 | # Distributed under the terms of the GNU General Public License v2 163 | 164 | EAPI=8 165 | 166 | PYTHON_COMPAT=( python3_{10..13} ) 167 | 168 | inherit python-any-r1 169 | 170 | DESCRIPTION="Programmable Completion for bash" 171 | HOMEPAGE="https://github.com/scop/bash-completion" 172 | SRC_URI="https://github.com/scop/bash-completion/releases/download/${PV}/${P}.tar.xz" 173 | 174 | LICENSE="GPL-2+" 175 | SLOT="0" 176 | KEYWORDS="~alpha amd64 arm ~arm64 ~hppa ia64 ~mips ppc ~ppc64 ~s390 ~sh sparc x86 ~amd64-linux ~x86-linux ~ppc-macos ~x64-macos ~x86-macos ~m68k-mint ~sparc-solaris ~sparc64-solaris" 177 | IUSE="test" 178 | RESTRICT="!test? ( test )" 179 | 180 | RDEPEND=">=app-shells/bash-4.3_p30-r1:0" 181 | BDEPEND=" 182 | test? ( 183 | ${RDEPEND} 184 | $(python_gen_any_dep ' 185 | dev-python/pexpect[${PYTHON_USEDEP}] 186 | dev-python/pytest[${PYTHON_USEDEP}] 187 | ') 188 | )" 189 | 190 | python_check_deps() { 191 | python_has_version -d "dev-python/pexpect[${PYTHON_USEDEP}]" && 192 | python_has_version -d "dev-python/pytest[${PYTHON_USEDEP}]" 193 | } 194 | 195 | pkg_setup() { 196 | use test && python-any-r1_pkg_setup 197 | } 198 | 199 | 200 | Additional conditional dependencies 201 | =================================== 202 | Another possible case is that Python is required unconditionally 203 | but some dependencies are required only conditionally to USE flags. 204 | The simplest way to achieve that is to use ``${PYTHON_DEPS}`` globally 205 | and ``python_gen_any_dep`` in USE-conditional block, then express 206 | a similar condition in ``python_check_deps()``: 207 | 208 | .. code-block:: bash 209 | :emphasize-lines: 17,21-26,29-32 210 | 211 | # Copyright 1999-2024 Gentoo Authors 212 | # Distributed under the terms of the GNU General Public License v2 213 | 214 | EAPI=8 215 | 216 | PYTHON_COMPAT=( python3_{10..13} ) 217 | 218 | inherit python-any-r1 cmake 219 | 220 | DESCRIPTION="Qt bindings for the Telepathy D-Bus protocol" 221 | HOMEPAGE="https://telepathy.freedesktop.org/" 222 | SRC_URI="https://telepathy.freedesktop.org/releases/${PN}/${P}.tar.gz" 223 | 224 | LICENSE="LGPL-2.1" 225 | SLOT="0" 226 | KEYWORDS="amd64 ~arm arm64 x86" 227 | IUSE="test" 228 | RESTRICT="!test? ( test )" 229 | 230 | BDEPEND=" 231 | ${PYTHON_DEPS} 232 | test? ( 233 | $(python_gen_any_dep ' 234 | dev-python/dbus-python[${PYTHON_USEDEP}] 235 | ') 236 | ) 237 | " 238 | 239 | python_check_deps() { 240 | use test || return 0 241 | python_has_version -b "dev-python/dbus-python[${PYTHON_USEDEP}]" 242 | } 243 | 244 | 245 | Multiple sets of conditional dependencies 246 | ========================================= 247 | The hardest case for this eclass is to declare multiple Python 248 | dependencies conditional to different USE flags. While there are 249 | multiple possible ways of doing that, the least error-prone is to move 250 | USE conditional blocks inside ``python_gen_any_dep``: 251 | 252 | .. code-block:: bash 253 | :emphasize-lines: 16,22-28,31-37,40 254 | 255 | # Copyright 1999-2024 Gentoo Authors 256 | # Distributed under the terms of the GNU General Public License v2 257 | 258 | EAPI=8 259 | 260 | PYTHON_COMPAT=( python3_{10..13} ) 261 | 262 | inherit gnome2 python-any-r1 263 | 264 | DESCRIPTION="GObject library for accessing the freedesktop.org Secret Service API" 265 | HOMEPAGE="https://wiki.gnome.org/Projects/Libsecret" 266 | 267 | LICENSE="LGPL-2.1+ Apache-2.0" # Apache-2.0 license is used for tests only 268 | SLOT="0" 269 | KEYWORDS="~alpha amd64 arm arm64 ia64 ~mips ppc ppc64 sparc x86" 270 | IUSE="+introspection test" 271 | RESTRICT="!test? ( test )" 272 | # Tests fail with USE=-introspection, https://bugs.gentoo.org/655482 273 | REQUIRED_USE="test? ( introspection )" 274 | 275 | BDEPEND=" 276 | test? ( 277 | $(python_gen_any_dep ' 278 | dev-python/mock[${PYTHON_USEDEP}] 279 | dev-python/dbus-python[${PYTHON_USEDEP}] 280 | introspection? ( dev-python/pygobject:3[${PYTHON_USEDEP}] ) 281 | ') 282 | ) 283 | " 284 | 285 | python_check_deps() { 286 | if use introspection; then 287 | python_has_version "dev-python/pygobject:3[${PYTHON_USEDEP}]" || return 1 288 | fi 289 | python_has_version "dev-python/mock[${PYTHON_USEDEP}]" && 290 | python_has_version --host-root "dev-python/dbus-python[${PYTHON_USEDEP}]" 291 | } 292 | 293 | pkg_setup() { 294 | use test && python-any-r1_pkg_setup 295 | } 296 | 297 | 298 | .. _python-any-r1.eclass(5): 299 | https://devmanual.gentoo.org/eclass-reference/python-any-r1.eclass/index.html 300 | -------------------------------------------------------------------------------- /basic.rst: -------------------------------------------------------------------------------- 1 | ============= 2 | Common basics 3 | ============= 4 | 5 | .. highlight:: bash 6 | 7 | The various eclasses in python-r1 try to follow a single design. You 8 | will probably use more than one of them, so it is worthwhile to shortly 9 | explain the common bits used by all of them, as well as the non-obvious 10 | differences between them. 11 | 12 | 13 | .. index:: PYTHON_COMPAT 14 | 15 | PYTHON_COMPAT 16 | ============= 17 | The ``PYTHON_COMPAT`` variable is used by all Python eclasses, and must 18 | be declared in all ebuilds before they are inherited. It specifies 19 | the list of Python implementations supported by the package. 20 | 21 | The valid values are: 22 | 23 | - ``pythonX_Y`` for CPython X.Y 24 | - ``pythonX_Yt`` for CPython X.Y freethreading 25 | - ``pypy3`` for PyPy3.10 26 | - ``pypy3_Y`` for PyPy3.Y (used with PyPy 3.11 and newer) 27 | 28 | Typical use:: 29 | 30 | PYTHON_COMPAT=( python3_{10..13} pypy3 pypy3_11 ) 31 | inherit python-single-r1 32 | 33 | 34 | .. index:: PYTHON_DEPS 35 | .. index:: PYTHON_REQUIRED_USE 36 | 37 | PYTHON_DEPS and PYTHON_REQUIRED_USE 38 | =================================== 39 | The ``python-any-r1``, ``python-single-r1`` and ``python-r1`` eclasses 40 | all assume that the developer is responsible for explicitly putting 41 | the dependency strings and USE requirements in correct variables. 42 | This is meant to consistently cover packages that use Python 43 | unconditionally and conditionally, at build time and at runtime. 44 | 45 | For ``python-single-r1`` and ``python-r1``, the most basic form to use 46 | Python unconditionally is to define the following:: 47 | 48 | REQUIRED_USE=${PYTHON_REQUIRED_USE} 49 | 50 | RDEPEND=${PYTHON_DEPS} 51 | BDEPEND=${RDEPEND} 52 | 53 | For ``python-any-r1``, only build-time dependencies are used:: 54 | 55 | BDEPEND=${PYTHON_DEPS} 56 | 57 | This does not apply to ``distutils-r1`` as it does the above assignment 58 | by default. 59 | 60 | 61 | .. index:: EPYTHON 62 | .. index:: PYTHON 63 | 64 | Python environment 65 | ================== 66 | The eclasses commonly use the concept of *Python environment*. This 67 | means a state of environment enforcing a particular Python 68 | implementation. Whenever the ebuild code is run inside this 69 | environment, two variables referring to the specific Python interpreter 70 | are being exported: 71 | 72 | - ``EPYTHON`` containing the interpreter's basename (also used 73 | as the implementation identifier), e.g. ``python3.10`` 74 | - ``PYTHON`` containing the absolute final path to the interpreter, 75 | e.g. ``/usr/bin/python3.10`` 76 | 77 | The full path should only be used to provide the value that should 78 | be embedded in the installed programs, e.g. in the shebangs. 79 | For spawning Python during the build, ``EPYTHON`` is preferable. 80 | 81 | .. Warning:: 82 | 83 | Using full path rather than the basename will bypass the virtualenv 84 | created by ``distutils-r1.eclass`` in PEP 517 mode. This may cause 85 | failures to import Python modules, or use of the previous installed 86 | version rather than the just-built one. Using ``${EPYTHON}`` 87 | resolves these problems. 88 | 89 | Wrappers for ``python``, ``pythonN`` and some common tools are provided 90 | in PATH, and ``/usr/bin/python`` etc. also enforce the specific 91 | implementation via python-exec (for programs that hardcode full path). 92 | 93 | The environment can be either established in local scope, or globally. 94 | The local scope generally applies to multi-impl packages, and is created 95 | either by calls to ``python_foreach_impl`` from ``python-r1``, or inside 96 | sub-phase functions in ``distutils-r1``. The global scope setup is done 97 | via calling ``python_setup``, either directly or via default 98 | ``pkg_setup`` in ``python-any-r1`` and ``python-single-r1``. 99 | 100 | 101 | Dependencies in Python packages 102 | =============================== 103 | .. Note:: 104 | 105 | The following sections focus specifically on dependencies that 106 | are Python packages. Python software often depends on external 107 | tools, libraries written in other programming languages, etc. 108 | For these dependencies, the usual Gentoo rules apply. 109 | 110 | 111 | .. index:: BDEPEND 112 | .. index:: DEPEND 113 | .. index:: RDEPEND 114 | 115 | The most common dependency types 116 | -------------------------------- 117 | The dependencies found in Python packages can usually be classified 118 | into two categories: runtime dependencies and build-time dependencies. 119 | 120 | *Runtime dependencies* are packages that are required to be present 121 | in order for the installed Python modules and scripts to be usable. 122 | In general, these are all packages whose modules are imported 123 | in the installed Python files. Generally runtime dependencies 124 | are not needed at build time and therefore the build systems 125 | do not verify whether they are installed. However, modern Python 126 | scripts based on entry points often refuse to run if their dependencies 127 | are not satisfied. Runtime dependencies should be placed 128 | in ``RDEPEND``. 129 | 130 | A special subclass of runtime dependencies are *optional runtime 131 | dependencies* (often called 'extra' dependencies). The dependencies are 132 | optional if the package can still be meaningfully functional when they 133 | are not installed. This usually means that the package either handles 134 | failing imports gracefully, or that they are imported only in a subset 135 | of package's installed modules and that the package can still be 136 | meaningfully used without importing these modules. 137 | 138 | There are multiple approaches to handling optional dependencies. 139 | Depending on the specifics, they can: 140 | 141 | 1. be added unconditionally to ``RDEPEND`` (if they are considered 142 | important and/or light enough); 143 | 144 | 2. be listed as an informational message in ``pkg_postinst`` (usually 145 | utilizing ``optfeature.eclass``); 146 | 147 | 3. be added to ``RDEPEND`` conditionally to USE flags (this is only 148 | acceptable if the package is cheap to rebuild). 149 | 150 | *Build-time dependencies* are the packages needed for the package 151 | to be built and installed. In general, they include the packages 152 | providing the build system. In some cases, they may also include some 153 | runtime dependencies, e.g. when they are needed to import 154 | the ``__init__.py`` of the package. As a rule of thumb, if the package 155 | can be built correctly when the specific dependency is not installed, 156 | it does not need to be listed as a build dependency. Most of the time, 157 | build dependencies belong in ``BDEPEND``. 158 | 159 | The ``distutils-r1`` class generally takes care of adding the dependency 160 | on the build system and basic tooling. However, additional plugins 161 | (e.g. ``dev-python/setuptools_scm``) need to be listed explicitly. 162 | 163 | A special class of build-time dependencies are requirements specific 164 | to running the test suite and building documentation. Most of the time 165 | the former include not only the test runner but also all runtime 166 | dependencies of the package (since the test suite runs its code). 167 | Sometimes this is also required to build documentation. These classes 168 | of dependencies go into ``BDEPEND`` under ``test`` and ``doc`` USE flags 169 | respectively. 170 | 171 | Note that sometimes test dependencies can also be optional (including 172 | optional runtime dependencies). They should generally be added 173 | unconditionally to ensure maximum test coverage. Also note that 174 | (as explained further in the Guide), some test dependencies 175 | (e.g. on linters or coverage reporting tools) may actually 176 | be undesirable. 177 | 178 | Again, ``distutils-r1`` provides functions to conveniently add support 179 | for common test runner and Sphinx-based documentation. The former also 180 | takes care of copying ``RDEPEND`` into test dependencies. 181 | 182 | Some Python packages include C extensions that depend on external 183 | libraries. In this case, similarly to non-Python packages, 184 | the dependency on packages providing these libraries needs to go 185 | into ``RDEPEND`` and ``DEPEND`` (not ``BDEPEND``). 186 | 187 | Finally, there are Python packages providing C headers such 188 | as ``dev-python/numpy``. If the package in question uses both headers 189 | and Python code from NumPy, the dependency may need to be included 190 | in all three of ``RDEPEND``, ``DEPEND`` and ``BDEPEND`` (unconditionally 191 | or for tests). 192 | 193 | 194 | Finding dependency lists from build systems 195 | ------------------------------------------- 196 | Most of the modern Python build systems include all the package metadata 197 | in the ``pyproject.toml`` file. Setuptools are using ``setup.cfg`` 198 | and/or ``setup.py``. Some packages also include custom code to read 199 | dependencies from external files; it is usually worthwhile to look 200 | for ``requirements`` in the name. 201 | 202 | .. Warning:: 203 | 204 | Unconditional runtime dependencies and unconditional build-time 205 | dependencies are often enforced by the script wrappers and build 206 | systems respectively. If upstream lists spurious dependencies, 207 | they often need to be explicitly stripped rather than just ommitted 208 | from ebuild. 209 | 210 | The keys commonly used to list specific kinds of dependencies in common 211 | Python build systems: 212 | 213 | 1. Runtime dependencies (unconditional): 214 | 215 | - `PEP 621`_ metadata: ``project.dependencies`` 216 | - older flit versions: ``tool.flit.metadata.requires`` 217 | - poetry: ``tool.poetry.dependencies`` (note: this also includes 218 | special ``python`` entry to indicate compatible Python versions) 219 | - setuptools: ``install_requires`` 220 | 221 | 2. Optional runtime and/or build-time dependencies: 222 | 223 | - `PEP 621`_ metadata: ``project.optional-dependencies`` 224 | - older flit versions: ``tool.flit.metadata.requires-extra`` 225 | - poetry: ``tool.poetry.dependencies`` with ``optional = true``, 226 | sometimes grouped using ``tool.poetry.extras`` 227 | - setuptools: ``extras_require`` 228 | 229 | 3. Build-time dependencies (unconditional): 230 | 231 | - all ``pyproject.toml`` build systems: ``build-system.requires`` 232 | - poetry: ``tool.poetry.dev-dependencies`` 233 | - setuptools: ``setup_requires`` (deprecated) 234 | 235 | 4. Test dependencies (in addition to ``RDEPEND``): 236 | 237 | - often listed as ``test`` key in optional dependencies 238 | - setuptools: ``tests_require`` (deprecated) 239 | - in some cases they can also be found in ``tox.ini`` 240 | or ``noxfile.py`` 241 | 242 | 5. Doc building dependencies: 243 | 244 | - often listed as ``doc`` key in optional dependencies 245 | 246 | 6. Python version compatibility: 247 | 248 | - `PEP 621`_ metadata: ``project.requires-python`` 249 | - older flit versions: ``tool.flit.metadata.requires-python`` 250 | - poetry: ``python`` in ``tool.poetry.dependencies`` 251 | - setuptools: ``python_requires`` 252 | 253 | .. _PEP 621: https://www.python.org/dev/peps/pep-0621 254 | -------------------------------------------------------------------------------- /concept.rst: -------------------------------------------------------------------------------- 1 | ================= 2 | Advanced concepts 3 | ================= 4 | 5 | .. highlight:: python 6 | 7 | Namespace packages 8 | ================== 9 | 10 | Hierarchical package structure 11 | ------------------------------ 12 | Traditionally, Python packages were organized into a hierarchical 13 | structure with modules and subpackages being located inside the parent 14 | package directory. When submodules are imported, they are represented 15 | as attributes on the parent module. Consider the following session:: 16 | 17 | >>> import sphinx.addnodes 18 | >>> sphinx 19 | 20 | >>> sphinx.addnodes 21 | 22 | 23 | This works fine most of the time. However, it start being problematic 24 | when multiple Gentoo packages install parts of the same top-level 25 | package. This may happen e.g. with some plugin layouts where plugins 26 | are installed inside the package. More commonly, it happens when 27 | upstream wishes all their packages to start with a common component. 28 | 29 | This is the case with Zope framework. Different Zope packages share 30 | common ``zope`` top-level package. ``dev-python/zope-interface`` 31 | installs into ``zope.interface``, ``dev-python/zope-event`` 32 | into ``zope.event``. For this to work using the hierarchical layout, 33 | a common package has to install ``zope/__init__.py``, then other Zope 34 | packages have to depend on it and install sub-packages inside that 35 | directory. As far as installed packages are concerned, this is entirely 36 | doable. 37 | 38 | The real problem happens when we wish to test a freshly built package 39 | that depends on an installed package. In that case, Python imports 40 | ``zope`` from build directory that contains only ``zope.interface``. 41 | It will not be able to import ``zope.event`` that is installed in system 42 | package directory:: 43 | 44 | >>> import zope.interface 45 | >>> zope 46 | 47 | >>> zope.interface 48 | 49 | >>> import zope.event 50 | Traceback (most recent call last): 51 | File "", line 1, in 52 | ModuleNotFoundError: No module named 'zope.event' 53 | 54 | Now, this could be worked around by copying all other subpackages back 55 | to the build directory. However, there is a better solution. 56 | 57 | 58 | Namespace package structure 59 | --------------------------- 60 | Unlike traditional packages, namespace packages act as a kind of proxy. 61 | They are not strictly bound to the containing directory, and instead 62 | permit loading subpackages from all directories found in module search 63 | path. If we make ``zope`` a namespace package, we can import both 64 | the locally built ``zope.interface`` and system ``zope.event`` 65 | packages:: 66 | 67 | >>> import zope.interface 68 | >>> import zope.event 69 | >>> zope 70 | 71 | >>> zope.interface 72 | 73 | >>> zope.event 74 | 75 | 76 | There are three common methods of creating namespace packages: 77 | 78 | 1. `PEP 420`_ namespaces implemented in Python 3.3 and newer, 79 | 80 | 2. Using pkgutil_ standard library module, 81 | 82 | 3. Using `namespace package support in setuptools`_ (discouraged). 83 | 84 | PEP 420 namespaces are created implicitly when a package directory 85 | does not contain ``__init__.py`` file. While earlier versions 86 | of Python (including Python 2.7) ignored such directories and did not 87 | permit importing Python modules within them, Python 3.3 imports such 88 | directories as namespace packages. 89 | 90 | pkgutil namespaces use ``__init__.py`` with the following content:: 91 | 92 | __path__ = __import__('pkgutil').extend_path(__path__, __name__) 93 | 94 | setuptools namespace can use ``__init__.py`` with the following 95 | content:: 96 | 97 | __import__('pkg_resources').declare_namespace(__name__) 98 | 99 | Alternatively, setuptools normally installs a ``.pth`` file that is 100 | automatically loaded by Python and implicitly injects the namespace 101 | into Python. 102 | 103 | Both pkgutil and setuptools namespaces are portable to all versions 104 | of Python. 105 | 106 | More general information on the topic can be found under `packaging 107 | namespace packages`_ in Python Packaging User Guide. 108 | 109 | 110 | Determining whether namespaces are used 111 | --------------------------------------- 112 | The exact method of detecting namespace packages depends on the type 113 | of namespace used. 114 | 115 | PEP 420 namespaces can generally be recognized by the lack 116 | of ``__init__.py`` in an installed package directory. However, since 117 | they do not require any specific action, distinguishing them is not very 118 | important. 119 | 120 | pkgutil namespaces can be recognized through the content of their 121 | ``__init__.py``. Generally, you should find it suspicious if it is 122 | the only file in a top-level package directory, and if the name of this 123 | directory is less specific than the package name (e.g. ``zope`` for 124 | ``zope.interface``, ``ruamel`` for ``ruamel.yaml``). If you miss this, 125 | then you will learn about it when the ``__init__.py`` file collides 126 | between multiple packages. 127 | 128 | setuptools namespaces usually do not install ``__init__.py`` but 129 | do install a ``.pth`` file instead. Prior to installation, they can 130 | also be recognized by ``namespace_packages`` option in ``setup.py`` 131 | or ``setup.cfg``. However, some packages use a custom ``__init__.py`` 132 | file that does enable setuptools namespaces. 133 | 134 | 135 | Adding new namespace packages to Gentoo 136 | --------------------------------------- 137 | If the package uses PEP 420 namespaces, no special action is required. 138 | Per PEP 420 layout, the package must not install ``__init__.py`` files 139 | for namespaces. 140 | 141 | If the package uses the regular setuptools namespace install method 142 | (i.e. ``namespace_packages`` option), then the eclass detects that 143 | and strips the namespaces automatically, e.g.:: 144 | 145 | * python3_11: running distutils-r1_run_phase distutils-r1_python_install 146 | * Stripping pkg_resources-style namespace ruamel 147 | * Stripping pkg_resources-style namespace ruamel.std 148 | 149 | If the package uses pkgutil-style or setuptools-style namespaces 150 | via ``__init__.py`` files, these files need to be removed manually. 151 | This is done after the PEP 517 build phase: 152 | 153 | .. code-block:: bash 154 | 155 | python_compile() { 156 | distutils-r1_python_compile 157 | rm "${BUILD_DIR}/install$(python_get_sitedir)"/jaraco/__init__.py || die 158 | } 159 | 160 | Note that in some extreme cases, upstream combines namespace support 161 | and other code in the ``__init__.py`` file. Naturally, this file cannot 162 | be removed. No good solution has been found for this problem yet. 163 | 164 | Some packages include an explicit ``setuptools`` runtime dependency 165 | (``install_requires``) when using namespaces. If this is the only 166 | use of ``pkg_resources`` and ``setuptools`` in installed package 167 | sources, this dependency needs to be stripped, e.g.: 168 | 169 | .. code-block:: bash 170 | 171 | src_prepare() { 172 | # strip rdep specific to namespaces 173 | sed -i -e "/'setuptools'/d" setup.py || die 174 | distutils-r1_src_prepare 175 | } 176 | 177 | 178 | Legacy namespace packages in Gentoo 179 | ----------------------------------- 180 | Historically, Gentoo has used ``dev-python/namespace-*`` packages 181 | to support namespaces. This method is deprecated and it is in process 182 | of being retired. 183 | 184 | 185 | .. _PEP 420: https://www.python.org/dev/peps/pep-0420/ 186 | 187 | .. _pkgutil: https://docs.python.org/3/library/pkgutil.html 188 | 189 | .. _namespace package support in setuptools: 190 | https://setuptools.readthedocs.io/en/latest/setuptools.html#namespace-packages 191 | 192 | .. _packaging namespace packages: 193 | https://packaging.python.org/guides/packaging-namespace-packages/ 194 | -------------------------------------------------------------------------------- /conf.py: -------------------------------------------------------------------------------- 1 | # Configuration file for the Sphinx documentation builder. 2 | # 3 | # This file only contains a selection of the most common options. For a full 4 | # list see the documentation: 5 | # https://www.sphinx-doc.org/en/master/usage/configuration.html 6 | 7 | # -- Path setup -------------------------------------------------------------- 8 | 9 | # If extensions (or modules to document with autodoc) are in another directory, 10 | # add these directories to sys.path here. If the directory is relative to the 11 | # documentation root, use os.path.abspath to make it absolute, like shown here. 12 | # 13 | # import os 14 | # import sys 15 | # sys.path.insert(0, os.path.abspath('.')) 16 | 17 | 18 | # -- Project information ----------------------------------------------------- 19 | 20 | project = 'Gentoo Python Guide' 21 | copyright = '2020, Michał Górny, license: CC BY 4.0' 22 | author = 'Michał Górny' 23 | 24 | 25 | # -- General configuration --------------------------------------------------- 26 | 27 | # Add any Sphinx extension module names here, as strings. They can be 28 | # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom 29 | # ones. 30 | extensions = [ 31 | ] 32 | 33 | # List of patterns, relative to source directory, that match files and 34 | # directories to ignore when looking for source files. 35 | # This pattern also affects html_static_path and html_extra_path. 36 | exclude_patterns = ['_build', '.tox'] 37 | 38 | 39 | # -- Options for HTML output ------------------------------------------------- 40 | 41 | # The theme to use for HTML and HTML Help pages. See the documentation for 42 | # a list of builtin themes. 43 | # 44 | html_theme = 'alabaster' 45 | -------------------------------------------------------------------------------- /depend.rst: -------------------------------------------------------------------------------- 1 | ===================== 2 | Advanced dependencies 3 | ===================== 4 | 5 | .. highlight:: bash 6 | 7 | .. index:: PYTHON_REQ_USE 8 | .. index:: python_gen_impl_dep 9 | 10 | Requesting USE flags on the Python interpreter 11 | ============================================== 12 | While the majority of Python standard library modules are available 13 | unconditionally, a few are controlled by USE flags. For example, 14 | the sqlite3_ module requires ``sqlite`` flag to be enabled 15 | on the interpreter. If a package requires this module, it needs 16 | to enforce the matching flag via a USE dependency. 17 | 18 | In order to create a USE dependency on the Python interpreter, set 19 | ``PYTHON_REQ_USE`` before inheriting the eclass. This will cause 20 | the eclass to generate appropriate dependency string in ``PYTHON_DEPS``. 21 | 22 | .. code-block:: bash 23 | :emphasize-lines: 7 24 | 25 | # Copyright 1999-2024 Gentoo Authors 26 | # Distributed under the terms of the GNU General Public License v2 27 | 28 | EAPI=8 29 | 30 | PYTHON_COMPAT=( python3_12 ) 31 | PYTHON_REQ_USE="sqlite" 32 | 33 | inherit python-r1 gnome2-utils meson xdg-utils 34 | 35 | DESCRIPTION="Modern music player for GNOME" 36 | HOMEPAGE="https://wiki.gnome.org/Apps/Lollypop" 37 | SRC_URI="https://adishatz.org/${PN}/${P}.tar.xz" 38 | KEYWORDS="~amd64" 39 | 40 | LICENSE="GPL-3" 41 | SLOT="0" 42 | REQUIRED_USE="${PYTHON_REQUIRED_USE}" 43 | 44 | DEPEND="${PYTHON_DEPS} 45 | ..." 46 | 47 | Full USE dependency syntax is permitted. For example, you can make 48 | the dependency conditional to a flag on the package: 49 | 50 | .. code-block:: bash 51 | :emphasize-lines: 8,22 52 | 53 | # Copyright 1999-2024 Gentoo Authors 54 | # Distributed under the terms of the GNU General Public License v2 55 | 56 | EAPI=8 57 | 58 | DISTUTILS_USE_PEP517=setuptools 59 | PYTHON_COMPAT=( python3_12 ) 60 | PYTHON_REQ_USE="sqlite?" 61 | 62 | inherit distutils-r1 63 | 64 | DESCRIPTION="A lightweight password-manager with multiple database backends" 65 | HOMEPAGE="https://pwman3.github.io/pwman3/" 66 | SRC_URI=" 67 | https://github.com/pwman3/pwman3/archive/v${PV}.tar.gz 68 | -> ${P}.tar.gz 69 | " 70 | 71 | LICENSE="GPL-3" 72 | SLOT="0" 73 | KEYWORDS="~amd64" 74 | IUSE="mongodb mysql postgres +sqlite" 75 | 76 | Finally, there are cases when the problem cannot be fully solved using 77 | a single USE dependency. Additional Python interpreter dependencies 78 | with specific USE flags can be constructed using ``python_gen_impl_dep`` 79 | helper then. For example, the following ebuild requires Python with 80 | SQLite support when running tests: 81 | 82 | .. code-block:: bash 83 | :emphasize-lines: 26 84 | 85 | # Copyright 1999-2024 Gentoo Authors 86 | # Distributed under the terms of the GNU General Public License v2 87 | 88 | EAPI=8 89 | 90 | DISTUTILS_USE_PEP517=setuptools 91 | PYTHON_COMPAT=( python3_{10..13} pypy3 ) 92 | 93 | inherit distutils-r1 pypi 94 | 95 | DESCRIPTION="Let your Python tests travel through time" 96 | HOMEPAGE=" 97 | https://github.com/spulec/freezegun 98 | https://pypi.org/project/freezegun/ 99 | " 100 | 101 | LICENSE="Apache-2.0" 102 | SLOT="0" 103 | KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~amd64-linux ~x86-linux ~ppc-macos ~x64-macos ~x86-macos" 104 | 105 | RDEPEND=" 106 | >dev-python/python-dateutil-2.7[${PYTHON_USEDEP}] 107 | " 108 | BDEPEND=" 109 | test? ( 110 | $(python_gen_impl_dep sqlite) 111 | ) 112 | " 113 | 114 | distutils_enable_tests pytest 115 | 116 | 117 | .. index:: python_gen_cond_dep; for conditional deps 118 | 119 | Dependencies conditional to Python version 120 | ========================================== 121 | When packaging software for multiple Python versions, it is quite likely 122 | that you'll find yourself needing some packages only with some 123 | of the versions, and not with others. This is the case with backports 124 | and other compatibility packages. It also happens if some 125 | of the optional dependencies do not support the full set 126 | of implementations your package supports. 127 | 128 | A dependency that applies only to a subset of ``PYTHON_COMPAT`` can 129 | be created using ``python_gen_cond_dep`` function (the same as used 130 | in ``python-single-r1``). It takes a dependency string template, 131 | followed by zero or more implementation arguments. The dependencies 132 | are output for every matching implementation. 133 | 134 | The dependency template should contain literal (usually escaped through 135 | use of single quotes) ``${PYTHON_USEDEP}`` that will be substituted 136 | with partial USE dependency by the eclass function (when using 137 | ``python-single-r1``, ``${PYTHON_SINGLE_USEDEP}`` is also permitted). 138 | 139 | The implementation arguments can be: 140 | 141 | 1. Literal implementation names. For example, if a particular feature 142 | is only available on a subset of Python implementations supported 143 | by the package:: 144 | 145 | RDEPEND=" 146 | cli? ( 147 | $(python_gen_cond_dep ' 148 | dev-python/black[${PYTHON_USEDEP}] 149 | dev-python/click[${PYTHON_USEDEP}] 150 | ' python3_{8..10}) 151 | ) 152 | " 153 | 154 | 2. ``fnmatch(3)``-style wildcard against implementation names. 155 | For example, CFFI is part of PyPy's stdlib, so the explicit package 156 | needs to be only installed for CPython:: 157 | 158 | RDEPEND=" 159 | $(python_gen_cond_dep ' 160 | dev-python/cffi[${PYTHON_USEDEP}] 161 | ' 'python*') 162 | " 163 | 164 | Remember that the patterns need to be escaped to prevent filename 165 | expansion from happening. 166 | 167 | 3. Python standard library versions that are expanded into appropriate 168 | implementations by the eclass. For example, this makes it convenient 169 | to depend on backports:: 170 | 171 | RDEPEND=" 172 | $(python_gen_cond_dep ' 173 | dev-python/backports-zoneinfo[${PYTHON_USEDEP}] 174 | ' 3.8) 175 | " 176 | 177 | The advantage of this form is that it matches all the Python 178 | implementations (currently CPython and PyPy) using a specific stdlib 179 | version. 180 | 181 | An important feature of ``python_gen_cond_dep`` is that it handles 182 | removal of old implementations gracefully. When one of the listed 183 | implementations is no longer supported, it silently ignores it. This 184 | makes it possible to remove old implementations without having to update 185 | all dependency strings immediately. 186 | 187 | For example, in the following example the dependency became empty when 188 | Python 3.7 was removed:: 189 | 190 | RDEPEND=" 191 | $(python_gen_cond_dep ' 192 | dev-python/importlib_metadata[${PYTHON_USEDEP}] 193 | ' python3_7)" 194 | 195 | 196 | .. index:: cffi, greenlet 197 | 198 | Dependencies on CFFI and greenlet 199 | ================================= 200 | The PyPy distribution includes special versions of the cffi_ 201 | and greenlet_ packages. For this reason, packages using CFFI 202 | and/or greenlet and supporting PyPy3 need to make the explicit 203 | dependencies conditional to CPython:: 204 | 205 | RDEPEND=" 206 | $(python_gen_cond_dep ' 207 | >=dev-python/cffi-1.1.0:=[${PYTHON_USEDEP}] 208 | ' 'python*') 209 | " 210 | 211 | 212 | .. _sqlite3: https://docs.python.org/3.8/library/sqlite3.html 213 | .. _cffi: https://pypi.org/project/cffi/ 214 | .. _greenlet: https://pypi.org/project/greenlet/ 215 | 216 | 217 | .. index:: test-rust 218 | 219 | Optional test suite dependencies on Rust packages 220 | ================================================= 221 | When the test suite of a high-profile Python package starts depending 222 | on Python-Rust packages, it may not be feasible to mask the package 223 | on all architectures that are not supported by Rust. In this case, 224 | it is preferable to skip the tests that require the particular 225 | dependencies. 226 | 227 | If upstream does not handle missing dependencies gracefully and refuses 228 | to merge a patch to do so, it is possible to conditionally deselect 229 | tests from the ebuild based on whether the particular dependencies are 230 | installed:: 231 | 232 | python_test() { 233 | local EPYTEST_DESELECT=() 234 | 235 | if ! has_version "dev-python/trustme[${PYTHON_USEDEP}]"; then 236 | EPYTEST_DESELECT+=( 237 | tests/test_requests.py::TestRequests::test_https_warnings 238 | ) 239 | fi 240 | 241 | epytest 242 | } 243 | 244 | Note that if the modules are imported in outer scope, ignoring the whole 245 | test file may be necessary. If a file contains both tests requiring 246 | the dependency and other useful tests, sometimes it is possible 247 | to convince upstream to move imports into specific test functions, 248 | in order to make it possible to deselect specific tests. 249 | 250 | If the tests requiring these packages are not very important, it is 251 | acceptable to skip the dependency and assume they would be run 252 | if the package was installed independently. However, if they are 253 | significant (e.g. tests for TLS support), the ``test-rust`` flag 254 | can be used to pull them in, e.g.:: 255 | 256 | IUSE="test test-rust" 257 | RESTRICT="!test? ( test )" 258 | 259 | BDEPEND=" 260 | test? ( 261 | test-rust? ( 262 | dev-python/trustme[${PYTHON_USEDEP}] 263 | ) 264 | ) 265 | " 266 | 267 | This flag is masked on profiles for architectures that do not provide 268 | a Rust toolchain, and forced on all the remaining profiles. This 269 | ensures that the respective tests are run whenever it is possible 270 | to run them. 271 | -------------------------------------------------------------------------------- /diagrams/eclass.dot: -------------------------------------------------------------------------------- 1 | digraph { 2 | node [shape=box]; 3 | "python-utils-r1" [fontcolor=grey]; 4 | "python-utils-r1" -> "python-any-r1"; 5 | "python-utils-r1" -> "python-single-r1"; 6 | "python-utils-r1" -> "python-r1"; 7 | "python-single-r1" -> "distutils-r1"; 8 | "python-r1" -> "distutils-r1"; 9 | } 10 | -------------------------------------------------------------------------------- /distutils-legacy.rst: -------------------------------------------------------------------------------- 1 | ============================ 2 | distutils-r1 legacy concepts 3 | ============================ 4 | 5 | .. highlight:: bash 6 | 7 | This section describes concepts specific to the legacy mode 8 | of the ``distutils-r1`` eclass. When using the modern PEP 517 mode, 9 | none of the features described here are available. 10 | 11 | 12 | .. index:: DISTUTILS_USE_SETUPTOOLS 13 | 14 | Different build system variations 15 | ================================= 16 | The commonly used build systems specific to Python packages can be 17 | classified for eclass support into following groups: 18 | 19 | 1. Plain distutils (built-in in Python). 20 | 2. Setuptools and its direct derivatives (e.g. pbr). 21 | 3. ``pyproject.toml``-based build systems (Flit, Poetry). 22 | 23 | The eclass supports the first two directly. Support for Flit and Poetry 24 | is provided through the ``dev-python/pyproject2setuppy`` package that 25 | converts the package's metadata to setuptools call. 26 | 27 | In addition to being a build system, setuptools provides runtime 28 | facilities via the ``pkg_resources`` module. If these facilities 29 | are used, the package needs to have a runtime dependency 30 | on ``dev-python/setuptools``. Otherwise, a build-time dependency 31 | is sufficient. 32 | 33 | 34 | DISTUTILS_USE_SETUPTOOLS 35 | ------------------------ 36 | The most common case right now is a package using setuptools as a build 37 | system, and therefore needing a build-time dependency only. This 38 | is the eclass' default. If your package does not fit this profile, 39 | you can set ``DISTUTILS_USE_SETUPTOOLS`` variable to one 40 | of the supported values: 41 | 42 | - ``no`` — pure distutils use (no extra dependencies). 43 | - ``bdepend`` — build-time use of setuptools (``BDEPEND`` 44 | on ``dev-python/setuptools``). 45 | - ``rdepend`` — build- and runtime use of setuptools (``BDEPEND`` 46 | and ``RDEPEND`` on ``dev-python/setuptools``). 47 | - ``pyproject.toml`` — use of Flit or Poetry (``BDEPEND`` 48 | on ``dev-python/pyproject2toml`` and ``dev-python/setuptools``). 49 | - ``manual`` — special value indicating that the eclass logic misbehaves 50 | and the dependency needs to be specified manually. Strongly 51 | discouraged, please report a bug and/or fix the package instead. 52 | 53 | The cases for particular values are explained in subsequent sections. 54 | 55 | The Gentoo repository includes a post-install QA check that verifies 56 | whether the value of ``DISTUTILS_USE_SETUPTOOLS`` is correct, 57 | and reports if it is most likely incorrect. This is why it is important 58 | to use the variable rather than specifying the dependency directly. 59 | An example report is:: 60 | 61 | * DISTUTILS_USE_SETUPTOOLS value is probably incorrect 62 | * have: DISTUTILS_USE_SETUPTOOLS=bdepend (or unset) 63 | * expected: DISTUTILS_USE_SETUPTOOLS=rdepend 64 | 65 | The value needs to be set before inheriting the eclass: 66 | 67 | .. code-block:: bash 68 | :emphasize-lines: 7 69 | 70 | # Copyright 1999-2020 Gentoo Authors 71 | # Distributed under the terms of the GNU General Public License v2 72 | 73 | EAPI=7 74 | 75 | PYTHON_COMPAT=( python3_{10..13} pypy3 ) 76 | DISTUTILS_USE_SETUPTOOLS=rdepend 77 | 78 | inherit distutils-r1 79 | 80 | DESCRIPTION="A configurable sidebar-enabled Sphinx theme" 81 | HOMEPAGE="https://github.com/bitprophet/alabaster" 82 | SRC_URI="mirror://pypi/${PN:0:1}/${PN}/${P}.tar.gz" 83 | 84 | LICENSE="BSD" 85 | KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~s390 ~sh ~sparc ~x86 ~x64-solaris" 86 | SLOT="0" 87 | 88 | 89 | distutils and setuptools build systems 90 | -------------------------------------- 91 | Distutils and setuptools are the two most common build systems 92 | for Python packages right now. Their common feature is that they use 93 | a ``setup.py`` script that interfaces with the build system. Generally, 94 | you can determine which of the two build systems are being used 95 | by looking at the imports in ``setup.py``, in particular from which 96 | module the ``setup`` function is imported. 97 | 98 | Distutils-based packages (``DISTUTILS_USE_SETUPTOOLS=no``) use e.g.: 99 | 100 | .. code-block:: python 101 | 102 | from distutils import setup 103 | 104 | Setuptools-based package (``DISTUTILS_USE_SETUPTOOLS=bdepend``, unset 105 | or possibly ``rdepend`` as indicated by the subsequent sections) use: 106 | 107 | .. code-block:: python 108 | 109 | from setuptools import setup 110 | 111 | In some cases, upstreams find it convenient to alternatively support 112 | both setuptools and distutils. A commonly used snippet is: 113 | 114 | .. code-block:: python 115 | 116 | try: 117 | from setuptools import setup 118 | except ImportError: 119 | from distutils import setup 120 | 121 | However, non-fixed build system choice can be problematic to Gentoo 122 | users. This is because pure distutils installs egg-info data as a 123 | single file, while setuptools install the same data as a directory 124 | (using the same path). Therefore, if you rebuild the same version 125 | of the package with a different build system than before, you end up 126 | trying to replace a file with a directory or the other way around. 127 | This is not permitted by the PMS and not handled cleanly by the package 128 | managers. 129 | 130 | You must always ensure that a single build system will be used 131 | unconditionally. In the case of the condition presented above, it is 132 | sufficient to leave ``DISTUTILS_USE_SETUPTOOLS`` at its default value 133 | as that will ensure that setuptools is installed and therefore 134 | the fallback will never take place. However, patching ``setup.py`` may 135 | be necessary if you want to force distutils (e.g. to enable clean 136 | bootstrap) or the upstream condition requiers that. 137 | 138 | 139 | Setuptools' entry points 140 | ------------------------ 141 | .. Important:: 142 | 143 | With removal of Python 3.7, the correct ``DISTUTILS_USE_SETUPTOOLS`` 144 | value for packages using entry points changed to ``bdepend``. 145 | 146 | *Entry points* provide the ability to expose some of the package's 147 | Python functions to other packages. They are commonly used to implement 148 | plugin systems and by setuptools themselves to implement wrapper scripts 149 | for starting programs. 150 | 151 | Entry points are defined as ``entry_points`` argument to the ``setup()`` 152 | function, or ``entry_points`` section in ``setup.cfg``. They are 153 | installed in the package's egg-info as ``entry_points.txt``. In both 154 | cases, they are grouped by entry point type, and defined as a dictionary 155 | mapping entry points names to the relevant functions. 156 | 157 | For our purposes, we are only interested in entry points used to define 158 | wrapper scripts, the ``console_scripts`` and ``gui_scripts`` groups, 159 | as they are installed with the package itself. As for plugin systems, 160 | it is reasonable to assume that the installed plugins are only 161 | meaningful to the package using them, and therefore that the package 162 | using them will depend on the appropriate metadata provider. 163 | 164 | Old versions of setuptools used to implement the script wrappers using 165 | ``pkg_resources`` package. Modern versions of setuptools use 166 | the following logic: 167 | 168 | 1. If ``importlib.metadata`` module is available (Python 3.8+), use it. 169 | In this case, no external dependencies are necessary. 170 | 171 | 2. If ``importlib_metadata`` backport is available, use it. It is 172 | provided by ``dev-python/importlib_metadata``. 173 | 174 | 3. Otherwise, fall back to ``pkg_resources``. It is provided 175 | by ``dev-python/setuptools``. 176 | 177 | Since Python 3.7 is no longer present in Gentoo, new ebuilds do not 178 | need any additional dependencies for entry points and should use 179 | the default value (i.e. remove ``DISTUTILS_USE_SETUPTOOLS``). 180 | 181 | For the time being, the QA check for incorrect values is accepting 182 | both the new value and the old ``rdepend`` value. If you wish to be 183 | reminded about the update, you can add the following variable to your 184 | ``make.conf``:: 185 | 186 | DISTUTILS_STRICT_ENTRY_POINTS=1 187 | 188 | Please note that in some cases ``rdepend`` can still be the correct 189 | value, if there are `other runtime uses of setuptools`_. In some cases 190 | the QA check will also trigger the wrong value because of leftover 191 | explicit dependencies on setuptools. 192 | 193 | 194 | Other runtime uses of setuptools 195 | -------------------------------- 196 | Besides the generated wrapper scripts, the package code itself may use 197 | the ``setuptools`` or ``pkg_resources`` packages. The common cases 198 | for this include getting package metadata and resource files. This 199 | could also be a case for plugin managers and derived build systems. 200 | 201 | As a rule of thumb, if any installed Python file imports ``setuptools`` 202 | or ``pkg_resources``, the package needs to use the value of ``rdepend``. 203 | 204 | The QA check determines that this is the case by looking at the upstream 205 | dependencies (``install_requires``) installed by the package. It is 206 | quite common for packages to miss the dependency, or have a leftover 207 | dependency. If ``install_requires`` does not match actual imports 208 | in the installed modules, please submit a patch upstream. 209 | 210 | 211 | pyproject.toml-based projects 212 | ----------------------------- 213 | The newer build systems used for Python packages avoid supplying 214 | ``setup.py`` and instead declare package's metadata and build system 215 | information in ``pyproject.toml``. Examples of these build systems 216 | are Flit and Poetry. 217 | 218 | These build systems are generally very heavy and do not support plain 219 | installation to a directory. For this reason, Gentoo is using 220 | ``dev-python/pyproject2setuppy`` to provide a thin wrapper for 221 | installing these packages using setuptools. 222 | 223 | To enable the necessary eclass logic and add appropriate build-time 224 | dependencies, specify the value of ``pyproject.toml`` 225 | to ``DISTUTILS_USE_SETUPTOOLS``. 226 | 227 | Strictly speaking, both Flit and Poetry do support entry points, 228 | and therefore some packages actually need a runtime dependency 229 | on setuptools. This is a known limitation, and it will probably 230 | not be addressed for the same reason as the logic for setuptools' entry 231 | points is not updated. 232 | 233 | 234 | .. index:: DISTUTILS_IN_SOURCE_BUILD 235 | 236 | In-source vs out-of-source builds 237 | ================================= 238 | In the general definition, an *out-of-source build* is a build where 239 | output files are placed in a directory separate from source files. 240 | By default, distutils and its derivatives always do out-of-source builds 241 | and place output files in subdirectories of ``build`` directory. 242 | 243 | Conversely, an *in-source build* happens when the output files are 244 | interspersed with source files. The closest distutils equivalent 245 | of an in-source build is the ``--inplace`` option of ``build_ext`` 246 | that places compiled C extensions alongside Python module sources. 247 | 248 | ``distutils-r1`` shifts this concept a little. When performing 249 | an out-of-source build (the default), it creates a dedicated output 250 | directory for every Python interpreter enabled, and then uses it 251 | throughout all build and install steps. 252 | 253 | It should be noted that unlike build systems such as autotools or CMake, 254 | out-of-source builds in distutils are not executed from the build 255 | directory. Instead, the setup script is executed from source directory 256 | and passed path to build directory. 257 | 258 | Sometimes out-of-source builds are incompatible with custom hacks used 259 | upstream. This could be a case if the setup script is writing 260 | implementation-specific changes to the source files (e.g. using ``2to3`` 261 | to convert them to Python 3) or relying on specific build paths. 262 | For better compatibility with those cases, the eclass provides 263 | an in-source build mode enabled via ``DISTUTILS_IN_SOURCE_BUILD``. 264 | 265 | In this mode, the eclass creates a separate copy of the source directory 266 | for each Python implementation, and then runs the build and install 267 | steps inside that copy. As a result, any changes done to the source 268 | files are contained within the copy used for the current interpreter. 269 | 270 | .. code-block:: bash 271 | :emphasize-lines: 23 272 | 273 | # Copyright 1999-2020 Gentoo Authors 274 | # Distributed under the terms of the GNU General Public License v2 275 | 276 | EAPI=7 277 | 278 | DISTUTILS_USE_SETUPTOOLS=no 279 | PYTHON_COMPAT=( python3_{10..13} pypy3 ) 280 | PYTHON_REQ_USE="xml(+)" 281 | 282 | inherit distutils-r1 pypi 283 | 284 | DESCRIPTION="Collection of extensions to Distutils" 285 | HOMEPAGE=" 286 | https://github.com/pypa/setuptools 287 | https://pypi.org/project/setuptools/ 288 | " 289 | 290 | LICENSE="MIT" 291 | SLOT="0" 292 | KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~riscv ~s390 ~sh ~sparc ~x86 ~x64-cygwin ~amd64-linux ~x86-linux ~ppc-macos ~x64-macos ~x86-macos ~sparc-solaris ~sparc64-solaris ~x64-solaris ~x86-solaris" 293 | 294 | # Force in-source build because build system modifies sources. 295 | DISTUTILS_IN_SOURCE_BUILD=1 296 | 297 | 298 | .. index:: distutils_install_for_testing 299 | 300 | Installing the package before testing 301 | ===================================== 302 | The tests are executed in ``src_test`` phase, after ``src_compile`` 303 | installed package files into the build directory. The eclass 304 | automatically adds appropriate ``PYTHONPATH`` so that the installed 305 | Python modules and extensions are used during testing. This works 306 | for the majority of packages. 307 | 308 | However, some test suites will not work correctly unless the package 309 | has been properly installed via ``setup.py install``. This may apply 310 | specifically to packages calling their executables that are created 311 | via entry points, various plugin systems or the use of package metadata. 312 | 313 | The ``distutils_install_for_testing`` function runs ``setup.py install`` 314 | into a temporary directory, and adds the appropriate paths to ``PATH`` 315 | and ``PYTHONPATH``. 316 | 317 | This function currently supports two install layouts: 318 | 319 | - the standard *root directory* layout that is enabled 320 | via ``--via-root``, 321 | 322 | - a virtualenv-alike *venv* layout that is enabled via ``--via-venv``. 323 | 324 | 325 | The eclass defaults to the root directory layout that is consistent 326 | with the layout used for the actual install. This ensures that 327 | the package's scripts are found on ``PATH``, and the package metadata 328 | is found via ``importlib.metadata`` / ``pkg_resources``. It should 329 | be sufficient to resolve the most common test problems. 330 | 331 | In some cases, particularly packages that do not preserve ``PYTHONPATH`` 332 | correctly, the virtualenv-alike layout (``--via-venv``) is better. 333 | Through wrapping the Python interpreter itself, it guarantees that 334 | the packages installed in the test environment are found independently 335 | of ``PYTHONPATH`` (just like a true venv). It should cover the few 336 | extreme cases. 337 | 338 | In EAPIs prior to 8, an additional legacy ``--via-home`` layout used 339 | to be supported. It historically used to be necessary to fix problems 340 | with some packages. However, the underlying issues probably went away 341 | along with old versions of Python, and the `removal of site.py hack`_ 342 | has broken it for most of the consumers. 343 | 344 | .. code-block:: bash 345 | 346 | python_test() { 347 | distutils_install_for_testing 348 | epytest --no-network 349 | } 350 | 351 | 352 | .. _removal of site.py hack: 353 | https://github.com/pypa/setuptools/commit/91213fb2e7eecde9f5d7582de485398f546e7aa8 354 | -------------------------------------------------------------------------------- /eclass.rst: -------------------------------------------------------------------------------- 1 | ================================ 2 | Choosing between Python eclasses 3 | ================================ 4 | 5 | Overview 6 | ======== 7 | The python-r1 eclass suite features 5 eclasses in total: 8 | 9 | 1. ``python-utils-r1.eclass`` that provides utility functions common 10 | to all eclasses. This eclass is rarely inherited directly. 11 | 12 | 2. ``python-any-r1.eclass`` that is suitable for packages using Python 13 | purely at build time. 14 | 15 | 3. ``python-single-r1.eclass`` that provides a base for simpler packages 16 | installing Python scripts and alike. 17 | 18 | 4. ``python-r1.eclass`` that provides a base for more complex packages, 19 | particularly these installing Python modules. 20 | 21 | 5. ``distutils-r1.eclass`` that provides convenient phase functions 22 | and helpers for packages that primarily involve installing Python 23 | files. 24 | 25 | .. figure:: diagrams/eclass.svg 26 | 27 | Inheritance graph of python-r1 suite eclasses. 28 | 29 | As a rule of thumb, the best eclass to use is the one that makes 30 | the ebuild the simplest while meeting its requirements. A more detailed 31 | process involves: 32 | 33 | 1. Determining whether Python is used purely at build time, 34 | or at runtime as well. In the former case, ``python-any-r1`` 35 | is the right choice. 36 | 37 | 2. Determining whether single-impl or multi-impl approach is more 38 | appropriate. For the former, ``python-single-r1`` is the correct 39 | base eclass. For the latter, ``python-r1``. 40 | 41 | 3. Determining whether the ebuild benefits from using ``distutils-r1``. 42 | If it does, this eclass should be use instead (potentially along 43 | with ``DISTUTILS_SINGLE_IMPL`` to switch the underlying eclass). 44 | 45 | 46 | Build time vs runtime use 47 | ========================= 48 | The first basis for choosing Python eclass is whether Python is used 49 | merely at build time or at runtime as well. 50 | 51 | A runtime use occurs if the package explicitly needs Python to be 52 | installed along with it, in order for it to function correctly. This 53 | generally happens if the package installs Python modules, extensions, 54 | scripts, or executables calling the Python interpreter or linking 55 | to libpython. This also applies to bash scripts or other executables 56 | that call python inline. 57 | 58 | A build time use occurs if the package calls the Python interpreter 59 | or any kind of aforementioned executables during package's build 60 | (or install) phases. 61 | 62 | If the package uses Python purely at build time, the ``python-any-r1`` 63 | eclass is appropriate. Otherwise, ``python-single-r1``, ``python-r1`` 64 | or ``distutils-r1`` are to be used. 65 | 66 | A specific exception to that rule is when the package is only calling 67 | external Python scripts directly (i.e. not via ``python /usr/bin/foo``). 68 | If the called executables can be considered fully contained 69 | dependency-wise, there is no need to use an eclass. 70 | 71 | For example, when using ``dev-util/meson`` to build a package, there is 72 | no need to use a Python eclass since Meson abstracts away its Pythonic 73 | implementation details and works as a regular executable for your 74 | packages. However, ``dev-util/scons`` requires Python eclass since it 75 | loads Python code from the package and a compatible Python version must 76 | be enforced. 77 | 78 | 79 | Single-impl vs multi-impl 80 | ========================= 81 | The second important basis for packages using Python at runtime is 82 | whether the package in question should support multi-implementation 83 | install or not. 84 | 85 | A *single-impl* package is a package requiring the user to choose 86 | exactly one Python implementation to be built against. This means 87 | that the scripts installed by that package will be run via specified 88 | Python interpreter, and that the modules and extensions will be 89 | importable from it only. The package's Python reverse dependencies will 90 | also have to use the same implementation. Since the package can't 91 | support having more than one implementation enabled, its reverse 92 | dependencies have to be simple-impl as well. 93 | 94 | Single-impl packages use ``python-single-r1`` eclass. Writing ebuilds 95 | for them is easier since it is generally sufficient to call setup 96 | function early on, and the upstream build system generally takes care 97 | of using selected Python version correctly. Making packages single-impl 98 | is recommended when dealing with packages that are not purely written 99 | for Python or have single-impl dependencies. 100 | 101 | A *multi-impl* package allows user to enable multiple (preferably 102 | any number of) implementations. The modules, extensions and scripts 103 | installed by the package are installed separately for each enabled 104 | implementation, and can therefore be used from any of them. The package 105 | can have reverse dependencies enabling only a subset of its 106 | implementations. 107 | 108 | Multi-impl packages use ``python-r1`` eclass. Ebuilds are more complex 109 | since they need to explicitly repeat build and install steps for each 110 | enabled implementation. Using this model is recommended for packages 111 | providing Python modules or extensions only, or having multi-impl 112 | reverse dependencies. In some cases supporting multi-impl build 113 | requires applying hacks, e.g. ``dev-libs/boost[python]`` uses 114 | non-standard names to install ``libboost_python`` for multiple Python 115 | versions. 116 | 117 | The implementation for single-impl packages is selected 118 | via ``PYTHON_SINGLE_TARGET``, while multi-impl uses ``PYTHON_TARGETS``. 119 | These USE flag sets can be set independently to provide greater 120 | flexibility for developers and end users. 121 | 122 | Both single-impl and multi-impl installs are supported 123 | by the ``distutils-r1`` eclass. 124 | 125 | 126 | Python-first packages (distutils-r1 eclass) 127 | =========================================== 128 | The third step in choosing the eclass for runtime use of Python 129 | is determining whether the ebuild would benefit from ``distutils-r1``. 130 | This eclass is especially useful for packages that primarily focus 131 | on providing Python content. Its advantages include: 132 | 133 | - adding appropriate dependencies and ``REQUIRED_USE`` by default 134 | 135 | - a sub-phase function mechanism that makes installing Python modules 136 | in multi-impl mode easier 137 | 138 | - convenient support for building documentation using Sphinx 139 | and running tests using common Python test runners 140 | 141 | In general, ``distutils-r1`` should be preferred over the other eclasses 142 | if: 143 | 144 | - the package uses a PEP 517-compliant build system (i.e. has 145 | a ``pyproject.toml`` file with a ``build-system`` section) 146 | 147 | - the package uses a legacy distutils or setuptools build system 148 | (i.e. has a ``setup.py`` file) 149 | 150 | - the package primarily installs Python modules 151 | 152 | In general, for multi-impl packages ``distutils-r1`` is preferred 153 | over ``python-r1`` as it usually makes the ebuilds simpler. 154 | For single-impl packages, ``python-single-r1`` can sometimes be simpler. 155 | -------------------------------------------------------------------------------- /expert-multi.rst: -------------------------------------------------------------------------------- 1 | ====================== 2 | Expert python-r1 usage 3 | ====================== 4 | 5 | .. highlight:: bash 6 | 7 | The APIs described in this chapter are powerful but even harder to use 8 | than those described in ``python-r1`` chapter. You should not consider 9 | using them unless you have a proper ninja training. 10 | 11 | 12 | .. index:: python_gen_useflags 13 | 14 | Partially restricting Python implementation 15 | =========================================== 16 | There are packages that have been ported to Python 3 only partially. 17 | They may still have some optional dependencies that support Python 2 18 | only, they may have some components that do not support Python 3 yet. 19 | The opposite is also possible — some of the features being available 20 | only for Python 3. 21 | 22 | There are two approaches to this problem. You can either skip features 23 | (ignore USE flags) if the necessary implementation is not enabled, 24 | or you can use ``REQUIRED_USE`` to enforce at least one interpreter 25 | having the requested feature. 26 | 27 | Skipping specific tasks can be done via investigating ``${EPYTHON}``. 28 | If USE flags are involved, you will probably also need to use 29 | ``python_gen_cond_dep`` with additional parameters restricting 30 | dependencies to available targets. 31 | 32 | .. code-block:: bash 33 | :emphasize-lines: 37-43,49 34 | 35 | # Copyright 1999-2020 Gentoo Authors 36 | # Distributed under the terms of the GNU General Public License v2 37 | 38 | EAPI=7 39 | 40 | PYTHON_COMPAT=( python3_{10..13} pypy3 ) 41 | PYTHON_REQ_USE="threads(+)" 42 | inherit distutils-r1 43 | 44 | DESCRIPTION="HTTP library for human beings" 45 | HOMEPAGE="http://python-requests.org/" 46 | SRC_URI="mirror://pypi/${P:0:1}/${PN}/${P}.tar.gz" 47 | 48 | LICENSE="Apache-2.0" 49 | SLOT="0" 50 | KEYWORDS="~amd64 ~arm ~arm64 ~sparc ~x86 ~amd64-linux ~x86-linux" 51 | IUSE="socks5 +ssl test" 52 | RESTRICT="!test? ( test )" 53 | 54 | RDEPEND=" 55 | >=dev-python/certifi-2017.4.17[${PYTHON_USEDEP}] 56 | >=dev-python/chardet-3.0.2[${PYTHON_USEDEP}] 57 | =dev-python/idna-2.5[${PYTHON_USEDEP}] 59 | =dev-python/PySocks-1.5.6[${PYTHON_USEDEP}] ) 62 | ssl? ( 63 | >=dev-python/cryptography-1.3.4[${PYTHON_USEDEP}] 64 | >=dev-python/pyopenssl-0.14[${PYTHON_USEDEP}] 65 | ) 66 | " 67 | 68 | BDEPEND=" 69 | dev-python/setuptools[${PYTHON_USEDEP}] 70 | test? ( 71 | $(python_gen_cond_dep ' 72 | ${RDEPEND} 73 | dev-python/pytest[${PYTHON_USEDEP}] 74 | dev-python/pytest-httpbin[${PYTHON_USEDEP}] 75 | dev-python/pytest-mock[${PYTHON_USEDEP}] 76 | >=dev-python/PySocks-1.5.6[${PYTHON_USEDEP}] 77 | ' 'python*') 78 | ) 79 | " 80 | 81 | python_test() { 82 | # tests hang with PyPy 83 | [[ ${EPYTHON} == pypy3* ]] && continue 84 | 85 | epytest 86 | } 87 | 88 | Enforcing implementations is done via putting ``||`` block with all 89 | targets providing the feature in ``REQUIRED_USE``. The eclass provides 90 | ``python_gen_useflags`` function to print valid flag names for specified 91 | implementation list. Please always use this function instead of listing 92 | actual flag names as it handled phasing implementations out gracefully. 93 | ``python_gen_cond_dep`` should also be called with matching target 94 | list. 95 | 96 | .. code-block:: bash 97 | :emphasize-lines: 19,31-33 98 | 99 | # Copyright 1999-2020 Gentoo Authors 100 | # Distributed under the terms of the GNU General Public License v2 101 | 102 | EAPI=7 103 | 104 | PYTHON_COMPAT=( python3_{6..8} ) 105 | PYTHON_REQ_USE="sqlite" 106 | inherit distutils-r1 107 | 108 | DESCRIPTION="Toolkit to convert between many translation formats" 109 | HOMEPAGE="https://github.com/translate/translate" 110 | SRC_URI="https://github.com/translate/translate/releases/download/${PV}/${P}.tar.gz" 111 | 112 | LICENSE="GPL-2" 113 | SLOT="0" 114 | KEYWORDS="amd64 arm64 x86 ~amd64-linux ~x86-linux" 115 | IUSE="+subtitles" 116 | REQUIRED_USE="${PYTHON_REQUIRED_USE} 117 | subtitles? ( || ( $(python_gen_useflags python3_{6,7}) ) )" 118 | 119 | DEPEND=">=dev-python/six-1.10.0[${PYTHON_USEDEP}]" 120 | RDEPEND="${DEPEND} 121 | !dev-python/pydiff 122 | app-text/iso-codes 123 | >=dev-python/chardet-3.0.4[${PYTHON_USEDEP}] 124 | >=dev-python/lxml-3.5[${PYTHON_USEDEP}] 125 | >=dev-python/pycountry-18.5.26[${PYTHON_USEDEP}] 126 | >=dev-python/python-levenshtein-0.12.0[${PYTHON_USEDEP}] 127 | sys-devel/gettext 128 | subtitles? ( 129 | $(python_gen_cond_dep ' 130 | media-video/gaupol[${PYTHON_USEDEP}] 131 | ' python3_{6,7}) 132 | ) 133 | " 134 | 135 | .. index:: python_setup; with implementation parameter 136 | .. index:: DISTUTILS_ALL_SUBPHASE_IMPLS 137 | 138 | Restricting interpreters for python_setup 139 | ========================================= 140 | A specific case of the restriction described above is when the build 141 | step supports a subset of Python targets for the runtime part. This 142 | could happen e.g. if package's Python bindings have been ported 143 | to Python 3 but the test suite or building tooling still requires 144 | Python 2. 145 | 146 | To support this use case, ``python_setup`` can optionally take a list 147 | of implementations. This list must be a subset of ``PYTHON_COMPAT``, 148 | and only implementation on the list can be used by ``python_setup``. 149 | Note that you also need to set matching ``REQUIRED_USE``, as otherwise 150 | the function will fail if the user does not enable any of the supported 151 | targets. 152 | 153 | .. code-block:: bash 154 | :emphasize-lines: 19,27 155 | 156 | # Copyright 1999-2020 Gentoo Authors 157 | # Distributed under the terms of the GNU General Public License v2 158 | 159 | EAPI=6 160 | 161 | PYTHON_COMPAT=( python2_7 python3_{5..7} ) 162 | 163 | inherit python-r1 toolchain-funcs 164 | 165 | DESCRIPTION="Python extension module generator for C and C++ libraries" 166 | HOMEPAGE="https://www.riverbankcomputing.com/software/sip/intro" 167 | SRC_URI="https://www.riverbankcomputing.com/static/Downloads/${PN}/${PV}/${P}.tar.gz" 168 | 169 | # Sub-slot based on SIP_API_MAJOR_NR from siplib/sip.h 170 | SLOT="0/12" 171 | LICENSE="|| ( GPL-2 GPL-3 SIP )" 172 | KEYWORDS="alpha amd64 arm arm64 ~hppa ia64 ppc ppc64 ~sparc x86 ~amd64-linux ~x86-linux ~ppc-macos ~x64-macos ~x86-macos" 173 | REQUIRED_USE="${PYTHON_REQUIRED_USE} 174 | || ( $(python_gen_useflags 'python2*') )" 175 | 176 | RDEPEND="${PYTHON_DEPS}" 177 | DEPEND="${RDEPEND} 178 | sys-devel/bison 179 | sys-devel/flex 180 | 181 | src_prepare() { 182 | python_setup 'python2*' 183 | "${EPYTHON}" build.py prepare || die 184 | default 185 | } 186 | 187 | src_configure() { 188 | configuration() { 189 | local myconf=( 190 | "${EPYTHON}" 191 | "${S}"/configure.py 192 | --bindir="${EPREFIX}/usr/bin" 193 | --destdir="$(python_get_sitedir)" 194 | --incdir="$(python_get_includedir)" 195 | ) 196 | echo "${myconf[@]}" 197 | "${myconf[@]}" || die 198 | } 199 | python_foreach_impl run_in_build_dir configuration 200 | } 201 | 202 | src_compile() { 203 | python_foreach_impl run_in_build_dir default 204 | } 205 | 206 | src_install() { 207 | installation() { 208 | emake DESTDIR="${D}" install 209 | python_optimize 210 | } 211 | python_foreach_impl run_in_build_dir installation 212 | 213 | einstalldocs 214 | } 215 | 216 | The ``distutils-r1`` equivalent of ``python_setup`` parameters is 217 | the ``DISTUTILS_ALL_SUBPHASE_IMPLS`` variable. Alternatively to global 218 | scope, it can be set in an early phase function (prior to any sub-phase 219 | call). 220 | 221 | .. code-block:: bash 222 | :emphasize-lines: 22,28-30,46 223 | 224 | # Copyright 1999-2020 Gentoo Authors 225 | # Distributed under the terms of the GNU General Public License v2 226 | 227 | EAPI=5 228 | 229 | PYTHON_COMPAT=( 230 | pypy 231 | python3_5 python3_6 python3_7 232 | python2_7 233 | ) 234 | PYTHON_REQ_USE='bzip2(+),ssl(+),threads(+)' 235 | inherit distutils-r1 236 | 237 | DESCRIPTION="Portage is the package management and distribution system for Gentoo" 238 | HOMEPAGE="https://wiki.gentoo.org/wiki/Project:Portage" 239 | SRC_URI="mirror://gentoo/${P}.tar.bz2" 240 | 241 | LICENSE="GPL-2" 242 | KEYWORDS="~alpha ~amd64 ~arm ~arm64 ~hppa ~ia64 ~m68k ~mips ~ppc ~ppc64 ~riscv ~s390 ~sh ~sparc ~x86" 243 | SLOT="0" 244 | IUSE="epydoc" 245 | REQUIRED_USE="epydoc? ( $(python_gen_useflags 'python2*') )" 246 | 247 | DEPEND=" 248 | >=app-arch/tar-1.27 249 | >=sys-apps/sed-4.0.5 sys-devel/patch 250 | epydoc? ( 251 | $(python_gen_cond_dep ' 252 | >=dev-python/epydoc-2.0[${PYTHON_USEDEP}] 253 | ' 'python2*') 254 | )" 255 | RDEPEND=" 256 | >=app-arch/tar-1.27 257 | dev-lang/python-exec:2 258 | >=sys-apps/sed-4.0.5 259 | app-shells/bash:0[readline] 260 | >=app-admin/eselect-1.2 261 | elibc_glibc? ( >=sys-apps/sandbox-2.2 ) 262 | kernel_linux? ( sys-apps/util-linux ) 263 | >=app-misc/pax-utils-0.1.17" 264 | PDEPEND=" 265 | >=net-misc/rsync-2.6.4 266 | userland_GNU? ( >=sys-apps/coreutils-6.4 )" 267 | 268 | pkg_setup() { 269 | use epydoc && DISTUTILS_ALL_SUBPHASE_IMPLS=( python2.7 ) 270 | } 271 | 272 | python_compile_all() { 273 | if use epydoc; then 274 | esetup.py epydoc 275 | fi 276 | } 277 | 278 | Note that when the restriction is caused by dependencies rather than 279 | package's files, the any-r1 API described below is preferable to this. 280 | 281 | 282 | .. index:: python_gen_any_dep; python-r1 283 | .. index:: python_check_deps; python-r1 284 | 285 | Disjoint build dependencies (any-r1 API) 286 | ======================================== 287 | Some packages have disjoint sets of runtime and pure build-time 288 | dependencies. The former need to be built for all enabled 289 | implementations, the latter only for one of them. The any-r1 API 290 | in ``python-r1`` is specifically suited for expressing that. 291 | 292 | Let's consider an example package that uses Sphinx with a plugin 293 | to build documentation. Naturally, you're going to build the documents 294 | only once, not separately for every enabled target. 295 | 296 | 297 | Using regular python-r1 API 298 | --------------------------- 299 | If you were using the regular API, you'd have to use 300 | ``${PYTHON_USEDEP}`` on the dependencies. The resulting code could look 301 | like the following:: 302 | 303 | BDEPEND=" 304 | doc? ( 305 | dev-python/sphinx[${PYTHON_USEDEP}] 306 | dev-python/sphinx_rtd_theme[${PYTHON_USEDEP}] 307 | )" 308 | 309 | src_compile() { 310 | ... 311 | 312 | if use doc; then 313 | python_setup 314 | emake -C docs html 315 | fi 316 | } 317 | 318 | If your package is built with support for Python 3.6, 3.7 and 3.8, 319 | then this dependency string will enforce the same targets for Sphinx 320 | and the theme. However, in practice it will only be used through 321 | Python 3.8. Normally, this is not such a big deal. 322 | 323 | Now imagine your package supports Python 2.7 as well, while Sphinx 324 | does not anymore. This means that your package will force downgrade 325 | to the old version of ``dev-python/sphinx`` even though it will not 326 | be used via Python 2.7 at all. 327 | 328 | 329 | Using any-r1 API with python-r1 330 | ------------------------------- 331 | As the name suggests, the any-r1 API resembles the API used 332 | by ``python-any-r1`` eclass. The disjoint build-time dependencies 333 | are declared using ``python_gen_any_dep``, and need to be tested 334 | via ``python_check_deps()`` function. The presence of the latter 335 | function activates the alternate behavior of ``python_setup``. Instead 336 | of selecting one of the enabled targets, it will run it to verify 337 | installed dependencies and use one having all dependencies satisfied. 338 | 339 | .. code-block:: bash 340 | :emphasize-lines: 3-6,9-12,18 341 | 342 | BDEPEND=" 343 | doc? ( 344 | $(python_gen_any_dep ' 345 | dev-python/sphinx[${PYTHON_USEDEP}] 346 | dev-python/sphinx_rtd_theme[${PYTHON_USEDEP}] 347 | ') 348 | )" 349 | 350 | python_check_deps() { 351 | python_has_version "dev-python/sphinx[${PYTHON_USEDEP}]" && 352 | python_has_version "dev-python/sphinx_rtd_theme[${PYTHON_USEDEP}]" 353 | } 354 | 355 | src_compile() { 356 | ... 357 | 358 | if use doc; then 359 | python_setup 360 | emake -C docs html 361 | fi 362 | } 363 | 364 | Note that ``python_setup`` may select an implementation that is not even 365 | enabled via ``PYTHON_TARGETS``. The goal is to try hard to avoid 366 | requiring user to change USE flags on dependencies if possible. 367 | 368 | An interesting side effect of that is that the supported targets 369 | in the dependencies can be a subset of the one in package. For example, 370 | we have used this API to add Python 3.8 support to packages before 371 | ``dev-python/sphinx`` supported it — the eclass implicitly forced using 372 | another implementation for Sphinx. 373 | 374 | 375 | Different sets of build-time dependencies 376 | ----------------------------------------- 377 | Let's consider the case when Python is used at build-time for something 378 | else still. In that case, we want ``python_setup`` to work 379 | unconditionally but enforce dependencies only with ``doc`` flag enabled. 380 | 381 | .. code-block:: bash 382 | :emphasize-lines: 9-13,16 383 | 384 | BDEPEND=" 385 | doc? ( 386 | $(python_gen_any_dep ' 387 | dev-python/sphinx[${PYTHON_USEDEP}] 388 | dev-python/sphinx_rtd_theme[${PYTHON_USEDEP}] 389 | ') 390 | )" 391 | 392 | python_check_deps() { 393 | use doc || return 0 394 | python_has_version "dev-python/sphinx[${PYTHON_USEDEP}]" && 395 | python_has_version "dev-python/sphinx_rtd_theme[${PYTHON_USEDEP}]" 396 | } 397 | 398 | src_compile() { 399 | python_setup 400 | 401 | ... 402 | 403 | use doc && emake -C docs html 404 | } 405 | 406 | Note that ``python_setup`` behaves according to the any-r1 API here. 407 | While it will not enforce doc dependencies with ``doc`` flag disabled, 408 | it will use *any* interpreter that is supported and installed, even 409 | if it is not enabled explicitly in ``PYTHON_TARGETS``. 410 | 411 | 412 | Using any-r1 API with distutils-r1 413 | ---------------------------------- 414 | The alternate build dependency API also integrates with ``distutils-r1`` 415 | eclass. If ``python_check_deps()`` is declared, the ``python_*_all()`` 416 | sub-phase functions are called with the interpreter selected according 417 | to any-r1 rules. 418 | 419 | .. code-block:: bash 420 | :emphasize-lines: 3-6,9-13 421 | 422 | BDEPEND=" 423 | doc? ( 424 | $(python_gen_any_dep ' 425 | dev-python/sphinx[${PYTHON_USEDEP}] 426 | dev-python/sphinx_rtd_theme[${PYTHON_USEDEP}] 427 | ') 428 | )" 429 | 430 | python_check_deps() { 431 | use doc || return 0 432 | python_has_version "dev-python/sphinx[${PYTHON_USEDEP}]" && 433 | python_has_version "dev-python/sphinx_rtd_theme[${PYTHON_USEDEP}]" 434 | } 435 | 436 | python_compile_all() { 437 | use doc && emake -C docs html 438 | } 439 | 440 | Note that ``distutils-r1`` calls ``python_setup`` unconditionally, 441 | therefore ``python_check_deps()`` needs to account for that. 442 | 443 | Normally you won't have to use this API for Sphinx though — 444 | ``distutils_enable_sphinx`` does precisely that for you. 445 | 446 | 447 | Combining any-r1 API with implementation restrictions 448 | ===================================================== 449 | Both APIs described above can be combined. This can be used when 450 | build-time scripts support a subset of implementations supported 451 | by the package itself, and by its build-time dependencies. For example, 452 | if the package uses ``dev-util/scons`` build system with ``SConstruct`` 453 | files using Python 2 construct. 454 | 455 | There are two approaches to achieve that: either the build-time 456 | implementation list needs to be passed to ``python_setup``, 457 | or ``python_check_deps`` needs to explicitly reject unsupported targets. 458 | In both cases, a matching implementation list needs to be passed 459 | to ``python_gen_any_dep``. 460 | 461 | .. code-block:: bash 462 | :emphasize-lines: 25,28-30,46 463 | 464 | # Copyright 1999-2020 Gentoo Authors 465 | # Distributed under the terms of the GNU General Public License v2 466 | 467 | EAPI=7 468 | 469 | PYTHON_COMPAT=( python2_7 python3_6 ) 470 | inherit python-r1 toolchain-funcs 471 | 472 | DESCRIPTION="GPS daemon and library for USB/serial GPS devices and GPS/mapping clients" 473 | HOMEPAGE="https://gpsd.gitlab.io/gpsd/" 474 | SRC_URI="mirror://nongnu/${PN}/${P}.tar.gz" 475 | 476 | LICENSE="BSD" 477 | SLOT="0/24" 478 | KEYWORDS="~amd64 ~arm ~ppc ~ppc64 ~sparc ~x86" 479 | 480 | IUSE="python" 481 | REQUIRED_USE=" 482 | python? ( ${PYTHON_REQUIRED_USE} )" 483 | 484 | RDEPEND=" 485 | >=net-misc/pps-tools-0.0.20120407 486 | python? ( ${PYTHON_DEPS} )" 487 | DEPEND="${RDEPEND} 488 | $(python_gen_any_dep '>=dev-util/scons-2.3.0[${PYTHON_USEDEP}]' -2) 489 | virtual/pkgconfig" 490 | 491 | python_check_deps() { 492 | python_has_version ">=dev-util/scons-2.3.0[${PYTHON_USEDEP}]" 493 | } 494 | 495 | src_configure() { 496 | myesconsargs=( 497 | prefix="${EPREFIX}/usr" 498 | libdir="\$prefix/$(get_libdir)" 499 | udevdir="$(get_udevdir)" 500 | chrpath=False 501 | gpsd_user=gpsd 502 | gpsd_group=uucp 503 | nostrip=True 504 | manbuild=False 505 | $(use_scons python) 506 | ) 507 | 508 | # SConstruct uses py2 constructs 509 | python_setup -2 510 | } 511 | 512 | .. code-block:: bash 513 | :emphasize-lines: 25,28-31,46 514 | 515 | # Copyright 1999-2020 Gentoo Authors 516 | # Distributed under the terms of the GNU General Public License v2 517 | 518 | EAPI=7 519 | 520 | PYTHON_COMPAT=( python2_7 python3_6 ) 521 | inherit python-r1 toolchain-funcs 522 | 523 | DESCRIPTION="GPS daemon and library for USB/serial GPS devices and GPS/mapping clients" 524 | HOMEPAGE="https://gpsd.gitlab.io/gpsd/" 525 | SRC_URI="mirror://nongnu/${PN}/${P}.tar.gz" 526 | 527 | LICENSE="BSD" 528 | SLOT="0/24" 529 | KEYWORDS="~amd64 ~arm ~ppc ~ppc64 ~sparc ~x86" 530 | 531 | IUSE="python" 532 | REQUIRED_USE=" 533 | python? ( ${PYTHON_REQUIRED_USE} )" 534 | 535 | RDEPEND=" 536 | >=net-misc/pps-tools-0.0.20120407 537 | python? ( ${PYTHON_DEPS} )" 538 | DEPEND="${RDEPEND} 539 | $(python_gen_any_dep '>=dev-util/scons-2.3.0[${PYTHON_USEDEP}]' -2) 540 | virtual/pkgconfig" 541 | 542 | python_check_deps() { 543 | python_is_python3 && return 1 544 | python_has_version ">=dev-util/scons-2.3.0[${PYTHON_USEDEP}]" 545 | } 546 | 547 | src_configure() { 548 | myesconsargs=( 549 | prefix="${EPREFIX}/usr" 550 | libdir="\$prefix/$(get_libdir)" 551 | udevdir="$(get_udevdir)" 552 | chrpath=False 553 | gpsd_user=gpsd 554 | gpsd_group=uucp 555 | nostrip=True 556 | manbuild=False 557 | $(use_scons python) 558 | ) 559 | 560 | python_setup 561 | } 562 | -------------------------------------------------------------------------------- /helper.rst: -------------------------------------------------------------------------------- 1 | ======================= 2 | Common helper functions 3 | ======================= 4 | 5 | The functions described in this chapter are common to all three basic 6 | eclasses. To facilitate code reuse, they are declared 7 | in ``python-utils-r1.eclass``. However, you should not inherit this 8 | eclass directly and instead assume the functions are provided as part 9 | of the API of other eclasses. 10 | 11 | Eclass reference: `python-utils-r1.eclass(5)`_ 12 | 13 | 14 | .. index:: python_doexe 15 | .. index:: python_newexe 16 | .. index:: python_doscript 17 | .. index:: python_newscript 18 | .. index:: python_domodule 19 | .. index:: python_doheaders 20 | .. index:: python_scriptinto 21 | .. index:: python_moduleinto 22 | 23 | Install helpers 24 | =============== 25 | The install helpers are provided commonly for ``python-single-r1`` 26 | and ``python-r1`` eclasses. Their main purpose is to facilitate 27 | installing Python scripts, modules and extensions whenever the package 28 | lacks a build system or the build system is not suited for installing 29 | them. 30 | 31 | The API is consistent with the standard ``do*``, ``new*`` and ``*into`` 32 | helpers. There are four kinds of functions provided: 33 | 34 | 1. ``python_doexe`` and ``python_newexe`` that install executables 35 | wrapping them via python-exec, 36 | 2. ``python_doscript`` and ``python_newscript`` that install Python 37 | scripts, updating the shebangs and wrapping them via python-exec, 38 | 3. ``python_domodule`` that installs Python modules, or recursively 39 | installs packages (directories), 40 | 4. ``python_doheader`` that installs header files to Python-specific 41 | include directory. 42 | 43 | The install path for executables and scripts (1. and 2.) can be adjusted 44 | by calling ``python_scriptinto``. Note that this actually affects only 45 | the wrapper symlink install path; the actual scripts will be installed 46 | in the standard python-exec script directories. This also implies that 47 | no two executables can have the same name, even if final directory is 48 | different. The default install path is ``/usr/bin``. 49 | 50 | The install path for modules and packages (3.) can be adjusted 51 | by calling ``python_moduleinto``. This function accepts either absolute 52 | path or Python parent module name that causes modules to be installed 53 | in an appropriate subdirectory of the site-packages directory. 54 | The default install path is top-level site-packages (equivalent 55 | to ``python_moduleinto .``). 56 | 57 | The install path for headers (4.) cannot be adjusted. 58 | 59 | All the helpers can be used in ``src_install`` phase, in which case 60 | they install the files onto ``${D}``. Additionally, ``python_domodule`` 61 | can be use in other ebuild phases, in which case it installs 62 | into ``${BUILD_DIR}/install``. This provides improved integration 63 | with distutils-r1 eclass. 64 | 65 | ``python_doexe`` is generally used to install executables that reference 66 | Python but are not Python scripts. This could be e.g. a bash script 67 | that calls Python:: 68 | 69 | make_wrapper "${PN}.tmp" "${EPYTHON} $(python_get_sitedir)/${PN}/cropgtk.py" 70 | python_newexe "${ED%/}/usr/bin/${PN}.tmp" "${PN}" 71 | rm "${ED%/}/usr/bin/${PN}.tmp" || die 72 | 73 | Note that you need to ensure that the executable calls correct Python 74 | interpreter itself. 75 | 76 | ``python_doscript`` is generally used to install Python scripts 77 | to binary directories:: 78 | 79 | python_scriptinto /usr/sbin 80 | python_newscript pynslcd.py pynslcd 81 | 82 | It takes care of updating the shebang for you. 83 | 84 | ``python_domodule`` is used to install Python modules, extensions, 85 | packages, data files and in general anything that lands in site-packages 86 | directory:: 87 | 88 | python_moduleinto ${PN} 89 | python_domodule images application ${MY_PN}.py \ 90 | AUTHORS CHANGES COPYING DEPENDS TODO __init__.py 91 | 92 | It is roughly equivalent to ``doins -r``, except that it byte-compiles 93 | all Python modules found inside it. 94 | 95 | ``python_doheader`` is used in the very rare cases when Python packages 96 | install additional header files that are used to compile other 97 | extensions:: 98 | 99 | python_doheader src/libImaging/*.h 100 | 101 | 102 | .. index:: python_fix_shebang 103 | 104 | Fixing shebangs on installed scripts 105 | ==================================== 106 | If upstream build system installs Python scripts, it should also update 107 | their shebangs to match the interpreter used for install. Otherwise, 108 | the scripts could end up being run via another implementation, one 109 | that possible does not have the necessary dependencies installed. 110 | An example of correct shebang is:: 111 | 112 | #!/usr/bin/env python3.8 113 | 114 | However, if the build system installs a script with ``python3`` or even 115 | ``python`` shebang, it needs to be updated. The ``python_fix_shebang`` 116 | function is provided precisely for that purpose. It can be used to 117 | update the shebang on an installed file:: 118 | 119 | src_install() { 120 | default 121 | python_fix_shebang "${D}"/usr/bin/sphinxtrain 122 | } 123 | 124 | It can also be used in working directory to update a script that's used 125 | at build time or before it is installed:: 126 | 127 | src_prepare() { 128 | default 129 | python_fix_shebang openvpn-vulnkey 130 | } 131 | 132 | Finally, it can also be used on a directory to recursively update 133 | shebangs in all Python scripts found inside it:: 134 | 135 | src_install() { 136 | insinto /usr 137 | doins -r linux-package/* 138 | dobin linux-package/bin/kitty 139 | python_fix_shebang "${ED}" 140 | } 141 | 142 | Normally, ``python_fix_shebang`` errors out when the target interpreter 143 | is not compatible with the original shebang, e.g. when you are trying 144 | to install a script with ``python2`` shebang for Python 3. ``-f`` 145 | (force) switch can be used to override that:: 146 | 147 | src_prepare() { 148 | default 149 | python_fix_shebang -f "${PN}.py" 150 | } 151 | 152 | 153 | .. index:: python_optimize 154 | 155 | Byte-compiling Python modules 156 | ============================= 157 | Python modules are byte compiled in order to speed up their loading. 158 | Byte-compilation is normally done by the build system when the modules 159 | are installed. However, sometimes packages fail to compile them 160 | entirely, or byte-compile them only partially. Nowadays, QA checks 161 | detect and report that: 162 | 163 | .. code-block:: text 164 | 165 | * This package installs one or more Python modules that are not byte-compiled. 166 | * The following files are missing: 167 | * 168 | * /usr/lib/pypy2.7/site-packages/_feedparser_sgmllib.pyc 169 | * /usr/lib64/python2.7/site-packages/_feedparser_sgmllib.pyc 170 | * /usr/lib64/python2.7/site-packages/_feedparser_sgmllib.pyo 171 | * 172 | * Please either fix the upstream build system to byte-compile Python modules 173 | * correctly, or call python_optimize after installing them. For more 174 | * information, see: 175 | * https://wiki.gentoo.org/wiki/Project:Python/Byte_compiling 176 | 177 | The eclass provides a ``python_optimize`` function to byte-compile 178 | modules. The most common way of using it is to call it after installing 179 | the package to byte-compile all modules installed into site-packages:: 180 | 181 | src_install() { 182 | cmake_src_install 183 | python_optimize 184 | } 185 | 186 | If Python scripts are installed to a non-standard directory, the path 187 | to them can be passed to the function:: 188 | 189 | src_install() { 190 | cd "${S}"/client || die 191 | emake DESTDIR="${D}" LIBDIR="usr/lib" install 192 | python_optimize "${D}/usr/lib/entropy/client" 193 | } 194 | 195 | 196 | .. index:: python_get_sitedir 197 | .. index:: python_get_includedir 198 | .. index:: python_get_scriptdir 199 | .. index:: python_get_library_path 200 | .. index:: python_get_CFLAGS 201 | .. index:: python_get_LIBS 202 | .. index:: python_get_PYTHON_CONFIG 203 | .. index:: python_get_stdlib 204 | 205 | Querying the implementation information 206 | ======================================= 207 | Most of the time, various build systems manage to detect and query 208 | the Python implementation correctly for necessary build details. 209 | Ocassionally, you need to provide those values or override bad detection 210 | results. For this purpose, the eclasses provide a series of *getters*. 211 | 212 | The following generic getters are provided: 213 | 214 | - ``python_get_sitedir`` that outputs the absolute path to the target's 215 | site-packages directory (where Python modules are installed). 216 | 217 | - ``python_get_includedir`` that outputs the absolute path 218 | to the target-specific header directory. 219 | 220 | - ``python_get_scriptdir`` that outputs the absolute path 221 | to the python-exec script directory for the implementation. 222 | 223 | - ``python_get_stdlib`` that outputs the absolute path to the target's 224 | standard library directory (where standard modules can be found). 225 | 226 | The following getters are provided only for CPython targets: 227 | 228 | - ``python_get_library_path`` that outputs the absolute path 229 | to the ``python`` library. 230 | 231 | - ``python_get_CFLAGS`` that outputs the C preprocessor flags 232 | for linking against the Python library (equivalent to ``pkg-config 233 | --cflags ...``). 234 | 235 | - ``python_get_LIBS`` that outputs the linker flags for linking 236 | against the Python library (equivalent to ``pkg-config --libs ...``). 237 | 238 | - ``python_get_PYTHON_CONFIG`` that outputs the absolute path 239 | to the ``python-config`` executable. 240 | 241 | Note that all paths provided by getters include the offset-prefix 242 | (``${EPREFIX}``) already and they are not suitable to passing 243 | to ``*into`` helpers. If you need to install something, use `install 244 | helpers`_ instead. 245 | 246 | .. code-block:: bash 247 | 248 | src_configure() { 249 | local mycmakeargs=( 250 | ... 251 | ) 252 | use python && mycmakeargs+=( 253 | -DPYTHON_DEST="$(python_get_sitedir)" 254 | -DPYTHON_EXECUTABLE="${PYTHON}" 255 | -DPYTHON_INCLUDE_DIR="$(python_get_includedir)" 256 | -DPYTHON_LIBRARY="$(python_get_library_path)" 257 | ) 258 | 259 | cmake_src_configure 260 | } 261 | 262 | 263 | .. code-block:: bash 264 | 265 | python_test() { 266 | # prepare embedded executable 267 | emake \ 268 | CC="$(tc-getCC)" \ 269 | PYINC="$(python_get_CFLAGS)" \ 270 | PYLIB="$(python_get_LIBS)" \ 271 | check 272 | } 273 | 274 | 275 | .. _python-utils-r1.eclass(5): 276 | https://devmanual.gentoo.org/eclass-reference/python-utils-r1.eclass/index.html 277 | -------------------------------------------------------------------------------- /index.rst: -------------------------------------------------------------------------------- 1 | =================== 2 | Gentoo Python Guide 3 | =================== 4 | :Author: Michał Górny 5 | :License: `Creative Commons Attribution 4.0 International`_ 6 | 7 | .. toctree:: 8 | :maxdepth: 2 9 | :caption: Contents: 10 | 11 | preface 12 | interpreter 13 | eclass 14 | basic 15 | any 16 | single 17 | multi 18 | distutils 19 | test 20 | distutils-legacy 21 | pypi 22 | helper 23 | depend 24 | pytest 25 | concept 26 | expert-multi 27 | buildsys 28 | porting 29 | migration 30 | qawarn 31 | package-maintenance 32 | interpreter-maintenance 33 | 34 | 35 | Indices and tables 36 | ================== 37 | 38 | * :ref:`genindex` 39 | * :ref:`search` 40 | 41 | .. _Creative Commons Attribution 4.0 International: 42 | https://creativecommons.org/licenses/by/4.0/ 43 | -------------------------------------------------------------------------------- /interpreter-maintenance.rst: -------------------------------------------------------------------------------- 1 | ===================================== 2 | Maintenance of Python implementations 3 | ===================================== 4 | 5 | Notes specific to Python interpreters 6 | ===================================== 7 | CPython patchsets 8 | ----------------- 9 | Gentoo is maintaining patchsets for all CPython versions. These include 10 | some non-upstreamable Gentoo patches and upstream backports. While it 11 | is considered acceptable to add a new patch (e.g. a security bug fix) 12 | to ``files/`` directory, it should be eventually moved into 13 | the respective patchset. 14 | 15 | When adding a new version, it is fine to use an old patchset if it 16 | applies cleanly. If it does not, you should regenerate the patchset 17 | for new version. 18 | 19 | The origin for Gentoo patches are the ``gentoo-*`` tags the `Gentoo fork 20 | of CPython repository`_. The recommended workflow is to clone 21 | the upstream repository, then add Gentoo fork as a remote, e.g.:: 22 | 23 | git clone https://github.com/python/cpython 24 | cd cpython 25 | git remote add gentoo git@git.gentoo.org:fork/cpython.git 26 | git fetch --tags gentoo 27 | 28 | In order to rebase the patchset, check out the tag corresponding 29 | to the previous patchset version and rebase it against the upstream 30 | release tag:: 31 | 32 | git checkout gentoo-3.7.4 33 | git rebase v3.7.6 34 | 35 | You may also add additional changes via ``git cherry-pick``. Once 36 | the new patches are ready, create the tarball and upload it, then 37 | create the tag and push it:: 38 | 39 | mkdir python-gentoo-patches-3.7.6 40 | cd python-gentoo-patches-3.7.6 41 | git format-patch v3.7.6 42 | cd .. 43 | tar -cf python-gentoo-patches-3.7.6.tar python-gentoo-patches-3.7.6 44 | xz -9 python-gentoo-patches-3.7.6.tar 45 | scp python-gentoo-patches-3.7.6.tar.xz ... 46 | git tag gentoo-3.7.6 47 | git push --tags gentoo 48 | 49 | 50 | PyPy 51 | ---- 52 | Due to high resource requirements and long build time, PyPy on Gentoo 53 | is provided both in source and precompiled form. This creates a bit 54 | unusual ebuild structure (the package names given here are for 55 | the newest approach, used with PyPy3.11): 56 | 57 | - ``dev-lang/pypy3-exe`` provides the PyPy executable and generated 58 | files built from source, 59 | - ``dev-lang/pypy3-exe-bin`` does the same in precompiled binary form, 60 | - ``dev-lang/pypy`` combines the above with the common files. This 61 | is the package that satisfies the PyPy targets. 62 | 63 | When bumping PyPy, ``pypy3-exe`` needs to be updated first. Then it 64 | should be used to build a binary package and bump ``pypy3-exe-bin``. 65 | Technically, ``pypy`` can be bumped after ``pypy3-exe`` and used to test 66 | it but it should not be pushed before ``pypy3-exe-bin`` is ready, as it 67 | would force all users to switch to source form implicitly. 68 | 69 | The binary packages are built using Docker_ nowadays, using 70 | binpkg-docker_ scripts. To produce them, create a ``local.diff`` 71 | containing changes related to PyPy bump and run ``${arch}-pypy${ver}``. 72 | For example:: 73 | 74 | git clone https://github.com/mgorny/binpkg-docker 75 | cd binpkg-docker 76 | (cd ~/git/gentoo && git diff origin) > local.diff 77 | make {amd64,arm64,ppc64le,x86}{,-musl}-pypy{,3_10,3_11} 78 | 79 | The resulting binary packages will be placed in your home directory, 80 | in ``~/binpkg/${arch}/pypy``. Upload them and use them to bump 81 | ``pypy*-exe-bin``. 82 | 83 | 84 | Adding a new Python implementation 85 | ================================== 86 | Eclass and profile changes 87 | -------------------------- 88 | 89 | .. Note:: 90 | 91 | The best time to perform these changes is prior to adding the first 92 | beta of new Python version. This is because ``dev-lang/python`` 93 | has a conditional dependency on ``dev-lang/python-exec`` 94 | for non-alpha versions, so adding the target prior to committing it 95 | ensures depgraph consistency. 96 | 97 | When adding a new Python target, please remember to perform all 98 | the following tasks: 99 | 100 | - add the new target flags to ``profiles/desc/python_targets.desc`` 101 | and ``python_single_target.desc``. 102 | 103 | - force the new implementation on ``dev-lang/python-exec`` 104 | via ``profiles/base/package.use.force``. 105 | 106 | - mask the new target flags on stable profiles 107 | via ``profiles/base/use.stable.mask``. 108 | 109 | - update ``python-utils-r1.eclass``: 110 | 111 | 1. add the implementation to ``_PYTHON_ALL_IMPLS`` 112 | 113 | 2. update the patterns in ``_python_verify_patterns`` 114 | 115 | 3. update the patterns in ``_python_set_impls`` 116 | 117 | 4. update the patterns in ``_python_impl_matches`` 118 | 119 | 5. add the appropriate dependency to the case for ``PYTHON_PKG_DEP`` 120 | 121 | - update the tested version range in ``eclass/tests/python-utils-r1.sh`` 122 | 123 | - add the new implementation to the list 124 | in ``app-portage/gpyutils/files/implementations.txt``. 125 | 126 | - update the `implementation tables`_ on Gentoo wiki 127 | 128 | 129 | Porting initial packages 130 | ------------------------ 131 | The initial porting is quite hard due to a number of circular 132 | dependencies. To ease the process, it is recommended to temporarily 133 | limit testing of the packages that feature many additional test 134 | dependencies. The packages needing this have implementation conditions 135 | in place already. An example follows: 136 | 137 | .. code-block:: bash 138 | :emphasize-lines: 6,18,23 139 | 140 | PYTHON_TESTED=( python3_{8..10} pypy3 ) 141 | PYTHON_COMPAT=( "${PYTHON_TESTED[@]}" python3_11 ) 142 | 143 | BDEPEND=" 144 | test? ( 145 | $(python_gen_cond_dep ' 146 | dev-python/jaraco-envs[${PYTHON_USEDEP}] 147 | >=dev-python/jaraco-path-3.2.0[${PYTHON_USEDEP}] 148 | dev-python/mock[${PYTHON_USEDEP}] 149 | dev-python/pip[${PYTHON_USEDEP}] 150 | dev-python/sphinx[${PYTHON_USEDEP}] 151 | dev-python/pytest[${PYTHON_USEDEP}] 152 | dev-python/pytest-fixture-config[${PYTHON_USEDEP}] 153 | dev-python/pytest-virtualenv[${PYTHON_USEDEP}] 154 | dev-python/pytest-xdist[${PYTHON_USEDEP}] 155 | >=dev-python/virtualenv-20[${PYTHON_USEDEP}] 156 | dev-python/wheel[${PYTHON_USEDEP}] 157 | ' "${PYTHON_TESTED[@]}") 158 | ) 159 | " 160 | 161 | python_test() { 162 | has "${EPYTHON}" "${PYTHON_TESTED[@]/_/.}" || continue 163 | 164 | HOME="${PWD}" epytest setuptools 165 | } 166 | 167 | It is important to remember to update the implementation range 168 | and therefore enable testing once the test dependencies are ported. 169 | Please do not remove the conditions entirely, as they will be useful 170 | for the next porting round. 171 | 172 | If only a non-significant subset of test dependencies is a problem, 173 | it is better to make these dependencies conditional and run 174 | the remainder of the test suite. If tests are not skipped automatically 175 | due to missing dependencies, using ``has_version`` to skip them 176 | conditionally is preferred over hardcoding version ranges, e.g.: 177 | 178 | .. code-block:: bash 179 | :emphasize-lines: 3-6,12 180 | 181 | BDEPEND=" 182 | test? ( 183 | $(python_gen_cond_dep ' 184 | dev-python/pydantic[${PYTHON_USEDEP}] 185 | ' pypy3 python3_{8..10} # TODO: python3_11 186 | ) 187 | ) 188 | " 189 | 190 | python_test() { 191 | local EPYTEST_DESELECT=() 192 | if ! has_version "dev-python/pydantic[${PYTHON_USEDEP}]"; then 193 | EPYTEST_DESELECT+=( 194 | tests/test_comparison.py::test_close_to_now_{false,true} 195 | ) 196 | fi 197 | epytest 198 | } 199 | 200 | During the initial testing it is acceptable to be more lenient on test 201 | failures, and deselect failing tests on the new implementation when 202 | the package itself works correctly for its reverse dependencies. 203 | For example, during Python 3.11 porting we have deselected a few failing 204 | tests on ``dev-python/attrs`` to unblock porting ``dev-python/pytest``. 205 | Porting pytest in order to enable testing packages was far more 206 | important than getting 100% passing tests on ``dev-python/attrs``. 207 | 208 | The modern recommendation for the porting process is to focus 209 | on ``dev-python/pytest`` as the first goal. It is the most common test 210 | dependency for Python packages, and porting it makes it possible to 211 | start testing packages early. The initial ported package set should 212 | include all dependencies of pytest, except for test dependencies 213 | of the package with large test dependency graphs (``dev-python/pytest`` 214 | itself, ``dev-python/setuptools``). This amounts to around 40 packages. 215 | 216 | Note that emerging the initial set requires installing 217 | ``dev-python/pytest`` with ``USE=-test`` first. Once it is installed, 218 | the previously installed dependencies should be reinstalled with tests 219 | enabled. 220 | 221 | After pushing the initial batch, the next recommended goal 222 | is ``dev-python/urllib3``. It should be followed by focusing 223 | on reenabling tests in the packages where they were skipped. 224 | 225 | 226 | Removing a Python implementation 227 | ================================ 228 | Preparation 229 | ----------- 230 | The obsolescence of the implementation needs to be announced on mailing 231 | lists, along with request to proceed with porting packages to a newer 232 | implementation. The package lists found on `QA reports`_ can be used 233 | to track the porting progress. 234 | 235 | As the time for removal approaches, the packages showing no signs 236 | of porting effort (except for backports, i.e. packages dedicated 237 | to the old implementation and used only conditionally to it) need to 238 | be masked for removal. 239 | 240 | 241 | Eclass and profile changes 242 | -------------------------- 243 | Rather than being removed entirely, old targets are implicitly disabled 244 | via the eclass. This ensures that old entries in ``PYTHON_COMPAT`` 245 | do not trigger errors. 246 | 247 | When disabling an old target, please remember to: 248 | 249 | - remove the ``dev-lang/python-exec`` dependency from the interpreter 250 | ebuilds (with a revision bump) 251 | 252 | - mask and remove the remaining backport packages immediately (they 253 | would fail to source once the only listed implementation is disabled 254 | but the mask should remain to hint users at the removal) 255 | 256 | - update the tested version range in ``eclass/tests/python-utils-r1.sh`` 257 | 258 | - update ``python-utils-r1.eclass``: 259 | 260 | 1. move the implementation from ``_PYTHON_ALL_IMPLS`` 261 | to ``_PYTHON_HISTORICAL_IMPLS`` 262 | 263 | 2. update the patterns in ``_python_set_impls`` 264 | 265 | 3. remove the respective case for ``PYTHON_PKG_DEP`` 266 | 267 | - remove the use of implementation flags from ``profiles/`` tree 268 | 269 | - remove the flags from ``profiles/desc/python_targets.desc`` 270 | and ``profiles/desc/python_single_target.desc`` 271 | 272 | - set the implementation status to ``dead`` 273 | in ``app-portage/gpyutils/files/implementations.txt`` 274 | 275 | - update the `implementation tables`_ on Gentoo wiki 276 | 277 | 278 | Python build system bootstrap 279 | ============================= 280 | Python build systems are often facing the bootstrap problem — that is, 281 | the build system itself has some dependencies, while these dependencies 282 | require the same build system to build. The common upstream way 283 | (actually recommended in `PEP 517 build requirements`_ section) is 284 | to bundle the necessary dependencies as part of the build system. 285 | However, this goes against best Gentoo practices. 286 | 287 | The current Gentoo practice for bootstrap with dependency unbundling 288 | is to: 289 | 290 | 1. Install the dependencies of flit_core and the eclass PEP 517 logic 291 | (installer, tomli) manually using ``python_domodule``. 292 | 293 | 2. Install flit_core using the standalone PEP 517 backend. 294 | 295 | 3. Install the dependencies of setuptools using flit (writing trivial 296 | ``pyproject.toml`` within the ebuild if necessary). 297 | 298 | 4. Install setuptools using the standalone PEP 517 backend. 299 | 300 | 5. The dependencies of other build systems can be installed using 301 | flit, setuptools or other previously unbundled build systems. 302 | 303 | Note that for the purpose of bootstrap only obligatory baseline 304 | dependencies are considered significant. Non-obligatory dependencies 305 | (i.e. ones that can be missing during the bootstrap process) can be 306 | placed in ``PDEPEND``. Test suite dependencies can include cycles 307 | with the package itself — running tests is not considered obligatory 308 | during the bootstrap process. 309 | 310 | flit_core has been chosen as the base build system for unbundling since 311 | it has the fewest external dependencies (i.e. only depends on tomli). 312 | Its author indicates in the `flit_core vendoring README`_ that no other 313 | dependencies will be added or vendored into flit_core. 314 | 315 | 316 | .. _Gentoo fork of CPython repository: 317 | https://gitweb.gentoo.org/fork/cpython.git/ 318 | .. _Docker: https://www.docker.com/ 319 | .. _binpkg-docker: https://github.com/mgorny/binpkg-docker 320 | .. _implementation tables: 321 | https://wiki.gentoo.org/wiki/Project:Python/Implementations 322 | .. _QA reports: https://qa-reports.gentoo.org/ 323 | .. _PEP 517 build requirements: 324 | https://www.python.org/dev/peps/pep-0517/#build-requirements 325 | .. _flit_core vendoring README: 326 | https://github.com/pypa/flit/blob/main/flit_core/flit_core/vendor/README 327 | -------------------------------------------------------------------------------- /interpreter.rst: -------------------------------------------------------------------------------- 1 | =================== 2 | Python interpreters 3 | =================== 4 | 5 | Versions of Python 6 | ================== 7 | By a *version of Python* we usually mean the variant of Python language 8 | and standard library interface as used by a specific version 9 | of CPython_, the reference implementation of Python. 10 | 11 | Python versions are determined from the two first version components. 12 | The major version is incremented when major incompatible changes are 13 | introduced in the language, as was the case in Python 3. Along with 14 | minor version changes, the new releases introduce new features 15 | and remove deprecated APIs. The Python documentation generally 16 | indicates when a particular API was added or deprecated, and when it 17 | is planned to be removed. 18 | 19 | Practically speaking, this means that a program written purely 20 | for Python 2 is unlikely to work on Python 3, and requires major changes 21 | to achieve compatibility. On the other hand, a program written for 22 | Python 3.7 is very likely to work with Python 3.8, and reasonably likely 23 | to support Python 3.6 as well. If that is not the case, minor changes 24 | are usually sufficient to fix that. 25 | 26 | For example, Python 3.7 introduced a new `importlib.resources`_ module. 27 | If your program uses it, it will not work on Python 3.6 without 28 | a backwards compatibility code. 29 | 30 | Python 3.8 removed the deprecated `platform.linux_distribution()`_ 31 | function. If your program used it, it will not work on Python 3.8 32 | without changes. However, it was deprecated since Python 3.5, so if you 33 | were targetting 3.7, you should not have been using it in the first 34 | place. 35 | 36 | Gentoo supports building packages against Python 2.7 and a shifting 37 | window of 3-4 versions of Python 3. They are provided as slots 38 | of ``dev-lang/python``. 39 | 40 | 41 | Life cycle of a Python implementation 42 | ===================================== 43 | Every Python implementation (understood as a potential target) in Gentoo 44 | follows roughly the following life cycle: 45 | 46 | 1. The interpreter is added to ``~arch`` for initial testing. At this 47 | point, packages can not declare support for the implementation yet. 48 | 49 | New CPython releases enter this stage when upstream releases 50 | the first alpha version. Since the feature set is not stable yet, 51 | it is premature to declare the package compatibility with these 52 | versions. 53 | 54 | 2. The new Python target is added. It is initially stable-masked, 55 | so only ``~arch`` users can use it. At this point, packages start 56 | being tested against the new target and its support starts being 57 | declared in ``PYTHON_COMPAT``. 58 | 59 | CPython releases enter this stage when the first beta release 60 | is made. These versions are not fully stable yet either and package 61 | regressions do happen between beta releases but they are stable 62 | enough for initial testing. 63 | 64 | 3. When ready, the new interpreter is stabilized. The target is not yet 65 | available for stable users, though. 66 | 67 | CPython releases enter this stage roughly 30 days after the first 68 | stable release (i.e. X.Y.0 final) is made. This is also the stage 69 | where PyPy3 releases are in Gentoo for the time being. 70 | 71 | 4. The stable-mask for the target is removed. For this to happen, 72 | the inconsistencies in stable graph need to be addressed first 73 | via stabilizing newer versions of packages. 74 | 75 | CPython releases enter this stage after the interpreter is marked 76 | stable on all architectures and all the packages needed to clear 77 | the stable depenendency graph are stable as well. 78 | 79 | 5. Over time, developers are repeatedly asked to push testing packages 80 | for the new target forward and stabilize new versions supporting it. 81 | Eventually, the final push for updates happens and packages 82 | not supporting the new target start being removed. 83 | 84 | 6. If applicable, the new target becomes the default. The developers 85 | are required to test new packages against it. The support for old 86 | target is slowly being discontinued. 87 | 88 | Currently, we try to plan the switch to the next CPython version 89 | around June — July every year, to make these events more predictable 90 | to Gentoo users. 91 | 92 | 7. Eventually, the target becomes replaced by the next one. When it 93 | nears end of life, the final packages requiring it are masked for 94 | removal and the target flags are disabled. 95 | 96 | We generally aim to preserve support for old targets for as long 97 | as they are found still needed by Gentoo users. However, as more 98 | upstream packages remove support for older versions of Python, 99 | the cost of preserving the support becomes too great. 100 | 101 | 8. The compatibility declarations are cleaned up from ``PYTHON_COMPAT``, 102 | and obsolete ebuild and eclass code is cleaned up. 103 | 104 | 9. Finally, the interpreter is removed when it becomes no longer 105 | feasible to maintain it (usually because of the cost of fixing 106 | vulnerabilities or build failures). 107 | 108 | 109 | Stability guarantees of Python implementations 110 | ============================================== 111 | The language and standard library API of every Python version is 112 | expected to be stable since the first beta release of the matching 113 | CPython version. However, historically there were cases of breaking 114 | changes prior to a final release (e.g. the revert of ``enum`` changes 115 | in Python 3.10), as well as across minor releases (e.g. ``urlsplit()`` 116 | URL sanitization / security fix). 117 | 118 | The ABI of every CPython version is considered stable across bugfix 119 | releases since the first RC. This includes the public ABI of libpython, 120 | C extensions and compiled Python modules. Prior to the first RC, 121 | breaking changes to either may still happen. Gentoo currently does not 122 | account for these changes to the high cost of using slot operators, 123 | and therefore users using ``~arch`` CPython may have to occasionally 124 | rebuild Python packages manually. 125 | 126 | Additionally, modern versions of CPython declare so-called 'stable ABI' 127 | that remains forward compatible across Python versions. This permits 128 | upstreams to release wheels that can be used with multiple CPython 129 | versions (contrary to the usual case of building wheels separately 130 | for each version). However, this does not affect Gentoo packaging 131 | at the moment. 132 | 133 | PyPy does not hold specific ABI stability guarantees. Gentoo packages 134 | use subslots to declare the current ABI version, and the eclasses use 135 | slot operators in dependencies to enforce rebuilds whenever the ABI 136 | version changes. Fortunately, this has not happened recently. 137 | 138 | 139 | Alternative Python implementations 140 | ================================== 141 | CPython is the reference and most commonly used Python implementation. 142 | However, there are other interpreters that aim to maintain reasonable 143 | compatibility with it. 144 | 145 | PyPy_ is an implementation of Python built using in-house RPython 146 | language, using a Just-in-Time compiler to achieve better performance 147 | (generally in long-running programs running a lot of Python code). 148 | It maintains quite good compatibility with CPython, except when programs 149 | rely on its implementation details or GC behavior. 150 | 151 | PyPy upstream provides PyPy variants compatible with Python 2.7 152 | and at least one version of Python 3. Gentoo supports building packages 153 | against the current version of PyPy3 (and the previous version during 154 | transition periods). The different versions are available as slots 155 | of ``dev-lang/pypy``. 156 | 157 | Jython_ is an implementation of Python written in Java. Besides being 158 | a stand-alone Python interpreter, it supports bidirectional interaction 159 | between Python and Java libraries. 160 | 161 | Jython development is very slow paced, and it is currently bound 162 | to Python 2.7. Gentoo does not provide Jython anymore. 163 | 164 | IronPython_ is an implementation of Python for the .NET framework. 165 | Alike Jython, it supports bidirectional interaction between Python 166 | and .NET Framework. It is currently bound to Python 2.7. It is not 167 | packaged in Gentoo. 168 | 169 | Brython_ is an implementation of Python 3 for client-side web 170 | programming (in JavaScript). It provides a subset of Python 3 standard 171 | library combined with access to DOM objects. 172 | 173 | MicroPython_ is an implementation of Python 3 aimed for microcontrollers 174 | and embedded environments. It aims to maintain some compatibility 175 | with CPython while providing stripped down standard library 176 | and additional modules to interface with hardware. It is packaged 177 | as ``dev-lang/micropython``. 178 | 179 | Tauthon_ is a fork of Python 2.7 that aims to backport new language 180 | features and standard library modules while preserving backwards 181 | compatibility with existing code. It is not packaged in Gentoo. 182 | 183 | 184 | Support for multiple implementations 185 | ==================================== 186 | The support for simultaneously using multiple Python implementations 187 | is implemented primarily through USE flags. The packages installing 188 | or using Python files define either ``PYTHON_TARGETS`` 189 | or ``PYTHON_SINGLE_TARGET`` flags that permit user to choose which 190 | implementations are used. 191 | 192 | Modules and extensions are installed separately for each interpreter, 193 | in its specific site-packages directory. This means that a package 194 | can run using a specific target correctly only if all its dependencies 195 | were also installed for the same implementation. This is enforced 196 | via USE dependencies. 197 | 198 | Additionally, ``dev-lang/python-exec`` provides a mechanism for 199 | installing multiple variants of each Python script simultaneously. This 200 | is necessary to support scripts that differ between Python versions 201 | (particularly between Python 2 and Python 3) but it is also used 202 | to prevent scripts from being called via unsupported interpreter 203 | (i.e. one that does not have its accompanying modules or dependencies 204 | installed). 205 | 206 | This also implies that all installed Python scripts must have their 207 | shebangs adjusted to use a specific Python interpreter (not ``python`` 208 | nor ``python3`` but e.g. ``python3.7``), and all other executables must 209 | also be modified to call specific version of Python directly. 210 | 211 | 212 | Implementation support policy 213 | ============================= 214 | Gentoo gives the following guarantees with regards to Python support: 215 | 216 | 1. We aim to provide support for new CPython targets as soon as that 217 | becomes possible, and switch the default to the new slot as soon 218 | as porting is considered reasonably complete. This usually happens 219 | around June, the next year after the first stable release on a given 220 | branch. 221 | 222 | 2. We will continue providing support for the previous CPython branch 223 | for at least one year after the next branch becomes stable. However, 224 | in reality this will probably extend until PyPy switches to the newer 225 | slot, as we aim to support parallel versions of PyPy and CPython. 226 | 227 | 3. We will continue providing interpreters (possibly without package 228 | target support) for as long as they are maintained upstream, i.e. 229 | until their end-of-life date. Afterwards, we may continue providing 230 | EOL interpreters for as long as patching them continues being 231 | feasible. 232 | 233 | 4. Experimental targets (e.g. freethreading CPython, PyPy) are provided 234 | on best-effort basis, with no support guarantees. 235 | 236 | 237 | Backports 238 | ========= 239 | A common method of improving compatibility with older versions of Python 240 | is to backport new standard library modules or features. Packages doing 241 | that are generally called *backports*. 242 | 243 | Ideally, backports copy the code from the standard library with minimal 244 | changes, and provide a matching API. In some cases, new versions 245 | of backports are released as the standard library changes, and their 246 | usability extends from providing a missing module to extending older 247 | version of the module. For example, the ``dev-python/funcsigs`` package 248 | originally backported function signatures from Python 3.3 to older 249 | versions, and afterwards was updated to backport new features from 250 | Python 3.6, becoming useful to versions 3.3 through 3.5. 251 | 252 | Sometimes, the opposite happens. ``dev-python/mock`` started 253 | as a stand-alone package, and was integrated into the standard library 254 | as unittest.mock_ later on. Afterwards, the external package became 255 | a backport of the standard library module. 256 | 257 | In some cases backports effectively replace external packages. Once 258 | lzma_ module has been added to the standard library, its backport 259 | ``dev-python/backports-lzma`` has effectively replaced the competing 260 | LZMA packages. 261 | 262 | Individual backports differ by the level of compatibility with 263 | the standard library provided, and therefore on the amount of additional 264 | code needed in your program. The exact kind of dependencies used 265 | depends on that. 266 | 267 | ``dev-python/ipaddress`` is a drop-in backport of the ipaddress_ module 268 | from Python 3.3. It is using the same module name, so a code written 269 | to use this module will work out-of-the-box on Python 2.7 if the package 270 | is installed. As a side note, since Python always prefers built-in 271 | modules over external packages, there is no point in enabling Python 3 272 | in this package as the installed module would never be used. 273 | Appropriately, you should depend on this package only for the Python 274 | versions needing it. 275 | 276 | ``dev-python/mock`` is a compatible backport of the unittest.mock_ 277 | module. It can't use the same name as the standard library module, 278 | therefore the packages need to use it conditionally, e.g.:: 279 | 280 | try: 281 | from unittest.mock import Mock 282 | except ImportError: # py<3.3 283 | from mock import Mock 284 | 285 | or:: 286 | 287 | import sys 288 | if sys.hexversion >= 0x03030000: 289 | from unittest.mock import Mock 290 | else: 291 | from mock import Mock 292 | 293 | However, the actual API remains compatible, so the programs do not need 294 | more compatibility code than that. In some cases, upstreams fail (or 295 | even refuse) to use the external ``mock`` package conditionally — 296 | in that case, you either need to depend on this package unconditionally, 297 | or patch it. 298 | 299 | ``dev-python/trollius`` aimed to provide a backport of asyncio_ 300 | for Python 2. Since the asyncio framework relies on new Python syntax, 301 | the backport cannot be API compatible and requires using a different 302 | syntax than native asyncio code. 303 | 304 | 305 | .. _CPython: https://www.python.org/ 306 | 307 | .. _importlib.resources: 308 | https://docs.python.org/3.7/library/importlib.html#module-importlib.resources 309 | 310 | .. _platform.linux_distribution(): 311 | https://docs.python.org/3.7/library/platform.html#platform.linux_distribution 312 | 313 | .. _PyPy: https://www.pypy.org/ 314 | 315 | .. _Jython: https://www.jython.org/ 316 | 317 | .. _IronPython: https://ironpython.net/ 318 | 319 | .. _Brython: https://www.brython.info/ 320 | 321 | .. _MicroPython: https://micropython.org/ 322 | 323 | .. _Tauthon: https://github.com/naftaliharris/tauthon 324 | 325 | .. _unittest.mock: 326 | https://docs.python.org/3.3/library/unittest.mock.html 327 | 328 | .. _lzma: https://docs.python.org/3.3/library/lzma.html 329 | 330 | .. _ipaddress: https://docs.python.org/3.3/library/ipaddress.html 331 | 332 | .. _asyncio: https://docs.python.org/3.4/library/asyncio.html 333 | -------------------------------------------------------------------------------- /migration.rst: -------------------------------------------------------------------------------- 1 | ================ 2 | Migration guides 3 | ================ 4 | 5 | 6 | .. index:: PYTHON_MULTI_USEDEP 7 | 8 | Migrating from old PYTHON_USEDEP syntax in python-single-r1 9 | =========================================================== 10 | Prior to February 2020, ``python-single-r1`` used to provide a single 11 | ``PYTHON_USEDEP`` variable alike the two other eclasses. However, 12 | getting it to work correctly both on single-impl and multi-impl packages 13 | required a gross hack. 14 | 15 | The current eclass API requires using ``python_gen_cond_dep`` function 16 | to generate multi-impl deps instead, with ``PYTHON_USEDEP`` serving 17 | as a placeholder. Single-impl deps can be expressed with 18 | ``PYTHON_SINGLE_USEDEP`` that can be used either as placeholder, 19 | or directly as a variable. 20 | 21 | During a transitional period, ``PYTHON_USEDEP`` was banned entirely 22 | and ``PYTHON_MULTI_USEDEP`` was used instead. As of EAPI 8, 23 | the opposite is true — ``PYTHON_USEDEP`` is to be used, 24 | and ``PYTHON_MULTI_USEDEP`` was removed. 25 | 26 | The recommended rule of thumb for migrating old ebuilds is to: 27 | 28 | 1. Replace all instances of ``${PYTHON_USEDEP}`` for simple-impl deps 29 | with ``${PYTHON_SINGLE_USEDEP}``. If you don't know the type 30 | of given dep, dependency check (repoman, pkgcheck) will tell you 31 | if you chose wrong. 32 | 33 | 2. Wrap the dependencies using ``${PYTHON_USEDEP}`` in a single 34 | ``python_gen_cond_dep`` block (reordering may be desirable). 35 | 36 | 3. Run ``pkgcheck scan`` or ``repoman full``. If you get syntax errors, 37 | you probably missed ``python_gen_cond_dep`` or did not escape 38 | the ``$`` in placeholder properly. If you get unmatched dependency, 39 | you probably got single-impl vs. multi-impl wrong. 40 | 41 | This way, you can generally convert ebuilds using trial-and-error 42 | method. 43 | 44 | 45 | .. index:: EAPI 8 46 | 47 | Migrating from EAPI 7 to EAPI 8 48 | =============================== 49 | EAPI 8 has banned everything that's been deprecated in EAPI 7, as well 50 | as some other obsolete stuff. The following table lists all banned 51 | things along with their suggested replacements. 52 | 53 | +-------------------------------+------------------------------------+ 54 | | Deprecated thing | Replacement | 55 | +===============================+====================================+ 56 | | Private eclass API | 57 | +-------------------------------+------------------------------------+ 58 | | python_export | python_setup / getters | 59 | +-------------------------------+------------------------------------+ 60 | | python_wrapper_setup | python_setup | 61 | +-------------------------------+------------------------------------+ 62 | | Obsolete API | 63 | +-------------------------------+------------------------------------+ 64 | | distutils_install_for_testing | no argument (``--via-root``) | 65 | | ``--via-home`` | or ``--via-venv`` | 66 | +-------------------------------+------------------------------------+ 67 | | python_gen_usedep | python_gen_cond_dep | 68 | +-------------------------------+------------------------------------+ 69 | | PYTHON_MULTI_USEDEP | PYTHON_USEDEP | 70 | +-------------------------------+------------------------------------+ 71 | | mydistutilsargs rename | 72 | +-------------------------------+------------------------------------+ 73 | | mydistutilsargs | DISTUTILS_ARGS | 74 | +-------------------------------+------------------------------------+ 75 | | Post-Python 2 cleanup | 76 | +-------------------------------+------------------------------------+ 77 | | python_gen* -2 / python2 | remove entirely | 78 | | / pypy | | 79 | +-------------------------------+------------------------------------+ 80 | | python_gen* -3 | make unconditional | 81 | +-------------------------------+------------------------------------+ 82 | | python_is_python3 | always assume true | 83 | +-------------------------------+------------------------------------+ 84 | 85 | The changes can be split roughly into four groups: ban of now-private 86 | eclass API, ban of obsolete API functions, mydistutilsargs rename 87 | and bans related to post-Python 2 cleanup. 88 | 89 | The private eclass API part involves ``python_export`` 90 | and ``python_wrapper_setup``. Both were deprecated in March 2020, 91 | and they were never covered in this guide. The former was historically 92 | used to get information about the Python interpreter (either the current 93 | ``${EPYTHON}`` or an arbitrary choice), the latter to create the wrapper 94 | directory containing ``python`` and other executables. 95 | 96 | When the functions were used to establish a Python build environment, 97 | the replacement for both is a single ``python_setup`` call. When 98 | ``python_export`` was used to grab additional details about the Python 99 | interpreter, the various ``python_get*`` functions should be used 100 | instead. 101 | 102 | .. code-block:: bash 103 | 104 | src_configure() { 105 | # ... 106 | 107 | # OLD: 108 | local PYTHON_INCLUDEDIR PYTHON_LIBPATH 109 | python_export PYTHON_INCLUDEDIR PYTHON_LIBPATH 110 | mycmakeargs+=( 111 | -DPython3_INCLUDE_DIR="${PYTHON_INCLUDEDIR}" 112 | -DPython3_LIBRARY="${PYTHON_LIBPATH}" 113 | ) 114 | 115 | # NEW: 116 | mycmakeargs+=( 117 | -DPython3_INCLUDE_DIR="$(python_get_includedir)" 118 | -DPython3_LIBRARY="$(python_get_library_path)" 119 | ) 120 | } 121 | 122 | The second group involves sundry API that were deprecated earlier. 123 | These are: 124 | 125 | 1. ``distutils_install_for_testing --via-home`` layout that stopped 126 | working correctly at some point. The default ``--via-root`` should 127 | work most of the time, and ``-via-venv`` replace the remaining cases 128 | for the removed layout. 129 | 130 | 2. ``python_gen_usedep`` function that was historically used to generate 131 | partial USE dependencies, and was generally combined with 132 | ``REQUIRED_USE`` to force specific (usually old) Python interpreters 133 | for specific features. This was really ugly. Nowadays, you should 134 | really use ``python_gen_cond_dep`` instead. 135 | 136 | 3. ``PYTHON_MULTI_USEDEP`` placeholder that was temporarily used 137 | in python-single-r1 ebuilds. ``PYTHON_USEDEP`` is equivalent now. 138 | 139 | The third group is a sole rename of ``mydistutilsargs`` variable. 140 | Since you usually need to pass the same arguments in all phase 141 | functions, this variable was not really used in local scope. It has 142 | been renamed to uppercase ``DISTUTILS_ARGS`` to follow the common 143 | pattern for global scope variables. 144 | 145 | Finally, the fourth group involves banning some of the features that 146 | were specifically used in order to support distinguish between Python 2 147 | and Python 3. This is meant to force cleaning up old cruft from 148 | ebuilds. It comes in three parts: 149 | 150 | 1. Banning arguments to ``python_gen*`` that reference Python 2 151 | (e.g. ``-2``, ``python2*``, ``python2_7``, ``pypy``). Since Python 2 152 | is no longer supported in the relevant code paths, the relevant calls 153 | should just be removed. 154 | 155 | 2. Banning the ``-3`` short-hand to ``python_gen*``. Since all 156 | supported interpreters are compatible with Python 3 now, the relevant 157 | code should be made unconditional. Note that ``python3*`` is still 158 | useful, as it distinguishes CPython from PyPy3. 159 | 160 | 3. Banning the ``python_is_python3`` function. Since the removal 161 | of Python 2 support, it always evaluated to true. 162 | 163 | All the aforementioned replacements are available in all EAPIs. 164 | 165 | 166 | Migrating to PEP 517 builds 167 | =========================== 168 | As of January 2022, the ``distutils-r1`` can use PEP 517 build backends 169 | instead of calling setuptools directly. The new mode is particularly 170 | useful for: 171 | 172 | - packages using flit and poetry, as a better replacement for 173 | the deprecated ``dev-python/pyproject2setuppy`` hack 174 | 175 | - packages using other PEP 517 build systems (such as pdm) that are not 176 | supported in legacy mode at all 177 | 178 | - packages using setuptools without ``setup.py`` 179 | 180 | - packages using plain distutils, as the mode handles the switch from 181 | deprecated stdlib distutils to the version vendored in setuptools 182 | safely 183 | 184 | The PEP 517 mode provides the test phase with venv-style installed 185 | package tree (alike ``distutils_install_for_testing --via-venv``) 186 | that should make testing more streamlined. 187 | 188 | Unfortunately, the new mode can cause issues with customized distutils 189 | and setuptools build systems. It is important to verify the installed 190 | file list after the migration. Packages that require custom configure 191 | phases or passing arguments are not supported at the moment. 192 | 193 | For simple packages, the migration consists of: 194 | 195 | 1. Adding ``DISTUTILS_USE_PEP517`` above the inherit line. The value 196 | indicates the build system used, e.g. ``flit``, ``poetry``, 197 | ``setuptools`` (used also for distutils). 198 | 199 | 2. Removing ``DISTUTILS_USE_SETUPTOOLS``. If the previous value was 200 | ``rdepend`` (and indeed a runtime dependency is required), then 201 | ``dev-python/setuptools`` needs to be explicitly added to 202 | ``RDEPEND``. 203 | 204 | 3. Removing ``distutils_install_for_testing`` and/or ``--install`` 205 | option to ``distutils_enable_tests``. This should no longer be 206 | necessary and tests should work out of the box. 207 | 208 | 209 | .. index:: multipart 210 | .. index:: python-multipart 211 | 212 | multipart vs. python-multipart packages 213 | ======================================= 214 | Prior to November 2024, two PyPI packages claimed the ``multipart`` 215 | import name: multipart_ and python-multipart_. Originally, Gentoo 216 | packaged only the latter, as it was necessary to satisfy dependencies. 217 | However, with the former being listed as the recommended replacement 218 | for the `deprecated cgi module`_, new packages started depending on it. 219 | 220 | At this point, Gentoo decided to package multipart as well, 221 | and to rename python-multipart in anticipation of upstream rename 222 | already in progress, in ``>=dev-python/python-multipart-0.0.12-r100``. 223 | However, it should be noted that the Gentoo rename does not install 224 | a compatibility loader to permit importing it via ``multipart`` import 225 | name, as that would work only if ``dev-python/multipart`` 226 | was not installed. Instead, packages need to be explicitly patched 227 | to use the new import name of ``python_multipart``. 228 | 229 | Therefore, the dependencies on these packages should be handled 230 | in the following manner: 231 | 232 | 1. If the package depends on multipart, a dependency 233 | on ``dev-python/multipart`` should be added. 234 | 235 | 2. If the package bundles multipart, the dependency should be 236 | unbundled. 237 | 238 | 3. If the package depends on python-multipart and uses the new import 239 | name (i.e. ``import python_multipart``), a dependency 240 | on ``>=dev-python/python-multipart-0.0.12-r100`` (or a newer version) 241 | should be added. 242 | 243 | 4. If the package depends on python-multipart and uses the old import 244 | name (i.e. ``import multipart``), it should be patched to use 245 | the new import name instead, and the dependency should be added 246 | as above. 247 | 248 | The following snippet provides an example of patching 249 | the python-multipart import: 250 | 251 | .. code-block:: bash 252 | 253 | RDEPEND=" 254 | >=dev-python/python-multipart-0.0.12-r100[${PYTHON_USEDEP}] 255 | " 256 | 257 | src_prepare() { 258 | distutils-r1_src_prepare 259 | 260 | # python-multipart package renamed in Gentoo to python_multipart 261 | find -name "*.py" -exec sed \ 262 | -e "s:from multipart:from python_multipart:" \ 263 | -e "s:import multipart:import python_multipart as multipart:" \ 264 | -i {} + || die 265 | } 266 | 267 | 268 | .. _multipart: https://pypi.org/project/multipart/ 269 | .. _python-multipart: https://pypi.org/project/python-multipart/ 270 | .. _deprecated cgi module: https://docs.python.org/3.12/library/cgi.html 271 | -------------------------------------------------------------------------------- /multi.rst: -------------------------------------------------------------------------------- 1 | =============================== 2 | python-r1 — multi-impl packages 3 | =============================== 4 | 5 | .. highlight:: bash 6 | 7 | The ``python-r1`` eclass is used to install multi-impl packages. 8 | It is considered an expert eclass — when possible, you should prefer 9 | using ``python-single-r1`` instead. For packages using distutils 10 | or a similar Python build system, ``distutils-r1`` eclass should be used 11 | instead. 12 | 13 | Eclass reference: `python-r1.eclass(5)`_ 14 | 15 | 16 | .. index:: python_foreach_impl 17 | 18 | Manual install 19 | ============== 20 | The simplest case of multi-impl package is a package without a specific 21 | build system. The modules need to be installed manually here, 22 | and ``python_foreach_impl`` function is used to repeat the install step 23 | for all enabled implementations. 24 | 25 | For simple use cases, the install command can be inlined: 26 | 27 | .. code-block:: bash 28 | :emphasize-lines: 6,8,18,19,28 29 | 30 | # Copyright 1999-2024 Gentoo Authors 31 | # Distributed under the terms of the GNU General Public License v2 32 | 33 | EAPI=8 34 | 35 | PYTHON_COMPAT=( python3_{10..13} ) 36 | 37 | inherit python-r1 38 | 39 | DESCRIPTION="Enhanced df with colors" 40 | HOMEPAGE="http://kassiopeia.juls.savba.sk/~garabik/software/pydf/" 41 | SRC_URI="http://kassiopeia.juls.savba.sk/~garabik/software/pydf/${PN}_${PV}.tar.gz" 42 | 43 | LICENSE="public-domain" 44 | SLOT="0" 45 | KEYWORDS="amd64 arm ~arm64 ppc ppc64 ~riscv x86 ~amd64-linux ~x86-linux" 46 | 47 | REQUIRED_USE="${PYTHON_REQUIRED_USE}" 48 | RDEPEND="${PYTHON_DEPS}" 49 | BDEPEND="${RDEPEND}" 50 | 51 | src_prepare() { 52 | default 53 | sed -i -e "s#/etc/pydfrc#${EPREFIX}/etc/pydfrc#" "${PN}" || die 54 | } 55 | 56 | src_install() { 57 | python_foreach_impl python_doscript "${PN}" 58 | insinto /etc 59 | doins "${PN}rc" 60 | doman "${PN}.1" 61 | einstalldocs 62 | } 63 | 64 | While ``python_foreach_impl`` can be repeated multiple times, it is 65 | generally better to declare a function when multiple install commands 66 | need to be executed: 67 | 68 | .. code-block:: bash 69 | :emphasize-lines: 35-42,45 70 | 71 | # Copyright 1999-2024 Gentoo Authors 72 | # Distributed under the terms of the GNU General Public License v2 73 | 74 | EAPI=8 75 | 76 | PYTHON_COMPAT=( python3_{10..13} ) 77 | 78 | inherit python-r1 79 | 80 | DESCRIPTION="Check for mapped libs and open files that are marked as deleted" 81 | HOMEPAGE="https://github.com/klausman/lib_users" 82 | SRC_URI=" 83 | https://github.com/klausman/${PN}/archive/v${PV}.tar.gz 84 | -> ${P}.tar.gz 85 | " 86 | 87 | LICENSE="GPL-2" 88 | SLOT="0" 89 | KEYWORDS="~alpha amd64 ~arm arm64 ~hppa ppc ppc64 ~sparc x86" 90 | IUSE="test" 91 | RESTRICT="!test? ( test )" 92 | 93 | REQUIRED_USE="${PYTHON_REQUIRED_USE}" 94 | 95 | DEPEND="${PYTHON_DEPS} 96 | test? ( 97 | dev-python/nose2[${PYTHON_USEDEP}] 98 | )" 99 | RDEPEND="${PYTHON_DEPS}" 100 | 101 | src_test() { 102 | python_foreach_impl nose2 --verbosity=2 103 | } 104 | 105 | python_install() { 106 | python_newscript lib_users.py lib_users 107 | python_newscript fd_users.py fd_users 108 | # lib_users_util/ contains a test script we don't want, so do things by hand 109 | python_moduleinto lib_users_util 110 | python_domodule lib_users_util/common.py 111 | python_domodule lib_users_util/__init__.py 112 | } 113 | 114 | src_install() { 115 | python_foreach_impl python_install 116 | dodoc README.md TODO 117 | } 118 | 119 | .. index:: PYTHON_USEDEP; python-r1 120 | 121 | Dependencies 122 | ============ 123 | When depending on other Python packages, USE dependencies need to be 124 | declared in order to ensure that the dependencies would be built against 125 | all the Python implementations enabled for the package. This is easily 126 | done via appending the USE dependency string from ``${PYTHON_USEDEP}`` 127 | to the dependencies:: 128 | 129 | RDEPEND="${PYTHON_DEPS} 130 | sys-apps/portage[${PYTHON_USEDEP}] 131 | " 132 | DEPEND="${RDEPEND}" 133 | 134 | 135 | .. index:: run_in_build_dir 136 | 137 | Pure Python autotools package 138 | ============================= 139 | Another typical case for this eclass is to handle a pure Python package 140 | with a non-standard build system. In this case, it is generally 141 | necessary to call phase functions via ``python_foreach_impl``. Whenever 142 | possible, out-of-source builds are recommended (i.e. installing to 143 | separate directories from a single source directory). 144 | 145 | 146 | .. code-block:: bash 147 | :emphasize-lines: 34,38,42,50 148 | 149 | # Copyright 1999-2023 Gentoo Authors 150 | # Distributed under the terms of the GNU General Public License v2 151 | 152 | EAPI=7 153 | 154 | PYTHON_COMPAT=( python3_{10..13} ) 155 | inherit autotools python-r1 156 | 157 | DESCRIPTION="Python wrapper for libcangjie" 158 | HOMEPAGE="http://cangjians.github.io/" 159 | SRC_URI="https://github.com/Cangjians/py${PN}/releases/download/v${PV}/${P#py}.tar.xz" 160 | 161 | LICENSE="LGPL-3+" 162 | SLOT="0" 163 | KEYWORDS="amd64 x86" 164 | 165 | REQUIRED_USE="${PYTHON_REQUIRED_USE}" 166 | 167 | RDEPEND="${PYTHON_DEPS} 168 | app-i18n/libcangjie" 169 | DEPEND="${RDEPEND}" 170 | BDEPEND="dev-python/cython[${PYTHON_USEDEP}] 171 | virtual/pkgconfig" 172 | 173 | src_prepare() { 174 | default 175 | eautoreconf 176 | } 177 | 178 | src_configure() { 179 | python_configure() { 180 | ECONF_SOURCE="${S}" econf 181 | } 182 | python_foreach_impl run_in_build_dir python_configure 183 | } 184 | 185 | src_compile() { 186 | python_foreach_impl run_in_build_dir default 187 | } 188 | 189 | src_test() { 190 | python_foreach_impl run_in_build_dir default 191 | } 192 | 193 | src_install() { 194 | python_install() { 195 | default 196 | python_optimize 197 | } 198 | python_foreach_impl run_in_build_dir python_install 199 | einstalldocs 200 | 201 | find "${D}" -name '*.la' -delete || die 202 | } 203 | 204 | Note the use of ``run_in_build_dir`` helper from ``multibuild`` eclass 205 | (direct inherit is unnecessary here, as it is considered implicit part 206 | of ``python-r1`` API). It changes the directory to ``BUILD_DIR`` (which 207 | is set by ``python_foreach_impl`` to a unique directory for each 208 | implementation) and runs the specified command there. In this case, 209 | the ebuild performs autotools out-of-source build in a dedicated 210 | directory for every interpreter enabled. 211 | 212 | Also note that the in-build-dir call to ``default`` does not install 213 | documentation from source directory, hence the additional 214 | ``einstalldocs`` call. Libtool-based packages install ``.la`` files 215 | that are unnecessary for Python extensions, hence they are removed 216 | afterwards. 217 | 218 | If the package in question does not support out-of-source builds 219 | (e.g. due to a buggy build system), ``python_copy_sources`` function 220 | can be used to duplicate the package's sources in build directories 221 | for each implementation. The same ebuild easily can be changed 222 | to do that: 223 | 224 | .. code-block:: bash 225 | :emphasize-lines: 28,35,39,43,51 226 | 227 | # Copyright 1999-2023 Gentoo Authors 228 | # Distributed under the terms of the GNU General Public License v2 229 | 230 | EAPI=7 231 | 232 | PYTHON_COMPAT=( python3_{10..13} ) 233 | inherit autotools python-r1 234 | 235 | DESCRIPTION="Python wrapper for libcangjie" 236 | HOMEPAGE="http://cangjians.github.io/" 237 | SRC_URI="https://github.com/Cangjians/py${PN}/releases/download/v${PV}/${P#py}.tar.xz" 238 | 239 | LICENSE="LGPL-3+" 240 | SLOT="0" 241 | KEYWORDS="amd64 x86" 242 | 243 | REQUIRED_USE="${PYTHON_REQUIRED_USE}" 244 | 245 | RDEPEND="${PYTHON_DEPS} 246 | app-i18n/libcangjie" 247 | DEPEND="${RDEPEND}" 248 | BDEPEND="dev-python/cython[${PYTHON_USEDEP}] 249 | virtual/pkgconfig" 250 | 251 | src_prepare() { 252 | default 253 | eautoreconf 254 | python_copy_sources 255 | } 256 | 257 | src_configure() { 258 | python_configure() { 259 | ECONF_SOURCE="${S}" econf 260 | } 261 | python_foreach_impl run_in_build_dir python_configure 262 | } 263 | 264 | src_compile() { 265 | python_foreach_impl run_in_build_dir default 266 | } 267 | 268 | src_test() { 269 | python_foreach_impl run_in_build_dir default 270 | } 271 | 272 | src_install() { 273 | python_install() { 274 | default 275 | python_optimize 276 | } 277 | python_foreach_impl run_in_build_dir python_install 278 | einstalldocs 279 | 280 | find "${D}" -name '*.la' -delete || die 281 | } 282 | 283 | Note that besides adding ``python_copy_sources`` call, ``ECONF_SOURCE`` 284 | has been removed in order to disable out-of-source builds. 285 | 286 | 287 | Conditional Python use 288 | ====================== 289 | When the package installs Python components conditionally to a USE flag, 290 | the respective USE conditional needs to be consistently used in metadata 291 | variables and in ``python_foreach_impl`` calls. 292 | 293 | .. code-block:: bash 294 | :emphasize-lines: 15,16,20-22,42-48 295 | 296 | # Copyright 1999-2020 Gentoo Authors 297 | # Distributed under the terms of the GNU General Public License v2 298 | 299 | EAPI=6 300 | PYTHON_COMPAT=( python2_7 ) 301 | 302 | inherit gnome2 python-r1 303 | 304 | DESCRIPTION="Canvas widget for GTK+ using the cairo 2D library for drawing" 305 | HOMEPAGE="https://wiki.gnome.org/GooCanvas" 306 | 307 | LICENSE="LGPL-2" 308 | SLOT="2.0" 309 | KEYWORDS="~alpha amd64 ia64 ppc ppc64 sparc x86" 310 | IUSE="python" 311 | REQUIRED_USE="python? ( ${PYTHON_REQUIRED_USE} )" 312 | 313 | # python only enables python specific binding override 314 | RDEPEND=" 315 | python? ( 316 | ${PYTHON_DEPS} 317 | >=dev-python/pygobject-2.90.4:3[${PYTHON_USEDEP}] ) 318 | " 319 | DEPEND="${RDEPEND}" 320 | 321 | src_prepare() { 322 | # Python bindings are built/installed manually. 323 | sed -e "/SUBDIRS = python/d" -i bindings/Makefile.am \ 324 | bindings/Makefile.in || die 325 | 326 | gnome2_src_prepare 327 | } 328 | 329 | src_configure() { 330 | gnome2_src_configure \ 331 | --disable-python 332 | } 333 | 334 | src_install() { 335 | gnome2_src_install 336 | 337 | if use python; then 338 | sub_install() { 339 | python_moduleinto $(python -c "import gi;print gi._overridesdir") 340 | python_domodule bindings/python/GooCanvas.py 341 | } 342 | python_foreach_impl sub_install 343 | fi 344 | } 345 | 346 | Note that in many cases, you will end up having to disable upstream 347 | rules for installing Python files as they are suitable only for 348 | single-impl installs. 349 | 350 | 351 | .. index:: python_setup; for python-r1 352 | 353 | Additional build-time Python use 354 | ================================ 355 | Some packages additionally require Python at build time, independently 356 | of Python components installed (i.e. outside ``python_foreach_impl``). 357 | The eclass provides extensive API for this purpose but for now we'll 358 | focus on the simplest case where the global code does not have any 359 | dependencies or they are a subset of dependencies declared already. 360 | 361 | In this case, it is sufficient to call ``python_setup`` before 362 | the routine requiring Python. It will choose the most preferred 363 | of enabled implementations, and set the global environment for it. Note 364 | that it is entirely normal that the same environment will be set inside 365 | ``python_foreach_impl`` afterwards. 366 | 367 | .. code-block:: bash 368 | :emphasize-lines: 18,19,21,22,25,29-35,39-41 369 | 370 | # Copyright 1999-2024 Gentoo Authors 371 | # Distributed under the terms of the GNU General Public License v2 372 | 373 | EAPI=8 374 | 375 | PYTHON_COMPAT=( python3_{10..13} ) 376 | PYTHON_REQ_USE="ncurses,readline" 377 | 378 | inherit python-r1 379 | 380 | DESCRIPTION="QEMU + Kernel-based Virtual Machine userland tools" 381 | HOMEPAGE="http://www.qemu.org http://www.linux-kvm.org" 382 | SRC_URI="http://wiki.qemu-project.org/download/${P}.tar.xz" 383 | 384 | LICENSE="GPL-2 LGPL-2 BSD-2" 385 | SLOT="0" 386 | KEYWORDS="amd64 ~arm64 ~ppc ~ppc64 x86" 387 | IUSE="python" 388 | REQUIRED_USE="${PYTHON_REQUIRED_USE}" 389 | 390 | BDEPEND="${PYTHON_DEPS}" 391 | RDEPEND="python? ( ${PYTHON_DEPS} )" 392 | 393 | src_configure() { 394 | python_setup 395 | ./configure || die 396 | } 397 | 398 | qemu_python_install() { 399 | python_domodule "${S}/python/qemu" 400 | 401 | python_doscript "${S}/scripts/kvm/vmxcap" 402 | python_doscript "${S}/scripts/qmp/qmp-shell" 403 | python_doscript "${S}/scripts/qmp/qemu-ga-client" 404 | } 405 | 406 | src_install() { 407 | default 408 | if use python; then 409 | python_foreach_impl qemu_python_install 410 | fi 411 | } 412 | 413 | Note that the parts affecting installation of runtime components 414 | (``RDEPEND``, ``python_foreach_impl``) are made conditional to the USE 415 | flag, while parts affecting build time (``REQUIRED_USE``, ``BDEPEND``, 416 | ``python_setup``) are unconditional. 417 | 418 | 419 | .. _python-r1.eclass(5): 420 | https://devmanual.gentoo.org/eclass-reference/python-r1.eclass/index.html 421 | -------------------------------------------------------------------------------- /package-maintenance.rst: -------------------------------------------------------------------------------- 1 | ========================== 2 | Python package maintenance 3 | ========================== 4 | 5 | .. highlight:: bash 6 | 7 | Package name policy 8 | =================== 9 | All packages in ``dev-python/*`` that are published on PyPI_, must be 10 | named to match their respective PyPI names, after performing 11 | `normalization specified in PEP 503`_, i.e. after replacing all runs 12 | of dot, hyphen and underscore characters (``[-_.]+``) with a single 13 | hyphen (``-``), and all uppercase letters with lowercase. Notably, 14 | prefixes and suffixes such as ``python-`` must not be removed or added. 15 | 16 | Since it is not uncommon for multiple packages to be published with very 17 | similar names, it is crucial that all packages conform to this policy. 18 | Gentoo-specific renames not only make it harder for users to find 19 | the specific package they need but can also create name conflicts when 20 | another package needs to be added. For example, adding 21 | ``python-bugzilla`` package as ``dev-python/bugzilla`` would conflict 22 | with the ``bugzilla`` package that is also present on PyPI. 23 | 24 | The following table illustrates mapping PyPI package names to Gentoo 25 | package names. 26 | 27 | ================= =================================================== 28 | PyPI package name Correct Gentoo package name 29 | ================= =================================================== 30 | ConfigArgParse ``dev-python/configargparse`` 31 | Flask ``dev-python/flask`` 32 | flask-babel ``dev-python/flask-babel`` 33 | github3.py ``dev-python/github3-py`` 34 | python-bugzilla ``dev-python/python-bugzilla`` 35 | sphinx_pytest ``dev-python/sphinx-pytest`` 36 | sphinx-tabs ``dev-python/sphinx-tabs`` 37 | txAMQP ``dev-python/txamqp`` 38 | zope.interface ``dev-python/zope-interface`` 39 | zstd ``dev-python/zstd`` 40 | ================= =================================================== 41 | 42 | Note that the presented table already presents some inconsistency 43 | in upstream package names. For example, ``Flask`` itself uses title 44 | case name, while ``flask-babel`` has recently switched to lowercase. 45 | Using normalized names for all packages avoids this inconsistency within 46 | Gentoo. It may also be a good idea to point upstream to `PEP 423`_ 47 | that specifies package naming recommendations. 48 | 49 | PyPI automatically redirects to canonical package URLs. However, please 50 | make sure to use these canonical URLs in ``HOMEPAGE``, ``SRC_URI`` 51 | and ``remote-id`` to avoid unnecessary redirects. These are reported 52 | by ``pkgcheck scan --net``. 53 | 54 | When packaging software that is not published on PyPI, or adding 55 | multiple Gentoo packages corresponding to the same PyPI project, please 56 | bear potential future collisions in mind when naming it. Avoid names 57 | that are already used for other PyPI projects. When in doubt, prefer 58 | more verbose names that are less likely to be reused in the future. You 59 | may also suggest that upstream publishes to PyPI, or at least pushes 60 | an empty package to reserve the name. 61 | 62 | 63 | Support for Python 2 64 | ==================== 65 | Gentoo does not support building Python packages for Python 2 anymore. 66 | We are keeping PyPy2.7 (built stripped down, by default) to facilitate 67 | building PyPy3, and CPython 2.7 for PyPy2.7 bootstrap. Technically, 68 | they could still be used to run Python 2 code standalone, but this 69 | is discouraged and poses security risk. 70 | 71 | 72 | Which implementations to test new packages for? 73 | =============================================== 74 | The absolute minimum set of targets are the current default targets 75 | found in ``profiles/base/make.defaults``. However, developers 76 | are strongly encouraged to test at least the next Python 3 version 77 | in order to ease future transition, and preferably all future versions. 78 | 79 | Marking for PyPy3 is optional. At this moment, we do not aim for wide 80 | coverage of PyPy3 support. 81 | 82 | 83 | Adding new Python implementations to existing packages 84 | ====================================================== 85 | New Python implementations can generally be added to existing packages 86 | without a revision bump. This is because the new dependencies are added 87 | conditionally to new USE flags. Since the existing users can not have 88 | the new flags enabled, the dependencies do not need to be proactively 89 | added to existing installations. 90 | 91 | This usually applies to stable packages as well as new Python targets 92 | are generally ``use.stable.mask``-ed. This means that stable users 93 | will not be able to enable newly added flags and therefore the risk 94 | of the change breaking stable systems is minimal. 95 | 96 | 97 | Which packages can be (co-)maintained by the Python project? 98 | ============================================================ 99 | A large part of the Python ecosystem is fairly consistent, making it 100 | feasible for (co-)maintenance by the Gentoo Python team. 101 | 102 | As a rule of thumb, Python team is ready to maintain packages specific 103 | to the Python ecosystem and useful for the general population of Python 104 | programmers. This includes Python interpreters and tooling, packages 105 | purely providing Python modules and extensions and utilities specific 106 | to the Python language. 107 | 108 | However, the Python team has limited manpower, therefore it may reject 109 | packages that have high maintenance requirements. As a rule, Python 110 | team does not accept packages without working tests. 111 | 112 | If your package matches the above profile, feel free to ask a member 113 | of the Python project whether they would like to (co-)maintain 114 | the package. However, if you are not a member of the project, please 115 | do not add us without asking first. 116 | 117 | 118 | Porting packages to a new EAPI 119 | ============================== 120 | When porting packages to a new EAPI, please take care not to port 121 | the dependencies of Portage prematurely. This generally includes 122 | ``app-portage/gemato``, ``dev-python/setuptools`` and their recursive 123 | dependencies. 124 | 125 | Ideally, these ebuilds carry an appropriate note above their EAPI line, 126 | e.g.:: 127 | 128 | # please keep this ebuild at EAPI 7 -- sys-apps/portage dep 129 | EAPI=7 130 | 131 | This does not apply to test dependencies — they are not strictly 132 | necessary to install a new Portage version. 133 | 134 | 135 | Monitoring new package versions 136 | =============================== 137 | 138 | PyPI release feeds 139 | ------------------ 140 | The most efficient way to follow new Python package releases are 141 | the feeds found on PyPI_. These can be found in the package's 142 | "Release history" tab, as "RSS feed". 143 | 144 | The Gentoo Python project maintains a comprehensive `list of PyPI feeds 145 | for packages`_ in ``dev-python/`` category (as well as other important 146 | packages maintained by the Python team) in OPML format. 147 | 148 | 149 | Checking via pip 150 | ---------------- 151 | The `pip list -\-outdated`_ command described in a followup section 152 | can also be used to verify installed packages against their latest PyPI 153 | releases. However, this is naturally limited to packages installed 154 | on the particular system, and does not account for newer versions being 155 | already available in the Gentoo repository. 156 | 157 | 158 | Repology 159 | -------- 160 | Repology_ provides a comprehensive service for tracking distribution 161 | package versions and upstream releases. The easiest ways to find Python 162 | packages present in the Gentoo repository is to search by their 163 | maintainer's e-mail or category (e.g. ``dev-python``). When searching 164 | by name, the majority of Python-specific package use ``python:`` prefix 165 | in their Repology names. 166 | 167 | Unfortunately, Repology is very susceptible to false positives. 168 | Examples of false positives include other distributions using custom 169 | version numbers, replacing packages with forks or simply Repology 170 | confusing different packages with the same name. If you find false 171 | positives, please use the 'Report' option to request a correction. 172 | 173 | Please also note that Repology is unable to handle the less common 174 | version numbers that do not have a clear mapping to Gentoo version 175 | syntax (e.g. ``.post`` releases). 176 | 177 | 178 | Stabilization recommendations 179 | ============================= 180 | 181 | Policy 182 | ------ 183 | The Python landscape is changing dynamically, and therefore the test 184 | suites in packages — if not whole packages — often start failing early. 185 | For this reason, we recommend lowering the baseline stabilization delay 186 | to 14 days. 187 | 188 | In addition to that: 189 | 190 | 1. When stabilizing across a major version bump (e.g. from 1.1.x 191 | to 1.2.x), prefer waiting until the newest minor version becomes 192 | the stable candidate (i.e. do not stabilize from 1.1.4 to 1.2.0 193 | if 1.2.1 is available). When stabilizing over a minor version bump 194 | (e.g. from 1.1.4 to 1.1.5), feel free to proceed immediately. 195 | 196 | 2. If reverse dependencies block upgrade of a package (e.g. through 197 | ``<`` dependencies), consider stabilizing the newest versions 198 | matching the restriction as well. The same is worth considering 199 | if upstream maintains multiple versions simultaneously with major 200 | API changes, even if there are no explicit ``<`` dependencies 201 | (e.g. ``dev-python/django``). 202 | 203 | 3. If a new release is likely to cause major compatibility issues 204 | (e.g. major releases of ``dev-python/sphinx``), consider delaying 205 | the stabilization and/or explicitly testing its reverse dependencies, 206 | in order to ensure that necessary ``<`` dependencies are added first. 207 | 208 | 4. Avoid stabilizing prereleases (alpha, beta and RC versions), unless 209 | it is necessary and upstream does not provide a final release 210 | for a significant time. 211 | 212 | 5. Ideally, aim for tests to pass on all relevant architectures. Add 213 | deselects if necessary, as this will ensure that future 214 | stabilizations will be handled faster. 215 | 216 | 217 | Tooling 218 | ------- 219 | The recommended way of filing stabilization requests is to use 220 | ``stablereq-*`` tools from ``app-portage/mgorny-dev-scripts`` package, 221 | combined with ``pkgdev`` from ``dev-util/pkgdev``. 222 | 223 | To prepare the initial stabilization list and open it in an editor:: 224 | 225 | export PKGCHECK_ARGS="--stabletime 14" 226 | git grep -l python@ '**/metadata.xml' | 227 | cut -d/ -f1-2 | 228 | grep -v dev-python/ | 229 | xargs stablereq-make-list 'dev-python/*' 230 | 231 | Simultaneously, the following call can be used to run ``eshowkw`` 232 | to display current keywords on all stabilization candidates:: 233 | 234 | export PKGCHECK_ARGS="--stabletime 14" 235 | git grep -l python@ '**/metadata.xml' | 236 | cut -d/ -f1-2 | 237 | grep -v dev-python/ | 238 | xargs stablereq-eshowkw 'dev-python/*' 239 | 240 | Edit the list as desirable, save into a file and then feed the file 241 | into pkgdev:: 242 | 243 | pkgdev bugs --auto-cc-arches=* $(<"${file_path}") 244 | 245 | 246 | Routine checks on installed Python packages 247 | =========================================== 248 | The following actions are recommended to be run periodically on systems 249 | used to test Python packages. They could be run e.g. via post-sync 250 | actions. 251 | 252 | 253 | pip check 254 | --------- 255 | ``pip check`` (provided by ``dev-python/pip``) can be used to check 256 | installed packages for missing dependencies and version conflicts: 257 | 258 | .. code-block:: text 259 | 260 | $ python3.10 -m pip check 261 | meson-python 0.6.0 requires ninja, which is not installed. 262 | cx-freeze 6.11.1 requires patchelf, which is not installed. 263 | openapi-spec-validator 0.4.0 has requirement openapi-schema-validator<0.3.0,>=0.2.0, but you have openapi-schema-validator 0.3.0. 264 | cx-freeze 6.11.1 has requirement setuptools<=60.10.0,>=59.0.1, but you have setuptools 62.6.0. 265 | 266 | This tool checks the installed packages for a single Python 267 | implementation only, so you need to run it for every installed 268 | interpreter separately. 269 | 270 | In some cases the issues are caused by unnecessary version pins 271 | or upstream packages listing optional dependencies as obligatory. 272 | The preferred fix is to fix the package metadata rather than modifying 273 | the dependencies in ebuild. 274 | 275 | .. Warning:: 276 | 277 | pip does not support the ``Provides`` metadata, so it can 278 | produce false positives about ``certifi`` dependency. Please ignore 279 | these: 280 | 281 | .. code-block:: text 282 | 283 | httpcore 0.15.0 requires certifi, which is not installed. 284 | httpx 0.23.0 requires certifi, which is not installed. 285 | sphobjinv 2.2.2 requires certifi, which is not installed. 286 | requests 2.28.0 requires certifi, which is not installed. 287 | 288 | 289 | pip list -\-outdated 290 | -------------------- 291 | ``pip list --outdated`` (provided by ``dev-python/pip``) can be used 292 | to check whether installed packages are up-to-date. This can help 293 | checking for pending version bumps, as well as to detect wrong versions 294 | in installed metadata: 295 | 296 | .. code-block:: text 297 | 298 | $ pip3.11 list --outdated 299 | Package Version Latest Type 300 | ------------------------ ----------------- ------- ----- 301 | dirty-equals 0 0.4 wheel 302 | filetype 1.0.10 1.0.13 wheel 303 | mercurial 6.1.3 6.1.4 sdist 304 | node-semver 0.8.0 0.8.1 wheel 305 | PyQt-builder 1.12.2 1.13.0 wheel 306 | PyQt5 5.15.6 5.15.7 wheel 307 | PyQt5-sip 12.10.1 12.11.0 sdist 308 | PyQtWebEngine 5.15.5 5.15.6 wheel 309 | Routes 2.5.1.dev20220522 2.5.1 wheel 310 | selenium 3.141.0 4.3.0 wheel 311 | sip 6.6.1 6.6.2 wheel 312 | sphinxcontrib-websupport 1.2.4.dev20220515 1.2.4 wheel 313 | uri-template 0.0.0 1.2.0 wheel 314 | watchfiles 0.0.0 0.15.0 wheel 315 | watchgod 0.0.dev0 0.8.2 wheel 316 | 317 | Again, the action applies to a single Python implementation only 318 | and needs to be repeated for all of them. 319 | 320 | Particularly note the packages with versions containing only zeroes 321 | in the above list — this is usually a sign that the build system 322 | does not recognize the version correctly. In some cases, the only 323 | working solution would be to sed the correct version in. 324 | 325 | The additional ``dev`` suffix is usually appended via ``tag_build`` 326 | option in ``setup.cfg``. This causes the version to be considered 327 | older than the actual release, and therefore the respective options need 328 | to be stripped. 329 | 330 | 331 | gpy-verify-deps 332 | --------------- 333 | ``gpy-verify-deps`` (provided by ``app-portage/gpyutils``) compares 334 | the ebuild dependencies of all installed Python packages against their 335 | metadata. It reports the dependencies that are potentially missing 336 | in ebuilds, as well as dependencies potentially missing 337 | ``[${PYTHON_USEDEP}]``. For the latter, it assumes that all 338 | dependencies listed in package metadata are used as Python modules. 339 | 340 | .. code-block:: text 341 | 342 | $ gpy-verify-deps 343 | [...] 344 | =dev-python/tempest-31.0.0: missing dependency: dev-python/oslo-serialization [*] 345 | =dev-python/tempest-31.0.0: missing dependency: dev-python/cryptography [*] 346 | =dev-python/tempest-31.0.0: missing dependency: dev-python/stestr [*] 347 | =dev-python/versioningit-2.0.0: missing dependency: dev-python/tomli [*] 348 | =dev-python/versioningit-2.0.0: missing dependency: dev-python/importlib_metadata [python3.8 python3.9] 349 | =dev-python/wstools-0.4.10-r1: missing dependency: dev-python/setuptools [*] 350 | 351 | The check is done for all installed interpreters. The report indicates 352 | whether the dependency upstream is unconditional (``[*]``) or specific 353 | to a subset of Python implementations. 354 | 355 | Similarly to ``pip check`` results, every dependency needs to be 356 | verified. In many cases, upstream metadata lists optional or build-time 357 | dependencies as runtime dependencies, and it is preferable to strip them 358 | than to copy the mistakes into the ebuild. 359 | 360 | 361 | .. _PyPI: https://pypi.org/ 362 | 363 | .. _normalization specified in PEP 503: 364 | https://peps.python.org/pep-0503/#normalized-names 365 | 366 | .. _PEP 423: https://peps.python.org/pep-0423/ 367 | 368 | .. _list of PyPI feeds for packages: 369 | https://projects.gentoo.org/python/release-feeds.opml 370 | 371 | .. _Repology: https://repology.org/ 372 | -------------------------------------------------------------------------------- /preface.rst: -------------------------------------------------------------------------------- 1 | ======= 2 | Preface 3 | ======= 4 | 5 | Gentoo provides one of the best frameworks for providing Python support 6 | in packages among operating systems. This includes support for 7 | running multiple versions of Python (while most other distributions 8 | avoid going beyond simultaneous support for Python 2 and one version 9 | of Python 3), alternative implementations of Python, reliable tests, 10 | deep QA checks. While we aim to keep things simple, this is not always 11 | possible. 12 | 13 | At the same time, the available documentation is limited and not always 14 | up-to-date. Both the `built-in eclass documentation`_ and `Python 15 | project wiki page`_ provide bits of documentation but they are mostly 16 | in reference form and not very suitable for beginners nor people who 17 | do not actively follow the developments within the ecosystem. This 18 | results in suboptimal ebuilds, improper dependencies, missing tests. 19 | 20 | This document aims to fill the gap by providing a good, complete, 21 | by-topic (rather than reference-style) documentation for the ecosystem 22 | in Gentoo and the relevant eclasses. Combined with examples, it should 23 | help you write good ebuilds and solve common problems as simply 24 | as possible. 25 | 26 | `Gentoo Python Guide sources`_ are available on GitHub. Suggestions 27 | and improvements are welcome. 28 | 29 | 30 | .. _built-in eclass documentation: 31 | https://devmanual.gentoo.org/eclass-reference/index.html 32 | 33 | .. _Python project wiki page: 34 | https://wiki.gentoo.org/wiki/Project:Python 35 | 36 | .. _Gentoo Python Guide sources: 37 | https://github.com/mgorny/python-guide/ 38 | -------------------------------------------------------------------------------- /pypi.rst: -------------------------------------------------------------------------------- 1 | ====================================== 2 | pypi — helper eclass for PyPI archives 3 | ====================================== 4 | 5 | .. highlight:: bash 6 | 7 | The ``pypi`` eclass is a small eclass to facilitate fetching sources 8 | from PyPI. It abstracts away the complexity of PyPI URLs, and makes it 9 | easier to adapt ``SRC_URI`` to their future changes. 10 | 11 | Please note that PyPI archives are not always the best choice 12 | for distfiles. In particular, they frequently are missing tests 13 | and other files important to Gentoo packaging. Should that be the case, 14 | other archives should be used. Read the :ref:`source archives` section 15 | for more information. 16 | 17 | Eclass reference: `pypi.eclass(5)`_ 18 | 19 | 20 | PyPI URLs 21 | ========= 22 | 23 | Modern and legacy URLs 24 | ---------------------- 25 | The modern form of PyPI URLs include a hash of the distribution file, 26 | e.g.:: 27 | 28 | https://files.pythonhosted.org/packages/20/2e/36e46173a288c1c40853ffdb712c67e0e022df0e1ce50b7b1b50066b74d4/gpep517-13.tar.gz 29 | 30 | This makes them inconvenient for use in ebuilds, as the hash would have 31 | to be updated manually on every version bump. For this reason, Gentoo 32 | settled on using the legacy form of URLs instead, e.g.:: 33 | 34 | https://files.pythonhosted.org/packages/source/g/gpep517/gpep517-13.tar.gz 35 | 36 | It should be noted that legacy URLs are no longer supported and may stop 37 | working at an arbitrary time. Using the eclass is going to make 38 | the migration to an alternative provider easier than inline URLs. 39 | 40 | The path part of a legacy URL for a source package has the general form 41 | of:: 42 | 43 | /packages/source/${project::1}/${project}/${filename} 44 | 45 | whereas the path for a binary package (wheel) is:: 46 | 47 | /packages/${pytag}/${project::1}/${project}/${filename} 48 | 49 | In both cases, ``${project}`` refers to the PyPI project name. 50 | Technically, PyPI accepts any name that matches the PyPI project name 51 | after `package name normalization`_. However, this behavior is not 52 | guaranteed and using the canonical project name is recommended. 53 | 54 | The filenames and ``${pytag}`` are described in the subsequent sections. 55 | 56 | 57 | Source distribution filenames 58 | ----------------------------- 59 | The filename of a source distribution (sdist) has the general form of:: 60 | 61 | ${name}-${version}.tar.gz 62 | 63 | where ``${name}`` is the project name normalized according to `PEP 625`_ 64 | and ``${version}`` is the version following `PEP 440`_. The project 65 | name normalization transforms all uppercase letters to lowercase, 66 | and replaces all contiguous runs of ``._-`` characters with a single 67 | underscore. 68 | 69 | For example, package ``Flask-BabelEx`` version ``1.2.3`` would use 70 | the following filename:: 71 | 72 | flask_babelex-1.2.3.tar.gz 73 | 74 | However, note that source distributions predating PEP 625 are still 75 | commonly found, and they use non-normalized project names, e.g.:: 76 | 77 | Flask-BabelEx-1.2.3.tar.gz 78 | 79 | In both instances, the top directory inside the archive has the same 80 | name as the archive filename, minus ``.tar.gz`` suffix. Historically, 81 | ``.zip`` distributions were used as well. 82 | 83 | 84 | Binary distribution filenames 85 | ----------------------------- 86 | The filename of a binary distribution (wheel) has the general form of:: 87 | 88 | ${name}-${version}-${pytag}-${abitag}-${platformtag}.whl 89 | 90 | Similarly to source distributions, ``${name}`` and ``${version}`` 91 | specify the normalized project name and version (normalization follows 92 | the same rules and is specified in `binary distribution format`_ 93 | specification). 94 | 95 | ``${pytag}`` is the tag specifying Python version that the wheel was 96 | built for. This can be ``py2.py3`` or ``py3`` for pure Python wheels, 97 | or e.g. ``cp39`` for CPython 3.9 wheels. ``${abitag}`` specifies 98 | the appropriate ABI (``none`` for pure Python wheels), 99 | ``${platformtag}`` the platform (``any`` for pure Python wheels). 100 | 101 | For example, the modern wheel for the aforementioned package would be 102 | named:: 103 | 104 | flask_babelex-1.2.3-py3-none-any.whl 105 | 106 | However, some distributions use older normalization rules specified 107 | in `PEP 427`_. In this case, runs of characters other than alphanumeric 108 | characters and dots (``[^\w\d.]+``) are replaced by a single underscore. 109 | Notably, this means that uppercase letters and dots are left in place. 110 | In this normalization, the example wheel is named:: 111 | 112 | Flask_BabelEx-1.2.3-py3-none-any.whl 113 | 114 | 115 | Packages with matching name and version 116 | ======================================= 117 | In the most common case, the upstream package will have exactly the same 118 | name as the Gentoo package, and the version numbers will be entirely 119 | compatible. In this case, it is sufficient to inherit the eclass, 120 | and it will automatically set a suitable default ``SRC_URI``. 121 | The result will be roughly equivalent to:: 122 | 123 | SRC_URI=" 124 | https://files.pythonhosted.org/packages/source/${PN::1}/${PN}/${P}.tar.gz 125 | " 126 | S=${WORKDIR}/${P} 127 | 128 | with ``${P}`` actually using the normalized project name and the Gentoo 129 | version being translated to its PyPI equivalent. The version 130 | translation performs the following replacements: 131 | 132 | ============= =========== 133 | Gentoo suffix PyPI suffix 134 | ============= =========== 135 | _alpha a 136 | _beta b 137 | _rc rc 138 | _p .post 139 | ============= =========== 140 | 141 | If the project in question uses a build system that is not compliant 142 | with `PEP 625`_ and has uppercase letters or dots in its name, you may 143 | need to set ``PYPI_NO_NORMALIZE`` to a non-empty value to disable name 144 | normalization, e.g.:: 145 | 146 | PYPI_NO_NORMALIZE=1 147 | 148 | inherit distutils-r1 pypi 149 | 150 | Note that ``SRC_URI`` is not combined between eclasses and ebuilds. 151 | Should you need to fetch additional files, you need to explicitly append 152 | to the variable or the default will be overwritten, e.g.:: 153 | 154 | inherit distutils-r1 pypi 155 | 156 | SRC_URI+=" 157 | https://github.com/pytest-dev/execnet/commit/c0459b92bc4a42b08281e69b8802d24c5d3415d4.patch 158 | -> ${P}-pytest-7.2.patch 159 | " 160 | 161 | 162 | Package with a different name 163 | ============================= 164 | If the project name used on PyPI differs from the Gentoo package name, 165 | the ``PYPI_PN`` variable can be used to use another name. This is 166 | especially useful for project that use uppercase letters or dots 167 | in their names. 168 | 169 | For example, a package using lowercase package name in Gentoo and title 170 | case on PyPI could use:: 171 | 172 | PYPI_PN=${PN^} 173 | 174 | inherit distutils-r1 pypi 175 | 176 | Note that projects whose distfiles do not conform to PEP 625 will also 177 | need to explicitly disable filename normalization. For example, 178 | a package using dot in its filename and setuptools would use:: 179 | 180 | PYPI_NO_NORMALIZE=1 181 | PYPI_PN=${PN/-/.} 182 | 183 | inherit distutils-r1 pypi 184 | 185 | 186 | Customizing the generated URL 187 | ============================= 188 | The default value may not be suitable for your package if it uses 189 | a different project name than the Gentoo package name, a version number 190 | that needs to be translated differently or the legacy ``.zip`` sdist 191 | format. The ``pypi_sdist_url`` function can be used to generate URLs 192 | in that case. Its usage is:: 193 | 194 | pypi_sdist_url [--no-normalize] [ [ []]] 195 | 196 | with package defaulting to ``${PYPI_PN}``, version to translated ``${PV}`` 197 | and suffix to ``.tar.gz``. The generated filename uses `PEP 625`_ 198 | normalization, unless ``--no-normalize`` is provided 199 | (``PYPI_NO_NORMALIZE`` does not affect explicit function calls). 200 | For example, the Gentoo ``dev-python/markups`` package uses title-case 201 | ``Markups`` project name and legacy filename, and so the ebuild needs 202 | to use:: 203 | 204 | inherit distutils-r1 pypi 205 | 206 | SRC_URI="$(pypi_sdist_url --no-normalize "${PN^}")" 207 | S=${WORKDIR}/${P^} 208 | 209 | Should the package start using source distributions with normalized 210 | filenames, then only project name would need to be overriden 211 | and the default ``S`` would be correct (``Markups`` and ``markups`` 212 | normalize the same):: 213 | 214 | inherit distutils-r1 pypi 215 | 216 | SRC_URI="$(pypi_sdist_url "${PN^}")" 217 | 218 | Note that due to project name normalization, the ebuild would also work 219 | without ``SRC_URI`` override. However, it is recommended to pass 220 | the canonical project name, as normalization is not guaranteed. 221 | 222 | 223 | Fetching wheels 224 | =============== 225 | In very specific cases, it may be necessary to fetch wheels 226 | (i.e. prebuilt Python packages) instead. The ``pypi_wheel_url`` 227 | function is provided to aid this purpose. Its usage is:: 228 | 229 | pypi_wheel_url [--unpack] [ [ [ []]]] 230 | 231 | with package defaulting to ``${PYPI_PN}``, version to translated ``${PV}``, 232 | python-tag to ``py3`` and abi-platform-tag to ``none-any`` 233 | (i.e. indicating a pure Python package). For example, 234 | ``dev-python/ensurepip-setuptools`` does:: 235 | 236 | inherit pypi 237 | SRC_URI="$(pypi_wheel_url "${PN#ensurepip-}")" 238 | 239 | Note that wheels are ZIP archives suffixed ``.whl``, and they are not 240 | unpacked by the package manager automatically. Should you need them 241 | unpacked, you can pass ``--unpack`` option to include a ``->`` operator 242 | that renames the wheel to use ``.whl.zip`` suffix, causing it to be 243 | unpacked. Remember to add an explicit dependency on ``app-arch/unzip`` 244 | in that case. 245 | 246 | The ``pypi_wheel_filename`` function is provided to aid getting 247 | the wheel filename. It has a matching synopsis:: 248 | 249 | pypi_wheel_filename [ [ [ []]]] 250 | 251 | 252 | .. _pypi.eclass(5): 253 | https://devmanual.gentoo.org/eclass-reference/pypi.eclass/index.html 254 | .. _package name normalization: 255 | https://packaging.python.org/en/latest/specifications/name-normalization/ 256 | .. _PEP 625: https://peps.python.org/pep-0625/ 257 | .. _PEP 440: https://peps.python.org/pep-0440/ 258 | .. _binary distribution format: 259 | https://packaging.python.org/en/latest/specifications/binary-distribution-format/ 260 | .. _PEP 427: https://peps.python.org/pep-0427/ 261 | -------------------------------------------------------------------------------- /pytest.rst: -------------------------------------------------------------------------------- 1 | ============== 2 | pytest recipes 3 | ============== 4 | 5 | .. highlight:: bash 6 | 7 | Skipping tests based on markers 8 | =============================== 9 | A few packages use `custom pytest markers`_ to indicate e.g. tests 10 | requiring Internet access. These markers can be used to conveniently 11 | disable whole test groups, e.g.:: 12 | 13 | python_test() { 14 | epytest -m 'not network' dask 15 | } 16 | 17 | 18 | Skipping tests based on paths/names 19 | =================================== 20 | There are two primary methods of skipping tests based on path (and name) 21 | in pytest: using ``--ignore`` and ``--deselect``. 22 | 23 | ``--ignore`` causes pytest to entirely ignore a file or a directory 24 | when collecting tests. This works only for skipping whole files but it 25 | ignores missing dependencies and other failures occurring while 26 | importing the test file. 27 | 28 | ``--deselect`` causes pytest to skip the specific test or tests. It can 29 | be also used to select individual tests or even parametrized variants 30 | of tests. 31 | 32 | Both options can be combined to get tests to pass without having 33 | to alter the test files. They are preferable over suggestions from 34 | skipping problematic tests when tests are installed as part 35 | of the package. They can also be easily applied conditionally to Python 36 | interpreter. 37 | 38 | The modern versions of eclasses provide two control variables, 39 | ``EPYTEST_IGNORE`` and ``EPYTEST_DESELECT`` that can be used to list 40 | test files or tests to be ignored or deselected respectively. These 41 | variables can be used in global scope to avoid redefining 42 | ``python_test()``. However, combining them with additional conditions 43 | requires using the local scope. 44 | 45 | :: 46 | 47 | python_test() { 48 | local EPYTEST_IGNORE=( 49 | # ignore whole file with missing dep 50 | tests/test_client.py 51 | ) 52 | local EPYTEST_DESELECT=( 53 | # deselect a single test 54 | 'tests/utils/test_general.py::test_filename' 55 | # deselect a parametrized test based on first param 56 | 'tests/test_transport.py::test_transport_works[eventlet' 57 | ) 58 | [[ ${EPYTHON} == python3.6 ]] && EPYTEST_DESELECT+=( 59 | # deselect a test for py3.6 only 60 | 'tests/utils/test_contextvars.py::test_leaks[greenlet]' 61 | ) 62 | epytest 63 | } 64 | 65 | 66 | Avoiding the dependency on pytest-runner 67 | ======================================== 68 | pytest-runner_ is a package providing ``pytest`` command to setuptools. 69 | While it might be convenient upstream, there is no real reason to use 70 | it in Gentoo packages. It has no real advantage over calling pytest 71 | directly. 72 | 73 | Some packages declare the dependency on ``pytest-runner`` 74 | in ``setup_requires``. As a result, the dependency is enforced whenever 75 | ``setup.py`` is being run, even if the user has no intention of running 76 | tests. If this is the case, the dependency must be stripped. 77 | 78 | The recommended method of stripping it is to use sed:: 79 | 80 | python_prepare_all() { 81 | sed -i -e '/pytest-runner/d' setup.py || die 82 | distutils-r1_python_prepare_all 83 | } 84 | 85 | 86 | .. index:: PYTEST_DISABLE_PLUGIN_AUTOLOAD 87 | .. index:: PYTEST_PLUGINS 88 | 89 | Disabling plugin autoloading 90 | ============================ 91 | Normally, when running a test suite pytest loads all plugins installed 92 | on the system. This is often convenient for upstreams, as it makes it 93 | possible to use the features provided by the plugins (such as ``async`` 94 | test function support, or fixtures) without the necessity to explicitly 95 | enable them. However, there are also cases when additional plugins 96 | could make the test suite fail or become very slow (especially if pytest 97 | is called recursively). 98 | 99 | The modern recommendation for these cases is to disable plugin 100 | autoloading via setting the ``PYTEST_DISABLE_PLUGIN_AUTOLOAD`` 101 | environment variable, and then explicitly enable specific plugins 102 | if necessary. 103 | 104 | .. Note:: 105 | 106 | Previously we used to recommend explicitly disabling problematic 107 | plugins via ``-p no:``. However, it is rarely obvious 108 | which plugin is causing the problems, and it is entirely possible 109 | that another plugin will cause issues in the future, so an opt-in 110 | approach is usually faster and more reliable. 111 | 112 | The easier approach to enabling plugins is to use the ``-p`` option, 113 | listing specific plugins. The option can be passed multiple times, 114 | and accepts a plugin name as specified in the package's 115 | ``entry_points.txt`` file:: 116 | 117 | python_test() { 118 | local -x PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 119 | epytest -p asyncio -p tornado 120 | } 121 | 122 | However, this approach does not work when the test suite calls pytest 123 | recursively (e.g. you are testing a pytest plugin). In this case, 124 | the ``PYTEST_PLUGINS`` environment variable can be used instead. It 125 | takes a comma-separated list of plugin *module names*:: 126 | 127 | python_test() { 128 | local -x PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 129 | local -x PYTEST_PLUGINS=xdist.plugin,xdist.looponfail,pytest_forked 130 | 131 | epytest 132 | } 133 | 134 | Please note that failing to enable all the required plugins may cause 135 | some of the tests to be skipped implicitly (especially if the test suite 136 | is using ``async`` functions and no async plugin is loaded). Please 137 | look at skip messages and warnings to make sure everything works 138 | as intended. 139 | 140 | 141 | .. index:: EPYTEST_XDIST 142 | 143 | Using pytest-xdist to run tests in parallel 144 | =========================================== 145 | pytest-xdist_ is a plugin that makes it possible to run multiple tests 146 | in parallel. This is especially useful for programs with large test 147 | suites that take significant time to run single-threaded. 148 | 149 | Using pytest-xdist is recommended if the package in question supports it 150 | (i.e. it does not cause semi-random test failures) and its test suite 151 | takes significant time. This is done via setting ``EPYTEST_XDIST`` 152 | to a non-empty value prior to calling ``distutils_enable_tests``. 153 | It ensures that an appropriate depedency is added, and that ``epytest`` 154 | adds necessary command-line options. 155 | 156 | .. code-block:: 157 | 158 | EPYTEST_XDIST=1 159 | distutils_enable_tests pytest 160 | 161 | Please note that some upstream use pytest-xdist even if there is no real 162 | gain from doing so. If the package's tests take a short time to finish, 163 | please avoid the dependency and strip it if necessary. 164 | 165 | Not all test suites support pytest-xdist. Particularly, it requires 166 | that the tests are written not to collide one with another. Sometimes, 167 | xdist may also cause instability of individual tests. In some cases, 168 | it is possible to work around this using the same solution as when 169 | `dealing with flaky tests`_. 170 | 171 | When only a few tests are broken or unstable because of pytest-xdist, 172 | it is possible to use it and deselect the problematic tests. It is up 173 | to the maintainer's discretion to decide whether this is justified. 174 | 175 | 176 | Dealing with flaky tests 177 | ======================== 178 | A flaky test is a test that sometimes passes, and sometimes fails 179 | with a false positive result. Often tests are flaky because of too 180 | steep timing requirements or race conditions. While generally it is 181 | preferable to fix the underlying issue (e.g. by increasing timeouts), 182 | it is not always easy. 183 | 184 | Sometimes upstreams use such packages as ``dev-python/flaky`` 185 | or ``dev-python/pytest-rerunfailures`` to mark tests as flaky and have 186 | them rerun a few minutes automatically. If upstream does not do that, 187 | it is also possible to force a similar behavior locally in the ebuild:: 188 | 189 | python_test() { 190 | # plugins make tests slower, and more fragile 191 | local -x PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 192 | # some tests are very fragile to timing 193 | epytest -p rerunfailures --reruns=10 --reruns-delay=2 194 | } 195 | 196 | Note that the snippet above also disables plugin autoloading to speed 197 | tests up and therefore reduce their flakiness. Sometimes forcing 198 | explicit rerun also makes it possible to use xdist on packages that 199 | otherwise randomly fail with it. 200 | 201 | 202 | .. index:: EPYTEST_TIMEOUT 203 | 204 | Using pytest-timeout to prevent deadlocks (hangs) 205 | ================================================= 206 | pytest-timeout_ plugin adds an option to terminate the test if its 207 | runtime exceeds the specified limit. Some packages decorate specific 208 | tests with timeouts; however, it is also possible to set a baseline 209 | timeout for all tests. 210 | 211 | A timeout causes the test run to fail, and therefore using it is 212 | not generally necessary for test suites that are working correctly. 213 | If individual tests are known to suffer from unfixable hangs, it is 214 | preferable to deselect them. However, setting a general timeout is 215 | recommended when a package is particularly fragile, or has suffered 216 | deadlocks in the past. A proactive setting can prevent it from hanging 217 | and blocking arch testing machines. 218 | 219 | The plugin can be enabled via setting ``EPYTEST_TIMEOUT`` to the timeout 220 | in seconds, prior to calling ``distutils_enable_tests``. This ensures 221 | that an appropriate depedency is added, and that ``epytest`` adds 222 | necessary command-line options. 223 | 224 | .. code-block:: 225 | 226 | : ${EPYTEST_TIMEOUT:=1800} 227 | distutils_enable_tests pytest 228 | 229 | The timeout applies to every test separately, i.e. the above example 230 | will cause a single test to time out after 30 minutes. If multiple 231 | tests hang, the total run time will multiply consequently. 232 | 233 | When deciding on a timeout value, please take into the consideration 234 | that the tests may be run on a low performance hardware, and on a busy 235 | system, and choose an appropriately high value. 236 | 237 | It is a good idea to use the default assignment form, as in the snippet 238 | above, as that permits the user to easily override the timeout 239 | if necessary. 240 | 241 | .. Note:: 242 | 243 | ``EPYTEST_TIMEOUT`` can also be set by user in ``make.conf`` 244 | or in the calling environment. This can be used as a general 245 | protection against hanging test suites. However, please note that 246 | this does not control dependencies, and therefore the user may need 247 | to install ``dev-python/pytest-timeout`` explicitly. 248 | 249 | 250 | Avoiding dependencies on other pytest plugins 251 | ============================================= 252 | There is a number of pytest plugins that have little value to Gentoo 253 | users. They include plugins for test coverage 254 | (``dev-python/pytest-cov``), coding style (``dev-python/pytest-flake8``) 255 | and more. Generally, packages should avoid using those plugins. 256 | 257 | .. Warning:: 258 | 259 | As of 2022-01-24, ``epytest`` disables a few undesirable plugins 260 | by default. As a result, developers have a good chance 261 | of experiencing failures due to hardcoded pytest options first, 262 | even if they have the relevant plugins installed. 263 | 264 | If your package *really* needs to use the specific plugin, you need 265 | to pass ``-p `` explicitly to reenable it. 266 | 267 | In some cases, upstream packages only list them as dependencies 268 | but do not use them automatically. In other cases, you will need 269 | to strip options enabling them from ``pytest.ini`` or ``setup.cfg``. 270 | 271 | :: 272 | 273 | src_prepare() { 274 | sed -i -e 's:--cov=wheel::' setup.cfg || die 275 | distutils-r1_src_prepare 276 | } 277 | 278 | 279 | TypeError: _make_test_flaky() got an unexpected keyword argument 'reruns' 280 | ========================================================================= 281 | If you see a test error resembling the following:: 282 | 283 | TypeError: _make_test_flaky() got an unexpected keyword argument 'reruns' 284 | 285 | This means that the tests are being run via flaky_ plugin while 286 | the package in question expects pytest-rerunfailures_. This is 287 | because both plugins utilize the same ``@pytest.mark.flaky`` marker 288 | but support different set of arguments. 289 | 290 | To resolve the problem, explicitly disable the ``flaky`` plugin and make 291 | sure to depend on ``dev-python/pytest-rerunfailures``:: 292 | 293 | BDEPEND=" 294 | test? ( 295 | dev-python/pytest-rerunfailures[${PYTHON_USEDEP}] 296 | )" 297 | 298 | python_test() { 299 | epytest -p no:flaky 300 | } 301 | 302 | 303 | ImportPathMismatchError 304 | ======================= 305 | An ``ImportPathMismatchError`` generally indicates that the same Python 306 | module (or one that supposedly looks the same) has been loaded twice 307 | using different paths, e.g.:: 308 | 309 | E _pytest.pathlib.ImportPathMismatchError: ('path', '/usr/lib/pypy3.7/site-packages/path', PosixPath('/tmp/portage/dev-python/jaraco-path-3.3.1/work/jaraco.path-3.3.1/jaraco/path.py')) 310 | 311 | These problems are usually caused by pytest test discovery getting 312 | confused by namespace packages. In this case, the ``jaraco`` directory 313 | is a Python 3-style namespace but pytest is treating it as a potential 314 | test directory. Therefore, instead of loading it as ``jaraco.path`` 315 | relatively to the top directory, it loads it as ``path`` relatively 316 | to the ``jaraco`` directory. 317 | 318 | The simplest way to resolve this problem is to restrict the test 319 | discovery to the actual test directories, e.g.:: 320 | 321 | python_test() { 322 | epytest test 323 | } 324 | 325 | or:: 326 | 327 | python_test() { 328 | epytest --ignore jaraco 329 | } 330 | 331 | 332 | Failures due to missing files in temporary directories 333 | ====================================================== 334 | As of 2024-01-05, ``epytest`` overrides the default temporary directory 335 | retention policy of pytest. By default, directories from successful 336 | tests are removed immediately, and the temporary directories 337 | from the previous test run are replaced by the subsequent test run. 338 | This frequently reduces disk space requirements from test suites, 339 | but it can rarely cause tests to fail. 340 | 341 | If you notice test failures combined with indications that a file was 342 | not found, and especially regarding the pytest temporary directories, 343 | try if overriding the retention policy helps, e.g.:: 344 | 345 | python_test() { 346 | epytest -o tmp_path_retention_policy=all 347 | } 348 | 349 | 350 | fixture '...' not found 351 | ======================= 352 | Most of the time, a missing fixture indicates that some pytest plugin 353 | is not installed. In rare cases, it can signify an incompatible pytest 354 | version or package issue. 355 | 356 | The following table maps common fixture names to their respective 357 | plugins. 358 | 359 | =================================== ==================================== 360 | Fixture name Package 361 | =================================== ==================================== 362 | event_loop dev-python/pytest-asyncio 363 | freezer dev-python/pytest-freezegun 364 | httpbin dev-python/pytest-httpbin 365 | loop dev-python/pytest-aiohttp 366 | mocker dev-python/pytest-mock 367 | =================================== ==================================== 368 | 369 | 370 | .. index:: filterwarnings 371 | .. index:: Werror 372 | .. index:: pytest.warns 373 | 374 | Warnings and ``pytest.raises()`` 375 | ================================ 376 | Some projects set pytest to raise on warnings, using options such as: 377 | 378 | .. code-block:: toml 379 | 380 | filterwarnings = [ 381 | "error", 382 | # ... 383 | ] 384 | 385 | This may be desirable for upstream CI systems, as it ensures that pull 386 | requests do not introduce new warnings, and that any deprecations are 387 | handled promptly. However, it is undesirable for downstream testing, 388 | as new deprecations in dependencies can cause the existing versions 389 | to start failing. 390 | 391 | To avoid this problem, ``epytest`` explicitly forces ``-Wdefault``. 392 | Most of the time, this does not cause any issues, besides causing pytest 393 | to verbosely report warnings that are normally ignored by the test 394 | suite. However, if some packages incorrectly use ``pytest.raises()`` 395 | to check for warnings, their test suites will fail, for example:: 396 | 397 | ============================= FAILURES ============================= 398 | ________________ test_ser_ip_with_unexpected_value _________________ 399 | 400 | def test_ser_ip_with_unexpected_value() -> None: 401 | ta = TypeAdapter(ipaddress.IPv4Address) 402 | 403 | > with pytest.raises(UserWarning, match='serialized value may not be as expected.'): 404 | E Failed: DID NOT RAISE 405 | 406 | tests/test_types.py:6945: Failed 407 | ========================= warnings summary ========================= 408 | tests/test_types.py::test_ser_ip_with_unexpected_value 409 | /tmp/pydantic/pydantic/type_adapter.py:458: UserWarning: Pydantic serializer warnings: 410 | PydanticSerializationUnexpectedValue(Expected `` but got `` with value `'123'` - serialized value may not be as expected.) 411 | return self.serializer.to_python( 412 | 413 | -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html 414 | 415 | The solution is to replace ``pytest.raises()`` with more correct 416 | ``pytest.warns()``. The latter will work correctly, independently 417 | of ``filterwarnings`` value. 418 | 419 | 420 | .. _custom pytest markers: 421 | https://docs.pytest.org/en/stable/example/markers.html 422 | .. _pytest-runner: https://pypi.org/project/pytest-runner/ 423 | .. _pytest-xdist: https://pypi.org/project/pytest-xdist/ 424 | .. _pytest-timeout: https://pypi.org/project/pytest-timeout/ 425 | .. _flaky: https://github.com/box/flaky/ 426 | .. _pytest-rerunfailures: 427 | https://github.com/pytest-dev/pytest-rerunfailures/ 428 | -------------------------------------------------------------------------------- /qawarn.rst: -------------------------------------------------------------------------------- 1 | ====================== 2 | QA checks and warnings 3 | ====================== 4 | 5 | .. highlight:: text 6 | 7 | This section explains Python-related QA checks and the resulting QA 8 | warnings that can be output while running the package manager or related 9 | tooling. 10 | 11 | 12 | Improved QA warning reporting in Portage 13 | ======================================== 14 | Normally, Portage outputs QA warnings at specific phases of the build 15 | process. They are usually interspersed with other verbose output, 16 | and they are easy to miss, especially when building multiple packages 17 | in a single batch. 18 | 19 | To make them easier to catch, Portage's elog system can be used 20 | to repeat all the QA warnings once emerge exits. The required "echo" 21 | module is already enabled by default, however it skips QA warnings 22 | by default. To change that, set in your ``make.conf``: 23 | 24 | .. code-block:: bash 25 | 26 | PORTAGE_ELOG_CLASSES="log warn error qa" 27 | 28 | For more information on using Portage's elog system, please refer 29 | to make.conf.example_ included in the Portage distribution. 30 | 31 | 32 | Compiled bytecode-related warnings 33 | ================================== 34 | To improve performance, the Python interpreter compiles Python sources 35 | into bytecode. CPython and PyPy3 feature three optimization levels 36 | that impact the bytecode size: 37 | 38 | 1. no optimizations (the default) 39 | 40 | 2. ``-O`` that removes assert statements from code 41 | 42 | 3. ``-OO`` that removes assert statements and docstrings 43 | 44 | Normally, the compiled bytecode is stored on disk in ``.pyc`` files 45 | (historically, Python 2 has also used ``.pyo`` files). 46 | When these files are present and up-to-date, Python loads 47 | the precompiled bytecode from them rather than creating it from 48 | the source code, improving module loading time. When they are missing, 49 | Python normally creates them automatically if it has write permissions 50 | to the respective directory. 51 | 52 | In Gentoo, we aim for packages to always byte-compile all their Python 53 | files to all supported optimization levels. Besides improving module 54 | loading performance, it ensures that the Python interpreters will not 55 | attempt to write compiled bytecode themselves, effectively creating 56 | files that are not monitored by the package manager (and therefore e.g. 57 | are not removed when the respective package is uninstalled) or causing 58 | sandbox violations while building other packages. 59 | 60 | The Gentoo repository features a QA check to ensure that all installed 61 | Python modules are byte-compiled for all optimization levels supported 62 | by the respective Python interpreter, and that no stray compiled files 63 | exist. This check is implemented using an auxiliary command found 64 | in ``app-portage/gpep517-7`` and newer. This check can also be run 65 | manually (with machine-readable output) using e.g.:: 66 | 67 | $ python3.10 -m gpep517 verify-pyc --destdir=/tmp/portage/dev-python/trimesh-3.12.9/image 68 | missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-1.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py 69 | missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-2.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py 70 | missing:/usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.pyc:/usr/lib/python3.10/site-packages/trimesh/resources/templates/blender_boolean.py 71 | 72 | 73 | Modules are not byte-compiled 74 | ----------------------------- 75 | The most common QA warning that can be noticed while building packages 76 | indicates that at least some of the expected ``.pyc`` files are missing. 77 | For example:: 78 | 79 | * QA Notice: This package installs one or more Python modules that are 80 | * not byte-compiled. 81 | * The following files are missing: 82 | * 83 | * /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-1.pyc 84 | * /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.opt-2.pyc 85 | * /usr/lib/python3.10/site-packages/trimesh/resources/templates/__pycache__/blender_boolean.cpython-310.pyc 86 | 87 | * QA Notice: This package installs one or more Python modules that are 88 | * not byte-compiled. 89 | * The following files are missing: 90 | * 91 | * /usr/lib/python3.10/site-packages/blueman/__pycache__/Constants.cpython-310.opt-2.pyc 92 | * /usr/lib/python3.10/site-packages/blueman/__pycache__/DeviceClass.cpython-310.opt-2.pyc 93 | * /usr/lib/python3.10/site-packages/blueman/__pycache__/Functions.cpython-310.opt-2.pyc 94 | * /usr/lib/python3.10/site-packages/blueman/__pycache__/Sdp.cpython-310.opt-2.pyc 95 | [...] 96 | 97 | There are three common causes for these warnings: 98 | 99 | 1. The package's build system nor the ebuild do not byte-compile 100 | installed Python modules — the warning lists all optimization levels 101 | for all installed modules. 102 | 103 | 2. The package's build system byte-compiles installed modules only for 104 | a subset of optimization levels — the warning lists all modules 105 | but only for a subset of levels (the second example in the example 106 | snippet). 107 | 108 | 3. The package installs ``.py`` files with incorrect syntax that can not 109 | be byte-compiled and usually trigger syntax errors during the install 110 | phase (the first example in the above snippet). 111 | 112 | For the first two cases, the better solution is to patch the respective 113 | build system to perform byte-compilation for all optimization levels. 114 | An acceptable workaround is to call ``python_optimize`` from the ebuild 115 | (note that in PEP517 mode, distutils-r1 does that unconditionally). 116 | 117 | For the third case, the only real solution is to submit a fix upstream 118 | that renames files that do not contain valid Python modules to use 119 | another suffix. For example, the template triggering the QA warning 120 | in trimesh package could be renamed from ``.py`` to ``.py.tmpl``. 121 | 122 | 123 | Stray compiled bytecode 124 | ----------------------- 125 | The following QA warning indicates that there are stray ``.pyc`` files 126 | that are not clearly matching any installed Python module-implementation 127 | pair:: 128 | 129 | * QA Notice: This package installs one or more compiled Python modules 130 | * that do not match installed modules (or their implementation). 131 | * The following files are stray: 132 | * 133 | * /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-35.pyc 134 | * /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-36.pyc 135 | * /usr/lib/python3.10/site-packages/SCons/Tool/docbook/__pycache__/__init__.cpython-38.pyc 136 | 137 | There are two common causes for this: 138 | 139 | 1. The package is shipping precompiled ``.pyc`` files and installing 140 | them along with ``.py`` modules. The ebuild should remove the stray 141 | files in ``src_prepare`` then. 142 | 143 | 2. The ebuild is attempting to remove some ``.py`` files after they have 144 | been byte-compiled. It needs to be modified to either remove them 145 | prior to the byte-compilation stage, or to fix the build system 146 | not to install them in the first place. 147 | 148 | 149 | Stray top-level files in site-packages 150 | ====================================== 151 | distutils-r1 checks for the common mistake of installing unexpected 152 | files that are installed top-level into the site-packages directory. 153 | An example error due to that looks like the following:: 154 | 155 | * The following unexpected files/directories were found top-level 156 | * in the site-packages directory: 157 | * 158 | * /usr/lib/python3.10/site-packages/README.md 159 | * /usr/lib/python3.10/site-packages/LICENSE 160 | * /usr/lib/python3.10/site-packages/CHANGELOG 161 | * 162 | * This is most likely a bug in the build system. More information 163 | * can be found in the Python Guide: 164 | * https://projects.gentoo.org/python/guide/qawarn.html#stray-top-level-files-in-site-packages 165 | 166 | In general, it is desirable to prepare a fix for the build system 167 | and submit it upstream. However, it is acceptable to remove the files 168 | locally in the ebuild while waiting for a release with the fix. 169 | 170 | The subsequent sections describe the common causes and the suggested 171 | fixes. 172 | 173 | 174 | Example for test packages installed by setuptools 175 | ------------------------------------------------- 176 | Many packages using the setuptools build system utilize the convenient 177 | ``find_packages()`` method to locate the Python sources. In some cases, 178 | this method also wrongly grabs top-level test directories or other files 179 | that were not intended to be installed. 180 | 181 | For example, the following invocation will install everything that looks 182 | like a Python package from the source tree: 183 | 184 | .. code-block:: python 185 | 186 | setup( 187 | packages=find_packages()) 188 | 189 | The correct fix for this problem is to add an ``exclude`` parameter 190 | that restricts the installed package list, for example: 191 | 192 | .. code-block:: python 193 | 194 | setup( 195 | packages=find_packages(exclude=["tests", "tests.*"])) 196 | 197 | Note that if the top-level ``tests`` package has any subpackages, both 198 | ``tests`` and ``tests.*`` need to be listed. 199 | 200 | If ``setup.cfg`` is used instead, the excludes are specified as follows: 201 | 202 | .. code-block:: ini 203 | 204 | [options.packages.find] 205 | exclude = 206 | tests 207 | tests.* 208 | 209 | If ``pyproject.toml`` is used: 210 | 211 | .. code-block:: toml 212 | 213 | [tool.setuptools.packages.find] 214 | exclude = [ 215 | "tests", 216 | "tests.*", 217 | ] 218 | 219 | For reference, see `custom discovery in setuptools documentation`_. 220 | 221 | 222 | Documentation files installed by Poetry 223 | --------------------------------------- 224 | It is a relatively common problem that packages using the Poetry build 225 | system are installing documentation files (such as ``README``) 226 | to the site-packages directory. This is because of incorrect 227 | ``include`` use in ``pyproject.toml``. For example, consider 228 | the following configuration: 229 | 230 | .. code-block:: toml 231 | 232 | include = [ 233 | "CHANGELOG", 234 | "README.md", 235 | "LICENSE" 236 | ] 237 | 238 | The author meant to include these files in the source distribution 239 | packages. However, the ``include`` key applies to wheels as well, 240 | effectively including them in files installed into ``site-packages``. 241 | 242 | To fix that, you need to specify file formats explicitly, for every 243 | entry: 244 | 245 | .. code-block:: toml 246 | 247 | include = [ 248 | { path = "CHANGELOG", format = "sdist" }, 249 | { path = "README.md", format = "sdist" }, 250 | { path = "LICENSE", format = "sdist" }, 251 | ] 252 | 253 | For reference, see `include and exclude in Poetry documentation`_. 254 | 255 | 256 | Deprecated PEP 517 backends 257 | =========================== 258 | 259 | flit.buildapi 260 | ------------- 261 | Some packages are still found using the historical flit build backend. 262 | Their ``pyproject.toml`` files contain a section similar 263 | to the following: 264 | 265 | .. code-block:: toml 266 | 267 | [build-system] 268 | requires = ["flit"] 269 | build-backend = "flit.buildapi" 270 | 271 | This backend requires installing the complete flit package manager. 272 | Instead, the package should be fixed upstream to use flit_core 273 | per `flit build system section documentation`_ instead: 274 | 275 | .. code-block:: toml 276 | 277 | [build-system] 278 | requires = ["flit_core"] 279 | build-backend = "flit_core.buildapi" 280 | 281 | flit_core produces identical artifacts to flit. At the same time, it 282 | reduces the build-time dependency footprint and therefore makes isolated 283 | PEP 517 builds faster. 284 | 285 | 286 | poetry.masonry.api 287 | ------------------ 288 | A similar problem applies to the packages using poetry. The respective 289 | ``pyproject.toml`` files contain: 290 | 291 | .. code-block:: toml 292 | 293 | [build-system] 294 | requires = ["poetry>=0.12"] 295 | build-backend = "poetry.masonry.api" 296 | 297 | Instead, the lightweight poetry-core module should be used per `poetry 298 | PEP-517 documentation`_: 299 | 300 | .. code-block:: toml 301 | 302 | [build-system] 303 | requires = ["poetry_core>=1.0.0"] 304 | build-backend = "poetry.core.masonry.api" 305 | 306 | poetry-core produces identical artifacts to poetry. It has smaller 307 | dependency footprint and makes isolated builds much faster. 308 | 309 | 310 | setuptools.build_meta:__legacy__ 311 | -------------------------------- 312 | Some packages using setuptools specify the following: 313 | 314 | .. code-block:: toml 315 | 316 | [build-system] 317 | requires = ["setuptools>=40.8.0", "wheel"] 318 | build-backend = "setuptools.build_meta:__legacy__" 319 | 320 | This is incorrect, as the legacy backend is intended to be used only 321 | as an implicit fallback. All packages should be using the regular 322 | backend instead: 323 | 324 | .. code-block:: toml 325 | 326 | [build-system] 327 | requires = ["setuptools>=40.8.0"] 328 | build-backend = "setuptools.build_meta" 329 | 330 | Please also note that the ``wheel`` package should *not* be listed 331 | as a dependency, as it is an implementation detail and it was always 332 | implicitly returned by the backend. Unfortunately, due to prolonged 333 | documentation error, a very large number of packages still specifies it, 334 | and other packages tend to copy that mistake. 335 | 336 | 337 | .. _make.conf.example: 338 | https://gitweb.gentoo.org/proj/portage.git/tree/cnf/make.conf.example#n330 339 | .. _custom discovery in setuptools documentation: 340 | https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#custom-discovery 341 | .. _include and exclude in Poetry documentation: 342 | https://python-poetry.org/docs/pyproject/#include-and-exclude 343 | .. _flit build system section documentation: 344 | https://flit.readthedocs.io/en/latest/pyproject_toml.html#build-system-section 345 | .. _poetry PEP-517 documentation: 346 | https://python-poetry.org/docs/pyproject/#poetry-and-pep-517 347 | -------------------------------------------------------------------------------- /single.rst: -------------------------------------------------------------------------------- 1 | ======================================= 2 | python-single-r1 — single-impl packages 3 | ======================================= 4 | 5 | .. highlight:: bash 6 | 7 | The ``python-single-r1`` eclass is used to install single-impl packages. 8 | It is probably the easiest eclass to use, and it is recommended over 9 | ``python-r1`` whenever multi-impl support would add unnecessary 10 | complexity. However, for packages using distutils or a similar Python 11 | build system, ``distutils-r1`` eclass should be used instead. 12 | 13 | Eclass reference: `python-single-r1.eclass(5)`_ 14 | 15 | 16 | Basic use for unconditional Python 17 | ================================== 18 | The defining feature of this eclass is that it defines a ``pkg_setup`` 19 | phase. It normally calls ``python_setup`` function in order 20 | to determine the interpreter selected by user, and set the global 21 | environment appropriately. 22 | 23 | This means that a most trivial package using an autotools-compatible 24 | build system along with unconditional dependency on Python could look 25 | like the following: 26 | 27 | .. code-block:: bash 28 | :emphasize-lines: 6,8,17,20 29 | 30 | # Copyright 1999-2024 Gentoo Authors 31 | # Distributed under the terms of the GNU General Public License v2 32 | 33 | EAPI=8 34 | 35 | PYTHON_COMPAT=( python3_{10..13} ) 36 | 37 | inherit python-single-r1 38 | 39 | DESCRIPTION="Scripts to prepare and plot VOACAP propagation predictions" 40 | HOMEPAGE="https://www.qsl.net/h/hz1jw//pythonprop/" 41 | SRC_URI="mirror://sourceforge/${PN}/${P}.tar.gz" 42 | 43 | LICENSE="GPL-2+" 44 | SLOT="0" 45 | KEYWORDS="~amd64 ~x86" 46 | REQUIRED_USE="${PYTHON_REQUIRED_USE}" 47 | 48 | RDEPEND=" 49 | ${PYTHON_DEPS} 50 | ... 51 | " 52 | BDEPEND="${RDEPEND}" 53 | 54 | This ebuild demonstrates the absolute minimum working code. Only 55 | the four highlighted lines are specific to Python eclasses, plus 56 | the implicitly exported ``pkg_setup`` phase. 57 | 58 | 59 | .. index:: PYTHON_SINGLE_USEDEP 60 | .. index:: PYTHON_USEDEP; python-single-r1 61 | .. index:: python_gen_cond_dep; for python-single-r1 62 | 63 | Dependencies 64 | ============ 65 | When depending on other Python packages, USE dependencies need to be 66 | declared in order to ensure that the dependencies would be built against 67 | the Python implementation used for the package. The exact dependency 68 | string depends on whether the target package is single-impl 69 | or multi-impl. 70 | 71 | When depending on other single-impl packages, the eclass-defined 72 | ``${PYTHON_SINGLE_USEDEP}`` variable can be used to inject the correct 73 | USE dependency:: 74 | 75 | RDEPEND=" 76 | ... 77 | dev-python/basemap[${PYTHON_SINGLE_USEDEP}] 78 | " 79 | 80 | When depending on multi-impl packages, a more complex construct must 81 | be used. The ``python_gen_cond_dep`` generator function is used 82 | to copy the specified dependency template for all supported 83 | implementations, and substitute ``${PYTHON_USEDEP}`` template inside 84 | it:: 85 | 86 | RDEPEND=" 87 | ... 88 | $(python_gen_cond_dep ' 89 | dev-python/matplotlib-python2[gtk2,${PYTHON_USEDEP}] 90 | ') 91 | " 92 | 93 | Please note that in this context, ``${...}`` is used as a literal 94 | template substitution key, so it must be escaped to prevent bash from 95 | substituting it immediately. In the above example, single quotes 96 | are used for this purpose. When other variables are used, double quotes 97 | with explicit escapes have to be used:: 98 | 99 | RDEPEND=" 100 | ... 101 | $(python_gen_cond_dep " 102 | dev-python/wxpython:${WX_GTK_VER}[\${PYTHON_USEDEP}] 103 | ")" 104 | 105 | As demonstrated above, the USE dependency string can be combined with 106 | other USE dependencies. ``PYTHON_SINGLE_USEDEP`` can be used both 107 | inside and outside ``python_gen_cond_dep``, while ``PYTHON_USEDEP`` only 108 | inside it. 109 | 110 | 111 | Conditional Python use 112 | ====================== 113 | The examples so far assumed that Python is used unconditionally. 114 | If Python support is conditional to a USE flag, appropriate USE 115 | conditionals need to be used in metadata variables, and ``pkg_setup`` 116 | needs to be rewritten to call the default implementation conditionally: 117 | 118 | .. code-block:: bash 119 | :emphasize-lines: 17,19,24,25,27-31,34,39 120 | 121 | # Copyright 1999-2024 Gentoo Authors 122 | # Distributed under the terms of the GNU General Public License v2 123 | 124 | EAPI=8 125 | 126 | PYTHON_COMPAT=( python3_{11..12} ) 127 | 128 | inherit python-single-r1 129 | 130 | DESCRIPTION="Yet more Objects for (High Energy Physics) Data Analysis" 131 | HOMEPAGE="http://yoda.hepforge.org/" 132 | SRC_URI="http://www.hepforge.org/archive/${PN}/${P}.tar.bz2" 133 | 134 | LICENSE="GPL-2" 135 | SLOT="0/${PV}" 136 | KEYWORDS="~amd64 ~x86 ~amd64-linux ~x86-linux" 137 | IUSE="python root" 138 | REQUIRED_USE=" 139 | python? ( ${PYTHON_REQUIRED_USE} ) 140 | root? ( python ) 141 | " 142 | 143 | RDEPEND=" 144 | python? ( ${PYTHON_DEPS} ) 145 | root? ( sci-physics/root:=[python=,${PYTHON_SINGLE_USEDEP}] )" 146 | DEPEND="${RDEPEND} 147 | python? ( 148 | $(python_gen_cond_dep ' 149 | dev-python/cython[${PYTHON_USEDEP}] 150 | ') 151 | )" 152 | 153 | pkg_setup() { 154 | use python && python-single-r1_pkg_setup 155 | } 156 | 157 | src_configure() { 158 | econf \ 159 | $(use_enable python pyext) \ 160 | $(use_enable root) 161 | } 162 | 163 | 164 | A hybrid: build-time + conditional runtime 165 | ========================================== 166 | A fairly common pattern is for Python to be required unconditionally 167 | at build time but only conditionally at runtime. This happens e.g. when 168 | the package is calling some helper scripts at build time, and optionally 169 | installing Python bindings. In this case, the build time dependency 170 | is expressed unconditionally, and the runtime dependency is made 171 | USE-conditional: 172 | 173 | .. code-block:: bash 174 | :emphasize-lines: 18,19,23,30,38 175 | 176 | # Copyright 1999-2024 Gentoo Authors 177 | # Distributed under the terms of the GNU General Public License v2 178 | 179 | EAPI=8 180 | 181 | PYTHON_COMPAT=( python3_{10..13} ) 182 | PYTHON_REQ_USE="threads(+)" 183 | 184 | inherit waf-utils python-single-r1 185 | 186 | DESCRIPTION="Samba talloc library" 187 | HOMEPAGE="https://talloc.samba.org/" 188 | SRC_URI="https://www.samba.org/ftp/${PN}/${P}.tar.gz" 189 | 190 | LICENSE="GPL-3 LGPL-3+ LGPL-2" 191 | SLOT="0" 192 | KEYWORDS="~alpha amd64 arm ~arm64 ~hppa ia64 ~m68k ~mips ppc ppc64 ~riscv ~s390 ~sh ~sparc x86 ~amd64-linux ~x86-linux ~x64-macos ~sparc-solaris ~x64-solaris" 193 | IUSE="+python" 194 | REQUIRED_USE="${PYTHON_REQUIRED_USE}" 195 | 196 | RDEPEND=" 197 | ... 198 | python? ( ${PYTHON_DEPS} ) 199 | " 200 | DEPEND=" 201 | ${RDEPEND} 202 | ... 203 | " 204 | BDEPEND=" 205 | ${PYTHON_DEPS} 206 | ... 207 | " 208 | 209 | WAF_BINARY="${S}/buildtools/bin/waf" 210 | 211 | src_configure() { 212 | local extra_opts=( 213 | $(usex python '' --disable-python) 214 | ) 215 | waf-utils_src_configure "${extra_opts[@]}" 216 | } 217 | 218 | Note that eclass-exported ``pkg_setup`` is used unconditionally here. 219 | 220 | 221 | Multiple USE conditions 222 | ======================= 223 | Finally, let's give an example of a package where Python is needed 224 | for two independent conditions. To make it more complex, one of them 225 | applies to build time (tests) while the other to runtime (bindings). 226 | 227 | .. code-block:: bash 228 | :emphasize-lines: 17,20,21,25,28,32-34,39,40 229 | 230 | # Copyright 1999-2020 Gentoo Authors 231 | # Distributed under the terms of the GNU General Public License v2 232 | 233 | EAPI=8 234 | 235 | PYTHON_COMPAT=( python3_{10..13} ) 236 | 237 | inherit cmake python-single-r1 238 | 239 | DESCRIPTION="Sound design and signal processing system for composition and performance" 240 | HOMEPAGE="https://csound.github.io/" 241 | SRC_URI="https://dev.gentoo.org/~fordfrog/distfiles/${P}-distributable.tar.xz" 242 | 243 | LICENSE="LGPL-2.1 doc? ( FDL-1.2+ )" 244 | SLOT="0" 245 | KEYWORDS="~amd64 ~x86" 246 | IUSE="python test" 247 | RESTRICT="!test? ( test )" 248 | REQUIRED_USE=" 249 | python? ( ${PYTHON_REQUIRED_USE} ) 250 | test? ( ${PYTHON_REQUIRED_USE} )" 251 | 252 | BDEPEND=" 253 | python? ( dev-lang/swig ) 254 | test? ( ${PYTHON_DEPS} ) 255 | " 256 | RDEPEND=" 257 | python? ( ${PYTHON_DEPS} ) 258 | " 259 | 260 | pkg_setup() { 261 | if use python || use test ; then 262 | python-single-r1_pkg_setup 263 | fi 264 | } 265 | 266 | src_configure() { 267 | local mycmakeargs=( 268 | -DBUILD_PYTHON_INTERFACE=$(usex python) 269 | -DBUILD_PYTHON_OPCODES=$(usex python) 270 | ) 271 | 272 | cmake_src_configure 273 | } 274 | 275 | Please note that in general, the condition in ``pkg_setup`` must match 276 | the one in ``REQUIRED_USE``, and that one is a superset of conditions 277 | used in dependencies. 278 | 279 | 280 | Manual install 281 | ============== 282 | Some packages do not include Python files in their build systems, 283 | or do not install all of them. In this case, the necessary files 284 | can be installed via one of the installation helpers. 285 | 286 | .. code-block:: bash 287 | :emphasize-lines: 23,24 288 | 289 | # Copyright 1999-2020 Gentoo Authors 290 | # Distributed under the terms of the GNU General Public License v2 291 | 292 | EAPI=8 293 | 294 | PYTHON_COMPAT=( python3_{10..13} ) 295 | 296 | inherit python-single-r1 297 | 298 | DESCRIPTION="Arabic dictionary based on the DICT protocol" 299 | HOMEPAGE="https://www.arabeyes.org/Duali" 300 | SRC_URI="mirror://sourceforge/arabeyes/${P}.tar.bz2" 301 | 302 | LICENSE="BSD" 303 | SLOT="0" 304 | KEYWORDS="~alpha amd64 ~hppa ~ia64 ~mips ~ppc ~sparc x86" 305 | REQUIRED_USE="${PYTHON_REQUIRED_USE}" 306 | 307 | DEPEND="${PYTHON_DEPS}" 308 | RDEPEND="${DEPEND}" 309 | 310 | src_install() { 311 | python_domodule pyduali 312 | python_doscript duali dict2db trans2arabic arabic2trans 313 | } 314 | 315 | 316 | .. _python-single-r1.eclass(5): 317 | https://devmanual.gentoo.org/eclass-reference/python-single-r1.eclass/index.html 318 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py3 3 | skipsdist = True 4 | 5 | [testenv] 6 | deps = 7 | sphinx 8 | whitelist_externals = 9 | make 10 | 11 | commands = 12 | make {posargs:html} 13 | --------------------------------------------------------------------------------