Browse Source

Initial version of the Guide

Signed-off-by: Michał Górny <>
Michał Górny 1 year ago
No known key found for this signature in database GPG Key ID: 639ADAE2329E240E
  1. 20
  2. 73
  3. 52
  4. 111
  5. 21
  6. 210
  7. 31
  8. 259


@ -0,0 +1,20 @@
# Minimal makefile for Sphinx documentation
# You can set these variables from the command line, and also
# from the environment for the first two.
SPHINXBUILD ?= sphinx-build
BUILDDIR = _build
# Put it first so that "make" without argument is like "make help".
.PHONY: help Makefile
# Catch-all target: route all unknown targets to Sphinx using the new
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
%: Makefile


@ -0,0 +1,73 @@
Common basics
.. highlight:: bash
The various eclasses in python-r1 try to follow a single design. You
will probably use more than one of them, so it is worthwhile to shortly
explain the common bits used by all of them, as well as the non-obvious
differences between them.
The ``PYTHON_COMPAT`` variable is used by all Python eclasses, and must
be declared in all ebuilds before they are inherited. It specifies
the list of Python implementations supported by the package.
The valid values are:
- ``pythonX_Y`` for CPython X.Y
- ``pypy3`` for PyPy3.
Typical use::
PYTHON_COMPAT=( python3_{6,7,8} pypy3 )
inherit python-single-r1
The ``python-any-r1``, ``python-single-r1`` and ``python-r1`` eclasses
all assume that the developer is responsible for explicitly putting
the dependency strings and USE requirements in correct variables.
This is meant to consistently cover packages that use Python
unconditionally and conditionally, at build time and at runtime.
For ``python-single-r1`` and ``python-r1``, the most basic form to use
Python unconditionally is to define the following::
For ``python-any-r1``, only build-time dependencies are used::
This does not apply to ``distutils-r1`` as it does the above assignment
by default.
Python environment
The eclasses commonly use the concept of *Python environment*. This
means a state of environment enforcing a particular Python
implementation. Whenever the ebuild code is run inside this
environment, ``EPYTHON`` variable indicates which implementation
is being used (by its executable name, e.g. ``python3.8``).
Additionally, ``PYTHON`` provides the absolute path to the interpreter
(however, using ``EPYTHON`` is preferable whenever possible). Wrappers
for ``python``, ``pythonN`` and some common tools are provided in PATH,
and ``/usr/bin/python`` etc. also enforce the specific implementation
via python-exec (for programs that hardcode full path).
The environment can be either established in local scope, or globally.
The local scope generally applies to multi-impl packages, and is created
either by calls to ``python_foreach_impl`` from ``python-r1``, or inside
sub-phase functions in ``distutils-r1``. The global scope setup is done
via calling ``python_setup``, either directly or via default
``pkg_setup`` in ``python-any-r1`` and ``python-single-r1``.


@ -0,0 +1,52 @@
# Configuration file for the Sphinx documentation builder.
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
# import os
# import sys
# sys.path.insert(0, os.path.abspath('.'))
# -- Project information -----------------------------------------------------
project = 'Gentoo Python Guide'
copyright = '2020, Michał Górny'
author = 'Michał Górny'
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
# -- Options for HTML output -------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'alabaster'
# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']


@ -0,0 +1,111 @@
Choosing between Python eclasses
Build-time vs runtime use
The first basis for choosing Python eclass is whether Python is used
merely at build-time or at runtime as well.
A runtime use occurs if the package explicitly needs Python to be
installed along with it, in order for it to function correctly. This
generally happens if the package installs Python modules, extensions,
scripts, or executables calling the Python interpreter or linking
to libpython. This also applies to bash scripts or other executables
that call python inline.
A build-time use occurs if the package calls the Python interpreter
or any kind of aforementioned executables during package's build
(or install) phases.
If the package uses Python purely at build-time, the ``python-any-r1``
eclass is appropriate. Otherwise, ``python-single-r1``, ``python-r1``
or their derivatives are to be used.
A specific exception to that rule is when the package is only calling
external Python scripts directly (i.e. not via ``python /usr/bin/foo``).
If the called executables can be considered fully contained
dependency-wise, there is no need to use an eclass.
For example, when using ``dev-util/meson`` to build a package, there is
no need to use a Python eclass since Meson abstracts away its Pythonic
implementation details and works as a regular executable for your
packages. However, ``dev-util/scons`` requires Python eclass since it
loads Python code from the package and a compatible Python version must
be enforced.
Single-impl vs multi-impl
The second important basis for packages using Python at runtime is
whether the package in question should support multi-implementation
install or not.
A *single-impl* package is a package requiring the user to choose
exactly one Python implementation to be built against. This means
that the scripts installed by that package will be run via specified
Python interpreter, and that the modules and extensions will be
importable from it only. The package's Python reverse dependencies will
also have to use the same implementation. Since the package can't
support having more than one implementation enabled, its reverse
dependencies have to be simple-impl as well.
Single-impl packages use ``python-single-r1`` eclass. Writing ebuilds
for them is easier since it is generally sufficient to call setup
function early on, and the upstream build system generally takes care
of using selected Python version correctly. Making packages single-impl
is recommended when dealing with packages that are not purely written
for Python or have single-impl dependencies.
A *multi-impl* package allows user to enable multiple (preferably
any number of) implementations. The modules, extensions and scripts
installed by the package are installed separately for each enabled
implementation, and can therefore be used from any of them. The package
can have reverse dependencies enabling only a subset of its
Multi-impl packages use ``python-r1`` eclass. Ebuilds are more complex
since they need to explicitly repeat build and install steps for each
enabled implementation. Using this model is recommended for packages
providing Python modules or extensions only, or having multi-impl
reverse dependencies. In some cases supporting multi-impl build
requieres applying hacks, e.g. ``dev-libs/boost[python]`` uses
non-standard names to install ``libboost_python`` for multiple Python
The implementation for single-impl packages is selected
via ``PYTHON_SINGLE_TARGET``, while multi-impl uses ``PYTHON_TARGETS``.
These USE flag sets can be set independently to provide greater
flexibility for developers and end users.
Distutils and related build systems
The third basis for choosing an eclass is the build system used.
If the project uses one of Python-specific build systems, that is
distutils, setuptools, flit or poetry, the ``distutils-r1`` eclass
should be used instead of the other eclasses. As a rule of thumb,
this happens when either ```` or ``pyproject.toml`` file exists
in the distribution.
``distutils-r1`` builds on either ``python-r1`` or ``python-single-r1``,
therefore it can be used to create both multi-impl and single-impl
packages. It provides full set of default phase functions, making
writing ebuilds much easier.
A rule of thumb
As a rule of thumb, the following checklist can be used to determine
the eclass to use:
1. If the package has ```` or ``pyproject.toml`` file,
use ``distutils-r1``.
2. If the package primarily installs Python modules or extensions
or has multi-impl reverse dependencies, use ``python-r1``.
3. If the package (possibly conditionally) qualifies as using Python
at runtime, use ``python-single-r1``.
4. If the package uses Python at build time only, use ``python-any-r1``.


@ -0,0 +1,21 @@
Gentoo Python Guide
.. toctree::
:maxdepth: 2
:caption: Contents:
Indices and tables
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`


@ -0,0 +1,210 @@
Python interpreters
Versions of Python
By a *version of Python* we usually mean the variant of Python language
and standard library interface as used by a specific version
of CPython_, the reference implementation of Python.
Python versions are determined from the two first version components.
The major version is incremented when major incompatible changes are
introduced in the language, as was the case in Python 3. Along with
minor version changes, the new releases introduce new features
and remove deprecated APIs. The Python documentation generally
indicates when a particular API was added or deprecated, and when it
is planned to be removed.
Practically speaking, this means that a program written purely
for Python 2 is unlikely to work on Python 3, and requires major changes
to achieve compatibility. On the other hand, a program written for
Python 3.7 is very likely to work with Python 3.8, and reasonably likely
to support Python 3.6 as well. If that is not the case, minor changes
are usually sufficient to fix that.
For example, Python 3.7 introduced a new `importlib.resources`_ module.
If your program uses it, it will not work on Python 3.6 without
a backwards compatibility code.
Python 3.8 removed the deprecated `platform.linux_distribution()`_
function. If your program used it, it will not work on Python 3.8
without changes. However, it was deprecated since Python 3.5, so if you
were targetting 3.7, you should not have been using it in the first
Gentoo supports building packages against Python 2.7 and a shifting
window of 3-4 versions of Python 3. They are provided as slots
of ``dev-lang/python``.
Alternative Python implementations
CPython is the reference and most commonly used Python implementation.
However, there are other interpreters that aim to maintain reasonable
compatibility with it.
PyPy_ is an implementation of Python built using in-house RPython
language, using a Just-in-Time compiler to achieve better performance
(generally in long-running programs running a lot of Python code).
It maintains quite good compatibility with CPython, except when programs
rely on its implementation details or GC behavior.
PyPy upstream provides PyPy variants compatible with Python 2.7
and one version of Python 3. Gentoo supports building packages against
PyPy3. PyPy2.7 is provided as ``dev-python/pypy``, while PyPy3 is
provided as ``dev-python/pypy3``.
Jython_ is an implementation of Python written in Java. Besides being
a stand-alone Python interpreter, it supports bidirectional interaction
between Python and Java libraries.
Jython development is very slow paced, and it is currently bound
to Python 2.7. Gentoo does not support building packages for Jython
anymore. The interpreter is still provided as ``dev-java/jython``.
IronPython_ is an implementation of Python for the .NET framework.
Alike Jython, it supports bidirectional interaction between Python
and .NET Framework. It is currently bound to Python 2.7. It is not
packaged in Gentoo.
Brython_ is an implementation of Python 3 for client-side web
programming (in JavaScript). It provides a subset of Python 3 standard
library combined with access to DOM objects. It is packaged in Gentoo
as ``dev-python/brython``.
MicroPython_ is an implementation of Python 3 aimed for microcontrollers
and embedded environments. It aims to maintain some compatibility
with CPython while providing stripped down standard library
and additional modules to interface with hardware. It is packaged
as ``dev-lang/micropython``.
Tauthon_ is a fork of Python 2.7 that aims to backport new language
features and standard library modules while preserving backwards
compatibility with existing code. It is not packaged in Gentoo.
Support for multiple implementations
The support for simultaneously using multiple Python implementations
is implemented primarily through USE flags. The packages installing
or using Python files define either ``PYTHON_TARGETS``
or ``PYTHON_SINGLE_TARGET`` flags that permit user to choose which
implementations are used.
Modules and extensions are installed separately for each interpreter,
in its specific site-packages directory. This means that a package
can run using a specific target correctly only if all its dependencies
were also installed for the same implementation. This is enforced
via USE dependencies.
Additionally, ``dev-lang/python-exec`` provides a mechanism for
installing multiple variants of each Python script simultaneously. This
is necessary to support scripts that differ between Python versions
(particularly between Python 2 and Python 3) but it is also used
to prevent scripts from being called via unsupported interpreter
(i.e. one that does not have its accompanying modules or dependencies
This also implies that all installed Python scripts must have their
shebangs adjusted to use a specific Python interpreter (not ``python``
nor ``python3`` but e.g. ``python3.7``), and all other executables must
also be modified to call specific version of Python directly.
A common method of improving compatibility with older versions of Python
is to backport new standard library modules or features. Packages doing
that are generally called *backports*.
Ideally, backports copy the code from the standard library with minimal
changes, and provide a matching API. In some cases, new versions
of backports are released as the standard library changes, and their
usability extends from providing a missing module to extending older
version of the module. For example, the ``dev-python/funcsigs`` package
originally backported function signatures from Python 3.3 to older
versions, and afterwards was updated to backport new features from
Python 3.6, becoming useful to versions 3.3 through 3.5.
Sometimes, the opposite happens. ``dev-python/mock`` started
as a stand-alone package, and was integrated into the standard library
as unittest.mock_ later on. Afterwards, the external package became
a backport of the standard library module.
In some cases backports effectively replace external packages. Once
lzma_ module has been added to the standard library, its backport
``dev-python/backports-lzma`` has effectively replaced the competing
LZMA packages.
Individual backports differ by the level of compatibility with
the standard library provided, and therefore on the amount of additional
code needed in your program. The exact kind of dependencies used
depends on that.
``dev-python/ipaddress`` is a drop-in backport of the ipaddress_ module
from Python 3.3. It is using the same module name, so a code written
to use this module will work out-of-the-box on Python 2.7 if the package
is installed. As a side note, since Python always prefers built-in
modules over external packages, there is no point in enabling Python 3
in this package as the installed module would never be used.
Appropriately, you should depend on this package only for the Python
versions needing it.
``dev-python/mock`` is a compatible backport of the unittest.mock_
module. It can't use the same name as the standard library module,
therefore the packages need to use it conditionally, e.g.::
from unittest.mock import Mock
except ImportError: # py<3.3
from mock import Mock
import sys
if sys.hexversion >= 0x03030000:
from unittest.mock import Mock
from mock import Mock
However, the actual API remains compatible, so the programs do not need
more compatibility code than that. In some cases, upstreams fail (or
even refuse) to use the external ``mock`` package conditionally —
in that case, you either need to depend on this package unconditionally,
or patch it.
``dev-python/trollius`` aimed to provide a backport of asyncio_
for Python 2. Since the asyncio framework relies on new Python syntax,
the backport cannot be API compatible and requires using a different
syntax than native asyncio code.
.. _CPython:
.. _importlib.resources:
.. _platform.linux_distribution():
.. _PyPy:
.. _Jython:
.. _IronPython:
.. _Brython:
.. _MicroPython:
.. _Tauthon:
.. _unittest.mock:
.. _lzma:
.. _ipaddress:
.. _asyncio:


@ -0,0 +1,31 @@
Gentoo provides one of the best frameworks for providing Python support
in packages among operating systems. This includes support for
running multiple versions of Python (while most other distributions
avoid going beyond simultaneous support for Python 2 and one version
of Python 3), alternative implementations of Python, reliable tests,
deep QA checks. While we aim to keep things simple, this is not always
At the same time, the available documentation is limited and not always
up-to-date. Both the `built-in eclass documentation`_ and `Python
project wiki page`_ provide bits of documentation but it is mostly
in reference form and not very suitable for beginners nor people who
do not actively follow the developments within the ecosystem. This
results in suboptimal ebuilds, improper dependencies, missing tests.
This document aims to fill the gap by providing a good, complete,
by-topic (rather than reference-style) documentation for the ecosystem
in Gentoo and the relevant eclasses. Combined with examples, it should
help you write good ebuilds and solve common problems as simply
as possible.
.. _built-in eclass documentation:
.. _Python project wiki page:


@ -0,0 +1,259 @@
python-single-r1 — single-impl packages
.. highlight:: bash
The ``python-single-r1`` eclass is used to install single-impl packages.
It is probably the easiest eclass to use, and it is recommended over
``python-r1`` whenever multi-impl support would add unnecessary
complexity. However, for packages using distutils or a similar Python
build system, ``distutils-r1`` eclass should be used instead.
Basic use for unconditional Python
The defining feature of this eclass is that it defines a ``pkg_setup``
phase. It normally calls ``python_setup`` function in order
to determine the interpreter selected by user, and set the global
environment appropriately.
This means that a most trivial package using an autotools-compatible
build system along with unconditional dependency on Python could look
like the following:
.. code-block:: bash
:emphasize-lines: 6,7,17,20
# Copyright 1999-2020 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
PYTHON_COMPAT=( python2_7 )
inherit python-single-r1
DESCRIPTION="Scripts to prepare and plot VOACAP propagation predictions"
KEYWORDS="~amd64 ~x86"
This ebuild demonstrates the absolute minimum working code. Only
the four highlighted lines are specific to Python eclasses, plus
implicitly exported ``pkg_setup`` phase
When depending on other Python packages, USE dependencies need to be
declared in order to ensure that the dependencies would be built against
the Python implementation used for the package. The exact dependency
string depends on whether the target package is single-impl
or multi-impl.
When depending on other single-impl packages, the eclass-defined
``${PYTHON_SINGLE_USEDEP}`` variable can be used to inject the correct
USE dependency::
When depending on multi-impl packages, a more complex construct must
be used. The ``python_gen_cond_dep`` generator function is used
to copy the specified dependency template for all supported
implementations, and substitute ``${PYTHON_MULTI_USEDEP}`` template
inside it::
$(python_gen_cond_dep '
Please note that in this context, ``${...}`` is used as a literal
template substitution key, so it must be escaped to prevent bash from
substituting it immediately. In the above example, single quotes
are used for this purpose. When other variables are used, double quotes
with explicit escapes have to be used::
$(python_gen_cond_dep "
As demonstrated above, the USE dependency string can be combined with
other USE dependencies. ``PYTHON_SINGLE_USEDEP`` can be used both
inside and outside ``python_gen_cond_dep``, while
``PYTHON_MULTI_USEDEP`` only inside it.
Conditional Python use
The examples so far assumed that Python is used unconditionally.
If Python support is conditional to a USE flag, appropriate USE
conditionals need to be used in metadata variables, and ``pkg_setup``
needs to be rewritten to call the default implementation conditionally:
.. code-block:: bash
:emphasize-lines: 16,17,20,21,23-27,30,35
# Copyright 1999-2020 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
PYTHON_COMPAT=( python2_7 )
inherit python-single-r1
DESCRIPTION="Yet more Objects for (High Energy Physics) Data Analysis"
KEYWORDS="~amd64 ~x86 ~amd64-linux ~x86-linux"
IUSE="python root"
python? ( ${PYTHON_DEPS} )
root? ( sci-physics/root:=[python=,${PYTHON_SINGLE_USEDEP}] )"
python? (
$(python_gen_cond_dep '
pkg_setup() {
use python && python-single-r1_pkg_setup
src_configure() {
econf \
$(use_enable python pyext) \
$(use_enable root)
A hybrid: build-time + conditional runtime
A fairly common pattern is for Python to be required unconditionally
at build time but only conditionally at runtime. This happens e.g. when
the package is calling some helper scripts at build time, and optionally
installing Python bindings. In this case, the build time dependency
is expressed unconditionally, and the runtime dependency is made
.. code-block:: bash
:emphasize-lines: 18,19,23,26,32
# Copyright 1999-2020 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
PYTHON_COMPAT=( python3_{6,7,8} )
inherit waf-utils python-single-r1
DESCRIPTION="Samba talloc library"
KEYWORDS="~alpha amd64 arm ~arm64 ~hppa ia64 ~m68k ~mips ppc ppc64 ~riscv ~s390 ~sh ~sparc x86 ~amd64-linux ~x86-linux ~x64-macos ~sparc-solaris ~x64-solaris"
python? ( ${PYTHON_DEPS} )"
src_configure() {
local extra_opts=(
$(usex python '' --disable-python)
waf-utils_src_configure "${extra_opts[@]}"
Note that eclass-exported ``pkg_setup`` is used unconditionally here.
Multiple USE conditions
Finally, let's give an example of a package where Python is needed
for two independent conditions. To make it more complex, one of them
applies to build time (tests) while the other to runtime (bindings).
.. code-block:: bash
:emphasize-lines: 16,18,19,23,26,30-32,37,38
# Copyright 1999-2020 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2
PYTHON_COMPAT=( python3_{6,7,8} )
inherit cmake python-single-r1
DESCRIPTION="Sound design and signal processing system for composition and performance"
LICENSE="LGPL-2.1 doc? ( FDL-1.2+ )"
KEYWORDS="~amd64 ~x86"
IUSE="python test"
python? ( ${PYTHON_REQUIRED_USE} )
python? ( dev-lang/swig )
test? ( ${PYTHON_DEPS} )
python? ( ${PYTHON_DEPS} )
pkg_setup() {
if use python || use test ; then
src_configure() {
local mycmakeargs=(
Please note that in general, the condition in ``pkg_setup`` must match
the one in ``REQUIRED_USE``, and that one is a superset of conditions
used in dependencies.