python-r1 Developer's Guide

1.  Introductory information

Purpose of the eclasses

The python‑r1 suite consists of five eclasses:

1. python-utils-r1,
2. python-r1,
3. python-single-r1,
4. python-any-r1,
5. distutils-r1.

The python‑utils‑r1 eclass is the utility eclass in the suite. It does not export any phase functions nor set any metadata. It does not require the package to fit any specific model.

The python‑r1 eclass is the second fundamental eclass. It extends the former eclass with the metadata and variables suited for packages supporting multiple Python implementations: the implementation choice flags, dependency strings. It does not, however, enforce any dependencies directly nor export phase functions.

The python‑single‑r1 eclass is an alternative to python‑r1 for packages which do not support being installed for multiple Python implementations. It exports similar metadata and variables, and a pkg_setup phase function handling the target implementation choice.

The python‑any‑r1 eclass is designed to be used on packages which do not need explicit implementation choice or rely on specific implementation installed being invariable. This mostly involves packages having strictly build-time dependency on Python. It exports metadata suitable for setting an appropriate dependency, and a pkg_setup phase function handling finding the best installed implementation.

The distutils‑r1 eclass extends the suite with a set of basic phase functions to build and install packages using the distutils build system of the inbuilt Python module distutils. It follows the common practices for build system eclasses, including patching and installing documentation.

The choice of eclass for your package could follow the following algorithm:

1. If the package uses the distutils build system, then use distutils‑r1.
2. If the package installs Python modules or scripts which can be simultaneously installed for multiple Python implementations, use python‑r1.
3. If the package installs Python modules or scripts which can be installed for a single Python implementation only, or embeds Python, then use python‑single‑r1.
4. If the package has a strictly build-time dependency on Python, then use python‑any‑r1.

A conversion from python.eclass can follow the following algorithm:

1. If the package inherits distutils, then use distutils‑r1.
2. If the package enables SUPPORT_PYTHON_ABIS, then use python‑r1.
3. If the package has run-time dependency on Python, then use python‑single‑r1.
4. If the package has only build-time dependency on Python, then use python‑any‑r1.

The active Python interpreter in ebuild scope

A key concept in the workflow of python‑r1 suite is the setting of the active Python interpreter. Its value is stored and exported in the EPYTHON environment variable.

The setting of active Python interpreter can affect;

1. the interpreter used to run external Python scripts,
2. the implementation used by helpers, such as python_domodule and python_doscript,
3. the default implementation for python_export and the associated getter functions.

The EPYTHON variable is set implicitly in the following scopes;

1. in commands and functions run by python_foreach_impl,
2. in implementation-specific sub-phases of distutils‑r1,
3. in ebuilds using the python‑single‑r1 eclass after running python‑single‑r1_pkg_setup.

If it is otherwise necessary to set the active Python interpreter, it can be either set using the python_export function or by setting the EPYTHON variable directly. For the details on the former solution, please refer to the Obtaining Python implementation information section.

If EPYTHON is set directly, it needs to have one of the following values;

1. pythonX.Y for CPython X.Y;
2. pypy-cX.Y for PyPy X.Y;
3. jythonX.Y for Jython X.Y.

pkg_setup() {
	# Variant 1
	# The variables are not local, therefore the setting is preserved
	# through phase functions.
	python_export python2_7
}

src_compile() {
	# Variant 2
	# The variables are kept local.
	local EPYTHON PYTHON
	python_export python2_7

	# …
}

src_install() {
	# Variant 3
	local EPYTHON=python2.7

	# …
}

2.  Common metadata variables

General notes

Listing supported Python implementations (PYTHON_COMPAT)

The first and most important task in writing ebuilds both for python‑r1 and distutils‑r1 is to specify the list of Python implementations supported by the ebuild.

This list is specified using the PYTHON_COMPAT variable. It is an obligatory array which need be declared before the inherit command. It should list all supported implementations using the following naming scheme:

1. pythonX_Y for CPython X.Y;
2. pypyX_Y for PyPy X.Y;
3. jythonX_Y for Jython X.Y.

PYTHON_COMPAT should be treated with respect similar to the KEYWORDS variable. No implementation should be added there without prior testing, and all package dependencies must support that implementation.

The PYTHON_COMPAT variable can be considered a replacement for python.eclass' PYTHON_DEPEND and RESTRICT_PYTHON_ABIS variables.

# Any version of CPython 2.
PYTHON_COMPAT=( python2_5 python2_6 python2_7 )

# Python 2 or 3, using brace expansion.
PYTHON_COMPAT=( python{2_5,2_6,2_7,3_1,3_2,3_3} )

# Python 2.6+ and compliant implementations.
PYTHON_COMPAT=( python{2_6,2_7} pypy{1_9,2_0} )

# inherit follows PYTHON_COMPAT
inherit python-r1

Depending on Python (PYTHON_DEPS)

In order to efficiently handle various kinds of package dependencies on Python, the python‑r1 eclass does not set the ebuild dependencies directly. Instead, it prepares the proper dependency string and stores it in a variable named PYTHON_DEPS.

The PYTHON_DEPS variable is unconditionally set by the eclass to a list of proper USE-conditional dependencies on enabled Python implementations. It also holds any additional dependencies necessary — most notably, a dependency on dev-python/python-exec.

The ebuild should reference the PYTHON_DEPS variable in RDEPEND and/or DEPEND as necessary. Those dependencies may require use of an appropriate USE-conditional block.

# Unconditional Python run-time + build-time dependency.
RDEPEND=${PYTHON_DEPS}
DEPEND=${RDEPEND}

# Unconditional Python build-time dependency.
DEPEND=${PYTHON_DEPS}

# USE-conditional Python run-time dependency.
RDEPEND="python? ( ${PYTHON_DEPS} )"

The actual value assigned to this variable differs according to the inherited eclass as listed in the following table:

Depending on other Python packages (PYTHON_USEDEP)

Whenever two Python packages involve one of the packages loading Python modules installed by the other, both packages need to have the support for the same Python implementations enabled. That constraint needs to be enforced through the use of PYTHON_USEDEP variable.

The PYTHON_USEDEP variable is set unconditionally by the eclass. It contains a bare compact USE dependency string which can be used directly in dependency specifications.

It should be noted that dependencies not involving direct Python module loading do not require PYTHON_USEDEP. For example, running external Python scripts does not need those constraints enforced.

PYTHON_USEDEP can not be used against packages not supporting PYTHON_TARGETS. Developers are strongly encouraged to convert dependencies before committing the actual package. If that is not feasible, the dependency has to omit the use of PYTHON_USEDEP.

# Simple dependency on a python‑r1 package.
RDEPEND="dev-python/setuptools[${PYTHON_USEDEP}]"

# Dependency with additional USE flags requested.
RDEPEND="dev-python/lxml[threads,${PYTHON_USEDEP}]"

# Dependency on a package using python.eclass.
RDEPEND="dev-python/wxpython:2.8"

The compatibilities of PYTHON_USEDEP with packages using different eclasses according to the inherited eclass of the ebuild being developed are listed in the following table:

Requesting optional Python features (PYTHON_REQ_USE)

Sometimes it is necessary to depend on a feature or module which belongs to the Python implementation itself but is not always available. The availability can depend both on Python version and USE flags. These features can be divided into two types; those having replacement packages and those lacking them.

If a particular module has a replacement package, the correct way to require it is to depend on the respective virtual package. For example, a script using the argparse module should depend on virtual/python-argparse. The virtual will either enforce a particular USE constraints or pull in the replacement package as necessary.

If a particular module or feature is provided by the Python implementation only, and is conditional upon the setting of a USE flag, PYTHON_REQ_USE should be used instead.

The PYTHON_REQ_USE variable is an optional variable which can be used to enforce a set of USE flag requirements upon the Python implementation. It takes a bare USE dependency string and appends it to the dependency on every Python implementation supported. Sometimes it may be necessary to use EAPI 4 USE defaults to handle cases when a particular flag is enabled unconditionally in some of the supported implementations.

The PYTHON_REQ_USE variable can be considered a replacement for python.eclass' PYTHON_USE_WITH variable.

# Run-time dependency on argparse module (through virtuals).
RDEPEND="virtual/python-argparse[${PYTHON_USEDEP}]"

# Simple dependency on ncurses support.
PYTHON_REQ_USE="ncurses"

# Dependency on bzip2 support (conditional in PyPy, always enabled in CPython).
PYTHON_REQ_USE="bzip2(+)"

3.  The distutils‑r1 eclass

Tasks performed by distutils‑r1

The distutils‑r1 exports a set of phase functions performing common tasks related to building and installing the package. In many cases, those tasks will suffice for a typical Python package.

The tasks performed by distutils‑r1 are (in execution order):

1. applying patches listed in the PATCHES array;
2. applying user patches (the epatch_user function);
3. building and installing the package using setup.py;
4. ensuring that Python scripts are installed in multiple variants according to the enabled Python implementations;
5. installing additional documentation (using DOCS and HTML_DOCS arrays).

PATCHES=(
	# bug #nnnnnn, a random error with something.
	"${FILESDIR}"/${P}-foo.patch

	# bug #nnnnnn, some other error.
	"${FILESDIR}"/${P}-bar.patch
)

DOCS=( README.md )
HTML_DOCS=( doc/html/ )

The ‘partial’ phase functions

The distutils‑r1 eclass utilises a mechanism inspired by phase functions to make writing ebuilds relatively easy. For each of the src_* phases, two ‘partial’ phases are used; the implementation-specific sub-phase and the implementation-common sub-phase.

Each of the implementation-specific sub-phases is named according to its corresponding src_* phase but with the src_ prefix replaced with python_. For example, src_compile() becomes python_compile(). It is called once for each enabled Python implementation. For the call scope, the default set of informational variables is exported (EPYTHON, PYTHON, BUILD_DIR, PYTHONPATH if necessary).

Each of the implementation-common sub-phases has the suffix _all appended. For example, the python_install_all() sub-phase corresponds to src_install(). It is called only once during the build process. It is invoked in the main source directory. For the call scope, the variables corresponding to the best enabled Python implementation are exported.

The distutils‑r1 eclass provides default functions for all implementation-specific sub-phases and for python_prepare_all and python_install_all. If you are defining any of those phase functions, you ought call the respective distutils‑r1 phase function.

python_compile_all() {
	if use doc; then
		python_export_best
		"${PYTHON}" setup.py doc || die
	fi
}

python_test() {
	"${PYTHON}" setup.py test || die
}

python_install_all() {
	use doc && local HTML_DOCS=( doc/* )

	distutils-r1_python_install_all
}

Out-of-source and in-source builds

There are two modes of building packages with distutils‑r1; ‘out-of-source builds’ (the default) and ‘in-source builds’.

When an out-of-source build is performed, the build directory (${BUILD_DIR}) is used to store output files, while the sources are kept in their original location. Therefore, the build system must not attempt to modify any of the input files or write to ${S}.

In order to support out-of-source builds, the build directory is appended to PYTHONPATH and esetup.py command adds build directory paths to the setup.py invocation. Due to technical limitations of distutils, this results in the package being built on the first esetup.py invocation.

The in-source builds are implemented for packages where out-of-source builds are problematic. They are enabled either explicitly when the DISTUTILS_IN_SOURCE_BUILD variable is set (to any value), or implicitly when the python_prepare sub-phase is declared.

When an in-source build is performed, the build directory acts both as a source and output directory. The package sources are copied there first, and all implementation-specific sub-phases are executed in the build directory. Therefore, the build system is allowed to modify the sources freely.

Parallel build support

A parallel build occurs when more than one task is performed concurrently. For example, many common build systems including autotools and cmake compile multiple files in parallel to make a better use of the computing power of modern multicore CPUs and multiprocessor systems.

Sadly, distutils itself is not capable of parallel builds, making the build of multiple Python extensions inefficient and time-consuming. In order to circumvent that limitation, the distutils‑r1 eclass runs the whole build process for multiple Python implementations in parallel.

This usually causes no issues itself but it can trigger issues consequent to the modifying of source files by the build system. Those issues can generally be managed by enabling the in-source build option by setting DISTUTILS_IN_SOURCE_BUILD to a non-null value.

When a parallel build is performed, the ebuild is disallowed from modifying installed files during the implementation-specific sub-phase. Any changes necessary need be delayed until the implementation-common sub-phase. Otherwise, the files are prone to being clobbered in the middle of performing the change.

If the build system itself actually uses a parallel build (e.g. distutils running make) or any other issue makes parallel build undesirable, it can be disabled by setting DISTUTILS_NO_PARALLEL_BUILD to a non-null value.

4.  Advanced python‑r1 functions

Repeating commands for multiple Python implementations

If it is necessary to run a command repeatedly for multiple Python implementations, the python_foreach_impl function can be used. It takes a command and a list of its parameters and runs it once for each Python implementation enabled. The command can also name a local function.

The commands are run with the following environment variables set:

src_install() {
	# I wonder how many ebuilds will copy that name…
	sub_install() {
		# BUILD_DIR contains a module built for appropriate impl
		python_domodule "${BUILD_DIR}"/mymodule
	}

	# Run the function for each enabled implementation
	python_foreach_impl sub_install

	# Run python_doscript (install script) for each implementation
	python_foreach_impl python_doscript mymodule-ui
}

5.  Eclasses for use of a single Python implementation

Introduction

The python‑r1 suite provides two eclasses for work with a single Python implementation:

1. python-single-r1,
2. python-any-r1.

The python‑single‑r1 eclass is intended for packages where the choice of the used Python implementation is made available to the user. This group includes packages which install Python modules or scripts specific to the chosen implementation.

The python‑any‑r1 eclass is intended for packages which are not bound to a single Python implementation and its selection, therefore, is unimportant. These are mostly packages with build-time tools written in Python.

Both eclasses share the same core API being a superset of the core python‑r1 API. It includes;

1. use of PYTHON_COMPAT to list supported Python implementations,
2. setting of PYTHON_DEPS to a proper dependency string, respecting PYTHON_REQ_USE,
3. export of pkg_setup() phase function which performs the choice of Python implementation and exports implementation-specific variables.

It should be noted that if the Python-related code in the ebuild is conditional upon a USE flag, a proper USE-conditional need be used around ${PYTHON_DEPS} and the python-single-r1_pkg_setup or python-any-r1_pkg_setup invocation need be used conditionally.

PYTHON_COMPAT=( python{2_6,2_7} )
inherit python-single-r1

IUSE="python"

RDEPEND="python? ( ${PYTHON_DEPS} )"
DEPEND=${RDEPEND}

pkg_setup() {
	use python && python-single-r1_pkg_setup
}

python‑single‑r1

The python‑single‑r1 eclass, much alike python‑r1, provides an explicit implementation choice with the help of USE flags. However, in order not to collide with common setups enabling multiple Python implementations, it uses a different variable — PYTHON_SINGLE_TARGET.

The python-single-r1_pkg_setup phase function obtains the value of PYTHON_SINGLE_TARGET and exports the default set of variables specific to the chosen Python implementation.

python‑any‑r1

The python‑any‑r1 eclass performs an implicit choice of Python implementation. It accepts any of the supported Python interpreters (therefore using an any-of dependency in PYTHON_DEPS), and chooses the best installed one.

The python-any-r1_pkg_setup phase function obtains the best Python implementation installed. The choice algorithm follows one used by python‑exec tool, that is in order:

1. the implementation specified in EPYTHON,
2. the implementation chosen by eselect‑python,
3. the default implementation preference.

6.  Helper functions in python‑utils‑r1 eclass

General notes

Obtaining Python implementation information

Sometimes it is necessary to obtain information specific to a particular Python implementations, in particular interpreter-specific paths. The python‑utils‑r1 eclass provides the following means of obtaining that information:

1. python_export function,
2. python_export_best function,
3. getter functions.

The python_export and python_export_best functions take an optional list of variable names and export the requested variables. When no variable names are passed, the default variable list is used.

The python_export function can additionally take a Python implementation as a first argument, either in the form of a USE flag or an EPYTHON value. If such a form is used, the values specific to the passed implementation are exported. Otherwise, the currently active implementation is used.

The python_export_best exports variables for the ‘best’ of the currently enabled implementations; that is, the newest interpreter version from the most preferred group. These groups are, in order of preference:

1. CPython 2,
2. CPython 3,
3. PyPy,
4. Jython.

The getter functions print the value of a specific property to standard output. The output can be captured using bash command substitution. These functions take an optional parameter specifying the requested Python implementation. If not specified, the implementation set as EPYTHON will be used.

The following table lists all the available variables, along with the respective getter functions:

Installing Python scripts and modules manually

The python‑utils‑r1 eclass provides two major helpers which could be used to install Python scripts and modules manually. They can be used whenever the build system is not capable of installing them correctly, or the package maintainer wishes to install additional files.

These functions are:

1. python_domodule,
2. python_doscript.

The python_domodule helper takes a list of file and/or directory names and installs the named modules and packages recursively. The files are installed in the site-packages directory by default. The destination can be changed through setting the python_moduleroot variable or using the python_moduleinto function. It can be either an absolute path or relative to the site-packages root.

The installed modules will be byte-compiled using the currently active Python implementation. The selected implementation determines the site-packages location as well.

src_install() {
	python_export python2_7 EPYTHON PYTHON PYTHON_SITEDIR

	# Installs /usr/lib*/python2.7/site-packages/mypackage directory
	python_domodule mypackage

	# Installs /usr/lib*/python2.7/site-packages/epython.py (and .pyc, .pyo…)
	python_domodule epython.py

	# Installs …/site-packages/foo/bar.py (and .pyc, .pyo…)
	python_moduleinto foo
	python_domodule bar.py

	# Installs /usr/lib/portage/pym (and compiles for py2.7)
	python_moduleinto /usr/lib/portage
	python_domodule pym

	# Installs /usr/lib*/python*/site-packages/mypackage
	local python_moduleroot
	python_foreach_impl python_domodule mypackage
}

The python_doscript helper takes a list of script names and installs them. The files are installed in /usr/bin by default. The destination can be changed through setting the python_scriptpath variable or using the python_scriptinto function.

The installed scripts will be renamed to end with an implementation-specific suffix specific to the currently active Python interpreter. A wrapper will be linked in place of the original name.

src_install() {
	python_export python3_2 EPYTHON

	# Installs /usr/bin/frobnicate-python3.2
	# and a wrapper symlink at /usr/bin/frobnicate
	python_doscript frobnicate

	# Installs /usr/sbin/whyamihere-python3.2
	# and a wrapper symlink at /usr/sbin/whyamihere
	python_scriptinto /usr/sbin
	python_doscript whyamihere

	# Installs /usr/bin/myscript-* for all implementations
	# and a wrapper symlink at /usr/bin/myscript
	local python_scriptroot
	python_foreach_impl python_doscript myscript
}

Compiling installed Python modules

There are a few packages which are able to install the Python modules correctly but either do not compile them at all, or fail to do it properly. For those packages, python_optimize is the tool of choice.

The python_optimize function takes an optional list of directory paths. If the list is provided, it compiles all Python modules in the directories listed. Otherwise, it compiles the modules installed into the standard module install locations, including the site-packages directory.

The modules are compiled using the currently active Python implementation.

src_install() {
	python_export python2_7 EPYTHON PYTHON

	# Compile modules in custom location using python2.7
	# (note: you can not rely on ${PYTHONPATH})
	python_optimize "${D}"/usr/lib/portage/pym

	# Compile modules installed to site-packages
	# for all enabled implementations
	python_foreach_impl python_optimize
}

The contents of this document, unless otherwise expressly stated, are licensed under the CC-BY-SA-3.0 license. The Gentoo Name and Logo Usage Guidelines apply.