Gentoo Python Developers Guide

1.  Bumps, Drops, Stabilization

Version Bumps and Fixing Bugs

Never bump one of the following packages yourself if permission to do so is not explicitly granted by a (Co-)Lead:

• dev-lang/python
• app-admin/eselect-python

While doing version bumps is surely appreciated, don't do it blindly. There were many bugs in the past which had been carried from version to version without being noticed. Make also sure that you check bugzilla before the bump to see whether there are open bugs for the package. Reading the ChangeLog of a package is also a good idea to catch new, changed or optional dependencies.

Not all existing ebuilds in the tree use the eclasses properly (see below), so please fix bugs on sight. Build the packages you're bumping or fixing even on small changes. Not all ebuilds have been written carefully while others might have been perfect when they have been committed. But over time, practice and rules change.

The same goes for fixing bugs in the ebuilds. Please check whether there is a new version of the package out and do a version bump accordingly. Closing bugs is good, but not enough. Your primary objective should not be to close bugs but to maintain the packages in the herd.

Ask for and do peer reviews. With such a practice, ebuild quality increases and it is a good way to transfer knowledge.

Dropping old versions and Stabilization

Every team member should try to keep the package directories clean and uncluttered. Besides the obvious checks (last stable for an arch, last not p.masked, other packages depend on an exact version of this package), there are some other things which you should consider before dropping an old version:

• When dropping an unstable version in the presence of a stable one: Does the version you are going to drop have serious bugs which avoid stabilization? Otherwise you might keep it and open a stabilization bug.
• The same consideration also applies if there is no stable version yet: Are there users who might want a stable version? Is this package mature enough to go stable? If you decide to stabilize it, also think about how arch team members could test it.
• Do not stabilize alpha and beta versions nor release candidates wherever possible. There are exceptions to this (if upstream just produces beta-ware or the package is desperately needed for another app). If unsure, talk to the Lead first.

2.  Writing of ebuilds

Types of packages

There are 2 types of packages handled by python.eclass:

• Packages supporting installation for multiple versions of Python
• Packages not supporting installation for multiple versions of Python

Packages supporting installation for multiple versions of Python install some files (usually .py or .so) into directories specific to given versions of Python.

Packages not supporting installation for multiple versions of Python usually exhibit at least one of the following properties:

• They install Python modules outside of site-packages directories.
• They do not install any files into directories specific to given versions of Python.
• They install libraries linked against Python library (e.g. libpython2.7.so) outside of directories specific to given versions of Python (e.g. in /usr/lib) and filenames of these libraries do not contain Python version.
• They import Python modules installed by packages not supporting installation for multiple versions of Python.

Specification of dependency on Python

Ebuilds should properly specify dependency on supported version(s) of Python. python.eclass supports PYTHON_DEPEND helper variable, which allows to specify minimal and maximal version of Python. PYTHON_DEPEND variable should be set before 'inherit'. PYTHON_DEPEND variable should contain 1 or 2 groups of version components and can optionally begin with USE flag conditional in the form of "flag? " or "!flag? ". Each group of version components should contain major version ("2", "3" or "*") and can optionally contain minimal version (e.g. "2.6") and maximal version (e.g. "3.1"). Version components should be separated by colons. Colons followed only by empty components can be ommitted. "*" major version means that versions of both Python 2 and Python 3 are accepted. Minimal and maximal versions should contain major and minor versions.

# Dependency on any version of Python.
PYTHON_DEPEND="*"

# Dependency on any version of Python 2.
PYTHON_DEPEND="2"

# Dependency on any version of Python 3.
PYTHON_DEPEND="3"

# Dependency on any version of Python 2, which is at least 2.6.*.
PYTHON_DEPEND="2:2.6"

# Dependency on any version of Python 3, which is at least 3.2.*.
PYTHON_DEPEND="3:3.2"

# Dependency on any version of Python 2 or 3, which is at least 2.6.*.
PYTHON_DEPEND="*:2.6"

# Dependency on any version of Python 2, which is at least 2.7.*, or a version of Python 3, which is at least 3.2.*.
PYTHON_DEPEND="2:2.7 3:3.2"

# Dependency on any version of Python 2, which is at most 2.6.*.
PYTHON_DEPEND="2::2.6"

# Dependency on any version of Python 3, which is at most 3.2.*.
PYTHON_DEPEND="3::3.2"

# Dependency on any version of Python 2 or 3, which is at most 3.2.*.
PYTHON_DEPEND="*::3.2"

# Dependency on any version of Python 2, which is at least 2.5.* and at most 2.6.*.
PYTHON_DEPEND="2:2.5:2.6"

# Dependency on any version of Python 3, which is at least 3.1.* and at most 3.2.*.
PYTHON_DEPEND="3:3.1:3.2"

# Dependency on any version of Python 2 or 3, which is at least 2.6.* and at most 3.1.*.
PYTHON_DEPEND="*:2.6:3.1"

# Dependency on any version of Python 2, which is at least 2.5.* and at most 2.6.*, or a version of Python 3, which is at least 3.1.* and at most 3.2.*.
PYTHON_DEPEND="2:2.5:2.6 3:3.1:3.2"

# Dependency on any version of Python 2, when "python" USE flag is enabled.
PYTHON_DEPEND="python? 2"

# Dependency on any version of Python 2, which is at least 2.5.*, when "python" USE flag is enabled.
PYTHON_DEPEND="python? 2:2.5"

# Dependency on any version of Python 3, when "minimal" USE flag is disabled.
PYTHON_DEPEND="!minimal? 3"

Checking of USE flags of Python

Ebuilds can set PYTHON_USE_WITH or PYTHON_USE_WITH_OR before 'inherit' and call python_pkg_setup() to check if Python has been installed with specific USE flags. All USE flags specified in PYTHON_USE_WITH must be enabled, but at least one USE flag specified in PYTHON_USE_WITH_OR must be enabled. PYTHON_USE_WITH_OPT can specify a name of a USE flag, which conditionalizes PYTHON_USE_WITH and PYTHON_USE_WITH_OR. If python_set_active_version() (described below) is used, then it must be called before python_pkg_setup().

PYTHON_USE_WITH="tk"

inherit python

...

pkg_setup() {
    python_set_active_version 2
    python_pkg_setup
}

Supporting of installation for multiple versions of Python

Ebuilds should set SUPPORT_PYTHON_ABIS="1" before 'inherit'.

Ebuilds not working with some versions of Python should set RESTRICT_PYTHON_ABIS variable (e.g. after DEPEND/RDEPEND), which should contain list of space-separated fnmatch patterns. Such patterns can contain '*' character.

# Package not supporting Python 2.4.
RESTRICT_PYTHON_ABIS="2.4"

# Package not supporting Python 3.
RESTRICT_PYTHON_ABIS="3.*"

# Package not supporting Python 2.4 and 2.5.
RESTRICT_PYTHON_ABIS="2.[45]"

# Package not supporting Python 2.4 and 3.
RESTRICT_PYTHON_ABIS="2.4 3.*"

# Package not supporting Python 2.4, 2.6, 2.7 and 3.
RESTRICT_PYTHON_ABIS="2.4 2.[67] 3.*"

Separate build directories must be used for different Python versions. Distutils by default uses "build" directory, which can be changed by "-b" option of "build" command of setup.py. Packages, which do not use Distutils, and very small number of packages, which use Distutils, usually need to use build directories outside of "${S}". Functions from distutils.eclass by default use "${S}/build-${PYTHON_ABI}" build directories. Packages, which do not use "${S}/build-${PYTHON_ABI}" build directories, need to call python_copy_sources() function, which copies sources to separate build directories.

src_prepare() {
    epatch "${FILESDIR}/${P}-fix_build.patch"
    python_copy_sources
}

python_copy_sources() can be also used to copy a subdirectory of "${S}" to separate build directories. It can be useful in ebuilds of packages, which optionally build Python bindings.

src_prepare() {
    default

    if use python; then
        python_copy_sources bindings/python
    fi
}

python_execute_function() is used to perform appropriate actions with all enabled Python versions. This function requires one argument, which is name of function or -d / --default-function option. This function accepts some optional arguments. python_execute_function() executes a function, which needs to be defined earlier. To improve readability, it's recommended to define functions, which are used only in 1 place in ebuilds, directly before passing their names to python_execute_function().

src_test() {
    testing() {
        PYTHONPATH="build-${PYTHON_ABI}/lib" "$(PYTHON)" runtests.py
    }
    python_execute_function testing
}

When given actions should be executed in separate build directories created by python_copy_sources(), then -s / --separate-build-dirs option must be passed to python_execute_function().

src_prepare() {
    epatch "${FILESDIR}/${P}-fix_syntax.patch"
    python_copy_sources
}
src_configure() {
    configuration() {
        "$(PYTHON)" configure.py \
            --bindir="${EPREFIX}/usr/bin" \
            --destdir="${EPREFIX}$(python_get_sitedir)"
    }
    python_execute_function -s configuration
}

If build directories are subdirectories of "${S}", then additionally --source-dir option and path to source directory must be passed to python_execute_function().

src_configure() {
    econf \
        $(use_with java) \
        $(use_with python) \
        $(use_with ruby)

    # Python bindings are built/tested/installed manually.
    sed -e "/SUBDIRS =/s/ python//" -i bindings/Makefile || die "sed failed"
}

src_compile() {
    default

    if use python; then
        python_copy_sources bindings/python
        building() {
            # Override paths stored in bindings/python-${PYTHON_ABI}/Makefile files by 'configure'.
            emake PYTHON="$(PYTHON)" PYTHON_INCLUDEDIR="$(python_get_includedir)" PYTHON_LIBDIR="$(python_get_libdir)"
        }
        python_execute_function -s --source-dir bindings/python building
    fi
}

-d / --default-function option is useful in cases, when the same actions, which are executed in default phase functions (e.g. emake in src_compile()), need to be executed. This option can be used only in a subset of ebuild phases.

src_compile() {
    python_execute_function -d -s
}

python.eclass defines the following phase functions, which can be used to simplify some ebuilds:

• python_src_prepare
• python_src_configure
• python_src_compile
• python_src_test
• python_src_install

python_src_prepare() calls 'python_copy_sources', while other phase functions call 'python_execute_function -d -s'. If PYTHON_EXPORT_PHASE_FUNCTIONS="1" variable has been set before 'inherit', then these phase functions are exported.

PYTHON_ABI variable can be checked inside function executed by python_execute_function().

src_prepare() {
    python_copy_sources

    patching() {
        [[ "${PYTHON_ABI}" == 3.* ]] && return
        epatch "${FILESDIR}/${P}-python-3.patch"
    }
    python_execute_function --action-message 'Applying patches with $(python_get_implementation) $(python_get_version)' -s patching
}

Sometimes another eclass defines a specialized function, which performs building, installation etc., but is designed for non-Python packages. In such cases, it's possible to call python_execute_function() with name of such a function.

src_configure() {
    python_execute_function -s gnome2_src_configure
}

Setting of active version of Python in packages not supporting installation for multiple versions of Python

If given package supports only Python 2 or only Python 3, then python_set_active_version() function should be called to set active version of Python. Usually major version of Python should be passed to this function.

pkg_setup() {
    python_set_active_version 2
}

If given package supports only 1 version of Python, then Python ABI (in the form of ${major_version}.${minor_version}) can be passed to python_set_active_version(). It will cause that python-updater will ignore this package.

pkg_setup() {
    python_set_active_version 3.1
}

Getter functions

• PYTHON prints filename of Python interpreter (e.g. python3.2).
• PYTHON with -a / --absolute-path option prints absolute path to Python interpreter (e.g. /usr/bin/python3.2).
• python_get_implementation prints name of Python implementation (e.g. CPython).
• python_get_implementational_package prints category, name and slot of package providing Python implementation (e.g. dev-lang/python:3.2).
• python_get_includedir prints path to Python include directory (e.g. /usr/include/python3.2).
• python_get_libdir prints path to Python library directory (e.g. /usr/lib64/python3.2).
• python_get_sitedir prints path to Python site-packages directory (e.g. /usr/lib64/python3.2/site-packages).
• python_get_library prints path to Python library (e.g. /usr/lib64/libpython3.2.so).
• python_get_library with -l / --linker-option option prints Python library in the form of -l${library} linker option (e.g. -lpython3.2).
• python_get_version prints major and minor version of Python (e.g. 3.2).
• python_get_version with --major option prints major version of Python (e.g. 3).
• python_get_version with --minor option prints minor version of Python (e.g. 2).
• python_get_version with --micro option prints micro version of Python (e.g. 0).

src_install() {
    ...

    install_headers() {
        insinto "$(python_get_includedir)"
        doins include/*.h
    }
    python_execute_function -q install_headers
}

In ebuilds supporting installation for multiple versions of Python, sometimes given action needs to be executed only once (e.g. generation of documentation). In such cases it should be executed with the final Python ABI from the list of enabled ABI, which is performed by passing -f / --final-ABI option to appropriate getter functions.

src_compile() {
    distutils_src_compile

    if use doc; then
        "$(PYTHON -f)" setup.py pudge
    fi
}

Shebangs in installed scripts

Shebangs in installed scripts should be correct to avoid problems when a different version of Python is set as main active version of Python. If given package does not support some versions of Python and build system installs scripts with too generic shebangs, then python_convert_shebangs() should be called to convert shebangs. This function requires Python version and files / directories. Directories can be passed only with -r / --recursive option.

src_prepare() {
    python_convert_shebangs -r 2 .
}

Handling of byte-compiled modules

Byte-compilation of Python modules is usually disabled. python_enable_pyc() enables it, while python_disable_pyc() disables it.

python_mod_optimize() is used to compile and optimize Python modules in pkg_postinst() phase. python_mod_cleanup() is used to remove compiled and optimized Python modules in pkg_postrm(). Ebuilds, which use distutils.eclass and install Python modules into site-packages directories, usually do not need to directly call python_mod_optimize() or python_mod_cleanup(). Paths of modules installed into site-packages directories should be relative to site-packages directories. Other paths should be relative to ${ROOT}. python_mod_cleanup() removes empty directories after cleaning up .py files.

pkg_postinst() {
    python_mod_optimize PyQt4
}

pkg_postrm() {
    python_mod_cleanup PyQt4
}

Usage of distutils.eclass

• distutils_src_compile(), distutils_src_test() and distutils_src_install() internally perform actions appropriate for given type of package. In ebuilds supporting installation for multiple versions of Python, they define some functions and pass their names to python_execute_function().
• If the ebuild name (in ${PN}) differs from the directory created by the package in site-packages/, then ebuild should define a variable PYTHON_MODNAME variable to tell distutils_pkg_postinst() and distutils_pkg_postrm() paths of Python modules.
• Ebuilds can set DOCS variable to tell distutils_src_install() to install additional (pure-text) documentation files.

PYTHON_MODNAME="IPython"

default_docs="AUTHORS Change* CHANGELOG CONTRIBUTORS KNOWN_BUGS MAINTAINERS MANIFEST* NEWS PKG-INFO README* TODO"

3.  Common Problems and Mistakes

Below are common problems you may face and common mistakes made when writing ebuilds for python packages.

setuptools: *_require and use_setuptools()

Packages that use setuptools to install use _require options like tests_require,install_require,setup_requires in setup.py. These are nice to learn about dependencies but you don't want them in setup.py when you're installing the package. The following is from the setuptools homepage section on setup_requires:

A string or list of strings specifying what other distributions need to be present in order for the setup script to run. setuptools will attempt to obtain these (even going so far as to download them using EasyInstall) before processing the rest of the setup script or commands.

—setuptools developer's guide

We have lovely package managers which can download stuff for us and verify their digests thus we don't want to download any packages using EasyInstall. There are other options like tests_require, install_requires that behave the same way.

Some packages have a ez_setup.py along with the usual setup.py. This is a python script to download and install appropriate setuptools. To do this use_setuptools() is called from ez_setup.py before importing setuptools.

def use_setuptools(
    version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
    download_delay=15
):
    """Automatically find/download setuptools and make it available on sys.path
    [...]

Just like the _require options, if a setup.py script calls use_setuptools() from ez_setup.py you should remove it. Below is an example which illustrates how to do it.

from ez_setup import use_setuptools
use_setuptools()
from setuptools import setup, find_packages

[...]

install_requires=["Routes >= 1.0", "Paste", "PasteDeploy", "PasteScript"],

[...]

src_unpack() {
    unpack ${A}
    cd "${S}"
    sed -i \
        -e '/use_setuptools/d' \
        -e '/install_requires=\[.*\],/d' \
        setup.py || die "sed failed"
}

src_test and PYTHONPATH

When testing python packages it's important to make sure we're actually testing the package that is going to be merged not the already installed package. We can solve the problem by setting the PYTHONPATH environment variable which augments the default search path for module files. Here are two examples:

src_test() {
    testing() {
        PYTHONPATH="build-${PYTHON_ABI}/lib" "$(PYTHON)" test.py
    }
    python_execute_function testing
}

src_test() {
    testing() {
        PYTHONPATH="$(ls -d build-${PYTHON_ABI}/lib.*)" "$(PYTHON)" test.py
    }
    python_execute_function testing
}

As you may have noticed if the module is written in languages like C, C++, etc. the name of the directory in build varies between architectures but it always starts with lib.

Is dev-python/setuptools an RDEPEND or DEPEND?

repoman may issue a warning saying dev-python/setuptools is a suspicious RDEPEND. Note however that setuptools is quite often a run-time dependency by code that installs commands in /usr/bin, uses pkg_resources to require specific package versions or makes use of entry points in a plugin system.

If you emerge a package that uses setuptools and it installs commands in /usr/bin you can look at those commands and easily determine if setuptools is required at run-time.

    #!/usr/bin/python
    # EASY-INSTALL-ENTRY-SCRIPT: 'doapfiend==0.3.4','console_scripts','doapfiend'
    __requires__ = 'doapfiend==0.3.4'
    import sys
    from pkg_resources import load_entry_point

If the script imports pkg_resources then dev-python/setuptools must be in RDEPEND.

You can also use dev-python/yolk to determine if a package uses setuptools entry points by giving it the Python module name (not Gentoo package name).

    $ yolk --entry-map paste

If there is any output, then dev-python/setuptools should be in RDEPEND.

The contents of this document, unless otherwise expressly stated, are licensed under the CC-BY-SA-2.5 license. The Gentoo Name and Logo Usage Guidelines apply.