Gentoo Python Developers Guide

Tiziano Müller Author
Ali Polatel Author
Rob Cakebread Author

Updated June 20, 2008

1. Bumps, Drops, Stabilization

Never bump one of the following packages yourself if permission to do so is not explicitly granted by a (Co-)Lead:

dev-lang/python
dev-python/pycrypto

While doing version bumps is surely appreciated, don't do it blindly. There were many bugs in the past which had been carried from version to version without being noticed. Make also sure that you check bugzilla before the bump to see whether there are open bugs for the package. Reading the ChangeLog of a package is also a good idea to catch new, changed or optional dependencies.

Not all existing ebuilds in the tree use the eclasses properly (see below), so please fix bugs on sight. Build the packages you're bumping or fixing even on small changes. Not all ebuilds have been written carefully while others might have been perfect when they have been committed. But over time, practice and rules change.

The same goes for fixing bugs in the ebuilds. Please check whether there is a new version of the package out and do a version bump accordingly. Closing bugs is good, but not enough. Your primary objective should not be to close bugs but to maintain the packages in the herd.

Ask for and do peer reviews. With such a practice, ebuild quality increases and it is a good way to transfer knowledge.

Dropping old versions and Stabilization

Every team member should try to keep the package folders clean and uncluttered. Besides the obvious checks (last stable for an arch, last not p.masked, other packages depend on an exact version of this package), there are some other things which you should consider before dropping an old version:

When dropping an unstable version in the presence of a stable one: Does the version you are going to drop have serious bugs which avoid stabilization? Otherwise you might keep it and open a stabilization bug.
The same consideration also applies if there is no stable version yet: Are there users who might want a stable version? Is this package mature enough to go stable? If you decide to stabilize it, also think about how arch team members could test it.
Do not stabilize alpha and beta versions nor release candidates wherever possible. There are exceptions to this (if upstream just produces beta-ware or the package is desperately needed for another app). If unsure, talk to the Lead first.

2. Correct eclass usage

There are currently 3 python related eclasses: python, distutils and twisted.

python.eclass

Here are some important things to remember when using this eclass:

Call python_version before using PYVER, PYVER_MINOR or PYVER_MAJOR.
Insert a line "NEED_PYTHON=MY_VERSION" before the inherit to depend on python. If you don't need a specific version, put virtual/python in DEPEND/RDEPEND.
If you don't use distutils (and/or your package installs .py files somewhere else than site-packages), you have to call python_mod_optimize and python_mod_cleanup yourself. If your package doesn't install itself into a subdir, omit "/YOURPACKAGE". You can pass any path you want to python_mod_optimize/cleanup. python_mod_cleanup will also remove empty directories after the cleanup. If you call python as root, .py files will get byte-compiled to .pyc but not recorded in the package's CONTENTS. They therefore won't get removed on update/removal of the package.

Code Listing 2.1: Optimize/cleanup idiom

inherit python multilib

[...]

pkg_postinst() {
    python_mod_optimize $(python_get_sitedir)/YOURPACKAGE
}

pkg_postrm() {
    python_mod_cleanup
}

Important: If the package's setup byte-compiles installed .py files, it's a good idea to disable this and use python_mod_optimize to prevent unexpected problems.

distutils.eclass

To depend on a specific version of python, put NEED_PYTHON=MY_VERSION before the inherit (rather than DEPEND/RDEPEND on dev-lang/python-MY_VERSION) or leave it away to implicitly depend on virtual/python through the distutils eclass.
If the ebuild name (in ${PN}) differs from the folder created by the package in site-packages/, you have to define a variable PYTHON_MODNAME to tell distutils where to look for the module.
Set the DOCS variable to install additional (pure-text) docs. If you write the src_install-function, you can put the definition of the DOCS var in the function but before calling distutils_src_install.

Code Listing 2.2: PYTHON_MODNAME usage (example from ipython-0.7.3.ebuild)

# Copyright 1999-2007 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: /var/cvsroot/gentoo/xml/htdocs/proj/en/Python/developersguide.xml,v 1.11 2009/10/03 17:22:17 sping Exp $

NEED_PYTHON=2.3

inherit distutils

DESCRIPTION="An advanced interactive shell for Python."
HOMEPAGE="http://ipython.scipy.org/"
SRC_URI="http://ipython.scipy.org/dist/${P}.tar.gz"

LICENSE="BSD"
SLOT="0"
KEYWORDS="~amd64 ~ia64 ~ppc ~s390 ~x86"
IUSE="doc examples emacs gnuplot test"

DEPEND="test? ( dev-python/pexpect )"
RDEPEND="gnuplot? ( dev-python/gnuplot-py )"

PYTHON_MODNAME="IPython"

[...]

Code Listing 2.3: DOCS idiom

src_install() {
    DOCS="foo.txt bar.txt"
    distutils_src_install
    dohtml foo.html bar.png
}

Note: distutils.eclass defines the DDOCS variable with common doc file names and they are installed by default as docs. If the name of the doc file you want to install is there you don't have to specify a DOCS variable.

Code Listing 2.4: DDOCS variable in distutils.eclass

DDOCS="CHANGELOG KNOWN_BUGS MAINTAINERS PKG-INFO CONTRIBUTORS TODO NEWS"
DDOCS="${DDOCS} Change* MANIFEST* README*"

twisted.eclass

TBD

3. Ebuild writing

Here are some important things to keep in mind when writing ebuilds for python packages.

Calling python in ebuilds

To call python in one of your ebuilds, use "${python}" to do it (defined in distutils.eclass).

Checking for tkinter support

To check whether python has tkinter support you can use one of python_tkinter_exists (from python.eclass) or distutils_python_tkinter (from distutils.eclass) in pkg_setup. These two functions are identical.

4. Common Problems and Mistakes

Below are common problems you may face and common mistakes made when writing ebuilds for python packages.

setuptools: *_require and use_setuptools()

Important: For setuptools-0.6a9 and newer you no longer have to remove _require options other than tests_require because starting with this version --single-version-externally-managed is made automatic when --root is used which solves the problem. The new distutils_src_unpack function handles use_setuptools() problems. The methods explained in this section - i.e. removing _requires and use_setuptools() with sed - shouldn't be used anymore.

Packages that use setuptools to install use _require options like tests_require,install_require,setup_requires in setup.py. These are nice to learn about dependencies but you don't want them in setup.py when you're installing the package. The following is from the setuptools homepage section on setup_requires:

A string or list of strings specifying what other distributions need to be present in order for the setup script to run. setuptools will attempt to obtain these (even going so far as to download them using EasyInstall) before processing the rest of the setup script or commands.

—setuptools developer's guide

We have lovely package managers which can download stuff for us and verify their digests thus we don't want to download any packages using EasyInstall. There are other options like tests_require, install_requires that behave the same way.

Some packages have a ez_setup.py along with the usual setup.py. This is a python script to download and install appropriate setuptools. To do this use_setuptools() is called from ez_setup.py before importing setuptools.

Code Listing 4.1: use_setuptools() from ez_setup.py

def use_setuptools(
    version=DEFAULT_VERSION, download_base=DEFAULT_URL, to_dir=os.curdir,
    download_delay=15
):
    """Automatically find/download setuptools and make it available on sys.path
    [...]

Just like the _require options, if a setup.py script calls use_setuptools() from ez_setup.py you should remove it. Below is an example which illustrates how to do it.

Code Listing 4.2: setup.py of dev-python/myghty-1.1

from ez_setup import use_setuptools
use_setuptools()
from setuptools import setup, find_packages

[...]

install_requires=["Routes >= 1.0", "Paste", "PasteDeploy", "PasteScript"],

[...]

Code Listing 4.3: myghty-1.1.ebuild

src_unpack() {
    unpack ${A}
    cd "${S}"
    sed -i \
        -e '/use_setuptools/d' \
        -e '/install_requires=\[.*\],/d' \
        setup.py || die "sed failed"
}

src_test and PYTHONPATH

When testing python packages it's important to make sure we're actually testing the package that is going to be merged not the already installed package. We can solve the problem by setting the PYTHONPATH environment variable which augments the default search path for module files. Here are two examples:

Code Listing 4.4: src_test of dev-python/mechanize-0.1.7b -- a pure python module

src_test() {
    PYTHONPATH=build/lib/ "${python}" test.py || die "tests failed"
}

Code Listing 4.5: src_test of dev-python/pyme-0.6.0-r1 -- a python module written in C

src_test() {
    PYTHONPATH="$(ls -d build/lib.*)" "${python}" examples/genkey.py || die "genkey test failed"
}

As you may have noticed if the module is written in languages like C, C++, etc. the name of the directory in build varies between architectures but it always starts with lib.

Is dev-python/setuptools an RDEPEND or DEPEND?

repoman may issue a warning saying dev-python/setuptools is a suspicious RDEPEND. Note however that setuptools is quite often a run-time dependency by code that installs commands in /usr/bin, uses pkg_resources to require specific package versions or makes use of entry points in a plugin system.

If you emerge a package that uses setuptools and it installs commands in /usr/bin you can look at those commands and easily determine if setuptools is required at run-time.

Code Listing 4.6: Example of code that requires setuptools at run-time

    #!/usr/bin/python
    # EASY-INSTALL-ENTRY-SCRIPT: 'doapfiend==0.3.4','console_scripts','doapfiend'
    __requires__ = 'doapfiend==0.3.4'
    import sys
    from pkg_resources import load_entry_point

If the script imports pkg_resources then dev-python/setuptools must be in RDEPEND.

You can also use dev-python/yolk to determine if a package uses setuptools entry points by giving it the Python module name (not Gentoo package name).

Code Listing 4.7: Example of a Python module that requires setuptools at run-time

    $ yolk --entry-map paste

If there is any output, then dev-python/setuptools should be in RDEPEND.

The contents of this document are licensed under the Creative Commons - Attribution / Share Alike license.