python-r1 Policy Guide

1.  Common policies

State of Python implementation support in Gentoo

The Python implementations supported by the python-r1 suite can be divided into four groups:

1. Current stable implementations — the implementations which are enabled by the default value of PYTHON_TARGETS. All Python packages are supposed to support at least one.
2. Supported implementations — the new versions being tested as well as the old ones still being supported. Developers are encouraged to ensure support for those implementations and apply the necessary patches.
3. Unsupported implementations — the implementations which are still in the tree but their maintenance is undesirably resource consuming. Developers are not expected to support those packages by patching, and can drop them from PYTHON_COMPAT at will.
4. Deprecated implementations — the implementations being phased out. Developers should not add those implementations to PYTHON_COMPAT without prior request from user and discussion with the Python team.

The state of Python implementations is listed in the following table:

Adding implementations to PYTHON_COMPAT

The PYTHON_COMPAT variable should be treated with respect similar to that given for KEYWORDS. No implementation should ever be listed without prior testing.

Any developer is allowed to enable additional implementations in an ebuild of his own accord or upon a user's request. However, the package should be tested with an adequate level of scrutiny on at least one of the keyworded architectures. The testing may involve running a test suite, testing the package's Python scripts, or a basic subset of the package's API.

Before enabling the support for a Python implementation, the developer should search Gentoo Bugzilla for any open bugs regarding the package and the implementation in question.

The list of enabled implementations in stable packages must not be changed. Therefore, if a package is stable, it should be revision-bumped and the new implementation added to the new, non-stable ebuild.

When a Python package is being submitted for stabilization, some of the supported implementations may lack stable keywords. In that case, the ebuild should be revision-bumped and the conflicting Python implementations removed from the ebuild being stabilized.

When a new Python package is being added to a repository, it must be tested with at least both current stable implementations. Developers are encouraged to test additional Python implementations. The deprecated implementations must not be added without prior permission from the Python team.

Use of virtual packages

The virtual packages are used mostly to provide a consistent way of depending on Python packages. They serve as a replacement for dependencies which would normally need to be made conditional upon a particular set of Python implementations.

# Non-virtual solution
RDEPEND="
	python_targets_python2_5? ( dev-python/argparse[python_targets_python2_5] )
	python_targets_python2_6? ( dev-python/argparse[python_targets_python2_6] )
	python_targets_jython2_5? ( dev-python/argparse[python_targets_jython2_5] )"

# Virtual solution
RDEPEND="virtual/python-argparse[${PYTHON_USEDEP}]"

The Python ebuilds should use virtuals whenever they need to express a dependency which varies through enabled Python implementations. The following table lists all virtual packages available to date and cases when they can be replaced with direct dependencies:

Dealing with test suite failures

Test suite failures are a common problem in Python package maintenance. However, test suites are designed to provide the first sign of problems with a given package, or an implementation-package combo and therefore must not be disregarded.

Each test failure should be thoroughly analyzed. A test failure can be a symptom of;

1. an error in the package code or its incompatibility with a given Python implementation,
2. an error in the test suite or its incompatibility with a given Python implementation,
3. an error in the ebuild.

In the former two cases, the issue need be reported upstream. Unless a fix (patch) is provided by the ebuild, a matching bug report in Gentoo Bugzilla need also be filed.

If the issue is specific to a Python implementation and is a result of incompatible package code, then that implementation should be dropped from PYTHON_COMPAT. If the origin of the issue is unknown, the same course of action is acceptable.

If dropping would affect a major Python implementation or cause major dependency issues and an older package version does not exhibit the issue, the new version should be masked. If there is no other useful version of the package available, the implementation may be preserved.

Skipping or restricting the tests conditionally upon a given Python implementation is unacceptable. It is acceptable for a package to fail tests but it is not to pretend that there are no issues.

Adding patches, restrictions and work-arounds

Whenever there is a need to fix a bug affecting a package via introducing changes in the installed Python code, please remember to open a bug upstream and submit the patch there. If the issue is not urgent, it is preferable to await upstream acceptance of the patch and backport the change performed by upstream whenever appropriate.

It is generally preferable to use patch files rather than sed statements. If an ebuild uses sed statements, those statements need be reviewed on each version bump and removed when no longer necessary.

In cases that require any of the following;

• adding patches or sed statements to an ebuild,
• restricting tests in an ebuild (RESTRICT=test),
• removing (or restraining from adding) Python implementations from PYTHON_COMPAT,
• disabling out-of-source builds or parallel builds,

an appropriate comment should be added to the ebuild, documenting the change and the reasoning for it. This will help other maintainers avoid mistakes and yield better testing of the ebuild when next bumped.

2.  Development tips

Running tests in Python packages

If a Python package provides tests suitable for automated testing, the ebuild shall run those tests in the python_test() (or src_test()) phase.

There are a number of common solutions for running tests — including built-in unittest module, dev-python/nose, dev-python/pytest. Often the solutions are compatible with one another, enough to be able to run a test suite designed for one of the other tools.

There are two common rules when choosing the test runner:

1. the test runner shall introduce the least possible number of dependencies,
2. the test runner chosen by upstream ought be preferred.

When in doubt, it is often useful to review the package's test modules for imports. If a package belonging to a test suite is imported, the runner for that test suite shall be used. The common test suites are listed in the following table:

If the package defines the test command for setup.py and uses one of the fore-mentioned test suites, the best solution is the one requiring the least effort or smallest number of dependencies.

There are a number of packages which require the unittest2 module only in versions of Python 2 older than 2.7 and Python 3 older than 3.2. Those packages shall depend on virtual/python-unittest2.

If a particular package requires unittest2 for Python 2.7+ and 3.2+ only for setuptools discovery module, it is preferred to depend on the virtual package along with a python_test() function similar to the following snippet:

DEPEND="test? ( virtual/python-unittest2[${PYTHON_USEDEP}] )"

python_test() {
	local runner=( "${PYTHON}" -m unittest )
	if [[ ${EPYTHON} == python2.[56] || ${EPYTHON} == python3.1 ]]; then
		runner=( unit2.py )
	fi

	"${runner[@]}" discover # ...
}

The tests need to die on failure.

3.  Project-specific policies

Deprecating and removing Python implementations

If the Python team agrees on deprecating and removing an old Python implementation, the following steps need be taken first;

1. the deprecation and its timeline for it shall be announced, at minimum, on the project mailing list,
2. the implementation shall be marked as ‘deprecated’ in the policy guide,
3. the implementation package and USE flags shall be masked.

Post deprecation, the ‘phasing out’ period begings. The developers must neither enable the deprecated implementation on new packages, nor remove it from existing packages. Developers must especially take care not to break any reverse dependencies.

When the phasing out period ends, the following steps need be taken;

1. the implementation shall be removed from the policy guide,
2. the eclasses shall be modified to ignore occurences of the implementation in PYTHON_COMPAT,
3. the implementation package shall be treecleaned and the relevant USE flags shall be removed from profiles.

Upon completion of all the fore-mentioned steps, the implementation occurences in PYTHON_COMPAT can be removed at leisure.

The contents of this document, unless otherwise expressly stated, are licensed under the CC-BY-SA-3.0 license. The Gentoo Name and Logo Usage Guidelines apply.