python-r1 User's Guide

1.  Multi-implementation support

Short introduction

Currently, Gentoo supports three different Python interpreters — CPython, PyPy and Jython. CPython has two distinct major branches — called Python 2 & Python 3. These are the four interpreter groups.

Each group is further sub-divided into branches, which are identified by minor version numbers: Python 2.7, 2.6, 2.5, 3.3, 3.2, 3.1; PyPy 1.9 and 1.8; and lastly, Jython 2.5. All these versions can be installed in Gentoo concurrently, and are collectively referred to as Python implementations.

There may be various reasons to install multiple Python implementations. The most important one is that many Python packages in Gentoo still don't support Python 3 while a few require it. Therefore, in order to maintain compatibility with the both kinds of packages, Gentoo enables both versions by default.

Having two or more implementations installed allows our users to run all kinds of scripts. It gives them an ability to test their own scripts against multiple Python implementations.

PYTHON_TARGETS and PYTHON_SINGLE_TARGET settings

Having a number of implementations to choose from, the modern Python packages provide users with an ability to explicitly select one or more Python implementations. This selection is made via USE flags.

There are two relevant expanded USE variables:

1. PYTHON_TARGETS,
2. PYTHON_SINGLE_TARGET.

The PYTHON_TARGETS variable is used whenever more than one Python implementation can be enabled at the same time. Therefore, it can list multiple interpreters.

The PYTHON_SINGLE_TARGET variable is used whenever only a single Python implementation can be enabled, and it takes precedence over PYTHON_TARGETS. It can list one implementation only.

By default, Python 2.7 and 3.2 are enabled, with Python 2.7 being used for packages not supporting multiple implementations. Those defaults can be overridden through specifying the relevant variables in make.conf. The following values can be used:

1. pythonX_Y for CPython X.Y;
2. pypyX_Y for PyPy X.Y;
3. jythonX_Y for Jython X.Y.

PYTHON_TARGETS="python2_6 python2_7 python3_2"
PYTHON_SINGLE_TARGET="python2_7"

If it is necessary to change the enabled implementations for a single package, the expanded form of the USE flags can be used in package.use file:

app-portage/flaggie python_targets_python3_1
net-libs/libproxy python_targets_python3_1 python_single_target_python3_1 -python_single_target_python2_7

The implementations enabled for a package can be listed by the output of emerge --verbose. If a package lists the PYTHON_SINGLE_TARGET, the enabled implementation is used, rendering PYTHON_TARGETS moot. Otherwise, all implementations enabled in PYTHON_TARGETS are used.

$ emerge -pv app-portage/flaggie net-libs/libproxy

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R    ] app-portage/flaggie-0.2-r2::gentoo-cvs
                   PYTHON_TARGETS="python2_6 python2_7 python3_2 -python3_1" 0 kB
[ebuild   R    ] net-libs/libproxy-0.4.10-r1::gentoo-cvs
                   PYTHON_SINGLE_TARGET="python2_7 -python2_6"
                   PYTHON_TARGETS="python2_6 python2_7" 0 kB

Total: 2 packages (2 reinstalls), Size of downloads: 0 kB

Selecting active Python implementation

The app-admin/eselect-python package provides a tool to select the currently active Python implementations. It provides the choice of ‘active’ Python 2 and Python 3 interpreters, and which one is preferred.

The selections done through eselect-python influence:

1. the interpreter started by python, python2 and python3 commands,
2. the interpreter used to start non-versioned Python scripts.

In order to obtain a list of installed Python interpreters supported by eselect-python, use the eselect python list command:

$ eselect python list
Available Python interpreters:
  [1]   python2.5
  [2]   python2.6
  [3]   python2.7 *
  [4]   python3.1
  [5]   python3.2

The asterisk highlights the currently active ‘main’ Python interpreter. The choice of main Python interpreter both influences the Python 2/3 preference and the active Python version in the appropriate group. In order to set another implementation, use eselect python set:

$ eselect python set python3.2
$ eselect python show
python3.2

Aside from setting the main Python interpreter, eselect-python is capable of setting the active version of Python 2 and Python 3. In order to enable that mode, the --python2 or --python3 parameter need be added after the command:

$ eselect python list --python3
Available Python 3 interpreters:
  [1]   python3.1
  [2]   python3.2 *
$ eselect python set --python3 python3.2
$ eselect python show --python3
python3.2
$ eselect python show
python2.7

The previous example emphasizes that setting the active Python 3 interpreter does not influence the main version of Python interpreter if it is Python 2. The same rule applies to setting active Python 2 when the main interpreter is Python 3.

Python script renaming

You may have noticed already that the default installation of Python scripts in Gentoo differs to that of other distributions (and Python packages themselves). This is done in order to completely support multiple Python implementations being used on the same system.

$ ls -1 /usr/bin/flaggie*
/usr/bin/flaggie
/usr/bin/flaggie-python2.6
/usr/bin/flaggie-python2.7
/usr/bin/flaggie-python3.2

Firstly, all Python scripts installed by packages are installed in per-implementation variants. This ensures that any changes necessary for a given Python implementation are preserved in the scripts.

Secondly, a wrapper is installed in place of the standard script. The wrapper is a very simple tool which checks which of the Python implementations that the script supports would be ‘preferred’, and runs that script variant.

2.  Compatibility with python.eclass

The dependency compatibility problem

Currently, most of the Python packages in Gentoo do not support explicit Python implementation selection via PYTHON_TARGETS. For that reason, it is common that packages supporting use of PYTHON_TARGETS variable can depend on those which don't, and vice versa. Therefore, it is necessary to ensure that those inter-dependent packages work correctly.

Solutions for common cases

Before proceeding, the current values of PYTHON_TARGETS and USE_PYTHON need be known. They can be displayed by running emerge --info.

This solution can be used only if PYTHON_TARGETS lists at most two targets; one CPython 2 version and one CPython 3 version. In any other case, the USE_PYTHON variable-based solution need be used.

The USE_PYTHON variable need be unset, i.e. not appear in the output of emerge --info. If it has been assigned any value and one of the following solutions is to be used, it should first be removed from make.conf.

$ emerge --info | grep -o '\(USE_PYTHON\|PYTHON_TARGETS\)="[^"]*"'
PYTHON_TARGETS="python2_7 python3_2"

The Python versions used in PYTHON_TARGETS should be the active Python interpreter versions. They can be set active using eselect python.

$ eselect python set --python2 python2.7
$ eselect python set --python3 python3.2

Additionally, if your PYTHON_TARGETS setting lists only a single Python version, no version from the other group may be installed. That is, if only CPython 2 is desired, no version of CPython 3 can be installed.

$ emerge --unmerge python:{3.1,3.2,3.3}

USE_PYTHON-based solution

If none of the fore-mentioned solutions suit you or you don't want to change the active Python versions, you can use the USE_PYTHON variable to enforce specific targets to python.eclass. The variable must list the same implementations as PYTHON_TARGETS. However, it uses different name semantics, therefore the values need to be translated as detailed in the following table:

PYTHON_TARGETS="python2_6 python2_7 python3_2 python3_3 pypy1_8 pypy1_9"
USE_PYTHON="2.6 2.7 3.2 3.3 2.7-pypy-1.8 2.7-pypy-1.9"

3.  Common problems

REQUIRED_USE flag constraints unsatisfied

In some cases, an attempt to install or upgrade a package may result in portage throwing an error of the type ‘unsatisfied REQUIRED_USE flag constraint’.

$ emerge -1 argparse

These are the packages that would be merged, in order:

Calculating dependencies \

!!! Problem resolving dependencies for dev-python/argparse
... done!

!!! The ebuild selected to satisfy "argparse" has unmet requirements.
- dev-python/argparse-1.2.1-r2::gentoo-cvs USE="(multilib)" PYTHON_TARGETS="-jython2_5 -python2_5 -python2_6 -python3_1"

  The following REQUIRED_USE flag constraints are unsatisfied:
    any-of ( python_targets_python2_5 python_targets_python2_6 python_targets_python3_1 python_targets_jython2_5 )

This error message means that you are attempting to install a package which does not support any of the Python implementations enabled on your system.

In the most common case, the package in question does not support the particular implementation of your choice. The issue could be solved through enabling one of the listed implementations by adding it to PYTHON_TARGETS.

In a few very specific cases, this might also mean that a particular package is unsuitable or unnecessary on your system. For example, the argparse Python module is included in CPython versions 2.7, 3.2 and newer, and thus installing it is not meaningful on systems using only those versions of Python. Then the issue should be solved through removing the offending package.

The contents of this document, unless otherwise expressly stated, are licensed under the CC-BY-SA-3.0 license. The Gentoo Name and Logo Usage Guidelines apply.