Gentoo Prefix Use Cases

1.  Gentoo Prefix - what's in it?

Introduction

Usually when one thinks of Gentoo, its Linux distribution comes to mind. However, there is more that Gentoo has to offer. To the extreme, Gentoo has its Alt project that deals with non-Linux operating systems. Most visible in Gentoo/Alt are currently the BSD and Prefix projects. While the former focusses on providing some form of Gentoo on all kinds of BSD systems, the latter targets nearly every platform including Gentoo Linux itself. What both have in common, is that they use Gentoo Portage to install software on the target systems.

Among the various Linux distributions, Gentoo is one of the few that is source based. In fact, Gentoo is a meta-distribution, which more or less means that Gentoo provides how to install a package, instead of the package itself in a binary form. This property allows to easily reuse the meta-data that exists for a package to install it on another kind of platform. This is one of the reasons why the Gentoo Linux distribution targets so many (rare) architectures: reuse allows for easy maintenance. The Gentoo/Alt project builds further on this particular property. With relatively small efforts, the meta-packages (ebuilds) can be made to install from the same sources on non-Linux operating systems.

In this article we focus on the Prefix project. The project itself is experimental, thus not intended for use in any settings where an experimental nature is not appreciated, such as production environments. However, the early experiences with the project are interesting enough to be discussed in this article. Gentoo Prefix has some unusual properties, which makes it useful for a number of occasions. Two use cases follow.

Use Cases

Case 1: A student at the University

A student X wants to compile and run some self-developed software on the Sun workstations provided by his university. The development tools required for the student's project go far beyond what the system administrators like to install, and other than that, they don't have time to look into it. With the student obviously having no administrative privileges to the Sun workstations, he cannot "just" install the required software tools himself. Even if the student wants to use non-official package installers, he still needs administrative privileges, as those installers like to place the binaries in places which by default are not writeable for the student. The only option left for the student is to compile and install the software he needs manually himself. This typically requires some time, expertise and work. Not always does software compile out of the box. Sometimes due to "dependencies" on other packages, and quite often a number of packages needs to be installed in the right order before it all works. A lot of work, just to get the self-developed package only to compile.

Case 2: Companies deploying software

Company Y has its own product P. By the nature of P, it is compiled at the customer's machines, highly tuned and configured for the customer's needs. Also the maintenance, patches and requirement changes are done by Y. Unfortunately, the customers of Y are very much bound to the machines and operating systems they use on that. Big contracts, and secured "trust" in the form of SLAs and years of experience make Y's customers want to use whatever they have been using for years. Even though administrative privileges are within reach of Y for the systems to build on, Y likes to avoid that since not only Y's software runs on the systems. A fix for one, can break things for the other. The nature of Y's customers simply doesn't allow breakage, hence changing the system is the very last resort. However, for Y to install its product, it needs some more recent, or extra development tools. Like the student from case 1, the only option left, is to manually compile and install what is necessary, including the burden of tracking the "dependencies".

Next to these two use cases, many others exist. It is not hard to think of not being at an university, but on a company workstation or laptop. Or just that a user doesn't like to "pollute" the main system by installing (non-vendor) packages. In all cases, Gentoo Prefix provides a solution. Simply put, Gentoo Prefix allows to install packages from source, without administrative privileges, into a custom location in the file system hierarchy (the prefix). Since Gentoo Prefix uses the Portage package manager, it also deals with all (possible) dependencies, and compile time options to enable or disable certain functionality. Last, but not least, the packages provided are -- as usual within Gentoo -- fairly up-to-date, so the chance for "outdated software" is really small.

Installing Gentoo Prefix

How does Gentoo Prefix work then? Without the details, it works like a normal Gentoo Linux system: packages are simply emerged. However, before one can do that, Gentoo Prefix first has to be installed. While at this time of writing there is no installer or live-CD, the only way to install Gentoo Prefix is by bootstrapping using a special script. Detailed descriptions of how to do that on for example Mac OS X can be found on the project website. Installing via that documentation remains the only way for as long as "one button" installers are not available. For users that are familiar with installing Gentoo Linux without the installer just by following the documentation, will find some resemblance in how Gentoo Prefix is installed. Step by step instructions, with per step explanation of why and what for.

The bootstrapping process consists of three phases, 1) installing the core tools, 2) using Portage to emerge a base system, and 3) using portage to rebuild all of the installed packages with custom optimisations and features.

While the whole process is boring overall, it results in an up-to-date and ready to use developer (sub-)system. Phase 1) is mainly installing the tools that Portage itself needs to function properly. Packages that are installed here are a recent Python, findutils, and so on, in total a number of 10 packages. The final step of this phase is to install Portage itself. This allows to continue with the next phase, where packages are only installed using Portage. Phase 2) starts by emerging build utilities in an order such that dependencies are met. Because these utilities are so "basic", Portage often doesn't take them into account as package dependencies, since they belong to the base profile. Hence, the dependency calculation of Portage will not create a correct graph for these packages, resulting in failures during compilation. For this reason, A number of packages is emerged without dependency calculation in a fixed order. After this the final step of this phase is to use the dependency system of Portage to emerge the special target "system", which includes all the packages that should be available on a system to function properly. Once all these packages are available, the last phase, 3), can be executed. In this final phase, the tree with ebuilds is updated to the most current version, and the compile time flags and features can be set. After that, it mainly does a recompilation of all packages compiled before, and adds some packages that may be necessary for the updated features ("USE flags"). While recompiling all packages previously compiled may sound like a waste of time, it is done on purpose. First, in order to have the new compile time flags become active, packages have to be re-emerged. Since these compile time flags (CFLAGS) often contain optimisations which greatly affect the speed and performance of the utilities, it is worth to recompile for this. Second, packages emerged in the second phase, sometimes still use tools provided by the operating system being built on, instead of those tools installed by Portage. These need to be corrected, which is done by a recompilation.

After these phases, the (sub-)system is ready for use, and packages of choice can be emerged in the system. These packages can be libraries (libpcre, libxml2, ...), applications (vim, mutt, ...) or whole graphical toolkits, such as QT and GTK.

What is this Prefix?

So far we have discussed what one can do with Gentoo Prefix. The curious reader, however, may wonder how it is done, and why it can do all of this installing without causing trouble. The first thing that needs some explanation, is the name of the project itself: Prefix. A prefix is some part put "in front of" something else. In this case, it is a path put in front of all other paths that Portage deals with. The main reason why administrative privileges are often required, is because the software is finally installed into a location that a normal user in the system can write to. This is good, as it protects the system from getting broken by mistakes from unknowing users, but prevents the user from installing the software. So the approach taken in Prefix Portage is to deviate from these default locations, and shift everything into an offset. This offset, which is prefixed to every path Portage uses, can be freely chosen. This way, a user can install into his own home directory, or to some large disk space location which he has write permissions for. And here is the reason why Prefix works for what it wants. It installs into there where the user has full privileges to do what is necessary: installing software.

Guarded with this offset, knowing that everything is installed into this location, usage of the prefix consists of using everything from this prefix, before even looking at what the main operating system has installed. This is what Portage in the prefix does. It simply changes the search paths such that the prefix comes first. Simple as that, as far as Portage is concerned. Knowledgeable readers on compiler and linker workings will immediately wonder how Portage deals with them and the offset. The simple answer is that compilers and linkers installed by Portage in the prefix are configured such that they look into the prefix first as well. Not only that, but also does it make sure that compiled programs will keep on looking in the prefix via rpath directions, which are a bit too detailed for this article to discuss. The conclusion is that programs compile and run using prefix provided tools and libraries.

Conclusions

Gentoo Prefix is an easy way to enable using software installed on a user base. This in particular is interesting for situations where administrative privileges are not available or simply not suitable. Gentoo Prefix is like Fink, MacPorts or pkgsrc for Mac OS X systems, but not limited to this operating system alone. The nature of Portage -- from the Gentoo philosophy -- allows fine grained control over how packages are installed, which is brought to users of many platforms. Last but absolutely not least, we like to stress that Gentoo Prefix is not a "finished" product. It is a proof-of-concept that currently (quite successful) explores the possibilities of using Gentoo Portage on systems other than Gentoo Linux. Unprivileged installation of Gentoo Prefix is one of the drives that allow the project to be quite successful so far.

The contents of this document, unless otherwise expressly stated, are licensed under the CC-BY-SA-2.5 license. The Gentoo Name and Logo Usage Guidelines apply.