Gentoo for Mac OS X project targets

1.  Targets

Global

The Gentoo for Mac OS X project tries to make the rich Portage tree available for Mac OS X users. The nature of the OS X operating system, which is based on the open source Darwin kernel, allows many applications to be compiled and run on OS X.

There are many ways in which Gentoo can be applied to Mac OS X. This document tries to outline these ways and point out the current focus of development.

There are two relevant concepts that need to be understood before we can delve into the different routes there are. On the one hand, there is the Portage software, which is unique for the Gentoo distribution. Portage is sometimes considered to be "just another package manager", but is primarily focussed on building applications from source. On the other hand, there finally is the 'tree' that the Portage software uses. The tree contains the packages, how to download, compile and install them. Both are useless without each other, as an engine without fuel clearly doesn't run, while fuel without an engine doesn't make you move either.

The 'Gentoo' in "Gentoo Portage" not only tells the Portage software is developed and used by Gentoo, but also that the tree is utilises is one that is built for the Gentoo distribution: a GNU/Linux distribution. It is important to note that the tree encodes logic and assumptions. For the Gentoo tree this currently means it contains GNU/Linux logic and assumptions for mainly the x86 architecture.

Finally, a package manager can be primary, or secondary. In the first case, the package manager is responsible for building the (core) system, while in the latter case, the package manager builds on top of an existing system to enhance it. Please note that this is by no means meant as a formal definition of the general concept "package manager". Within this document it is used as entity that is natively responsible for managing software packages on the OS it works on. Effectively, this means that Gentoo Portage is not the primary package manager on Mac OS X. Gentoo Portage is the primary package manager on its GNU/Linux distribution, and as such, the Gentoo Portage tree matches this goal.

Gentoo for Mac OS X

Gentoo for Mac OS X is the Gentoo Portage software, and its tree put on top of Mac OS X, just as it would be on a GNU/Linux system where Portage would be the primary package manager. However, Portage will not manage the (core) system packages, because doing so might break the OS it is running on (which has its own package manager).

The current incorporation of the Gentoo for Mac OS X project in the Gentoo tree is like this. It installs files in the same places as it would on a GNU/Linux system, in the same directory as the OS itself places its files. This behaviour has its advantages and disadvantages.

• Because Portage installs next to the OS, in the same directories, no fancy path settings are necessary, everything is installed in default places. Note that the relatively large overlap of the Gentoo Linux file system layout and Darwin's one, causes this advantage.
• Portage can use tools provided by the OS, this avoids unnecessary compilation and waste of space.
• Once a tool provided by the OS is outdated, or not recent enough, Portage cannot update it without overwriting the OS provided one. Note that this doesn't necessarily have to be a drawback, as from a progressive viewpoint this allows more control over the system, e.g. breaking a possible lock-in by the OS vendor. In that sense, also downgrading a package version applies.
• The vendor of the OS that provides a tool, might have changed the behaviour of the tool for some reason. The Gentoo Portage tree doesn't know about this, which can cause trouble. In such case the tree has to be modified in order to use it.
• Because Portage installs its files next to where the OS installs them, the primary package manager of the OS might install a package and overwrite files provided by Portage. It might be hard for Portage to figure this out in time, and possibly execute correcting actions.

In reality, this route seems to be a nightmare, even though the advantages look nice. Many of the tools provided by Mac OS X are too outdated or just slightly different. This causes a lot of packages not being able to compile because of unmet dependencies, which would collide with the OS. Many changes made to the Portage tree are also not always trivial.

Darwin Portage

To overcome the problems of outdated software, and Portage not being able to update them -- because it is a secondary package manager -- Portage can be made the primary package manager. This involves support in Portage's tree to build an OS X system up from the ground.

The open source projects behind Apple's Mac OS X are made available through the Darwin project. As such, a tree can be built which uses these sources. Non open source proprietary Apple packages can be installed on top of it using official Apple Mac OS X distribution media.

• Because Portage is the primary package manager, it is in full control, and no collisions with another package manager will happen.
• Since Portage will be the one to decide how and when the OS and extra tools are installed, it will be possible to leave out much unwanted software.
• Building a system from scratch requires skills, usually not available to Mac OS X users. It also involves more knowledge of the system, as Apple's tools to update the system will probably not work, or hork the system.
• A tree that encodes the logic and assumptions of the Darwin software packages needs to be built. The question remains how much of the efforts of the Gentoo Portage tree can be reused for this.

All in all, this way is attractive to skilled system administrators, and it offers a wide variety of solutions or extra opportunities. However, it seriously affects the system, and it remains questionable whether such system can be called a Mac OS X system. Several efforts will have to be spent to make this work.

Gentoo Portage for Mac OS X

With a slight difference in the name from the current project form, Gentoo Portage for Mac OS X tries to use the Gentoo Portage tree to its full extend on an Mac OS X system by letting Portage be as close as possible to a primary package manager, while in fact being a secondary package manager.

Although it might sound like a contradiction in terminis, in this direction Portage is made a primary package manager, while in fact it functions as a secondary one. This is possible by giving Portage its own corner in the file system, allowing it to fill it as it desires to build up a system there. This requires Portage to be able to install in a so called 'prefix', which is an offset given to all installed applications and files, such that they don't collide with the OS provided utils.

• By installing in a prefixed location, no collisions with OS provided files can exist, allowing Portage to have tools as up-to-date as it likes, or compiled the way it wants to.
• Because Portage has control over each and every application it installs, the Gentoo Portage tree can be used for Mac OS X, since logic and assumptions can largerly be applied to the sub-system Portage creates on Mac OS X. (The prefix.)
• Because Portage creates a whole sub-system, the prefix location is perfectly suitable for a removable medium, such that Portage and its installed applications can be shared, carried with one, or whatever. The sub-system itself should run on each system that has a sufficient hardware architecture, which can be a great feature for many reasons.
• Because Portage has to provide every tool it uses itself (it builds its own system), this means that many tools that are already available from the OS are not used, although perfectly suitable for the job. Additionally, this means space wise many applications will be double, hence many bytes wasted.

Even though this solution wastes space, it does provide a solution for insufficient OS provided packages, as well as a keen focus on shareability and protection from side effects caused by OS upgrades or repairs.

An obvious extension to this route would be to sacrify shareability for space, and use OS provided tools as long as they are sufficient. This would require a way for Portage to figure out what is available in the OS, probably more dynamic than what is done now for "Gentoo for Mac OS X".

2.  Roadmap

Current Situation

The current state of the Mac OS X project within Gentoo implements the "Gentoo for Mac OS X" route. This route has proven to be unsuitable for Mac OS X, as the OS itself becomes a too big limitation for Portage to sucessfully emerge many packages, if one is not willing to overwrite packages, as in the progressive profile. Since many people prefer not to touch the OS provided files, another direction is necessary.

Because "Darwin Portage" requires efforts towards a tree to be made, and because it is a rather big difference from a regular Mac OS X system, it was chosen to change the direction towards "Gentoo Portage for Mac OS X", to enable Gentoo's Mac OS X project to continue and innovate again. The prefixed location will avoid collissions, hence allowing many more packages to install.

This implies that development has shifted from the live system in portage to a new system only available to developers. Maintenance is being done, mainly for security bumps, but large investments are not being done, unless they are directly reusable in the prefixed environment.

The Gentoo for Mac OS X installer that floats around the net currently is old and contains many bugs. It has been removed from the official project pages, because the use of it is no longer being supported. We hope to release an updated installer to fill in this gap.

Current Status

Because "Gentoo Portage for Mac OS X" requires Portage to be able to install in a prefix in order to create a sub-system, Portage first has to be able to properly do this. After this, the Gentoo Portage tree has to be converted to be prefix-aware.

Prefix support in Portage has reached an alpha level in the development branches. This -- at the moment clumsy -- but more or less usuable product is used to find out what changes will be necessary to the Gentoo Portage tree. The major work of making Portage prefix aware has been done, and the functionality has to mature. The feature is considered all but stable. Portage development in this area is done with support from the Portage Development Team.

Because the Prefixed portage is more or less usuable, a few Gentoo for Mac OS X developers have it installed to experiment with it. Because the changes that need to be made require ebuild modifications that are not backwards compatible, applications are currently ported in an overlay. This overlay currently contains applications like perl, python, apple-gcc, vim, sed, awk, find, etc. A full base install is covered, based on ebuilds in the Gentoo Portage tree.

Milestones

A few key points on the road towards implementation of Gentoo Portage for Mac OS X can be indentified. These milestones are as follows:

1. Portage Prefix Support - partial done
   In order to start doing anything, Portage has to be changed to support prefixed installs. This is a large and non-trivial change that takes both time and lots of testing.
2. Prefix-aware Portage Tree - limited core done
   A Portage with prefix support, needs prefix-aware ebuilds, just like a diesel engine doesn't run with gasoline. Because the aims are for minimal changes to the Gentoo tree, it should be a trivial task to convert ebuilds from the live tree into something suitable for Portage with prefix support. Many more complicated packages, however, need special treatments in order to recognise the prefix they should install in correctly. Ebuilds that have been ported are put in the prefix overlay. This overlay should at minimal contain 20-30% of the live tree in order to be considered as representative sample.
3. Utilise Prefix-aware Portage
   Although the previous milestones implicitly require the product to be used, this milestone is here to explicitly require the prefix-aware portage to be used. This does not solely mean for OS X systems only, but also for Linux systems or any other party that would like to join in this development. It is very important to have a few use cases, in order to underline the fact that a prefix-aware portage isn't just an OS X thing. This milestone forms funding for the next.
4. Make Prefix Aware Portage Mainline Product
   This step will require a lot of efforts in the form of a Gentoo Linux Enhancement Proposal (GLEP), but is a very critical step in the whole path to get the changes made accepted by the Gentoo community as a whole. The changes required for prefix-aware portage are everything but trivial, and hence need approval by the community at large. At this stage it helps to have several use cases, a fair share of the Gentoo Portage tree converted and some really strong armour. This step won't be easy, and will require a lot of efforts in writing the GLEP. However, only if the GLEP will be accepted, we can move on to the next milestone. In case of a reject, we are outlaws.
5. Make Gentoo Portage Tree Prefix-aware
   A logical conclusion of the accepted GLEP is to get the full tree ported to the prefix-aware format. Since the GLEP is accepted, this might be done with some help from others to speed up the process. Again this task is non-trivial, since updates occur all the time on the live tree, and they need to be reflected in the new tree somehow.
6. Roll Out Prefix-aware Portage
   This is not our job, but it is the end of this road. In order to be able to use the features of the tree, a capable Portage has to be made the mainline product.

The contents of this document are licensed under the Creative Commons - Attribution / Share Alike license.