Gentoo Linux Kernel Maintainers Guide
This guide aims to document the processes used in the maintenance of Gentoo's
core kernel package, gentoo-sources. It is written in hope that it will help
future contributors get involved with these maintenance efforts.
At the core of gentoo-sources is genpatches: a generic kernel patchset,
designed to be minimal, Gentoo-independent, and accessible to other
distributions and projects. Please ensure you have a good
understanding of genpatches before continuing: read through the
This document was written by Daniel Drake, the kernel project lead. References
to "me" in this document refer to Daniel, but please do not feel that this is
a one-man project. Other members of the team are happy to help you as well.
Please subscribe to the gentoo-kernel mailing list. Subscription
instructions can be found
here, and archives
can be found here.
This list is low-traffic. I will announce modifications to this document on
You should also subscribe to the linux-kernel mailing list (LKML). This list is
extremely high traffic. You shouldn't even attempt to read it all,
but glance through the subject lines every day or two, and keep an eye out
for topics that interest you. It's also interesting to read the release
announcements from Linus and the -stable crew, and Andrew Morton's -mm release
announcements to keep up with what's going on in the kernel community.
Keep the LKML archived in its own folder in a good mail client. It becomes
very useful later on when you are researching bugs reported by users on the
Gentoo bugzilla. I try to keep at least the most recent 30,000 emails around.
To subscribe to the LKML, send an email to
firstname.lastname@example.org with the following in the body of the
Code Listing 2.1: Message body
subscribe linux-kernel YOUR-EMAIL-ADDRESS
You may also be interested in reading the
LKML FAQ. It contains lots of
information about how the mailing list is used and the kernel community in
general. I don't expect you to have any need to write to the LKML initially,
but do be sure to read this FAQ before you send your first email there.
Please be active on our IRC channel, #gentoo-kernel on freenode. The Gentoo
website has more information on
While learning, please ask us questions! This is the best way to learn. If we don't respond, ask again another time. We do want to help you, since hopefully that will result in you helping us in the future!
The primary kernel maintainers are Daniel Drake (IRC username dsd_) and Mike Pagano (IRC username mpagano).
The bulk of Gentoo kernel maintenance happens on
bugzilla - bug reports, keyword
requests, feature requests, etc. Make sure you have your own personal account.
You should now configure bugzilla to send you copies of all bugmails related
to kernel maintenance. To do this, go to
preferences and add email@example.com to your user watch list.
I suggest you set up a mail filter to move these mails to their own folder.
Another useful feature is bugzilla saved searches. You can perform a bugzilla
query, and save it with a user-defined name for quick access later. Links to
these saved searches are present on the bottom of every bugzilla screen. For
each of the following queries, please scroll to the bottom of the resultant
buglist and use the "Remember search" functionality to save each one:
Gentoo kernel bugs: This query lists all open bugs assigned to
firstname.lastname@example.org except ebuild submissions, tracker bugs, and 2.4 bugs. This
is the main buglist you will be working from.
Kernel regressions: This query lists all open regression bugs. A
regression is a bug which was introduced in a certain kernel release, whereas
the same functionality worked in releases beforehand. In many ways, regressions
are the most critical form of bugs.
Linux bugzilla: This query lists all bugs reported on the Gentoo bugzilla
which have then been reported upstream at the
We report the majority of bugs upstream to the kernel bugzilla,
should create an account here, and again, put email@example.com on your
email user watch list.
You'll probably find it handy to keep a subversion checkout of genpatches
Code Listing 2.2: Checking out genpatches from anonymous SVN
# svn co svn://anonsvn.gentoo.org/linux-patches/genpatches-2.6/trunk genpatches
Here's a list of useful websites which you'll see me referring to.
The kernel is a huge codebase, and is one of the packages which is present
on almost every users system and is used all the time. Our maintenance style
may seem strange at first, but please remember that kernel maintenance is
overall a huge task, and as such, one of our primary objectives is minimizing
the amount of work that we have to do (and overall make things more manageable).
One way we reduce our workload is to attempt to deviate away from the upstream
kernel releases as little as possible: we aim to make genpatches as small as
possible. The less patches we have to handle/backport/forwardport, the less
work we have on our hands.
Another approach we take is quick adoption of new kernel releases. Each new
kernel release fixes many bugs and adds more hardware support, which
invariantly affects proportions of our userbase. Instead of going to the
pains of backporting these code changes, we reduce our workload by encouraging
quick adoption of new kernel releases where this new code originates. Further
in line with these goals, we tend to stop supporting older kernels fairly soon
after a new release has been cut.
Even more time consuming than patch handling is bug fixing. Being such a huge
codebase, we get a steady flow of kernel bug reports on our bugzilla. Some of
them take considerable time/effort to get resolved. We work to streamline our
bug handling processes here.
Even after 3 years of bug handling, I personally am familiar with only a very
small proportion of the kernel codebase. It's important that we raise these
bugs to the attention of the upstream kernel developers who do know what they
are doing, and it's important that we do this quickly. As you will learn, we
don't actually do much of the fixing downstream in Gentoo.
With respect to the above forwarding of bugs to upstream developers, it's
important that we maintain good relationships with upstream developers. In
general, upstream developers are wary of distro-reported kernel bugs - after
all, most distributions patch their kernel releases so heavily that they aren't
easily supportable by people outside of those distributions. I recently
had to dig a patch out of the Mandriva patchset, and just out of curiousity, I
ran diffstat against their patches. The data set was so great that it revealed
an obscure bug in diffstat itself!
One way we attempt to get respect from upstream developers is by our patching
policy: other than in exceptional circumstances, we don't add any patches
until they have been merged into the official development tree upstream.
Additionally, this helps us to respect upstream decisions, and helps us ensure
quality (and upstream support) of our patchset.
The bugfixing process certainly varies from bug to bug, but many have things
in common. During the process I usually aim to do the following:
- Verify that it is not a user or configuration error
- Verify that it is not a Gentoo-specific distro bug
- Gather detailed information to classify the problem
- Send the bug upstream
With that in mind, bugs often follow a process such as the following:
- Initial evaluation
- Information gathering (including kernel test requests)
- Filing upstream
A user reports a bug to the Gentoo bugzilla. A bug wrangler reassigns the bug
You are notified of this new bug by email. You first run a few sanity checks
over it: is it an obvious user error? Is it a duplicate of another bug that
has already been reported?
The initial bug report is usually lacking in information, by no fault of the
reporter. Examples of common questions you might ask in response to the
intial report include:
Which kernel version are you using? This is important information
which is often omitted from the initial report.
Which other kernel versions have you tried? Sometimes the user knows that this is a long-term bug, and sometimes this information can be useful to
Are there any known previous working kernels? Here you're basically
asking "is this a regression?"
Can you reproduce this? Sometimes people post crash dumps with no
indication of whether this was a one-off after years of stability, or if it
reliably occurs soon after boot, or if they know of a way to trigger the crash
whenever they feel like it.
There are typically many other such questions that could be asked for most
bug reports, but many of these can be answered by requesting output of certain
commands or certain files. I find myself commonly requesting the following:
dmesg - this dumps the kernel log from the current boot to the
console. It contains all sorts of interesting information about the hardware
and drivers, as well as information about any kernel-level errors that may
.config - sometimes it's useful to verify exactly how the user has
configured their kernel.
lsmod output - when the user has built certain components as
modules, this is one way you can verify that they have been loaded.
lspci output - this provides a useful overview of the system
hardware and can be used to get an idea of which drivers the user should be
After requesting one piece of information, you might then realise that you
need to request something else. That's fine, just post a new comment with a
We generally accept bug reports for 2 kernel versions at any one time:
- The most recent kernel with stable keywords
- The most recent kernel with testing keywords
For example, at the time of writing, the most recent stable kernel is 2.6.20.
2.6.21 has just been released, and is present in the testing tree, but it
needs substantially more testing time before we are confident enough to mark
it stable. During this time we will accept bugs for the latest
gentoo-sources-2.6.20 release, and the latest gentoo-sources-2.6.21 release.
Even if a user files a bug on a currently supported kernel, we often ask them
to test the latest development kernel (e.g. 2.6.22-rc1 in the above example).
Why? We're moving to get this bug reported upstream as soon as possible, and
as a kernel developer, there simply isn't any point diagnosing issues on a
codebase which isn't current.
With that in mind, and the above example of current versions, there are several
situations which can arise:
- A user may report a bug with 2.6.19 or older. These kernels are not
supported. You first ask them to reproduce on any supported kernel (give them
the option of either 2.6.20 or 2.6.21), and if the bug still exists there, ask
them to test the latest development kernel (2.6.22-rc1).
- The user may report a bug with 2.6.20 (current supported stable
kernel). Ask them to test the current testing kernel (2.6.21), and if the bug
still exists there, request that they test the latest development kernel
- The user may report the bug with 2.6.21 (current supported testing kernel).
Ask them to reproduce with the latest development kernel (e.g.
For 2 weeks after the upstream release of a 'production' kernel, there is
a period in which there is no development kernel available. For example, 2.6.21
has only just come out and 2.6.22-rc1 will not be released for at least a week.
In this case, you can skip the stage where you ask the user to test the latest
development kernel -- reproducing the issue on 2.6.21 is enough to request
the upstream developers to work on it.
Researching the bug
When you're happy that you have enough information about the problem, you
should attempt to find other reports of the same issue. In some cases you can
do this without asking the user to test all the new kernels.
It is quite often the case that the bug has already been reported upstream and
there is already some discussion happening about how to fix it, or maybe a
patch is already available.
This part is really just common sense: stick some search terms into google.
If you have a distinguishable error message, search for that. Otherwise, search
for "linux 2.6.20 sky2" if you're researching a reliability regression in
2.6.20 with the sky2 driver.
Additionally, query the linux bugzilla with similar search terms, and use your
email client to search through your local LKML archives.
If you find something, that's great, reference it on the bug. You may even
have found a fix for the problem. It's certainly not uncommon for standard
searching not to find any references though.
Occasionally we get bug reports from people using vanilla-sources.
Whether or not we accept these bugs is a matter of judgement. vanilla-sources
is provided to aid testing, but is not supported: if it's broken, we don't
That said, if someone reports a bug in vanilla-sources-18.104.22.168 (the latest
upstream release) it is likely to also exist in the latest gentoo-sources-2.6.21
release, so it probably makes sense to accept the bug and go through the usual
diagnosis. However, if you feel that gentoo-sources may have a patch to fix
this, don't hesitate to remind them that vanilla-sources is unsupported and
they need to reproduce this on gentoo-sources before we'll look into it.
Sometimes, users report -rc kernel regressions, e.g. they report bugs which
exist in vanilla-sources-2.6.22-rc1 but not in 2.6.21. In these cases, you
can close the bug right away: you should thank the user for reporting the
issue but point out that they should report it upstream at the kernel bugzilla
directly. gentoo-sources does not do -rc releases.
Binary and out-of-kernel modules
It may become apparent through information gathering that the user is using
kernel modules/drivers which are not part of the official kernel sources. One
example of such a package would be x11-drivers/nvidia-drivers.
Again, these situations require some judgement. There's simply no way we can
support kernels which are modified at runtime in this way, so it is perfectly
reasonable to ask the user to reproduce the issue without those external
modules loaded (and also request that they have not been loaded since
boot, as opposed to them booting, loading and unloading the modules, and then
reproducing the issues).
That said, you can sometimes conclude that the bug exists through other means,
in which case there is no need to ask them to reproduce without the external
modules. For example, you might find an identical report on a non-tainted
kernel on LKML.
If you determine that a bug is a regression, there are a few extra things to
consider. First, you should tag it as so (read about the Status Whiteboard later
in this document). Secondly, you should note that the Linux source code
management utility (git) has a useful bisection feature which can identify
the exact commit which caused a regression. This really makes the bug a lot
easier to solve.
However, performing a bisection is a time consuming process: for identifying
a regression between versions 2.6.x and 2.6.x+1 it usually requires that you
build and test approximately 13 different kernels. So, unless you sense that
the user is particularly keen, you should probably exhaust other options first.
You could get the bug forwarded upstream and after a few days if there hasn't
been any progress, ask them if they will consider doing a bisection.
I have some pre-written instructions on how to do a bisection, which are
easy to follow. If you are going to suggest a bisection, I would suggest you
just drop them a link to
If the user doesn't respond
It's unfortunate, but fairly often, users turn unresponsive and don't provide
the extra information that you requested. While we aren't paid to fix these
issues and have plenty more to be working on, it's best just to discard these
After 14 days, if the user has not responded to my most recent request, I close
the bug as NEEDINFO with a comment asking that they reopen the bug when the
information has been provided. Quite often this spurs the user into providing
what was requested.
Reporting bugs upstream
You've established that this appears to be a kernel bug, you've gathered
information which agrees, and it's been reproduced on the latest development
kernel. The next step is to get the bug reported upstream.
There are 2 ways you can do this: by asking the user to report the bug at
the upstream kernel bugzilla, and by yourself sending an email to the relevent
lists and maintainers.
It's very important that we make it easy for the upstream developers here.
When reporting bugs, state clearly which versions are affected, mentioning
which ones are unmodified (usually we can say "it's present in 2.6.19-gentoo
but is also reproducible on unmodified 2.6.20-rc5").
While the procedure does involve linking to the downstream bug, make sure all
information is presented upstream - don't expect them to click and read through
the downstream bug, which probably contains some degree of noise anyway.
The procedure outlined below includes modifying the status whiteboard field,
which is explained further later in this document.
Reporting bugs on the kernel bugzilla
The kernel bugzilla is the primary mechanism we use to report bugs upstream.
Actually, the way we do it is by getting the user to file the bug report, and
then us tracking it. Here is the process I follow:
- Mark the Gentoo bug as UPSTREAM, and add "linux-bugzilla-pending" to the
status whiteboard. While doing so, add a comment asking the user to file the
bug upstream. Give them the kernel bugzilla URL, ask them to post the new bug
URL on the Gentoo bug when done, and provide some basic guidance for what
they should include in their bug report
- When the user responds with the upstream bug URL, place that URL in the
URL field of the Gentoo bug. Remove "linux-bugzilla-pending" from the status
whiteboard, and add "watch-linux-bugzilla". Add a quick comment stating that
we'll keep an eye on the upstream bug.
- Go to the upstream bug report. Read through it. Did the user miss any
important information from their description? If so, add a comment with that
information. If there are useful attachments on the Gentoo bug which the user
didn't attach to the upstream bug, attach them yourself with brief
- Add firstname.lastname@example.org to the CC list. If the user didn't mention the URL
to the Gentoo bug on the upstream bug report, add a comment with the URL.
If the user doesn't respond to the request to file the bug upstream, discard
the bug like you would for other unresponsive cases. Since you closed
it as UPSTREAM, it will drop off the buglist anyway -- so do remember to
continue the process when they do respond with the upstream bug URL. The linux-bugzilla-pending keyword allows us to measure how often bugs get lost at this point, and allows us to follow up.
When the bug gets updated upstream, email@example.com will get the bugmail,
and since you have that address on your watch list, you'll see it too. When
the bug gets fixed, reopen then downstream bug with a quick summary of the
solution, removing the "watch-linux-bugzilla" tag.
If someone closes the upstream bug without a fix (for example, the user
didn't respond to an information request, or the bug got marked as invalid),
reflect the changes downstream: mark the equivalent Gentoo bug as CLOSED and
remove the "watch-linux-bugzilla" tag.
Sending email bug reports upstream
To be written later. This is basically used as a fallback if the upstream
bug isn't getting the attention it needs.
Other bugzilla stuff
This document includes references to manipulating bugs on the Gentoo bugzilla
(closing/reopening/reassigning/etc). Unless you're a Gentoo developer or you're
the bug reporter, you don't actually have access to do this though.
If a certain action should be taken on a bug which you don't have permission
for, you should ask in #gentoo-kernel on IRC for someone to perform the
operation. If there is no response, leave a simple comment on the bug
requesting the change, e.g. "please close this bug as NEEDINFO"
I realise this is a pain, but it's only a temporary one. Once you've made
several contributions to bugs in this form, I will increase your bugzilla
access permissions so that you can perform these tasks on your own.
The other aspect of this problem is manipulating bugs on the upstream kernel
bugzilla. This needs to be done much less than bugs downstream. I personally
have developer access to the upstream bugzilla, so if anything needs doing
(bugs closing/reassigning/etc), prod me on IRC. In future we could look into
getting more people developer access here.
Bugzilla has a single-line status whiteboard text field for each bug, which
you are free to put anything in. I use it for certain kinds of tagging.
If a bug is present in 2.6.20, but fixed in 2.6.21 and the fix is not yet
known, I place "linux-2.6.21" in the status whiteboard field. This is
effectively tagging the bug saying "this problem goes away when 2.6.21 gets
If the bug is a 2.6.21 regression, I place "linux-2.6.21-regression" in the
status whiteboard. This is used for one of the saved searches I linked to
earlier. If the bug is a regression but the exact version which introduced
the bug is unknown (e.g. 2.6.16 worked, 2.6.20 is broken), I place
"linux-2.6.??-regression" in the whiteboard. For regressions I also prefix the
summary field with (e.g.) [2.6.21 regression], so that things are
immediately viewable from the search results.
If the bug has been reported upstream, I place the upstream bug URL in the
URL field, and "watch-linux-bugzilla" in the status whiteboard field. This is
used by one of the saved searches.
When a bug has been solved but is not fixed in portage (i.e. is pending a patch
being added to genpatches, or pending the next genpatches/gentoo-sources
release), we add the InSVN bugzilla keyword.
The bug is then closed with an appropriate message when a new kernel release
(incorporating the fix) is added to the portage tree.
genpatches and linux-stable
- This section is aimed at documenting the process for Gentoo kernel developers only (rather than just prospective ones), but if you're really keen, feel free to ask on IRC if you'd like to submit a patch.
- The patch must conform to the stable kernel rules, see Documentation/stable_kernel_rules.txt
- The patch must have been shipped in at least one gentoo-sources release. The reasoning here is that some of the time, patches committed to genpatches are untested -- full testing is carried out before release.
- The patch must not already be in the stable queue. Check the
stable-queue.git web interface.
Prepare an email in the following way:
- Addressed to firstname.lastname@example.org
- CC email@example.com
- CC the patch author, who's contact details can be found in the patch
- Use the same subject line as the one in the patch, prepended with [PATCH] if it is not there already.
- In the message body, include a very brief description with a reference to the Gentoo bug report, for example: This patch fixes a bootup crash in the snd-intel-hda driver as detailed at http://bugs.gentoo.org/12345
Attach the patch to the email, directly from genpatches. The patch should have come from gitweb or similar and already have the authorship details, commit ID, etc.
What to do now?
Look at some bug reports from the saved searches you configured earlier. If
you can contribute straight off, go right ahead. However, I expect you'll
probably be unsure what should happen next. So, pick a bug, and question us
about it, preferably on IRC in #gentoo-kernel. Alternatively, you could write
to the gentoo-kernel mailing list.
It may seem that many of the open bugs are already in good hands, waiting on
the user, or something like that. However, please stick around, because we
get a steady flow of new bugs!
While this document has introduced you to the processes, you may not feel
capable of jumping into problems and solving bugs. That's fine! Because you
have put firstname.lastname@example.org on your bugzilla watchlist, you will see bugs as
they come in and the way that we fix them. Observe how we handle and
solve the issues. Sooner or later, you will be able to mimick our activities
on other bugs, and you'll be away!
Thank you for your interest in helping us!
The contents of this document, unless otherwise expressly stated, are licensed under the CC-BY-SA-2.5 license. The Gentoo Name and Logo Usage Guidelines apply.