Gentoo Linux Kernel Maintainers GuideThis guide aims to document the processes used in the maintenance of Gentoo's core kernel package, gentoo-sources. It is written in hope that it will help future contributors get involved with these maintenance efforts. At the core of gentoo-sources is genpatches: a generic kernel patchset, designed to be minimal, Gentoo-independent, and accessible to other distributions and projects. Please ensure you have a good understanding of genpatches before continuing: read through the genpatches homepage. This document was written by Daniel Drake, the kernel project lead. References to "me" in this document refer to Daniel, but please do not feel that this is a one-man project. Other members of the team are happy to help you as well. Please subscribe to the gentoo-kernel mailing list. Subscription instructions can be found here, and archives can be found here. This list is low-traffic. I will announce modifications to this document on that list. You should also subscribe to the linux-kernel mailing list (LKML). This list is extremely high traffic. You shouldn't even attempt to read it all, but glance through the subject lines every day or two, and keep an eye out for topics that interest you. It's also interesting to read the release announcements from Linus and the -stable crew, and Andrew Morton's -mm release announcements to keep up with what's going on in the kernel community. Keep the LKML archived in its own folder in a good mail client. It becomes very useful later on when you are researching bugs reported by users on the Gentoo bugzilla. I try to keep at least the most recent 30,000 emails around. To subscribe to the LKML, send an email to majordomo@vger.kernel.org with the following in the body of the email:
You may also be interested in reading the LKML FAQ. It contains lots of information about how the mailing list is used and the kernel community in general. I don't expect you to have any need to write to the LKML initially, but do be sure to read this FAQ before you send your first email there. Please be active on our IRC channel, #gentoo-kernel on freenode. The Gentoo website has more information on IRC channels. While learning, please ask us questions! This is the best way to learn. If we don't respond, ask again another time. We do want to help you, since hopefully that will result in you helping us in the future! The primary kernel maintainers are Daniel Drake (IRC username dsd_) and Mike Pagano (IRC username mpagano). The bulk of Gentoo kernel maintenance happens on bugzilla - bug reports, keyword requests, feature requests, etc. Make sure you have your own personal account. You should now configure bugzilla to send you copies of all bugmails related to kernel maintenance. To do this, go to bugzilla email preferences and add kernel@gentoo.org to your user watch list. I suggest you set up a mail filter to move these mails to their own folder. Another useful feature is bugzilla saved searches. You can perform a bugzilla query, and save it with a user-defined name for quick access later. Links to these saved searches are present on the bottom of every bugzilla screen. For each of the following queries, please scroll to the bottom of the resultant buglist and use the "Remember search" functionality to save each one: Gentoo kernel bugs: This query lists all open bugs assigned to kernel@gentoo.org except ebuild submissions, tracker bugs, and 2.4 bugs. This is the main buglist you will be working from. Kernel regressions: This query lists all open regression bugs. A regression is a bug which was introduced in a certain kernel release, whereas the same functionality worked in releases beforehand. In many ways, regressions are the most critical form of bugs. Linux bugzilla: This query lists all bugs reported on the Gentoo bugzilla which have then been reported upstream at the kernel bugzilla. We report the majority of bugs upstream to the kernel bugzilla, http://bugzilla.kernel.org. You should create an account here, and again, put kernel@gentoo.org on your email user watch list. You'll probably find it handy to keep a subversion checkout of genpatches locally.
Here's a list of useful websites which you'll see me referring to.
The kernel is a huge codebase, and is one of the packages which is present on almost every users system and is used all the time. Our maintenance style may seem strange at first, but please remember that kernel maintenance is overall a huge task, and as such, one of our primary objectives is minimizing the amount of work that we have to do (and overall make things more manageable). One way we reduce our workload is to attempt to deviate away from the upstream kernel releases as little as possible: we aim to make genpatches as small as possible. The less patches we have to handle/backport/forwardport, the less work we have on our hands. Another approach we take is quick adoption of new kernel releases. Each new kernel release fixes many bugs and adds more hardware support, which invariantly affects proportions of our userbase. Instead of going to the pains of backporting these code changes, we reduce our workload by encouraging quick adoption of new kernel releases where this new code originates. Further in line with these goals, we tend to stop supporting older kernels fairly soon after a new release has been cut. Even more time consuming than patch handling is bug fixing. Being such a huge codebase, we get a steady flow of kernel bug reports on our bugzilla. Some of them take considerable time/effort to get resolved. We work to streamline our bug handling processes here. Even after 3 years of bug handling, I personally am familiar with only a very small proportion of the kernel codebase. It's important that we raise these bugs to the attention of the upstream kernel developers who do know what they are doing, and it's important that we do this quickly. As you will learn, we don't actually do much of the fixing downstream in Gentoo. With respect to the above forwarding of bugs to upstream developers, it's important that we maintain good relationships with upstream developers. In general, upstream developers are wary of distro-reported kernel bugs - after all, most distributions patch their kernel releases so heavily that they aren't easily supportable by people outside of those distributions. I recently had to dig a patch out of the Mandriva patchset, and just out of curiousity, I ran diffstat against their patches. The data set was so great that it revealed an obscure bug in diffstat itself! One way we attempt to get respect from upstream developers is by our patching policy: other than in exceptional circumstances, we don't add any patches until they have been merged into the official development tree upstream. Additionally, this helps us to respect upstream decisions, and helps us ensure quality (and upstream support) of our patchset. The bugfixing process certainly varies from bug to bug, but many have things in common. During the process I usually aim to do the following:
With that in mind, bugs often follow a process such as the following:
A user reports a bug to the Gentoo bugzilla. A bug wrangler reassigns the bug to kernel@gentoo.org. You are notified of this new bug by email. You first run a few sanity checks over it: is it an obvious user error? Is it a duplicate of another bug that has already been reported? The initial bug report is usually lacking in information, by no fault of the reporter. Examples of common questions you might ask in response to the intial report include:
There are typically many other such questions that could be asked for most bug reports, but many of these can be answered by requesting output of certain commands or certain files. I find myself commonly requesting the following:
After requesting one piece of information, you might then realise that you need to request something else. That's fine, just post a new comment with a new request. We generally accept bug reports for 2 kernel versions at any one time:
For example, at the time of writing, the most recent stable kernel is 2.6.20. 2.6.21 has just been released, and is present in the testing tree, but it needs substantially more testing time before we are confident enough to mark it stable. During this time we will accept bugs for the latest gentoo-sources-2.6.20 release, and the latest gentoo-sources-2.6.21 release. Even if a user files a bug on a currently supported kernel, we often ask them to test the latest development kernel (e.g. 2.6.22-rc1 in the above example). Why? We're moving to get this bug reported upstream as soon as possible, and as a kernel developer, there simply isn't any point diagnosing issues on a codebase which isn't current. With that in mind, and the above example of current versions, there are several situations which can arise:
For 2 weeks after the upstream release of a 'production' kernel, there is a period in which there is no development kernel available. For example, 2.6.21 has only just come out and 2.6.22-rc1 will not be released for at least a week. In this case, you can skip the stage where you ask the user to test the latest development kernel -- reproducing the issue on 2.6.21 is enough to request the upstream developers to work on it. When you're happy that you have enough information about the problem, you should attempt to find other reports of the same issue. In some cases you can do this without asking the user to test all the new kernels. It is quite often the case that the bug has already been reported upstream and there is already some discussion happening about how to fix it, or maybe a patch is already available. This part is really just common sense: stick some search terms into google. If you have a distinguishable error message, search for that. Otherwise, search for "linux 2.6.20 sky2" if you're researching a reliability regression in 2.6.20 with the sky2 driver. Additionally, query the linux bugzilla with similar search terms, and use your email client to search through your local LKML archives. If you find something, that's great, reference it on the bug. You may even have found a fix for the problem. It's certainly not uncommon for standard searching not to find any references though. Occasionally we get bug reports from people using vanilla-sources. Whether or not we accept these bugs is a matter of judgement. vanilla-sources is provided to aid testing, but is not supported: if it's broken, we don't patch it. That said, if someone reports a bug in vanilla-sources-2.6.21.1 (the latest upstream release) it is likely to also exist in the latest gentoo-sources-2.6.21 release, so it probably makes sense to accept the bug and go through the usual diagnosis. However, if you feel that gentoo-sources may have a patch to fix this, don't hesitate to remind them that vanilla-sources is unsupported and they need to reproduce this on gentoo-sources before we'll look into it. Sometimes, users report -rc kernel regressions, e.g. they report bugs which exist in vanilla-sources-2.6.22-rc1 but not in 2.6.21. In these cases, you can close the bug right away: you should thank the user for reporting the issue but point out that they should report it upstream at the kernel bugzilla directly. gentoo-sources does not do -rc releases. Binary and out-of-kernel modules It may become apparent through information gathering that the user is using kernel modules/drivers which are not part of the official kernel sources. One example of such a package would be x11-drivers/nvidia-drivers. Again, these situations require some judgement. There's simply no way we can support kernels which are modified at runtime in this way, so it is perfectly reasonable to ask the user to reproduce the issue without those external modules loaded (and also request that they have not been loaded since boot, as opposed to them booting, loading and unloading the modules, and then reproducing the issues). That said, you can sometimes conclude that the bug exists through other means, in which case there is no need to ask them to reproduce without the external modules. For example, you might find an identical report on a non-tainted kernel on LKML. If you determine that a bug is a regression, there are a few extra things to consider. First, you should tag it as so (read about the Status Whiteboard later in this document). Secondly, you should note that the Linux source code management utility (git) has a useful bisection feature which can identify the exact commit which caused a regression. This really makes the bug a lot easier to solve. However, performing a bisection is a time consuming process: for identifying a regression between versions 2.6.x and 2.6.x+1 it usually requires that you build and test approximately 13 different kernels. So, unless you sense that the user is particularly keen, you should probably exhaust other options first. You could get the bug forwarded upstream and after a few days if there hasn't been any progress, ask them if they will consider doing a bisection. I have some pre-written instructions on how to do a bisection, which are easy to follow. If you are going to suggest a bisection, I would suggest you just drop them a link to this article. It's unfortunate, but fairly often, users turn unresponsive and don't provide the extra information that you requested. While we aren't paid to fix these issues and have plenty more to be working on, it's best just to discard these bugs. After 14 days, if the user has not responded to my most recent request, I close the bug as NEEDINFO with a comment asking that they reopen the bug when the information has been provided. Quite often this spurs the user into providing what was requested. You've established that this appears to be a kernel bug, you've gathered information which agrees, and it's been reproduced on the latest development kernel. The next step is to get the bug reported upstream. There are 2 ways you can do this: by asking the user to report the bug at the upstream kernel bugzilla, and by yourself sending an email to the relevent lists and maintainers. It's very important that we make it easy for the upstream developers here. When reporting bugs, state clearly which versions are affected, mentioning which ones are unmodified (usually we can say "it's present in 2.6.19-gentoo but is also reproducible on unmodified 2.6.20-rc5"). While the procedure does involve linking to the downstream bug, make sure all information is presented upstream - don't expect them to click and read through the downstream bug, which probably contains some degree of noise anyway. The procedure outlined below includes modifying the status whiteboard field, which is explained further later in this document. Reporting bugs on the kernel bugzilla The kernel bugzilla is the primary mechanism we use to report bugs upstream. Actually, the way we do it is by getting the user to file the bug report, and then us tracking it. Here is the process I follow:
If the user doesn't respond to the request to file the bug upstream, discard the bug like you would for other unresponsive cases. Since you closed it as UPSTREAM, it will drop off the buglist anyway -- so do remember to continue the process when they do respond with the upstream bug URL. The linux-bugzilla-pending keyword allows us to measure how often bugs get lost at this point, and allows us to follow up. When the bug gets updated upstream, kernel@gentoo.org will get the bugmail, and since you have that address on your watch list, you'll see it too. When the bug gets fixed, reopen then downstream bug with a quick summary of the solution, removing the "watch-linux-bugzilla" tag. If someone closes the upstream bug without a fix (for example, the user didn't respond to an information request, or the bug got marked as invalid), reflect the changes downstream: mark the equivalent Gentoo bug as CLOSED and remove the "watch-linux-bugzilla" tag. Sending email bug reports upstream To be written later. This is basically used as a fallback if the upstream bug isn't getting the attention it needs. This document includes references to manipulating bugs on the Gentoo bugzilla (closing/reopening/reassigning/etc). Unless you're a Gentoo developer or you're the bug reporter, you don't actually have access to do this though. If a certain action should be taken on a bug which you don't have permission for, you should ask in #gentoo-kernel on IRC for someone to perform the operation. If there is no response, leave a simple comment on the bug requesting the change, e.g. "please close this bug as NEEDINFO" I realise this is a pain, but it's only a temporary one. Once you've made several contributions to bugs in this form, I will increase your bugzilla access permissions so that you can perform these tasks on your own. The other aspect of this problem is manipulating bugs on the upstream kernel bugzilla. This needs to be done much less than bugs downstream. I personally have developer access to the upstream bugzilla, so if anything needs doing (bugs closing/reassigning/etc), prod me on IRC. In future we could look into getting more people developer access here. Bugzilla has a single-line status whiteboard text field for each bug, which you are free to put anything in. I use it for certain kinds of tagging. If a bug is present in 2.6.20, but fixed in 2.6.21 and the fix is not yet known, I place "linux-2.6.21" in the status whiteboard field. This is effectively tagging the bug saying "this problem goes away when 2.6.21 gets released". If the bug is a 2.6.21 regression, I place "linux-2.6.21-regression" in the status whiteboard. This is used for one of the saved searches I linked to earlier. If the bug is a regression but the exact version which introduced the bug is unknown (e.g. 2.6.16 worked, 2.6.20 is broken), I place "linux-2.6.??-regression" in the whiteboard. For regressions I also prefix the summary field with (e.g.) [2.6.21 regression], so that things are immediately viewable from the search results. If the bug has been reported upstream, I place the upstream bug URL in the URL field, and "watch-linux-bugzilla" in the status whiteboard field. This is used by one of the saved searches. When a bug has been solved but is not fixed in portage (i.e. is pending a patch being added to genpatches, or pending the next genpatches/gentoo-sources release), we add the InSVN bugzilla keyword. The bug is then closed with an appropriate message when a new kernel release (incorporating the fix) is added to the portage tree. 7. genpatches and linux-stable
Prepare an email in the following way:
Look at some bug reports from the saved searches you configured earlier. If you can contribute straight off, go right ahead. However, I expect you'll probably be unsure what should happen next. So, pick a bug, and question us about it, preferably on IRC in #gentoo-kernel. Alternatively, you could write to the gentoo-kernel mailing list. It may seem that many of the open bugs are already in good hands, waiting on the user, or something like that. However, please stick around, because we get a steady flow of new bugs! While this document has introduced you to the processes, you may not feel capable of jumping into problems and solving bugs. That's fine! Because you have put kernel@gentoo.org on your bugzilla watchlist, you will see bugs as they come in and the way that we fix them. Observe how we handle and solve the issues. Sooner or later, you will be able to mimick our activities on other bugs, and you'll be away! Thank you for your interest in helping us! The contents of this document are licensed under the Creative Commons - Attribution / Share Alike license. |
|
