Cleaning up after rootkits
An article over at Network World opines that experts are divided over rootkit removal. Odd… the experts I’ve encountered all agree that once a box has become compromised, the only way to be sure it’s clean is to perform a complete re-install.
A closer reading of the article reveals that really it’s only some security vendors who claim otherwise… vendors who happen to sell rootkit detection software that has a removal feature. Even then, they don’t claim they can always remove a rootkit.
So despite the lurid headline, there is no debate. When we get called in to look at a compromised box, clients always asks if the malware can be cleaned, instead of reinstalling the entire server. That’s understandable… rebuilding can take a long time. However, the answer is always the same:
- It might be possible to remove everything, but you will never be sure.
- Identifying and removing the malware will take longer than reinstalling
Read Mark Russinovitch’s story regarding the Sony rootkit to get an idea of the work required to identify and removing malware. Of course, that’s on Windows which provides many more nooks and crannies for malware to hide than Unix-like OSes. On Linux, removal is rarely that much of a horror but the bottom line still stands: it’s impossible to be certain all the malware was removed and that critical OS components didn’t become broken in the process.
What’s a rootkit?
The term “rootkit” has become diluted lately, especially since the popular press started covering the Sony debacle in 2005. Rootkit just means a set of pre-written tools an intruder can use to escalate privileges on a computer and hide the presence of the malicious software they install. An attacker who gains access to a server will use the components of their rootkit to gain administrator privilege and corrupt operating system binaries so that they selectively hide certain running processes and certain files on disk. These are the tedious tasks of owning a system… a rootkit puts everything together and automates much of it.
An anecdote
Back in 1999 a client contacted us about their servers “running slow”. These were Redhat Linux servers that had been connected to the Internet without any firewall or system hardening. On the systems the “ps” command showed no unusual processes, and “ls” showed nothing but the expected binaries. However, each server’s load average was at 4 and network packet counts were increasing rapidly. No running process could account for that much system activity.
We uploaded static compiled versions of the “ps” and “ls” binaries and what do you know… the process table suddenly showed IRC bots, file sharing software and other interesting items. The uncorrupted “ls” command showed many directories and files that didn’t show up before.
It turned out that the servers had been compromised for over a year… almost since the minute they were first powered up. Multiple intruders were using them to attack other Internet servers and to trade files. It was only when the intruders got greedy and overused the servers that the owner noticed something was wrong.
In that particular case, the “rootkit” the attacker used was pretty simple… it replaced many of the standard system binaries like “ls” and “ps” with custom-compiled versions. These versions had a list of the process and file names of the installed malware and omitted them from any output. Comparing file sizes and checksums of the binaries with ones from the original Redhat CD-ROM revealed they had been altered. Redhat also has the “rpm –verify” command that compares files on disk with the original .rpm packages.
A more sophisticated rootkit would have modified the kernel or installed a loadable kernel module to hide the intruder’s presence. Detecting modifications at the kernel level is more difficult than just uploading a few uncorrupted binaries… even good binaries may still show nothing, since the kernel can control what processes and files can be seen at a low level.
Naturally the client asked if the damage could simply be undone without re-installing everything. Once it was made clear that yes, maybe it could but there was no guarantee and it would take nearly as long as reinstalling, they opted for the rebuild.
We left the compromised servers running while offline the replacements were installed and hardened. We also readied a firewall that could be activated instantly. Some attackers get angry when you interfere with their work… denial of service attacks, wiping the servers or worse can result. Depending on how serious their activities are, it’s often better to let them continue until clean systems and defenses are ready. Then you quickly power off the compromised boxes, activate the firewall and other defenses, and bring the new servers online. In this case the attackers didn’t retaliate when the old servers went dark, though they did spend a few days trying to get back in.
In the end the client got three freshly installed servers with the OS and all services configured as much as was possible at that time to resist attack, plus a separate firewall and content filters. They haven’t been compromised since… as far as anyone can tell.
Related posts:
- Windows .NET rootkits are easy
- The most important Windows security tool
- Disabling the NX bit for specific apps
Tags: endpoint security, incident handling, Malware