Into.the.Void.

Yesterday morning, phone rings to notify my of a new sms. Someone could not access his website on some server that I am root/administer.
I tried to ping the server and got 1 reply every 10-15 packets so my initial thought was that the hosting provider had fucked up. I pinged other machines in the “neighborhood”, they replied just fine. So the problem lied in my server. I got console access through IPMI, you know…the ones with the cipher zero bug, and I managed to login. An apache2 process was constantly using 100% of a core and the machine sent gazillion packets towards a certain destination.

Since I wanted to investigate what exactly this process did, I put an iptables entry in my OUTPUT chain to block packets towards that destination. The machine became responsive again, though the apache process still ran at 100%. Since I run my vhosts using apache2 mpm_itk module, I knew through the apache2 PIDs’ username which site had been hacked. I grepped the logs for any POST, but I couldn’t see anything. Unfortunately the logs only go back 2 days (NOT my policy! and a very bad one actually…but anyway).

strace -p PID did not yield anything interesting, just the process trying to create sockets to send packets towards the destination.

socket(PF_NETLINK, SOCK_RAW, 0) = 417
bind(417, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(417, {sa_family=AF_NETLINK, pid=11398, groups=00000000}, [12]) = 0
sendto(417, "\24\0\0\0\26\0\1\3\233\323\354Q\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
recvmsg(417, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"0\0\0\0\24\0\2\0\233\323\354Q\206,\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 588
recvmsg(417, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0\233\323\354Q\206,\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 128
recvmsg(417, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0\233\323\354Q\206,\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
close(417) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 417
fcntl(417, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(417, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(417, {sa_family=AF_INET, sin_port=htons(4883), sin_addr=inet_addr("X.Y.Z.W")}, 16) = 0
fcntl(417, F_SETFL, O_RDWR) = 0
sendto(417, "\207\25\312P\322t\0#\317}jf\2(W\374\375\232h\213\220\31\355\277)\320[\255\273\276\221\374"..., 8192, MSG_DONTWAIT, NULL, 0) = -1 EPERM (Operation not permitted)
close(417) = 0

lsof -n -p PID output had hundreds of open log files and a few connections. Grepping out the logs I noticed one that was quite interesting, it went towards another server at port 5555.
apache2 11398 XXXXXXX 416u IPv4 831501972 0t0 TCP A.B.C.D:59210->B.C.D.E:5555 (CLOSE_WAIT)

I run tcpdump there, and of course it was an irc connection. I started capturing everything.

lsof also revealed this:
apache2 11398 XXXXXXX cwd DIR 8,7 4096 2474373 /var/www/vhosts/XXXXXXX/httpdocs/libraries/phpgacl
which I could have have also seen it doing ls /proc/PID/cwd/ …but anyway.

Looking inside that dir I found a file named gacl_db.php. It was base64 encoded. Well actually it was multiple times base64 encoded and obfsuscated by using character substitutions, so I had to de-obfuscate it. It was quite easy using php and some bash scripting.

This is the original base64 encoded/obfuscated file: Original gacl_db.php
This is the final result: Deobfuscated gacl_db.php
(I have removed the irc server details from the deobfuscated file, it’s still there in the original file for whoever wants it though)

It’s just an IRC bot containing a perl reverse shell as well. It has commands to flood other servers, and that’s what my server was doing.

I joined the IRC server and at that time there were more than 90 bots inside. Right now that I’m writing this blog post there are less than 50. Every bot joining the channel outputs a text like this:

[uname!]: FreeBSD a.b.c.d 8.1-RELEASE-p5 FreeBSD 8.1-RELEASE-p5 #10: Fri Sep 30 14:45:56 MSK 2011 root@a.b.c.d:/path/to/to/to/sth pl#27 amd64 (safe: off)
[vuln!]: http://www.a-vhost-name.TLD/libraries/phpgacl/gacl_db.php
[uname!]: Linux x.y.z.w 3.2.0-43-generic #68-Ubuntu SMP Wed May 15 03:33:33 UTC 2013 x86_64 (safe: off)
[vuln!]: http://www.another-vhost-name.another-TLD/libraries/phpgacl/gacl_db.php

So if you run servers or websites, do a locate gacl_db.php.

Since all the bot/servers entering post a [vuln!] message about phpgacl, my guess is that the original vulnerability that allowed the attacker to gain access is right there. I haven’t had time to look into it yet, but I’ve warned my clients to remove this library from their websites as a precaution. You should probably do the same.

Filed by kargig at 15:04 under Internet,Linux,Networking
Tags: apache, apache2, DDoS, debian, flood, hack, hacked, Linux, php
6 Comments | 16,800 views

A friend’s site was recently hit by the massive infections/hacks on Dreamhost‘s servers, so I decided to do some scanning on some servers that I administrate for base64_decode references.

The simple command I used to find suspect files was:
# find . -name \*.php -exec grep -l "eval(base64_decode" {} \;

The results could be sorted in just 2 categories. Malware and stupidity. There was no base64_decode reference that did something useful in any possible way.

The best malware I found was a slightly modified version of the c99 php shell on a hacked joomla installation (the site has been hacked multiple times but the client insists on just re-installing the same joomla installation over and over and always wonders how the hell do they find him and hack him…oh well). c99 is impressive though…excellent work. I won’t post the c99 shell here…google it, you can even find infected sites running it and you can “play” with them if you like…

And now comes the good part, stupidity.
My favorite php code containing a base64_decode reference that I found:

$hash  = 'aW5jbHVkZSgnLi4vLi';
$hash .= '4vaW5jX2NvbmYvY29u';
$hash .= 'Zi5pbmMucGhwJyk7aW';
$hash .= '5jbHVkZSgnLi4vLi4v';
$hash .= 'aW5jX2xpYi9kZWZhdW';
$hash .= 'x0LmluYy5waHAnKTtl';
$hash .= 'Y2hvICRwaHB3Y21zWy';
$hash .= 'd2ZXJzaW9uJ107';
eval(base64_decode($hash));

Let’s see what this little diamond does:

% base64 -d 
aW5jbHVkZSgnLi4vLi4vaW5jX2NvbmYvY29uZi5pbmMucGhwJyk7aW5jbHVkZSgnLi4vLi4vaW5jX2xpYi9kZWZhdWx0LmluYy5waHAnKTtlY2hvICRwaHB3Y21zWyd2ZXJzaW9uJ107
include('../../inc_conf/conf.inc.php');include('../../inc_lib/default.inc.php');echo $phpwcms['version'];

So this guy used a series of strings which all of them together create a base64 encoded string in order to prevent someone from changing the version tag of his software. That’s not software, that’s crapware. Hiding the code where the version string appears ? That’s how you protect your software ? COME OOOOON….

Filed by kargig at 01:36 under Internet,Linux,Privacy
Tags: base64_decode, crapware, hack, joomla, Linux, malware, php
No Comments | 7,990 views

We have been talking with Patroklos (argp of census-labs.com) about going to a CCC event for years. This year though we were determined. So on late September 2009 we booked our flight tickets to Berlin. A couple of weeks later some other friends expressed their wish to come with us. So in the end me, Patroklos, huku and SolidSNK (of grhack.net) and Christine formed up a group to visit 26c3 Here Be Dragons. Another group of Greeks also came to 26c3, among them Ithilgore, xorl, sin , gorlist and one more that I have no idea who he was, sorry 🙂

After a canceled flight on the 26th of December due to fog on SKG airport we finally flew on the 27th and went to Berlin. After arriving there we immediately went to the hotel we had booked and then straight to the Berliner Congress Center where the 26c3 was taking place.

BCC is an excellent conference center, nothing close to anything I have ever seen in Greece. It looks great both from the outside and from the inside. When we entered BCC we saw a huge number of diverse people. You could see and feel the difference with all the other IT conferences. People were very relaxed, very talkative and extremely friendly. What makes CCC so special is it’s community. There were soooo many CCC volunteers inside the BCC willing to help you with any information you might need. More on that later on…

After paying just 80€ for the whole conference, 4 days, we started walking around the ground floor. There were many information desks of various projects, free PCs to use (loaded with Ubuntu), the huge lounge which included a bar for food and drinks with lots of seats for people and 2 rooms for presentations. On the upper floor there were many more projects and another large room for presentations.

What made BCC so lively were all these projects around the presentation rooms. There were always hundreds of people sitting outside of the presentation rooms hacking on their projects, discussing with other people, selling merchandise, etc. Because it was our first time in the conference we were not experienced enough to use our time wisely between the lectures so I only managed to visit very few projects, Cacert, Gentoo and Debian. I am sure that there were people who did not attend any lectures at all and just sat all day at their projects’ infodesk.

Before I continue with the presentations we went to I want to make a note about volunteers again. Volunteers at 26c3 were called angels and they did an EXCELLENT job. They would not allow you to sit wherever you liked at a lecture, they would try to find you a seat or they would put you on a place where you could stand without blocking others. Nobody was allowed to sit at the corridors, nobody. Everything was in order and I never ever heard a single person complain about angels’ policy. They were strict and firm on one hand but helpful, fair and polite on the other. They were probably the best volunteers I have ever faced anywhere. All of them were carrying an ID and a DECT phone on them to cooperate with other angels (oh yes, the conference had it’s own DECT network…AND it’s own GSM network!!!) Funny quote: Angels at the entrance and exit doors wore t-shirts that wrote “Physical ACL”, heh.

The very first presentation we attended was “Here Be Electric Dragons“, and then we moved to see “Exposing Crypto bugs through reverse engineering“. After a break we tried to go to the “GSM: SRSLY?” lecture but it was SOO full that we were not allowed to go inside the presentation room. So we went to the “Tor and censorship: lessons learned” presentation which was more interesting than I expected. The final talks we saw on the first day were: “UNBILD – Pictures and Non-Pictures” which was in German and of course “cat /proc/sys/net/ipv4/fuckups“. Since none of us spoke German there was no urge to see the UNBILD lecture, but as we painfully understood by not being able to even enter the presentation room for the “GSM: SRSLY?” lecture, you have to go a LOT earlier to see a good lecture. We definetely wanted to see fabs lecture so we went there an hour earlier to find some seats. By the way, outside of the presentation rooms were TVs with live streaming from inside for people who couldn’t go inside or for people who didn’t want to. As I said earlier a lot of people preferred sitting at their projects’ infodesk and watched the streams of the presentations.

On the next day we saw: “Milkymist“, “Advanced microcontroller programming“, “Fuzzing the Phone in your Phone“, “Defending the Poor, Preventing Flash exploits“, “Haste ma’n netblock?” and “SCCP hacking, attacking the SS7 & SIGTRAN applications one step further and mapping the phone system“.

On the third day just “Playing with the GSM RF Interface“, “Using OpenBSC for fuzzing of GSM handsets” and “Black Ops Of PKI” since we decided to do some sightseeing as well 🙂

Finally on the last day we went to “secuBT” and from that to another German lecture about a distributed portscanner called Wolpertinger that replaced a canceled lecture on IBM AS/400. Afterwards we went to the realtime English translation stream of “Security Nightmares” and to the “Closing Event“.

I had a really great time and I certainly want to be there again next year. If I manage to go there again though I will try take a lot more days off work so I can visit many more places around the city. The whole event was excellent, the organization was almost perfect and the people who contributed to it deserve a huge applaud, especially the angels.

Congratulations to all.

Necessary pics:

P.S. I don’t want to go into specific details about the lectures I attended. Some were REALLY good, some were average and some were totally boring. If you follow the news you already know which streams of lectures you should certainly download and see. You can find every lecture on CCC’s FTP server.

P.S.2 What a great wiki for an event…I was amazed by the amount of information one can find in there…

P.S.3 To Greeks only…please download the closing event presentation to see how we should start organizing events. Just check on the efforts of the people who contributed to the 26c3 event. I don’t want to write anything more about this issue because the difference with any Greek event I’ve ever attended to, or even the mentality of the people attending “our” events is SO SO SO HUUUUGE that it makes me really sad. I hope that this might fire up something. If more Greeks attended events organized abroad then maybe one day we might get more serious about our events as well.

Filed by kargig at 15:41 under Internet,Linux,Networking,Privacy
Tags: 26c3, bcc, Berlin, cacert, ccc, conference, debian, dragons, Gentoo, germany, hack, here be dragons, ubuntu
12 Comments | 12,223 views