Into.the.Void.

This post was originally written on 09/11/2012 but for various reasons it couldn’t be published earlier. It won’t be very technical, it’s mostly a behind the scenes view of how some of us at NOC GRNET tried to cope with an extreme spike of demands of a specific service we support.

Intro
At GRNET we provide a live streaming service to the Hellenic Parliament, who also connect to the Internet through our network. What we’re doing is that we’re serving the official TV channel of the Hellenic Parliament (the WebTV program actually). We have a machine that encodes (transcodes actually) video/audio and a separate streamer that serves the content to the clients via RTSP, RTMP and HTTP. While this service has been running for quite some time, and the streaming server has been used in various other occasions, it typically serves no more than 100 concurrent clients. With a stream at 640Kbps on average that’s abound 60Mbps of streaming traffic. The number of viewers usually goes up only when there’s something exciting going on. Our previous traffic spike was at 577Mbps on October 31st, when the Minister of Finance was presenting the budget for 2013 to the Parliament. What happened that day was that news reporters were on strike and it seems that one of the few news sources available at the time for people to use was the Parliament channel, either on the TV or through our stream. The viewership that day surpassed all previous expectations but the streaming system actually performed quite well considering we didn’t have any complaints.

Remember, remember the 7th of November
On the 7th of November almost everyone was on strike in Greece. There was a huge march arranged at 17:00 outside of the Greek Parliament to protest against the new harsh measures of Memorandum III that the parliament was to vote for at midnight.

We started seeing some traffic arriving, well leaving is more accurate, our streaming server at around 10:00 in the morning. Some news sites and blogs (tanea.gr, left.gr, protothema.gr) had already started posting links to our parliament’s streaming service for people to watch the discussion that was already taking place. At around 10:30 we were already serving more than 100Mbps while the traffic was steadily rising. A bit after 12:00 though things got really nasty. Websites like tovima.gr, zougla.gr and newsit.gr had posted our streaming links in their very first page. The discussions both inside the parliament and on the social networks had also started heating up. At 12:25 the traffic starting ramping up extremely fast!

12:25 160Mbps
12:32 367Mbps
12:38 597Mbps
12:42 756Mbps
12:49 779Mbps

That’s when traffic stopped growing any more. We actually saw some drop in the traffic and then up to 780Mbps and down again. It was obvious that we had reached some limit. It wasn’t very obvious though what the bottleneck was since the streaming system load was quite low, around 0.6-1.0 in a 4 vCPU VM and it also had a virtio-net/gigabit[1] network card. Theoretically it should be able to push at least 100-150 more Mbps. In order to cope with the increasing demand we changed the player web page in order to off-load some users to an experimental service that relies on IP multicast from the server and P2P between the clients for delivering streams. This bought us some time, about 2-3 hours. Also maybe because it was noon time, or due to more people using the experimental service or not being satisfied from the performance of the streaming video and disconnecting, traffic later on dropped “down” to an average of 650-700Mbits.

We started discussing short-term actions for improving the service so as to be able to cope with more demand. The streaming server was running on GRNET’s virtualization platform, which is based on ganeti, and the first thought we had was to try and add more network cards to the VM and somehow bond the cards together. Could we push more than 1Gbps from the same VM? The problem was that we couldn’t do any 802.3ad bonding since the “virtual switch” inside the virtualization platform did not support such features. One other solution would be to add another virtual network card and use Linux bonding mode 5 (balance-tlb) or 6 (balance-alb). After a bit of reading this mode was rejected as well. These two bonding modes expect from the ethernet card running on the machine to support ethtool to read their speed. Our VMs use virtio-net which doesn’t support this functionality for ethtool. We could switch to another type of network card, e1000 for example, but this could have an unmeasured/untested performance penalty which could actually negate the addition of a second network card.

Hundreds of thousands of Greeks were planning to join this huge protest starting at 17:00 and so were some of us working at GRNET, at least I was. We had to make a decision though, drop our plans and improve the streaming service to help more people, especially Greeks living abroad, to access the stream or just join others at Syntagma square? Our shift was ending soon anyway. We decided to stay at work and try and improve the service as far as we could.

The path we chose in order to “scale” was to create to a second instance of the streamer and try to “balance” client requests to both streamers using round robin DNS. That involved creating a checkpoint of the running VM at our storage system, copy the running VM’s image from that checkpoint as a new image and run a new VM with that image. So we would be cloning the current streaming service while the service was running and serving clients (take that Windows!).

So we provisioned a new Debian server through LDAP, added the copied image disks at the VM’s configuration, booted the VM, run puppet to change the configuration files according to the new hostname and IP, and we were ready to serve more clients. After some testing we created a RR DNS entry, “streamer-frontend.domain.gr” which pointed to both first-streamer.domain.gr and second-streamer.domain.gr and had a TTL of 60. Changing the landing page though to serve streamer-frontend.domain.gr as the streamer url wasn’t enough. The first problem was that there were news sites and blogs that had copied directly our first streamer’s URL instead of the live.grnet.gr/paliament/ landing page, which actually runs on a different system and is served by an Apache2 server (more on that later). Having already more than 1000 streams on the first streamer and then starting to do RR DNS on both servers meant that there would still be a large number of clients served by the first server and new people getting directed there would only make matters worse, while people directed to the second server would have a much better service experience. So what we actually did was point streamer-frontend.domain.gr not on both first-streamer.domain.gr and second-streamer.domain.gr but just second-streamer.domain.gr. We also changed the IP address of first-streamer A,AAAA records to second-streamer’s IP and flushed the first-streamer RR at our caching resolvers. Since many people use our caching resolvers, especially students with DSL lines, we were able to direct even more people towards the second-streamer.

Before adding second-streamer.domain.gr:

* first-streamer.domain.gr -> 1.2.3.4 TTL 86400
* live.grnet.gr/parliament/ pointing the streamer URL at first-streamer.domain.gr

After adding second-streamer.domain.gr:

* streamer-frontend.domain.gr -> 1.2.3.5 TTL 60
* first-streamer.domain.gr -> 1.2.3.5 TTL 60
* second-streamer.domain.gr -> 1.2.3.5 TTL 60
* live.grnet.gr/parliament/ pointing the streamer URL at streamer-frontend.domain.gr

That way people disconnecting from the stream and reconnecting some time later on were pointed to the new server, if they used our landing page. That meant better quality streaming for both “old” clients getting their stream from first-streamer, and for the new clients that were being served from second-streamer.

Within 30 minutes since booting up, the second streamer was serving more than 250Mbps and after another 30′ its traffic had climbed up to 450Mbps while first-streamer was steadily serving more than 550Mbits. That lead us to an all time record of 1.02Gbps at 17:45. At that point we were serving more than 2000 concurrent streams. A number we never expected for this humble streaming service.

While traffic was increasing through the addition of the second streamer, another problem came up. Some misconfigured clients and HTTP proxies were opening up connections at the Apache 2 serving the web page and never closed them down. We hit Apache’s internal ServerLimit variable of 256 around 17:00. We had to increase ServerLimit inside Apache’s config and restart it:

ServerLimit 500
<IfModule mpm_prefork_module>
    StartServers         10 
    MinSpareServers      15 
    MaxSpareServers      25 
    MaxClients          500
    MaxRequestsPerChild   0  
</IfModule>

After some minutes of happiness another “unexpected” issue came up. It started to rain in Athens, so people who were at the protest at Syntagma square would probably leave soon and go home. There would certainly be an increase in traffic. And what about midnight, which is when the MPs were supposed to vote on the new measures? People would be sitting at their computers, bashing the politicians on Twitter and Facebook while watching the Parliament’s stream. At the same time we started seeing a big increase in international traffic, since more Greeks living abroad were tuning in; this stream was probably the only way they could watch what was happening at the Parliament.

That only meant one thing. Hello third-streamer! 10′ later we had another streamer running. But we also wanted to make some changes to the first two VMs. We wanted to add more vCPUs, going from 4 to 8, add more RAM, going from 4096Mb to 6144Mb, and do some minor performance tuning on the software. That meant restarting the VMs in order to activate the additional resources… luckily the chairman of the Parliament announced a 10 minute recess some time around 19:30. That was our chance… we changed back first-streamer’s IP address, added it to streamer-frontend RR DNS while also adding third-streamer to it. Then we rebooted the first two VMs. After 30″ we had 3 streamers running and waiting for clients to join them. third-streamer instantly got more than 200Mbps at the very moment we rebooted the other 2 VMs.

After adding third-streamer.domain.gr:

* streamer-frontend.domain.gr -> {1.2.3.4 | 1.2.3.5 | 1.2.3.6} TTL 60
* first-streamer.domain.gr -> 1.2.3.4 TTL 60
* second-streamer.domain.gr -> 1.2.3.5 TTL 60
* third-streamer.domain.gr -> 1.2.3.6 TTL 60
* live.grnet.gr/parliament/ pointing the streamer URL at streamer-frontend.domain.gr

At 20:10, some of us decided to leave work after 11 hours, having 3 streamers running that had already reached 1.2Gbps of traffic. There were more than 2800 concurrent users at the time…

Our assumptions for midnight came true. Around that time, more than 4000 people had tuned in to the parliament’s stream and we were pushing about 1.66 Gbps. That was probably the all-time record for a single service in GRNET.

A big problem we didn’t solve that night
Unfortunately, each client gets a unique session id from the streamer, also when requesting an HTTP stream. These are not shared among the streamers, so if you send a subsequent HTTP request to a different streamer, who does not track that session id, it will not serve the request as expected but rather prompt the client to start a new session. Since some browsers did not cache the RRs of streamer-frontend long enough (remember that low TTL?) they would spread their requests to all the streamers. We knew this was happening but there was no way to solve this without extensive changes to the setup, which would have to be tested of course would mean significant downtime for the service. Therefore, depending on the browser people used, some got better or worse service than others. The good thing is that we know how to fix this in the future.

Some Stats & Graphs
more than 31000 unique IPs connecting to the streaming service
more than 4000 concurrent users at peak time
more than 4.7 TB of data streamed

first-streamer ethernet traffic:

second-streamer ethernet traffic:

third-streamer ethernet traffic:

Streamer client types

Streamer LAN traffic aggregates:

Create your own graphs here: http://mon.grnet.gr/rg/157184/details/

Epilogue
Yeah, there were hiccups in the stream for many many people, we acknowledge that. But we certainly did the best we could to keep the service running. Could we have done better ? Yes we could, but we would have to re-design the service and make some drastic changes that were never tested before. We didn’t want to risk making drastic changes that might not work at all while we could somewhat “scale” our service using a “working” solution. I wonder what will happen on Sunday when the Parliament votes for 2013 budget…Will we exceed our previous peak? We’ve already discussed alternatives to cope with the extra traffic and we’ll definitely be better prepared [2]!

I think that zmousm, alexandros and me did a fairly good job regarding the circumstances. We should buy each other a beer sometime..We’ll treat faidonl another one though for his helpful consulting 🙂

Since no external CDNs or other services provided by companies abroad were used, this might be the event with the biggest ever demand in network resources served from within Greece.

Do you like solving such problems ? Then there might be a lurking sysadmin inside you 😉

[1] After a talk with apoikos, he corrected me saying that virtio-net does not have a “gigabit” capacity. Virtio-net’s capacity is only limited by the node resources available, so it can theoretically perform better than gigabit. So our thoughts on using balance-alb/tlb to cope with the extra bandwidth were wrong. This clue points out that the cause of bottleneck on the streamer that couldn’t go over 800Mbps was the streamer software itself since both VM’s and hardware node’s system load were low.

[2] On 11/11/2012 the Greek Parliament voted for 2013 budget. What we did prior to that day to cope with the traffic was to setup a new version of the streamer software, that actually came out on 08/11/2012 just one day after the initial spike, that had the option to disable session IDs. Then it was quite straightforward to add some varnish caches in front of the streamers to serve the HTTP streams to the clients. Unfortunately client demand was somewhat lower, it only reached 0.92Gbps.

Filed by kargig at 11:15 under Internet,IPv6,Linux,Networking
Tags: debian, ganeti, grnet, Linux, parliament, puppet, scaling, spike, streaming, virtio-net, virtual machine, virtualization
5 Comments | 22,443 views

On Sunday the 11th of November we finally had our first CryptoParty in Athens, Greece. We hosted it at the Athens Hackerspace.

Organizing
We organized our first CryptoParty in a very ad-hoc way. A pad was set up and advertised on Twitter/Facebook. Almost immediately people started writing their thoughts, views and interests there. We soon had a list of topics that people were interested in and another list of people willing to give presentations/workshops. Later on we set up a doodle so people would choose the most convenient dates for them. From the group of 50 people that originally expressed their interest to attend the CryptoParty, at least 20 voted on the doodle. That’s how the final date of November the 11th was chosen.

It was surprising/refreshing that even though everything was organized through an anonymously editable pad, nobody tried to vandalize it.

The actual event
Through the pad, we chose 3 topics for the first meeting. “Using SSL/TLS for your Internet communications”, an “introduction to Tor” and another “introduction to I2P”.
The time for the event was set for 12:00 in the morning, probably a very bad choice. The next one should definitely be later in the afternoon or even night. We learn by our mistakes though…People started showing up at around 11:30, but the event didn’t start until 12:30 when someone from hackerspace.gr gave a 5′ intro talk about what the hackerspace is to people who had never been there before. People kept coming even until 13:00 and the audience had grown to more than 30 people.
After the three workshops/presentations around 10-15 people stayed and we ordered pizza.

All in all I’d say it was fairly successful since more than 30 people came and actually did things to improve their security.

The presentations/workshops
“Using SSL/TLS for your Internet communications” (in English) was my effort to show people how cleartext data travels through the Internet and how any intermediate “bad guy”/LEA can easily read or manipulate your data. People were instructed to install wireshark so they could actually see for themselves what the actual problem is. It was very “nice” to see their surprise upon watching cleartext packets flowing through their network cards. It was even nicer to see their surprise when I used tcpdump on hackerspace’s router to redirect traffic to wireshark running on a Debian laptop to display their data, without having “direct” access to their computer. Then people were introduced to the idea of Transport Layer Security (SSL/TLS), and how HTTPS protects their web data from prying eyes. After this tiny “privacy apocalypse” it was very easy to convince users to install HTTPS-Everywhere. And so they did. Afterwards they got instructions on how they should change SSL/TLS settings for their E-email and IM clients.
My original intention was to “scare” people a bit. It was funny to see their faces when they logged in to yahoo mail and they could see their emails cleartext on wireshark. People don’t understand how data travels through the Internet unless they experience it for themselves. I’m glad that people who had absolutely no idea about HTTPS are now using HTTPS-Everywhere to protect themselves. Hopefully they’ll show that to their friends as well.

“Introduction to Tor” (in Greek) gave people an idea at what anonymity is, how it differs from security and how users should be combining both TLS and Tor usage for security and anonymity at the same time. A brief explanation of what hidden services are was given as well. Even though George asked people to download and install Tor Browser Bundle and use it, we’ll definitely need more “hands on” Tor workshops in the future. It will be interesting to convince more people to actually use it and why not, even set up their own hidden services.

“Invisible Internet Project a.k.a. I2P” (in English) by @alafroiskiotos was probably the hardest of the three presentations to keep up for people that had no previous idea about anonymity networks. It’s unique architecture and some difficulties in it’s usage raised a lot of interesting questions by attendees.

Thoughts on future CryptoParties
After the end of the workshops/presentations we had a lengthy discussion with the attendees as to what they would like to see/experience in the future CryptoParties. Unfortunately people were not very vocal. Very few participated and openly expressed their thoughts/opinions. A great part of the discussion was spent trying to figure out whom should CryptoParty presentations/workshops target at, users? developers? geeks? It’s obviously very hard to target all groups of people at the same time.

So here are my thoughts on what future CryptoParties should be. CryptoParties should be about changing user habits, they should be closer to workshops than presentations. They should be focused mainly on users not developers nor computer science students. Just simple users. People don’t want theoretical talks about cryptography, they need advice they can use in their daily lives. It’s already very hard to talk about modern crypto to people who haven’t got a strong mathematical background, you have to oversimplify things. Oversimplifying things then makes geeks/nerds unhappy and still doesn’t “teach” people about proper crypto. Even a fairly “simple” HTTPS negotiation contains key crypto concepts that are very difficult for a “crypto-newbie” to grasp. So it’s a lose-lose situation.

We need to teach, or better convince, users on using good, secure, audited tools and not just tell them about technologies and concepts. We, weirdos, might like that, but most users don’t. People need our help to learn how to avoid “fancy” tools and false security prophets. We need to show them how security should be applied in a layered approach. Getting people to care about their own privacy is key to the success of CryptoParties in the way I see them. To achieve that, we, people that know a few things more than the average Joe, should all become volunteers to such efforts. We should be joining CryptoParties in order to help others and not in order to improve ourselves and our knowledge. (Actually when you study in order to make a good workshop/presentation you improve your own knowledge as well, but let’s leave that beside for now.) We can have our separate geeky/nerdy events to present fancy tech and cool crypto stuff, but let’s keep CryptoParties simple and practical. Oh and we’ll need to repeat things again and again and again. That’s the only way people might change their habits.

If you want to find out more about the next Athens CryptoParty keep an eye at Hackerspace’s events and the athens cryptoparty pad. Join us!

Good luck to all the CryptoParties worldwide!

Filed by kargig at 21:23 under Encryption,Internet,Linux,Networking,Privacy
Tags: anonymity, athens, CryptoParty, HSTS, https, HTTPS-Everywhere, I2P, Networking, presentation, Privacy, security, SSL/TLS, tcpdump, tor, wireshark, workshop
No Comments | 10,088 views

When you have a misbehaving system and you are not sure what the problem is, always bet on a poorly written application.

Here’s a small example of how another poorly written web application caused system issues.

I was sitting at my office today I when I got this nagios alert for a host.

Date/Time: Tue Nov 6 19:15:11 EET 2012
Additional Info:
SWAP CRITICAL – 0% free (0 MB out of 509 MB)

Logging in actually showed all the swap’s been used and so was RAM, 0.95/1Gb. Lots of apache2 server instances were running. I did a netstat and I saw a lot of ESTABLISHED connections:

tcp6       0      0 2001:DB8:f00::1:35571 2001:DB8:bar::100:80 ESTABLISHED 9631/apache2    
tcp6       0      0 2001:DB8:f00::1:35777 2001:DB8:bar::100:80 ESTABLISHED 9656/apache2    
tcp6       0      0 2001:DB8:f00::1:36531 2001:DB8:bar::100:80 ESTABLISHED 11578/apache2   
tcp6       0      0 2001:DB8:f00::1:36481 2001:DB8:bar::100:80 ESTABLISHED 11158/apache2   
tcp6       0      0 2001:DB8:f00::1:36295 2001:DB8:bar::100:80 ESTABLISHED 11115/apache2   
tcp6       0      0 2001:DB8:f00::1:34831 2001:DB8:bar::100:80 ESTABLISHED 8312/apache2  

2001:DB8:f00::1 -> my server
2001:DB8:bar::100 -> dst server

As one can easily see my server is connecting to port 80 of dst, possibly asking for something over HTTP.

# netstat -antpW | grep 2001:DB8:bar::100 | wc -l
111

# dig -x 2001:DB8:bar::100 +short                                 
crl.randomcertauthority.com.

tailing the log files didn’t show anything weird happening. I run a tcpdump for that dst server but there wasn’t at that time any traffic going on.

So, I took a look at munin to see when this problem started developing.

As it’s obvious from the above graphs, the problem started around 14:00. So I took another look at the apache logs and I saw a bot crawling a specific url from my server. I visited that url on my server using curl and I saw traffic flowing through tcpdump going from my server to dst server. So visiting that URL was definitely causing problems. But why?

I restarted apache, swap and memory were released, all the stale ESTABLISHED connections went away and I saw hundreds of FIN/RST packets going back and forth at tcpdump.

I tried to open a few concurrent connections from my PC to my server’s url using curl. After a couple of tries netstat showed that I had managed to create stale ESTABLISHED connections towards dst server. It was an HTTP connection asking for a crl. So I was both able to reproduce the problem and I also knew the specific url of the dst server that caused the connection hanging issues.
Next thing I did was to try to open direct HTTP connections from my server to the dst url using curl. After a few concurrent connections I managed to make curl hang. So the problem was definitely not on my server, but at the dst server.

Since it was already quite late, my first (re)action was to install mod_evasive to try and minimize the problem so I could take a better look the next day.

# aptitude install libapache2-mod-evasive
# a2enmod mod_evasive
### edit /etc/apache2/sites-enabled/site-name and add the following
       <IfModule mod_evasive20.c>
            DOSHashTableSize    3097
            DOSPageCount        1   
            DOSSiteCount        50  
            DOSPageInterval     1   
            DOSSiteInterval     1   
            DOSBlockingPeriod   10  
            DOSEmailNotify myemail@mydomain.gr
        </IfModule>
# /etc/init.d/apache2 reload

I tried to curl my server’s URL from my PC and I got blocked after the second concurrent try. But after some repetitions I was still able to create one or two stale ESTABLISHED connections from my server to the dst server. Far fewer than before but the problem was still somewhat reproducible.

Then I decided to take a look at the site’s PHP code. Finding the culprit was quite easy, I just had to find the code segment where PHP requested the dst server’s url.
Here’s the code segment:

$ch = curl_init($this->crl_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
$crl_content = curl_exec($ch);

The developer had never thought that the remove server might keep the connection open for whatever reason (rate limiting anyone?)

Patching it was quite simple:

$ch = curl_init($this->crl_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT,'60');
curl_setopt($ch, CURLOPT_TIMEOUT,'60');
$crl_content = curl_exec($ch);

After this everything worked fine again. Connections were getting ESTABLISHED but after 60 seconds they got torn down, automagically. No more stale ESTABLISHED connections. Hooray!

A letter to every developer:

Dear developer,

please test your code before shipping. Pretty please take corner cases into account. We know you’re competent enough, don’t be lazy.

Your kind sysadmin

Filed by kargig at 22:36 under Internet,IPv6,Linux,Networking
Tags: apache2, bug, curl, debian, HTTP, libcurl, Linux, mod_evasive, munin, nagios3, php, tcpdump
1 Comment | 8,325 views

*WARNING* 14/01/2014 This post is quite deprecated. For example obfsproxy has been completely rewritten in python and there is a newer and more secure replacement of obfs2, named obfs3. Please read this obfsproxy-debian-instructions for any updates.

Some countries like China, Iran, Ethiopia, Kazakhstan and others, like installing some nasty little boxes at the edges of their country’s “internet feed” to monitor and filter traffic. These little boxes are called DPI (Deep Packet Inspection) boxes and what they do, is sniff out every little packet flowing through them to find specific patterns and then they provide their administrator with the option to block traffic that matches these patterns. These boxes are very sophisticated and they don’t just filter traffic by src, dst or port, they filter traffic by the content (payload) the packets carry.
Unfortunately, it’s not just these countries that deploy DPI technologies, but some private companies also use such devices in order to monitor their employees.

The 10 thousand feet view
Tor is a nice way to avoid basic censorship technologies, but sometimes DPI technology is so good that it can fingerprint Tor traffic, which is already encrypted, and block it. In response to that, Tor people devised a technology called Pluggable Transports whose job is to obfuscate traffic in various ways so that it looks like something different than it actually is. For example it can make Tor traffic look like a skype call using SkypeMorph or one can use Obfsproxy to obfuscate traffic to look like…nothing, or at least nothing easily recognizable. What’s cool about obfsproxy though is that one can even use it separately from Tor, to obfuscate any connection he needs to.

A warning
Even though obfsproxy encrypts traffic and makes it look completely random, it’s not a fool proof solution for everything. It’s basic job is to defend against DPI that can recognize/fingerprint TLS connections. If someone has the resources he could potentially train his DPI box to “speak” the obfsproxy protocol and eventually decrypt the obfuscated traffic. What this means is that obfsproxy should not be used as a single means of protection and it should just be used as a wrapper _around_ already encrypted SSL traffic.
If you’re still in doubt about what can obfsproxy protect you from and from what it can’t, please read the Obfsproxy Threat Model document.

Two use cases
Obfuscate an SSH and an OpenVPN connection.
Obviously one needs a server outside the censorship perimeter that he or someone else will run the obfsproxy server part. Instructions on installing obfsproxy on Debian/Ubuntu are given in my previous blog post setting up tor + obfsproxy + brdgrd to fight censhorship. Installing netcat, the openbsd version; package name is netcat-openbsd on Debian/Ubuntu, is also needed for the SSH example.

What both examples do is obfuscate a TLS connection through an obfsproxy server so that it looks innocent. Assuming that the most innocent looking traffic is HTTP, try running the obfsproxy server part on port 80.

SSH connection
Scenario:
USER: running ssh client
HOST_A (obfsproxy): running obfsproxy on port 80 and redirecting to HOST_B port 22
HOST_B (dst): Running SSH server on port 22

What one needs to do is setup the following “tunneling”:
ssh client —> [NC SOCKS PROXY] —> obfsproxy client (USER)—> obfsproxy server (HOST_A) —> ssh server (HOST_B)

Steps:
1. on HOST_A setup obfsproxy server to listen for connection and redirect to HOST_B:
# screen obfsproxy --log-min-severity=info obfs2 --dest=HOST_B:22 server 0.0.0.0:80

2. on USER’s box, then configure obfsproxy client to setup a local socks proxy that will obfuscate all traffic passing through it:
$ screen obfsproxy --log-min-severity=info obfs2 socks 127.0.0.1:9999
Then instead of SSH-ing directly to HOST_B, the user has to ssh to HOST_A port 80 (where obfsproxy server is listening).

3. on USER’s box again, edit ~/.ssh/config and add something along the following lines:

Host HOST_A
    ProxyCommand /bin/nc.openbsd -x 127.0.0.1:9999 %h %p

This will force all SSH connections to HOST_A to pass through the local (obfsproxy) socks server listening on 127.0.0.1:9999

4. Finally run the ssh command:
$ ssh -p 80 username@HOST_A

That’s it. The connection will now pass get obfuscated locally, pass through obfsproxy server at HOST_A and then finally reach it’s destination at HOST_B.

OpenVPN connection
Scenario:
USER: running OpenVPN client
HOST_A (obfsproxy): running obfsproxy on port 80 and redirecting to HOST_B TCP port 443
HOST_B (dst): Running OpenVPN server on port 443

What one needs to do is setup the following “tunneling”:
openvpn client —> obfsproxy client (USER)—> obfsproxy server (HOST_A) —> OpenVPN server (HOST_B)

Steps:
1. on HOST_A setup obfsproxy server to listen for connection and redirect to HOST_B:
# screen obfsproxy --log-min-severity=info obfs2 --dest=HOST_B:443 server 0.0.0.0:80

2. on USER’s box, then configure obfsproxy client to setup a local socks proxy that will obfuscate all traffic passing through it:
$ screen obfsproxy --log-min-severity=info obfs2 socks 127.0.0.1:9999
Then instead of connecting the OpenVPN client directly to HOST_B, the user has edit OpenVPN config file to connect to HOST_A port 80 (where obfsproxy server is listening).

3. on USER’s box again, edit your openvpn config file, change the ‘port’ and ‘remote’ lines and add a ‘socks-proxy’ one:

port 80
remote HOST_A
socks-proxy 127.0.0.1 9999

This will instruct the OpenVPN client to connect to HOST_A passing through the local (obfsproxy) socks server listening on 127.0.0.1:9999

4. Finally run the openvpn client command:
$ openvpn client.config

That’s it.

Security Enhancement
You can “enhance” obfproxy’s security by adding a shared secret parameter to command line, so anyone who doesn’t have this secret key won’t be able to use the obfsproxy server, decryption of packets will fail:
# screen obfsproxy --log-min-severity=info obfs2 --shared-secret="foobarfoo" --dest=HOST_B:443 server 0.0.0.0:80
$ screen obfsproxy --log-min-severity=info obfs2 --shared-secret="foobarfoo" socks 127.0.0.1:9999

Documentation
Or at least…some documentation.

Your best chance to understand the internals of obfsproxy is to read the protocol specification

For more info about obfsproxy client part, read the documentation here: obfsproxy client external
For more info about obfsproxy server part, read the documentation here: obfsproxy server external

Screenshots
For those who like meaningless screenshots, here’s what wireshark (which is certainly NOT a DPI) can tell about a connection without and with obfsproxy:

Without obfsproxy

With obfsproxy

*Update*
one can find openvpn configuration files for use with obfsproxy here:
Linux Client
Windows Client

Filed by kargig at 01:54 under Encryption,Internet,Linux,Networking,Privacy
Tags: censorship, china, debian, Encryption, firewall, gfw, Linux, Networking, obfsproxy, openvpn, Privacy, security, ssh, tls, tor, vpn
5 Comments | 70,443 views

*WARNING* 14/01/2014 This post is quite deprecated. For example obfsproxy has been completely rewritten in python and there is a newer and more secure replacement of obfs2, named obfs3. Please read this obfsproxy-debian-instructions for any updates.

*Updated* look at the bottom for list of changes

This post is a simple guide to create a debian/ubuntu packages out of the latest versions of Tor, obfsproxy and brdgrd in order to setup a “special gateway” and help people who face censorship issues. Sharing some of your bandwidth helps a lot of people get back their freedom.

Tor
I guess most people already know what Tor is, quoting from Tor’s website:

Tor is a network of virtual tunnels that allows people and groups to improve their privacy and security on the Internet. It also enables software developers to create new communication tools with built-in privacy features. Tor provides the foundation for a range of applications that allow organizations and individuals to share information over public networks without compromising their privacy.

obfsproxy

obfsproxy is a tool that attempts to circumvent censorship, by transforming the Tor traffic between the client and the bridge. This way, censors, who usually monitor traffic between the client and the bridge, will see innocent-looking transformed traffic instead of the actual Tor traffic.

brdgrd

brdgrd is short for “bridge guard”: A program which is meant to protect Tor bridges from being scanned (and as a result blocked) by the Great Firewall of China.

Combining these to work together is quite easy if you follow this simple guide/howto.

////// Become root
$ sudo su -

////// Get build tools/packages
# cd /usr/src/
# apt-get install build-essential libssl-dev devscripts git-core autoconf debhelper autotools-dev libevent-dev dpatch pkg-config
# apt-get install hardening-includes asciidoc docbook-xml docbook-xsl xmlto
# apt-get install screen libnetfilter-queue-dev

////// Get latest versions of tor/obfsproxy/brdgrd
# git clone https://git.torproject.org/debian/obfsproxy.git
# git clone https://git.torproject.org/debian/tor.git
# git clone https://git.torproject.org/brdgrd.git

////// Compile obfsproxy & create package
# cd obfsproxy/
# ./autogen.sh 
# debuild -uc -us 

////// Compile tor & create package
# cd ../tor/
# ./autogen.sh 
# debuild -uc -us 

////// Install packages
////// The following package versions might be different depending on your configuration. Change them appropriately by looking at the deb files in your path: ls *.deb

# cd ..
# dpkg -i tor-geoipdb_0.2.4.3-alpha-1_all.deb obfsproxy_0.1.4-2_amd64.deb tor_0.2.4.3-alpha-1_amd64.deb

////// Create Tor configuration
////// PLEASE SEE THE CHANGEME_X VARIABLE BELOW BEFORE RUNNING THE FOLLOWING COMMAND

# cat > /etc/tor/torrc << EOF 
AvoidDiskWrites 1
DataDirectory /var/lib/tor
ServerTransportPlugin obfs2 exec /usr/bin/obfsproxy --managed
Log notice file /var/log/tor/notices.log
## If you want to enable management port uncomment the following 2 lines and add a password
## ControlPort 9051
## HashedControlPassword 16:CHANGEME
## CHANGEME_1 -> provide a nickname for your bridge, can be anything you like.
Nickname CHANGEME_1
## CHANGEME_2 -> How many KB/sec will you share. Don't be stingy! Try putting _at least_ 20 KB.
RelayBandwidthRate CHANGEME_2 KB
## CHANGEME_3 -> Put a slightly higher value than your previous one. e.g if you put 500 on CHANGEME_2, put 550 on CHANGEME_3.
RelayBandwidthBurst CHANGEME_3 KB
ExitPolicy reject *:* 
## CHANGEME_4 -> If you want others to be able to contact you uncomment this line and put your GPG fingerprint for example.
#ContactInfo CHANGEME_4
ORPort 443 
#ORPort [2001:db8:1234:5678:9012:3456:7890:1234]:443
BridgeRelay 1
## CHANGEME_5 -> If you don't want to publish your bridge in BridgeDB, so you can privately share it with your friends uncomment the following line
#PublishServerDescriptor 0
EOF

////// Restart Tor

# /etc/init.d/tor restart

////// Compile and run brdgrd
////// If you've changed ORport in Tor config above, be sure to change the "--sport 443" port below as well
////// brdgrd does not help since obfsproxy is already running in front of the bridge, but won't hurt either.

# cd brdgrd/
# make
# iptables -A OUTPUT -p tcp --tcp-flags SYN,ACK SYN,ACK --sport 443 -j NFQUEUE --queue-num 0
////// brdgrd Can't do IPv6 yet...so the next line is commented out
////// ip6tables -A OUTPUT -p tcp --tcp-flags SYN,ACK SYN,ACK --sport 443 -j NFQUEUE --queue-num 0
////// You can run brdgrd without root, just by setting some correct cap_net_admin rights
////// Instead of: screen -dmS brdgrd ./brdgrd -v
$ sudo screen -dmS brdgrd setcap cap_net_admin=ep ./brdgrd -v

# tail -f /var/log/tor/notices.log

The above guide has been tested on Debian Squeeze and Ubuntu 12.04.

That’s it. You just made the world a better place.

*Update*
I’ve made some changes to the post according to comments on the blog post and #tor-dev.
a) Changed URLs for the git clone operations to https:// instead of git://
b) Changed brdgrd git url to gitweb.torproject.org instead of github.
c) Changed config sections of torrc file
d) Added some more info on brdgrd

*Update2*
Tor has published “official” instructions for setting up obfsproxy bridges on Debian boxes –> Setting up an Obfsproxy Bridge on Debian/Ubuntu

*Update3*
Update sample config to inform about unpublished bridges.

Filed by kargig at 18:50 under Encryption,IPv6,Linux,Networking,Privacy
Tags: 12.04, brdgrd, bridgedb, censorship, debian, guide, HOWTO, ipv6, Linux, obfsproxy, Precise Pangolin, Privacy, Squeeze, tor, ubuntu
7 Comments | 30,872 views

Bind9, since version 9.5, offers an experimental embedded web server which can provide statistics abound Bind through HTTP. Upon enabling, one can access this web server and get an XML response full with various statistics.

Enabling the feature is quite easy. One just needs to add some lines like the following inside Bind’s configuration file:

statistics-channels {
          inet 127.0.0.1 port 8053 allow {127.0.0.1;};
};

Restart Bind and try connecting to the statistics-channel. For example through curl:

$ curl http://127.0.0.1:8053
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="/bind9.xsl"?>
<isc version="1.0">
  <bind>
    <statistics version="2.2">
      <views>
        <view>
          <name>_default</name>
          <zones>
            <zone>
              <name>0.in-addr.arpa/IN</name>
              <rdataclass>IN</rdataclass>
              <serial>1</serial>
            </zone>
            <zone>
              <name>10.in-addr.arpa/IN</name>
              <rdataclass>IN</rdataclass>
              <serial>1</serial>
            </zone>
...
...
...
...
         <TotalUse>810834110</TotalUse>
          <InUse>327646360</InUse>
          <BlockSize>740556800</BlockSize>
          <ContextSize>9954232</ContextSize>
          <Lost>0</Lost>
        </summary>
      </memory>
    </statistics>
  </bind>
</isc>

The output will be quite big, for example the current output from a busy recursive resolver is around 2.5Mb.

People usually use munin to monitor bind through 2 different ways. The first, usually applied to servers that are not very busy, for example servers with less than 100queries/sec, is to parse the output of the query log. Then a munin plugin that comes with the default munin installation and is called bind9 can parse the query log and create some graphs. The second way for a bit busier servers is to have query log disabled and monitor bind is through the output of the rndc stats command. A munin plugin that also comes with the default munin package and is called bind9_rndc can parse the created named.stats output file and create some nice graphs. To configure either of the two ways one can take a look at this informational wiki: http://wiki.debuntu.org/wiki/Munin/Bind9_Plugin

A third way is to have a new munin plugin get the XML output from the statistics-channel described above and create some even fancier and richer graphs. Luckily a guy called Andrew Duquette has created such a perl plugin for munin. You need to exercise your google skills to find it though, (that is if you don’t cheat searching with his name as a search term!) I took that plugin, made some changes to it and uploaded it to github. The plugin is called bind9_statchannel and you can find it in my github munin-plugins repo.

The changes I made were the following:

• Use STACK instead of just AREA for some graphs
• Add warning and critical levels for type ANY queries (to warn for DDoS through such queries)
• Raise timeout to 300 seconds
• Add sorting to stacked graphs so they are a bit more readable/comparable

The plugin creates the following type of graphs:

• Cache DB RRsets for various views (or the _default if you don’t have any others)
• Memory Context “In Use”
• Memory Usage Summary
• Queries In
• Queries Out
• Resolver Statistics for various views (or the _default if you don’t have any others)
• Server Statistics
• Socket I/O Statistics
• Tasks

As you can see it can give many more details compared to the other bind9 munin plugins. It’s also the only bind9 munin plugin that can tell you how many incoming vs outgoing queries there are, as well as the only one that can tell you how many queries you have over IPv4 vs IPv6.

To use it on Debian you need to at least install the following two perl packages: libxml-simple-perl, liblwp-useragent-determined-perl

Here are some sample graphs from the bind9_statchannel plugin:

If you have any bug fixes, code changes, etc please don’t hesitate to fork and open a pull request on github.

Download: bind9_statchannel

Filed by kargig at 23:04 under General,Internet,Linux,Networking
Tags: bind9, bind9_statchannel, debian, dns, ipv6, Linux, munin, perl, plugin, statistics, statistics-channel
7 Comments | 25,082 views

HTTPS Everywhere is a browser addon by EFF whose job is to redirect you to the HTTPS versions of certain, whitelisted, web sites. What this means is that HTTPS Everywhere protects your communication with those websites by forcing them to be encrypted.

The current HTTPS Everywhere ruleset lacks any Greek websites, so I started yet-another-list to create rules for Greek websites. This is the fourth list I’m maintaing after GrRBL, Greek Spammers Blacklist and Greek AdblockPlus Filter rules and it is the only one where being included is actually a good thing.

You can find some more info about Greek rules for HTTPS Everywhere on my github page.

Until the rules get adopted upstream by HTTPS Everywhere team, in order to use them you should download the rules and place them inside your Firefox profile directory. But first of all you need to install the plugin/extension/addon/call-me-whatever-you-want by going to HTTPS Everywhere page.

Step 1: Instructions for Linux users
Go to the HTTPSEverywhereUserRules directory inside your firefox profile directory:

$ cd .mozilla/firefox/XYZXYZXYZ.default/HTTPSEverywhereUserRules/
(XYZXYZXYZ will be different in your machine)

and download the current Greek ruleset:
$ wget https://raw.github.com/kargig/https-everywhere-greek-rules/master/Greek.xml

Step 1: Instructions for Windows users
Download https://raw.github.com/kargig/https-everywhere-greek-rules/master/Greek.xml with your favorite browser.
Then, according to this Mozilla support page, open Fifefox, go to Help->Troubleshooting Information and under the Application Basics section, click on Open Containing Folder. There a window will appear and you should copy the previously downloaded Greek.xml file inside the HTTPSEverywhereUserRules folder.

Step 2: Instructions for any OS
Either restart your browser to load the new rules or click the HTTPS Everywhere icon beside the url bar, select “Disable HTTPS Everywhere”, then click it again and select “Enable HTTPS Everywhere”. The new rules should now be loaded, you can test by going to http://void.gr and it should immediately redirect you to https://void.gr

Some notes
The ruleset is experimental. If you find any problems please report them as issues to github.
If you want a Greek website added to the list, either report it as a new issue on github or fork the repository, add your own rules and open a pull request.

A small rant
I found some webmails in Greece that don’t even offer HTTPS as an option to the user. They ‘POST’ user details, including passwords of course, over unencrypted HTTP connections. I will be updating a text file called hallofshame.txt inside the github reposity of Greek rules for HTTPS Everywhere with such websites. I am planning to inform the operators of such websites every now and then, so if you know any other cases please open up new issues so we can help protect innocent users.

A big rant on current HTTPS status of top Greek websites
The status of HTTPS support on top 100 Greek websites (according to Alexa) is SAD. No wait, it is EXTEMELY SAD. Out of these 100 websites, taking into account only the ones that are actually run by Greeks, that means excluding Google, Facebook, Youtube, LinkedIn, etc, only 2, yes you read correctly, just two websites offer HTTPS support.
The reason 95% the others don’t is probably because they are based on Akamai-zed services and either don’t have the money to buy Akamai’s HTTPS products or don’t have the technical skills to do it properly.

If you don’t run an Akamai-zed website and want a completely free 1-year SSL certificate please visit https://www.startssl.com/. If you need professional help with your setup please don’t hesitate to contact.

There’s a very good (financial) explanation why these high traffic Greek sites have prefered Akamai’s services and haven’t deployed their own servers in Greece but this will be the content of another blog post coming soon.

Filed by kargig at 19:51 under Encryption,Greek,Internet,Linux,Networking,Privacy
Tags: EFF, Encryption, https, HTTPS-Everywhere, Linux, Networking, Privacy, security, ssl, tls
2 Comments | 18,410 views

On a 24-core Linux machine I wanted to monitor interrupts/sec. The reason is quite complicated, let’s say that there are more than 100.000 interrupts/sec and I want to monitor who needs what.

First I installed Debian’s itop package. It didn’t work at all on the 24-core machine, just showed an empty display. Then I tried a perl itop implementation I found online at a redhat server, let’s call it redhat’s itop. That still didn’t work well. Lot’s of parsing issues and it certainly wasn’t ready to deal with 24-cores.

So I tweaked it a bit and made it ready for a 24-core server. You can find it online at my github repository: kargig’s itop

Since displaying 24 columns of continuously changing numbers doesn’t fit all screens, I added some command line options to the tool.
By default it shows all interrupts/sec per CPU if the number of CPU cores is not higher than 8. If it’s higher than 8 it just displays the total (adding all per core numbers) interrupts/sec.
Command line flag ‘-a’ forces displaying of ALL cores and command line flag ‘-t’ forces displaying just the TOTAL number of interrupts/sec.

So if you too need to monitor your server’s interrupts/sec, give my itop a try. If you want to make changes, just fork it on github. Simple as that…I promise to merge any interesting changes/pull requests you people send me!

oh and here’s a screenshot of ‘itop -t’

Filed by kargig at 18:22 under Linux
Tags: debian, interrupts, interrupts per second, itop, Linux, perl, system administration
No Comments | 21,833 views

Από δελτίο τύπου του Οργανισμού Πνευματικής Ιδιοκτησίας:

…στις 16 Μαΐου 2012 δημοσιεύθηκε η απόφαση 4658/2012 του Μονομελούς Πρωτοδικείου Αθηνών, η οποία έκανε δεκτό αίτημα οργανισμών συλλογικής διαχείρισης δικαιωμάτων επί μουσικών και οπτικοακουστικών έργων να υποχρεωθούν εκτός άλλων οι ελληνικές εταιρίες παροχής υπηρεσιών σύνδεσης στο διαδίκτυο να λάβουν τεχνολογικά μέτρα προκειμένου να καταστεί αδύνατη η πρόσβαση των συνδρομητών τους σε διαδικτυακές τοποθεσίες μέσω των οποίων πραγματοποιείται παράνομη παρουσίαση και ανταλλαγή έργων. Η απόφαση εφαρμόζει ουσιαστικά για πρώτη το άρθρο 64 Α του ν. 2121/1993 που ενσωματώνει πρόβλεψη Οδηγίας της Ευρωπαϊκής Ένωσης για τη δυνατότητα λήψης ασφαλιστικών μέτρων κατά των διαμεσολαβητών (παρόχων υπηρεσιών διαδικτύου), οι υπηρεσίες των οποίων χρησιμοποιούνται από τρίτο για την προσβολή του δικαιώματος του δημιουργού ή συγγενικού δικαιώματος. Παρόμοιες αποφάσεις έχουν ήδη εκδοθεί σε άλλα κράτη μέλη της Ευρωπαϊκής Ένωσης και αποσκοπούν στην προστασία της πνευματικής ιδιοκτησίας στο διαδίκτυο χωρίς να θίγονται τα δικαιώματα των χρηστών….

Την πλήρη απόφαση μπορείτε να την διαβάσετε εδώ: 4658/2012
*Update*
Επειδή το site του ΟΠΙ δεν παρέχει πια την απόφαση, την έχω ανεβάσει εδώ: Απόφαση-του-Πρωτοδικείου-Αθηνών-για-την-αντιμετώπιση-της-διαδικτυακής-πειρατείας

Γιατί είναι σημαντική αυτή η απόφαση για τους χρήστες
Για πρώτη φορά στην Ελλάδα δικαστήριο επιβάλλει συγκεκριμένα τεχνικά/τεχνολογικά μέτρα παρεμπόδισης της πρόσβασης χρηστών σε ιστοσελίδες/servers. Σήμερα μπορεί να είναι μια ιστοσελίδα που παρέχει “πειρατικό” περιεχόμενο και ο ιδιοκτήτης της βγάζει χρήματα μέσω των διαφημίσεων, αύριο μπορεί να είναι ένα site που ο ιδιοκτήτης του δεν βγάζει χρήματα και μεθαύριο ένα πολιτικό site, ένα θρησκευτικό site, ένα blog που διαφωνεί με τις μεθόδους μιας εταιρίας, μιας κυβέρνησης, κτλ. Οπότε πρέπει ως χρήστες να ξέρουμε τι επιβάλλει το δικαστήριο και να δούμε πως εμείς, ως μέλη της κοινωνίας του Internet, μπορούμε να κάνουμε κάτι για να ακυρώσουμε στην πράξη μια τέτοια απόφαση αν πιστεύουμε πως αυτή είναι λανθασμένη.

Τι περιγράφει η απόφαση
Η απόφαση περιέχει μια λεπτομερή τεχνική έκθεση που εξηγεί πως δουλεύει ένα site, ποια πρωτόκολλα χρησιμοποιούνται από τα μηχανήματα των χρηστών/πελατών για να αποκτήσουν πρόσβαση στο site και έπειτα περιγράφει τρόπους να διακοπεί η σύνδεση των χρηστών με ένα site. Οι τρόποι που παρουσιάζονται είναι οι εξής 2:
Ι) Εφαρμογή κατάλληλων φίλτρων στους δρομολογητές (routers) των ISPs ώστε να αποκλειστεί οποιαδήποτε κίνηση καταλήγει σε συγκεκριμένη IP.
ΙΙ) Εφαρμογή κατάλληλης ανακατεύθυνσης, μέσω τροποποίησης των DNS εγγραφών στους nameservers του κάθε ISP ώστε, ώστε τα αιτήματα προς συγκεκριμένα domains να καταλήγουν σε διαφορετικούς ιστοτόπους. Αυτοί οι ιστότοποι θα μπορούσαν να περιέχουν και ένα προειδοποιητικό μύνημα ώστε να καταλαβαίνουν οι χρήστες γιατί δεν έχουν πρόσβαση στο κανονικό site, όπως αναφέρει το η έκθεση.

Από αυτούς τους 2 τρόπους, στην απόφαση επιβάλλεται η χρήση μόνο του τρόπου (I) ως τεχνολογικό μέτρο διακοπής της πρόσβασης στις “παραβατικές” ιστοσελίδες.

Τα προβλήματα της απόφασης
Τα προβλήματα της απόφασης για μένα είναι αρκετά. Κάποια αναφέρονται και στην ίδια την τεχνική έκθεση που περιέχεται στην απόφαση.
Συγκεκριμένα αναφέρει:

Αν και υπάρχουν δυνατότητες παράκαμψης των συγκεκριμένων τεχνικών μέσων από την μεριά των χρηστών του διαδυκτύου, οι τεχνικές αυτές είναι άγνωστες στη μεγάλη πλειονότητα των πελατών (συνδρομητών) των ISP, που είναι οι δυνητικοί επισκέπτες των ιστοτόπων στους οποίους έχει διακοπεί η πρόσβαση.

Θα αναφερθώ μόνο στα πολύ βασικά όμως…
α) Καταρχήν τα sites έχουν αλλάξει IPs. Το www.ellinadiko.com πλέον δεν δείχνει στην IP που αναφέρεται στην απόφαση, για την ακρίβεια δεν δείχνει πουθενά αυτή τη στιγμή, ενώ το www.music-bazaar.com λειτουργεί αλλά δείχνει σε διαφορετική IP. Άρα η εφαρμογή της οδηγίας (Ι) είναι πρακτικά άχρηστη ως προς τους σκοπούς της απόφασης χωρίς πολλά πολλά. Από την άλλη όμως μπορεί να δημιουργήσει προβλήματα πρόσβασης σε άλλα sites που μπορεί αυτή τη στιγμή να φιλοξενούνται σε εκείνες τις IP για τις οποίες πρέπει να μπουν φίλτρα. Άρα αν εφαρμοστεί η απόφαση ως έχει κινδυνεύει να διακοπεί η πρόσβαση στο site μιας ελληνικής ή ξένης εταιρίας ή προσώπου χωρίς να φταίει σε τίποτα! Ακόμα να μην είχαν αλλάξει IPs τα sites αυτά όμως, πάλι προκύπτει πρόβλημα. Η σύγχρονη τεχνολογία, των τελευταίων 15+ ετών, επιτρέπει την φιλοξενία πολλαπλών ιστοτόπων στην ίδια IP μέσω της τεχνολογίας virtual hosting, κάτι που εφαρμόζεται κατά κόρον ώστε να εξοικονομηθούν IPs. Αυτό έχει σαν αποτέλεσμα πως αν αποτραπεί η κίνηση προς μία συγκεκριμένη IP από ένα φίλτρο ενός ISP, τότε παρεμποδίζεται και η κίνηση προς όλα τα υπόλοιπα sites που φιλοξενούνται στην ίδια IP. Άρα υπάρχει πιθανότητα “τιμωρίας” αθώων ανθρώπων που δεν έχουν κάνει απολύτως τίποτα.

β) Η τεχνική έκθεση και η απόφαση περιέχει συγκεκριμένα domains που θα πρέπει να εφαρμοστεί το (II). Αυτό όμως δεν εμποδίζει σε τίποτα τον διαχειριστή της “προβληματικής” ιστοσελίδας να αλλάξει αύριο domain κρατώντας ακριβώς το ίδιο περιεχόμενο. Οπότε εμποδίζοντας την πρόσβαση στους πελάτες πίσω από ένα ISP σε ένα συγκεκριμένο domain δεν καταφέρνεις και πολλά. Ακόμα όμως και να μην αλλάξει domain ο διαχειριστής μιας και υπάρχουν ελέυθεροι nameservers (Google Public DNS, OpenDNS, κ.α) στο Internet, το μόνο που θα είχε να κάνει ο χρήστης θα ήταν να χρησιμοποιήσει αυτούς έναντι των nameservers του ISP του. Άρα πάλι τα τεχνικά μέτρα είναι εντελώς ανεπαρκή ως προς τον σκοπό της απόφασης. Πέραν αυτού και λόγω της προτεινόμενης ανακατεύθυνσης που προτείνει η τεχνική έκθεση τίθεται και ένα θέμα ιδιωτικότητας σε περίπτωση που εφαρμοζόταν το μέτρο (ΙΙ). Λόγω της ανακατεύθυνσης όλοι οι πελάτες θα “πήγαιναν” σε μία νέα ιστοσελίδα που θα ήταν υπό τη διαχείριση (μάλλον?) του ISP, άρα ο ISP αποκτάει πολύ εύκολα πρόσβαση στο ποιός θέλει να επισκεφτεί τον ιστότοπο αυτό. Τίθεται λοιπόν ζήτημα παρακολούθησης της κίνησης των πελατών. Προσωπικά το θεωρώ απαράδεκτο, όπως απαράδεκτο είναι να προσπαθείς να αλλάξεις τον τρόπο που λειτουργεί το internet. Άλλωστε όπως έχει πει ο John Gilmore:

The Net interprets censorship as damage and routes around it

Μετάφραση:

Το Δίκτυο ερμηνέυει τη λογοκρισία ως ζημιά και δρομολογεί (την κίνηση) γύρω από αυτό (ξεπερνώντας την ζημιά)

Τι θα μπορούσαν να κάνουν οι χρήστες για να παρακάμψουν το “πρόβλημα” αν τους επηρέαζε
Σε περίπτωση εφαρμογής του (II), όπως αναφέρθηκε παραπάνω το μόνο που θα είχαν να κάνουν οι χρήστες θα ήταν να αλλάξουν nameservers στο PC/δίκτυο τους. Αυτό εξηγείται αναλύτικά στις σελίδες της Google Public DNS αλλά και του OpenDNS. Τόσο απλά. Είναι υπόθεση 1 λεπτού αν έχει ο οποιοσδήποτε τις οδηγίες μπροστά του.

Σε περίπτωση εφαρμογής της τεχνικής (Ι) και την στιγμή που το site δεν μπορεί για τους Χ λόγους να αλλάξει IP, αυτό που πρέπει να κάνουν οι χρήστες είναι να χρησιμοποιήσουν κάποιον proxy server, ένα VPN ή κάποιο άλλο δίκτυο που δρομολογεί διαφορετικά τις συνδέσεις τους, για παράδειγμα το Tor. Ο ευκολότερος τρόπος να βρει κάποιος δωρεάν proxies στο δίκτυο είναι να ψάξει στο Google, ενώ η αγορά ενός VPN ξεκινά από τα 3€. Η χρήση του tor είναι πλεόν αρκετά απλή και το μόνο που απαιτείται είναι να κατεβάσει κανείς το Tor Browser Bundle και να τρέξει το Vidalia. Όταν κάποιος τρέξει το Vidalia θα ανοίξει ένας νέος browser (Firefox) και έπειτα η δρομολόγηση των πακέτων προς το site που θέλει να επισκευτεί κανείς γίνεται μέσω του Tor δικτύου το οποίο είναι αρκετά δύσκολο να το σταματήσουν οι ISPs. Σίγουρα πάντως η απόφαση ασφαλιστικών μέτρων 4658/2012 δεν είναι ικανή να σταματήσει το Tor ή οποιονδήποτε άλλο από τους παραπάνω τρόπους παράκαμψης του “προβλήματος”.

Τι πρέπει να γνωρίζουν οι χρήστες του Internet
Οι χρήστες του internet πρέπει να γνωρίζουν πως ανά πάσα στιγμή μια τέτοια απόφαση μπορεί να τους αλλάξει τις συνήθειές τους αλλά και να τους κόψει την πρόσβαση από πηγές πληροφορίας που μέχρι τώρα είχαν ελεύθερη πρόσβαση. Για να μην βρεθούν τελευταία στιγμή να αναρωτιούνται τί και πώς πρέπει να φροντίζουν να ενημερώνονται για τους κινδύνους και τα προβλήματα. Είναι μάλιστα επιτακτικό ο ένας χρήστης να ενημερώνει τους άλλους. Γι αυτούς ακριβώς τους λόγους τους τελευταίους 2-3 μήνες έχει ξεκινήσει μια προσπάθεια ενημέρωσης των Ελλήνων χρηστών για τα ψηφιακά τους δικαιώματα, τους κινδύνους που υπάρχουν στο διαδίκτυο, πως προστατεύει κανείς τα προσωπικά του δεδομένα και πως αποφεύγει προσπάθειες εταιρικής ή κρατικής λογοκρισίας μέσω κάποιων παρουσιάσεων που γίνονται στο hackerspace της Αθήνας. Η επόμενη παρουσίαση γίνεται στις 30/05/2012 και αφορά την χρήση του δικτύου Tor. Όσοι ενδιαφέρονται είναι ευπρόσδεκτοι να έρθουν να ακούσουν και φυσικά να ρωτήσουν για τυχόν απορίες που ίσως έχουν σχετικά με την ψηφιακή τους ζωή.

Οργανωθείτε!
Αν σας ενδιαφέρει να παλέψετε και εσείς για τα ψηφιακά δικαιώματα και τις ελευθερίες στην Ελλάδα καλό θα ήταν να διαβάσετε το κείμενο θέσεων του Δικτύου για την Ψηφιακή Απελευθέρωση (Digital Liberation Network) και αν συμφωνείτε να εγγραφείτε στην mailing list του DLN.

Filed by kargig at 21:06 under Encryption,General,Greek,Internet,Linux,MacOSX,Networking,Privacy
Tags: 4658/2012, access list, censorship, dns redirection, freedom_of_speech, google dns, Greek, hackerspace, Networking, opendns, Privacy, proxy, security, tor, vpn, ανακατεύθυνση, λογοκρισία, ΟΠΙ
10 Comments | 20,069 views

Alternate title: “Being a lamb around a pack of wolves” … A venue full of hackers that are eager to attack your systems…

On 3-4/05/2012 the third AthCon conference was held in Athens. AthCon is an international security conference whose motto is “The First HIGHLY TECHNICAL Security Conference in Greece”.

Even though I am not a security professional, my daily job title is “Systems and Services Engineer” which of course includes various aspects of security but I am certainly not a security researcher, I had decided months ago that I would be attending this year’s AthCon. Since I like messing a lot with IPv6 for the past 2-3 years, I decided that I could try and submit an introductory talk about IPv6 security issues. My talk was accepted, so I was not only attending AthCon this year but I was going to give a presentation as well.

My presentation – Are you ready for IPv6 insecurities ? was during the first day of the conference. I am always worried when I give presentations on IPv6 that the people attending have probably no clue about this ‘not-so-new’ protocol. Most people think that IPv6 is like IPv4 with bigger addresses and ‘:’ instead of ‘.’ to separate the address groups, which is of course a HUGE mistake/misunderstanding. I was hopeful that this wouldn’t be the case in AthCon, so when I started my presentation and I asked the crowd ‘how many of you know what SLAAC is ?’ and I only saw 3-4 hands raised I kinda froze, I was expecting at least a double digit…I was going to give a presentation on IPv6 security concepts to people that have absolutely no idea what I’m talking about. Being prepared for the fact that some people would need some ‘refreshing’ on their IPv6 knowledge, I had prepared around 20 introductory slides explaining some IPv6 concepts before I entered the security details, but I doubt these were enough for most people there. I am hopeful though that some of the attendees might be motivated to read more about the protocol since I think my security slides contained enough details, references and links to get people started. If someone needs more details feel free to contact me.

Enough with my presentation, what about other presentations ?
My personal view is that this year’s AthCon had some great talks, some that were ok and some that I didn’t like. I won’t mention which ones I didn’t like, but I noticed that a LOT of people were gossiping about these in the hallways. I will only mention here the ones that I really liked.

Day 1:
“Packing Heat!” by Dimitrios Glynos
A presentation that every pentester should download/watch somehow. Techniques about packing your executables to avoid detection by anti-virus programs, need I say more ? Great content and very well presented. Congrats Dimitris!

“PostScript: Danger Ahead” by Andrei Costin
How to use PostScript programming language to take advantage of Printers, OS, etc. Very interesting concepts were presented and also the examples/demos shown were pretty cool and easy to understand.

Day 2:
“Apple vs. Google Client Platforms” by Felix ‘FX’ Lindner
I guess mostly everyone reading this blog knows FX and what a great speaker he is. If you don’t then start watching his previous presentations and start reading about his work. His presentation at AthCon, apart from being the best one in terms of “presenting it”, was also extremely interesting. He connected the security concepts behind Apple’s iOS and Google’s Chromebook with their business tactics and policies. Just wait for AthCon to publish the videos and watch it. Probably the best talk at AthCon 2012.

“Advances in BeEF: RESTful API, WebSockets, XssRays enhancements” by Michele Orru
Jaw-dropping. That’s all I have to say about BeEF. Scary. Watch it to see what browsers and IDS have to face and defend against…not in the future but right now.

“Exploitation and state machines” by Halvar Flake
This presentation was about exploitation techniques and why automated exploitation engines don’t work that well. Even though reversing and exploitation is far from my interest topics I enjoyed the talk a lot. Very well structured and very clear points. Too bad this talk did not appear on the schedule and was there as “tbc”, I am sure many more people would come just to listen to this talk and speak to Halvar.

If I were to suggest a couple of things for next year…
a) Please put the CTF in separate slots within the day, not at the same time with the presentations. In a conference of 150-200 people (just guessing here) having 30+ people leaving the presentation room and just attending the CTF all day long leaves the main room a bit empty. I am pretty sure there were people that wanted to attend both the presentations and the CTF, unfortunately they had to make a choice.
b) Send some details/info to the speakers about the conference a few days earlier. Maybe non-greek presenters were given but we weren’t, at least I wasn’t.
c) The venue is really nice, but maybe it would help if the next AthCon was organized somewhere downtown. Yeah I can understand that the cost would be higher but number of people attending would also raise (I think).
d) Give us even more highly technical presentations/speakers! People starve for these kind of talks!

My congratulations fly to AthCon people for organizing the conference. See you next year!

You can find some of the pics I took from the speakers at: AthCon 2012 speaker pics (if any of the speakers wants his pic removed please contact me ASAP)

Filed by kargig at 15:07 under Encryption,Internet,IPv6,Linux,Networking,Privacy
Tags: apple, athcon, athens, beef, conference, ctf, exploitation, google, hacking, ipv6, packing, postscript, presentation, security, vulnerability, web application, xss
No Comments | 6,742 views

My presentation for AthCon 2012 is now available online: Are you ready for IPv6 insecurities ?

Filed by kargig at 23:41 under IPv6,Linux
Tags: athcon, attack, DHCPv6, dns, firewall, ids, ipv6, Linux, presentation, scanning, security, slaac, thc-ipv6
1 Comment | 7,692 views

Browsers on Linux should use xdg-open to open URLs. From xdg-open’s description:

xdg-open opens a file or URL in the user’s preferred application. If a URL is
provided the URL will be opened in the user’s preferred web browser. If a file
is provided the file will be opened in the preferred application for files of
that type. xdg-open supports file, ftp, http and https URLs.

Magnet is a new type of URLs that’s currently not supported by xdg-open. It’s very easy to add a quick hack to support it though. Just copy xdg-open to a local binary path (e.g /usr/local/bin or preferably $HOME/bin) and apply the following patch. Change deluge with your preferred torrent application.

--- /usr/bin/xdg-open  2010-09-15 14:08:29.000000000 +0300
+++ bin/xdg-open  2012-01-24 22:05:03.935338593 +0200
@@ -437,6 +437,11 @@
                 exit_success
             fi
         fi
+    elif (echo "$1" | grep -q '^magnet:'); then
+        deluge "$1" 
+        if [ $? -eq 0 ]; then
+            exit_success
+        fi
     fi
 
     sensible-browser "$1"

For those who don’t know how to patch a file, here it is in full: patched xdg-open to open magnet URLs

Works fine on my Debian using Chrome which properly uses xdg-open to handle URLs. Firefox after version 3.5 became a piece of bloatware and it’s not possible to open magnet links using about:config settings or xdg-open. For Opera just go to Settings->Preferences->Programs and add new protocol with name magnet. At Open with other application, use the path that you saved the patched xdg-open.

Now go download some creative commons licensed files.

*UPDATE*
Patch for latest xdg-utils (since some people complained that previous patch does not apply):

--- xdg-open  2012-02-18 14:22:48.058497027 +0200
+++ xdg-open  2012-02-18 14:24:04.326875223 +0200
@@ -440,6 +440,11 @@
                 exit_success
             fi
         fi
+    elif (echo "$1" | grep -q '^magnet:'); then
+        deluge "$1" 
+        if [ $? -eq 0 ]; then
+            exit_success
+        fi
     fi
 
     IFS=":"

*UPDATE 2*
For Firefox/Iceweasel one can do the following:
go to about:config and right-click. Then click on New->Boolean->network.protocol-handler.expose.magnet -> Value -> false
Upon the next click Firefox/Iceweasel will ask you to choose a program to open magnet links, choose the patched xdg-open posted above.

Filed by kargig at 23:39 under Internet,Linux
Tags: bash, chrome, debian, deluge, firefox, iceweasel, Linux, magnet, opera, torrent, xdg
14 Comments | 70,849 views

I use a Mac Mini with OS X 10.5.8 as a media center connected to my TV and I wanted to install Music Player Daemon on it so I could control the music remotely from my laptop or phone. I mostly followed the OS X guide from MPD’s wiki to do it but I ran into some problems while trying to daemonize mpd.

I got the following error while running mpd without –no-daemon:

The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.

When I ran mpd –no-daemon everything was fine though. So in order to “solve” this problem I’ve changed the plist file to include a screen invocation.

My mpd.plist looks like that now:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
    <key>Label</key>
    <string>mpd</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/local/bin/daemondo</string>
        <string>--label=mpd</string>
        <string>--start-cmd</string>
        <string>screen</string> <!-- path to screen -->
        <string>-dmS</string>
        <string>mpd</string>
        <string>/opt/local/bin/mpd</string> <!-- path to MPD -->
        <string>-v</string>
        <string>--no-daemon</string>
        <string>/Users/kargig/.mpd/mpd.conf</string> <!-- path to MPD config -->
        <string>;</string>
        <string>--stop-cmd</string>
        <string>/opt/local/bin/mpd</string> <!-- path to MPD, again -->
        <string>--kill</string>
        <string>/Users/kargig/.mpd/mpd.conf</string> <!-- path to MPD config, again -->
        <string>;</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>Disabled</key>
    <false/>
    </dict>
</plist>

So launchctl calls daemondo which calls screen which runs mpd –no-daemon, so mpd doesn’t crash.

I use mpdscribble for scrobbling my music to last.fm. Clients-wise, I use Theremin on OS X, Gnome Music Player Client/gmpc on Debian Linux and MPDroid on Android. And all those connections over IPv6 of course, over my LAN’s Unique Local Addresses to be exact, mpd and all clients listed above work fine with IPv6.

 # lsof -n -i | grep ESTABLISHED | grep 6600
mpd       43025         kargig   12u  IPv6 0x49c719c      0t0    TCP [fdbf:aaaa:aab0:447d:216:XXff:feaa:11XX]:6600->[fdbf:aaaa:aab0:447d:222:XXff:fe1e:d8XX]:48703 (ESTABLISHED)
mpd       43025         kargig   15u  IPv6 0x3127cd4      0t0    TCP [fdbf:aaaa:aab0:447d:216:XXff:feaa:11XX]:6600->[fdbf:aaaa:aab0:447d:fadb:XXff:fe4f:aXX]:51113 (ESTABLISHED)

Apart from MPD’s wiki there’s another nice blog post you can read to help you install mpd on OS X, Integrating MPD with OS X.
For general reference on setting up mpd, Archilinux has a fine wiki entry.

Filed by kargig at 22:52 under IPv6,Linux,MacOSX
Tags: debian, gmpc, ipv6, last.fm, Linux, Mac, mac mini, mpd, MPDroid, mpdscribble, Music Player Daemon, OS X, scrobble, Theremin, ULA, Unique Local Address
1 Comment | 24,335 views

Yes, this is another post on the internet about properly aligning your SSD partitions on Linux. It’s mostly my notes that I have gathered from other posts around the net. Please read the whole post before starting to create partitions on your SSD.

Intro
I bought myself a brand new SSD for Xmas, OCZ Agilty 3 120Gb. But I also bought a CDROM caddy so that I could replace my useless macbook CDROM drive, last time I used it was probably 2009 or 2010. So my plan was to put the old, original macbook SATA hard disk inside the caddy and use the SSD as the primary one. Sounds easy right ? Well you just need patience, lots of patience in order to remove all necessary screws in order to get the CDROM drive out and replace it with the caddy. Instructions for this procedure can be found at iFixIt.com.

Create Partitions on the SSD disk
Before one begins some definitions!

Heads = Tracks per cylinder
Sectors = Sectors per track

The goal here is to have the partitions aligned to the SSD’s Erase Block Size.
Googling around the net I found out that OCZ always uses 512Kb as Erase Block Size. If one uses fdisk with 32 Heads and 32 Sectors that makes a cylinder size of 1024b = 1Kb. Multiplying with 512 (sector size), which is fdisk’s default unit size, that makes it 512kb (=32*32*512)! Exactly the Erase Block Size that’s needed. So one needs to start fdisk issuing the following command:
# fdisk -H32 -S32 /dev/sdb
where /dev/sdb is the SSD.

It is very important to remember to start the first partition from the 2nd unit (or 512kb if you prefer). Due to MS-DOS compatibility if the first partition were to start at the first cylinder, it would skip one track. So it would actually start at 32(sectors)*512(sector size)=16Kb, messing up the alignment.

Then create necessary partitions as needed.

LVM alignment
So, the partitions on the SSD are aligned, but what if one wants to use LVM ? Then LVM’s overhead has to be taken into account as well.
To create an aligned PV based on the partitions that have already been created one needs to use the “–dataalignment” option found in newer versions of LVM utilities.
# pvcreate --dataalignment 512k /dev/sdb3
To check the alignment use the following command:

# pvs /dev/sdb3 -o+pe_start
  PV         VG   Fmt  Attr PSize   PFree  1st PE 
  /dev/sdb3  ssd  lvm2 a-   111.46g 81.46g 512.00k

Check that “1st PE” is what is actually needed for the alignment.

Proceed creating VGs and LVs as needed.

Formatting Partitions with ext4
There’s no reason to use ext3 on SSD, one needs to take advantage of ext4 SSD features. I prefer 4K as block size.
For a further explanation of the following formulas read Linux RAID Wiki – RAID setup
stride = chunk (Sector size) / block size = 512Kb / 4K = 128
stripe-width is usually calculated by a formula that uses multiple disks. Since there’s only one disk in this scenario, stripe-width is equal to stride.
stripe-width = 128

# mkfs.ext4 -O extent -b 4096 -E stride=128,stripe-width=128 /dev/mapper/ssd-debian

Mounting the partition
To enable SSD TRIM support, which protects the disk from wearing off, one needs to enable the discard option while mounting the partition. Edit /etc/fstab and add the discard mount option (and noatime if you want to).
/dev/mapper/ssd-deb / ext4 discard,noatime,errors=remount-ro 0 1

Note 1: As of 2.6.37 Linux Kernel supports TRIM on device mapper. Previous kernel versions will report errors upon trying to mount an LVM partition with discard mount option. If you have an older kernel either don’t use LVM on your SSD yet or upgrade your kernel!
Note 2: Read the links posted below for a complete blog post over TRIM command. Apparently it’s not always the best choice

That’s basically it…

Extra – copying the old root partition to the new disk

# mkdir /mnt/ssd/
# mount /dev/mapper/ssd-debian /mnt/ssd/
# rsync -aPEHv --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/mnt --exclude=/var/cache/apt/archives/ / /mnt/ssd/
# mkdir /mnt/ssd/dev
# mkdir /mnt/ssd/proc
# mkdir /mnt/ssd/sys
# mkdir /mnt/ssd/mnt
# cp -avp /dev/console /mnt/ssd/dev/
# cp -avp /dev/urandom /mnt/ssd/dev/
# cp -avp /dev/zero /mnt/ssd/dev/
# cp -avp /dev/random /mnt/ssd/dev/
# Edit /mnt/ssd/etc/fstab to change the device names
# Update grub

Using the above commands one avoids copying unneeded directories like /dev, /sys, etc that will be recreated later. Don’t forget to at least copy the above 4 devices in the new /mnt/ssd/dev dir, else the partition won’t be bootable.

References
1. Aligning Filesystems to an SSD’s Erase Block Size (link goes to archive.org since original article has unfortunately disappeared from the web)
2. [linux-lvm] Re: Aligning PVs on SSDs?
3. Aligning an SSD on Linux (excellent article!)
4. ArchWiki – Solid State Drives
5. SSD performance tips for RHEL6 and Fedora
6. Re: [dm-devel] trim support (discard)
7. How to check TRIM is working on your SSD running Linux
8. Impact of ext4′s discard option on my SSD (very useful insight on TRIM command, read the comments as well)

Thanks
Thanks fly to @apoikos for helping me with the CDROM replacement and @faidonl for his original SSD alignment tips 🙂

Filed by kargig at 21:49 under Linux
Tags: alignment, debian, erase block size, ext4, fdisk, Linux, LVM, macbook, partitions, rsync, SSD
13 Comments | 91,709 views

One of the worst things a webmaster or a anyone else that runs some web application can do, is to constantly send “informative newsletters” to people. Most CMS applications make it really easy to send such emails. These are 99% spam, and as such there are many good reasons that you should limit the amount of such outgoing “newsletters” coming out of your email server. Else there’s a good chance you might get added to a blacklist, and you don’t want your legitimate clients to have their emails blocked because of some irresponsible people. I recently had to deploy such a solution to a hosting server that serves multiple (>300) domains. The server already ran postfix, so I had to implement something useful around it.

The problem with postfix is that you can’t really rate-limit the outgoing queue per sender domain/address. There are only generic settings that control the general mail server’s capabilities of sending emails. What I wanted though is to have the ability to restrict specific domains to some specific email message count per day. This is something that a postfix addon named postfix-policyd can do by deferring/greylisting, but still just on the incoming queue. One would think that the problems would be solved by just applying this, but truth is that they don’t. Applying a defer/greylisting policy on the incoming queue is fine while the client on the remote side is another SMTP server that can happily store the deferred email on its queue and retry some minutes/hours later. What happens though if the SMTP client is a PHP application that connects through the mail() function ? There you have no queue and if you defer a message at the SMTP server it will get forever lost, PHP can’t resend it. So the solution would be to apply an intermediate SMTP queue between PHP and the primary SMTP server, that is another local postfix installation that would only serve as a queue that relays emails to the primary.

Using a “simple” diagram sending an email from PHP should follow this path upon a successful installation:

PHP mail() –(sendmail binary)–> intermediate_POSTFIX –(SMTP relay)–> POSTFIX –(smtpd_sender_restrictions)–> POLICYD –(pickup)–> POSTFIX –(SMTP)–> REMOTE SERVER

Here are the steps I took on a Debian Squeeze server to install this little monster.

1. Create a new postfix configuration directory for the new intermediate postfix instance
I named my intermediate postfix config dir as postfix2525, name comes from the port that it will listen on but you can definitely be more creative.

# mkdir /etc/postfix2525
# cp -av /etc/postfix /etc/postfix2525

Remove everything from /etc/postfix2525/main.cf and just add the following lines:

data_directory = /var/lib/postfix2525
queue_directory = /var/spool/postfix2525
relayhost = 127.0.0.1:12525

This defines a new data and queue directory and instructs this postfix to relay all emails through another one that listens on the localhost, the primary one, on port 12525. More about this port later when you will create some special config on the primary postfix.

Remove previous contents of /etc/postfix2525/master.cf and just add these lines:

127.0.0.1:2525      inet  n       -       -       -       2       smtpd
        -o syslog_name=postfix2525
pickup    fifo  n       -       -       60      1       pickup
cleanup   unix  n       -       -       -       0       cleanup
qmgr      fifo  n       -       n       300     1       qmgr
#qmgr     fifo  n       -       -       300     1       oqmgr
tlsmgr    unix  -       -       -       1000?   1       tlsmgr
rewrite   unix  -       -       -       -       -       trivial-rewrite
bounce    unix  -       -       -       -       0       bounce
defer     unix  -       -       -       -       0       bounce
trace     unix  -       -       -       -       0       bounce
verify    unix  -       -       -       -       1       verify
flush     unix  n       -       -       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       -       -       -       smtp
# When relaying mail as backup MX, disable fallback_relay to avoid MX loops
relay     unix  -       -       -       -       -       smtp
        -o smtp_fallback_relay=
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       -       -       -       showq
error     unix  -       -       -       -       -       error
retry     unix  -       -       -       -       -       error
discard   unix  -       -       -       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       -       -       -       lmtp
anvil     unix  -       -       -       -       1       anvil
scache    unix  -       -       -       -       1       scache

Obviously the most important part here is the first line. It defines that this postfix instance will listen for SMTP connections on localhost, port 2525 and it’s syslog output name will be postfix2525 so that it’s easier to tell apart which SMTP instance spits which errors.

After this is done you need to run the following command that will create all necessary directories with their proper permissions.

# postfix -c /etc/postfix2525/ check

Also make sure you add the following line to the main.cf file of your main postfix installation:
alternate_config_directories = /etc/postfix2525

You will also need a new init script. Since the script by itself is quite big and there are only a few lines that actually differ, I will post my diff here:

--- /etc/init.d/postfix  2011-05-04 21:17:47.000000000 +0200
+++ /etc/init.d/postfix2525  2011-12-19 19:22:09.000000000 +0100
@@ -17,8 +17,10 @@
 # Description:       postfix is a Mail Transport agent
 ### END INIT INFO
 
+CONFDIR=/etc/postfix2525
 PATH=/bin:/usr/bin:/sbin:/usr/sbin
 DAEMON=/usr/sbin/postfix
+DAEMON_OPTIONS="-c /etc/postfix2525"
 NAME=Postfix
 TZ=
 unset TZ
@@ -28,13 +30,13 @@
 
 test -f /etc/default/postfix && . /etc/default/postfix
 
-test -x $DAEMON && test -f /etc/postfix/main.cf || exit 0
+test -x $DAEMON && test -f /etc/postfix2525/main.cf || exit 0
 
 . /lib/lsb/init-functions
 #DISTRO=$(lsb_release -is 2>/dev/null || echo Debian)
 
 running() {
-    queue=$(postconf -h queue_directory 2>/dev/null || echo /var/spool/postfix)
+    queue=$(postconf -c $CONFDIR -h queue_directory 2>/dev/null || echo /var/spool/postfix2525)
     if [ -f ${queue}/pid/master.pid ]; then
   pid=$(sed 's/ //g' ${queue}/pid/master.pid)
   # what directory does the executable live in.  stupid prelink systems.
@@ -66,7 +68,7 @@
       fi
 
       # see if anything is running chrooted.
-      NEED_CHROOT=$(awk '/^[0-9a-z]/ && ($5 ~ "[-yY]") { print "y"; exit}' /etc/postfix/master.cf)
+      NEED_CHROOT=$(awk '/^[0-9a-z]/ && ($5 ~ "[-yY]") { print "y"; exit}' /etc/postfix2525/master.cf)
 
       if [ -n "$NEED_CHROOT" ] && [ -n "$SYNC_CHROOT" ]; then
     # Make sure that the chroot environment is set up correctly.
@@ -111,7 +113,7 @@
     umask $oldumask
       fi
 
-      if start-stop-daemon --start --exec ${DAEMON} -- quiet-quick-start; then
+      if start-stop-daemon --start --exec ${DAEMON} -- ${DAEMON_OPTIONS} quiet-quick-start; then
     log_end_msg 0
       else
     log_end_msg 1
@@ -123,7 +125,7 @@
   RUNNING=$(running)
   log_daemon_msg "Stopping Postfix Mail Transport Agent" postfix
   if [ -n "$RUNNING" ]; then
-      if ${DAEMON} quiet-stop; then
+      if ${DAEMON} ${DAEMON_OPTIONS} quiet-stop; then
     log_end_msg 0
       else
     log_end_msg 1

If everything went well up to now you should be able to start your new postfix instance and check that it is actually running.

# /etc/init.d/postfix2525 start
# netstat -antp | grep 2525
tcp        0      0 127.0.0.1:2525          0.0.0.0:*               LISTEN      6138/master

2. Configure main postfix to accept emails from the intermediate
Edit /etc/postfix/master.cf and add this line at the bottom:

127.0.0.1:12525 inet n - - - - smtpd  -o smtp_fallback_relay= -o smtpd_client_restrictions=  -o smtpd_helo_restrictions=  -o smtpd_recipient_restrictions=permit_mynetworks,reject  -o smtpd_data_restrictions=  -o receive_override_options=no_unknown_recipient_checks

This defines a special port for the main postfix instance that has (or maybe it hasn’t actually) some special restrictions.
Actually you will have to change this line later on upon installing postfix-policyd, but this should be good enough for now, in order for you to do some testing.
Restart postfix

# /etc/init.d/postfix restart
# netstat -antp | grep 2525
tcp        0      0 127.0.0.1:12525         0.0.0.0:*               LISTEN      26799/master    
tcp        0      0 127.0.0.1:2525          0.0.0.0:*               LISTEN      6138/master   

The intermediate postfix listens on 127.0.0.1:2525 and the main one has another special listening port on 127.0.0.1:12525.

3. Test your intermediate postfix instance
You can do this in a gazillion different ways. One of my favorite ways to test SMTP connectivity is through telnet (—> shows data entry):

# telnet localhost 2525
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 server.mydomain.gr ESMTP Postfix
---> EHLO koko.gr
250-server.mydomain.gr
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
---> MAIL FROM: lala@koko.gr
250 2.1.0 Ok
---> RCPT TO: koko@destination.gr
250 2.1.5 Ok
---> DATA
354 End data with <CR><LF>.<CR><LF>
---> THIS IS A TEST
---> .
250 2.0.0 Ok: queued as C41E21C84FF
---> quit

If you were keeping an eye on syslog messages you should have seen some connection messages both from postfix2525 and from postfix. If everything went well your email _should_ have arrived at it’s destination. If this is true then your primary postfix instance now works as a relay for your intermediate queue.

Don’t read the next parts of this post if you haven’t previously managed this step!

4. Install and configure postfix-policyd

# aptitude install postfix-policyd

To run policyd you need to create a database and import policyd SQL schema to it. Your distro has probably already taken care of the previous step, if it hasn’t…do it manually and think about changing distro!
Then edit the config file usually located at /etc/postfix-policyd.conf. The options I chose to play with were the following:

SENDERTHROTTLE=1
SENDER_THROTTLE_SASL=1
SENDER_THROTTLE_HOST=0

Since all emails will be relayed through localhost there’s no point in throttling per host, what is needed is throttling per envelope sender.
You should manually review your desired limits though. I won’t post mine here because everyone has different needs and there’s no sane config for everyone.

Start postfix-policyd
# /etc/init.d/postfix-policyd start

If you get weird startup errors like:
postfix-policyd: fatal: didn't find priority 'LOG_IFOO', exiting
Edit /etc/postfix-policyd.conf, find the following line:
SYSLOG_FACILITY="LOG_MAIL | LOG_INFO"
and change it to (mind the removed spaces):
SYSLOG_FACILITY="LOG_MAIL|LOG_INFO"

5. Configure main postfix instance to use postifix-policyd
Edit /etc/postfix/main.cf and add this:
webclient_restrictions = check_policy_service inet:127.0.0.1:10031

Then edit /etc/postfix/master.cf again and change the line you had previously added to the bottom of the file with this:

127.0.0.1:12525 inet n - - - - smtpd  -o smtp_fallback_relay= -o smtpd_client_restrictions=  -o smtpd_helo_restrictions=  -o smtpd_recipient_restrictions=permit_mynetworks,reject  -o smtpd_data_restrictions=  -o receive_override_options=no_unknown_recipient_checks -o smtpd_sender_restrictions=${webclient_restrictions}

The difference is
-o smtpd_sender_restrictions=${webclient_restrictions}
which practically instructs postfix to use postfix-policyd for emails that arrive on port 12525, which is the port that the intermediate postfix instance uses to relay all emails.

6. Test your intermediate postfix instance again
If everything went well, the main postfix instance should now be able to enforce sender policies. Try sending a new email through the intermediate postfix again, yes using telnet, and you should pickup some new log lines at your syslog:

Dec 19 21:56:40 myserver postfix-policyd: connection from: 127.0.0.1 port: 45635 slots: 0 of 4096 used
Dec 19 21:56:40 myserver postfix-policyd: rcpt=5, greylist=new, host=127.0.0.1 (unknown), from=lala@koko.gr, to=koko@lalala.gr, size=348
Dec 19 21:56:40 myserver postfix/smtpd[9168]: NOQUEUE: reject: RCPT from unknown[127.0.0.1]: 450 4.7.1 : Sender address rejected: Policy Rejection- Please try later.; from= to= proto=ESMTP helo=
Dec 19 21:56:40 myserver postfix/smtp[8970]: C41E21C84FF: to=, relay=127.0.0.1[127.0.0.1]:12525, delay=20, delays=20/0/0.01/0, dsn=4.7.1, status=deferred (host 127.0.0.1[127.0.0.1] said: 450 4.7.1 : Sender address rejected: Policy Rejection- Please try later. (in reply to RCPT TO command))

The above means that greylisting through policyd works.

7. make PHP use your new intermediate postfix instance
PHP on linux by default uses the sendmail binary to send emails via the mail() function. That would use the main postfix instance though, so one needs to edit /etc/php/apache2/php.ini and change the following line:
sendmail_path = "sendmail -C /etc/postfix2525 -t -i"

The -C directive instructs sendmail to use the alternate config dir, so that emails will be sent to the new intermediate postfix instance and then to the main one, passing through policyd of course.

To check the queue size of the intermediate postfix:
# postqueue -p -c /etc/postfix2525/

If any PHP applications that are hosted have explicit SMTP server/port directives, then be sure to notify your clients/developers that they _MUST_ use localhost:2525 to send their emails to and not the default localhost:25. This is one of the shortcomings of the above method, if someone manually sets up his application to use the default localhost:25 his emails will get right through. But being a good sysadmin, you should monitor such behavior and punish those users accordingly!

That’s about it…with the above configuration and some tweaking to the thresholds you have very good chances of avoiding getting blacklisted because someone decided to send a few thousand spams emails. And most importantly, your normal mail service will continue to work flawlessly, no matter how big the queue of the intermediate mail server is.

Enjoy!

Reference for policyd: http://policyd.sourceforge.net/readme.html

Filed by kargig at 23:53 under Internet,Linux
Tags: antispam, blacklist, debian, email, Linux, mail, php, policyd, postfix, postfix-policyd, relay, sendmail, smtp, spam
12 Comments | 81,746 views