Firejail with Tor HOWTO

A few years ago I created a set of scripts to start applications inside a linux namespace and automatically “Tor-ify” their network traffic. The main reason behind this effort was to provide some isolation and Tor support for applications that don’t have socks5 support, for example claws-mail. While this worked it was hard to keep adding sandboxing features like the ones firejail already provided. So I decided to take a look at how I could automatically send/receive traffic from a firejail-ed application through Tor.

This blog post is NOT meant to be used as copy/paste commands but to explain why each step is needed and how to overcome the problems found in the path.
If you have reasons to proxy all your traffic through Tor as securely as possible use Tails on a different machine, this guide is NOT for you.

A dedicated bridge
First of all create a Linux bridge and assign an IP address to it. Use this bridge to attach the veth interfaces that firejail creates when using the ‘net’ option. This option creates a new network namespace for each sandboxed application.

# brctl addbr tornet
# ip link set dev tornet up
# ip addr add 10.100.100.1/24 dev tornet

NAT
Then enable NAT from/to your “external” interface (eno1 in my case) for tcp connections and udp port 53 (DNS) and enable IP(v4) forwarding, if you don’t already use it. Some rules about sane default policy for FORWARD chain are added here well, modify to your needs.

# sysctl -w net.ipv4.conf.all.forwarding=1
# iptables -P FORWARD DROP
# iptables -A INPUT -m state --state INVALID -j DROP
# iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# iptables -A FORWARD -i tornet -o eno1 -p tcp -j ACCEPT
# iptables -A FORWARD -i tornet -o eno1 -p udp --dport=53 -j ACCEPT
# iptables -t nat -A POSTROUTING -s 10.100.100.0/24 -o eno1 -j MASQUERADE

This configuration is enough to start a sandboxed application that will have it’s traffic NAT-ed from the Linux host.

$ firejail --net=tornet /bin/bash
Parent pid 26730, child pid 26731

Interface        MAC                IP               Mask             Status
lo                                  127.0.0.1        255.0.0.0        UP    
eth0             72:cc:f6:d8:6a:09  10.100.100.29    255.255.255.0    UP    
Default gateway 10.100.100.1

$ host www.debian.org
www.debian.org has address 5.153.231.4
Host www.debian.org not found: 3(NXDOMAIN)
Host www.debian.org not found: 4(NOTIMP)
$ host whoami.akamai.net
whoami.akamai.net has address 83.235.72.202
$ curl wtfismyip.com/text
3.4.5.6

(where 3.4.5.6 is your real IP and 83.235.72.202 should be the IP address of the final DNS recursive resolver requesting information from whois.akamai.net)

So NAT works and the shell is sandboxed.

“Tor-ify” traffic
Edit /etc/tor/torrc and enable TransPort and VirtualAddrNetwork Tor features to transparently proxy to the Tor network connections landing on Tor daemon’s port 9040. DNSPort is used to resolve DNS queries through the Tor network. You don’t have to use IsolateDestAddr for your setup, but I like it.

TransPort 9040
VirtualAddrNetwork 172.30.0.0/16
DNSPort 5353 IsolateDestAddr

Then use iptables to redirect traffic from tornet bridge to TransPort and DNSPort specified in torrc. You also need to ACCEPT that traffic in your INPUT chain if your policy is DROP (it is right ?)

# iptables -t nat -A PREROUTING -i tornet -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.1:5353
# iptables -t nat -A PREROUTING -i tornet -p tcp -j DNAT --to-destination 127.0.0.1:9040
# iptables -A INPUT -i tornet -p tcp --dport 9040 -j ACCEPT
# iptables -A INPUT -i tornet -p udp --dport 5353 -j ACCEPT

Run your sandbox again and try to access the same website:

$ firejail --net=tornet /bin/bash
$ curl wtfismyip.com/text
curl: (7) Failed to connect to wtfismyip.com port 80: Connection timed out

aaaand nothing happens. The problem is that you have tried to route traffic from a “normal” interface to loopback which is considered a “martian” and is not allowed by default by the Linux kernel.

sysctl magic
To enable loopback to be used for routing the route_localnet sysctl setting must be set.
# sysctl -w net.ipv4.conf.tornet.route_localnet=1

Try again:

$ firejail --net=tornet /bin/bash
$ host whoami.akamai.net
whoami.akamai.net has address 74.125.181.10
Host whoami.akamai.net not found: 3(NXDOMAIN)
Host whoami.akamai.net not found: 4(NOTIMP)
$ curl wtfismyip.com/text
176.10.104.243
$ host 176.10.104.243
243.104.10.176.in-addr.arpa domain name pointer tor2e1.digitale-gesellschaft.ch.

it works!

You can actually run any program you want like that:
$ firejail --net=tornet google-chrome

Accessing onion services
There’s one problem left though, accessing onion services.
If you try and access www.debian.org onion service from your firejail+tor setup you will get an error.

$ firejail --net=tornet /bin/bash
$ curl http://sejnfjrq6szgca7v.onion/
curl: (6) Could not resolve host: sejnfjrq6szgca7v.onion

To fix that you need to modify /etc/tor/torrc again and add AutomapHostsOnResolve option.
AutomapHostsOnResolve 1

$ firejail --net=tornet /bin/bash
$ curl -I http://sejnfjrq6szgca7v.onion/
HTTP/1.1 200 OK
Date: Fri, 09 Dec 2016 12:05:56 GMT
Server: Apache
Content-Location: index.en.html
Vary: negotiate,accept-language,Accept-Encoding
TCN: choice
Last-Modified: Thu, 08 Dec 2016 15:42:34 GMT
ETag: "3a40-543277c74dd5b"
Accept-Ranges: bytes
Content-Length: 14912
Cache-Control: max-age=86400
Expires: Sat, 10 Dec 2016 12:05:56 GMT
X-Clacks-Overhead: GNU Terry Pratchett
Content-Type: text/html
Content-Language: en

Accessing onion services works as well now.

Applications supporting socks5
If you already have some of your applications proxying connections to tor using 127.0.0.1:9050 then you need to add another iptables rule to redirect the socks traffic from inside firejail’s namespace to Tor SocksPort.
# iptables -t nat -A PREROUTING -i tornet -p tcp -m tcp --dport 9050 -j DNAT --to-destination 127.0.0.1:9050

keys.void.gr – A GPG Keyserver in Greece

After some months of entertaining the idea of setting up a public gpg keyserver I finally managed to find some time and do it this weekend.

Habemus keys.void.gr Keyserver!

Some history
The first time I set up a gpg keyserver was 3 years ago. Its purpose was to make it possible for a researcher to get more results than the default on a single query from a keyserver. Using that keyserver the Greek PGP Web of Trust 2012 edition was created. After the original import of the keys, I refreshed the keys just 2 or 3 times in the following years.

The setup
The keyserver is running on Debian Linux with SKS version 1.1.5. Port 80 and 443 are being handled by nginx which acts as a reverse proxy for SKS. I originally had port 11371, the default port that gpg client uses, behind nginx as well but I had to remove it due to the following issue. I like using HSTS header for the HTTPS port, but browsers trying to access http://keys.void.gr:11371, were switching to https://keys.void.gr:11371 (because of HSTS) which couldn’t work because port 11371 does not use TLS. So once a browser visited https://keys.void.gr and got the HSTS header, every future connection towards http://keys.void.gr:11371 would fail. The solution was to use a protocol multiplexer called sslh. What this does, is that it sniffs the connections coming towards port 11371 and if it finds a TLS connection, it sends it to port 443, if it finds an HTTP connection it sends it to port 80. That way you can either visit http://keys.void.gr:11371 or https://keys.void.gr:11371 and they both work.

For ports 80,443 the connection path looks like: client -> nginx -> sks
For port 11317 the connection path looks like: client -> sslh -> nginx -> sks

keys.void.gr is available in both IPv4 and IPv6.

I’ve also setup an onion/hidden service for the keyserver, so if you prefer visiting the onion address, here it is: wooprzddebtxfhnq.onion (available on port 11371 as well).

Difficulties
I’m not sure if it’s the Debian package’s fault or I did something stupid, but if you plan on running your own keyserver be very careful with permissions on the your filesystem. sks errors are not very friendly. Make sure that /var/spool/sks, /var/lib/sks and /var/log/sks are all owned by debian-sks:debian-sks.
# chown -R debian-sks:debian-sks /var/spool/sks /var/lib/sks /var/log/sks
Don’t run the DB building script as root, run it as debian-sks user:
# sudo -u debian-sks /usr/lib/sks/sks_build.sh
There are a quite some tunables referenced in the sks man page regarding pagesizes, I went with the default options for now.

The pool
To enter the pool of keyservers and start interacting with other keyservers you have to join the sks-devel mailing list and announce your server existence by sending your “membership line” which looks like this:
keys.void.gr 11370 # George K. <keyserver [don't spam me] void [a dot goes here] gr> #0x721006E470459C9C

If people place this line in their membership config file and you place theirs, then the keyservers start communicating, or “gossiping” as it is called in the sks language. It needs to be mutual.

Because of the minimal traffic I was seeing on the mailing list archives I thought that finding peers would take weeks, if not months. I was very very wrong. I got 6 replies to my email in less than 2 hours. Impressive. Thanks a lot people!

UI
I’ve taken the boostrap-ed HTML from https://github.com/mattrude/pgpkeyserver-lite.

TODO
hkps support will be added in the following days or weeks.

Stats
keys.void.gr Keyserver statistics
sks-keyservers.net pool Status for keys.void.gr

Enjoy!

Creating a new GPG key with subkeys

A few weeks ago I created my new GPG/PGP key with subkeys and a few people asked me why and how. The rationale for creating separate subkeys for signing and encryption is written very nicely in the subkeys page of the debian wiki. The short answer is that having separate subkeys makes key management a lot easier and protects you in certain occasions, for example you can create a new subkey when you need to travel or when your laptop gets stolen, without losing previous signatures. Obviously you need to keep your master key somewhere very very safe and certainly not online or attached to a computer.

You can find many other blog posts on the net on the subject, but most of them are missing a few parts. I’ll try to keep this post as complete as possible. If you are to use gpg subkeys you definitely need an encrypted usb to store the master key at the end. So if you don’t already have an encrypted USB go and make one first.

When this process is over you will have a gpg keypair on your laptop without the master key, you will be able to use that for everyday encryption and signing of documents but there’s a catch. You won’t be able to sign other people’s keys. To do that you will need the master key. But that is something that does not happen very often so it should not be a problem in your everyday gpg workflow. You can read about signing other people’s keys at the end of this post. AFAIK you can’t remove your master key using some of the gpg GUIs, so your only hope is the command line. Live with it…

First some basic information that will be needed later.
When listing secret keys with gpg -K keys are marked with either ‘sec’ or ‘ssb’. When listing (public) keys with gpg -k keys are marked with ‘pub’ or ‘sub’.

sec => 'SECret key'
ssb => 'Secret SuBkey'
pub => 'PUBlic key'
sub => 'public SUBkey'

When editing a key you will see a usage flag on the right. Each key has a role and that is represented by a character. These are the roles and their corresponding characters:

Constant           Character      Explanation
─────────────────────────────────────────────────────
PUBKEY_USAGE_SIG      S       key is good for signing
PUBKEY_USAGE_CERT     C       key is good for certifying other signatures
PUBKEY_USAGE_ENC      E       key is good for encryption
PUBKEY_USAGE_AUTH     A       key is good for authentication

Before doing anything make sure you have a backup of your .gnupg dir.
$ umask 077; tar -cf $HOME/gnupg-backup.tar -C $HOME .gnupg

Secure preferences
Now edit your .gnupg/gpg.conf and add or change the following settings (most are stolen from Riseup: OpenPGP Best Practices):

# when outputting certificates, view user IDs distinctly from keys:
fixed-list-mode
# long keyids are more collision-resistant than short keyids (it's trivial to make a key with any desired short keyid)
keyid-format 0xlong
# when multiple digests are supported by all recipients, choose the strongest one:
personal-digest-preferences SHA512 SHA384 SHA256 SHA224
# preferences chosen for new keys should prioritize stronger algorithms:
default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 BZIP2 ZLIB ZIP Uncompressed
# If you use a graphical environment (and even if you don't) you should be using an agent:
# (similar arguments as https://www.debian-administration.org/users/dkg/weblog/64)
use-agent
# You should always know at a glance which User IDs gpg thinks are legitimately bound to the keys in your keyring:
verify-options show-uid-validity
list-options show-uid-validity
# when making an OpenPGP certification, use a stronger digest than the default SHA1:
cert-digest-algo SHA256
# prevent version string from appearing in your signatures/public keys
no-emit-version 

Create new key
Time to create the new key. I’m marking user input with bold (↞) arrows

$ gpg --gen-key
gpg (GnuPG) 1.4.12; Copyright (C) 2012 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection?
Your selection?  1 ↞↞↞↞ 
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048)  4096 ↞↞↞↞ 
Requested keysize is 4096 bits
Please specify how long the key should be valid.
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0)  0 ↞↞↞↞ 
Key does not expire at all
Is this correct? (y/N)  y ↞↞↞↞ 
You need a user ID to identify your key; the software constructs the user ID
from the Real Name, Comment and Email Address in this form:
    "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>"
Real name: foo bar ↞↞↞↞ 
Email address: foobar@void.gr ↞↞↞↞ 
Comment:
You selected this USER-ID:
"foo bar <foobar@void.gr>"

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit?  O ↞↞↞↞ 
You need a Passphrase to protect your secret key.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
.............+++++
..+++++

gpg: key 0x6F87F32E2234961E marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   3  signed:  14  trust: 0-, 0q, 0n, 0m, 0f, 3u
gpg: depth: 1  valid:  14  signed:   9  trust: 13-, 1q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2014-03-18
pub   4096R/0x6F87F32E2234961E 2013-12-01
      Key fingerprint = 407E 45F0 D914 8277 3D28  CDD8 6F87 F32E 2234 961E
uid                 [ultimate] foo bar <foobar@void.gr>
sub   4096R/0xD3DCB1F51C37970B 2013-12-01

Optionally, you can add another uid and add it as the default:

$ gpg --edit-key 0x6F87F32E2234961E                                      
gpg (GnuPG) 1.4.12; Copyright (C) 2012 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

pub  4096R/0x6F87F32E2234961E  created: 2013-12-01  expires: never       usage: SC  
                               trust: ultimate      validity: ultimate
sub  4096R/0xD3DCB1F51C37970B  created: 2013-12-01  expires: never       usage: E   
[ultimate] (1). foo bar <foobar@void.gr>
gpg> adduid ↞↞↞↞ 
Real name: foo bar ↞↞↞↞ 
Email address: foobar@riseup.net ↞↞↞↞ 
Comment: 
You selected this USER-ID:
    "foo bar <foobar@riseup.net>"
Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit?  O ↞↞↞↞ 
You need a passphrase to unlock the secret key for
user: "foo bar <foobar@void.gr>"
4096-bit RSA key, ID 0x6F87F32E2234961E, created 2013-12-01

pub  4096R/0x6F87F32E2234961E  created: 2013-12-01  expires: never       usage: SC  
                               trust: ultimate      validity: ultimate
sub  4096R/0xD3DCB1F51C37970B  created: 2013-12-01  expires: never       usage: E   
[ultimate] (1)  foo bar <foobar@void.gr>
[ unknown] (2). foo bar <foobar@riseup.net>
gpg> uid 2 ↞↞↞↞ 
gpg> primary ↞↞↞↞ 
gpg> save ↞↞↞↞ 

Let’s see what we’ve got until now, 0x6F87F32E2234961E is the master key (SC flags) and 0xD3DCB1F51C37970B (E flag)is a separate subkey for encryption.

Add new signing subkey
Since we already have a separate encryption subkey, it’s time for a new signing subkey. Expiration dates for keys is a very hot topic. IMHO there’s no point in having an encryption subkey with an expiration date, expired keys are working just fine for decryption anyways, so I’ll leave it without one, but I want the signing key that I’m regularly using to have an expiration date. You can read more about this topic on the gnupg manual (Selecting expiration dates and using subkeys).

$ gpg --edit-key 0x6F87F32E2234961E
gpg (GnuPG) 1.4.12; Copyright (C) 2012 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

pub  4096R/0x6F87F32E2234961E  created: 2013-12-01  expires: never       usage: SC  
                               trust: ultimate      validity: ultimate
sub  4096R/0xD3DCB1F51C37970B  created: 2013-12-01  expires: never       usage: E   
[ultimate] (1). foo bar <foobar@riseup.net>
[ultimate] (2)  foo bar <foobar@void.gr>
gpg> addkey ↞↞↞↞ 

Key is protected.

You need a passphrase to unlock the secret key for
user: “foo bar <foobar@riseup.net>”
4096-bit RSA key, ID 0x6F87F32E2234961E, created 2013-12-01

Please select what kind of key you want:
(3) DSA (sign only)
(4) RSA (sign only)
(5) Elgamal (encrypt only)
(6) RSA (encrypt only)

Your selection? 4 ↞↞↞↞ 
RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096 ↞↞↞↞ 
Requested keysize is 4096 bits
Please specify how long the key should be valid.
         0 = key does not expire
        = key expires in n days
      w = key expires in n weeks
      m = key expires in n months
      y = key expires in n years
Key is valid for? (0) 5y ↞↞↞↞ 
Key expires at Fri 30 Nov 2018 03:36:47 PM EET
Is this correct? (y/N) y ↞↞↞↞ 
Really create? (y/N) y ↞↞↞↞ 
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
+++++
...............+++++

pub  4096R/0x6F87F32E2234961E  created: 2013-12-01  expires: never       usage: SC  
                               trust: ultimate      validity: ultimate
sub  4096R/0xD3DCB1F51C37970B  created: 2013-12-01  expires: never       usage: E   
sub  4096R/0x296B12D067F65B03  created: 2013-12-01  expires: 2018-11-30  usage: S   
[ultimate] (1). foo bar <foobar@riseup.net>
[ultimate] (2)  foo bar <foobar@void.gr>
gpg> save ↞↞↞↞ 

As you can see there’s a new subkey 0x296B12D067F65B03 with just the S flag, that the signing subkey.
Before moving forward it’s wise to create a revocation certificate:

$ gpg --output 0x6F87F32E2234961E.gpg-revocation-certificate --armor --gen-revoke 0x6F87F32E2234961E
sec  4096R/0x6F87F32E2234961E 2013-12-01 foo bar <foobar@riseup.net>

Create a revocation certificate for this key? (y/N) y
Please select the reason for the revocation:
  0 = No reason specified
  1 = Key has been compromised
  2 = Key is superseded
  3 = Key is no longer used
  Q = Cancel
(Probably you want to select 1 here)
Your decision?  1 ↞↞↞↞ 
Enter an optional description; end it with an empty line:
> This revocation certificate was generated when the key was created. ↞↞↞↞ 
> 
Reason for revocation: Key has been compromised
This revocation certificate was generated when the key was created.
Is this okay? (y/N) y ↞↞↞↞ 
You need a passphrase to unlock the secret key for
user: "foo bar <foobar@riseup.net>"
4096-bit RSA key, ID 0x6F87F32E2234961E, created 2013-12-01

Revocation certificate created.

Please move it to a medium which you can hide away; if Mallory gets
access to this certificate he can use it to make your key unusable.
It is smart to print this certificate and store it away, just in case
your media become unreadable.  But have some caution:  The print system of
your machine might store the data and make it available to others!

Encrypt this file and store it someplace safe, eg your encrypted USB. You should definitely not leave it at your laptop’s hard disk. You can even print it and keep it in this form, it’s small enough so one could type it if needed.

Remove Master key
And now the interesting part, it’s time to remove the master key from your laptops’s keychain and just leave the subkeys. You will store the master key in the encrypted usb so it stays safe.

First go and backup your .gnupg dir on your encrypted USB. Don’t move forward until you do that. DON’T!

$ rsync -avp $HOME/.gnupg /media/encrypted-usb
or
$ umask 077; tar -cf /media/encrypted-usb/gnupg-backup-new.tar -C $HOME .gnupg

Did you backup your key? Are you sure ?

Then it’s time to remove the master key!

$ gpg --export-secret-subkeys 0x6F87F32E2234961E > /media/encrypted-usb/subkeys
$ gpg --delete-secret-key 0x6F87F32E2234961E
$ gpg --import /media/encrypted-usb/subkeys
$ shred -u /media/encrypted-usb/subkeys

What you’ve accomplished with this process is export the subkeys to /media/encrypted-usb/subkeys then delete the master key and re-import just the subkeys. Master key resides only on the encrypted USB key now. Don’t lose that USB key. USB keys are extremely cheap, make multiple copies of the encrypted key and place them in safe places, you can give one such key to your parents or your closest friend in case of emergency. For safety, make sure there’s at least one copy outside of your residence.

You can see the difference of the deleted master key by comparing the listing of the secret keys in your .gnupg and your /media/encrypted-usb/.gnupg/ dir.

$ gpg -K 0x6F87F32E2234961E                                             
sec#   4096R/0x6F87F32E2234961E 2013-12-01
uid                            foo bar <foobar@riseup.net>
uid                            foo bar <foobar@void.gr>
ssb   4096R/0xD3DCB1F51C37970B 2013-12-01
ssb   4096R/0x296B12D067F65B03 2013-12-01 [expires: 2018-11-30] 
$ gpg --home=/media/encrypted-usb/.gnupg/ -K 0x6F87F32E2234961E                                             
sec   4096R/0x6F87F32E2234961E 2013-12-01
uid                            foo bar <foobar@riseup.net>
uid                            foo bar <foobar@void.gr>
ssb   4096R/0xD3DCB1F51C37970B 2013-12-01
ssb   4096R/0x296B12D067F65B03 2013-12-01 [expires: 2018-11-30] 

Notice the pound (#) in the ‘sec’ line from your ~/.gnupg/. That means that the master key is missing.

Upload your new key to the keyservers if you want to…

Key Migration
In case you’re migrating from an older key you need to sign your new key with the old one (not the other way around!)
$ gpg --default-key 0xOLD_KEY --sign-key 0x6F87F32E2234961E

Write a transition statement and sign it with both the old and the new key:

$ gpg --armor -b -u 0xOLD_KEY -o sig1.txt gpg-transition.txt
$ gpg --armor -b -u 0x6F87F32E2234961E -o sig2.txt gpg-transition.txt

That’s about it…upload the transition statement and your signatures to some public space (or mail it to your web of trust).

Signing other people’s keys
Because your laptop’s keypair does not have the master key anymore and the master key is the only one with the ‘C’ flag, when you want to sign someone else’s key, you will need to mount your encrypted USB and then issue a command that’s using that encrypted directory:
$ gpg --home=/media/encrypted-usb/.gnupg/ --sign-key 0xSomeones_keyid
Export your signature and send it back to people whose key you just signed..

Things to play with in the future
Next stop ? An OpenPGP Smartcard! (eshop) or a yubikey NEO, (related blogpost). Any Greeks want to join me for a mass (5+) order?

References
https://wiki.debian.org/subkeys
https://help.riseup.net/en/security/message-security/openpgp/best-practices
https://alexcabal.com/creating-the-perfect-gpg-keypair/

P.S. 0x6F87F32E2234961E is obviously just a demo key. You can find my real key here.
P.S.2 The above commands were executed on gpg 1.4.12 on Debian Wheezy. In the future the output of the commands will probably differ.

Anonymize headers in postfix

E-mail headers usually leak some information about the person sending the email. Most servers reveal the sender’s originating IP, but sometimes we might not want this behavior. Here’s a simple way to modify your postfix server to remove just the IP of the sender. The original idea is from https://we.riseup.net/debian/mail but with postfix 2.9 version (Debian Wheezy) using the way proposed in the riseup article you will also be anonymizing all intermediate ‘Received: from’ headers and not just the sender’s. The setup proposed by riseup article seems to work fine with postfix 2.7 (Debian Squeeze).

1. Install postfix-pcre if you haven’t already.
# apt-get install postfix-pcre


2.
Create a file /etc/postfix/smtp_header_checks with content:
/^\s*(Received: from)[^\n]*(.*)/ REPLACE $1 [127.0.0.1] (localhost [127.0.0.1])$2


3.
Edit /etc/postfix/master.cf
Find the section about submission and add at the end of it: -o cleanup_service_name=subcleanup
e.g.

submission inet n       -       -       -       -       smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

submission inet n       -       -       -       -       smtpd
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING
  -o cleanup_service_name=subcleanup

Then at the end of /etc/postfix/master.cf file add the following:

subcleanup unix n       -       -       -       0       cleanup
    -o header_checks=pcre:/etc/postfix/smtp_header_checks

That’s it, reload your postfix and you’re done. When you’ll be sending emails over submission (you do use submission instead of smtp to send your emails, right?) then the first ‘Received’ header will be modified like the following example.
Instead of:

Received: from foo.bar (abcd.efgh.domain.tld [111.222.100.200])
        by mail.domain.tld (Postfix) with ESMTPA id BAB8A1A0224
        for <user@dst.domain2.tld>; Sun, 24 Nov 2013 15:47:50 +0100 (CET)

It will be:

Received: from [127.0.0.1] (localhost [127.0.0.1])
        by mail.domain.tld (Postfix) with ESMTPA id BAB8A1A0224
        for <user@dst.domain2.tld>; Sun, 24 Nov 2013 15:47:50 +0100 (CET)

Extra
If you want to anonymize even more headers, try adding the following to /etc/postfix/smtp_header_checks

/^\s*User-Agent/        IGNORE
/^\s*X-Enigmail/        IGNORE
/^\s*X-Mailer/          IGNORE
/^\s*X-Originating-IP/  IGNORE

Logging
As the riseup article says, be very careful of what is being logged at the server. If you don’t want to log the replacements done by pcre then add something like the following in your rsyslog.conf before any other rule:
:msg, contains, "replace: header Received:" ~

Another day another hacked website

Yesterday morning, phone rings to notify my of a new sms. Someone could not access his website on some server that I am root/administer.
I tried to ping the server and got 1 reply every 10-15 packets so my initial thought was that the hosting provider had fucked up. I pinged other machines in the “neighborhood”, they replied just fine. So the problem lied in my server. I got console access through IPMI, you know…the ones with the cipher zero bug, and I managed to login. An apache2 process was constantly using 100% of a core and the machine sent gazillion packets towards a certain destination.

Since I wanted to investigate what exactly this process did, I put an iptables entry in my OUTPUT chain to block packets towards that destination. The machine became responsive again, though the apache process still ran at 100%. Since I run my vhosts using apache2 mpm_itk module, I knew through the apache2 PIDs’ username which site had been hacked. I grepped the logs for any POST, but I couldn’t see anything. Unfortunately the logs only go back 2 days (NOT my policy! and a very bad one actually…but anyway).

strace -p PID did not yield anything interesting, just the process trying to create sockets to send packets towards the destination.

socket(PF_NETLINK, SOCK_RAW, 0) = 417
bind(417, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 0
getsockname(417, {sa_family=AF_NETLINK, pid=11398, groups=00000000}, [12]) = 0
sendto(417, "\24\0\0\0\26\0\1\3\233\323\354Q\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
recvmsg(417, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"0\0\0\0\24\0\2\0\233\323\354Q\206,\0\0\2\10\200\376\1\0\0\0\10\0\1\0\177\0\0\1"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 588
recvmsg(417, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"@\0\0\0\24\0\2\0\233\323\354Q\206,\0\0\n\200\200\376\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 128
recvmsg(417, {msg_name(12)={sa_family=AF_NETLINK, pid=0, groups=00000000}, msg_iov(1)=[{"\24\0\0\0\3\0\2\0\233\323\354Q\206,\0\0\0\0\0\0\1\0\0\0\24\0\1\0\0\0\0\0"..., 4096}], msg_controllen=0, msg_flags=0}, 0) = 20
close(417) = 0
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 417
fcntl(417, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(417, F_SETFL, O_RDWR|O_NONBLOCK) = 0
connect(417, {sa_family=AF_INET, sin_port=htons(4883), sin_addr=inet_addr("X.Y.Z.W")}, 16) = 0
fcntl(417, F_SETFL, O_RDWR) = 0
sendto(417, "\207\25\312P\322t\0#\317}jf\2(W\374\375\232h\213\220\31\355\277)\320[\255\273\276\221\374"..., 8192, MSG_DONTWAIT, NULL, 0) = -1 EPERM (Operation not permitted)
close(417) = 0

lsof -n -p PID output had hundreds of open log files and a few connections. Grepping out the logs I noticed one that was quite interesting, it went towards another server at port 5555.
apache2 11398 XXXXXXX 416u IPv4 831501972 0t0 TCP A.B.C.D:59210->B.C.D.E:5555 (CLOSE_WAIT)

I run tcpdump there, and of course it was an irc connection. I started capturing everything.

lsof also revealed this:
apache2 11398 XXXXXXX cwd DIR 8,7 4096 2474373 /var/www/vhosts/XXXXXXX/httpdocs/libraries/phpgacl
which I could have have also seen it doing ls /proc/PID/cwd/ …but anyway.

Looking inside that dir I found a file named gacl_db.php. It was base64 encoded. Well actually it was multiple times base64 encoded and obfsuscated by using character substitutions, so I had to de-obfuscate it. It was quite easy using php and some bash scripting.

This is the original base64 encoded/obfuscated file: Original gacl_db.php
This is the final result: Deobfuscated gacl_db.php
(I have removed the irc server details from the deobfuscated file, it’s still there in the original file for whoever wants it though)

It’s just an IRC bot containing a perl reverse shell as well. It has commands to flood other servers, and that’s what my server was doing.

I joined the IRC server and at that time there were more than 90 bots inside. Right now that I’m writing this blog post there are less than 50. Every bot joining the channel outputs a text like this:

[uname!]: FreeBSD a.b.c.d 8.1-RELEASE-p5 FreeBSD 8.1-RELEASE-p5 #10: Fri Sep 30 14:45:56 MSK 2011 root@a.b.c.d:/path/to/to/to/sth pl#27 amd64 (safe: off)
[vuln!]: http://www.a-vhost-name.TLD/libraries/phpgacl/gacl_db.php
[uname!]: Linux x.y.z.w 3.2.0-43-generic #68-Ubuntu SMP Wed May 15 03:33:33 UTC 2013 x86_64 (safe: off)
[vuln!]: http://www.another-vhost-name.another-TLD/libraries/phpgacl/gacl_db.php

So if you run servers or websites, do a locate gacl_db.php.

Since all the bot/servers entering post a [vuln!] message about phpgacl, my guess is that the original vulnerability that allowed the attacker to gain access is right there. I haven’t had time to look into it yet, but I’ve warned my clients to remove this library from their websites as a precaution. You should probably do the same.

World city map of Tor nodes

Some months ago I started playing with the idea of creating a world map that would have every Tor node on it. Obviously I wan’t the first one…I soon discovered Moritz Bartl’s post on the same topic. Luckilly he had his code posted on Github so I could fork it and add features that I wanted. The original python script parsed the consensus and the misrodescriptors, put Tor nodes into some classes and created a KML file with some description on each node.

Some differences
I changed some parts of the python script to better suit my needs.
a. Create a separate kml files for each Tor node class.
b. Add new classes: Bad, Authority and Named.
c. Pay more attention on requesting every external URL over HTTPS.
d. Generate HTML code that displays those KMLs on a Google Maps overlay.
e. Add some small randomization to each nodes’s coordinates so that nodes in the same city don’t overlap.

You can find a complete changelog at kargig/tormap GitHub repo.

And here’s the outcome: World city map of Tor nodes at https://tormap.void.gr/
One of my main goals was to have selectable classes of nodes that will appear on the map.

To produce the map overlay, a cron script runs every hour, which is also the period it takes for Tor Authority nodes to produce a new consensus, and creates some static files which are then served by nginx.

I’m not a web developer/designer and I don’t really know any javascript. So please, feel free to fork my code and make it look better, run faster and add your own features. I’ll happily accept patches/pull requests!

Extras
On kargig/tormap repo you will also find a handy script, ‘runme.sh’, that downloads all necessary files that need to be parsed by the python script.

Missplaced nodes on the map
Well, blame MaxMind’s GeoIP City database for that. But I think it’s kinda funny to see Tor nodes in Siberia and in the middle of the sea though (look at the West coast of Africa), heh. For those wondering, these nodes are gathered there because their geoip Lat,Long is set to 0,0.
Really though, what’s “Ben’s Cat Shaque” diplayed there next to all those nodes in the west coast of Africa? Anyone has some clue ?

Conspriracy people
I’m sure that people who love conspriracy theories will start posting about those ‘Bad’ Tor nodes in Iran and Syria. Why do you think these are there ? What does it mean ? Let the flames begin!

Future TODO
a. OpenStreetMap
I have started working on an OpenStreetMap implementation of the above using OpenLayers. The biggest hurdle is that OSM does not provide a server that serves map tiles over HTTPS. Makes me wonder…is that actually so difficult ?
b. More stats
I would like to add small graphs on how the number of nodes in each class evolves.

Other Tor mapping efforts
https://b.kentbackman.com/2010/10/04/view-tor-exit-nodes-in-google-earth/
http://freehaven.net/~ioerror/maps/v3-tormap.html

Don’t forget, you can always help Tor by running a node/bridge or sending some money to Tor or EFF!

Scaling a small streaming system from 50 to 4000+ users

This post was originally written on 09/11/2012 but for various reasons it couldn’t be published earlier. It won’t be very technical, it’s mostly a behind the scenes view of how some of us at NOC GRNET tried to cope with an extreme spike of demands of a specific service we support.

Intro
At GRNET we provide a live streaming service to the Hellenic Parliament, who also connect to the Internet through our network. What we’re doing is that we’re serving the official TV channel of the Hellenic Parliament (the WebTV program actually). We have a machine that encodes (transcodes actually) video/audio and a separate streamer that serves the content to the clients via RTSP, RTMP and HTTP. While this service has been running for quite some time, and the streaming server has been used in various other occasions, it typically serves no more than 100 concurrent clients. With a stream at 640Kbps on average that’s abound 60Mbps of streaming traffic. The number of viewers usually goes up only when there’s something exciting going on. Our previous traffic spike was at 577Mbps on October 31st, when the Minister of Finance was presenting the budget for 2013 to the Parliament. What happened that day was that news reporters were on strike and it seems that one of the few news sources available at the time for people to use was the Parliament channel, either on the TV or through our stream. The viewership that day surpassed all previous expectations but the streaming system actually performed quite well considering we didn’t have any complaints.

Remember, remember the 7th of November
On the 7th of November almost everyone was on strike in Greece. There was a huge march arranged at 17:00 outside of the Greek Parliament to protest against the new harsh measures of Memorandum III that the parliament was to vote for at midnight.

We started seeing some traffic arriving, well leaving is more accurate, our streaming server at around 10:00 in the morning. Some news sites and blogs (tanea.gr, left.gr, protothema.gr) had already started posting links to our parliament’s streaming service for people to watch the discussion that was already taking place. At around 10:30 we were already serving more than 100Mbps while the traffic was steadily rising. A bit after 12:00 though things got really nasty. Websites like tovima.gr, zougla.gr and newsit.gr had posted our streaming links in their very first page. The discussions both inside the parliament and on the social networks had also started heating up. At 12:25 the traffic starting ramping up extremely fast!

12:25 160Mbps
12:32 367Mbps
12:38 597Mbps
12:42 756Mbps
12:49 779Mbps

That’s when traffic stopped growing any more. We actually saw some drop in the traffic and then up to 780Mbps and down again. It was obvious that we had reached some limit. It wasn’t very obvious though what the bottleneck was since the streaming system load was quite low, around 0.6-1.0 in a 4 vCPU VM and it also had a virtio-net/gigabit[1] network card. Theoretically it should be able to push at least 100-150 more Mbps. In order to cope with the increasing demand we changed the player web page in order to off-load some users to an experimental service that relies on IP multicast from the server and P2P between the clients for delivering streams. This bought us some time, about 2-3 hours. Also maybe because it was noon time, or due to more people using the experimental service or not being satisfied from the performance of the streaming video and disconnecting, traffic later on dropped “down” to an average of 650-700Mbits.

We started discussing short-term actions for improving the service so as to be able to cope with more demand. The streaming server was running on GRNET’s virtualization platform, which is based on ganeti, and the first thought we had was to try and add more network cards to the VM and somehow bond the cards together. Could we push more than 1Gbps from the same VM? The problem was that we couldn’t do any 802.3ad bonding since the “virtual switch” inside the virtualization platform did not support such features. One other solution would be to add another virtual network card and use Linux bonding mode 5 (balance-tlb) or 6 (balance-alb). After a bit of reading this mode was rejected as well. These two bonding modes expect from the ethernet card running on the machine to support ethtool to read their speed. Our VMs use virtio-net which doesn’t support this functionality for ethtool. We could switch to another type of network card, e1000 for example, but this could have an unmeasured/untested performance penalty which could actually negate the addition of a second network card.

Hundreds of thousands of Greeks were planning to join this huge protest starting at 17:00 and so were some of us working at GRNET, at least I was. We had to make a decision though, drop our plans and improve the streaming service to help more people, especially Greeks living abroad, to access the stream or just join others at Syntagma square? Our shift was ending soon anyway. We decided to stay at work and try and improve the service as far as we could.

The path we chose in order to “scale” was to create to a second instance of the streamer and try to “balance” client requests to both streamers using round robin DNS. That involved creating a checkpoint of the running VM at our storage system, copy the running VM’s image from that checkpoint as a new image and run a new VM with that image. So we would be cloning the current streaming service while the service was running and serving clients (take that Windows!).

So we provisioned a new Debian server through LDAP, added the copied image disks at the VM’s configuration, booted the VM, run puppet to change the configuration files according to the new hostname and IP, and we were ready to serve more clients. After some testing we created a RR DNS entry, “streamer-frontend.domain.gr” which pointed to both first-streamer.domain.gr and second-streamer.domain.gr and had a TTL of 60. Changing the landing page though to serve streamer-frontend.domain.gr as the streamer url wasn’t enough. The first problem was that there were news sites and blogs that had copied directly our first streamer’s URL instead of the live.grnet.gr/paliament/ landing page, which actually runs on a different system and is served by an Apache2 server (more on that later). Having already more than 1000 streams on the first streamer and then starting to do RR DNS on both servers meant that there would still be a large number of clients served by the first server and new people getting directed there would only make matters worse, while people directed to the second server would have a much better service experience. So what we actually did was point streamer-frontend.domain.gr not on both first-streamer.domain.gr and second-streamer.domain.gr but just second-streamer.domain.gr. We also changed the IP address of first-streamer A,AAAA records to second-streamer’s IP and flushed the first-streamer RR at our caching resolvers. Since many people use our caching resolvers, especially students with DSL lines, we were able to direct even more people towards the second-streamer.

Before adding second-streamer.domain.gr:

* first-streamer.domain.gr -> 1.2.3.4 TTL 86400
* live.grnet.gr/parliament/ pointing the streamer URL at first-streamer.domain.gr

After adding second-streamer.domain.gr:

* streamer-frontend.domain.gr -> 1.2.3.5 TTL 60
* first-streamer.domain.gr -> 1.2.3.5 TTL 60
* second-streamer.domain.gr -> 1.2.3.5 TTL 60
* live.grnet.gr/parliament/ pointing the streamer URL at streamer-frontend.domain.gr

That way people disconnecting from the stream and reconnecting some time later on were pointed to the new server, if they used our landing page. That meant better quality streaming for both “old” clients getting their stream from first-streamer, and for the new clients that were being served from second-streamer.

Within 30 minutes since booting up, the second streamer was serving more than 250Mbps and after another 30′ its traffic had climbed up to 450Mbps while first-streamer was steadily serving more than 550Mbits. That lead us to an all time record of 1.02Gbps at 17:45. At that point we were serving more than 2000 concurrent streams. A number we never expected for this humble streaming service.

While traffic was increasing through the addition of the second streamer, another problem came up. Some misconfigured clients and HTTP proxies were opening up connections at the Apache 2 serving the web page and never closed them down. We hit Apache’s internal ServerLimit variable of 256 around 17:00. We had to increase ServerLimit inside Apache’s config and restart it:

ServerLimit 500
<IfModule mpm_prefork_module>
    StartServers         10 
    MinSpareServers      15 
    MaxSpareServers      25 
    MaxClients          500
    MaxRequestsPerChild   0  
</IfModule>

After some minutes of happiness another “unexpected” issue came up. It started to rain in Athens, so people who were at the protest at Syntagma square would probably leave soon and go home. There would certainly be an increase in traffic. And what about midnight, which is when the MPs were supposed to vote on the new measures? People would be sitting at their computers, bashing the politicians on Twitter and Facebook while watching the Parliament’s stream. At the same time we started seeing a big increase in international traffic, since more Greeks living abroad were tuning in; this stream was probably the only way they could watch what was happening at the Parliament.

That only meant one thing. Hello third-streamer! 10′ later we had another streamer running. But we also wanted to make some changes to the first two VMs. We wanted to add more vCPUs, going from 4 to 8, add more RAM, going from 4096Mb to 6144Mb, and do some minor performance tuning on the software. That meant restarting the VMs in order to activate the additional resources… luckily the chairman of the Parliament announced a 10 minute recess some time around 19:30. That was our chance… we changed back first-streamer’s IP address, added it to streamer-frontend RR DNS while also adding third-streamer to it. Then we rebooted the first two VMs. After 30″ we had 3 streamers running and waiting for clients to join them. third-streamer instantly got more than 200Mbps at the very moment we rebooted the other 2 VMs.

After adding third-streamer.domain.gr:

* streamer-frontend.domain.gr -> {1.2.3.4 | 1.2.3.5 | 1.2.3.6} TTL 60
* first-streamer.domain.gr -> 1.2.3.4 TTL 60
* second-streamer.domain.gr -> 1.2.3.5 TTL 60
* third-streamer.domain.gr -> 1.2.3.6 TTL 60
* live.grnet.gr/parliament/ pointing the streamer URL at streamer-frontend.domain.gr

At 20:10, some of us decided to leave work after 11 hours, having 3 streamers running that had already reached 1.2Gbps of traffic. There were more than 2800 concurrent users at the time…

Our assumptions for midnight came true. Around that time, more than 4000 people had tuned in to the parliament’s stream and we were pushing about 1.66 Gbps. That was probably the all-time record for a single service in GRNET.

A big problem we didn’t solve that night
Unfortunately, each client gets a unique session id from the streamer, also when requesting an HTTP stream. These are not shared among the streamers, so if you send a subsequent HTTP request to a different streamer, who does not track that session id, it will not serve the request as expected but rather prompt the client to start a new session. Since some browsers did not cache the RRs of streamer-frontend long enough (remember that low TTL?) they would spread their requests to all the streamers. We knew this was happening but there was no way to solve this without extensive changes to the setup, which would have to be tested of course would mean significant downtime for the service. Therefore, depending on the browser people used, some got better or worse service than others. The good thing is that we know how to fix this in the future.

Some Stats & Graphs
more than 31000 unique IPs connecting to the streaming service
more than 4000 concurrent users at peak time
more than 4.7 TB of data streamed

first-streamer ethernet traffic:

second-streamer ethernet traffic:

third-streamer ethernet traffic:

Streamer client types

Streamer LAN traffic aggregates:

Create your own graphs here: http://mon.grnet.gr/rg/157184/details/

Epilogue
Yeah, there were hiccups in the stream for many many people, we acknowledge that. But we certainly did the best we could to keep the service running. Could we have done better ? Yes we could, but we would have to re-design the service and make some drastic changes that were never tested before. We didn’t want to risk making drastic changes that might not work at all while we could somewhat “scale” our service using a “working” solution. I wonder what will happen on Sunday when the Parliament votes for 2013 budget…Will we exceed our previous peak? We’ve already discussed alternatives to cope with the extra traffic and we’ll definitely be better prepared [2]!

I think that zmousm, alexandros and me did a fairly good job regarding the circumstances. We should buy each other a beer sometime..We’ll treat faidonl another one though for his helpful consulting 🙂

Since no external CDNs or other services provided by companies abroad were used, this might be the event with the biggest ever demand in network resources served from within Greece.

Do you like solving such problems ? Then there might be a lurking sysadmin inside you 😉

[1] After a talk with apoikos, he corrected me saying that virtio-net does not have a “gigabit” capacity. Virtio-net’s capacity is only limited by the node resources available, so it can theoretically perform better than gigabit. So our thoughts on using balance-alb/tlb to cope with the extra bandwidth were wrong. This clue points out that the cause of bottleneck on the streamer that couldn’t go over 800Mbps was the streamer software itself since both VM’s and hardware node’s system load were low.

[2] On 11/11/2012 the Greek Parliament voted for 2013 budget. What we did prior to that day to cope with the traffic was to setup a new version of the streamer software, that actually came out on 08/11/2012 just one day after the initial spike, that had the option to disable session IDs. Then it was quite straightforward to add some varnish caches in front of the streamers to serve the HTTP streams to the clients. Unfortunately client demand was somewhat lower, it only reached 0.92Gbps.

When in doubt, always blame the application

When you have a misbehaving system and you are not sure what the problem is, always bet on a poorly written application.

Here’s a small example of how another poorly written web application caused system issues.

I was sitting at my office today I when I got this nagios alert for a host.

Date/Time: Tue Nov 6 19:15:11 EET 2012
Additional Info:
SWAP CRITICAL – 0% free (0 MB out of 509 MB)

Logging in actually showed all the swap’s been used and so was RAM, 0.95/1Gb. Lots of apache2 server instances were running. I did a netstat and I saw a lot of ESTABLISHED connections:

tcp6       0      0 2001:DB8:f00::1:35571 2001:DB8:bar::100:80 ESTABLISHED 9631/apache2    
tcp6       0      0 2001:DB8:f00::1:35777 2001:DB8:bar::100:80 ESTABLISHED 9656/apache2    
tcp6       0      0 2001:DB8:f00::1:36531 2001:DB8:bar::100:80 ESTABLISHED 11578/apache2   
tcp6       0      0 2001:DB8:f00::1:36481 2001:DB8:bar::100:80 ESTABLISHED 11158/apache2   
tcp6       0      0 2001:DB8:f00::1:36295 2001:DB8:bar::100:80 ESTABLISHED 11115/apache2   
tcp6       0      0 2001:DB8:f00::1:34831 2001:DB8:bar::100:80 ESTABLISHED 8312/apache2  

2001:DB8:f00::1 -> my server
2001:DB8:bar::100 -> dst server

As one can easily see my server is connecting to port 80 of dst, possibly asking for something over HTTP.

# netstat -antpW | grep 2001:DB8:bar::100 | wc -l
111

# dig -x 2001:DB8:bar::100 +short                                 
crl.randomcertauthority.com.

tailing the log files didn’t show anything weird happening. I run a tcpdump for that dst server but there wasn’t at that time any traffic going on.

So, I took a look at munin to see when this problem started developing.



As it’s obvious from the above graphs, the problem started around 14:00. So I took another look at the apache logs and I saw a bot crawling a specific url from my server. I visited that url on my server using curl and I saw traffic flowing through tcpdump going from my server to dst server. So visiting that URL was definitely causing problems. But why?

I restarted apache, swap and memory were released, all the stale ESTABLISHED connections went away and I saw hundreds of FIN/RST packets going back and forth at tcpdump.

I tried to open a few concurrent connections from my PC to my server’s url using curl. After a couple of tries netstat showed that I had managed to create stale ESTABLISHED connections towards dst server. It was an HTTP connection asking for a crl. So I was both able to reproduce the problem and I also knew the specific url of the dst server that caused the connection hanging issues.
Next thing I did was to try to open direct HTTP connections from my server to the dst url using curl. After a few concurrent connections I managed to make curl hang. So the problem was definitely not on my server, but at the dst server.

Since it was already quite late, my first (re)action was to install mod_evasive to try and minimize the problem so I could take a better look the next day.

# aptitude install libapache2-mod-evasive
# a2enmod mod_evasive
### edit /etc/apache2/sites-enabled/site-name and add the following
       <IfModule mod_evasive20.c>
            DOSHashTableSize    3097
            DOSPageCount        1   
            DOSSiteCount        50  
            DOSPageInterval     1   
            DOSSiteInterval     1   
            DOSBlockingPeriod   10  
            DOSEmailNotify myemail@mydomain.gr
        </IfModule>
# /etc/init.d/apache2 reload

I tried to curl my server’s URL from my PC and I got blocked after the second concurrent try. But after some repetitions I was still able to create one or two stale ESTABLISHED connections from my server to the dst server. Far fewer than before but the problem was still somewhat reproducible.

Then I decided to take a look at the site’s PHP code. Finding the culprit was quite easy, I just had to find the code segment where PHP requested the dst server’s url.
Here’s the code segment:

$ch = curl_init($this->crl_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
$crl_content = curl_exec($ch);

The developer had never thought that the remove server might keep the connection open for whatever reason (rate limiting anyone?)

Patching it was quite simple:

$ch = curl_init($this->crl_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT,'60');
curl_setopt($ch, CURLOPT_TIMEOUT,'60');
$crl_content = curl_exec($ch);

After this everything worked fine again. Connections were getting ESTABLISHED but after 60 seconds they got torn down, automagically. No more stale ESTABLISHED connections. Hooray!

A letter to every developer:

Dear developer,

please test your code before shipping. Pretty please take corner cases into account. We know you’re competent enough, don’t be lazy.

Your kind sysadmin

Bypassing censorship devices by obfuscating your traffic using obfsproxy

*WARNING* 14/01/2014 This post is quite deprecated. For example obfsproxy has been completely rewritten in python and there is a newer and more secure replacement of obfs2, named obfs3. Please read this obfsproxy-debian-instructions for any updates.

Some countries like China, Iran, Ethiopia, Kazakhstan and others, like installing some nasty little boxes at the edges of their country’s “internet feed” to monitor and filter traffic. These little boxes are called DPI (Deep Packet Inspection) boxes and what they do, is sniff out every little packet flowing through them to find specific patterns and then they provide their administrator with the option to block traffic that matches these patterns. These boxes are very sophisticated and they don’t just filter traffic by src, dst or port, they filter traffic by the content (payload) the packets carry.
Unfortunately, it’s not just these countries that deploy DPI technologies, but some private companies also use such devices in order to monitor their employees.

The 10 thousand feet view
Tor is a nice way to avoid basic censorship technologies, but sometimes DPI technology is so good that it can fingerprint Tor traffic, which is already encrypted, and block it. In response to that, Tor people devised a technology called Pluggable Transports whose job is to obfuscate traffic in various ways so that it looks like something different than it actually is. For example it can make Tor traffic look like a skype call using SkypeMorph or one can use Obfsproxy to obfuscate traffic to look like…nothing, or at least nothing easily recognizable. What’s cool about obfsproxy though is that one can even use it separately from Tor, to obfuscate any connection he needs to.

A warning
Even though obfsproxy encrypts traffic and makes it look completely random, it’s not a fool proof solution for everything. It’s basic job is to defend against DPI that can recognize/fingerprint TLS connections. If someone has the resources he could potentially train his DPI box to “speak” the obfsproxy protocol and eventually decrypt the obfuscated traffic. What this means is that obfsproxy should not be used as a single means of protection and it should just be used as a wrapper _around_ already encrypted SSL traffic.
If you’re still in doubt about what can obfsproxy protect you from and from what it can’t, please read the Obfsproxy Threat Model document.

Two use cases
Obfuscate an SSH and an OpenVPN connection.
Obviously one needs a server outside the censorship perimeter that he or someone else will run the obfsproxy server part. Instructions on installing obfsproxy on Debian/Ubuntu are given in my previous blog post setting up tor + obfsproxy + brdgrd to fight censhorship. Installing netcat, the openbsd version; package name is netcat-openbsd on Debian/Ubuntu, is also needed for the SSH example.

What both examples do is obfuscate a TLS connection through an obfsproxy server so that it looks innocent. Assuming that the most innocent looking traffic is HTTP, try running the obfsproxy server part on port 80.

SSH connection
Scenario:
USER: running ssh client
HOST_A (obfsproxy): running obfsproxy on port 80 and redirecting to HOST_B port 22
HOST_B (dst): Running SSH server on port 22

What one needs to do is setup the following “tunneling”:
ssh client —> [NC SOCKS PROXY] —> obfsproxy client (USER)—> obfsproxy server (HOST_A) —> ssh server (HOST_B)

Steps:
1. on HOST_A setup obfsproxy server to listen for connection and redirect to HOST_B:
# screen obfsproxy --log-min-severity=info obfs2 --dest=HOST_B:22 server 0.0.0.0:80

2. on USER’s box, then configure obfsproxy client to setup a local socks proxy that will obfuscate all traffic passing through it:
$ screen obfsproxy --log-min-severity=info obfs2 socks 127.0.0.1:9999
Then instead of SSH-ing directly to HOST_B, the user has to ssh to HOST_A port 80 (where obfsproxy server is listening).

3. on USER’s box again, edit ~/.ssh/config and add something along the following lines:

Host HOST_A
    ProxyCommand /bin/nc.openbsd -x 127.0.0.1:9999 %h %p

This will force all SSH connections to HOST_A to pass through the local (obfsproxy) socks server listening on 127.0.0.1:9999

4. Finally run the ssh command:
$ ssh -p 80 username@HOST_A

That’s it. The connection will now pass get obfuscated locally, pass through obfsproxy server at HOST_A and then finally reach it’s destination at HOST_B.

OpenVPN connection
Scenario:
USER: running OpenVPN client
HOST_A (obfsproxy): running obfsproxy on port 80 and redirecting to HOST_B TCP port 443
HOST_B (dst): Running OpenVPN server on port 443

What one needs to do is setup the following “tunneling”:
openvpn client —> obfsproxy client (USER)—> obfsproxy server (HOST_A) —> OpenVPN server (HOST_B)

Steps:
1. on HOST_A setup obfsproxy server to listen for connection and redirect to HOST_B:
# screen obfsproxy --log-min-severity=info obfs2 --dest=HOST_B:443 server 0.0.0.0:80

2. on USER’s box, then configure obfsproxy client to setup a local socks proxy that will obfuscate all traffic passing through it:
$ screen obfsproxy --log-min-severity=info obfs2 socks 127.0.0.1:9999
Then instead of connecting the OpenVPN client directly to HOST_B, the user has edit OpenVPN config file to connect to HOST_A port 80 (where obfsproxy server is listening).

3. on USER’s box again, edit your openvpn config file, change the ‘port’ and ‘remote’ lines and add a ‘socks-proxy’ one:

port 80
remote HOST_A
socks-proxy 127.0.0.1 9999

This will instruct the OpenVPN client to connect to HOST_A passing through the local (obfsproxy) socks server listening on 127.0.0.1:9999

4. Finally run the openvpn client command:
$ openvpn client.config

That’s it.

Security Enhancement
You can “enhance” obfproxy’s security by adding a shared secret parameter to command line, so anyone who doesn’t have this secret key won’t be able to use the obfsproxy server, decryption of packets will fail:
# screen obfsproxy --log-min-severity=info obfs2 --shared-secret="foobarfoo" --dest=HOST_B:443 server 0.0.0.0:80
$ screen obfsproxy --log-min-severity=info obfs2 --shared-secret="foobarfoo" socks 127.0.0.1:9999

Documentation
Or at least…some documentation.

Your best chance to understand the internals of obfsproxy is to read the protocol specification

For more info about obfsproxy client part, read the documentation here: obfsproxy client external
For more info about obfsproxy server part, read the documentation here: obfsproxy server external

Screenshots
For those who like meaningless screenshots, here’s what wireshark (which is certainly NOT a DPI) can tell about a connection without and with obfsproxy:

Without obfsproxy

With obfsproxy

*Update*
one can find openvpn configuration files for use with obfsproxy here:
Linux Client
Windows Client

setting up tor + obfsproxy + brdgrd to fight censhorship

*WARNING* 14/01/2014 This post is quite deprecated. For example obfsproxy has been completely rewritten in python and there is a newer and more secure replacement of obfs2, named obfs3. Please read this obfsproxy-debian-instructions for any updates.

*Updated* look at the bottom for list of changes

This post is a simple guide to create a debian/ubuntu packages out of the latest versions of Tor, obfsproxy and brdgrd in order to setup a “special gateway” and help people who face censorship issues. Sharing some of your bandwidth helps a lot of people get back their freedom.

Tor
I guess most people already know what Tor is, quoting from Tor’s website:

Tor is a network of virtual tunnels that allows people and groups to improve their privacy and security on the Internet. It also enables software developers to create new communication tools with built-in privacy features. Tor provides the foundation for a range of applications that allow organizations and individuals to share information over public networks without compromising their privacy.

obfsproxy

obfsproxy is a tool that attempts to circumvent censorship, by transforming the Tor traffic between the client and the bridge. This way, censors, who usually monitor traffic between the client and the bridge, will see innocent-looking transformed traffic instead of the actual Tor traffic.

brdgrd

brdgrd is short for “bridge guard”: A program which is meant to protect Tor bridges from being scanned (and as a result blocked) by the Great Firewall of China.

Combining these to work together is quite easy if you follow this simple guide/howto.



////// Become root
$ sudo su -

////// Get build tools/packages
# cd /usr/src/
# apt-get install build-essential libssl-dev devscripts git-core autoconf debhelper autotools-dev libevent-dev dpatch pkg-config
# apt-get install hardening-includes asciidoc docbook-xml docbook-xsl xmlto
# apt-get install screen libnetfilter-queue-dev

////// Get latest versions of tor/obfsproxy/brdgrd
# git clone https://git.torproject.org/debian/obfsproxy.git
# git clone https://git.torproject.org/debian/tor.git
# git clone https://git.torproject.org/brdgrd.git

////// Compile obfsproxy & create package
# cd obfsproxy/
# ./autogen.sh 
# debuild -uc -us 

////// Compile tor & create package
# cd ../tor/
# ./autogen.sh 
# debuild -uc -us 

////// Install packages
////// The following package versions might be different depending on your configuration. Change them appropriately by looking at the deb files in your path: ls *.deb

# cd ..
# dpkg -i tor-geoipdb_0.2.4.3-alpha-1_all.deb obfsproxy_0.1.4-2_amd64.deb tor_0.2.4.3-alpha-1_amd64.deb

////// Create Tor configuration
////// PLEASE SEE THE CHANGEME_X VARIABLE BELOW BEFORE RUNNING THE FOLLOWING COMMAND

# cat > /etc/tor/torrc << EOF 
AvoidDiskWrites 1
DataDirectory /var/lib/tor
ServerTransportPlugin obfs2 exec /usr/bin/obfsproxy --managed
Log notice file /var/log/tor/notices.log
## If you want to enable management port uncomment the following 2 lines and add a password
## ControlPort 9051
## HashedControlPassword 16:CHANGEME
## CHANGEME_1 -> provide a nickname for your bridge, can be anything you like.
Nickname CHANGEME_1
## CHANGEME_2 -> How many KB/sec will you share. Don't be stingy! Try putting _at least_ 20 KB.
RelayBandwidthRate CHANGEME_2 KB
## CHANGEME_3 -> Put a slightly higher value than your previous one. e.g if you put 500 on CHANGEME_2, put 550 on CHANGEME_3.
RelayBandwidthBurst CHANGEME_3 KB
ExitPolicy reject *:* 
## CHANGEME_4 -> If you want others to be able to contact you uncomment this line and put your GPG fingerprint for example.
#ContactInfo CHANGEME_4
ORPort 443 
#ORPort [2001:db8:1234:5678:9012:3456:7890:1234]:443
BridgeRelay 1
## CHANGEME_5 -> If you don't want to publish your bridge in BridgeDB, so you can privately share it with your friends uncomment the following line
#PublishServerDescriptor 0
EOF

////// Restart Tor

# /etc/init.d/tor restart

////// Compile and run brdgrd
////// If you've changed ORport in Tor config above, be sure to change the "--sport 443" port below as well
////// brdgrd does not help since obfsproxy is already running in front of the bridge, but won't hurt either.

# cd brdgrd/
# make
# iptables -A OUTPUT -p tcp --tcp-flags SYN,ACK SYN,ACK --sport 443 -j NFQUEUE --queue-num 0
////// brdgrd Can't do IPv6 yet...so the next line is commented out
////// ip6tables -A OUTPUT -p tcp --tcp-flags SYN,ACK SYN,ACK --sport 443 -j NFQUEUE --queue-num 0
////// You can run brdgrd without root, just by setting some correct cap_net_admin rights
////// Instead of: screen -dmS brdgrd ./brdgrd -v
$ sudo screen -dmS brdgrd setcap cap_net_admin=ep ./brdgrd -v

# tail -f /var/log/tor/notices.log

The above guide has been tested on Debian Squeeze and Ubuntu 12.04.

That’s it. You just made the world a better place.

*Update*
I’ve made some changes to the post according to comments on the blog post and #tor-dev.
a) Changed URLs for the git clone operations to https:// instead of git://
b) Changed brdgrd git url to gitweb.torproject.org instead of github.
c) Changed config sections of torrc file
d) Added some more info on brdgrd

*Update2*
Tor has published “official” instructions for setting up obfsproxy bridges on Debian boxes –> Setting up an Obfsproxy Bridge on Debian/Ubuntu

*Update3*
Update sample config to inform about unpublished bridges.

Bind9 statistics-channels munin plugin

Bind9, since version 9.5, offers an experimental embedded web server which can provide statistics abound Bind through HTTP. Upon enabling, one can access this web server and get an XML response full with various statistics.

Enabling the feature is quite easy. One just needs to add some lines like the following inside Bind’s configuration file:

statistics-channels {
          inet 127.0.0.1 port 8053 allow {127.0.0.1;};
};

Restart Bind and try connecting to the statistics-channel. For example through curl:

$ curl http://127.0.0.1:8053
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="/bind9.xsl"?>
<isc version="1.0">
  <bind>
    <statistics version="2.2">
      <views>
        <view>
          <name>_default</name>
          <zones>
            <zone>
              <name>0.in-addr.arpa/IN</name>
              <rdataclass>IN</rdataclass>
              <serial>1</serial>
            </zone>
            <zone>
              <name>10.in-addr.arpa/IN</name>
              <rdataclass>IN</rdataclass>
              <serial>1</serial>
            </zone>
...
...
...
...
         <TotalUse>810834110</TotalUse>
          <InUse>327646360</InUse>
          <BlockSize>740556800</BlockSize>
          <ContextSize>9954232</ContextSize>
          <Lost>0</Lost>
        </summary>
      </memory>
    </statistics>
  </bind>
</isc>

The output will be quite big, for example the current output from a busy recursive resolver is around 2.5Mb.

People usually use munin to monitor bind through 2 different ways. The first, usually applied to servers that are not very busy, for example servers with less than 100queries/sec, is to parse the output of the query log. Then a munin plugin that comes with the default munin installation and is called bind9 can parse the query log and create some graphs. The second way for a bit busier servers is to have query log disabled and monitor bind is through the output of the rndc stats command. A munin plugin that also comes with the default munin package and is called bind9_rndc can parse the created named.stats output file and create some nice graphs. To configure either of the two ways one can take a look at this informational wiki: http://wiki.debuntu.org/wiki/Munin/Bind9_Plugin

A third way is to have a new munin plugin get the XML output from the statistics-channel described above and create some even fancier and richer graphs. Luckily a guy called Andrew Duquette has created such a perl plugin for munin. You need to exercise your google skills to find it though, (that is if you don’t cheat searching with his name as a search term!) I took that plugin, made some changes to it and uploaded it to github. The plugin is called bind9_statchannel and you can find it in my github munin-plugins repo.

The changes I made were the following:

  • Use STACK instead of just AREA for some graphs
  • Add warning and critical levels for type ANY queries (to warn for DDoS through such queries)
  • Raise timeout to 300 seconds
  • Add sorting to stacked graphs so they are a bit more readable/comparable

The plugin creates the following type of graphs:

  • Cache DB RRsets for various views (or the _default if you don’t have any others)
  • Memory Context “In Use”
  • Memory Usage Summary
  • Queries In
  • Queries Out
  • Resolver Statistics for various views (or the _default if you don’t have any others)
  • Server Statistics
  • Socket I/O Statistics
  • Tasks

As you can see it can give many more details compared to the other bind9 munin plugins. It’s also the only bind9 munin plugin that can tell you how many incoming vs outgoing queries there are, as well as the only one that can tell you how many queries you have over IPv4 vs IPv6.

To use it on Debian you need to at least install the following two perl packages: libxml-simple-perl, liblwp-useragent-determined-perl

Here are some sample graphs from the bind9_statchannel plugin:

If you have any bug fixes, code changes, etc please don’t hesitate to fork and open a pull request on github.

Download: bind9_statchannel

itop – Interrupts ‘top-like’ utility for Linux

On a 24-core Linux machine I wanted to monitor interrupts/sec. The reason is quite complicated, let’s say that there are more than 100.000 interrupts/sec and I want to monitor who needs what.

First I installed Debian’s itop package. It didn’t work at all on the 24-core machine, just showed an empty display. Then I tried a perl itop implementation I found online at a redhat server, let’s call it redhat’s itop. That still didn’t work well. Lot’s of parsing issues and it certainly wasn’t ready to deal with 24-cores.

So I tweaked it a bit and made it ready for a 24-core server. You can find it online at my github repository: kargig’s itop

Since displaying 24 columns of continuously changing numbers doesn’t fit all screens, I added some command line options to the tool.
By default it shows all interrupts/sec per CPU if the number of CPU cores is not higher than 8. If it’s higher than 8 it just displays the total (adding all per core numbers) interrupts/sec.
Command line flag ‘-a’ forces displaying of ALL cores and command line flag ‘-t’ forces displaying just the TOTAL number of interrupts/sec.

So if you too need to monitor your server’s interrupts/sec, give my itop a try. If you want to make changes, just fork it on github. Simple as that…I promise to merge any interesting changes/pull requests you people send me!

oh and here’s a screenshot of ‘itop -t’

open magnet URLs with xdg-open

Browsers on Linux should use xdg-open to open URLs. From xdg-open’s description:

xdg-open opens a file or URL in the user’s preferred application. If a URL is
provided the URL will be opened in the user’s preferred web browser. If a file
is provided the file will be opened in the preferred application for files of
that type. xdg-open supports file, ftp, http and https URLs.

Magnet is a new type of URLs that’s currently not supported by xdg-open. It’s very easy to add a quick hack to support it though. Just copy xdg-open to a local binary path (e.g /usr/local/bin or preferably $HOME/bin) and apply the following patch. Change deluge with your preferred torrent application.

--- /usr/bin/xdg-open  2010-09-15 14:08:29.000000000 +0300
+++ bin/xdg-open  2012-01-24 22:05:03.935338593 +0200
@@ -437,6 +437,11 @@
                 exit_success
             fi
         fi
+    elif (echo "$1" | grep -q '^magnet:'); then
+        deluge "$1" 
+        if [ $? -eq 0 ]; then
+            exit_success
+        fi
     fi
 
     sensible-browser "$1"

For those who don’t know how to patch a file, here it is in full: patched xdg-open to open magnet URLs

Works fine on my Debian using Chrome which properly uses xdg-open to handle URLs. Firefox after version 3.5 became a piece of bloatware and it’s not possible to open magnet links using about:config settings or xdg-open. For Opera just go to Settings->Preferences->Programs and add new protocol with name magnet. At Open with other application, use the path that you saved the patched xdg-open.

Now go download some creative commons licensed files.

*UPDATE*
Patch for latest xdg-utils (since some people complained that previous patch does not apply):



--- xdg-open  2012-02-18 14:22:48.058497027 +0200
+++ xdg-open  2012-02-18 14:24:04.326875223 +0200
@@ -440,6 +440,11 @@
                 exit_success
             fi
         fi
+    elif (echo "$1" | grep -q '^magnet:'); then
+        deluge "$1" 
+        if [ $? -eq 0 ]; then
+            exit_success
+        fi
     fi
 
     IFS=":"

*UPDATE 2*
For Firefox/Iceweasel one can do the following:
go to about:config and right-click. Then click on New->Boolean->network.protocol-handler.expose.magnet -> Value -> false
Upon the next click Firefox/Iceweasel will ask you to choose a program to open magnet links, choose the patched xdg-open posted above.

Music Player Daemon on OS X

I use a Mac Mini with OS X 10.5.8 as a media center connected to my TV and I wanted to install Music Player Daemon on it so I could control the music remotely from my laptop or phone. I mostly followed the OS X guide from MPD’s wiki to do it but I ran into some problems while trying to daemonize mpd.

I got the following error while running mpd without –no-daemon:

The process has forked and you cannot use this CoreFoundation functionality safely. You MUST exec().
Break on __THE_PROCESS_HAS_FORKED_AND_YOU_CANNOT_USE_THIS_COREFOUNDATION_FUNCTIONALITY___YOU_MUST_EXEC__() to debug.

When I ran mpd –no-daemon everything was fine though. So in order to “solve” this problem I’ve changed the plist file to include a screen invocation.

My mpd.plist looks like that now:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
    <key>Label</key>
    <string>mpd</string>
    <key>ProgramArguments</key>
    <array>
        <string>/opt/local/bin/daemondo</string>
        <string>--label=mpd</string>
        <string>--start-cmd</string>
        <string>screen</string> <!-- path to screen -->
        <string>-dmS</string>
        <string>mpd</string>
        <string>/opt/local/bin/mpd</string> <!-- path to MPD -->
        <string>-v</string>
        <string>--no-daemon</string>
        <string>/Users/kargig/.mpd/mpd.conf</string> <!-- path to MPD config -->
        <string>;</string>
        <string>--stop-cmd</string>
        <string>/opt/local/bin/mpd</string> <!-- path to MPD, again -->
        <string>--kill</string>
        <string>/Users/kargig/.mpd/mpd.conf</string> <!-- path to MPD config, again -->
        <string>;</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>Disabled</key>
    <false/>
    </dict>
</plist>

So launchctl calls daemondo which calls screen which runs mpd –no-daemon, so mpd doesn’t crash.

I use mpdscribble for scrobbling my music to last.fm. Clients-wise, I use Theremin on OS X, Gnome Music Player Client/gmpc on Debian Linux and MPDroid on Android. And all those connections over IPv6 of course, over my LAN’s Unique Local Addresses to be exact, mpd and all clients listed above work fine with IPv6.

 # lsof -n -i | grep ESTABLISHED | grep 6600
mpd       43025         kargig   12u  IPv6 0x49c719c      0t0    TCP [fdbf:aaaa:aab0:447d:216:XXff:feaa:11XX]:6600->[fdbf:aaaa:aab0:447d:222:XXff:fe1e:d8XX]:48703 (ESTABLISHED)
mpd       43025         kargig   15u  IPv6 0x3127cd4      0t0    TCP [fdbf:aaaa:aab0:447d:216:XXff:feaa:11XX]:6600->[fdbf:aaaa:aab0:447d:fadb:XXff:fe4f:aXX]:51113 (ESTABLISHED)

Apart from MPD’s wiki there’s another nice blog post you can read to help you install mpd on OS X, Integrating MPD with OS X.
For general reference on setting up mpd, Archilinux has a fine wiki entry.

Linux SSD partition alignment tips

Yes, this is another post on the internet about properly aligning your SSD partitions on Linux. It’s mostly my notes that I have gathered from other posts around the net. Please read the whole post before starting to create partitions on your SSD.

Intro
I bought myself a brand new SSD for Xmas, OCZ Agilty 3 120Gb. But I also bought a CDROM caddy so that I could replace my useless macbook CDROM drive, last time I used it was probably 2009 or 2010. So my plan was to put the old, original macbook SATA hard disk inside the caddy and use the SSD as the primary one. Sounds easy right ? Well you just need patience, lots of patience in order to remove all necessary screws in order to get the CDROM drive out and replace it with the caddy. Instructions for this procedure can be found at iFixIt.com.

Create Partitions on the SSD disk
Before one begins some definitions!

Heads = Tracks per cylinder
Sectors = Sectors per track

The goal here is to have the partitions aligned to the SSD’s Erase Block Size.
Googling around the net I found out that OCZ always uses 512Kb as Erase Block Size. If one uses fdisk with 32 Heads and 32 Sectors that makes a cylinder size of 1024b = 1Kb. Multiplying with 512 (sector size), which is fdisk’s default unit size, that makes it 512kb (=32*32*512)! Exactly the Erase Block Size that’s needed. So one needs to start fdisk issuing the following command:
# fdisk -H32 -S32 /dev/sdb
where /dev/sdb is the SSD.

It is very important to remember to start the first partition from the 2nd unit (or 512kb if you prefer). Due to MS-DOS compatibility if the first partition were to start at the first cylinder, it would skip one track. So it would actually start at 32(sectors)*512(sector size)=16Kb, messing up the alignment.

Then create necessary partitions as needed.

LVM alignment
So, the partitions on the SSD are aligned, but what if one wants to use LVM ? Then LVM’s overhead has to be taken into account as well.
To create an aligned PV based on the partitions that have already been created one needs to use the “–dataalignment” option found in newer versions of LVM utilities.
# pvcreate --dataalignment 512k /dev/sdb3
To check the alignment use the following command:

# pvs /dev/sdb3 -o+pe_start
  PV         VG   Fmt  Attr PSize   PFree  1st PE 
  /dev/sdb3  ssd  lvm2 a-   111.46g 81.46g 512.00k

Check that “1st PE” is what is actually needed for the alignment.

Proceed creating VGs and LVs as needed.

Formatting Partitions with ext4
There’s no reason to use ext3 on SSD, one needs to take advantage of ext4 SSD features. I prefer 4K as block size.
For a further explanation of the following formulas read Linux RAID Wiki – RAID setup
stride = chunk (Sector size) / block size = 512Kb / 4K = 128
stripe-width is usually calculated by a formula that uses multiple disks. Since there’s only one disk in this scenario, stripe-width is equal to stride.
stripe-width = 128

# mkfs.ext4 -O extent -b 4096 -E stride=128,stripe-width=128 /dev/mapper/ssd-debian

Mounting the partition
To enable SSD TRIM support, which protects the disk from wearing off, one needs to enable the discard option while mounting the partition. Edit /etc/fstab and add the discard mount option (and noatime if you want to).
/dev/mapper/ssd-deb / ext4 discard,noatime,errors=remount-ro 0 1

Note 1: As of 2.6.37 Linux Kernel supports TRIM on device mapper. Previous kernel versions will report errors upon trying to mount an LVM partition with discard mount option. If you have an older kernel either don’t use LVM on your SSD yet or upgrade your kernel!
Note 2: Read the links posted below for a complete blog post over TRIM command. Apparently it’s not always the best choice

That’s basically it…

Extra – copying the old root partition to the new disk

# mkdir /mnt/ssd/
# mount /dev/mapper/ssd-debian /mnt/ssd/
# rsync -aPEHv --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/mnt --exclude=/var/cache/apt/archives/ / /mnt/ssd/
# mkdir /mnt/ssd/dev
# mkdir /mnt/ssd/proc
# mkdir /mnt/ssd/sys
# mkdir /mnt/ssd/mnt
# cp -avp /dev/console /mnt/ssd/dev/
# cp -avp /dev/urandom /mnt/ssd/dev/
# cp -avp /dev/zero /mnt/ssd/dev/
# cp -avp /dev/random /mnt/ssd/dev/
# Edit /mnt/ssd/etc/fstab to change the device names
# Update grub

Using the above commands one avoids copying unneeded directories like /dev, /sys, etc that will be recreated later. Don’t forget to at least copy the above 4 devices in the new /mnt/ssd/dev dir, else the partition won’t be bootable.

References
1. Aligning Filesystems to an SSD’s Erase Block Size (link goes to archive.org since original article has unfortunately disappeared from the web)
2. [linux-lvm] Re: Aligning PVs on SSDs?
3. Aligning an SSD on Linux (excellent article!)
4. ArchWiki – Solid State Drives
5. SSD performance tips for RHEL6 and Fedora
6. Re: [dm-devel] trim support (discard)
7. How to check TRIM is working on your SSD running Linux
8. Impact of ext4′s discard option on my SSD (very useful insight on TRIM command, read the comments as well)

Thanks
Thanks fly to @apoikos for helping me with the CDROM replacement and @faidonl for his original SSD alignment tips 🙂