Into.the.Void.

A few months ago I created a list of tips for software (Slack, Gmail, Github, Zoom, Jira, etc) that are commonly used in many companies (Elastic that I work for included) and I believe can improve productivity, efficiency and enhance collaboration, especially in remote/distributed companies. There’s nothing ground breaking or new here, but I like having all of them in a single place. I published it internally, it was very well received and a few colleagues provided their feedback and new ideas (thank you!). Here’s the current list of work productivity tips:

Slack

• Use “Save for later” to create a list of important tasks/events/conversations you need to get back to
• Use “Remind me about this” when you’re expecting a follow up from someone or you want to follow up with them at a later time
• You can use all slack commands in a DM
• Use slack channel sections to group channels into different categories
  • Having a slack section for ‘Active Incidents’ is a great way to make following up easy, and not losing visibility of lower level incidents you’re involved in.
• Mute channels you don’t frequently use to reduce the number of unread channels which may act as a distraction for some
  • You still get notified if someone mentions you
• Use keywords in Slack preference → Notifications if you want to know all that is happening about a specific keyword in the channels you’ve joined
• If you want to enhance your focus time use “Pause Notifications”
  • If you are part of an (important) zoom/meeting, use “Pause notifications” in Slack to avoid getting distracted
• Adjust your notification settings to receive alerts for important channels or direct messages only.
• Use “Slack status” to declare when you’re on PTO or on focus time, it will help people understand why you’re not responding to them
  • This can be automagically synced from Google Calendar (docs) so you only have to add ad-hoc things like 💇 or 🚗
• Make use of Slack’s advanced search modifiers to find specific messages or files quickly. For example, in:channel to search within a specific channel, from:@username to search messages from a specific user
• When messaging users outside of their normal working hours, consider using the “Schedule for Later” option. This accomplishes two things, reduces off-hours work messages to users and will ensure the recipient doesn’t forget about your message overnight.
• You can DM yourself things you may want to remember that weren’t from an existing message (even leveraging above mentioned “Save for later”, “Remind me”, and “Scheduled message”! It’s totally healthy to have an inner monologue 🙂
• Use threads for replies to maintain organized conversations and minimize clutter in the main channel

Email/Gmail

• Use filters, many filters. Refine them often
  • Ask your teammates about their filters
  • Exchange your filters and share them with your team
  • Having good email filters can enable you to monitor Jira and Github updates
  • If you are on a Mac, Mimestream allows you to make/manage gmail filters from within the program
  • The ‘gmailctl’ program is a CLI to import/export filters in a format that is easy to read/update/create
• Use email as a more formal method of communication. Responses in emails usually have more thought behind them than slack messages
  • Slack messages, especially in channels, can easily be forgotten/ignored
• Expand “Google Tasks” and drag and drop emails that you want to respond to later into Tasks
• Star emails that you believe are important to keep separately and to find them more easily
• Don’t read your emails if you don’t have time to respond
  • If you did read something but you want to respond to it later, mark it as unread
• Disable notifications for emails
  • Elastic often uses emails as broadcast messages, there are soooo many emails every day and getting notifications for those emails can be distracting
• Schedule how often you read your emails, you don’t have to do it constantly
  • Elastic uses Slack for urgent notifications/messages
  • Every 2-4hrs is probably good enough
• Consider avoiding checking your emails first thing in the morning; focus instead on your big rocks for the day (that you’d have identified the prior evening). This will ensure you get the most important done and avoid getting sucked-into the vortex of emails follow-ups
• Use “Vacation responder” to declare when you’re on PTO, it will help people understand why you’re not responding to them. Feel free to mention a few delegates
• If you had direct reports in the past that have left the company, you can ask IT to stop forwarding emails to your account once you know there’s nothing valuable coming through anymore

Github

• Subscribe to “All” events for issues/PRs you care deeply about
• Subscribe to “Status changes” for issues/PRs you want to know if/when they are closed
• Use Github “busy” status to avoid getting assigned issues/PR reviews when you’re on PTO
• If you are reviewing a PR and you have identified problems, apart from comments please provide code suggestions.
  • This can severely expedite the time it takes to get to acceptable quality, especially in a distributed environment like Elastic where people from different timezones need to work together
• The Github Notifications page is very useful if you find the emails too spammy or difficult to filter to your liking

Zoom

• AI companion
  • Use AI companion during the meeting to ask questions or generate a summary up to that point
  • Ask the meeting host to send the “Meeting summary” after the meeting
  • If AI companion is not enabled by IT, ask them to do so
• If you don’t like AI stuff, take meeting notes yourself so you have a record of what was discussed and what are the action items
  • Delegate a person in a meeting to be the one taking notes
  • For recurring meetings, cycle between meeting participants
• Use keyboard shortcuts:
  • Mute/Unmute mic: 🐧Alt + A / 🍎 Command + Shift + A
  • Start/Stop your video: 🐧Alt + V / 🍎Command + Shift + V
  • Raise/Lower your hand: 🐧Alt + Y / 🍎 Option + Y
• Settings:
  • Audio
    • ✅ Automatically join audio by computer when joining a meeting
    • ✅ Mute my microphone when joining a meeting
  • Team chat
    • Add keywords to receive notifications when someone mentions those keywords in zoom chat
• Go into your zoom settings via the web URL to change how you record zoom meetings – you can record to computer, or to the web. To the web means if you disconnect, your recording won’t pause! Setting this to fire and forget means one less thing to remember
• Don’t unmute your microphone unless you intend to be the active person speaking
• In larger meetings, use the “Raise Hand” feature to let others know you want to speak – this helps minimize people talking over each other. (If multiple hands are raised, the tiles will sort in the order they were raised)

Jira

• “Watch” issues your interested at
• Create browser search engine so you can search directly in jira from your browser URL bar
  • e.g. “https://CHANGEME.atlassian.net/issues/?smartQueryDisabled=true&wildcardAdded=true&jql=text%20%7E%20%22%s*%22”
• You can create filters and dashboards to organise your work better. Remember with filters you need to save them so you can add them to dashboards, and if you need to share a filter or dashboard you have to explicitly enable that. (They are private by default)
• Jira has its own query language (JQL) that’s similar to SQL – don’t be afraid of the advanced search, you’ll likely prefer it. Help is a google search away.
• The ‘Your Work’ page is a ready made summary of where you’re at with your issues.  https://CHANGEME.atlassian.net/jira/your-work

Efficient meetings

• If there’s no agenda, ask for one before the meeting
  • If you’re not provided with one, kindly decline the meeting
• If it’s a recurring meeting between a large group (8-10+ people) ask the host if your presence is required for the upcoming meeting
• Start the meeting by clarifying what decisions or action points need covering
• Be a timekeeper; if you don’t know why you’re gathered in the first 5 minutes you’re falling behind. Save the last 5 mins to recap the next steps
• Use Google Calendar’s Speedy Meetings settings (example) to make sure folks have a comfort break between meetings
• End the meeting with summarizing action items or decisions, to ensure everyone is aligned. If there are no next steps, it’s not an productive/effective meeting!
• It’s a good habit to email the meeting invite any notes on key decisions or action items, so there’s no mistakes on what needs to be done
• Notes/Summary/Action Items
  • Ask the host to either take notes or designate a moderator/note taker
  • Leave some time before the end of the meeting to communicate action points/summary and verify there’s agreement
  • Ask the host or the designated moderator/note taker to send notes/meeting summary/action items after the meeting
• If you want to discuss a technical issue, come with a few diagrams (before vs after / option1 vs option2)
  • Don’t waste time trying to describe complex technical flows with words, use drawings
  • Excalidraw is great for quick drawings
  • Consider sending docs or diagrams to the attendees at least 24 hours in advance, so you all start from the same place of knowledge. (Bonus – some people want more time to reflect, so you’re supporting their needs too!)

Enjoying PTO / Out of Office

• No one wants to contact you when you’re out of office so if you’re away for a week:
  • Make sure you send your manager, or your team, a summary of the work you have in progress and what the next steps are. This helps them if you need to be out longer, or something emergent comes up – but it also helps you find the threads of where you’re at.
    • Example of message – Context of work, key decisions or dates that you’re waiting on, next steps, by when is this a problem, key link to the jira issue, points of contact if working with others.
• If you have a lot of work in progress, consider writing a short handover document and work to assign out pieces to others to keep things moving whilst you’re away.
• Ask people to send you a DM with urgent / not urgent prefacing their message, so you know what to prioritize when you get back. This goes for anything they hand back to you in your handover document!
• Set your slack, calendar and email to be out of office and include when you’ll be back in office, and who to contact for urgent issues. Decline the meetings you’re missing (out of office meeting type by GCal can do this for you)
• If you’re working with other teams, be sure to let them know when you’re out, when you’re back and who to reach out to if they get blocked
• Make sure you weren’t due to be on call!
• Bring back a nice picture to make everyone envious about your time away 😉
• If you’re coming back from sick leave, let your manager know how you’re doing – they care about you and want to know how to support your return to office
• Either keep work-related applications in a separate mobile phone (a <200$ android phone is more than enough), or disable Elastic slack, calendar, and email on your primary device so you can enjoy the time without notifications!

Hope it helps you as well

P.S. First post after 4,5 years…I could barely remember how to use wordpress, it has changed so much in recent years. Let’s see how long it takes for the next post

Filed by kargig at 11:58 under General,Work
Tags: collaboration, email, github, gmail, jira, meetings, productivity, slack, work, zoom
No Comments | 72 views

2 months ago I wrote a blog post describing the really bad state of STARTTLS support of Greek email providers. Things have slightly gotten better since then.

Updates on STARTTLS support per provider
The following is current as of 2016/03/26 and are only the updates since the previous blog post.
FORTHNET: Supports TLS 1.2 (at least since 2016/02/03)
VODAFONE: Supports TLS 1.2 for vodafone.gr but NOT for vodafone.com.gr (at least since 2016/03/10)

Updates on Certificate status per provider (that have STARTTLS support)
FORTHNET: uses a valid certificate (a wildcard *.forthnet.gr)
VODAFONE: uses a valid certificate (a wildcard *.megamailservers.eu)
MAILBOX: uses a valid signed certificate (for spamexperts.eu) (at least since 2016/03/26)

No other changes have been observed.

These updates indicate that 3 out of 5 commercial Greek ISPs currently use STARTTLS on their mail servers, OTE/COSMOTE, Forthnet and Vodafone. Way better than 1 out 5 which was the case 2 months ago. That means that the only ones left behind are Wind and Cyta, Since HOL has merged with Vodafone.

P.S. Thanks fly to @stsimb for notifying me of Forthnet updates with a comment on my blog

Filed by kargig at 21:02 under Encryption,Greek,Internet,Linux,Networking,Privacy
Tags: certificate, email, Internet, ISP, Networking, Privacy, providers, security, startls, tls, verification
No Comments | 9,938 views

I started looking into the STARTTLS support of Greek email providers completely by accident when one email of mine wasn’t being delivered for some reason to a friend who has an email address at a traditional Greek ISP. I started looking into the delivery issues by running swaks against the email server of the ISP and I just couldn’t believe it that the ISP’s mail server response did not include STARTTLS support. That made me wonder about the rest of the ISPs, so I created a very simple script that takes domains, finds their MX addresses and performs very simple TLS lookups using openssl. Yeah I know that there are websites that track the STARTTLS support of mail servers, but they usually don’t save the previous results and you can’t grep and compare.

What I’ve looked into is how emails are sent between servers (SMTP), not if users can read emails from the mail servers (POP3/IMAP) using encrypted connections.

TL;DR
The situation is BAD, REALLY BAD. Only 1,5 (yes, this is one and a half) commercial ISPs supports STARTTLS. OTE/COSMOTE has “proper” STARTTLS support while Wind has STARTTLS support only for windtools.gr domain, but not for their wind.gr.

I couldn’t believe the situation was SO, SO BAD before looking at the results. It seems that I had a lot more faith in those providers than I should have. Yeah I was wrong once again.

wtf is STARTTLS?
(please don’t read the next sentence if you know what TLS is)
If you have no idea about TLS and STARTTLS, then consider STARTTLS a way for servers to communicate and exchange data in encrypted form instead of cleartext. If mail servers don’t support STARTTLS then other servers can’t send them emails in encrypted form and everyone between those 2 servers can read the emails. It’s the equivalent of “https://” for mail servers. (There, I said it…).

TLS support per provider
The following is current as of 2016/01/23

Commercial Providers
OTE/COSMOTE: Some servers support TLS version 1.0 and some others 1.2 (more on that later)
WIND: Supports TLS version 1.0 on windtools.gr but does NOT support TLS on wind.gr (different mail servers)
CYTA: Does NOT support TLS on their mail servers
FORTHNET: Does NOT support TLS on their mail servers
HOL: Does NOT support TLS on their mail servers
VODAFONE: Does NOT support TLS on their mail servers

non-Commercial Providers
GRNET: Supports TLS 1.2
SCH: Supports TLS 1.0
TEE: Does NOT support TLS on their mail servers
MIL: Supports TLS 1.0

Universities
AUTH: Supports TLS 1.2
NTUA: Some servers support TLS 1.0 and one supports TLS 1.2
UPATRAS: Supports TLS 1.0

Free Providers
IN: Does NOT support TLS on their mail servers
FREEMAIL: Does NOT support TLS on their mail servers
MAILBOX: Supports TLS 1.2

Radical Providers
ESPIV: Supports TLS 1.2

Certificate status per provider (that have STARTTLS support)
OTE/COSMOTE: *.otenet.gr mail servers, which are the ones that support TLS 1.0, use a certificate that is valid for mailgate.otenet.gr, *.ote.gr mail servers have their own certificates, but all mail*.dt-one.com mail servers, which are the ones that use TLS 1.2, use the same self-signed certificate.
WIND: mx2.windtools.gr uses a valid certificate
GRNET: uses a valid certificate
SCH: uses a self-signed certificate (which has expired 5 years ago) signed by their own CA (which has expired 4 years ago)
MIL: uses a self-signed certificate (which has expired 1 year ago) signed by their own CA
AUTH: uses a certificate signed by their own CA called HARICA, whose certificate is now included in modern OSes, so I will consider this a valid certificate.
NTUA: all mail servers use a certificate that is valid for mail.ntua.gr
UPATRAS: uses a valid certificate
MAILBOX: uses a self-signed certificate (by plesk)
ESPIV: uses a valid certificate (a wildcard *.espiv.net)

Why does it matter
It makes a huge difference for users’ privacy. If a mail server does not support STARTTLS then anyone with the ability to look into packets traveling on the net from a source mail server to the destination mail server can read the emails in pure plaintext, as you read them on your mail client. Support of STARTTLS for a mail server forces an adversary that previously just passively monitored traffic to have to start a MITM (Man in the middle) attack in order to read those same emails. This converts the adversary from a passive to an active attacker. And this is both expensive and dangerous for the adversary, it can get caught in the act.

Security and privacy-minded people might start bashing me on my next proposal, but considering the current situation I think it’s OK for most of the users of those providers that don’t support TLS at all.
Dear providers, please install a certificate, even a self-signed one, and add support for STARTTLS on your mail servers today.

Even a self-signed certificate improves this situation. And it costs absolutely nothing. There’s really no excuse to not even have a self-signed certificate for your email server.

Self-signed vs CA-Signed
Truth is that it 99.9999% of email servers on the Internet do not verify the remote end’s certificate upon communication. That means that it makes absolutely no difference in most cases whether the certificate is CA-signed or self-signed. Most modern email servers support fingerprint verification for remote servers’ certificates but this can’t obviously scale on the Internet. If a user fears that some entity could MITM their email provider just to read their email, they already have bigger problems and certificate verification would not be able to help them a lot anyway. They either need to protect the contents of their email (gpg?) or start using alternate means of messaging/communication (pond?)

script
The script I used is on github: gr-mx. Feel free to make changes and send pull requests.
I plan to run the script once a week just to keep an archive of the results and be able to track and compare. Let’s see if something changes…

Various weirdness
* windtools.gr has 2 MX records, mx1.windtools.gr and mx2.windtools.gr. mx1.windtools.gr has been unreachable since I started running the script on 2016/01/08.
* mail{5,6,7,8}.dt-one.com mailservers used by OTE/COSMOTE did not have the self-signed certificate on 2016/01/08 while mail{1,2,3,4}.dt-one.com had it. The certificate was added at some point between 2016/01/11 and 2016/01/17

Filed by kargig at 15:55 under Encryption,Greek,Internet,Linux,Networking,Privacy
Tags: certificate, email, Internet, ISP, Networking, Privacy, providers, security, startls, tls, verification
3 Comments | 19,215 views

More and more privacy experts are nowdays calling people to move away from the email service provider giants (gmail, yahoo!, microsoft, etc) and are urging people to set up their own email services, to “decentralize”. This brings up many many other issues though, and one of which is that if only a small group people use a certain email server, even if they use TLS, it’s relatively easy for someone passively monitoring (email) traffic to correlate who (from some server) is communicating with whom (from another server). Even if the connection and the content is protected by TLS and GPG respectively, some people might feel uncomfortable if a third party knew that they are actually communicating (well these people better not use email, but let’s not get carried away).

This post is about sending SMTP traffic between two servers on the Internet over Tor, that is without someone being able to easily see who is sending what to whom. IMHO, it can be helpful in some situations to certain groups of people.

There are numerous posts on the Internet about how you can Torify all the SMTP connections of a postfix server, the problem with this approach is that most exit nodes are blacklisted by RBLs so it’s very probable that the emails sent will either not reach their target or will get marked as spam. Another approach is to create hidden services and make users send emails to each other at their hidden service domains, eg username@a2i4gzo2bmv9as3avx.onion. This is quite uncomfortable for users and it can never get adopted.

There is yet another approach though, the communication could happen over Tor hidden services that real domains are mapped to.

HOWTO
Both sides need to run a Tor client:
aptitude install tor torsocks

The setup is the following, the postmaster on the receiving side sets up a Tor Hidden Service for their SMTP service (receiver). This is easily done in his server (server-A) with the following line in the torrc:
HiddenServicePort 25 25. Let’s call this HiddenService-A (abcdefghijklmn12.onion). He then needs to notify other postmasters of this hidden service.

The postmaster on the sending side (server-B) needs to create 2 things, a torified SMTP service (sender) for postfix and a transport map that will redirect emails sent to domains of server-A to HiddenService-A.

Steps needed to be executed on server-B:
1. Create /usr/lib/postfix/smtp_tor with the following content:

#!/bin/sh

torsocks /usr/lib/postfix/smtp $@

2. Make it executable
chmod +x /usr/lib/postfix/smtp_tor

3. Edit /etc/postfix/master.cf and add a new service entry
smtptor unix - - - - - smtp_tor
For Debian Stretch and/or for postfix 2.11+ this should be:

smtptor      unix  -       -       -       -       -       smtp_tor
  -o smtp_dns_support_level=disabled

4. If you don’t already have a transport map file, create /etc/postfix/transport with content (otherwise just add the following to your transport maps file):

domain-a.net        smtptor:[abcdefghijklmn12.onion]
domain-b.com        smtptor:[bbbcccdddeeeadas.onion]

5. if you don’t already have a transport map file edit /etc/postfix/main.cf and add the following:
transport_maps = hash:/etc/postfix/transport

6. run the following:
postmap /etc/postfix/transport && service postfix reload

7. If you’re running torsocks version 2 you need to set AllowInbound 1 in /etc/tor/torsocks.conf. If you’re using torsocks version 1,you shouldn’t, no changes are necessary.

Conclusion
Well that’s about it, now every email sent from a user of server-B to username@domain-a.net will actually get sent over Tor to server-A on its HiddenService. Since HiddenServices are usually mapped on 127.0.0.1, it will bypass the usual sender restrictions. Depending on the setup of the receiver it might even evade spam detection software, so beware…If both postmasters follow the above steps then all emails sent from users of server-A to users of server-B and vice versa will be sent anonymously over Tor.

There is nothing really new in this post, but I couldn’t find any other posts describing such a setup. Since it requires both sides to actually do something for things to work, I don’t think it can ever be used widely, but it’s still yet another way to take advantage of Tor and Hidden Services.

!Open Relaying
When you setup a tor hidden service to accept connections to your SMTP server, you need to be careful that you aren’t opening your mail server up to be an open relay on the tor network. You need to very carefully inspect your configuration to see if you are allowing 127.0.0.1 connections to relay mail, and if you are, there are a couple ways to stop it.

You can tell if you are allowing 127.0.0.1 to relay mail if you have something like this in your postfix configuration by looking at the smtpd_recipient_restrictions and seeing if you have permit_mynetworks, and your mynetworks variable includes 127.0.0.1/8 (default). The tor hidden service will connect via 127.0.0.1, so if you allow that to send without authentication, you are an open relay on the tor network, and you don’t want that…

Three ways of dealing with this.

1. Remove remove 127.0.0.1 from mynetworks and use port 25/587 as usual.

2. Create a new secondary transport that has a different set of restrictions. Copy the restrictions from main.cf and remove ‘permit_mynetworks’ from them
/etc/postfix/master.cf

2525      inet  n       -       -       -       -       smtpd
   -o smtpd_recipient_restrictions=XXXXXXX
   -o smtpd_sender_restrictions=YYYYYY
   -o smtpd_helo_restrictions=ZZZZ

2587 inet n - - - - smtpd
   -o smtpd_enforce_tls=yes
   -o smtpd_tls_security_level=encrypt
   -o smtpd_sasl_auth_enable=yes
   -o smtpd_client_restrictions=permit_sasl_authenticated,reject
   -o smtpd_sender_restrictions=
   -o smtpd_recipient_restrictions=XXXXXXX
   -o smtpd_sender_restrictions=YYYYYY
   -o smtpd_helo_restrictions=XXXXX

Then edit your /etc/tor/torrc

HiddenServiceDir /var/lib/tor/smtp_onion
HiddenServicePort 25 2525
HiddenServicePort 587 2587

3. If your server is not used by other servers to relay email, then you can use the newer postfix variable that was designed for restricting relays smtpd_relay_restrictions (remember NOT to use permit_mynetworks there) to allow emails to be “relayed” by the onion service:
/etc/postfix/main.cf

smtpd_relay_restrictions = permit_sasl_authenticated,
        reject_unauth_destination

smtpd_recipient_restrictions =
        reject_unknown_recipient_domain,
        check_recipient_access hash:$checks_dir/recipient_access,
        permit_sasl_authenticated,
        permit_mynetworks,
        permit

Concerns
Can hidden services scale to support hundreds or thousands of connections e.g. from a mailing list ? who knows…
This type of setup needs the help of big fishes (large independent email providers like Riseup) to protect the small fishes (your own email server). So a new problem arises, bootstrapping and I’m not really sure this problem has any elegant solution. The more servers use this setup though, the more useful it becomes against passive adversaries trying to correlate who communicates with whom.
The above setup works better when there are more than one hidden services running on the receiving side so a passive adversary won’t really know that the incoming traffic is SMTP, eg when you also run a (busy) HTTP server as a hidden service at the same machine.
Hey, where did MX record lookup go ?

Trying it
If anyone wants to try it, you can send me an email using voidgrz25evgseyc.onion as the Hidden SMTP Service (in the transport map).

Links:
http://www.postfix.org/master.5.html
http://www.groovy.net/ww/2012/01/torfixbis
ehloonion/onionmx github repository

*Update 01/02/2015 Added information about !Open Relaying and torsocks version 2 configuration*
*Update 11/10/2016 Updated information about !Open Relaying*
*Update 14/06/2018 Added link to ehloonion/onionmx*

Filed by kargig at 22:14 under Encryption,Internet,Linux,Networking,Privacy
Tags: email, hidden service, postfix, Privacy, smtp, tor
6 Comments | 55,631 views

One of the worst things a webmaster or a anyone else that runs some web application can do, is to constantly send “informative newsletters” to people. Most CMS applications make it really easy to send such emails. These are 99% spam, and as such there are many good reasons that you should limit the amount of such outgoing “newsletters” coming out of your email server. Else there’s a good chance you might get added to a blacklist, and you don’t want your legitimate clients to have their emails blocked because of some irresponsible people. I recently had to deploy such a solution to a hosting server that serves multiple (>300) domains. The server already ran postfix, so I had to implement something useful around it.

The problem with postfix is that you can’t really rate-limit the outgoing queue per sender domain/address. There are only generic settings that control the general mail server’s capabilities of sending emails. What I wanted though is to have the ability to restrict specific domains to some specific email message count per day. This is something that a postfix addon named postfix-policyd can do by deferring/greylisting, but still just on the incoming queue. One would think that the problems would be solved by just applying this, but truth is that they don’t. Applying a defer/greylisting policy on the incoming queue is fine while the client on the remote side is another SMTP server that can happily store the deferred email on its queue and retry some minutes/hours later. What happens though if the SMTP client is a PHP application that connects through the mail() function ? There you have no queue and if you defer a message at the SMTP server it will get forever lost, PHP can’t resend it. So the solution would be to apply an intermediate SMTP queue between PHP and the primary SMTP server, that is another local postfix installation that would only serve as a queue that relays emails to the primary.

Using a “simple” diagram sending an email from PHP should follow this path upon a successful installation:

PHP mail() –(sendmail binary)–> intermediate_POSTFIX –(SMTP relay)–> POSTFIX –(smtpd_sender_restrictions)–> POLICYD –(pickup)–> POSTFIX –(SMTP)–> REMOTE SERVER

Here are the steps I took on a Debian Squeeze server to install this little monster.

1. Create a new postfix configuration directory for the new intermediate postfix instance
I named my intermediate postfix config dir as postfix2525, name comes from the port that it will listen on but you can definitely be more creative.

# mkdir /etc/postfix2525
# cp -av /etc/postfix /etc/postfix2525

Remove everything from /etc/postfix2525/main.cf and just add the following lines:

data_directory = /var/lib/postfix2525
queue_directory = /var/spool/postfix2525
relayhost = 127.0.0.1:12525

This defines a new data and queue directory and instructs this postfix to relay all emails through another one that listens on the localhost, the primary one, on port 12525. More about this port later when you will create some special config on the primary postfix.

Remove previous contents of /etc/postfix2525/master.cf and just add these lines:

127.0.0.1:2525      inet  n       -       -       -       2       smtpd
        -o syslog_name=postfix2525
pickup    fifo  n       -       -       60      1       pickup
cleanup   unix  n       -       -       -       0       cleanup
qmgr      fifo  n       -       n       300     1       qmgr
#qmgr     fifo  n       -       -       300     1       oqmgr
tlsmgr    unix  -       -       -       1000?   1       tlsmgr
rewrite   unix  -       -       -       -       -       trivial-rewrite
bounce    unix  -       -       -       -       0       bounce
defer     unix  -       -       -       -       0       bounce
trace     unix  -       -       -       -       0       bounce
verify    unix  -       -       -       -       1       verify
flush     unix  n       -       -       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       -       -       -       smtp
# When relaying mail as backup MX, disable fallback_relay to avoid MX loops
relay     unix  -       -       -       -       -       smtp
        -o smtp_fallback_relay=
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       -       -       -       showq
error     unix  -       -       -       -       -       error
retry     unix  -       -       -       -       -       error
discard   unix  -       -       -       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       -       -       -       lmtp
anvil     unix  -       -       -       -       1       anvil
scache    unix  -       -       -       -       1       scache

Obviously the most important part here is the first line. It defines that this postfix instance will listen for SMTP connections on localhost, port 2525 and it’s syslog output name will be postfix2525 so that it’s easier to tell apart which SMTP instance spits which errors.

After this is done you need to run the following command that will create all necessary directories with their proper permissions.

# postfix -c /etc/postfix2525/ check

Also make sure you add the following line to the main.cf file of your main postfix installation:
alternate_config_directories = /etc/postfix2525

You will also need a new init script. Since the script by itself is quite big and there are only a few lines that actually differ, I will post my diff here:

--- /etc/init.d/postfix  2011-05-04 21:17:47.000000000 +0200
+++ /etc/init.d/postfix2525  2011-12-19 19:22:09.000000000 +0100
@@ -17,8 +17,10 @@
 # Description:       postfix is a Mail Transport agent
 ### END INIT INFO
 
+CONFDIR=/etc/postfix2525
 PATH=/bin:/usr/bin:/sbin:/usr/sbin
 DAEMON=/usr/sbin/postfix
+DAEMON_OPTIONS="-c /etc/postfix2525"
 NAME=Postfix
 TZ=
 unset TZ
@@ -28,13 +30,13 @@
 
 test -f /etc/default/postfix && . /etc/default/postfix
 
-test -x $DAEMON && test -f /etc/postfix/main.cf || exit 0
+test -x $DAEMON && test -f /etc/postfix2525/main.cf || exit 0
 
 . /lib/lsb/init-functions
 #DISTRO=$(lsb_release -is 2>/dev/null || echo Debian)
 
 running() {
-    queue=$(postconf -h queue_directory 2>/dev/null || echo /var/spool/postfix)
+    queue=$(postconf -c $CONFDIR -h queue_directory 2>/dev/null || echo /var/spool/postfix2525)
     if [ -f ${queue}/pid/master.pid ]; then
   pid=$(sed 's/ //g' ${queue}/pid/master.pid)
   # what directory does the executable live in.  stupid prelink systems.
@@ -66,7 +68,7 @@
       fi
 
       # see if anything is running chrooted.
-      NEED_CHROOT=$(awk '/^[0-9a-z]/ && ($5 ~ "[-yY]") { print "y"; exit}' /etc/postfix/master.cf)
+      NEED_CHROOT=$(awk '/^[0-9a-z]/ && ($5 ~ "[-yY]") { print "y"; exit}' /etc/postfix2525/master.cf)
 
       if [ -n "$NEED_CHROOT" ] && [ -n "$SYNC_CHROOT" ]; then
     # Make sure that the chroot environment is set up correctly.
@@ -111,7 +113,7 @@
     umask $oldumask
       fi
 
-      if start-stop-daemon --start --exec ${DAEMON} -- quiet-quick-start; then
+      if start-stop-daemon --start --exec ${DAEMON} -- ${DAEMON_OPTIONS} quiet-quick-start; then
     log_end_msg 0
       else
     log_end_msg 1
@@ -123,7 +125,7 @@
   RUNNING=$(running)
   log_daemon_msg "Stopping Postfix Mail Transport Agent" postfix
   if [ -n "$RUNNING" ]; then
-      if ${DAEMON} quiet-stop; then
+      if ${DAEMON} ${DAEMON_OPTIONS} quiet-stop; then
     log_end_msg 0
       else
     log_end_msg 1

If everything went well up to now you should be able to start your new postfix instance and check that it is actually running.

# /etc/init.d/postfix2525 start
# netstat -antp | grep 2525
tcp        0      0 127.0.0.1:2525          0.0.0.0:*               LISTEN      6138/master

2. Configure main postfix to accept emails from the intermediate
Edit /etc/postfix/master.cf and add this line at the bottom:

127.0.0.1:12525 inet n - - - - smtpd  -o smtp_fallback_relay= -o smtpd_client_restrictions=  -o smtpd_helo_restrictions=  -o smtpd_recipient_restrictions=permit_mynetworks,reject  -o smtpd_data_restrictions=  -o receive_override_options=no_unknown_recipient_checks

This defines a special port for the main postfix instance that has (or maybe it hasn’t actually) some special restrictions.
Actually you will have to change this line later on upon installing postfix-policyd, but this should be good enough for now, in order for you to do some testing.
Restart postfix

# /etc/init.d/postfix restart
# netstat -antp | grep 2525
tcp        0      0 127.0.0.1:12525         0.0.0.0:*               LISTEN      26799/master    
tcp        0      0 127.0.0.1:2525          0.0.0.0:*               LISTEN      6138/master   

The intermediate postfix listens on 127.0.0.1:2525 and the main one has another special listening port on 127.0.0.1:12525.

3. Test your intermediate postfix instance
You can do this in a gazillion different ways. One of my favorite ways to test SMTP connectivity is through telnet (—> shows data entry):

# telnet localhost 2525
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 server.mydomain.gr ESMTP Postfix
---> EHLO koko.gr
250-server.mydomain.gr
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
---> MAIL FROM: lala@koko.gr
250 2.1.0 Ok
---> RCPT TO: koko@destination.gr
250 2.1.5 Ok
---> DATA
354 End data with <CR><LF>.<CR><LF>
---> THIS IS A TEST
---> .
250 2.0.0 Ok: queued as C41E21C84FF
---> quit

If you were keeping an eye on syslog messages you should have seen some connection messages both from postfix2525 and from postfix. If everything went well your email _should_ have arrived at it’s destination. If this is true then your primary postfix instance now works as a relay for your intermediate queue.

Don’t read the next parts of this post if you haven’t previously managed this step!

4. Install and configure postfix-policyd

# aptitude install postfix-policyd

To run policyd you need to create a database and import policyd SQL schema to it. Your distro has probably already taken care of the previous step, if it hasn’t…do it manually and think about changing distro!
Then edit the config file usually located at /etc/postfix-policyd.conf. The options I chose to play with were the following:

SENDERTHROTTLE=1
SENDER_THROTTLE_SASL=1
SENDER_THROTTLE_HOST=0

Since all emails will be relayed through localhost there’s no point in throttling per host, what is needed is throttling per envelope sender.
You should manually review your desired limits though. I won’t post mine here because everyone has different needs and there’s no sane config for everyone.

Start postfix-policyd
# /etc/init.d/postfix-policyd start

If you get weird startup errors like:
postfix-policyd: fatal: didn't find priority 'LOG_IFOO', exiting
Edit /etc/postfix-policyd.conf, find the following line:
SYSLOG_FACILITY="LOG_MAIL | LOG_INFO"
and change it to (mind the removed spaces):
SYSLOG_FACILITY="LOG_MAIL|LOG_INFO"

5. Configure main postfix instance to use postifix-policyd
Edit /etc/postfix/main.cf and add this:
webclient_restrictions = check_policy_service inet:127.0.0.1:10031

Then edit /etc/postfix/master.cf again and change the line you had previously added to the bottom of the file with this:

127.0.0.1:12525 inet n - - - - smtpd  -o smtp_fallback_relay= -o smtpd_client_restrictions=  -o smtpd_helo_restrictions=  -o smtpd_recipient_restrictions=permit_mynetworks,reject  -o smtpd_data_restrictions=  -o receive_override_options=no_unknown_recipient_checks -o smtpd_sender_restrictions=${webclient_restrictions}

The difference is
-o smtpd_sender_restrictions=${webclient_restrictions}
which practically instructs postfix to use postfix-policyd for emails that arrive on port 12525, which is the port that the intermediate postfix instance uses to relay all emails.

6. Test your intermediate postfix instance again
If everything went well, the main postfix instance should now be able to enforce sender policies. Try sending a new email through the intermediate postfix again, yes using telnet, and you should pickup some new log lines at your syslog:

Dec 19 21:56:40 myserver postfix-policyd: connection from: 127.0.0.1 port: 45635 slots: 0 of 4096 used
Dec 19 21:56:40 myserver postfix-policyd: rcpt=5, greylist=new, host=127.0.0.1 (unknown), from=lala@koko.gr, to=koko@lalala.gr, size=348
Dec 19 21:56:40 myserver postfix/smtpd[9168]: NOQUEUE: reject: RCPT from unknown[127.0.0.1]: 450 4.7.1 : Sender address rejected: Policy Rejection- Please try later.; from= to= proto=ESMTP helo=
Dec 19 21:56:40 myserver postfix/smtp[8970]: C41E21C84FF: to=, relay=127.0.0.1[127.0.0.1]:12525, delay=20, delays=20/0/0.01/0, dsn=4.7.1, status=deferred (host 127.0.0.1[127.0.0.1] said: 450 4.7.1 : Sender address rejected: Policy Rejection- Please try later. (in reply to RCPT TO command))

The above means that greylisting through policyd works.

7. make PHP use your new intermediate postfix instance
PHP on linux by default uses the sendmail binary to send emails via the mail() function. That would use the main postfix instance though, so one needs to edit /etc/php/apache2/php.ini and change the following line:
sendmail_path = "sendmail -C /etc/postfix2525 -t -i"

The -C directive instructs sendmail to use the alternate config dir, so that emails will be sent to the new intermediate postfix instance and then to the main one, passing through policyd of course.

To check the queue size of the intermediate postfix:
# postqueue -p -c /etc/postfix2525/

If any PHP applications that are hosted have explicit SMTP server/port directives, then be sure to notify your clients/developers that they _MUST_ use localhost:2525 to send their emails to and not the default localhost:25. This is one of the shortcomings of the above method, if someone manually sets up his application to use the default localhost:25 his emails will get right through. But being a good sysadmin, you should monitor such behavior and punish those users accordingly!

That’s about it…with the above configuration and some tweaking to the thresholds you have very good chances of avoiding getting blacklisted because someone decided to send a few thousand spams emails. And most importantly, your normal mail service will continue to work flawlessly, no matter how big the queue of the intermediate mail server is.

Enjoy!

Reference for policyd: http://policyd.sourceforge.net/readme.html

Filed by kargig at 23:53 under Internet,Linux
Tags: antispam, blacklist, debian, email, Linux, mail, php, policyd, postfix, postfix-policyd, relay, sendmail, smtp, spam
12 Comments | 81,749 views

Say that a user has an email account at the company he works for. Before going on vacation he activates his cool “vacation auto-reply” feature that adds

Out of Office – I will be back from holidays at the end of July.

on the top and then quotes the email he was sent.

During his vacation, he receives a call and he is told he has to urgently sent an email about some financial updates. He rushes to an internet cafe and sends the email. He makes a mistake though and mistypes one of the email addresses of the recipients. Instead of sending the email to “user@domain.com” he sends it at “usar@domain.com”.

His company’s SMTP server though receives the following error message from the remote SMTP server while trying to deliver the email:

<usar@domain.com>: host mx.domain.com[1.2.3.4] said: 550 5.1.1
   <usar@domain.com>... User unknown (in reply to RCPT TO command)

This means that his SMTP server will then send an email to him informing him about the error and quoting parts if not all of the email he had previously sent. The email will likely appear to be from “postmaster@company.com” or “do-not-reply@company.com” or something similar.
It will look like this:

This is the mail system at host mail.company.com.

I'm sorry to have to inform you that your message could not
be delivered to one or more recipients. It's attached below.

For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can
delete your own text from the attached returned message.

                  The mail system

<usar@domain.com>: host mx.domain.com[1.2.3.4] said: 550 5.1.1
   <usar@domain.com>... User unknown (in reply to RCPT TO command)
Reporting-MTA: dns; mail.company.com
X-Postfix-Queue-ID: AE4812AE328
X-Postfix-Sender: rfc822; employee1@company.com
Arrival-Date: Thu,  5 May 2011 20:05:27 +0200 (CEST)

Final-Recipient: rfc822; usar@domain.com
Original-Recipient: rfc822;usar@domain.com
Action: failed
Status: 5.1.1
Remote-MTA: dns; mx.domain.com
Diagnostic-Code: smtp; 550 5.1.1 <usar@domain.com>... User unknown

From: Loyal Employee <employee1@company.com>
Date: July 5, 2011 9:05:29 PM GMT+03:00
To: User User <usar@domain.com>
Subject: Re: Financial updates

Financial data goes here

But the user has still his vacation auto-reply turned on, so when the automatic postmaster’s email reaches his mailbox, the system will automatically reply back to the “postmaster@company.com” quoting the previous email and adding his auto-reply message:

Out of Office – I will be back from holidays at the end of July.

So the postmaster@company.com currently has all the financial details that he shouldn’t!

Apart from the fact that the user was sending financial data to somebody else in a clear text email instead of an encrypted one, the second biggest mistake that the user has made was that he has enabled vacation auto-replies that quote the email he was previously sent. That’s very very wrong. If you don’t want sensitive stuff ending at the postmaster’s inbox avoid quoting previous emails in your auto-replies by all means.

Based on a true story 🙂

Filed by kargig at 12:20 under Internet,Privacy
Tags: auto-reply, email, Privacy, vacation
2 Comments | 15,183 views

Lately the amount of spam with russian encoding/charset that I was getting had increased significantly. Spamassassin’s configuration options “ok_locales” and “ok_languages” were not enough because I didn’t want to whitelist some language, but I just wanted to blacklist some.

So the solution for my problems was the addition of the following lines in the configuration of spamassassin:

header LOCAL_CHARSET_RUSSIAN Subject:raw =~/\=\?koi8-r\?/i
score LOCAL_CHARSET_RUSSIAN 7
describe LOCAL_CHARSET_RUSSIAN Contains russian charset that is not acceptable

If you want to add even more charsets:

header LOCAL_CHARSET_BLOCKED Subject:raw =~/\=\?(koi8-r|windows-1251)\?/i
score LOCAL_CHARSET_BLOCKED 5
describe LOCAL_CHARSET_BLOCKED Contains charsets that are not acceptable

Be VERY careful with these rules if you place them in the global config (/etc/spamassassin/local.cf) because if any of your users are getting emails in russian those emails will be probably marked as spam!!

Filed by kargig at 13:27 under Internet
Tags: charset, email, russian, spam, spamassassin
2 Comments | 13,655 views

GrRBL
In the beginning of the year I announced my RBL for Greek spam emails. The blacklist is growing larger by the day, thanks to some really kind people forwarding me their Greek spam emails, and has reached more than 120 IP addresses of verified Greek spammers.This alone though is not enough.

Why
Some spammers use their aDSL lines which have dynamic IPs to send their massive email “newsletters”. These people are split into 2 sub-categories. The ones that use their own PC as an SMTP server and the ones who use their ISP’s mail server as SMTP. I’ve tried to complain to some of their ISPs…some replied back saying that they were willing to look into the issue (but did nothing at all in the end) and others did not even reply to me. For both sub-categories, GrRBL is ineffective since I can’t add dynamic IPs in the blacklist nor can I add the IPs of the email servers of those major Greek ISPs.

Another category of spammers is the one that uses their gmail/yahoo accounts to send their emails. GrRBL is ineffective for this category as well since I can’t add gmail/yahoo to the blacklist…

What
So there was no alternative but to gather all those email addresses of these 2 categories above and add them to a new blacklist, one that will contain email addresses. I use this blacklist with my spamassassin configuration to eliminate Greek spam that GrRBL can’t. Each time I receive (or someone forwards me) a new Greek spam, I add the “From:” email address to this new blacklist. This new blacklist grows far more aggressively than GrRBL since it’s a lot easier to gather the data and already has more than 140 addresses.

Distribution
There are two available formats of the blacklist, one ready for use by spamassassin and another one with clear formatting ready to be used even by SMTPs to drop these spam emails without even touching your inbox.
The blacklist is currently only distributed to a group of well trusted people and it is available only through rsync with a username/password.

I don’t want to make the list completely public yet, but if you are interested you can request it at the contact email of GrRBL and I will reply to you about accessing it.

Sidenote
If you need a good tool to check a host again some RBLs, adnsrblcheck by Yiorgos Adamopoulos is the way to go (and it includes GrRBL!)

Filed by kargig at 23:15 under Linux,Networking
Tags: blacklist, email, Greek, greek spam, grrbl, Internet, Linux, Networking, spam, spamassassin
3 Comments | 18,390 views

It’s been some months now that I’ve started collecting some IP addresses of well known Greek spammers and I’ve put them on an DNSBL. I’ve named this list GrRBL. The software I use to run the list is rbldnsd.

The list is strictly moderated by me and only me and I try to be very selective on hosts I add to the list. The list contains hosts not only in .gr zone but also “foreign” hosts used to send spam messages either in Greek language or of Greek interest.

There’s a minimalistic guide on using it with spamassassin, exim, sendmail and postfix on GrRBL’s website. There are currently no statistics and no public listing of IPs in the blacklist. If there’s enough demand for statistics I might create some.

There’s also NO automatic deletion support, once an IP is in the list there’s no automatic way out. Since I am the only one adding IPs to the list, I am also the only one removing them, manually of course.

Even though I use GrRBL in all of the mail servers I own/manage, still I consider the service as beta. I don’t think it’s ever going to eat your emails, but you are still the only one responsible if this happens.

To submit new spam messages for inclusion please send me an email with FULL headers of the spam message to grrbl [at] void [dot] gr and I will try to take a look at it as soon as possible.

If you use it, or plan to, please leave a comment or even better, submit some spam messages so the list gets bigger and better.

P.S. In case you wonder, yes the list contains the IPs of the notorious sofokleous10 spammer.

Filed by kargig at 15:48 under Internet,Linux
Tags: antispam, blacklist, dnsbl, email, exim, grrbl, postfix, rbl, sendmail, spam, spamassassin
11 Comments | 11,466 views