by Daniel Patterson on June 21, 2013
I reluctantly switched to GMail about six months ago, after using many so-called “replacements for GMail” (the last of which was Fastmail). All of them were missing one or more features that I require of email:
But, while GMail has all of these things, there were nagging reasons why I still wanted an alternative: handing an advertising company most of my personal and professional correspondance seems like a bad idea, having no (meaningful) way to either sign or encrypt email is unfortunate, and while it isn’t a true deal-breaker, having lightweight programmatic access to my email is a really nice thing (you can get a really rough approximation of this with the RSS feeds GMail provides). Furthermore, I’d be happy if I only get important email on my phone (ie, I want a whitelist on the phone - unexpected email is not something that I need to respond to all the time, and this allows me to elevate the notification for these messages, as they truly are important).
Over the past several months, I gradually put together a mail system that provides all the required features, as well as the three bonuses (encryption, easy programmatic access, and phone whitelisting). I’m describing it as a “Hacker’s Replacement for GMail” as opposed to just a “Replacement for GMail” because it involves a good deal of familiarity with Unix (or at least, to set up and debug the whole system it did. Perhaps following along is easier). But, the end result is powerful enough that for me, it is worth it.
This is somewhere between an outline and a HOWTO. I’ve organized it roughly in order of how I set things up, but some of the parts are more sketches than detailed instructions - supplement it with normal documentation for all the parts. Without further ado:
Exim4 as the mail server, Courier-IMAP for mobile usage, notmuch to manage the email database+tags+search, afew for managing notmuch tagging/email moving, the emacs client for notmuch on all computers, K9-Mail on android. Mail is received by the mail server and put in a Archive subdirectory which is not configured for push in K9-Mail. The mail is processed and tagged by afew, and any messages with the tag “important” are moved into the Important subdirectory. This directory is set up for push in K9-Mail, so I get all important email right away. No further tagging can be done through the mobile device, but that wasn’t a requirement. read/unread status will be synced two-way to notmuch, which is important.
The first and most important part is having a server. I’ve been really happy with VPSes I have from Digital Ocean (warning: that’s a referral link. Here’s one without.) - they provide big-enough VPSes for email and a simple website for $5/month. There are also many other providers. The important thing is to get a server, if you don’t already have one.
The next thing you’ll need is a domain name. You can use a subdomain of one you already have, but the simplest thing is to just get a new one. This is $10-15/year. Once you have it, you want to set a few records (these are set in the “Zone File”, and should be easy to set up through the online control panel of whatever registrar you used):
A mydomain.com IP.ADDR.OF.SERVER (mydomain.com might be written @)
MX 10 mydomain.com.This sets the domain to point to your server, and sets the mail record to point to that domain name. You will also need to set up a PTR record, or reverse DNS. If you got the server through Digital Ocean, you can set up the DNS records through them, and they allow you to set the PTR record for each server easily. Whereever you set it up, it should point at mydomain.com. (Note trailing period. Otherwise it will resolve to mydomain.com.mydomain.com - not what you want!).
/etc/exim4/exim4.conf.template that looks like:.ifdef MAIN_HARDCODE_PRIMARY_HOSTNAMEAnd adding above it, MAIN_HARDCODE_PRIMARY_HOSTNAME = mydomain.com. This is so that the header that the mail server displays matches the domain. If this isn’t the case, some mail servers won’t deliver messages. At this point, you can test the mail server by sending yourself emails, using the swaks tool, or running it through an online testing tool like MX Toolbox
.forward file in the home directory of the account on the server that is going to recieve mail. It should contain# Exim filter
save Maildir/.ArchiveWhat this does is put all mail that is recieved into the Archive subdirectory (the dots are convention of the version of the Maildir format that Courier-IMAP uses).
aptitude install notmuch python-notmuchRun notmuch setup and put in your name, email, and make sure that the directory to your email archive is “/home/YOURUSER/Maildir”. Run notmuch new to have it create the directories and, if you tested the mail server by sending yourself messages, import those initial messages.
Install afew from github.com/teythoon/afew. You can start with the default configuration, and then add filters that will add the tag ‘important’, as well as any other automatic tagging you want to have.
Some simple filters look like:
[Filter.0]
message = messages from someone
query = from:[email protected]
tags = +important
[Filter.1]
message = messages I don't care about
query = subject:Deal
tags = -unreadFor the [MailMover] section, you want the configuration to look something like:
[MailMover]
folders = Archive Important
max_age = 15
# rules
Archive = 'tag:important':.Important
Important = 'NOT tag:important':.ArchiveThis says to take anything in Archive with the important tag and put it in important. Note that the folders we are moving to are prefixed with a dot, but the names of the folders aren’t. Now we need to set everything up to run automatically.
incron, to watch for changes in our .Archive inbox and add files to the database, tag them, and move those that should be moved to .Important. On Debian, you can obtain incron with:aptitude install incronNow edit your incrontab (similar to crontab) with incrontab -e and put an entry like:
/home/MYUSER/Maildir/.Archive/new IN_MOVED_TO,IN_NO_LOOP /home/MYUSER/bin/my-notmuch-new.shThis says that we want to watch for IN_MOVED_TO events, we don’t want to listen while the script is running (if something goes wrong with your importing script, you could cause an infinite spawning of processes, which will take down the server). When these occur, we call a script we’ve written. You can obviously put this anywhere, just make it executable:
#!/bin/bash
/usr/local/bin/notmuch new >> /dev/null 2>&1
/usr/local/bin/afew -nt >> /dev/null 2>&1
/usr/local/bin/afew -m >> /dev/null 2>&1It is intentionally being very quiet because output from cron jobs will trigger emails… and thus if there were a mistake, we could be in infinite loop land again.
Now let’s set up the mobile client. Install K9-Mail, and set up your account with the incoming / outgoing mail server to be just ‘mydomain.com’. Click on the account, and it will show just Inbox (not helpful). Hit the menu button, then click folders, and check “display all folders”. Now hit the menu again and click folders and hit “refresh folders”. Provided at least one message has been put into Important and Archive, those should both show up now. Open the folder ‘Important’ and use the settings to enable push for it. Also add it to the Unified Inbox. Similarly, disable push on the Inbox (this latter doesn’t really matter, because we never deliver messages to the inbox). If you have trouble finding these settings (which I did for a while), note that the settings that are available are contingent upon the screen you are on. The folders settings only exist when you are looking at the list of folders (not the unified inbox / list of accounts, and not the contents of a folder).
Finally, the desktop client. I’m using the emacs client, because I spend most of my time inside emacs, but there are several other clients - one inside vim, one called ‘bower’ that is curses based, and a few others. alot, a python client, won’t work, because it assumes that the notmuch database is local (which is a really stupid assumption). The rest just assume that notmuch is in the path. This means that you can follow the instructions here: notmuchmail.org/remoteusage/. To test, run notmuch count on your local machine, and it should return the same thing (the total number of messages) as it does on the mail server.
Once this is working, install notmuch locally, so that you get the emacs bindings. You should now be able to run M-x notmuch in emacs and get to your inbox. Setting up mail sending is a little trickier - most of the documentation I found didn’t work!
The first thing to do, in case your ISP is like mine and blocks port 25, is to change the default listening port for the server. Open up /etc/default/exim4 and set SMTPLISTENEROPTIONS equal to -oX 25:587 -oP /var/run/exim4/exim.pid. This will have it listen on both 25 and 587.
Next, set up emacs to use your mail server to send mail, and to load notmuch. This incantation in my .emacs did the trick:
(require 'notmuch)
(setq smtpmail-starttls-credentials '(("mydomain.com" 587 nil nil))
smtpmail-auth-credentials (expand-file-name "~/.authinfo")
smtpmail-default-smtp-server "mydomain.com"
smtpmail-smtp-server "mydomain.com"
smtpmail-smtp-service 587)
(require 'smtpmail)
(setq message-send-mail-function 'smtpmail-send-it)
(require 'starttls)Now eval your .emacs (or restart emacs), and you are almost ready to send mail.
You just need to put a line like this into ~/.authinfo:
machine mydomain.com login MYUSERNAME password MYPASSWORD port 587With appropriate permissions (chmod 600 ~/.authinfo).
Now you can test this by typing C-x m or M-x notmuch and then from there, hit the ‘m’ key - both of these open the composition window. Type a message and who it is to, and then type C-c C-c to send it. It should take a second and then say it was sent at the bottom of the window.
At this point, you should be pretty much set up. Pat yourself on the back - it was a bit of setup, but you now have a mail system that is more powerful than what GMail gives you, and is extensible and completely controlled by you!