One of the most powerful ways of stopping spam entering users mailboxes, is to stop being received in the first place!
The way email is transmitted on the internet is via the SMTP protocol. So to try and stop spam entering the FastMail system, we apply a number of checks at the SMTP stage to stop spam senders.
RBLs (real-time blackhole lists) are a very common way to try and detect and stop known spam sending machines. The main problem with RBLs is that they often have "false positives", and block systems incorrectly causing email to be incorrectly bounced back to senders.
To stop that happening, FastMail only uses only one RBL for blocking, the SpamHaus XBL. The XBL is a highly accurate block list that lists the IPs of machines with known trojans and proxies. Independent testing shows that the ZEN RBL (of which XBL is a part) has a high block rate, with basically no false positives.
A number of other RBLs and URIBLs are used during the spam checking phase that occurs after email is receive, but these RBLs never outright block email, they're used as part of an overall scoring heuristic along with many other factors.
Address enumeration detection (AED) is designed to stop other people trying to find what addresses are valid email addresses at FastMail. It does this by trying to detect attempts to send to many different similar email addresses in a short period of time from a non-email server host.
Address enumeration detection provides a defense against spammers trying to discover valid email addresses simply by trying many different addresses over and over.
When a server is transferring email from one system to another, there's a series of conventions they have to follow defined in the SMTP RFC 2821. Again, spam bots are often poorly implemented, or try and use tricks to work around this conventions. By detecting some of the common violations, more spam bots can be detected and blocked
Through our large and long term experience with running an email service, we've been able to closely analyse how spam bots try to deliver email to our servers. By using this data, we're able to pre-emptively detect spam bots early only after they connect, and block them from delivering their spam email at all
Truedomain is an anti-phishing solution that provides a trust network built on DKIM signing of emails. For domains protected by Truedomain, we reject emails that don't have a DKIM signature or the signature is forged as these are clearly forged emails likely to be phishing attempts.
Greylisting is a process designed to detect and stop spam being accepted from poorly written sending email servers, of which almost all spam bots are. One of the main concerns users have with greylisting is that poort implementations will often delay all email. The FastMail implementation uses a number of tweaks to ensure that legitimate email is delivered with no delay, while spam email is delayed or stopped from delivery entirely.
Greylisting works by returning a temporary failure response (4xx code) to the first attempt to deliver an email, but accepts it on the second attempt. Every proper email server will attempt to redeliver a message after a temporary failure response, while currently almost all spam software does not, thus effectively blocking the emails from the spam software.
To avoid delaying proper email servers, our greylisting implementation performs a number of checks on a connection before applying the greylisting policy:
- We only greylist hosts that appear to be dialup/dsl hosts of some sort, or hosts that don't have any valid reverse DNS. This ensures that the vast majority of email servers are immediately not subject to greylisting, and their email is not delayed
- If a host has been greylisted, and it successfully passes greylisting twice in a 24 hours period (e.g. it correctly attempts to re-deliver a piece of email twice in 24 hours), then that host is whitelisted and not subject to greylisting for the next 24 hours. If it continues to deliver emails, each new delivery will extend the whitelist period. This means that any real email servers (a real email server will always retry) connected via dialup/dsl will quickly be whitelisted and not subject to email delays
- If a host opens an SMTP session with a HELO that is not an IP address, is not the same reverse DNS as it's connecting IP, but the forward DNS of the name does resolve to the connecting IP, then that host is not subject to greylisting (as suggested by "hadaso" on the forum). An example: The machine at IP 188.8.131.52 connects to us. The reverse DNS for 184.108.40.206 is 206-223-169-73.beanfield.net, which looks like a common dialup/dsl IP name, and would be a candidate for greylisting. However, the machine advertises itself to us with a "HELO mx3.hub.org" line. Doing a forward lookup of mx3.hub.org gives the IP 220.127.116.11, which is the same as the connecting IP, so we exclude it from greylisting
- If the SMTP MAIL FROM address passed in the SMTP session is in the users address book, then the email is not greylisted
When combined, these features provide an excellent balance of greylisting hosts which should not be sending email, and allowing those hosts which should be sending email to get their email straight through.
Additionally, to help with the tracking of any problems, once a message passes greylisting and is accepted, a new header is added "X-Spam-greylist". This header tells you how many seconds the email was delayed and whether that host has been whitelisted for 24 hours. (Technical: Actually, the delay figure indicates the length of the last delay for the ip/sender/recipient combination, so in the case of multiple emails from the same person, to the same person, from the same machine in a short time period, the figure will be a bit messy and hard to calculate.)
If you feel that email is being delayed by greylisting, please check the headers of your email for the "X-Spam-greylist" header first. If it's not present, then the email was NOT delayed by greylisting, and it was probably just held up on the sender's side, something beyond our control.
The result of all these tests is that much spam is blocked before it even enters our systems. See the domain management page for an example of how these systems improved spam filtering for a user when they pointed the MX records of their domain directly to us.