At the end of last year we took the decision to significantly upgrade our two connections to LINX – our busiest connections to the outside world.

This turned out to be a good plan as Mythic Beasts got a Christmas present in the form of a new company bandwidth record, thanks to two customers, Blinkbox Music and Raspberry Pi getting a substantial spike in hits as people unwrapped their Christmas presents.

And it seems that the excitement of all the presents hasn’t worn off, as the Christmas day record has just been toppled by a new all time high yesterday. With the Blinkbox apps very high in the free music app charts, we’re not expecting it to stand for long.

We aim to give our users the choice of a range of anti-spam measures. One of the options we provider is sender verify, a simple check whereby before you accept a mail, you check that the sender of that email exists, and would accept mail from you. You can argue about how effective this is as an anti-spam measure, but it seems a perfectly reasonable check to want to make, in the same way that many people choose to not answer their phone to those who withhold caller ID.

Unfortunately, some people object to you asking the question.

We recently had some complaints from users who said that they couldn’t receive mail from people with addresses hosted on Microsoft’s Hotmail servers, and sure enough, Hotmail have blacklisted one of our servers’ IPs for daring to enquire about whether particular sender addresses were valid. This affects not just hotmail.com, but various other Microsoft domains.

Sadly, Microsoft aren’t going to change their policy for us, so we needed to whitelist them. This isn’t entirely trivial as what matters is where the sender’s email address is hosted, which means looking up the MX records for that domain. Fortunately, Exim makes this easy enough, provided that you’re not offended by curly brackets. Adding the following condition to a sender verify ACL will disable the check for Hotmail hosted domains:

!condition = ${if forany{${lookup dnsdb{>: mxh=$sender_address_domain}{$value}fail}}{match {$item}{\Nmx.\.hotmail\.com\N}}}

I should note that for quite some time, we’ve used a dedicated IP address for performing our sender verify checks in order to minimise the impact of exactly this type of blacklisting. If we hadn’t done this, the blacklist would have made it impossible for any users to send mail to Hotmail-hosted addresses too. As it was, the problem only affected users who had elected to use sender verify on their domains.

You can now configure reverse DNS for IPv6 through our customer control panel. If you’ve previously been handling reverse DNS for your allocation through delegation and would prefer to use the control panel, then please get in touch.

If you’ve got a server with us and are interested in trying IPv6 and don’t already have an allocation then please email support and we’ll be happy to provide you with a block of addresses.

Our network has supported IPv6 for a while, but recently we’ve been making a concerted effort to enable IPv6 on more of our servers. What we’ve learned (mostly the hard way) is that the challenge in doing this is not so much in enabling specific services, such as making your webserver speak IPv6, but in the less obvious side effects of bringing up an IPv6 address on the server in question. Once you do this, the server will start making outgoing connections over IPv6 where possible, and that’s when you find out all the places that you’ve got IP-based access controls squirreled away.

One that caught us out recently when we brought up IPv6 addresses on our mail servers was an SPF record that listed our outgoing servers by their IP (v4) addresses. In hindsight, including IP addresses in an SPF record was never a great idea. It would be much better to use the “mx” or “a” SPF terms, referring to mail servers by name rather than address.

To help others avoid making the same mistake, we’ve added SPF record checking to our IPv6 Health Check. The rules on this are necessarily a bit arbitrary: if you have an explicit reference to an IPv4 address, it expects you to have at least one reference to an IPv6 address. In addition, any time that you use an MX term, it expects that MX to have both IPv4 and IPv6 addresses.

For an example of this, compare the results for twitter.com with the results for google.com. We fail twitter.com because of the “mx:one.textdrive.com” term. There are other parts of Twitter’s SPF that don’t appear to have IPv6 equivalents (e.g. “_netblocks.zdsys.com”) but there’s no easy way to determine which IPv6 address block corresponds to each IPv4 address block. Suggestions for better ways to categorise these test results gratefully received.

Last night we quietly upgraded the disks in our Sphinx shell server to a pair of SSD drives. Sphinx has been suffering under heavy I/O load for a while now, and it’s safe to say that the SSDs have resolved that problem for the foreseeable future.

The upgrade was without downtime, using the magic of LVM’s pvmove command.

It’s been upgraded with a pair of fiendishly expensive server-grade SSDs. We’re not normally ones to pay too much attention to whether kit is designated as “server-grade” but in the case of SSDs it really matters due to the limited number of write cycles on SSDs. The new disks are good for 8TB of writes per day for 5 years, whereas the equivalent consumer grade version is only rated for 20GB/day, which wouldn’t last very long in Sphinx.

Sphinx has a special place in our hearts as it’s the machine on which the company was founded nearly 14 years ago, and it’s been in pretty much continuous service ever since. Of course, the current hardware has absolutely nothing in common with the dual Celeron BP6 that we deposited in a Fulham datacentre back in 2000, and it now lives in Docklands, but it’s still the same machine (right?) which is why it still says:

[pdw@sphinx ~]$ rpm -q redhat-release
redhat-release-6.1-1

(don’t worry, that’s probably the only package from RH 6.1 that we’re still using…)

As if it’s not bad enough that we have to waste a huge amount of time, not to mention a non-trivial amount of hardware, bandwidth and electricity, trying to deal with spam, we also waste a fair amount of time dealing with the “ingenious” ways that the anti-spam brigade come up with to stop legitimate mail from getting through.

This week’s contribution was so special that it took three of us to confirm that they really were doing something as silly as it first appeared.

First some background: the IP address that you get given by your ISP for your broadband connection is usually dynamically allocated. This means that it may change every time you reconnect, and may be reallocated to other users when you’re not using it. This obviously makes it impossible to selectively blacklist the users of such addresses in response to spam complaints, so it is common practice for mail servers to block connections from all IPs that are known to be allocated on this basis, using something like the PBL. Users of such IP addresses are expected to use their ISP or hosting provider’s mail servers to send outgoing mail, and the administrators of those servers take responsibility for policing their customers (on pain of having their mail servers blacklisted).

Today, a customer complained that a legitimate email being sent via our server in this manner (using authenticated SMTP) was being blocked. On closer inspection, it turned out that the IP addresses that the receiving server was objecting to was not our server’s IP address, but the sender’s IP address on the basis of it having a “poor reputation”. Well duh – it’s a dynamically allocated IP: there’s a decent chance that at some point in its life it’s been allocated to an infected computer and used to send spam. They’s why you don’t accept mail from them directly, right?

The stupid thing is that the only way that the receiver knows the originating IP address is through a “Received” header that we add. That’s right, we could trivially defeat this anti-spam measure by configuring our mail server to not add the header.

Of course we’re not going to do that, as it would break the incredibly useful trace that is provided by the received headers, and is one of the few things that helps keep sane mail admins sane.

If you want to hear what we’ve got to say, but can only cope with 140 characters at a time then you can now follow us on Twitter.

We may use Twitter to give updates on service status when we’ve got problems, but the official source for such information remains our status page.

A few days ago we unveiled our IPv6 Health Check tool, and it very quickly proved its worth.

There are plenty of other IPv6 website checkers already out there that do a cursory check to make sure that you have some IPv6 addresses for your website, nameservers, and mail servers. Our checker attempts to dig a little deeper. Do your webservers actually respond over IPv6? On all addresses? Do all your MXs have working IPv6 reverse DNS? Are your DNS entries dependent on other zones that don’t have IPv6 nameservers?

One user pointed us towards the results for one of the Regional Internet Registries, initially because it broke the checker. A few bug-fixes (both in our code, and in CPAN modules) later, and we’d determined that there was a real problem behind it: www had two AAAA records, both servers were up, but only one was accepting connections on its IPv6 address. Connections to the other server eventually timed out.

Although this issue would cause real problems to an IPv6-only user, this is exactly the kind of problem that the Happy Eyeballs (aka Fast Fallback) algorithm does a perfect job of masking. If you pick the duff IPv6 server out of the DNS, it’ll almost immediately and silently fall back on an IPv4 server. Even if you use a tool like SixOrNot to show you what connection got used, it may not be obvious that something is amiss, as falling back to IPv4 becomes part of normal operation.

Even in a dual-stack world, such a problem isn’t without side effects, as it would likely lead to an imbalance in the load spread between the two web servers.

We’re continuing to broaden the range of tests performed by the tool in order to help catch the less obvious problems that can occur when IPv6-enabling your site.

Ever wondered whether users would get to your website in an IPv6-only world? Well, now you can find out. Our IPv6 Health Check tool checks not only that your web and mail servers are accessible via IPv6 addresses, but also that you can obtain those addresses from DNS using IPv6.

You can try it out by entering a domain name below:

Should you care? Well, typing domains for a few well-known websites into the checker reveals that the IPv6 Internet isn’t currently a very exciting place, so it’s going to be a little while before not having an IPv6 presence becomes a problem. On the other hand, there’s a growing number of users with both IPv4 and IPv6 connectivity, with the latter being preferred. Google recently announced that over 2% of their traffic was now over IPv6¹, so if you’re going to list IPv6 addresses for your servers then it’s important that they work.

1. Although you still need an IPv4 connection to find Google in the first place.

I’ve been keeping an eye on the effectiveness of the blacklist that we recently installed to block spam from Communicado Ltd.

The number of messages directed at our servers seems to vary significantly, but we’ve seen close to 1,000 in a single day rejected by the filter. Whilst not a huge number in the grand scheme of things (our servers reject several connections per second using IP blacklists), it’s a pretty significant number to be spread amongst a relatively small number of customers.

What we have noticed is that the domains that we’re now seeing have been registered increasingly recently, suggesting that the older domains are becoming unusable due to people blocking them. So, we have an arms race between our ability to keep our blacklist up to date, and their ability to keep buying and deploying domains. This is actually a good thing because the domains cost the spammers at Communicado real money, so if enough people use the blacklists and keep them up-to-date then sending spam in this way will become uneconomical. In the meantime, the only consolation of the spammers chucking all this money at Nominet, is that we occassionally get to drink it.

I did some mining of our mail logs, and identified another half dozen Communicado domains. Martin fed these into his Nominet search tool yielding another 1,000 or so domains, so we’ve now got over 5,000 on the list.

If you run a mail server and aren’t already using it, please add the blacklist to your configuration and keep it up-to-date.