I’ve been keeping an eye on the effectiveness of the blacklist that we recently installed to block spam from Communicado Ltd.

The number of messages directed at our servers seems to vary significantly, but we’ve seen close to 1,000 in a single day rejected by the filter. Whilst not a huge number in the grand scheme of things (our servers reject several connections per second using IP blacklists), it’s a pretty significant number to be spread amongst a relatively small number of customers.

What we have noticed is that the domains that we’re now seeing have been registered increasingly recently, suggesting that the older domains are becoming unusable due to people blocking them. So, we have an arms race between our ability to keep our blacklist up to date, and their ability to keep buying and deploying domains. This is actually a good thing because the domains cost the spammers at Communicado real money, so if enough people use the blacklists and keep them up-to-date then sending spam in this way will become uneconomical. In the meantime, the only consolation of the spammers chucking all this money at Nominet, is that we occassionally get to drink it.

I did some mining of our mail logs, and identified another half dozen Communicado domains. Martin fed these into his Nominet search tool yielding another 1,000 or so domains, so we’ve now got over 5,000 on the list.

If you run a mail server and aren’t already using it, please add the blacklist to your configuration and keep it up-to-date.

We’ve just put in place a new spam blacklist. What’s unusual about this list is that it’s a list of sender domains. Filtering on sender domains is not normally something we’d consider since standard operating procedure for spammers is to use a fake sender address containing a legitimate domain.

In this case, it seems that a particular company, Communicado Ltd, has gone to the trouble of registering a very large number of UK domains specifically for the purposes of spamming. We first noticed this in response to a customer complaint last week, and the thing that got my interest was that the various different domains had valid SPF records. For example:

$ host -t txt hurvabne.co.uk
hurvabne.co.uk descriptive text "v=spf1 a mx ip4:76.73.88.0/24 ip4:76.73.91.0/24 ~all"

SPF isn’t the world’s greatest anti-spam measure, but a pass on a record with specific IPs is generally a pretty positive indicator, as you have to either compromise the mail server or the DNS server – or own the domain.

What I didn’t twig was just how many domains were involved, or that they were all owned by the same company, until one of my colleagues came across Martin A. Brooks’ blog post, which identifies over 4,500 domains owned by this single outfit.

Martin has kindly shared his list of domains and we’re now filtering using it.

For what it’s worth, those domains will have cost them the best part of £17k for a year’s registration (somehow, I don’t think they’ll be renewed).

We’ve just extended our range of Virtual Servers to include a 512MB “VS Lite” option for just £7 + VAT per month – or the equivalent of £5.83/month if you pay annually.

We’ve been doing Virtual Servers for 10 years now, and we’ve always hosted them on machines with hot-swappable hardware RAID, meaning that we can replace failed disks without customers even being aware that there was a problem. One of the great things about virtualisation is that reliability shares really well – but it still adds a bit to the cost

A large proporption of our Dedicated Server customers use machines without hot-swappable drives, and we thought we’d extend the option of using cheaper hardware to our Virtual Server range in the form of the VS Lite, our most affordable Virtual Server yet.

The VS Lite host hardware still has monitored RAID, so we can cope with disk failures without data loss^*, but it requires downtime in order to replace the drive.

Our standard Virtual Servers start at £12.50 + VAT per month for a 1GB server, and all run RAID-10 on hot-swappable drives, and offer a choice of data centres, allowing you to spread multiple servers across geographically diverse sites.

In the unlikely event that even £7/month sounds expensive, then we’ve got an even more affordable alternative with a bit of a twist up our sleeves which we’ll be unveiling shortly. The first person to guess what we’re planning can have one free for a year, and don’t worry, as with all our Virtual Servers you can upgrade and downgrade easily, so don’t be put off by the prospect of an even better offer in the pipeline!

On the other hand, there are no prizes for guessing what hardware we’re using for the VS Lite host servers…

* RAID is not backup.

Last year we announced some improvements to the Mythic Beasts DNS API, and I asserted that this made it good for implementing a Dynamic DNS service. Dynamic DNS is simply a mechanism for programmatically updating a DNS record, typically used to provide a consistent name for a computer that is at the end of an internet connection with a dynamically assigned IP address.

Well, last weekend I had the opportunity to try implementing a Dynamic DNS service with our API, and realised that it actually makes the task unduly difficult. It can be done, but in order to change a record, you need to remove the old record, and to remove the old record you need to know what it is currently. This meant that you had to use the LIST command, grep out the old record, and then issue the necessary DELETE and ADD commands. Aside from being hassle, it introduces an unavoidable race condiition between the LIST and DELETE commands.

We’ve now implemented the obvious fix: a REPLACE command, which replaces all existing records for the specified host and type, and replaces them with the one provided. Obviously this doesn’t work if for some reason you want multiple records for a single host, but for the obvious use case it means that Dynamic DNS can be handled in a single command:

curl --data "domain=MY_DOMAIN&password=MY_PASSWORD&command=REPLACE \
myhost 300 A 1.2.3.4" https://ctrlpanel.mythic-beasts.com/customer/primarydnsapi

The DNS API is a standard feature included with all Mythic Beasts domain registrations. Full documentation can be be found here.

We’ve had an IPv6 aware network for quite some time, and we’ve been gradually rolling it out to our services with the aim of eventually having every service we offer fully available over IPv6 and IPv4. We host the Raspberry Pi website which has an IPv6 only internal network, IPv6 only virtual machines and IPv4 on the front end to help out those of you with the ‘legacy’ internet.

A quick skim over the logfiles suggests that about 96% of you still access the site through the legacy IPv4 network – about 4% of hosts are now connecting over IPv6 which is starting to become a non trivial fraction of the traffic. Of course this is much higher than typical sites, Raspberry Pi users are much more technically aware than the general population.

Yesterday we had our first real connectivity problem to investigate – an end user within Ja.net (the UK academic network) was unable to access files from the Raspberry Pi download server on about half of the occasions. Further investigation showed that they could access the load balancers in our Sovereign House site with connectivity via the London Internet Exchange Juniper LAN, but not the load balancers in our Harbour Exchange site with connectivity over the London Internet Exchange Extreme LAN.

When we started investigating we confirmed that it seemed to be a problem with the Extreme LAN, if we forced the connectivity via the Juniper LAN it worked from both sites, if we forced it via the Extreme LAN it failed from both sites. Odder and odder though, a packet dump on our LINX interface didn’t show us passing the packets on.

Our IPv4 peering worked fine, this was IPv6 specific.

We then started looking at the routing table on the router. Over IPv4 it looks like

131.111.0.0/16 via 195.66.236.15 dev eth0

and over IPv6 it looks like

2001:630::/32 via fe80::5e5e:abff:fe23:2fc2 dev eth0

That gives us the netblock, and the next hop to send the packet to.

So the next step is to check you can reach the gateway happily enough.

# ping 195.66.236.15
PING 195.66.236.15 (195.66.236.15) 56(84) bytes of data.
64 bytes from 195.66.236.15: icmp_seq=1 ttl=64 time=0.220 ms


and

# ping6 fe80::5e5e:abff:fe23:2fc2
connect: Invalid argument


Odd. Then I realised that fe80:: in IPv6 means a link local address – the address is specific to the network card so to ping it you have to specify the destination address and the network interface.

# ping6 fe80::5e5e:abff:fe23:2fc2 -I eth9
PING fe80::5e5e:abff:fe23:2fc2(fe80::5e5e:abff:fe23:2fc2) from fe80::21b:21ff:fe65:a4c5 eth9: 56 data bytes
64 bytes from fe80::5e5e:abff:fe23:2fc2: icmp_seq=1 ttl=64 time=0.451 ms


Then the penny dropped. The routing table has eth0 in it but we’re actually connected to eth9. Under IPv4 this is fine because the next-hop address is globally unique and only accessible over eth9 so we send the packets out of eth9 and they go to the correct destination. Under IPv6 it’s a link local address and therefore valid over any interface, so we obey the routing table and throw the packets out of eth0 whereupon they fall onto the floor because there’s no fibre connected.

Fixing the config to put the right interface description in made it all work, and our end user is happily able to access all the load balancers on all the v6 addresses in all of the buildings.

Obviously if you’re a Mythic Beasts customer and you don’t already have an IPv6 allocation for your real or virtual server, drop us an email and we’ll hand you your own address space to play with.

So our newest Mythic Beast started working for us today. The first task is to start installing your new laptop and reading and signing employment contracts. Today we had our newest employee fail at the first hurdle.

The laptop in question is a shiny Toshiba z930. This one came with Windows 8 and a fully charged battery. On first powering it on it comes up with the Windows 8 licence. This has a tickbox option for ‘I accept the license’ and a big button labelled ‘Accept’ to click on.

If you don’t tick the box, it tells you you have to. There’s no option to reject the license.

If you press the power button the laptop suspends itself. If you press and hold the power button the laptop still suspends itself. Ctrl-Alt-Delete doesn’t work. You can’t remove the battery as it’s built in. In frustration our newest employee suggested pouring his coffee over the damn thing to make it power cycle. This was a really stupid idea, not only does the laptop have a spill proof keyboard he’d also then have no coffee.

The best plan we could come up with was to wait for the batteries to run out which requires pressing a key about every five minutes to stop the thing suspending itself.

We’ve upgraded our DNS resolvers in our SOV and HEX data centres. New features include DNSSEC validation and IPv6.

The addresses are,

SOV : 2a00:1098:0:80:1000::12 / 93.93.128.2
HEX : 2a00:1098:0:82:1000::10 / 93.93.130.2

They’re now DNSSEC aware and validating resolvers. That means if a site has correctly configured DNSSEC and we receive an answer that fails the security check we will return no answer rather than an incorrect/forged one.

To demonstrate the difference,

a non dns sec validating resolver :
# dig +short sigfail.verteiltesysteme.net
134.91.78.139

a mythic beasts server using our resolvers
# dig +short sigfail.verteiltesysteme.net
<no answer>
#

and on the DNS server it logs an error,

debug.log:28-Aug-2013 15:44:57.565 dnssec: info: validating @0x7fba880b69e0: sigfail.verteiltesysteme.net A: no valid signature found

and correctly drops the reply.

Googles DNS servers on 8.8.8.8 work the same as ours so we’re fairly confident that there will be no compatibility issues.

With a customer of ours we have set them up their own full BGP network, split across two of our London sites. With advice from us we have

• Helped them join RIPE as an LIR
• Helped them apply for an IPv6 /32 and an ASN
• Set up a full BGP IPv6 only network
• Helped them apply for a final /22 of IPv4 space
• Configured this in the global routing table

They have the option to now cable or fibre direct to peering exchanges and other ISPs should they wish to do so on individual machines hosted within our rackspace. In the mean time they’re taking advantage of our co-location, out of band access to their routers via serial and our IPv4 and IPv6 transit.

We’ve now joined the London Internet Exchange and are present on both of their peering LANs for redundancy. We’re connected to the Juniper LAN in Sovereign House and the Extreme LAN in Harbour Exchange. We’re now connected to three peering exchanges – Edge-ix,LoNAP and LINX-juniper in Sovereign House, and two – LINX-extreme and LoNAP in Harbour Exchange.

You can see the current traffic over the LINX public exchanges here

which is best described as rather a lot. We’re in the process of setting up more direct peers in addition to the route servers which provided immediate peering with hundreds of ISPs and tens of thousands of routes. So many UK destinations are now a few hops shorter – which probably won’t be very noticeable – but we have improved redundancy and increased capacity.

Over the last twelve months we’ve made a series of networking changes and completely failed to blog about them. Our first announcement is that we now have a dark fibre ring around our core London sites.

This isn’t actually true. We now have a lit fibre ring around our core London sites. It’s currently running at 10Gbps and connects all of our routers together. All our routers connect to the local networks at 10Gbps so our entire network core is now 10Gbps. We also have some direct customer connections who are using our fibre as a layer 2 interlink between Telecity Sovereign House, Telecity Meridian Gate and Telecity Habour Exchange 6/7. Our standard is to offer a pair of ports in each site on redundant switches (so 6 x 1Gbps ports) with unlimited traffic between them.

As a result of our upgrade we’re able to continue to offer free traffic between all London hosted servers irrespective of the building the machines are in or which customer owns them – we bill only for traffic that leaves our network. Upgrading to progressively higher bandwidths is now straightforward as we can add CWDM / DWDM as required to increment in multiples of 10Gbits, or to 40Gbits or multiples of 40Gbits.

For those of you that are interested, the fibre lengths are

• MER <-> SOV : 1672ns (or 1122ft)
• SOV <-> HEX : 6423ns (or 4310ft)
• HEX <-> MER : 5456ns (or 3687ft)

and the latencies across the network from core router to core router (average over 10 pings) are

• MER <-> SOV : 0.096ms
• SOV <-> HEX : 0.082ms
• HEX <-> MER : 0.076ms

and from customer machine in SOV to customer machine in HEX, passing through at least two routers – 0.5ms.