tldr; Apple say IPv6 support vital, we offer £20pa off any VM that is IPv6 only.

Apple have announced that as of iOS 9, all apps require support for IPv6 and must run on an IPv6 only network. The motivation is fairly clear, T-Mobile USA runs an IPv6 only network, Comcast is deploying IPv6 and on IPv6 launch day in Finland 5% of users had IPv6 enabled simultaneously. Google sees around 7% of all traffic as IPv6 now.

Like it or not, IPv6 is here and the predictions of a lengthy period of being dual stack were wrong. Nobody bothered to turn on IPv6 until IPv4 ran out, then instead of IPv6 and Network Address Translation we’re skipping quickly to IPv6 only. If your application doesn’t work on an IPv6 only network an increasing fraction of users simply can’t use it.

At Mythic Beasts we’ve been using IPv6 for a long time. Two years ago we rebuilt the hosting infrastructure for Raspberry PI to be IPv6 only for all internal connections. A future article will explain our scale up to vastly more VMs, many IPv6 only. IPv6 at Mythic Beasts isn’t an add-on, if our IPv6 connectivity breaks, customers go offline. We’re steadily working on spreading IPv6 connectivity throughout other providers.

We’ve been offering developers IPv6 only Virtual Machines for experimentation for a while, and have one of the most comprehensive IPv6 connectivity checkers for hosted software which is very good at demonstrating that enabling a v6 address isn’t quite enough.

Every single connection to this website uses IPv6.

The best way to build the hosting infrastructure today, is to have an IPv6 only network for the whole thing and a single IPv4 address on the load balancer for ‘legacy’ IPv4 connections. To give everyone an incentive to do it right, today we’re extending our IPv6 only VM offer – all virtual machines that are IPv6 only will be discounted for the lifetime of the rental.

If you’re really interested, this presentation at the North American Operators Group about the largest US ISPs moving straight to IPv6 only deployments including the information that over 20% of US users have native IPv6.

SSL Certificates do two things. They encrypt the traffic between the end user and the website, and they provide authentication that confirms the website is who they say they are. As we previously wrote about at present the authentication step is done using a piece of maths called SHA-1.

What the SHA-1 function does, is to provide a signature that says ‘The Certificate Authority confirms that the public key for Mythic Beasts is ….’. It’s extremely important that nobody else can forge this signature, otherwise anybody could present their public key instead of the Mythic Beasts public key and intercept all of the data.

Now SHA-1 has been subject to a lot of analysis by people attempting to forge keys, and slowly progress has been made. SHA-1 has not been “broken”, but thanks to improvements in mathematics and computing, the estimated cost of forging a certificate has steadily fallen from more-money-than-exists to a-large-country-could-do-it and in the next 5 years is likely to reach script-kiddy-with-a-botnet-could-do-it.

So Google, Firefox and others now refuse to accept SHA-1 based certificates that will last into 2017. Whilst you can’t forge them now, in two years time it’s likely that well funded organizations may be able to do so. As a result, the Internet has had to migrate to SHA-2, a new function that achieves the same as SHA-1 for proving authenticity but has no known attacks: forging a SHA-2 signature is currently believed to be entirely infeasible. Google’s announcement of their intention to deprecate SHA-1 was greeted with dismay and anger, but in the end had the desired effect. The certification authorities moved quite quickly to make SHA-2 the default.

At Mythic Beasts this week, we replaced our SSL certificate for all our servers. As expected, the new certificate we were issued was SHA-2 based. Deployment of the new certificate went smoothly, sufficiently smoothly that not a single customer noticed. A short time later we realised that we now didn’t seem to receive any support mail at all.

Our ticket tracking system runs on top of mono, an open source reimplementation of .NET. The older version of mono it uses doesn’t have support for SHA-2 certificates, so our tracker was seeing the secure connection, failing to authenticate and refusing to send or receive email. Briefly we worked around this by turning encryption off for the support system – as the traffic is entirely within our network we aren’t so worried about it being intercepted.

However, we know that our end-users use a wide variety of different clients for email, some of which are quite old and obscure. So we thought it was rather likely that we were breaking email functionality for existing customers with the SHA-2 certificate. We decided the sensible thing to do would be to use the new SHA-2 certificate just for websites, and obtain a new SHA-1 certificate for mail applications.

We will face the same issue again in 12 months. (Except we don’t even know if the certification authority will still offer the choice of getting a SHA-1 certificate then.) We’re hoping that a year will force a number of updates to mail clients and system libraries such that next year we can deploy SHA-2 everywhere. Eventually, we will have to draw a line, and say that if our customers’ clients don’t support SHA-2, they will have to upgrade them, or use unencrypted access.

In a little known fact, here are two old men singing about SSL security beginning with a limited understanding of SHA hashes. It delightfully uses the metaphor of a journey to meet their loved one to show how the process of security is a continuous process that can never be fully achieved.

We’re please to announced that we can now set DS records for any domains registered with us.  At present, only UK domains can be configured  through the control panel.  For any other domains, please email support and we’ll put the records in place for you.

Control panel integration and other DNSSEC improvements will be coming soon.

We’ve just rolled out a beta of our snapshot functionality for our virtual servers.  This allows you to take an instantaneous image of your servers disk space which can then be restored at a later date to either the same or a different server.  This can be used for cloning a virtual server, for backups, or just to take a copy of your server before making significant configuration changes such as an operating system upgrade.

Snapshots are stored in our distributed storage cloud, which replicates the image across three separate data centres.

The system is in beta testing at the moment, and during this beta we’re offering free storage for images.  Once the beta is complete, storage space will become chargeable, but we’ll contact all customers who’ve made use of the service prior to issuing any bills.

If you want to try it out, simply use the snapshot panel for your server in the customer control panel, or use the snapshot command on the admin console.  Hopefully it’s self-explanatory, if it’s not, tell us and we’ll make it better!

Here at Mythic Beasts it’s fair to say that our staff hold a wide spectrum of political beliefs, but I think one thing we can all agree on is that all the major political parties have at least some irredeemably stupid policies (and possibly also that some of the minor parties only have stupid policies).

This makes voting for a political party a pretty depressing prospect. So, what about voting for an elected representative who will look after our interests?

Our founders reside in two constituencies with notable MPs: Witney and Cambridge.

The MP for Witney is notable for being the Prime Minister. The MP for Cambridge, Julian Huppert, is notable for being a Liberal Democrat and yet still being highly regarded by a large number of his constituents.

Now, if you want good data on whether your MP is any good or not, you should head over to the excellent They Work For You and find out what they’ve been up to in Parliament on your behalf.

But who wants good data when you can have some anecdotes? Let’s look at two issues that have got us wound up recently.

Firstly, the EU VAT MESS, which causes us an administrative burden far in excess of the value of the affected revenue.

Julian Huppert was very active on behalf of the constituents who contacted him on this issue (Mythic only got as far as a tweet…), including submitting written questions in parliament, which received a predictably useless response.

On the other hand, Paul wrote to David Cameron twice (the first letter went AWOL), and received only a hopeless response which completely failed to address any of the issues raised.

Secondly, banning secure encryption. As a hosting company, the ability to undertake transactions securely online is quite important to our everyday business (see previous notes).

The appalling jeering by other MPs, and the pathetic response given by Theresa May, to Julian Huppert’s questions asked in Parliament demonstrated the he was clearly one of the few MPs who actually grasped the implications of the proposal, rather just resorting to rhetoric that fuels the fear that terrorism relies on.

As for David Cameron, well, it’s his idea.

So what can we conclude from this? Not a lot, except that we’d probably be in a far better place if parliament were full of representatives who listened to and understood their constituents, rather than those who get in on the strength of a party political vote.

Jessie

The new stable version of Debian, named “Jessie” was released on Saturday.  The new version is now available for use on all of our Virtual Server hosts. Jessie is fully available at the Mythic Beasts mirror and we’re included in the default menu so you can easily install directly from our mirror.

Mythic Beasts make extensive use of Debian and would like to thank all the Debian developers by donating our usual firkin of beer from the every excellent Milton Brewery to the Summer Debian UK barbeque so everyone within the Debian community can have a pint on us. Possibly more than one.

At UKNOF31 we presented a talk entitled Catastrophic Unplanned Success, a slightly rushed history of how some of the rapid scale-up of RaspberryPi from the point of view of the hosting provider, detailing some of the issues we’ve dealt with during their extremely rapid scale up, and attempting to educate the teenagers into a proper DDoS rather than the half-hearted ones they’ve tried so far.

https://indico.uknof.org.uk/getFile.py/access?contribId=5&resId=0&materialId=slides&confId=33

We believe this talk was videoed, we’ll put the video up here too once it’s published.

Following on from recent upgrades to RAM and bandwidth for our Virtual Servers, we’re pleased to announce upgrades to Virtual Server storage options.

We’ve launched a new range of SSD Virtual Servers, offering the ultimate in I/O performance. The range starts with our VPS2 SSD which replaces the 40GB disk in our standard VPS 2 with a 10GB SSD drive.

Like our spinning rust-based Virtual Servers, our SSD storage is local to the host machine, and connected as RAID 1 mirrored pairs to a controller with a battery-backup unit.  This allows us to safely enable a large write cache, further boosting write performance.

We’ve also doubled the disk space available with all of our full HDD-based Virtual Servers, so our basic VPS2 now includes 40GB of disk, 2GB RAM and 1TB of monthly bandwidth.

Existing customers can upgrade to the new storage capacity by typing “upgrade” on the admin console, and then adding new partitions or resizing existing partitions to make use of the new capacity.

Every now and again, one of our users will either get their SMTP credentials stolen, or will get a machine on our network compromised. More often than not, the miscreants responsible will then proceed to send a whole bunch of adverts for V1@gr@ or whatever through our mail servers. This typically results in our mail servers getting (not unreasonably) added to various blacklists, which affects all our users, creates work for us and generally makes for sad times.

We’ve got various measures to counter this, one of which relies on the fact that spam lists are typically very dirty and will generate a lot of rejections. We can use this fact to freeze outgoing mail for a particular user or IP address if it is generating an unreasonable number of delivery failures. The approach we use is based on the, generally excellent, Block Cracking config.

Unfortunately, both we, and the author of the above, overlooked what happens when you start adding IPv6 addresses to a file which uses “:” as its key/value separator, such as that used by Exim’s lsearch lookup. Yesterday evening, a customer’s compromised machine started a spam run to us over IPv6.

Our system raises a ticket in our support queue every time it adds a new IP to our block list so that we can get in touch with the customer quickly. Unfortunately, if the lookup doesn’t work because you haven’t correctly escaped an IPv6 address, it’ll happily keep adding the same IP for each spam email seen, and raising a new ticket each time. Cue one very busy support queue.

Needless to say, the fix was simple enough, but the moral, if there is one is a) test everything that you do with both IPv6 and IPv4 and b) start preparing for IPv6 now, as it’s going to take you ages to find everything that it breaks.

Code making assumptions about what an IP address looks like that will be broken by IPv6 are almost certainly more prevalent than 2-digit year assumptions were 15 years ago.

Some time ago we were forwarded a plea by Liz Upton who’s sort of famous on the internet for some sort of cheap computer, on behalf of World Possible, which said

This brings us to good news / bad news.  Last month we pushed through 5TB of
FTP traffic, and over 20TB of FTP traffic on the year.  That's great, about
700 RACHEL downloads - but our web host isn't as excited with our success
and cut us off yesterday.

Liz thought this was the sort of thing we might be able to help with. So we got in contact and we’ve set them up with one of our older inexpensive servers to act as a new host. As it’s an educational project that we’d like to support; we thought we’d donate some bandwidth to help out. Since it nicely coincided with a substantial bandwidth upgrade in our Cambridge data centre we’d put the service there.

So far they seem pleased!

which is handy because some of their other suppliers who pay Amazon rates for bandwidth were a little bit annoyed with them.