Integration can be hard work

Last year we had a busy time acquiring Retrosnub, BHost and VMHaus. We’ve been steadily making progress in the background integrating the services the companies provide to reduce costs and complexity of management. We can now also announce our first significant feature upgrade for VMHaus. We’ve deployed a new virtual server cluster to our Amsterdam location and VMHaus services are now available in Amsterdam. VMHaus is using Mythic Beasts for colocation and network and in Amsterdam they will gain access to our extensive set of peers at AMSIX, LINX and LoNAP. Per hour billed virtual servers are available from VMHaus with payment through Paypal.

As you’d expect, every VM comes with a /64 of IPv6 space.

In the background we’ve also been migrating former-BHost KVM-based services to Mythic Beasts VM services in Amsterdam. Shortly we’ll be starting to migrate former-BHost and VMHaus KVM-based services in London to new VM clusters in the Meridian Gate data centre.

Sympl is so simple it’s even usable by Cambridge graduates

We’re pleased to announce that we are now supporting the Sympl open source project.  Sympl is a fork of Symbiosis, a platform that makes hosting websites and email on a virtual or dedicated server simple.  Once installed, configuring a new website, or creating a new email address and mailbox, is as simple as creating a new directory.  Web server, mail server and DNS configuration is all taken care of for you.

We’ve already taken the first steps towards integrating Sympl into our infrastructure by implementing support for our DNS API in OctoDNS.  For our next step, we will be adding support for OctoDNS to Sympl.  This means that it becomes possible to use Sympl with our DNS infrastructure, but equally you can use any other provider supported by OctoDNS (we don’t believe in lock in!)

We’re now very pleased to welcome Paul Cammish, the newest member of the Mythic Beasts team.  Paul has considerable experience, having worked at a number of different ISPs since 2000, most recently at Bytemark.  Paul created the Sympl project earlier this year, in order to provide ongoing support and enhancements for the platform.

We’re very excited by the possibilities that Sympl provides, and have some interesting ideas for future developments once we’ve dealt with the immediate priorities of DNS integration, and support for the upcoming Debian Buster release.

The source code for Sympl is now available in our self-hosted GitLab instance.

We’ve just rolled out a major overhaul of our DNS management interface. We hope that you’ll find the new interface faster and easier to use. As well as improvements to the user interface, we’ve also added the ability to import zone files. This means that if you’ve got a domain that is currently hosted with another provider, you can now easily transfer all of your DNS configuration to our servers in bulk (provided that you can get them to give you a copy of your current zone file).

Our DNS management interface is included with all domain registrations.  It’s also available for domains registered elsewhere for customers of our other services, including hosting accounts, virtual servers, dedicated servers and Raspberry Pi servers.

The DNS interface includes DNS API access, allowing you to support dynamic DNS and to automate other DNS management tasks.

We believe in retaining customers through good service rather than lock-in, so naturally there’s a corresponding zone file export feature.

X now does Y 🙂

— Mythic Beasts (@Mythic_Beasts) 19 March 2019

Sensible people don’t like to hard code IP addresses in lots of different places in DNS. Better to assign it a name, and then reference that name, as it makes it clearer what’s what and if you ever need to change that IP, you’ve only got to do it one place.

CNAME records can be a good way to do this, by aliasing a DNS name to an IP. Unfortunately, the DNS specs prevent you using CNAMEs in various places that you might want to, most commonly at the root-level of your domain (the dreaded “CNAME and other data” problem).

This is where ANAME pseudo-records come in. They look just like a CNAME record, but rather than being added to the DNS, our server converts them into A and AAAA records. This allows you to get the benefits of a CNAME in places where a CNAME is not legal.

This week a customer suggested another use for ANAME records that we’d not previously thought of: round robin DNS. That is, a single DNS name that points to multiple servers. As you can’t have multiple CNAME records for the same hostname, implementing round-robin DNS means hard-coding A and AAAA records into your zone file. Like this:

proxy.mythic-beasts.com. 3600	IN	A	93.93.129.174
proxy.mythic-beasts.com. 3600	IN	A	46.235.225.189
proxy.mythic-beasts.com. 3600	IN	AAAA	2a00:1098:0:80:1000:3b:1:1
proxy.mythic-beasts.com. 3600	IN	AAAA	2a00:1098:0:82:1000:3b:1:1

Which is messy. Wouldn’t it be nicer to use the names of the servers involved? Like this:

proxy.mythic-beasts.com. 3600	IN	CNAME	 rproxy46-sov-a.mythic-beasts.com.
proxy.mythic-beasts.com. 3600	IN	CNAME    rproxy46-hex-a.mythic-beasts.com.

Sadly, the spec says you can’t do that, but thanks to a minor tweak to our DNS control panel code, you can now do it with ANAME records. Simply specify multiple ANAME records for your host name, and we’ll go and find all A and AAAA records for all of the hosts that are referenced.

Thanks to @grayvsearth for the suggestion on this one.

ANAME records are available in our DNS management interface, which is included with all domain registrations, and available for free on other domains for customers of other services. Other features include a DNS API, allowing you to obtain Wildcard Let’s Encrypt certificates.

The art warehouses in Amsterdam look much prettier than the data warehouses.

Back in July 2018, Mythic Beasts acquired Bhost, giving us additional virtual machine (VM) clusters in London, Amsterdam and California.

Today we’re pleased to announce that we’ve deployed a substantial new VM cloud to Amsterdam, running our own VM platform. Virtual machines in Amsterdam are available to purchase immediately through our website in sizes from 1GB/1vCPU to 160GB/12vCPUs, and with both SSD and spinning rust disk options. Server management and backup options are also available.

Thanks to Brexit-related regulatory uncertainty, some of our existing clients informed us that they must be hosted outside of the UK before 29th March. Deploying capacity on our own platform in Amsterdam means that we can migrate virtual servers directly to the new location.

Once we’ve dealt with the immediate Brexit-driven server moves, we’ll be looking at migrating former-Bhost VMs into this new cloud, giving a significant performance boost in the process.

Deploying the Amsterdam VM cloud is a significant milestone in the integration of the Bhost infrastructure into our own. The integration provides improved performance and redundancy for both Mythic Beasts and Bhost customers whilst simultaneously cutting our operating costs. In preparation for this, we completed upgrades to our core network last October. The existing fibre ring around our three main London sites, which is currently lit at 50Gbps, is now complemented by a 10Gbps ring around London (HEX) ⟺ Cambridge ⟺ Amsterdam ⟺ London (MER). This replaces the old 2x1Gbps connectivity from Cambridge to London with diverse 10Gbps feeds to London and Amsterdam. Our network has gained an additional 10Gbps transit in Amsterdam (NTT) and we are also now connected on the Amsterdam Internet Exchange (AMS-IX).

On a trip to deploy new routers, Pete even managed a tour of the city on foot in just over three hours.

A Mythic Beast eating a Retrosnub (artists impression)

Just before Christmas we were approached by Malcolm Scott, director of Retrosnub, a small cloud hosting provider in Cambridge. His existing connectivity provider had run out of IPv4 addresses. They’d decided to deal with this issue by adding charges of £2 per IPv4 address per month to encourage existing customers to return unused IPv4 addresses to them. As a cloud hosting provider with a substantial number of virtual machines (VMs) on a small number of hosts this had the result of tripling the monthly colocation bill of Retrosnub.

Aware of my presentation on IPv6-only hosting at UKNOF, Malcolm knew that opportunities for significant expansion were severely limited due to the difficulty of obtaining large amounts of IPv4 address space. Retrosnub faced a future of bankruptcy or remaining a very niche provider. His connectivity providers seemed strongly in favour of Retrosnub going bust so they could reclaim and re-sell the IPv4 space for higher margin services.

There are no expansion opportunities for new cloud hosting providers.

As a larger provider with our own address space, we had sufficient spare capacity in our virtual machine cloud to absorb the entire customer base of Retrosnub with no additional expenditure. Our work in supporting IPv6-only virtual machines will also make it easier to significantly reduce the number of IPv4 addresses required to support Retrosnub services. We formed a deal and agreed to buy the customer base of Retrosnub.

Combining operations

Since agreeing the deal, we’ve been working hard to merge our operations with minimum disruption.

The top priority was the domain name services because domains expire if you don’t renew them. Doing a bulk transfer of domain names between registrars is something which Nominet, the body responsible for UK domains, makes extremely easy, as it just requires changing the “tag” on all the domains.

Unfortunately, just about all other TLDs follow a standard ICANN process, which requires that a domain be renewed for a year at the time of transfer, and that the owner of the domain approves the process. If you were designing a process to destroy competition in a market by making it hard for resellers to move between registrars, it would look quite like this.

We’ve now got the bulk of domains transferred, and the next steps will be to migrate the DNS records from Retrosnub to Mythic Beasts so that our control panel can be used to change the records.

At the same time, we rapidly formulated a plan to migrate all the virtual machines in to stem the financial losses. Moving the VMs required an unavoidable change in IP address, and we also wanted to get them migrated from their current platform (Citrix Xenserver with para-virtualisation) to our own platform (KVM with full hardware virtualisation).

In order to ease the transition, we arranged for a pair of servers to do IP forwarding: a server in our cloud that forwarded the new IP to the VM in the Retrosnub cloud until it was migrated in, and another in the Retrosnub cloud that forwarded the old IP after the server had been moved. By doing this we were able to give customers a one week window in which to complete their IP migration, rather than forcing it to be done at the time that we actually moved the VM.

In the process of this migration, all customers received a significant bandwidth upgrade and majority received disk, RAM and CPU upgrades too.

We completed this on schedule before the quarterly colocation bill arrived, so instead of paying the much increased bill, we cancelled the contract and removed the servers from the facility.

Next steps

Our next step will be to migrate all the web and email hosting customers into our standard shared hosting environment. This has some time pressure as Google have plans for Chrome to start marking all non-HTTPS websites as insecure. We offer one click HTTPS hosting using Let’s Encrypt on all of our hosting accounts.

Today is Pi Day where we celebrate all things mathematical. Today is a super special Pi day, because a new Raspberry Pi has been released.

It takes the previously excellent Raspberry Pi 3 (or 3B, to give it its full name) and upgrades it with an extra 200Mhz of CPU speed and gigabit ethernet over USB 2. It fixes many of the netboot issues which Pete highlighted at the last big Pi Birthday Party and will soon have a new smaller and cheaper Power over Ethernet HAT. These new features are of particular interest for our Raspberry Pi Cloud service, as we use netbooted Pis, with network file storage and Power over Ethernet to enable remote powercycling.

Raspberry Pi 3B+.

We’ve had one to play with, and we’ve run our favourite benchmark – Raspberry Pi’s own website. We installed the full stack (MySQL, WordPress & PHP7) under Debian Stretch onto a Pi 3B and a Pi 3B+, and tried it out with 32 concurrent connections. We’re running near identical setups on the two servers: both have their files stored over the network on an NFS file server and it’s the same operating system and applications; only the kernel differs.

The new model is about 15% faster than the old one which is almost exactly as expected from the boost in clock speed; WordPress is CPU limited.

Checksumming the 681MB database file shows up the gigabit ethernet rather effectively. All our storage is over the network so reading files is a benchmark of the network speed.

This is very nearly twice as fast as the previous model.

When is it coming to the Raspberry Pi Cloud?

The Raspberry Pi 3B+ is an obvious upgrade for our Raspberry Pi Cloud. We need to wait for the PoE HAT to become available. That will allow us better density and lower capital costs. However, the 3B+ consumes more power than the 3B so we need to do some thermal and airflow work before we can make it generally available.

As reported by The Register, sites which do not use HTTPS will soon be actively labelled as “insecure” by the Chrome browser. HTTPS is the secure form of HTTP that makes the little green padlock appear in browsers.

Ultimately, sites which use HTTP are going to be labelled like this:

Not subtle, eh?

The Reg article suggests that initial changes will be deployed July 2018, and will be a little more subtle, but with Chrome having 55-60% market share, it really is time to switch your website to HTTPS.

Fortunately, if you’re hosted with Mythic Beasts this is really easy.  All of our hosting accounts include free SSL (aka TLS) certificates (provided by Let’s Encrypt), and you can enable HTTPS hosting by just clicking a button in the control panel.  Here’s how:

Enabling HTTPS for your Mythic Beasts-hosted website

First, log in to our customer control panel, click on “Hosting and shell accounts”, and click through to the hosting account for your site.  Now find your site in the list, and click on “web settings”:

If you have both a “www” prefixed and bare version, as above, you’ll want to do both. 

On the web settings page, scroll down to the “security” section:

You almost certainly want the third option: this will enable HTTPS hosting, and ensure that users see the secure version of the site by default.  (Once you’re happy that your HTTPS site is working exactly as you want it, you could consider switching to the fourth option).

Click, hit “save changes”:

We’ve got plans to make this faster, but for the moment, you’ll need to wait a few minutes.  We’ll go and obtain a certificate for your site, and once installed update your site so that it redirects to the HTTPS by default.

Bingo!

If you haven’t got a working HTTPS site within 10 minutes, email us – we’re here to help.

Any gotchas?

The instructions here will only work if the HTTP version of your site is hosted by Mythic Beasts.   If you’re configuring a new site with Mythic Beasts, make sure that you can access your site via HTTP before enabling HTTPS.

If you’re transferring a site to us that is already using HTTPS, please see our transfer in instructions for how to do this with an interruption to service.

Managed hosting

We’ve been deploying HTTPS as the default for customers of our managed services for some time. We’re going to be doing an audit of all managed sites to warn customers of this upcoming change, but in the meantime, if you’re a managed customer with an http site, just email us and we’ll sort it out.

We don’t need these Giant Scary Laser stickers yet.

We’ve recently upgraded both our LONAP connections to 10Gbps at our two London POPs bring our total external capacity to 62Gbps.

We’ve been a member of LONAP, the London Network Access Point, since we first started running our own network. LONAP is an internet exchange, mutually owned by several hundred members. Each member connects to LONAP’s switches and can arrange to exchange traffic directly with other members without passing through another internet provider. This makes our internet traffic more stable because we have more available routes, faster because our connections go direct between source and recipient with fewer hops and usually cheaper too.

Since we joined, both we and LONAP have grown. Initially we had two 1Gbps connections, one in each of our two core sites. If one failed the other could take over the traffic. Recently we’ve been running both connections near capacity much of the time and in the event of failure of either link we’d have to fall back to a less direct, slower and more expensive route. Time to upgrade.

The upgrade involved moving from a copper CAT5e connection to optic fibre. As a company run by physics graduates this is an excellent excuse to add yet more LASERs to our collection. Sadly the LASERs aren’t very exciting, being 1310nm they’re invisible to the naked eye and for safety reasons they’re very low powered (~1mW). Not only will they not set things on fire (bad) they also won’t blind you if you accidentally look down the fibre (good). This is not universally true for all optic fibre though; DWDM systems can have nearly 100 invisible laser beams in the same fibre at 100x the power output each. Do not look down optic fibre!

The first upgrade at Sovereign House went smoothly, bringing online the first 10Gbps LONAP link. In Harbour Exchange proved a little more problematic.  We initially had a problem with an incompatible optical transceiver. Once replaced, we then saw a further issue with the link being unstable which was resolved by changing the switch port and optical transceiver at LONAP’s end. We then had further low level bit errors resulting in packet loss for large packets. This was eventually traced to a marginal optical patch lead. Many thanks to Rob Lister of LONAP support for quickly resolving this for us.

With the upgrade completed, we now have two 10Gbps connections to LONAP, in addition to our two 10Gbps connections into the London Internet Exchange and multiple 10Gbps transit uplinks, as well as some 1Gbps private connections to some especially important peers.

To celebrate this we’re dropping our bandwidth excess pricing to 1p/GB for all London based services.  The upgrades leave us even better placed to offer very competitive quotes on high bandwidth servers, as well as IPv6 and IPv4 transit in Harbour Exchange, Meridian Gate and Sovereign House.  Please contact us at sales@mythic-beasts.com for more information.

tl;dr: SRS changes the sender address when you forward email so it doesn’t get filed as spam.

We’ve just deployed an update to our hosting accounts that allows you to enable Sender Rewriting Scheme when forwarding mail for your domain.

We’ve previously mentioned how we’re seeing increased adoption of Sender Policy Framework (SPF), a system for ensuring that mail from a domain only comes from authorised servers. Whilst this may or may not reduce spam, it does very reliably break email forwarding.

If someone at sender.com sends you an email to you at yourdomain.com and you forward it on to your address at youremailprovider.com, the email that arrives at your final address will come from the mail server hosting yourdomain.com which almost certainly isn’t listed as a valid sender in the SPF record for sender.com.  Your email provider may reject the mail, or flag it as “untrusted”.

To fix this, we need a different TLA: SRS, or Sender Rewriting Scheme. As the name suggests, this rewrites the sender address of a forwarded email, from one in a domain that you don’t control (sender.com) to one that you do (yourdomain.com).

In the example above, the actual rewritten address would be something like:

SRS0-9oge=B5=sender.com=them@yourdomain.com

This includes an encoded version of the original address, and any email sent to it will be routed back to the sender.  This means that any bounces messages will end up in the right place.

The sender and recipient in these examples refer to the “envelope” sender and receiver.  The addresses that are normally visible to users are the “from” and “to” headers, which may be different and are unaffected by sender rewriting.  Applying SRS should be invisible to the end users.

SRS is now available as an option whenever you create or edit a forwarder using the customer control panel for email accounts hosted on our main hosting servers.  If your account is hosted on sphinx, we need to do a little extra magic to enable it, so please email support.