The European Bio-informatics Institute used x-rays and a lot of hard maths to draw the above picture of the main covid-19 protein.

We provide critical infrastructure to many other companies who rely on our services. Covid-19 could significantly change day to day life everywhere. At present we don’t believe it will have a significant effect on our operations.

Remote working

Mythic Beasts has always been a distributed company with no central office; all staff members normally work from home. As a result adopting remote working recommendations or enforcement will have no significant impact on our day to day operations. Normally, we have a weekly optional meeting for staff around Cambridgeshire and a compulsory all company meeting roughly once every six weeks. Migrating these meetings to conference calls will have minimal operational impact.

Reducing travel

Our sales process has always been online. We don’t routinely meet customers and have a very light attendance at conferences and industry events. Our next scheduled events are UKNOF (Manchester, April) and LINX (Manchester) both of which offer remote participation should it become necessary.

Financial stability

Mythic Beasts has been profitable every year since 2001 and carries no debt. We maintain significant cash reserves so we can self-finance routine expansion and other business opportunities, and weather unforeseen circumstances. We understand that many consider this an inefficient use of capital, but we can definitely pay our bills in the event of a global pandemic crippling the economy for a short period.

Sensible staff policies

Our staff members are provided with private healthcare. They also get sick leave which they are expected to take if ill. We also provide 30 days + bank holidays holiday to all staff members as standard and we strongly encourage them to have a two week contiguous holiday in order that we know we can operate without them in the event of sickness. We have sufficient staff levels that in the event of multiple staff members being ill for an extended period day to day operations can be maintained and only longer term projects should be delayed.

Supplier issues

We maintain stock of key components to cope with hardware failure although for some months we’ve been seeing very long lead times for hardware, especially CPUs and SSDs. This may impact our lead times for larger orders of dedicated servers or custom hardware. We think it likely that our data centre and connectivity providers may implement a change freeze as they do annually over the Christmas period and at other key times (e.g. the 2012 Olympics). This is a familiar operating environment and utilising multiple providers in multiple countries will help to mitigate this.

2020-03-12 : We run our own private phone conference and IRC service so we’re not affected by the reported load issues at the major public providers Slack, Teams, Zoom etc.

In summary, we think that day to day operations shouldn’t be affected but if you have any concerns get in touch at support@mythic-beasts.com.

We’ve got a new version of our DNS API under development.  One of the neat new features is the ability to accept input in bind zone file format (aka RFC 1035). 

One of the things that this makes very easy is adding SSHFP records to the DNS. SSHFP is a mechanism for lodging your server’s public SSH keys in the DNS so that your SSH client can automatically verify a server the first time you connect to it, rather than prompting you to confirm the host key.

Using our new API, you can add or update SSHFP keys by piping the output of ssh-keygen straight into curl:

ssh-keygen -r myhost | curl -X PUT -n https://api.mythic-beasts.com/zones/example.com/myhost/SSHFP -H 'Content-Type: text/dns' --data-binary @-

What’s going on here?

ssh-keygen -r outputs your server’s SSH public keys in RFC 1035 format:

$ ssh-keygen -r myhost
myhost IN SSHFP 1 1 e579ff6aabc2f0acf714deca53108a0c1ea7d799
myhost IN SSHFP 1 2 7c47d5dfb748ff1fd244b7289d815e83dad8c2c1652b92ac8aed8ff166733d07
myhost IN SSHFP 2 1 c5caf4cc8870acc7fd113e5a7c866822ec0d94de
myhost IN SSHFP 2 2 9f11843fa1d9da318aa4bc09bbcaacaf4a9868c4d83dfc4bad6853d0c9597a31
myhost IN SSHFP 3 1 eb8644f5fcfd555341f2063bd92044075e20da89
myhost IN SSHFP 3 2 60f3e9780f9b87e5b4d6344f2ab46decbf705123e96ef07c3247f714ca220fc4
myhost IN SSHFP 4 1 139426de48381ea46ad75dde4e412bf1c9b11e61
myhost IN SSHFP 4 2 6f094181b510bbb573048835665773eb1a2a65fd4341d95207479ed71296491b

We then pipe that into curl to make a request to the DNS API.

A PUT request to the /myhost/SSHFP endpoint replaces all existing myhost SSHFP records.

-n tells curl to get auth credentials from a .netrc file, and the “Content-Type” header tells our API that we’re providing the new records in zone file format.

What’s the point of SSHFP?

Having lodged these SSHFP records in the DNS, and provided that DNSSEC is enabled for your domain, it’s possible to connect to a server without being prompted to verify the server’s host key.

$ ssh -o VerifyHostKeyDNS=yes root@myhost.example.com 

You can avoid having to specify the -o by putting VerifyHostKeyDNS=yes in your ~/.ssh/config file.

Feedback wanted

The new DNS API isn’t quite ready for deployment, so if there are any features that you’d really like to see in the new API, now’s the time to tell us, either on Twitter or by email.

It’s now nearly five years since we started offering IPv6-only hosting, and what started out as a source of interesting projects for enthusiastic early-adopters has become our default for most hosting requirements.

A few things have changed over the years that have made this possible:

• The death of Windows XP, the last significant OS with a browser that didn’t support SNI (Server Name Indication). SNI makes it possible for us to proxy encrypted connections to IPv6-only hosts.
• The widespread adoption of secure services. This means that protocols that don’t have their own proxying features (such as POP3 or IMAP) can be proxied in their encrypted form thanks to SNI.
• Improvements to our hosting services, such as our SSH port forwarder.

This post gives a quick run-down of how we make IPv6-only hosting a reality.

Getting bytes in

There’s no getting away from the fact that an IPv6-only hosting server still needs to be able to talk to IPv4-only clients, but there’s now a good solution for doing so for pretty much all common scenarios.

Web traffic

This is the most common requirement, and also probably the easiest, as it can be handled by our v4 to v6 proxy.  The proxy is a set of servers with both IPv4 and IPv6 addresses that accept traffic for various protocols and forward it to an IPv6-only server.

The DNS for the hosted site points at our proxy servers, by means of either an ANAME or CNAME record to proxy.mythic-beasts.com.

Unencrypted HTTP traffic is easy to proxy as HTTP 1.1 is designed to support multiple websites on a single IP address.

HTTPS is also easy to proxy, thanks to the now-ubiquitous support for SNI (its successor, ESNI, may complicate this a bit in the future, but we’ll tackle that in a separate post).

Our proxy also supports PROXY protocol, which is a standard way of communicating the original client’s IP address on a proxied connection. Support for PROXY protocol is now a standard feature of NGINX and Apache.

IPv6 traffic can either follow the same route as IPv4 traffic through the proxy (as shown above) or can be routed directly to the hosting server by setting the AAAA records for the site to point at the server rather than the proxy:

This provides a slightly more direct route for IPv6 traffic, but can make the configuration on the server a little more complicated, particularly if you’re using PROXY protocol.

IMAP and POP3

These can both be proxied in their secure forms (IMAPS and POP3S) thanks to SNI, and thankfully these secure variants are now the default choice for all popular email clients.

SSH

Our customers typically want to administer their servers via SSH, and can’t guarantee that they’ll always be connecting from a v6-enabled network. The SSH protocol isn’t built on TLS/SSL so doesn’t have SNI support, and doesn’t have any equivalent features of its own.

We work around this by providing a port-forward to all virtual servers and Raspberry Pi servers from a host with a v4 IP address, so customers can make a connection to a different host on a non-standard port, and the connection will be forwarded to the IPv6 server on port 22. Details of the host and port can be found in our customer control panel.

SMTP

SMTP is a bit awkward. It’s used in two common scenarios:

1. “Submission”, where an end-user client sends outgoing mail using authenticated SMTP
2. Server-to-server delivery of email.

It has multiple ports in common use:

• 25 – the standard port for server-to-server email
• 465 – a port for SMTP over SSL
• 587 – the standard SMTP submission port

Port 25 doesn’t use SSL/TLS at connection time, but can be upgraded to a secure connection via the STARTTLS command, which means it can’t be proxied using SNI.

Port 465 has a confused history, having been allocated by IANA for secure SMTP, then revoked in favour of STARTTLS and allocated to a different service, and then reinstated for secure SMTP submission by RFC 8314.  Port 465 is supported by our proxy, and is a good choice for SMTP submission.

Port 587 was historically plain SMTP (RFC 2476) with STARTTLS, but is being migrated to SSL by default (RFC 8314) which is proxyable thanks to SNI.  Our proxy assumes that port 587 traffic is encrypted (because it can’t do anything useful if it’s not) and as such can also be used for SMTP submission, provided you use SSL/TLS rather than STARTTLS.

For server-to-server delivery, it’s possible to use our dual-stack MX servers to handle incoming mail. This can be done by having the highest priority MX record point to the v6-only server, and then have a lower priority record
pointing to our MX servers. v4-only servers will deliver to our MX servers, and we’ll then pass it on to your v6-only server.

This isn’t a perfect solution, as it means you can’t do connection-time filtering of incoming mail.

Our MX servers need to be configured to accept mail for your domain. At present, this needs to be done by emailing support.

Getting bytes out

Your server may need to make outgoing connections to v4-only servers. Fortunately this is straightforward using our NAT64 resolvers. These are DNS resolvers that when asked for an address for a host that does not have any AAAA records will provide an IPv6 address that is mapped to the host’s v4 address. The v6 address is actually an address on one of our NAT servers that will then forward the traffic to the v4 address.

There’s a 1:1 mapping between v4 addresses and v6 addresses on the NAT server – with IPv6 we can easily allocate the equivalent of the full 32-bit IPv4 address space to a single server!

NAT64 works very well in almost all cases. We have come across a few bits of software which explicitly request an A record when doing a DNS lookup, which obviously doesn’t work.

As with any NAT configuration, you’re sharing a v4 address with other users, which can cause issues for sites that perform IP-based filtering or rate limiting.

Make the switch

Like most providers, we now charge for IPv4 addresses, but unlike most other providers it’s a tax you probably don’t need pay. We offer IPv6-only versions of all of our virtual and dedicated servers, and our Raspberry Pi servers area all IPv6-only.

Learn more

If you’d like to hear more, here are some videos of a presentation that Pete gave at the UK Network Operators Forum (UKNOF).

Unfortunately Sympl doesn’t include easy to manage graphic designers.

Hot on the heels of the Debian Buster release, we’re pleased to announce our first release of Sympl, an open-source hosting management platform for Debian.

What is Sympl?

Sympl is easiest to explain by example.

Want to create a secure website for https://example.com?

Simply create a directory:

mkdir -p /srv/example.com/public/htdocs

That’s it. Point the DNS at your server and start uploading your content. An SSL certificate will be obtained automatically from Let’s Encrypt.

Want to create a new mailbox for Brian? Simply create a directory:

mkdir /srv/example.com/mailboxes/brian

Your server now accepts mail for brian@example.com.

Mail is accessible using webmail, or using any device via secure IMAP/SSL.

Configuration is all done over SSH, so you gain all the security advantages of a highly locked down server, with much easier configuration management.

Works with you, not against you

Unlike other solutions, which take an all-or-nothing approach to managing your server, Sympl happily accepts you customising the configuration and will avoid overwriting any configuration files that you alter.

When it writes configurations for you, Sympl automatically picks best practice options. This includes things like limiting permissions for PHP, secure connections for web and email, and of course, IPv6 support throughout. It’s built on Debian Linux and runs on our dedicated servers, virtual servers and we also build the packages for the Raspberry Pi.

Sympl is 100% open source. It’s completely free to use, irrespective of the number of servers or domains you might want to use with it.

Installing Sympl

If you have a Mythic Beasts virtual server running Debian Buster you can install Sympl easily by using the install script:

wget https://gitlab.mythic-beasts.com/sympl/install/raw/master/install.sh
bash install.sh

If you want a managed Sympl server, we’ll do this for you as part of the setup.

Server management

Sympl pairs well with our managed hosting service. We monitor your server 24/7, apply security updates and take a daily backup leaving you to manage the sites running on it.

Future plans

Future plans for Sympl include automatic DNS configuration using OctoDNS, which supports a wide range of DNS providers, updated Let’s Encrypt support allowing automatic wildcard SSL certificates, and a fully functional command line parser for day to day administration tasks.

Find out more info on Sympl at sympl.host, which is (of course) hosted using Sympl.

Integration can be hard work

Last year we had a busy time acquiring Retrosnub, BHost and VMHaus. We’ve been steadily making progress in the background integrating the services the companies provide to reduce costs and complexity of management. We can now also announce our first significant feature upgrade for VMHaus. We’ve deployed a new virtual server cluster to our Amsterdam location and VMHaus services are now available in Amsterdam. VMHaus is using Mythic Beasts for colocation and network and in Amsterdam they will gain access to our extensive set of peers at AMSIX, LINX and LoNAP. Per hour billed virtual servers are available from VMHaus with payment through Paypal.

As you’d expect, every VM comes with a /64 of IPv6 space.

In the background we’ve also been migrating former-BHost KVM-based services to Mythic Beasts VM services in Amsterdam. Shortly we’ll be starting to migrate former-BHost and VMHaus KVM-based services in London to new VM clusters in the Meridian Gate data centre.

Sympl is so simple it’s even usable by Cambridge graduates

We’re pleased to announce that we are now supporting the Sympl open source project.  Sympl is a fork of Symbiosis, a platform that makes hosting websites and email on a virtual or dedicated server simple.  Once installed, configuring a new website, or creating a new email address and mailbox, is as simple as creating a new directory.  Web server, mail server and DNS configuration is all taken care of for you.

We’ve already taken the first steps towards integrating Sympl into our infrastructure by implementing support for our DNS API in OctoDNS.  For our next step, we will be adding support for OctoDNS to Sympl.  This means that it becomes possible to use Sympl with our DNS infrastructure, but equally you can use any other provider supported by OctoDNS (we don’t believe in lock in!)

We’re now very pleased to welcome Paul Cammish, the newest member of the Mythic Beasts team.  Paul has considerable experience, having worked at a number of different ISPs since 2000, most recently at Bytemark.  Paul created the Sympl project earlier this year, in order to provide ongoing support and enhancements for the platform.

We’re very excited by the possibilities that Sympl provides, and have some interesting ideas for future developments once we’ve dealt with the immediate priorities of DNS integration, and support for the upcoming Debian Buster release.

The source code for Sympl is now available in our self-hosted GitLab instance.

We’ve just rolled out a major overhaul of our DNS management interface. We hope that you’ll find the new interface faster and easier to use. As well as improvements to the user interface, we’ve also added the ability to import zone files. This means that if you’ve got a domain that is currently hosted with another provider, you can now easily transfer all of your DNS configuration to our servers in bulk (provided that you can get them to give you a copy of your current zone file).

Our DNS management interface is included with all domain registrations.  It’s also available for domains registered elsewhere for customers of our other services, including hosting accounts, virtual servers, dedicated servers and Raspberry Pi servers.

The DNS interface includes DNS API access, allowing you to support dynamic DNS and to automate other DNS management tasks.

We believe in retaining customers through good service rather than lock-in, so naturally there’s a corresponding zone file export feature.

X now does Y 🙂

— Mythic Beasts (@Mythic_Beasts) 19 March 2019

Sensible people don’t like to hard code IP addresses in lots of different places in DNS. Better to assign it a name, and then reference that name, as it makes it clearer what’s what and if you ever need to change that IP, you’ve only got to do it one place.

CNAME records can be a good way to do this, by aliasing a DNS name to an IP. Unfortunately, the DNS specs prevent you using CNAMEs in various places that you might want to, most commonly at the root-level of your domain (the dreaded “CNAME and other data” problem).

This is where ANAME pseudo-records come in. They look just like a CNAME record, but rather than being added to the DNS, our server converts them into A and AAAA records. This allows you to get the benefits of a CNAME in places where a CNAME is not legal.

This week a customer suggested another use for ANAME records that we’d not previously thought of: round robin DNS. That is, a single DNS name that points to multiple servers. As you can’t have multiple CNAME records for the same hostname, implementing round-robin DNS means hard-coding A and AAAA records into your zone file. Like this:

proxy.mythic-beasts.com. 3600	IN	A	93.93.129.174
proxy.mythic-beasts.com. 3600	IN	A	46.235.225.189
proxy.mythic-beasts.com. 3600	IN	AAAA	2a00:1098:0:80:1000:3b:1:1
proxy.mythic-beasts.com. 3600	IN	AAAA	2a00:1098:0:82:1000:3b:1:1

Which is messy. Wouldn’t it be nicer to use the names of the servers involved? Like this:

proxy.mythic-beasts.com. 3600	IN	CNAME	 rproxy46-sov-a.mythic-beasts.com.
proxy.mythic-beasts.com. 3600	IN	CNAME    rproxy46-hex-a.mythic-beasts.com.

Sadly, the spec says you can’t do that, but thanks to a minor tweak to our DNS control panel code, you can now do it with ANAME records. Simply specify multiple ANAME records for your host name, and we’ll go and find all A and AAAA records for all of the hosts that are referenced.

Thanks to @grayvsearth for the suggestion on this one.

ANAME records are available in our DNS management interface, which is included with all domain registrations, and available for free on other domains for customers of other services. Other features include a DNS API, allowing you to obtain Wildcard Let’s Encrypt certificates.

The art warehouses in Amsterdam look much prettier than the data warehouses.

Back in July 2018, Mythic Beasts acquired Bhost, giving us additional virtual machine (VM) clusters in London, Amsterdam and California.

Today we’re pleased to announce that we’ve deployed a substantial new VM cloud to Amsterdam, running our own VM platform. Virtual machines in Amsterdam are available to purchase immediately through our website in sizes from 1GB/1vCPU to 160GB/12vCPUs, and with both SSD and spinning rust disk options. Server management and backup options are also available.

Thanks to Brexit-related regulatory uncertainty, some of our existing clients informed us that they must be hosted outside of the UK before 29th March. Deploying capacity on our own platform in Amsterdam means that we can migrate virtual servers directly to the new location.

Once we’ve dealt with the immediate Brexit-driven server moves, we’ll be looking at migrating former-Bhost VMs into this new cloud, giving a significant performance boost in the process.

Deploying the Amsterdam VM cloud is a significant milestone in the integration of the Bhost infrastructure into our own. The integration provides improved performance and redundancy for both Mythic Beasts and Bhost customers whilst simultaneously cutting our operating costs. In preparation for this, we completed upgrades to our core network last October. The existing fibre ring around our three main London sites, which is currently lit at 50Gbps, is now complemented by a 10Gbps ring around London (HEX) ⟺ Cambridge ⟺ Amsterdam ⟺ London (MER). This replaces the old 2x1Gbps connectivity from Cambridge to London with diverse 10Gbps feeds to London and Amsterdam. Our network has gained an additional 10Gbps transit in Amsterdam (NTT) and we are also now connected on the Amsterdam Internet Exchange (AMS-IX).

On a trip to deploy new routers, Pete even managed a tour of the city on foot in just over three hours.

A Mythic Beast eating a Retrosnub (artists impression)

Just before Christmas we were approached by Malcolm Scott, director of Retrosnub, a small cloud hosting provider in Cambridge. His existing connectivity provider had run out of IPv4 addresses. They’d decided to deal with this issue by adding charges of £2 per IPv4 address per month to encourage existing customers to return unused IPv4 addresses to them. As a cloud hosting provider with a substantial number of virtual machines (VMs) on a small number of hosts this had the result of tripling the monthly colocation bill of Retrosnub.

Aware of my presentation on IPv6-only hosting at UKNOF, Malcolm knew that opportunities for significant expansion were severely limited due to the difficulty of obtaining large amounts of IPv4 address space. Retrosnub faced a future of bankruptcy or remaining a very niche provider. His connectivity providers seemed strongly in favour of Retrosnub going bust so they could reclaim and re-sell the IPv4 space for higher margin services.

There are no expansion opportunities for new cloud hosting providers.

As a larger provider with our own address space, we had sufficient spare capacity in our virtual machine cloud to absorb the entire customer base of Retrosnub with no additional expenditure. Our work in supporting IPv6-only virtual machines will also make it easier to significantly reduce the number of IPv4 addresses required to support Retrosnub services. We formed a deal and agreed to buy the customer base of Retrosnub.

Combining operations

Since agreeing the deal, we’ve been working hard to merge our operations with minimum disruption.

The top priority was the domain name services because domains expire if you don’t renew them. Doing a bulk transfer of domain names between registrars is something which Nominet, the body responsible for UK domains, makes extremely easy, as it just requires changing the “tag” on all the domains.

Unfortunately, just about all other TLDs follow a standard ICANN process, which requires that a domain be renewed for a year at the time of transfer, and that the owner of the domain approves the process. If you were designing a process to destroy competition in a market by making it hard for resellers to move between registrars, it would look quite like this.

We’ve now got the bulk of domains transferred, and the next steps will be to migrate the DNS records from Retrosnub to Mythic Beasts so that our control panel can be used to change the records.

At the same time, we rapidly formulated a plan to migrate all the virtual machines in to stem the financial losses. Moving the VMs required an unavoidable change in IP address, and we also wanted to get them migrated from their current platform (Citrix Xenserver with para-virtualisation) to our own platform (KVM with full hardware virtualisation).

In order to ease the transition, we arranged for a pair of servers to do IP forwarding: a server in our cloud that forwarded the new IP to the VM in the Retrosnub cloud until it was migrated in, and another in the Retrosnub cloud that forwarded the old IP after the server had been moved. By doing this we were able to give customers a one week window in which to complete their IP migration, rather than forcing it to be done at the time that we actually moved the VM.

In the process of this migration, all customers received a significant bandwidth upgrade and majority received disk, RAM and CPU upgrades too.

We completed this on schedule before the quarterly colocation bill arrived, so instead of paying the much increased bill, we cancelled the contract and removed the servers from the facility.

Next steps

Our next step will be to migrate all the web and email hosting customers into our standard shared hosting environment. This has some time pressure as Google have plans for Chrome to start marking all non-HTTPS websites as insecure. We offer one click HTTPS hosting using Let’s Encrypt on all of our hosting accounts.