Sampling an Electromagnetic Field

June 13th, 2024 by
A Viewdata terminal with the EMF Schedule

A Viewdata terminal with the EMF Schedule

We went to Electromagnetic Field 2024 as a silver sponsor. Whilst there we found a lot of fantastic fun things and missed a vast number of others. The really amazing part of the festival was the massive variety of things the participants brought with them. Lock picking and blacksmithing courses were available. Geodesic domes were very popular and courses on how to build them were at the Maths Village. At least one dome integrated with the API from the bar, so the lighting changed colour based on what type of drinks were currently being ordered.

If you thought that ethernet and IP was a bit too modern there was a fully functioning DECT cordless phone network and you could access the live schedule information over ViewData (Prestel/Minitel).

We missed at least 98% of the talks. Fascinating ones we did see included a comprehensive explanation of the attempt to backdoor ssh with xzutils by Dr Matthew Garrett. Dr Matthew Bothwell gave a guide to Astrophysics for Supervillains covering things like ‘what happens if I crash the moon into the earth?’ (answer: you get a moon). Tim Hunkin of the Secret Life of Machines gave a short history of electric shocks and Ian B Dunne played the Theramin and musical saw. Much to our surprise, this was quite listenable.

Sadly the magic smoke came out of the Tesla Coil before the end so we had to make do with fire and lasers in additional to a traditional light show around the DJ area in the Null zone.

But this is a short summary, there was a fantastic kids creche, swap shop, night market for created things, crochet, hand built guitars, paper rockets, active satellite tracking, a 5km run, a fully stocked bar, a manual version of flappy bird to play and and and [approximately 100,000 further words cut to keep this post merely far beyond reasonable]

The now notorious swap shop gets a special mention. Not only did it have extremely dangerous materials like Linux install CDs from the late 1990s that may permanently corrupt young minds but some things that are rather harder to get hold of.

Warning sign from the swap shop asking not to drop off radioactive sources.

When we saw the warning sign we had to find out what incident motivated the creation.

It has a direct entry on wikipedia and is going to cause a lot of festivals to have to update their terms and conditions to prohibit bringing radioactive materials to the site. Fortunately an attendee was familiar with safe disposal procedure and quickly removed the offending sources.

The last and arguably best thing at Electromagnic Field was an incredibly secretive project, the Great Camp Hexpansion Question (GCHQ.net). This was a series of locations (mostly, but not all static) where you could plug the quest markers into the official badge which would record you’d found them. A cross between a scavenger hunt and a technology preview it encouraged wandering and looking around the whole camp to find many cool things that were tucked away.

GCHQ location

GCHQ location marker

We asked the organisers if this was an official GCHQ sanctioned project. They said no. But that’s what you’d expect GCHQ would say.

Out standing in two fields

May 20th, 2024 by

Fibre internet, in a field

Keen to build on our previous success at being outstanding in a field, for 2024 we’ve set ourselves a tough new target of being out standing in two completely different fields.

The Cambridge Beer Festival is being held this week on Jesus Green in Cambridge. A beer festival is pretty easy to organise: you need some virtual servers to handle the website and tickets, fibre to the field to give fast reliable connectivity for verifying tickets and accepting contactless payments, and perhaps satellite backup just in-case someone digs through the primary fibre.

There are also some other minor logistical requirements like a large quantity of beer, wine, mead, cider and other drinks, a very large marquee to keep it in,  a very large refrigeration system to keep the beer at cellar temperature, a huge cheese stall, a small army of volunteers and a makeshift road system to avoid damaging the park.

Mythic Beasts are providing the virtual servers and the internet transit to keep it all functioning.

Two weeks later, many of our staff are going to Electromagnetic Field. This is a camping festival with power and high speed internet to every tent, good beer and all kinds of amazing installations and demonstrations. In addition to being silver sponsors of the event, Mythic Beasts also donate internet transit and have sent a few sets of bidirectional fibre optics to carry traffic around the site.

If you’re going to be at either event, do come and say hello.  If you don’t know what we look like, drop an email to support or message us on social.mythic-beasts.com.

Electromagnetic Field 2024 sponsorship

May 1st, 2024 by

Electromagnetic Field Logo

We’re pleased to announce that we are silver sponsors of this year’s Electromagnetic Field festival.  As in previous years, we will also continue to support the event with free transit.  EMF is a long weekend camping in a field where people who are really very interested in things will tell you about the things that really interest them. There’s talks, demos, art installations and workshops on all kinds of creative things. In addition to camping, everyone gets power and high speed internet to their tent. Rumour has it there is also a bar.

Previous years have had an exceptionally wide variety of talks on a huge number of different subjects. The list of talks from the last festival in 2022 is long, but includes things as wild as:

  • Ship vs Oil Rig
  • The imitation game – using live data feeds from Network Rail to control a model railway
  • Building a home-made enigma machine

We’re not giving a talk this year as we didn’t come up with a good idea in time. For 2026 we’ve already rejected the following presentation titles :

  • I’ve got 99 problems and HEX ain’t one.
  • D. E. P. R. E. C. I. 8. The importance of correct accounting policies delivered through the medium of Aretha Franklin covers.
  • As a large language model I can’t assist with that. It’s illegal, unethical, and against my guidelines.

We’re looking forward to meeting up with lots of interesting people at EMF2024.

HEX-it complete

April 29th, 2024 by
Equinix invites you to celebrate international data centre day

We elected not to celebrate with Equinix

In March 2004 we moved all three of our servers into a single rack in the 6/7 Harbour Exchange data centre, operated at the time by Redbus.  The data centre has changed hands several times, and merged with the building next door to become what is now Equinix LD8. We’ve been continuously present for 20 years and 1 month. Normally moving out of a data centre is a difficult, expensive and time consuming operation that is best avoided, but Equinix offered us terms that made doing so make sense. In September 2023 we opened our new core point of presence in Telehouse South.

We’re happy to report this project is now complete and our footprint in Equinix LD8 is now reduced to an optical-only point of presence forwarding 10Gbps waves to our core site at City Lifeline.

Our new space in Telehouse South offers a considerable upgrade over what we could offer in LD8. All servers now have remotely switchable dual power feeds and with dual 10Gbps uplinks. We are able to offer offer cross-connects to anywhere in the Telehouse London campus and 10Gbps wavelengths back to our other sites. We already have some new colocation customers taking advantage of these additional services. We still include serial for out-of-band server management.

During this move, we live migrated our virtual server cloud to hosts in either City Lifeline or Sovereign House. Apart from a few special cases supporting very old virtual servers or ones with BGP transit services, this was done without interruption to the client. Dedicated servers and colocation customers moved in a series of windows to minimise downtime while the servers were relocated.

We brought on additional network capacity as part of the move including 10Gbps and 100Gbps links to transit providers and private peers within the Telehouse London campus. This provides a significant upgrade in connected external capacity.

It’s always DNS (why domain transfers suck)

April 3rd, 2024 by

It’s a popular meme that all mysterious internet problems are caused by issues with the Domain Name System (DNS). Like most memes, it gets over-used, but when it comes to transferring a domain between providers, the intricacies of DNS create some very real problems.

To make things easier, we’ve just rolled out a new feature to our DNS management system that allows you to fetch records from your old provider’s nameservers prior to transferring the DNS for your domain to us.

Screenshot of "fetch live records" control panel function.

Why is this needed?

This functionality can help achieve a seamless transfer of your hosting, by working around an annoying feature of the DNS system.

DNS is the system that converts internet names (like “www.mythic-beasts.com”) into IP addresses (like “93.93.129.174”) that can be used to locate the server for a particular service. This conversion is done by nameservers, and each domain has its own nameservers, usually provided by your hosting provider.

Graphic showing a client querying a nameserver for "www.mythic-beasts.com" and getting the answer.

When you transfer the hosting for your domain between providers, you’ll need to update your DNS records to point at your new web and email servers, but you will also typically change from using your old provider’s nameservers to your new provider’s.

The simple way to transfer your domain is to do these two things in one go.  Your old provider’s nameservers direct traffic for your domain at your old web and email servers, your new provider’s nameservers direct traffic at your new hosting service, so just change the nameservers for your domain from your old provider’s to your new provider’s and you’re done, right?

Graphic showing a client querying nameservers for "www.example.com" and getting a different answer before and after transferring the domain to Mythic Beasts.

This approach works, but it’s not ideal for domains that are in active use because of the delays created by caching.

Caching and TTL

One of the things that makes DNS so confusing is caching. When you look up a name, you’re told to remember the answer for a set period of time. IP addresses don’t change very often, so looking up a name every single time you need it would generate a lot of unnecessary traffic, and slow things down.

Graphic of client querying a namserver for "www.mythic-beasts.com" and getting the answer and the instruction to "remember this for 1 hour".

All DNS records have a “Time To Live” (“TTL”). This is the number of seconds that you’re allowed to remember it for before you have to do a new lookup to see if it’s changed. In the past, TTLs were usually set to hours, days or even a week. As the Internet has become faster, the overhead of DNS lookups has become less of a problem, and TTLs of one hour or a few minutes are now common.

Although caching helps improve performance in normal use, it creates a problem when you need to make changes. When you make a change to the DNS records for your domain, it won’t be picked up immediately by all users, because some people will have the old value cached.

If you know you’re going to need to change a DNS record, you can lower the TTL in advance (for example to 60 seconds), and then, when you come to change the record, all users will pick up the change very quickly.

If you’re planning to change hosting provider, it makes sense to lower the TTL on your DNS records in advance, so that when you come to make the change, all traffic is switched from the old provider to the new provider quickly.

Changing nameservers

When you have your own domain, you need to have some nameservers to answer DNS queries. As described above, when you transfer the hosting for your domain, you will typically also switch from using your old provider’s nameservers to your new provider’s.

The domain name system keeps a record of which nameservers provide the DNS for each domain. For example, DNS for mythic-beasts.com is provided by our nameservers (ns1.mythic-beasts.com and ns2.mythic-beasts.com). The problem is that these records are also subject to caching and usually have a fixed TTL of 48 hours.

Graphic showing a client querying the ".com registry nameserver" for the "example.com" nameservers, and being given the answer, and an instruction to remember it for "2 days". Followed by a query for "www.example.com", with the answer and an instruction to "remember this for one minute".

This means that even if you set a low TTL for your own records, when you change the nameservers for a domain, you have a two day period when queries for your domain might still end up at your old nameservers. If your old and new servers are serving different records, users will get a mix of different answers.

The trick to achieving a clean switch between hosting providers is to separate the move from your old provider’s nameservers to your new provider’s from changing the individual DNS records that control who provides your web and email hosting. In other words, get the old and new nameservers serving exactly the same records, so that during the 48 hour nameserver changeover period, it doesn’t matter which nameserver answers the query. Once that changeover is complete, you can switch your web and email hosting by updating low-TTL records.

Our new fetch live records feature makes it easier to copy the records from your old provider’s nameservers to ours, so that you can do a seamless nameserver handover before migrating your web and email hosting. Unfortunately, this tool can only check for commonly used records because there’s no reliable way to get a complete list. The best solution is to get an export of your current DNS records from your current provider, and use our import function, but many providers don’t have an export feature in their systems.

This stuff is hard – we’re here to help

Domain transfers, and DNS in general, are difficult and confusing. For many of our customers, changing providers is a once-per-decade thing, whereas we deal with domain transfers every single day.

We’re working hard to build tools that make the process easier, but our support team is always on hand to provide personalised help.

Green hosting

March 25th, 2024 by

Mythic Beasts is now a verified Green Hosting Provider according to the Green Web Foundation.

Green Web check for mythic-beasts.com

We’ve demonstrated to the Green Web Foundation that all our UK and EU data centres buy as much renewable electricity as they use. This hasn’t changed our operations; internally we met this requirement in 2018. What’s changed is that we’ve now provided all the documentation to meet the certification standards of the Green Web Foundation.

Of course this isn’t quite the same as saying that all the electricity we use comes from renewable power. Ultimately, the electrical energy from a wind farm isn’t tagged to flow directly to the data centres we use and there is also no requirement that the electricity is bought at exactly the same time it is used. Similarly, the data centres have fossil-fueled generator backup which means small amounts of fossil energy are still used.

That said, we do believe that this is an important and useful step in the right direction. By getting verified under this scheme we, and the 429 other verified companies, apply pressure on the data centre suppliers to buy and use renewable energy which strongly encourages the marketplace to build more renewable generation.

Some of our data centre providers are very large well-resourced companies and they place very large long term orders for renewable power. This means renewable power providers can secure funding to build out renewable power generation. When they want to build a data centre, they also have to fund the building of an equivalent amount of renewable generation to power it.

Mastodon security update

February 2nd, 2024 by

Yesterday, the following not-so-subtle notice appeared on the admin interface of all Mastodon instances:

The Mastodon team announced on Monday that this release was coming, so we were ready for it:

Details of the vulnerability are still limited, but from what we do know it sounds serious (“Remote account takeover“).

All our managed Mastodon instances were safely patched just over an hour after the new packages dropped. One instance gave us a bit of trouble, as the new version appeared to tickle a bug in Elasticsearch causing ES to consume all CPU on the server. After we eventually pinned down the cause, it was resolved by an upgrade of Elasticsearch. Turns out the ES upgrade didn’t fix it, and we’re still working with our customer to get this resolved.

Managed open source hosting

Open source software such as Mastodon, GitLab and Nextcloud can offer a great alternative to the lock-in associated with proprietary cloud equivalents, but the effort associated with hosting them can be significant: backups, monitoring, security patching, and the investigation and debugging required when a supposedly innocuous software upgrade leaves your CPU usage wedged at 100%.

Our managed open source hosting provides the best of both worlds: the convenience of a “cloud” solution, but without the lock-in. Your data is yours, and if you don’t like our service you can take your data and host it somewhere else (although we’re confident you won’t want to). And because there’s no lock-in, you get straightforward pricing based on the resources you’re using, rather than loss-leaders followed by price hikes once you’re hooked.

Read more about our managed hosting, or drop us an email at for more information.

Exim 0-day

October 4th, 2023 by
exim logo

We sponsor exim and provide a VM for their buildfarm.

Recently Trend Micro, through their Zero Day Initiative, published a critical flaw for the Exim mail server. It’s described as allowing remote attackers to execute arbitrary code on the Exim server without authentication. On the face of it, any server running Exim and listening on the internet can immediately be taken over by an attacker. What makes this worse is that they claim they reported this in June 2022, and the Exim team have ignored fixing it.

ZDI say ‘The only salient mitigation strategy is to restrict interaction with the application.’ and have allocated a scarily high severity score of 9.8/10.

Mythic Beasts make pretty heavy use of Exim in our mail infrastructure, and mitigating the security risk by turning off email is a pretty severe step while we wait for a fix. On top of that amongst servers we manage for ourselves and clients there’s nearly a thousand installed copies of Exim that will need to be updated.

The Exim team have a different view on the severity, as do other reputable security specialists. Watchtowr have a nice write-up explaining that, by default, none of the six issues can be exploited. Cross checking to Mythic Beasts mail infrastructure we can quickly confirm we’re not affected, and we believe that none of the managed customers should be either.

As this is now not especially time critical, we can wait for the supported operating systems to release updated packages which we can install.

Patching

The security issue is definitely significant enough to meet our 0-day policy of patching immediately as it’s network listening software with a risk of compromise. Debian released packages with the most important fixes on Monday 2nd October. Because this issue covers a very large number of affected machines, some of which are absolutely critical we decided to stage the rollout. First we did our staging servers, then one of our core mailhubs. We then paused for a short while to check no functionality was affected. Then we completed the full roll-out to all managed servers both customer and internal. The final step is our audit – recheck the Exim package on every managed server to make sure the update had applied everywhere. The full rollout and audit completed in around three hours.

We’re expecting updated packages from Ubuntu shortly, which will then be rolled out to all supported managed Ubuntu customers when available.

HEX-it

September 27th, 2023 by

Last year, we undertook a significant data centre migration, with the closure of Digital Realty’s Meridian Gate requiring us to move our entire presence there to Redcentric’s City Life Line. Having done it once, why not do it again?

Southern Serval, leaping

Our shared hosting server “serval” has already migrated to SOV. [ Photo by Wynand Uys]

This year, we’re planning a move out of Harbour Exchange (HEX), and starting a presence in Telehouse South. A lot of the things we learned during the previous move are making this move easier to manage, although it is still a prodigious effort, both physically and in terms of design and infrastructure.

One of the things we’ve been working on for some time is improved network infrastructure within our data centres. This introduces IP address portability so that IP addresses do not need to change when servers are moved between data centres, as well as significantly higher bandwidth uplinks for our virtual server hosts.

In the last year, we’ve live migrated over a thousand VMs across two data centres, with minimal interruption to service.

We’re about to start migrating all VMs out of our HEX data centre. We have two available London destinations, SOV and CLL. If you’re a customer with a VM in our HEX data centre, we’ll be emailing you over the next couple of weeks, to check if you have a preference (for instance because you have existing services in one of those data centres, and would prefer to be moved to the other to maintain fault-tolerance).

We will also soon be able to offer Telehouse South as a virtual server zone, in addition to SOV and CLL. This means we will continue to provide three London-based zones for our customers running distributed services. We’ll retain a small residual presence in HEX for connectivity.

PHP 8.2

September 25th, 2023 by

 

Last year we enhanced our web hosting service with the ability to choose your own PHP verison. You can choose a different PHP version for each website hosted with us, so you can upgrade your staging site and test before you upgrade the production one. With PHP 8.0 about to go end-of-life, the addition of PHP 8.2 provides more options for migrating production applications.

Screenshot of account control panel

Choose your PHP version in the control panel

Since the initial roll-out, we’ve added more PHP versions to help with moving and upgrading older applications. Not only is the newest version PHP 8.2 available, but you can also select the older 7.3 and 7.4 versions. We’re proud to sponsor Ondřej Surý who creates the debian packages we rely on.

Our hosting accounts still support unlimited websites, have free and automatic SSL through Let’s Encrypt to keep your sites secure, and include MariaDB databases.