Last month we attended Electromagnetic Field as a silver sponsor.  Despite being in a remote field in Herefordshire, the site had amazing connectivity, which we played a small part in providing.

We provided some optics to help get internet around the field and acted as an Internet Transit Provider to uplink the festival through our network.

We had a tour of the network operations centre. ElectromagneticField leased a single fibre to a telephone exchange in Gloucester and a donated private 40Gbps circuit hauls the traffic back to the London Network Access Point (LONAP). We used private VLANS over LONAP to link to the Mythic Beasts core network routers in Sovereign House and Telehouse and used this to provide our blend of transit providers and peers, including direct access over private fibre to some of the largest cloud providers.

The section from the field in Eastnor to Gloucester uses Dense Wavelength Division Multiplexing, a neat technology that uses multiple different frequencies to carry multiple signals on the same fibre at the same time. Each optical transceiver typically transmits at a specific wavelength on one fibre, and receives on the same wavelength on a second fibre. This is put into a multiplexer which combines the different frequencies from multiple optics into the same fibre and a second multiplexer splits them back out into the component frequencies at the other end, allowing multiple 10Gbps channels to operate over one fibre pair.

We use the same technique to multiply up the bandwidth in our core London network on our leased fibre that interconnects our core London points of presence.

To keep costs down at EMF there isn’t a fibre pair – just a single 60km fibre. The hack to get around this limitation is to use different frequencies in each direction and rely on the fact that the transceivers are frequency-specific for transmitting but not receiving – a transmitter that transmits at 1572.48nm will happily receive at 1572.89nm and vice versa. You can then use eight channels on one fibre as four bi-directional channels.

Around the campsite there were datenklo (a switch in a portaloo) which provided wifi and multiple 1Gbps wired uplinks. Each dataklo had a 10Gbps link back to the network operations centre to provide super-fast connectivity all around the site.

You can read more about some of the awesome things we saw at EMF 2024 in our previous blog post.

We went to Electromagnetic Field 2024 as a silver sponsor. Whilst there we found a lot of fantastic fun things and missed a vast number of others. The really amazing part of the festival was the massive variety of things the participants brought with them. Lock picking and blacksmithing courses were available. Geodesic domes were very popular and courses on how to build them were at the Maths Village. At least one dome integrated with the API from the bar, so the lighting changed colour based on what type of drinks were currently being ordered.

If you thought that ethernet and IP was a bit too modern there was a fully functioning DECT cordless phone network and you could access the live schedule information over ViewData (Prestel/Minitel).

We missed at least 98% of the talks. Fascinating ones we did see included a comprehensive explanation of the attempt to backdoor ssh with xzutils by Dr Matthew Garrett. Dr Matthew Bothwell gave a guide to Astrophysics for Supervillains covering things like ‘what happens if I crash the moon into the earth?’ (answer: you get a moon). Tim Hunkin of the Secret Life of Machines gave a short history of electric shocks and Ian B Dunne played the Theramin and musical saw. Much to our surprise, this was quite listenable.

Sadly the magic smoke came out of the Tesla Coil before the end so we had to make do with fire and lasers in additional to a traditional light show around the DJ area in the Null zone.

But this is a short summary, there was a fantastic kids creche, swap shop, night market for created things, crochet, hand built guitars, paper rockets, active satellite tracking, a 5km run, a fully stocked bar, a manual version of flappy bird to play and and and [approximately 100,000 further words cut to keep this post merely far beyond reasonable]

The now notorious swap shop gets a special mention. Not only did it have extremely dangerous materials like Linux install CDs from the late 1990s that may permanently corrupt young minds but some things that are rather harder to get hold of.

It has a direct entry on wikipedia and is going to cause a lot of festivals to have to update their terms and conditions to prohibit bringing radioactive materials to the site. Fortunately an attendee was familiar with safe disposal procedure and quickly removed the offending sources.

The last and arguably best thing at Electromagnic Field was an incredibly secretive project, the Great Camp Hexpansion Question (GCHQ.net). This was a series of locations (mostly, but not all static) where you could plug the quest markers into the official badge which would record you’d found them. A cross between a scavenger hunt and a technology preview it encouraged wandering and looking around the whole camp to find many cool things that were tucked away.

We asked the organisers if this was an official GCHQ sanctioned project. They said no. But that’s what you’d expect GCHQ would say.

Keen to build on our previous success at being outstanding in a field, for 2024 we’ve set ourselves a tough new target of being out standing in two completely different fields.

The Cambridge Beer Festival is being held this week on Jesus Green in Cambridge. A beer festival is pretty easy to organise: you need some virtual servers to handle the website and tickets, fibre to the field to give fast reliable connectivity for verifying tickets and accepting contactless payments, and perhaps satellite backup just in-case someone digs through the primary fibre.

There are also some other minor logistical requirements like a large quantity of beer, wine, mead, cider and other drinks, a very large marquee to keep it in,  a very large refrigeration system to keep the beer at cellar temperature, a huge cheese stall, a small army of volunteers and a makeshift road system to avoid damaging the park.

Mythic Beasts are providing the virtual servers and the internet transit to keep it all functioning.

Two weeks later, many of our staff are going to Electromagnetic Field. This is a camping festival with power and high speed internet to every tent, good beer and all kinds of amazing installations and demonstrations. In addition to being silver sponsors of the event, Mythic Beasts also donate internet transit and have sent a few sets of bidirectional fibre optics to carry traffic around the site.

If you’re going to be at either event, do come and say hello.  If you don’t know what we look like, drop an email to support or message us on social.mythic-beasts.com.

We’re pleased to announce that we are silver sponsors of this year’s Electromagnetic Field festival.  As in previous years, we will also continue to support the event with free transit.  EMF is a long weekend camping in a field where people who are really very interested in things will tell you about the things that really interest them. There’s talks, demos, art installations and workshops on all kinds of creative things. In addition to camping, everyone gets power and high speed internet to their tent. Rumour has it there is also a bar.

Previous years have had an exceptionally wide variety of talks on a huge number of different subjects. The list of talks from the last festival in 2022 is long, but includes things as wild as:

• Ship vs Oil Rig
• The imitation game – using live data feeds from Network Rail to control a model railway
• Building a home-made enigma machine

We’re not giving a talk this year as we didn’t come up with a good idea in time. For 2026 we’ve already rejected the following presentation titles :

• I’ve got 99 problems and HEX ain’t one.
• D. E. P. R. E. C. I. 8. The importance of correct accounting policies delivered through the medium of Aretha Franklin covers.
• As a large language model I can’t assist with that. It’s illegal, unethical, and against my guidelines.

We’re looking forward to meeting up with lots of interesting people at EMF2024.

In March 2004 we moved all three of our servers into a single rack in the 6/7 Harbour Exchange data centre, operated at the time by Redbus.  The data centre has changed hands several times, and merged with the building next door to become what is now Equinix LD8. We’ve been continuously present for 20 years and 1 month. Normally moving out of a data centre is a difficult, expensive and time consuming operation that is best avoided, but Equinix offered us terms that made doing so make sense. In September 2023 we opened our new core point of presence in Telehouse South.

We’re happy to report this project is now complete and our footprint in Equinix LD8 is now reduced to an optical-only point of presence forwarding 10Gbps waves to our core site at City Lifeline.

Our new space in Telehouse South offers a considerable upgrade over what we could offer in LD8. All servers now have remotely switchable dual power feeds and with dual 10Gbps uplinks. We are able to offer offer cross-connects to anywhere in the Telehouse London campus and 10Gbps wavelengths back to our other sites. We already have some new colocation customers taking advantage of these additional services. We still include serial for out-of-band server management.

During this move, we live migrated our virtual server cloud to hosts in either City Lifeline or Sovereign House. Apart from a few special cases supporting very old virtual servers or ones with BGP transit services, this was done without interruption to the client. Dedicated servers and colocation customers moved in a series of windows to minimise downtime while the servers were relocated.

We brought on additional network capacity as part of the move including 10Gbps and 100Gbps links to transit providers and private peers within the Telehouse London campus. This provides a significant upgrade in connected external capacity.

Mythic Beasts is now a verified Green Hosting Provider according to the Green Web Foundation.

We’ve demonstrated to the Green Web Foundation that all our UK and EU data centres buy as much renewable electricity as they use. This hasn’t changed our operations; internally we met this requirement in 2018. What’s changed is that we’ve now provided all the documentation to meet the certification standards of the Green Web Foundation.

Of course this isn’t quite the same as saying that all the electricity we use comes from renewable power. Ultimately, the electrical energy from a wind farm isn’t tagged to flow directly to the data centres we use and there is also no requirement that the electricity is bought at exactly the same time it is used. Similarly, the data centres have fossil-fueled generator backup which means small amounts of fossil energy are still used.

That said, we do believe that this is an important and useful step in the right direction. By getting verified under this scheme we, and the 429 other verified companies, apply pressure on the data centre suppliers to buy and use renewable energy which strongly encourages the marketplace to build more renewable generation.

Some of our data centre providers are very large well-resourced companies and they place very large long term orders for renewable power. This means renewable power providers can secure funding to build out renewable power generation. When they want to build a data centre, they also have to fund the building of an equivalent amount of renewable generation to power it.

Recently Trend Micro, through their Zero Day Initiative, published a critical flaw for the Exim mail server. It’s described as allowing remote attackers to execute arbitrary code on the Exim server without authentication. On the face of it, any server running Exim and listening on the internet can immediately be taken over by an attacker. What makes this worse is that they claim they reported this in June 2022, and the Exim team have ignored fixing it.

ZDI say ‘The only salient mitigation strategy is to restrict interaction with the application.’ and have allocated a scarily high severity score of 9.8/10.

Mythic Beasts make pretty heavy use of Exim in our mail infrastructure, and mitigating the security risk by turning off email is a pretty severe step while we wait for a fix. On top of that amongst servers we manage for ourselves and clients there’s nearly a thousand installed copies of Exim that will need to be updated.

The Exim team have a different view on the severity, as do other reputable security specialists. Watchtowr have a nice write-up explaining that, by default, none of the six issues can be exploited. Cross checking to Mythic Beasts mail infrastructure we can quickly confirm we’re not affected, and we believe that none of the managed customers should be either.

As this is now not especially time critical, we can wait for the supported operating systems to release updated packages which we can install.

Patching

The security issue is definitely significant enough to meet our 0-day policy of patching immediately as it’s network listening software with a risk of compromise. Debian released packages with the most important fixes on Monday 2nd October. Because this issue covers a very large number of affected machines, some of which are absolutely critical we decided to stage the rollout. First we did our staging servers, then one of our core mailhubs. We then paused for a short while to check no functionality was affected. Then we completed the full roll-out to all managed servers both customer and internal. The final step is our audit – recheck the Exim package on every managed server to make sure the update had applied everywhere. The full rollout and audit completed in around three hours.

We’re expecting updated packages from Ubuntu shortly, which will then be rolled out to all supported managed Ubuntu customers when available.

 

Last year we enhanced our web hosting service with the ability to choose your own PHP verison. You can choose a different PHP version for each website hosted with us, so you can upgrade your staging site and test before you upgrade the production one. With PHP 8.0 about to go end-of-life, the addition of PHP 8.2 provides more options for migrating production applications.

Since the initial roll-out, we’ve added more PHP versions to help with moving and upgrading older applications. Not only is the newest version PHP 8.2 available, but you can also select the older 7.3 and 7.4 versions. We’re proud to sponsor Ondřej Surý who creates the debian packages we rely on.

Our hosting accounts still support unlimited websites, have free and automatic SSL through Let’s Encrypt to keep your sites secure, and include MariaDB databases.

 

On Saturday the Debian team released the latest version of Debian, Bookworm. We’re pleased to announce that this is now available on our virtual and dedicated servers.

Bookworm is a fully supported operating system for our managed hosting and we already have it running on some of our internal production servers. Our preferred open source server management system, Sympl, has also been updated to support Bookworm. Other feature enhancements include much more control over PHP versions and settings. Our virtual server cloud has pre-built images for standard Bookworm and Bookworm with Sympl pre-installed.

There are many improvements in Bookworm, with PHP 8.2 support being the most anticipated by our customers. We would like to thank the Debian team for all their hard work in making this release.

Raspberry Pi Desktop is now available as a supported image in our Raspberry Pi Cloud on all Raspberry Pi 4 servers, providing a true remote desktop.

This is set up with the standard desktop operating system, a virtual 1080p monitor attached and VNC set up for immediate desktop access, and is secured using an SSH ‘tunnel’ to access your desktop so everything is encrypted between the client and server.

Install the desktop edition on your Raspberry Pi:

Add your SSH key:

Power the Raspberry Pi on:

You can then connect to the Rasberry Pi as normal, but now add “-L 5900:localhost:5900” to the SSH command line, which will bind port 5900 (the default VNC port) on your local computer to port 5900 on the Raspberry Pi on the other end of the SSH connection.

(If you’re using PuTTY or a similar SSH client, you should be able to find the relevant setting in Connection>SSH>Tunnels – you’ll want to set the source port to be “5900”, and the destination to be “localhost:5900″.)

Next, connect to the server (substitute ”1234″ for your SSH port, found in the control panel, and use the name of your hosted Pi) as root to establish the connection:

$ ssh -L 5900:localhost:5900 -p 1234 root@ssh.yourserver.hostedpi.com
The authenticity of host ...
ECDSA key fingerprint is SHA256:.....
Linux rpi-bullseye-arm64-vnc...
....

Then, set the password for the Pi desktop user:

$ passwd pi
New password: 
Retype new password: 
passwd: password updated successfully

And finally, connect with a local VNC client to “localhost” and up pops a desktop: