We’ve had a variety of customer being very complimentary recently. Andy Steven runs a series of web cams in the Shetland Islands that stream live views of the northern lights. The cameras relay the stream via one of our virtual servers in our MER data centre and the current bandwidth record is several Gbps.

I am proud to say that our new ‘AuroraCam’ network just delivers and for the first time I no longer break out in a sweat watching the demand increase from that AuroraWatchUK alert or a celebrity weather personality sending out a Tweet.

— Andy Steven, Shetland Webcams (full article)

Beautiful shot of the northern lights captured by Shetland Webcams. Could be improved by adding a kitten though.

We provide 10Gbps fibre connectivity to the Cambridge office of DarkTrace. Darktrace uses machine learning to identify and neutralise security threats in real time.

You’ve been much more transparent & approachable than any provider I’ve dealt with previously. Very happy with the service so far.

— Harry Godwin, Head of Business Infrastructure. Darktrace

The Web hosting review and advice site Hosting Advice interviewed us and wrote a great article about the management and infrastructure services we provide.

Recognizing that there is no one-size-fits-all approach to managed hosting, Mythic Beasts can take on varying responsibility levels as needed. This range of services includes everything from ensuring that servers are up and running to providing the extensive monitoring, security, and assistance necessary to keep custom web applications functioning reliably.

— Hosting Advice (full article)

Lastly our strong stance about returning Nominet to its public benefit roots garnered entirely positive responses at Twitter.

Very pleased to see that our domain registrar of choice @Mythic_Beasts has signed the call to improve governance of Nominet. https://t.co/z7p8Nfdi94 – we were going to ask them to sign but not surprised they’ve already done so. #kudos #domainnames #nominet

— The Familiar (@_The_Familiar) February 2, 2021

Well done, Mythic. Turning Nominet back into a public interest custodian of .uk is long overdue.https://t.co/d2lLGskK0Y https://t.co/yGUmBwg5Ph

— Jim Hague (@banbury_bill) February 1, 2021

We have signed up to Public Benefit, an effort to restore Nominet to its roots as a public benefit, not for profit organisation.

Nominet runs a world class registry for domains ending in .uk. Their technical execution is faultless and we’re extremely happy with all the services they provide for .uk domains.

A ccTLD domain registry is a natural monopoly, and a profitable one at that. For many years, Nominet have donated their surplus to the Social Tech Trust (formerly the Nominet Trust, which was renamed after they cut funding), a charity that uses technology for the public good.

Charitable donations have dwindled whilst prices have increased over the last five years, due to spending on loss making research projects such as self driving cars and Radio Spectrum management, not to mention last year’s £249,000 pay rise for the CEO (to £772,000).

We are strongly in favour of the proposal of Axel Pawlik, former MD of RIPE, as a director. Under Axel’s leadership, RIPE achieved many significant improvements to internet infrastructure including, but not limited, to:

• Managing IPv4 address exhaustion, balancing the needs of existing ISPs while preserving access for new entrants;
• Encouraging and facilitating IPv6 uptake;
• Encouraging uptake of RPKI to secure routing announcements (RIPE now has the highest participation rate of any RIR); and
• Creating RIPE Atlas, a communal tool to track routing that makes running an ISP much easier.

Sir Michael Lyons also appears to be a sound proposal, although beyond his earlier report on Nominet governance, we have no day-to-day experience of his work.

Nominet is structured such that the elected non-executive directors are out-numbered and are unable to achieve meaningful change, which is why after years of dissatisfaction this has come to an Extraordinary General Meeting to remove the existing directors. Voting is weighted in a complicated fashion, but the more domains the member controls the more important their vote is. As a result domain owners can effectively vote by switching registrars, and if you would like to support this proposal we would recommend moving any .uk domains to a registrar that has signed up to call the EGM. Nominet are very good at actually running the registry, and .uk domain transfers are very easy, and free.

Back in 2018, we acquired BHost, a virtual hosting provider with a presence in the UK, the Netherlands and the US. Since the acquisition, we’ve been working steadily to upgrade the US site from a single transit provider with incomplete IPv6 networking and a mixture of container-based and full virtualisation to what we have now:

• Dual redundant routers
• Two upstream network providers (HE.net, CenturyLink)
• A presence on two internet Exchanges (FCIX/SFMIX)
• Full IPv6 routing
• All customers on our own KVM-based virtualisation platform

With these improvements to our network, we’re now able to offer IPv4 and IPv6 transit connectivity to other customers in Hurricane Electric’s Fremont 2 data centre. We believe that standard services should have a standard price list, so here’s ours:

Prices start at £60/month on a one month rolling contract, with discounts for longer commits. You can order online by hitting the big green button, we’ll send you a cross-connect location within one working day, and we’ll have your session up within one working day of the cross connect being completed. If we don’t hit this timescale, your first month is free.

We believe that ordering something as simple as IP transit should be this straightforward, but it seems that it’s not the norm. Here’s what it took for us to get our second 10G transit link in place:

• 24th April – Contact sales representative recommended by another ISP.
• 1st May – Contact different sales representative recommended by UKNOF as one of their sponsors.
• 7th May – 1 hour video conference to discuss our requirements (a 10Gbps link).
• 4th June – Chase for a formal quote.
• 10th June – Provide additional details required for a formal quote.
• 10th June – Receive quote.
• 1st July – Clarify further details on quote, including commit.
• 2nd July – Approve quote, place order by email.
• 6th July – Answer clarifications, push for contract.
• 7th July – Quote cancelled. Provider realises that Fremont is in the US and they have sent EU pricing. Receive and accept higher revised quote.
• 10th July – Receive contract.
• 14th July – Return signed contract. Ask for cross connect location.
• 15th July – Reconfirm the delivery details from the signed contract.
• 16th July – Send network plan details for setting up the network.
• 27th July – Send IP space justification form. They remind us to provision a cross connect, we ask for details again.
• 6th August – Chase for cross connect location.
• 7th August – Delivery manager allocated who will process our order.
• 11th August – Ask for a cross connect location.
• 20th August – Ask for a cross connect location.
• 21st August – Circuit is declared complete within the 35 day working setup period. Billing for the circuit starts.
• 26th August – Receive a Letter Of Authorisation allowing us to arrange the cross connect. We immediately place order for cross connect.
• 26th August – Data centre is unable to fulfil cross connect order because the cross connect location is already in use.
• 28th August – Provide contact at data centre for our new provider to work out why this port is already in use.
• 1st September – Receive holding mail confirming they’re working on sorting our cross connect issue.
• 2nd September – Receive invoice for August + September. Refuse to pay it.
• 3rd September – Cross connect location resolved, circuit plugged in, service starts functioning.

Shortly after this we put our order form live and improved our implementation, we received our first order on the 9th September and provisioned a few days later. Our third transit customer is up and live, order form to fully working was just under twelve hours; comfortably within our promise of two working days.

Two new fans of our Raspberry Pi cloud.

It’s been less than two months since we launched the Raspberry Pi 4 into our public cloud. Take-up exceeded our predictions to the extent that we briefly ran out of stock and had to accelerate our expansion.

We now have Pi 4 servers back in stock, and we’ve also added OS images for 64-bit Raspberry Pi OS and Ubuntu.

64-bit operating systems offer significant benefits for some server applications. For example, MongoDB limits your database size to 2GB if you’re on a 32-bit host. It’s also the case that larger ARM servers only support 64-bit operating modes, so this addition brings us compatibility with the general ARM server ecosystem.

We’ve also boosted the cooling in our Raspberry Pi cloud by adding higher throughput fan trays. The new trays move 336m³/h, and the shelf is 0.05m³, so the air should change at least once per second. We are seeing maximum on chip temperatures (measured by vcgencmd measure_temp) of 59°C, which is considerably below the 80°C threshold where CPU throttling starts.

Cambridge Freegle pictured on a map backed by OpenStreetMap tiles from the Mythic Beasts hosted tile server.

We’re supporters of Freegle, a charity that recycles unwanted things by passing them on to new owners. As the COVID-19 lockdown is eased, many people have de-cluttered and have things available to be passed on to new owners. Similarly, a number of people have been struggling financially and will benefit from donations. Traffic on Freegle has rocketed.

Freegle used to use Google Maps for displaying items. In 2018, Google changes the terms for their maps service moving to pay-as-you-go, per-tile-served pricing model. Many sites are able to operate within the a $200/month fee credit, which buys 200,000 monthly tile requests. Freegle is now seeing enough usage to incur bills of over £750/month for map tiles — a significant expense for a small charity.

As is often the case with usage-based cloud services, a free, or very low, initial price can quickly escalate into a large and uncontrollable cost.

Fortunately, as is often the case, a comparable alternative based on open source software exists and can provide a much lower total overall cost.

Freegle contacted us looking for help in moving to their own tile server based on OpenStreetMap, providing lower – and just as importantly – fixed monthly costs.

Running an OpenStreetMap tile server

Freegle are using a Mythic Beasts virtual server to host OpenStreetMap docker image, fronted by NGINX to provide HTTPS and HTTP/2 support. The initial approach of rendering tiles on demand proved to be far too slow, so tiles are now pre-rendered and cached on SSD. Full details can be found in their article, Junking Google Maps for OpenStreetMap.

The initial pre-rendering is being done with a 256GB/16 core server. This is expected to complete within a few days, and once done, the server will be scaled down to 16GB/4 cores for normal production usage.

Costs for this custom solution? One working day of staff time, a few days of a fast virtual server (~£60), and the monthly cost of the product virtual server (~£50) which nets current monthly savings of £700 and gives long term guaranteed price stability.

The convenience of cloud without the price tag

Being based on open source software, there’s no risk of a future change in terms making the service unaffordable, and Freegle aren’t locked in to a single provider’s proprietary API. If we were to hike our prices, Freegle could easily move their service to another provider (although based on recent experience, we’re more likely to do the opposite).

Freegle implemented this service themselves on our VPS platform, but we can also offer this as a managed application, giving the convenience of a cloud-style service, but without the cloud-style lock-in and pricing.

Time passed and everything grew.

We have just rolled out a substantial update to our price list for virtual private servers.

The new price list is significantly better value, and also introduces the ability to specify storage independently of RAM and CPU. Servers can be configured with either SSD or HDD-backed storage, with sizes ranging from 5GB to 4TB.

This is immediately available in all six VPS zones: London UK (HEX, MER and SOV), Cambridge (UK), Amsterdam (NL) and Fremont (US).

Better prices

Our base prices for virtual servers have decreased, making them even better value with prices now starting from £47/year. CPU, RAM and disk space have all fallen in price. The only price we haven’t reduced is our IPv4 address pricing, but we have held that constant, despite the continuing depletion of the world’s limited supply of these legacy addresses.

More options

We have expanded our range of products. To meet customer demand for larger servers, we’ve now added 192GB and 256GB options with up to 16 cores. We’ve also introduced additional intermediate products.

More capacity

In addition to adding our US zone recently, we have added more capacity in all four of our UK zones to support upgrades and additional customers.

New OS images

We have also improved our standard OS images to support our new enhanced DNS infrastructure. We’re now automatically recreating and retesting them, rather than security updating on first install. This reduces the amount of time taken for your VPS to be provisioned in all of our sites.

Existing customers

We have always avoided unsustainable introductory pricing, and “new customer only” offers. We prefer to reward loyalty, which is why existing customers have already received an email with details of a specification upgrade that puts them on an even better deal than our new list pricing.

Our PI 4 servers all wear the Power over Ethernet HAT to provide power and cooling to the CPU.

We’re now offering these in our Raspberry Pi Cloud starting from £7.50/month or 1.2p/hour.

Since the release of the Raspberry Pi 4 last year, it’s been an obvious addition to our Raspberry Pi cloud, but it’s taken us a little while to make it happen. Our Raspberry Pi Cloud relies on network boot in order to ensure that customers can’t brick or compromise servers and, at launch, the Pi 4 wasn’t able to network boot. We now have a stable replacement firmware with full PXE boot support.

The Pi 4 represents a significant upgrade over the Pi 3; it is over twice as fast, has four times the RAM and the network card runs at full gigabit speed. On a network-booted server this gives you much faster file access in addition to more bandwidth out to the internet. We’ve done considerable back-end work to support the Pi 4. We’ve implemented:

• New operating system images that work on the Pi 4 for 32 bit Raspberry Pi OS and Ubuntu.
• A significant file server upgrade for faster IO performance.
• Supporting the different PXE boot mode of the Pi 4 without impacting our Pi 3 support.

Ben Nuttall has been running some secret beta testing with his project Pi Wheels which builds Python packages for the Raspberry Pi. We’re grateful for his help.

Is it any good?

tl;dr – YES

We’ve historically used WordPress as a benchmarking tool, mostly because it’s representative of web applications in general and as a hosting company we manage a lot of those. So we put the Raspberry Pi 4 up against a Well Known Cloud Provider that offers ARM instances. We benchmarked against both first generation (a1) and second generation (m6g) instances.

Our test was rendering 10,000 pages from a default WordPress install at a concurrency level of 50.

In both cases the Pi 4 is approximately twice as fast at a quarter of the price.

Notes:

• Raspberry Pi 4 monthly price based on on-demand per-second pricing.
• USD to GBP conversion from Google on 17th June 2020

Now offering servers at 122″ W

In 2018, we gained a small presence in the US thanks to our acquisition of BHost.  Establishing a US presence had been a long-standing company plan, and the acquisition gave us a very useful starting point.

Whilst we’ve been supporting and upgrading existing customers in the US, we wanted to implement some network and infrastructure changes before taking on any new customers.

In early February we combined a trip to the North American Network Operator Group in California with a substantial deployment of new equipment into the facility in Fremont, and we’re pleased to announce that our US site is now fully open for business.

New VPS cloud

We’ve deployed a cluster of brand new VPS host servers, and Fremont is now available as a zone when ordering a VPS. As we’ve done elsewhere, we’re also migrating all ex-BHost customers into our VPS cloud, upgrading everyone to KVM-based virtual machines with newer faster hardware. For ex-BHost users on the OpenVZ containerisation platform this is a significant upgrade to full virtualisation with no hardware contention and at no additional cost.

New DNS resolvers

Fremont to London latency is approximately 130ms.  To support US-based servers, we’ve deployed new resolvers in Fremont so that DNS resolution can be local and fast. This includes local DNS64/NAT64 servers for the benefit of IPv6-only hosting customers. We are also mirroring this improvement to our Cambridge and Amsterdam data centres for faster DNS resolution and local NAT64 in all our sites.

This is the Fremont Internet Exchange. We connect on the yellow fibre.

Network services

We’re now operating our own, fully routed network in the Fremont 2 data centre, and can offer network services to VPS and colocation customers in this facility. You can bring your own IP space to your virtual machine, and you can have BGP sessions to dynamically advertise your routes. Customers taking BGP will see a full routing table, combining transit routes from our upstreams with shorter, faster routes through the internet exchanges. We can also offer very low bandwidth connections (suitable for out-of-band connectivity), and transit connections with 95th percentile billing within the Fremont 2 data centre.

Network core

We’ve deployed a pair of routers to provide improved redundancy. These each have a full internet uplink and a link to a peering exchange. One connects at 10Gbps to the San Francisco Metropolitan Internet Exchange and the other at 10Gbps to the Fremont Cabal Internet Exchange. We’re peering as Autonomous System Number 60011 (in Europe we’re 44684) and now accepting peering requests over those exchanges. The BHost cloud is now behind this new routed network.

This means that in addition to improved and increased capacity, we’re also able to offer BGP to customers in our US site and transit sessions to other networks in the facility.

Virtual server features

Consolidating our US zone on our existing virtual server platform means that US virtual servers will benefit from the same technical advantages that we offer in other locations:

• VNC and virtual serial — virtual serial allows you to log your commands with working cut and paste. If your server crashes, the serial will log everything the kernel prints for later examination.
• Bring your own ISO — install any operating system you like.
• Optional BGP feed.
• Managed service options.
• IPv4 and IPv6 connectivity.
• Sympl, our open source server automation platform, is available out of the box.

Management in a different timezone

We offer full managed hosting on our US servers too, although we’ll be doing scheduled security updates starting from 7am US PST, not BST. We already run 24/7 operations so there is no difficulty in being able to offer our US customers the same management services that our EU customers get and we’re more than happy to schedule updates for in or out of your working hours.

Microscopy image of the coronavirus.
(Image copyright NIAID; licensed under CC-BY 2.0)

Covid-19 has dramatically changed life in the UK, and the lock-down announced on Monday has lead to further changes to how we, and our customers, are operating.  This page provides an update to our previously announced Covid-19 plan.

We’re happy to report that our staff member who had some of the symptoms of covid-19 is now fully recovered and has returned to our team after a few days of rest.

Data centre access changes

Our data centre suppliers have altered their operation: 24/7 walk in access is now prohibited and every visit needs booking and justification. The “remote hands” service remains available, but at a reduced capacity as they’ve moved as many staff as possible to home working to minimise their risks.

Equinix, who supply two of our core facilities, have completely closed their facilities in Italy, Germany, France and Spain to customers. Changes are only possible via their remote hands service in those countries.  We have a significant amount of equipment in two Equinix facilities in London and Amsterdam (LD8 and AM5).

Since this announcement on Sunday, we have been anticipating a similar closure being applied to London.  This has  now been announced and will be disruptive for us, as our normal operating procedures rely on being able to move spare equipment easily between the London data centres where we have a presence.  We have been taking steps to increase the spare hardware that we have at each of our London sites in order to mitigate the impact of this when it comes into force on Tuesday 31st March.

The data centres have well-considered policies to reduce risk, and to handle a confirmed case within the facility with rapid quarantine and deep clean procedures. No access is allowed to customers showing symptoms, and all customers’ temperatures are measured before entry to the facilities. Their operations are also robust in the event they need to manage the facility remotely for a period of time.

We’ve also altered our own policy for data centre access. Customer access to our data-centre space is suspended until further notice, including for documentation-related audits (e.g. ISO27001 compliance). This should have a minimal impact; we only allowed accompanied access previously and visits have always been exceedingly infrequent.

Items shipped to us will be quarantined for 24 hours before opening. Cardboard will self-decontaminate in 24 hours.

Staff members may not meet in a data centre unless it is specifically for a piece of work that requires two people for safety reasons (typically very heavy server deployment), and only if is to maintain an essential service. Staff members may not visit multiple key data centres in a single visit to minimise the risk of transmission between key sites, and may not visit if they are showing symptoms. Data centre visits are being minimised to reduce infection risk. This may limit the range of dedicated servers we are able to provision, and we have decided to stop offering Mac Minis with OS X due to the difficulty of provisioning them remotely.

Customer support

Unsurprisingly, a wholesale shift of the UK to remote working has a significant impact on all kinds of online systems, which are now critical for day to day operations. We’re supporting existing customers to make this transition, as well as provisioning new orders for services that now need to be in the cloud.

We run a system for POhWER that is used by all their advisors to track their cases. This is a critical system; if it’s offline, hundreds of people are unable to work. We maintain this as an active/standby pair split across two of our data centres.

“You’re supporting us to enable our vital work with the most vulnerable people in society to continue in these very trying times and, through your swift upgrade actions, our new fully remote working model is delivering the information, advice and advocacy our clients depend on.”
Sandra Black, Head of Training, Risk and Quality at POhWER.

The shift to remote working means that usage on their system has approximately doubled in ten days and has started to see performance limitations. We identified the bottleneck and proposed a cost-effective upgrade combined with some configuration improvements. We then made staff available to apply the changes in an emergency late night maintenance window, restoring their site to full performance by the next working day.

Direct efforts

We were approached at the weekend by a small team comprising local IT experts and doctors who are building an information website to efficiently distribute information to NHS staff members about how to use and select the correct protective equipment for the environment they are working in. We’re providing the virtual server, security updates, backups and 24/7 monitoring service for this free of charge, which has allowed the volunteer IT experts to concentrate on building the site. We’re expecting go-live in the next day or so once the content is checked.

We’re keen to hear of any other efforts where we may be able to assist.

Adding capacity

We’ve ordered more servers to expand the three busiest VM clouds to support existing customers scaling up, and new customers with urgent needs. We want to avoid cloud full and thankfully our server supplier is fully able to continue to build and deliver servers whilst maintaining 2m spacing between employees.

The European Bio-informatics Institute used x-rays and a lot of hard maths to draw the above picture of the main covid-19 protein.

We provide critical infrastructure to many other companies who rely on our services. Covid-19 could significantly change day to day life everywhere. At present we don’t believe it will have a significant effect on our operations.

Remote working

Mythic Beasts has always been a distributed company with no central office; all staff members normally work from home. As a result adopting remote working recommendations or enforcement will have no significant impact on our day to day operations. Normally, we have a weekly optional meeting for staff around Cambridgeshire and a compulsory all company meeting roughly once every six weeks. Migrating these meetings to conference calls will have minimal operational impact.

Reducing travel

Our sales process has always been online. We don’t routinely meet customers and have a very light attendance at conferences and industry events. Our next scheduled events are UKNOF (Manchester, April) and LINX (Manchester) both of which offer remote participation should it become necessary.

Financial stability

Mythic Beasts has been profitable every year since 2001 and carries no debt. We maintain significant cash reserves so we can self-finance routine expansion and other business opportunities, and weather unforeseen circumstances. We understand that many consider this an inefficient use of capital, but we can definitely pay our bills in the event of a global pandemic crippling the economy for a short period.

Sensible staff policies

Our staff members are provided with private healthcare. They also get sick leave which they are expected to take if ill. We also provide 30 days + bank holidays holiday to all staff members as standard and we strongly encourage them to have a two week contiguous holiday in order that we know we can operate without them in the event of sickness. We have sufficient staff levels that in the event of multiple staff members being ill for an extended period day to day operations can be maintained and only longer term projects should be delayed.

Supplier issues

We maintain stock of key components to cope with hardware failure although for some months we’ve been seeing very long lead times for hardware, especially CPUs and SSDs. This may impact our lead times for larger orders of dedicated servers or custom hardware. We think it likely that our data centre and connectivity providers may implement a change freeze as they do annually over the Christmas period and at other key times (e.g. the 2012 Olympics). This is a familiar operating environment and utilising multiple providers in multiple countries will help to mitigate this.

2020-03-12 : We run our own private phone conference and IRC service so we’re not affected by the reported load issues at the major public providers Slack, Teams, Zoom etc.

In summary, we think that day to day operations shouldn’t be affected but if you have any concerns get in touch at support@mythic-beasts.com.