A log cut into snail shells (public domain image from Simpon Speed)

On Friday 10th December we became aware of an extremely serious security issues in Log4J, a logging component in widespread use by applications written in the Java programming language.  The vulnerability has been nicknamed Log4Shell.

What is Log4Shell and Log4J

Log4J is a library to make writing data to a log file easier. It’s highly configurable to make it easy to send the right level of logging data to the right place and it includes bits of intelligence so you can log placeholders and have Log4J fill in the correct value for the environment. So if you’re logging an error in your application and you want to know what version of java is currently running your application you can log:

${java:version}

which will be replaced with the currently running version number of Java.

However, it is very common for log messages to contain user-supplied data.  For example, a login form might log the username from a failed login attempt, and many applications don’t check the data the user supplied for magic values like this.  So, if I were to attempt to log in with a username of ${java:version} instead of Pete, the logfiles will say:

Failed login attempt for user: "OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)"

rather than what the application developer expected which would be:

Failed login attempt for user: "${java:version}"

One of the other magic strings uses LightWeight Directory Access Protocol (LDAP) to look up data from a remote server and the remote server can specify additional software to install and run in order to process the answer from the LDAP server.

If an end user can set something that will go to a log file to a magic LDAP string pointing at a server they control they can make the java application request code from that server and make the target system execute code they just supplied. This effectively hands full control over the java application to the person that logged the magic LDAP string. Effectively you can turn a piece of data that is logged into an administrative shell on the target server, hence the name Log4Shell.

The vulnerability is very nasty for a number of reasons. Firstly, it’s a trivial-to-exploit remote code execution vulnerability. You literally send the application a URL to the code you want run and it runs it. Secondly, Log4J is very widely used, including in custom software, and many applications are likely to be vulnerable.

Managed customers

As part of our server management service, we monitor and assess all security advisories for operating system packages, applying serious 0-day vulnerabilities immediately to customer servers.

Unfortunately, Java applications almost never use system-provided libraries, and will instead bundle their dependencies as part of the application. From the point of view of our managed service, updating Java applications with an embedded Log4J is the technically the responsibility of the customer.

However, given the severity and ease of exploit of this vulnerability, we’ve been doing everything we can to help customers who may not even know that they’re reliant on Log4J, let alone where their application is vulnerable.

Going above and beyond

As part of our managed service we install an internally written package called Mythic Reporter. This logs a lot of data from servers every day about what the servers are doing. We then have a centralised process that reads the reports and automates auditing for common issues. With this we can spot things like:

• One of the hardware devices in your storage array is broken or is in a pre-failure state.
• Database replication appears not to be working.
• A filesystems has gone read-only.
• You have mirrored filesystems but not mirrored swap space.
• The cryptographic keys used by ssh that are weak or are blacklisted.
• You have a database running but no backups configured.
• You’re using the stock i40 network module for Debian which is unstable.
• Your server has thermally throttled.
• … and many others.

We can also utilise this dataset for other things. We log the full process list and listening network sockets for every managed server every day. So it’s a small matter of scripting on our reporter server to find the full list of client servers that have a network listening application written in Java. One staff member set about writing a customer notification, one understanding how nasty the security issue was and one building the full list of likely affected customers.

To every managed server customer running a java server process, we sent this email:

We have become aware of a serious security vulnerability in the log4j
logging package for Java. You're receiving this email because our 
records show that your managed server is running Java.

At this point, a full list of applications that are affected by this
vulnerability is not available, but given the widespread use of log4j, 
the severity of the vulnerability (remote code execution) and the
typical ease of exploitation, we strongly recommend investigating
proactively whether any Java applications that you are using are
vulnerable.

Your Mythic Beasts managed service includes monitoring and upgrading of
operating system packages, but does not cover software installed by
other means.  Java applications typically rely on JAR files that are not
provided by system packages, and in this case we are not able to detect
or apply necessary upgrades.

You can find more information on the vulnerability, and the affected
versions of log4j, here:

  https://www.lunasec.io/docs/blog/log4j-zero-day/

Whilst we cannot assess whether your server is vulnerable to this
vulnerability, we are happy to provide advice based on the information
that we have.

We detected Java running on the following servers:

-- list of servers --

We then opened tickets in our ticket tracking system for all affected customers so we could close them off once we’re confirmed they were either not vulnerable, or had been patched.

Auditing

We then started auditing the identified customer servers, scanning for installations of the Log4J library and notifying customers as to whether the libraries they have installed are vulnerable or not. We utilised reports from software providers to prioritise fixes. For example Jenkins may be affected depending on the plugins used.

We have worked through the list contacting every customer to confirm if we or they could upgrade the affected component or if we could mitigate through configuration changes, and this afternoon we have been chasing likely affected customers who haven’t responded to encourage them strongly to work with us to fix this issue.

If you run Java-based services and you’re not already a customer of our managed hosting service, then you’ve probably been quite busy over the last few days. If you haven’t been, then you may want to consider signing up.

Dependency management

Log4Shell is a somewhat vicious lesson in dependency management. Every time you import third party code, you need a process for monitoring security advisories for it, and for updating it as required. This is why we have a strong preference for using operating system packages wherever practical, as this delegates the whole problem to the operating system maintainers and makes automatically finding and updating affected libraries trivial. Being able to automatically find vulnerable packages is critical, as you can be guaranteed that when a serious vulnerability is discovered, the bad guys will automate it.

Yesterday (7th December) we attended the virtual IPv6 forum annual meeting. We were delighted that our director Pete Stevens has been added to the IPv6 Hall Of Fame as an IPv6 Deployment World Leader.

Unlike most awards we turn down, you can’t win this one just by paying for a hugely expensive table at an awards ceremony.

We also got an update on how IPv6 deployment is going through the UK. Happy to hear from BT that they’re making excellent progress replacing all the old HomeHubs with new IPv6-capable consumer routers. Sky Italia has deployed a consumer broadband network that’s effectively IPv6 only – IPv4 is provided as a service on top with MAP-T. As this is a form of carrier NAT they’ve managed one IPv4 address per 16 subscribers. This compares with their initial dual stack rollout we reported on from the 2019 council meeting.

Lastly it was noted that the cost of an IPv4 address on the open market is now around $60; increasing numbers of server providers are following our lead and making an IPv4 address an additional and removable option on the order form.

We’re delighted to announce our sponsorship of Organic Maps. Organic Maps is a simple, user-friendly application that downloads complete Open Street Map data to your phone, allowing you to use their mapping application offline complete with route planning from the on-device database.

This is a wonderful application. It doesn’t track you, advertise at you or flood you with non-notifications, and it works without mobile data and conserves battery life. So if you’ve ever been lost without signal or somewhere where roaming data is prohibitively expensive, or to a very busy location where the mobile networks were overloaded this application is genuinely better than the alternatives.

While the app avoids the need for mobile data, this comes at the cost of a significant up-front download of all the mapping data that you may or may not use offline. This won’t trouble typical home broadband, but for the servers at the other end it adds up quickly. We’ve stepped in and offered two 4GB virtual servers with 400TB/month of free bandwidth to Organic Maps, split between our London and Amsterdam zones, reducing the reliance on a traditional and bankruptcy-inducing large cloud provider.

“Use the cloud, it’s cheap,” people often say, incorrectly.

A competing quote from a slightly cheaper large cloud provider

At our list prices this would be somewhat cheaper:

Being 90% better value is achieved in part by not having to fund our own space programme.

A small galaxy hit the bullseye of NGC922 about 330m years ago. More information: www.spacetelescope.org/images/heic1218a/
Credit:
NASA, ESA

Congratulations to the Debian team for their new release of Debian Bullseye (11). Just over two years of hard work have resulted in over 40,000 package updates and 10,000 additions.

We’ve made images for our VPS cloud that are available in all regions and included the install ISO for customers who prefer to build their own OS images. Sympl, a management package for web and email hosting that we maintain has been updated to support Bullseye with packages available for download.

Our mirror server is up to date with the Debian Bullseye packages. We’ll now be looking at deploying new systems on Debian Bullseye and starting our upgrade program for Debian Stretch and Buster systems.

The UK Debian folks will be having a small party in Cambridge in a few days time and we’re sponsoring the beer to say thank you. It’s a weekend full of beer and barbeques.

Our Pi 4 servers all wear the Power over Ethernet HAT to provide power and cooling to the CPU.

Since the launch of the 8GB Raspberry Pi 4 we’ve had many requests to add these to our Raspberry Pi cloud. Meanwhile many Raspberry Pi users have read about overclocking the Raspberry Pi and running at a higher clock speed.

Overclocking further increases the computing power of the Pi, but brings significant operational issues for our Pi cloud. Not all Raspberry Pi hardware will run reliably at the higher clockspeed and the higher voltage required to support it. Increasing the clockspeed and voltage significantly increases the power consumption and thus the cooling requirements necessary to prevent overheating. We’ve spent a considerable amount of time testing and we’re now ready to launch our first 8GB Raspberry Pi 4 cluster. We’re offering them at two clock speeds: the stock 1.5GHz and overclocked to 2GHz.

The overclocked Raspberry Pis have all been run at a significant CPU load for several weeks to test their stability before release. Any that failed the stability test have been added to the cloud at the normal 1.5GHz clockspeed.

The 8GB Pi is available at 1.5GHz and 2GHz clock speeds. Supported operating systems are Raspberry Pi OS 64 and Ubuntu 64.

Larger fans provide more cooling to our 8GB Pi4 cloud so we can run at higher clockspeeds.

We’ve had a variety of customer being very complimentary recently. Andy Steven runs a series of web cams in the Shetland Islands that stream live views of the northern lights. The cameras relay the stream via one of our virtual servers in our MER data centre and the current bandwidth record is several Gbps.

I am proud to say that our new ‘AuroraCam’ network just delivers and for the first time I no longer break out in a sweat watching the demand increase from that AuroraWatchUK alert or a celebrity weather personality sending out a Tweet.

— Andy Steven, Shetland Webcams (full article)

Beautiful shot of the northern lights captured by Shetland Webcams. Could be improved by adding a kitten though.

We provide 10Gbps fibre connectivity to the Cambridge office of DarkTrace. Darktrace uses machine learning to identify and neutralise security threats in real time.

You’ve been much more transparent & approachable than any provider I’ve dealt with previously. Very happy with the service so far.

— Harry Godwin, Head of Business Infrastructure. Darktrace

The Web hosting review and advice site Hosting Advice interviewed us and wrote a great article about the management and infrastructure services we provide.

Recognizing that there is no one-size-fits-all approach to managed hosting, Mythic Beasts can take on varying responsibility levels as needed. This range of services includes everything from ensuring that servers are up and running to providing the extensive monitoring, security, and assistance necessary to keep custom web applications functioning reliably.

— Hosting Advice (full article)

Lastly our strong stance about returning Nominet to its public benefit roots garnered entirely positive responses at Twitter.

Very pleased to see that our domain registrar of choice @Mythic_Beasts has signed the call to improve governance of Nominet. https://t.co/z7p8Nfdi94 – we were going to ask them to sign but not surprised they’ve already done so. #kudos #domainnames #nominet

— The Familiar (@_The_Familiar) February 2, 2021

Well done, Mythic. Turning Nominet back into a public interest custodian of .uk is long overdue.https://t.co/d2lLGskK0Y https://t.co/yGUmBwg5Ph

— Jim Hague (@banbury_bill) February 1, 2021

We have signed up to Public Benefit, an effort to restore Nominet to its roots as a public benefit, not for profit organisation.

Nominet runs a world class registry for domains ending in .uk. Their technical execution is faultless and we’re extremely happy with all the services they provide for .uk domains.

A ccTLD domain registry is a natural monopoly, and a profitable one at that. For many years, Nominet have donated their surplus to the Social Tech Trust (formerly the Nominet Trust, which was renamed after they cut funding), a charity that uses technology for the public good.

Charitable donations have dwindled whilst prices have increased over the last five years, due to spending on loss making research projects such as self driving cars and Radio Spectrum management, not to mention last year’s £249,000 pay rise for the CEO (to £772,000).

We are strongly in favour of the proposal of Axel Pawlik, former MD of RIPE, as a director. Under Axel’s leadership, RIPE achieved many significant improvements to internet infrastructure including, but not limited, to:

• Managing IPv4 address exhaustion, balancing the needs of existing ISPs while preserving access for new entrants;
• Encouraging and facilitating IPv6 uptake;
• Encouraging uptake of RPKI to secure routing announcements (RIPE now has the highest participation rate of any RIR); and
• Creating RIPE Atlas, a communal tool to track routing that makes running an ISP much easier.

Sir Michael Lyons also appears to be a sound proposal, although beyond his earlier report on Nominet governance, we have no day-to-day experience of his work.

Nominet is structured such that the elected non-executive directors are out-numbered and are unable to achieve meaningful change, which is why after years of dissatisfaction this has come to an Extraordinary General Meeting to remove the existing directors. Voting is weighted in a complicated fashion, but the more domains the member controls the more important their vote is. As a result domain owners can effectively vote by switching registrars, and if you would like to support this proposal we would recommend moving any .uk domains to a registrar that has signed up to call the EGM. Nominet are very good at actually running the registry, and .uk domain transfers are very easy, and free.

Back in 2018, we acquired BHost, a virtual hosting provider with a presence in the UK, the Netherlands and the US. Since the acquisition, we’ve been working steadily to upgrade the US site from a single transit provider with incomplete IPv6 networking and a mixture of container-based and full virtualisation to what we have now:

• Dual redundant routers
• Two upstream network providers (HE.net, CenturyLink)
• A presence on two internet Exchanges (FCIX/SFMIX)
• Full IPv6 routing
• All customers on our own KVM-based virtualisation platform

With these improvements to our network, we’re now able to offer IPv4 and IPv6 transit connectivity to other customers in Hurricane Electric’s Fremont 2 data centre. We believe that standard services should have a standard price list, so here’s ours:

Prices start at £60/month on a one month rolling contract, with discounts for longer commits. You can order online by hitting the big green button, we’ll send you a cross-connect location within one working day, and we’ll have your session up within one working day of the cross connect being completed. If we don’t hit this timescale, your first month is free.

We believe that ordering something as simple as IP transit should be this straightforward, but it seems that it’s not the norm. Here’s what it took for us to get our second 10G transit link in place:

• 24th April – Contact sales representative recommended by another ISP.
• 1st May – Contact different sales representative recommended by UKNOF as one of their sponsors.
• 7th May – 1 hour video conference to discuss our requirements (a 10Gbps link).
• 4th June – Chase for a formal quote.
• 10th June – Provide additional details required for a formal quote.
• 10th June – Receive quote.
• 1st July – Clarify further details on quote, including commit.
• 2nd July – Approve quote, place order by email.
• 6th July – Answer clarifications, push for contract.
• 7th July – Quote cancelled. Provider realises that Fremont is in the US and they have sent EU pricing. Receive and accept higher revised quote.
• 10th July – Receive contract.
• 14th July – Return signed contract. Ask for cross connect location.
• 15th July – Reconfirm the delivery details from the signed contract.
• 16th July – Send network plan details for setting up the network.
• 27th July – Send IP space justification form. They remind us to provision a cross connect, we ask for details again.
• 6th August – Chase for cross connect location.
• 7th August – Delivery manager allocated who will process our order.
• 11th August – Ask for a cross connect location.
• 20th August – Ask for a cross connect location.
• 21st August – Circuit is declared complete within the 35 day working setup period. Billing for the circuit starts.
• 26th August – Receive a Letter Of Authorisation allowing us to arrange the cross connect. We immediately place order for cross connect.
• 26th August – Data centre is unable to fulfil cross connect order because the cross connect location is already in use.
• 28th August – Provide contact at data centre for our new provider to work out why this port is already in use.
• 1st September – Receive holding mail confirming they’re working on sorting our cross connect issue.
• 2nd September – Receive invoice for August + September. Refuse to pay it.
• 3rd September – Cross connect location resolved, circuit plugged in, service starts functioning.

Shortly after this we put our order form live and improved our implementation, we received our first order on the 9th September and provisioned a few days later. Our third transit customer is up and live, order form to fully working was just under twelve hours; comfortably within our promise of two working days.

Two new fans of our Raspberry Pi cloud.

It’s been less than two months since we launched the Raspberry Pi 4 into our public cloud. Take-up exceeded our predictions to the extent that we briefly ran out of stock and had to accelerate our expansion.

We now have Pi 4 servers back in stock, and we’ve also added OS images for 64-bit Raspberry Pi OS and Ubuntu.

64-bit operating systems offer significant benefits for some server applications. For example, MongoDB limits your database size to 2GB if you’re on a 32-bit host. It’s also the case that larger ARM servers only support 64-bit operating modes, so this addition brings us compatibility with the general ARM server ecosystem.

We’ve also boosted the cooling in our Raspberry Pi cloud by adding higher throughput fan trays. The new trays move 336m³/h, and the shelf is 0.05m³, so the air should change at least once per second. We are seeing maximum on chip temperatures (measured by vcgencmd measure_temp) of 59°C, which is considerably below the 80°C threshold where CPU throttling starts.

Cambridge Freegle pictured on a map backed by OpenStreetMap tiles from the Mythic Beasts hosted tile server.

We’re supporters of Freegle, a charity that recycles unwanted things by passing them on to new owners. As the COVID-19 lockdown is eased, many people have de-cluttered and have things available to be passed on to new owners. Similarly, a number of people have been struggling financially and will benefit from donations. Traffic on Freegle has rocketed.

Freegle used to use Google Maps for displaying items. In 2018, Google changes the terms for their maps service moving to pay-as-you-go, per-tile-served pricing model. Many sites are able to operate within the a $200/month fee credit, which buys 200,000 monthly tile requests. Freegle is now seeing enough usage to incur bills of over £750/month for map tiles — a significant expense for a small charity.

As is often the case with usage-based cloud services, a free, or very low, initial price can quickly escalate into a large and uncontrollable cost.

Fortunately, as is often the case, a comparable alternative based on open source software exists and can provide a much lower total overall cost.

Freegle contacted us looking for help in moving to their own tile server based on OpenStreetMap, providing lower – and just as importantly – fixed monthly costs.

Running an OpenStreetMap tile server

Freegle are using a Mythic Beasts virtual server to host OpenStreetMap docker image, fronted by NGINX to provide HTTPS and HTTP/2 support. The initial approach of rendering tiles on demand proved to be far too slow, so tiles are now pre-rendered and cached on SSD. Full details can be found in their article, Junking Google Maps for OpenStreetMap.

The initial pre-rendering is being done with a 256GB/16 core server. This is expected to complete within a few days, and once done, the server will be scaled down to 16GB/4 cores for normal production usage.

Costs for this custom solution? One working day of staff time, a few days of a fast virtual server (~£60), and the monthly cost of the product virtual server (~£50) which nets current monthly savings of £700 and gives long term guaranteed price stability.

The convenience of cloud without the price tag

Being based on open source software, there’s no risk of a future change in terms making the service unaffordable, and Freegle aren’t locked in to a single provider’s proprietary API. If we were to hike our prices, Freegle could easily move their service to another provider (although based on recent experience, we’re more likely to do the opposite).

Freegle implemented this service themselves on our VPS platform, but we can also offer this as a managed application, giving the convenience of a cloud-style service, but without the cloud-style lock-in and pricing.