Log4J security issues

December 13th, 2021 by

A log cut into snail shells (public domain image from Simpon Speed)

On Friday 10th December we became aware of an extremely serious security issues in Log4J, a logging component in widespread use by applications written in the Java programming language.  The vulnerability has been nicknamed Log4Shell.

What is Log4Shell and Log4J

Log4J is a library to make writing data to a log file easier. It’s highly configurable to make it easy to send the right level of logging data to the right place and it includes bits of intelligence so you can log placeholders and have Log4J fill in the correct value for the environment. So if you’re logging an error in your application and you want to know what version of java is currently running your application you can log:

${java:version}

which will be replaced with the currently running version number of Java.

However, it is very common for log messages to contain user-supplied data.  For example, a login form might log the username from a failed login attempt, and many applications don’t check the data the user supplied for magic values like this.  So, if I were to attempt to log in with a username of ${java:version} instead of Pete, the logfiles will say:

Failed login attempt for user: "OpenJDK Runtime Environment (build 11.0.11+9-Ubuntu-0ubuntu2.20.04)"

rather than what the application developer expected which would be:

Failed login attempt for user: "${java:version}"

One of the other magic strings uses LightWeight Directory Access Protocol (LDAP) to look up data from a remote server and the remote server can specify additional software to install and run in order to process the answer from the LDAP server.

If an end user can set something that will go to a log file to a magic LDAP string pointing at a server they control they can make the java application request code from that server and make the target system execute code they just supplied. This effectively hands full control over the java application to the person that logged the magic LDAP string. Effectively you can turn a piece of data that is logged into an administrative shell on the target server, hence the name Log4Shell.

The vulnerability is very nasty for a number of reasons. Firstly, it’s a trivial-to-exploit remote code execution vulnerability. You literally send the application a URL to the code you want run and it runs it. Secondly, Log4J is very widely used, including in custom software, and many applications are likely to be vulnerable.

Managed customers

As part of our server management service, we monitor and assess all security advisories for operating system packages, applying serious 0-day vulnerabilities immediately to customer servers.

Unfortunately, Java applications almost never use system-provided libraries, and will instead bundle their dependencies as part of the application. From the point of view of our managed service, updating Java applications with an embedded Log4J is the technically the responsibility of the customer.

However, given the severity and ease of exploit of this vulnerability, we’ve been doing everything we can to help customers who may not even know that they’re reliant on Log4J, let alone where their application is vulnerable.

Going above and beyond

As part of our managed service we install an internally written package called Mythic Reporter. This logs a lot of data from servers every day about what the servers are doing. We then have a centralised process that reads the reports and automates auditing for common issues. With this we can spot things like:

  • One of the hardware devices in your storage array is broken or is in a pre-failure state.
  • Database replication appears not to be working.
  • A filesystems has gone read-only.
  • You have mirrored filesystems but not mirrored swap space.
  • The cryptographic keys used by ssh that are weak or are blacklisted.
  • You have a database running but no backups configured.
  • You’re using the stock i40 network module for Debian which is unstable.
  • Your server has thermally throttled.
  • … and many others.

We can also utilise this dataset for other things. We log the full process list and listening network sockets for every managed server every day. So it’s a small matter of scripting on our reporter server to find the full list of client servers that have a network listening application written in Java. One staff member set about writing a customer notification, one understanding how nasty the security issue was and one building the full list of likely affected customers.

To every managed server customer running a java server process, we sent this email:

We have become aware of a serious security vulnerability in the log4j
logging package for Java. You're receiving this email because our 
records show that your managed server is running Java.

At this point, a full list of applications that are affected by this
vulnerability is not available, but given the widespread use of log4j, 
the severity of the vulnerability (remote code execution) and the
typical ease of exploitation, we strongly recommend investigating
proactively whether any Java applications that you are using are
vulnerable.

Your Mythic Beasts managed service includes monitoring and upgrading of
operating system packages, but does not cover software installed by
other means.  Java applications typically rely on JAR files that are not
provided by system packages, and in this case we are not able to detect
or apply necessary upgrades.

You can find more information on the vulnerability, and the affected
versions of log4j, here:

  https://www.lunasec.io/docs/blog/log4j-zero-day/

Whilst we cannot assess whether your server is vulnerable to this
vulnerability, we are happy to provide advice based on the information
that we have.

We detected Java running on the following servers:

-- list of servers --

We then opened tickets in our ticket tracking system for all affected customers so we could close them off once we’re confirmed they were either not vulnerable, or had been patched.

Auditing

We then started auditing the identified customer servers, scanning for installations of the Log4J library and notifying customers as to whether the libraries they have installed are vulnerable or not. We utilised reports from software providers to prioritise fixes. For example Jenkins may be affected depending on the plugins used.

We have worked through the list contacting every customer to confirm if we or they could upgrade the affected component or if we could mitigate through configuration changes, and this afternoon we have been chasing likely affected customers who haven’t responded to encourage them strongly to work with us to fix this issue.

If you run Java-based services and you’re not already a customer of our managed hosting service, then you’ve probably been quite busy over the last few days. If you haven’t been, then you may want to consider signing up.

Dependency management

Log4Shell is a somewhat vicious lesson in dependency management. Every time you import third party code, you need a process for monitoring security advisories for it, and for updating it as required. This is why we have a strong preference for using operating system packages wherever practical, as this delegates the whole problem to the operating system maintainers and makes automatically finding and updating affected libraries trivial. Being able to automatically find vulnerable packages is critical, as you can be guaranteed that when a serious vulnerability is discovered, the bad guys will automate it.

IPv6 Deployment World Leader

December 8th, 2021 by

Yesterday (7th December) we attended the virtual IPv6 forum annual meeting. We were delighted that our director Pete Stevens has been added to the IPv6 Hall Of Fame as an IPv6 Deployment World Leader.

Unlike most awards we turn down, you can’t win this one just by paying for a hugely expensive table at an awards ceremony.

We also got an update on how IPv6 deployment is going through the UK. Happy to hear from BT that they’re making excellent progress replacing all the old HomeHubs with new IPv6-capable consumer routers. Sky Italia has deployed a consumer broadband network that’s effectively IPv6 only – IPv4 is provided as a service on top with MAP-T. As this is a form of carrier NAT they’ve managed one IPv4 address per 16 subscribers. This compares with their initial dual stack rollout we reported on from the 2019 council meeting.

Lastly it was noted that the cost of an IPv4 address on the open market is now around $60; increasing numbers of server providers are following our lead and making an IPv4 address an additional and removable option on the order form.

Improving the world bit by expensive bit

October 6th, 2021 by

We’re delighted to announce our sponsorship of Organic Maps. Organic Maps is a simple, user-friendly application that downloads complete Open Street Map data to your phone, allowing you to use their mapping application offline complete with route planning from the on-device database.

This is a wonderful application. It doesn’t track you, advertise at you or flood you with non-notifications, and it works without mobile data and conserves battery life. So if you’ve ever been lost without signal or somewhere where roaming data is prohibitively expensive, or to a very busy location where the mobile networks were overloaded this application is genuinely better than the alternatives.

While the app avoids the need for mobile data, this comes at the cost of a significant up-front download of all the mapping data that you may or may not use offline. This won’t trouble typical home broadband, but for the servers at the other end it adds up quickly. We’ve stepped in and offered two 4GB virtual servers with 400TB/month of free bandwidth to Organic Maps, split between our London and Amsterdam zones, reducing the reliance on a traditional and bankruptcy-inducing large cloud provider.

Quote from unspecified cloud provider of $24,452 per month

“Use the cloud, it’s cheap,” people often say, incorrectly.

Quote from unspecified cloud provider of $20,591 per month

A competing quote from a slightly cheaper large cloud provider

 

At our list prices this would be somewhat cheaper:

Qty Item Item price Price
2 VPS4 virtual servers (4TB/mo bandwidth) £32.14 £64.28
396 Additional bandwidth (per TB) £5 £1,980.00
Total £2,044.28

Being 90% better value is achieved in part by not having to fund our own space programme.

Domain Management API

October 1st, 2021 by

We’ve just rolled out a new addition to our range of APIs for managing services: the Domain Management API. This new API allows you automate management of your Mythic Beasts domain registrations.

Access to the API is controlled by API keys, which can be managed in our customer control panel. As for our DNS API, the keys provide fine-grained control over access, allow you to grant permissions on individual domains, or all domains on your account, and to restrict a key’s access to specific actions.

API Key Configuration screenshot

Fine-grained access control

The API gives access to information about your domains, such as the expiry date, nameservers, and domain status.

At present, the API only supports a small number of actions, although we intend to expand this in the near future. At present, the following actions are supported:

  • Setting nameservers
  • Setting DS records
  • Locking/unlocking domains (where supported)

The ability to set DS records makes it possible to automate DNSSEC key roll-over, although it’s worth noting that we offer a free managed DNSSEC service which takes care of this for you, so you’ll only need to use this if you particularly want to control your DS records yourself.

The API is currently in public beta, and documentation can be found on our support site. We’d very much welcome feedback on the API, including suggestions for operations that you’d like to see supported. If you have any feedback, please contact us on support@mythic-beasts.com.

Bullseye, new Debian release

August 20th, 2021 by

A small galaxy hit the bullseye of NGC922 about 330m years ago. More information: www.spacetelescope.org/images/heic1218a/
Credit:
NASA, ESA

Congratulations to the Debian team for their new release of Debian Bullseye (11). Just over two years of hard work have resulted in over 40,000 package updates and 10,000 additions.

We’ve made images for our VPS cloud that are available in all regions and included the install ISO for customers who prefer to build their own OS images. Sympl, a management package for web and email hosting that we maintain has been updated to support Bullseye with packages available for download.

Our mirror server is up to date with the Debian Bullseye packages. We’ll now be looking at deploying new systems on Debian Bullseye and starting our upgrade program for Debian Stretch and Buster systems.

The UK Debian folks will be having a small party in Cambridge in a few days time and we’re sponsoring the beer to say thank you. It’s a weekend full of beer and barbeques.

8GB and overclocked Raspberry Pi servers

June 15th, 2021 by
Pi 4 with PoE HAT

Our Pi 4 servers all wear the Power over Ethernet HAT to provide power and cooling to the CPU.

Since the launch of the 8GB Raspberry Pi 4 we’ve had many requests to add these to our Raspberry Pi cloud. Meanwhile many Raspberry Pi users have read about overclocking the Raspberry Pi and running at a higher clock speed.

Overclocking further increases the computing power of the Pi, but brings significant operational issues for our Pi cloud. Not all Raspberry Pi hardware will run reliably at the higher clockspeed and the higher voltage required to support it. Increasing the clockspeed and voltage significantly increases the power consumption and thus the cooling requirements necessary to prevent overheating. We’ve spent a considerable amount of time testing and we’re now ready to launch our first 8GB Raspberry Pi 4 cluster. We’re offering them at two clock speeds: the stock 1.5GHz and overclocked to 2GHz.

The overclocked Raspberry Pis have all been run at a significant CPU load for several weeks to test their stability before release. Any that failed the stability test have been added to the cloud at the normal 1.5GHz clockspeed.

The 8GB Pi is available at 1.5GHz and 2GHz clock speeds. Supported operating systems are Raspberry Pi OS 64 and Ubuntu 64.

Larger fans provide more cooling to our 8GB Pi4 cloud so we can run at higher clockspeeds.

VPS API, on-demand billing and dormant VPSs

May 14th, 2021 by

Dormant mode means your VPS can have a nice snooze.

We’ve recently rolled out some new features that provide more flexibility to our VPS platform.

On-demand billing

Last year we added on-demand billing to our Raspberry Pi Cloud and we’ve now rolled this out to our VPS services, allowing you to add and remove VPSs at any time and pay by the second for the time that the server is provisioned. We continue to offer monthly, quarterly and annual billing options, with discounts for longer billing periods, allowing users to choose between the best pricing for long term usage and the convenience of on-demand, pay-as-you go pricing.

Dormant VPS mode

We’ve also added the ability to make a VPS dormant, so that you’re only charged for the server’s storage space (and any allocated IPv4 addresses) until you want to reactivate it. Dormant VPSs can be reactivated at any time, although it is not guaranteed that you will be able to re-provision to the same specification of server immediately. The RAM and CPU previously allocated to your server may have been reallocated, and a move to a different host server may be required.

VPS management API

We have also added an API for managing on-demand VPSs, allowing the creation and deletion of servers to be automated. The API is very similar to our API for managing Raspberry Pi Cloud servers. To get started, see our API docs.

Cloud-init user data

We use cloud-init to automate operating system installation when provisioning a new VPS. The installation can be customised using cloud-init user data, which can provide additional installation steps to be performed after the first boot. User data can be provided through both the control panel and the API. It also possible to store and re-use user data snippets in the control panel, making it easy to repeatably spin up new servers with your applications already installed and configured.

More capacity

We continue to add capacity to our cloud to keep up with customer demand with the most recent expansion being in our London Meridian Gate (MER) zone.

Private cloud improvements

Our Private Cloud service gets you the features and convenience of our public VPS platform, but provided on your own dedicated servers. We’ve recently rolled out improvements to our Private Cloud platform, allowing Private Cloud servers to be provisioned and managed via the API and control panel.

Teaching our network some MANRS

April 30th, 2021 by


We’ve recently rolled out software upgrades to our networks that enable improved routing security and we have joined the MANRS (Mutually Agreed Norms for Routing Security) initiative.

Our MANRS certification for our EU and US networks confirms that we block spoofed traffic, drop incorrect routing information and work with others to maintain routing security.

This is beneficial for any customer using our transit and IP services, which includes all dedicated server and virtual server customers.

Resource Public Key Infrastructure (RPKI)

Amazingly, up until the advent of RPKI the entire internet worked on a trust relationship. When another network told us that they were responsible for a range of internet addresses we’d believe them. Border Gateway Protocol (BGP) is how networks communicate routing data to each other and it had no mechanism to confirm that the route and address space being advertised to you were genuine.

Incorrect advertisements result in network traffic being delivered to the wrong destination and incidents, both deliberate and accidental, are common and can cause real harm. For example, $17m in crypto currency was stolen in 2018 via an IP address hijack aimed at Amazon. Youtube has been taken offline as have large parts of the Cloudflare network.

RPKI seeks to address this by providing signed proof that a network operator (identified by their Autonomous System Number) is permitted to originate a specific range of IP addresses. Once a range of IP addresses is signed you know that any announcement of the address space from any other provider is invalid and should be dropped.

Our transit providers are also certified by MANRS for further protection.

An RPKI example

RIPE Labs have created a deliberately invalid routing announcement that can be used to demonstrate and test RPKI. RIPE Labs have published a Resource Origination Authorisation (ROA) that says only AS0 is permitted to announce the prefix 209.24.0.0/24. They then announce that prefix under AS15562.

With RPKI we see that the network listed in the ROA does not match the network announcing the route, so that route is considered invalid and rejected as being a hijack.

Ripe Labs have published a checker that runs in your browser and detects whether you can see this invalid route on your ISP’s network.

From our network, we now get the big smiley face:

Internet Resource Registry (IRR)

RPKI complements another approach to routing security: filtering based on Internet Resource Registry (IRR) data. RPKI allows us to verify if a network is a valid ultimate destination for a particular IP range. Most networks we don’t see directly, we go through another transit providing network. IRR allows us to verify that the network advertising a given route is authorised to originate or transit that route.

The Regional Internet Registries (RIR) allow network providers to register a link between their network and an IP block. Various tools exist (e.g. bgpq3) to create a list of all the internet addresses that a network can originate or transit from their downstream customers. This is be used to generate a filter list that restricts what routes we will accept from peers and downstream customers.

These lists can be very long and change frequently – the list for our network (AS-MYTHIC) is usually 5000 or so records with tens to hundreds of changes per day.

Best Common Practice 38 (BCP 38)

Another issue with insecure routing is “spoofing” — sending IP packets with a fake source address. This is widely used by attackers to cause denial of service attacks. An attacker sends packets with a sender IP address faked to be that of the target machine. The recipient of these packets will send replies to the target machine instead of the originator. This makes it very easy to create distributed denial of service attacks.

BCP38 is a Best Common Practice which requires that networks filter packets that aren’t either to or from an address within their network.

Part of MANRS is not only to implement BCP 38 but also to host an active spoofer. This means if we drop our BCP38 filtering our non-compliance will be published including regular mailings to network operator groups.

Having good MANRS

By combining all these methods routing security is significantly improved. RPKI provides dynamic checking that doesn’t rely on us adding static route lists to our routers. This also provides excellent protection against accidental hijacks from a “route optimiser” gone wrong. IRR forces accurate routing data to generate filters. BCP38 reduces risks to other networks from spoofed packets. Combining all of these means we have much better MANRs at the price of terrible acronyms.

RPKI filtering is now fully deployed on our US and European network and they both now pass Cloudflare’s “Is BGP Safe Yet” test.

Restoring Nominet’s Purpose: update

February 22nd, 2021 by

Earlier this month we reported that we’d signed up to the Public Benefit campaign to reform Nominet, the company responsible for overseeing UK domain registrations.

The campaign was seeking 5% of Nominet’s membership in order to call an EGM to replace Nominet’s non-elected directors. The campaign quickly achieved this, the EGM request was delivered, and Nominet have now set the date for the EGM as 22nd March 2021. Members representing more than 17% of Nominet voting rights have now signed up to support the campaign. Typical AGM voting turnout is well under 10% suggesting that the vote is pretty much certain to succeed, at least according to The Register’s analysis.

If there was ever any doubt about the need for reform, Nominet’s response to the EGM letter has completely removed this.

Nominet’s CEO rushed out a statement hoping that:

all constituencies will be able to engage in a constructive way

At the same time, Nominet responded to Public Benefit’s email requesting member information by providing 575 printed pages:


This would seem to be more obstructive than constructive.

The EGM request made two motions: (1) sack the current directors; and (2) appoint two interim directors to take over. Nominet are claiming that the second motion is illegal (contrary to legal advice received by Public Benefit) and are refusing to put it on the EGM agenda. They now have the gall to claim that the EGM request destabilises Nominet because it does not provide a credible plan to replace the current leadership.

Is this just about reducing UK domain fees?

It’s been suggested that this campaign is about Nominet members, who are mostly companies like us that resell domain registrations, trying to reduce the price that they pay for domains. This seems to ignore the fact that the domain market is very competitive, and UK domains are particularly easy to transfer between registrars. Provided that the price is the same for all members, what that price is doesn’t make much difference to us.

Nonetheless, we’re very happy to make a public commitment that if the EGM process results in a reduction in the price that we pay for domains, we will pass on that saving in the price that we charge.

Testimonials

February 5th, 2021 by

We’ve had a variety of customer being very complimentary recently. Andy Steven runs a series of web cams in the Shetland Islands that stream live views of the northern lights. The cameras relay the stream via one of our virtual servers in our MER data centre and the current bandwidth record is several Gbps.

I am proud to say that our new ‘AuroraCam’ network just delivers and for the first time I no longer break out in a sweat watching the demand increase from that AuroraWatchUK alert or a celebrity weather personality sending out a Tweet.

— Andy Steven, Shetland Webcams (full article)

Beautiful shot of the northern lights captured by Shetland Webcams. Could be improved by adding a kitten though.

We provide 10Gbps fibre connectivity to the Cambridge office of DarkTrace. Darktrace uses machine learning to identify and neutralise security threats in real time.

You’ve been much more transparent & approachable than any provider I’ve dealt with previously. Very happy with the service so far.

— Harry Godwin, Head of Business Infrastructure. Darktrace

The Web hosting review and advice site Hosting Advice interviewed us and wrote a great article about the management and infrastructure services we provide.

Recognizing that there is no one-size-fits-all approach to managed hosting, Mythic Beasts can take on varying responsibility levels as needed. This range of services includes everything from ensuring that servers are up and running to providing the extensive monitoring, security, and assistance necessary to keep custom web applications functioning reliably.

— Hosting Advice (full article)

Lastly our strong stance about returning Nominet to its public benefit roots garnered entirely positive responses at Twitter.