PHPocalypse

It’s been a long week.

Dealing with urgent security updates is completely routine for us. We’re constantly monitoring and reviewing security announcements, and every week we’ll push out multiple fixes for packages that we consider to be “0-day worthy” – that is, a fix for a vulnerability that is serious enough that it needs patching immediately and so gets puts through our “0-day” process rather than waiting for the next scheduled upgrade window.

We have tooling that allows us to quickly find all managed servers where a given package is installed, send out notifications and automatically apply the updates.

What made this PHP issue so bad?

The main answer is ubiquity. PHP is installed on a large proportion of our managed servers, including our shared hosting servers where we offer multiple different versions of PHP.

Simply pushing out the updates as soon as they were available would take some time, and we were concerned that this would provide enough of a window to start seeing exploit attempts.

We therefore took the decision to take PHP-based sites on our shared hosting servers offline from the time of the announcement (3pm UK time on Wednesday) until the patches were applied. We took this decision in our incident planning meeting on Wednesday morning, but were pleased to see the NCSC including this approach in their list of recommendations.

Although many sites experienced a short, precautionary outage, the vast majority of servers under our control were fully patched and back in full service in well under an hour from the initial announcement.

The long tail

We try hard to keep servers on a current operating system version, and with a very strong preference for only installing software from operating system packages. This incident demonstrates why we do this, as fixing vulnerabilities becomes a straightforward task of upgrading an operating system package.

PHP is a bit of a special case, as a lot of PHP software targets PHP versions that are newer than those available in stable operating systems. There’s also software that’s well behind the curve and needs older PHP versions.

Fortunately, on Debian and Ubuntu at least, this is largely solved by the tireless efforts of Ondřej Surý to provide supported packages for multiple versions of PHP. The fix for this vulnerability has been back-ported as far as PHP 5.6, which formally went end-of-life in 2018!

Unfortunately, the world isn’t perfect, and we have a decent number of servers under our management that, for a variety of reasons, were more complicated than a simple package upgrade, and working through this long tail of systems has kept us very busy.

Some updates were also delayed because the Ubuntu infrastructure that the Sury Ubuntu PHP packages rely on couldn’t cope with the load, and we were up late ensuring that these packages were rolled out to all affected servers.

Extra support

The vulnerability hitting the front page of BBC News resulted in quite a few customers emailing us to ask if they were affected. The BBC were a bit late to the game, so we were happy to be able to respond to most of them confirming that it had been fixed the day before! We were less impressed by an obviously LLM-generated support query with a long list of questions to answer, which received the terse response that it deserved.

We’ve also been receiving lots of enquiries from customers of other hosting providers who experienced extended outages due to the issue.

What’s the lesson here?

Perhaps unsurprisingly, the mainstream news coverage mostly missed the real issue. The story made the news because high-profile organisations, including government organisations, took their websites offline, in some cases for an extended period of time. Predictably enough, news stories focused on the (overstated) role of LLMs in discovering the vulnerability, and not on the question of why these organisations didn’t have a better process in place for dealing with an entirely foreseeable event.

We have seen a significant increase in the number of serious vulnerabilities being reported, and customers with managed servers will have noticed an increase in the frequency of emails notifying of urgent updates to packages installed on their servers.

If you don’t already have a process for handling critical vulnerabilities in the packages that your service depends on, you may wish to talk to us about server management.

Ready for the next one?

Although it’s been a very long week for many of our staff there are definitely some positives coming out of this.

It’s often hard for us to persuade customers of the importance of having security support for their systems. It’s very easy to put off upgrading and then find out the hard way that you can’t respond quickly to a security vulnerability. The high profile of this incident, coupled with the NCSC’s “if you can’t patch it, switch it off” advice is actually very helpful.

And naturally, we’re reviewing what we can do to further streamline our processes ready for the next one.