Web hosting and Cloudflare

Not this kind of cloud or flare.

After careful consideration, we have reluctantly taken the decision that the use of Cloudflare will not be supported on our Web and Email Hosting service. First two clarifications:

• This only applies to our Web and Email Hosting services. It does not, and will not, apply to virtual servers (VPS), dedicated servers or Raspberry Pi servers.
• We have never formally supported or encouraged the use of Cloudflare with our Web and Email service. Until now, we have generally discouraged it, but we’ve tried to accommodate users who chose to use it.

The problem

As we’ve written about previously, we’ve seen a huge increase in the amount of abusive bot traffic hitting our web servers — much of it badly behaved AI scrapers — and frequently the volume of traffic is such that it overwhelms the server, and makes websites unavailable. At times we’ve seen over 95% of traffic coming from AI scrapers. Our Web and Email Hosting service is what’s often referred to as “shared hosting” meaning that we have websites for many customers on a single server. This is a very cost effective way to provide web hosting, but it does mean that any load issues caused by traffic to one website can affect other sites on the same server.

Our primary tool for dealing with abusive traffic is to identify the source of traffic and block the IPs that it’s coming from.

Cloudflare provides a service that seeks to protect websites by blocking abusive traffic. It does this by operating a “reverse proxy”; rather than web traffic arriving directly at the web server, it is instead sent to Cloudflare’s servers. Cloudflare inspects the traffic, filters out the abusive traffic, and then forwards the legitimate traffic to the actual web server. This has the effect that all traffic arriving at the web server appears to come from Cloudflare’s IP addresses, rather than the actual client IPs.

Some of our customers have chosen to front websites hosted on our shared hosting servers with Cloudflare. The problem is that Cloudflare isn’t perfect; it doesn’t succeed in filtering out all abusive traffic. This is particularly true of the free tier that we tend to see used in conjunction with our Web and Email Hosting service.

Unfortunately, when we see a large volume of abusive traffic arriving via Cloudflare, we are faced with a choice: either we block Cloudflare’s IP addresses, knowing that this will take all websites using Cloudflare completely offline, or we accept the traffic, which potentially has an impact on all websites hosted on that server. With the growth in volume of abusive traffic, we are being forced to make this choice increasingly often.

We have now taken the decision that we will treat Cloudflare’s IPs like any other IPs: if we see abusive traffic from them, we will block them. In the future, we may introduce a permanent block, or redirect traffic to a support page on our site that explains why Cloudflare is not supported on the service in order to avoid customers inadvertently using an unsupported configuration, but we will contact customers who appear to be using Cloudflare prior to taking this step.

FAQs

Obviously these questions are not frequently asked as this is the first announcement of this change, but whatever; here are some questions and answers:

What have you got against Cloudflare?

None of this is a criticism of the service that Cloudflare provides. The same would apply to any reverse proxy service placed in front of our servers that prevents us from seeing the actual source IP, thereby removing our ability to effectively filter traffic ourselves.

We do have concerns about any service that breaks the end-to-end encryption of web traffic, but that’s unrelated to this issue. Cloudflare (and any other such service) relies on terminating the TLS connection from the client on their servers, inspecting the traffic, and then making a new secure connection to the target web server.

Why do you think you’re better at this than Cloudflare?

We’re not, but when a particular attack is affecting our ability to provide our web hosting service to hundreds of customers, we’ve got a much stronger incentive to resolve it quickly.

Cloudflare definitely has some very impressive technology, but we suspect that much of it isn’t provided in the free tier of their service. Upgrading to a paid-for tier, or careful configuration of the free tier, might yield better filtering, but in our case, we’re affected by the most permissive configuration; it only takes one customer to point Cloudflare at our server with minimal filtering and we’ve got a mix of legitimate and abusive traffic arriving from the same IPs and we’re back where we started.

Cloudflare includes the client IP address in an HTTP header – why don’t you filter on that?

Filtering based on the source IP address of a TCP/IP connection can be done very efficiently because you only need to look at the packet header. Filtering based on an HTTP header requires accepting the connection, setting up a TLS connection and decrypting the content, and then parsing the headers, which is a significant overhead, and in extreme cases, the load from doing this is prohibitive.

Can I use Cloudflare on my VPS or dedicated server?

Yes, absolutely. The issues described here that make Cloudflare such a problem for our Web and Email Hosting service don’t apply to VPSs and dedicated servers, so if you want to put Cloudflare — or any other reverse proxy — in front of your server, you are free to do so. Of course, dealing with any abusive traffic that slips through is your responsibility.

Can I use Cloudflare on my managed server?

If you have a managed server with us then we will attempt to resolve any load issues caused by abusive traffic as part of this service. Please do not install Cloudflare (or similar) in front of your managed server without discussing it with us first, as it will hamper our ability to respond to any incidents. A special case is our WordPress Shared tier of managed WordPress hosting. As the name suggests, this is implemented on our shared hosting platform, and so we cannot support the use of Cloudflare with this service.