I would like to introduce our all new female GHOSTbusting team to tenuously tie in with a new Hollywood movie and gratuitously include a cool staff photo in this blog post, and for marketing reasons I’m going to ignore the reality that Toby did all the updates for GHOST.

Qualys found during a code audit a buffer overflow exploit for gethostbyname() in glibc which they’ve named GHOST. This means that any internet facing software that can be persuaded to do a DNS lookup is potentially vulnerable. To a first approximation that’s everything that’s listening on an internet socket.

The details are in CVE-2015-0235. Note this explains quite comprehensively how to exploit the vulnerability so we are expecting active exploitation to have already started.

The vulnerability was announced at 16:30 on Tuesday, at 16:40 the first ticket was opened in our queue automatically. We started reviewing the information shortly thereafter and deployed the updated packages to our shared hosting servers Tuesday evening. This gives a short window to discover any critical issues with the new packages before we start deploying updates to our managed hosting customers.

At 8:30am on Wednesday, we emailed every managed customer running vulnerable code (which is almost but not quite all of them) explaining the issue and indicating we’d be applying the patches immediately unless otherwise instructed not to. Giving customers a short window to reply before going ahead (some are automatically deploying via Puppet and don’t want us to update for them) we then applied the updates to the customer servers, which involved very brief interruptions to listening services as they restarted.

Subsequently spot auditing some customer machines indicates that the glibc update via the package manager may not have restarted every vulnerable process. We’re now writing some audit tools to check for missing service restarts. Tomorrow morning at 6am, our reporting package will report in lots of data about the status of all our managed customer machines including the complete process list and complete list of listening services, so on our reporting box we can do a complete audit for every listening process that hasn’t been restarted in the last 24 hours and investigate and fix where necessary.

If you aren’t a managed hosting customer of Mythic Beasts we implore you to update your systems as soon as possible, we strongly expect that someone is going to build a very big denial of service botnet very quickly from this vulnerability. If you have no idea how to update and audit your server please get in contact with us at support @ mythic-beasts.com even if you’re not hosted with Mythic Beasts.

8:30 : Wake up and get out of bed. Open the curtains to see the sun shining, put a dressing gown on and go downstairs to make some coffee.

8:40 : Take coffee to the home office and open up the laptop to start some work.

8:41 : Laptop does not ask for a password to decrypt the encrypted filesystem and refuses to work.

8:42 : Sip coffee and wait for desktop to boot.

8:43 : Log into desktop machine.
this wouldn’t actually work either, but we’re going to lie for narrative structure

8:45 : Open up web browser, default homepage is our support queue which displays message ‘I’m afraid this uses illegal encryption technology and you are not allowed to access this page’.

8:50 : Drink some more coffee.

8:55 : Realise there’s a copy of the customer support tickets in email, turn on email client.

8:56 : Wonder why email client gives strange connection errors that the mail server is refusing to allow it to connect with SSL turned on.

9:00 : Give up on email entirely, hurrah!

9:01 : Look at empty coffee cup, go downstairs to the kitchen to refill the coffee cup.

9:10 : Log on to company chat-room which fails to work with a connection error.

9:15 : Think this is all a bit bizarre so phone colleague on mobile, she answers to say that she’s having lots of problems too.

Spilled Coffee by Kenny Smith

9:20 : Conclude that the winning plan is clearly to spend the day updating some documentation while drinking coffee.

9:25 : Company wiki fails to load. Secure connection error.

9:30 : Decide to check the mrtg monitoring graphs to see if the network is working. Connection fails.

9:35 : Probably best to start fixing the mrtg monitoring server, first step, log into our bastion host which manages the access controls for servers on our network. Connection fails.

9:40 : This is getting really weird, probably best to go off and feed the cat who’s been miaowing for the last fifteen minutes demanding breakfast.

9:45 : Examine coffee carefully to check it’s not been tampered with and had hallucinogenic drugs added. Realise that if hallucinating could be hallicinating that no drugs were added when they were and how would you tell anyway. Conclude this is about to turn into a long, complex and ultimately nugatory philosophy problem.

10:00 : Return to desk, decide that the best plan is to audit our assets database and resolve some discrepancies between reality and the database by visiting the data centre.

10:05 : Unable to book visit to data centre, the data centre portal doesn’t work, connection errors.

10:10 : Unable to load the assets database, secure connection error.

10:11 : Unable to book car, Zipcar is down.

10:12 : Unable to look at map, Google Maps is down.

10:15 : Decide that the winning plan is to just give up, drink coffee and watch cat videos on youtube. Youtube fails to load with a secure connection error.

10:17 : Skim the news which has some article about a new government and some encryption technology. Click on a link in the forum which surprisingly fails to rick-roll.

10:20 : Now really very annoyed, going to have to waste time on facebook. Facebook refuses to load with a secure connection error.

10:30 : Phone company conference number for conference call to organise the day. Connection error.

10:35 : Really running out of ideas now of what to do. Go for a walk outside to a coffee shop. Mildly surprised that the sunshine is still working.

10:55 : Arrive in coffee shop to be greeted as Arthur Dent. Realise still wearing dressing gown, and for forms sake must now try and order a cup of tea.

11:00 : Order tea, coffee shop tells us that the credit card payment machine isn’t working and we’ll have to pay in cash. Observe that our wallet is empty. Leave coffee shop to go to cash machine.

11:10 : Cash machine is out of order.

11:30 : Return home and get dressed. Then collect cheque book, return to coffee shop. Persuade them that they can accept a cheque and order tea.

12:30 : Reflect that todays achievements so far consist of buying a cup of brown liquid that was almost but not quite entirely unlike tea. Go back home to face the afternoon.

13:00 : Decide that this is pointless and book tomorrow off. Holiday booking system doesn’t work, connection error.

13:10 : Decide this is lunacy and want to resign. Go to Linked In to update CV and find new job. Connection error.

13:20 : New job will probably be as crap as this one. Just resign. Fire up word processor, write resignation letter and email to boss.

13:30 : Email doesn’t work. Print it out.

13:40 : Printing doesn’t print either. Give up and copy it off the screen with a pen onto a piece of paper ready to post to boss. Realise there’s no stamps and with no cash it’s going to be hard to buy one.

13:45 : That’s it! Game over man! Game over! What the **** are we gonna do now? What are we gonna do? Maybe we could build a fire and sing a couple of songs? Why don’t we try that?

13:50 : Stop panicking and hit upon a cunning plan, steal all the money from the company and flee to a more sensible country than this one.

14:00 : Try to book a flight to Athens. Shopping cart fails with a connection error.

14:10 : Try to go to the bank website to withdraw all the money. Fails with a connection error.

14:20 : Visit the bank in person to steal all the money. Bank has a massive queue of people complaining because they can’t withdraw their money, apparently there’s ‘computer problems’.

14:30 : Give up on humanity entirely and go and find a park bench on which to live, in the vague hope that someone has a gold brick with which to wrap around a slice of lemon for brain smashing purposes.

The new Mythic Beasts Offices (public domain)

Those of you who follow us on Twitter are probably bored of us banging on about this, but the true lunacy of the EU VAT MESS has only just come to light. It turns out that the UK and other states are going to compensate the tax haven at the centre of this to the tune of €1.1bn for loss of tax revenue as a result of the rule change.

Let’s re-cap:

1. Large companies such as Amazon indulge in VAT tourism, by paying very low Luxembourg VAT when supplying to customers in the UK and other countries.

2. The EU declares this to be unfair tax avoidance, and decides to close the “loop hole” by making electronic services subject to VAT in the customer’s country rather than the seller’s.

3. Faced with the prospect of thousands of companies having to register for and operate VAT in 28 member states, HMRC sets up their VAT “one stop shop”, the VAT MOSS. This avoids filing separate returns to different states, but still requires sellers to track the multitude of different VAT rates in operation by different states, including obscure regional variations such as the Portuguese Azores.

4. The legislation does not include any thresholds for inter-state VAT, meaning if you sell a single item to a consumer in another EU state you must register for the VAT MOSS and charge EU VAT.

5. You can’t register for the VAT MOSS unless you are registered for UK VAT, meaning that if you make a single sale to another EU state, you’re obliged to start operating UK VAT on all your sales even if you’re well below the UK VAT threshold of £81k.

6. The guidance requires companies to collect an often impossible set of non-conflicting data to prove the consumer’s location, and then retain those records for 10 years.

7. HMRC’s original impact assessment recognised but dismissed this problem by vastly underestimating the number of businesses affected, and claiming that most small companies sell through online market places, giving as much as 70% of their revenue to companies such as… Amazon.

8. HMRC back-tracks on (5) by stating that you can avoid charging VAT on your UK sales by splitting your sales into two separate businesses – a technique known as revenue splitting which is normally considered illegal tax evasion.

9. Many companies (ourselves included) realise that the cost of compliance is greater than the affected revenue, and consider simply not supplying to consumers in other EU states, but are warned that this may be illegal under EU anti-discrimination laws.

10. Companies start to indulge in farcical discussions with HMRC about what constitutes an e-service. In some cases, by making the business less efficient, for example by manually attaching a PDF to an email rather sending it automatically, the service will no longer be considered an e-service.

11. Despite acknowledging that the change would impact businesses that are not currently registered for UK VAT, HMRC apparently did nothing to communicate the change to the rules to anyone other than VAT registered companies. Vince Cable then has the gall to tell companies just finding out about the change that he has done a lot to communicate the change.

12. Recognising that it’s about to lose a huge wedge of tax revenue, Luxembourg ups its VAT rate to 17%, a move which would probably have significantly curtailed VAT tourism on its own.

13. The UK and other member states agree to compensate Luxembourg to the tune of €1.1bn for the VAT revenue that they will lose as a result of companies ceasing to use Luxembourg as their tax avoidance state of choice.

You really couldn’t make it up.

As it stands, many micro-businesses are planning to simply shut up shop rather be killed by VAT bureaucracy.

For the record, we were already VAT registered, and did find out about the changes early this year, but the work of updating our billing system to cope with a plethora of different VAT rates, and the necessary “proof of residence” steps has been a massive and expensive distraction from doing useful things like upgrading our Virtual Servers.

We were planning to announce upgrades to our Virtual Servers today, but unfortunately we’ve had to spend time dealing with the #VATMESS

One of these coins is worth something. The other carries an obligation for a decade of document storage, 80 tax returns and tax rates for nearly 30 different countries. Guess which one we prefer?

At the moment, VAT on “e-services” sold within the EU is paid based on where the supplier is, so if you’re a small UK company selling, say, hosting services, you pay UK VAT to HMRC, irrespective of where the customer is.

If you’re a large company selling lots of such services then you’ll be paying enough VAT that it’s worth your while to move your operations to the member state with the lowest VAT rate, which is Luxembourg.

Of course, big companies avoiding tax is Evil, Bad and Wrong, so the EU has taken action.

The very short summary is, if you’re a non-VAT-registered customer in an EU state other than the UK, then we’re going to have to start charging you VAT at your local rate, rather than the UK rate. Good news if you’re in Luxembourg, bad news if you’re in Hungary.

The rather longer rant summary is that we’ve been forced to waste a significant amount of time understanding and complying with new regulations for VAT on electronic services which come into force on 1st January 2015.

Whilst cutting down on large companies undertaking VAT rate tourism might seem like a nice idea, charging VAT based on where the customer of an online service is creates a whole bunch of new problems:

1. How do we establish where a customer is based?

The guidance tells us that we need two non-contradictory pieces of evidence to establish the customer’s location, the most readily available being the billing address and the customer’s IP address. Setting aside the unreliability of geolocating IP addresses, what happens when a customer is enjoying their right to roam the EU freely and places an order whilst in another country?

Well, the guidance tells us we can use:

• location of the bank (we don’t collect this information)
• the country code of SIM card used by the customer (not applicable)
• the location of the customer’s fixed land line through which the service is supplied to him (not applicable)
• other commercially relevant information (suggestions on a postcard)

In the event that we succeed in obtaining the necessary evidence, we’re legally required to hang onto it for 10 years.

2. How do we find the correct VAT rate for a state?

Presumably, recognising that a huge proportion of companies in the EU now need to regularly lookup current VAT rates for different states, the EU will have created a convenient web service providing this information in a computer-readable format?

Well, the guidance sends you to this site which allows you to select “all states” and has an “Export selection” button. Looks promising until you try it and discover that it buries the data in a generated PDF.

Fortunately, some helpful soul has created what we actually want: a simple JSON feed.

Unfortunately, that site makes the amateur mistake of thinking that ISO 2 digit country codes will be enough to cope with all the VAT rates in the EU, forgetting that the Portuguese Azores and Portuguese Madeira have their own VAT rates, but not their own country codes. As it happens, the EU site listed above also denies knowledge of the VAT rates applicable in these regions.

3. How do we report and pay our VAT?

HMRC are proud to tell us that they’re saving us the burden of registering for VAT in each member state in which we do business by letting us use MOSS, their “One Stop Shop”, but we still now have to complete two separate quarterly VAT returns and, of course, the quarters don’t even align.

Bulk upload of our VAT data is supported using that well known open data-interchange standard: a spreadsheet. A particular highlight is that: “When completing HMRC’s spreadsheet you can’t use country codes (for example GB, UK, NL or DE) or country descriptions (for example Great Britain, the UK or The Netherlands). You must only use the following EU country names:”. That’s right, HMRC have eschewed ISO country codes for its preferred list of country names and spellings, and not even for the obvious reason that some states have multiple rates: Portugal is only listed once.

Tax doesn’t have to be taxing … but it is

The net result of these new rules is that it’s now much harder for us to sell to consumers in other EU states than it is for us to sell to consumers outside of the EU – surely the exact opposite of what the single market is supposed to achieve?

The amount of our business that is affected by these new rules is tiny, as most of the EU business we do have is to VAT-registered entities to whom an entirely different set of rules apply. The amount of profit we make in a year from the affected services is almost certainly less than the upfront compliance cost, if not the ongoing cost, so we have seriously considered simply refusing to sell to consumers in other EU states, although it has been suggested that this could be illegal under EU law!

It could be worse

These VAT changes are a nuisance for us, but we’re already well above the UK VAT threshold so already have processes in place to deal with the burden of UK VAT reporting. For very small companies, as we were not so long ago, these changes are absolutely horrific as there is no VAT threshold for inter-state VAT. The government accepts that requiring all businesses to operate UK VAT would be an unreasonable and stifling burden on small businesses, which is why we have a VAT threshold (currently £81k). But there is no such threshold for inter-state VAT, despite it being significantly more complicated to administer.

There is a growing storm of angry micro-businesses who, through virtue of not being VAT-registered, weren’t notified of the upcoming changes. Indeed, it seems that HMRC’s assessment of the impact of these changes not only vastly underestimated the cost of implementing them, but also completely forgot about several hundred thousand micro businesses that would get shafted by these changes.

(HMRC’s original impact assessment stated that “businesses currently unregistered in the UK who choose to register for MOSS in the UK will also have to obtain a UK VAT registration and their UK supplies will therefore also become liable to VAT”, meaning that if you sold a single e-service to an EU consumer you were pretty much obliged to start operating UK VAT too. HMRC have back-tracked on this by publicly endorsing the practice of splitting EU from UK revenue – despite revenue splitting normally being considered an illegal VAT-evasion practice)

Hip Hop is not only a style of music, but also the name of a virtual machine written by Facebook which compiles PHP Just In Time to make it go quickly.

Now we receive lots of unsolicited advice about how to run a not very popular wordpress blog and cope with the volume of traffic. Usually this involves ripping and replacing the entire infrastructure from a standard Linux/Apache/MySQL/PHP stack to something different (Nginx/MariaDB/PostgreSQL) which may not even be able to run WordPress at all (e.g. node.js).

At Mythic Beasts we like to understand what we’re doing, rather than blindly installing Magic Go Faster Solution Number 7. So we set up a test 2GB dual core virtual machine, that runs WordPress and a selection of popular plugins ( WordPress SEO, Akismet, Safe Report Comments, Liveblog, Facebook, Yet Another Related Posts Plugin, WordPress Supercache and Jetpack, no endorsement implied). Then we benchmarked with siege and managed the following results.

Apache/mod_php : 5.10 trans/sec

and when you turn supercache on and serve cached pages you get

Apache/mod_php/supercache : 873.50 trans/sec

So this gives us two scenarios, pages which we have to generate content for which can easily cause load issues, and pages served from supercache in which our VM is fast enough for all practical purposes and will easily weather even very big traffic spikes from news websites or television adverts.

Now, it’s very popular to tell us to use Ngnix as it’s faster than Apache. Is it though?

Nginx/php-fpm: 5.70 trans/sec
Nginx/php-fpm/supercache: 2230.58 trans/sec

Wow! Nginx is three times quicker than Apache at serving cached pages. This is amazing, but not very helpful. It means when our webserver is serving pages really quickly, we serve pages at three times really quickly, but when we’re generating pages on demand, it’s about 10% quicker. That’s not very special and doesn’t justify a rip and replace of the whole installation for a 10% performance improvement.

A quick look at the VM during the testing tells us that the bottleneck is executing the PHP code which creates WordPress pages. The choice of webserver is basically irrelevant; almost all the server time is spent executing PHP and reading data from the database.

Enter HipHop Virtual Machine.

This is nothing to do with the HipHop Virtual Machine. But we like tea and Banging Tunes

It has one focus, to execute PHP quickly for Facebook. Facebook have a lot of servers and spend hundreds of millions to billions per year on servers and data centres. A 50% performance improvement in PHP saves them huge sums of money in data centres and servers alone, so it’s clearly worth them trying to optimise as much as possible.

Here’s what happens with Apache/Nginx running HHVM.

Apache/HHVM :           35.93 trans/sec
Apache/HHVM/supercache: 928.70 trans/sec
Nginx/HHVM :            33.78 trans/sec
Nginx/HHVM/supercache : 2137.67 trans/sec

This is a huge improvement for non cached pages – seven times faster. Cached pages are bottlenecked in the webserver so it makes minimal difference, but they were already so fast we weren’t worried about them. Again Apache/Nginx are still pretty much the same speed for generated pages, we’re still dominated by the code execution time but a seven fold performance improvement is worth seriously considering.

Whilst we can reconfigure servers standing on our heads, we usually don’t.
Photo credit: Mark Dolby, Flickr, CC-BY.

All I need to do now is see if I can find someone with a very busy WordPress site and a million complaining users who would like to test it to see if it’s really as good as the lab tests suggest it might be.

Very sorry to hear the news that Big Bank Hank who co-wrote the first ever hit Rap track Rappers Delight died earlier this week from kidney complications related to cancer.

You see, he was six foot one, and he was tons of fun

At Mythic Beasts we try very hard to keep our customers happy, and to do our absolute best to meet their requirements in requests, even if they’re occasionally a little bit unusual.

One of our long standing customers is refreshing some of their hardware, and we had the following exchange to sort out the details

customer> The following 8 servers have been decommissioned and now need removing: 

mythic-beasts> We can sort that for you. Do you want to collect the servers or shall we recycle them for you?

customer> The drives can be kept for spares but you can ditch the servers or make a fort out of them or something..

a 1U server fort

Now it’s not really our field of expertise, but we think we’ve got a reasonable start on building a defensible concentric castle although we ran out of servers before we could start building the outer curtain wall.

Managed server customers receive as standard 24/7 monitoring of their servers, we check that the machines are up, that ssh is running, that the web-server is delivering the correct content amongst other checks. In the event a check fails our staff are alerted via SMS/pager to investigate the issue.

We’ve now enhanced this service for managed server customers, in the unlikely event you have a service affecting issue that the automated monitoring hasn’t caught, you can file an urgent ticket through our control panel which will create a new support ticket and alert our staff via SMS to deal with your issue.

This was a feature request from a customer in a meeting last Thursday and went into production as a service enhancement on Tuesday, we’re always receptive to suggestions from customers to make our offering better.

bzip2 is one of the great unix tools. It compresses and uncompresses data, and it does it very well. We’ve been using it within Mythic Beasts for years and it’s operated absolutely flawlessly.

We’re happy to report that we’re now hosting the main distribution site for bzip2

Thanks to Jonathan Wright who runs a very big website, for a nomination.

I’ve nominated Matt Smith, Rob McQueen and Neil McGovern.

Thanks to Ben Howe, our gap year student who’s adequately demonstrated to his colleagues the definition of a career limiting move by dunking a bucked of ice over his boss, The Haymakers for kindly providing the location for the company meeting, the chilled water and the ice, and the rest of my Mythic Beasts colleagues for filming and laughing.

Mythic Beasts have added paypal functionality to our billing system. You can now pay by credit or debit card, paypal, direct debit, BACS transfer or even cheque. Just don’t post us an envelope full of used fivers – save those for the sorts of services where you don’t get a VAT invoice.