We were planning to announce upgrades to our Virtual Servers today, but unfortunately we’ve had to spend time dealing with the #VATMESS

One of these coins is worth something. The other carries an obligation for a decade of document storage, 80 tax returns and tax rates for nearly 30 different countries. Guess which one we prefer?

At the moment, VAT on “e-services” sold within the EU is paid based on where the supplier is, so if you’re a small UK company selling, say, hosting services, you pay UK VAT to HMRC, irrespective of where the customer is.

If you’re a large company selling lots of such services then you’ll be paying enough VAT that it’s worth your while to move your operations to the member state with the lowest VAT rate, which is Luxembourg.

Of course, big companies avoiding tax is Evil, Bad and Wrong, so the EU has taken action.

The very short summary is, if you’re a non-VAT-registered customer in an EU state other than the UK, then we’re going to have to start charging you VAT at your local rate, rather than the UK rate. Good news if you’re in Luxembourg, bad news if you’re in Hungary.

The rather longer rant summary is that we’ve been forced to waste a significant amount of time understanding and complying with new regulations for VAT on electronic services which come into force on 1st January 2015.

Whilst cutting down on large companies undertaking VAT rate tourism might seem like a nice idea, charging VAT based on where the customer of an online service is creates a whole bunch of new problems:

1. How do we establish where a customer is based?

The guidance tells us that we need two non-contradictory pieces of evidence to establish the customer’s location, the most readily available being the billing address and the customer’s IP address. Setting aside the unreliability of geolocating IP addresses, what happens when a customer is enjoying their right to roam the EU freely and places an order whilst in another country?

Well, the guidance tells us we can use:

• location of the bank (we don’t collect this information)
• the country code of SIM card used by the customer (not applicable)
• the location of the customer’s fixed land line through which the service is supplied to him (not applicable)
• other commercially relevant information (suggestions on a postcard)

In the event that we succeed in obtaining the necessary evidence, we’re legally required to hang onto it for 10 years.

2. How do we find the correct VAT rate for a state?

Presumably, recognising that a huge proportion of companies in the EU now need to regularly lookup current VAT rates for different states, the EU will have created a convenient web service providing this information in a computer-readable format?

Well, the guidance sends you to this site which allows you to select “all states” and has an “Export selection” button. Looks promising until you try it and discover that it buries the data in a generated PDF.

Fortunately, some helpful soul has created what we actually want: a simple JSON feed.

Unfortunately, that site makes the amateur mistake of thinking that ISO 2 digit country codes will be enough to cope with all the VAT rates in the EU, forgetting that the Portuguese Azores and Portuguese Madeira have their own VAT rates, but not their own country codes. As it happens, the EU site listed above also denies knowledge of the VAT rates applicable in these regions.

3. How do we report and pay our VAT?

HMRC are proud to tell us that they’re saving us the burden of registering for VAT in each member state in which we do business by letting us use MOSS, their “One Stop Shop”, but we still now have to complete two separate quarterly VAT returns and, of course, the quarters don’t even align.

Bulk upload of our VAT data is supported using that well known open data-interchange standard: a spreadsheet. A particular highlight is that: “When completing HMRC’s spreadsheet you can’t use country codes (for example GB, UK, NL or DE) or country descriptions (for example Great Britain, the UK or The Netherlands). You must only use the following EU country names:”. That’s right, HMRC have eschewed ISO country codes for its preferred list of country names and spellings, and not even for the obvious reason that some states have multiple rates: Portugal is only listed once.

Tax doesn’t have to be taxing … but it is

The net result of these new rules is that it’s now much harder for us to sell to consumers in other EU states than it is for us to sell to consumers outside of the EU – surely the exact opposite of what the single market is supposed to achieve?

The amount of our business that is affected by these new rules is tiny, as most of the EU business we do have is to VAT-registered entities to whom an entirely different set of rules apply. The amount of profit we make in a year from the affected services is almost certainly less than the upfront compliance cost, if not the ongoing cost, so we have seriously considered simply refusing to sell to consumers in other EU states, although it has been suggested that this could be illegal under EU law!

It could be worse

These VAT changes are a nuisance for us, but we’re already well above the UK VAT threshold so already have processes in place to deal with the burden of UK VAT reporting. For very small companies, as we were not so long ago, these changes are absolutely horrific as there is no VAT threshold for inter-state VAT. The government accepts that requiring all businesses to operate UK VAT would be an unreasonable and stifling burden on small businesses, which is why we have a VAT threshold (currently £81k). But there is no such threshold for inter-state VAT, despite it being significantly more complicated to administer.

There is a growing storm of angry micro-businesses who, through virtue of not being VAT-registered, weren’t notified of the upcoming changes. Indeed, it seems that HMRC’s assessment of the impact of these changes not only vastly underestimated the cost of implementing them, but also completely forgot about several hundred thousand micro businesses that would get shafted by these changes.

(HMRC’s original impact assessment stated that “businesses currently unregistered in the UK who choose to register for MOSS in the UK will also have to obtain a UK VAT registration and their UK supplies will therefore also become liable to VAT”, meaning that if you sold a single e-service to an EU consumer you were pretty much obliged to start operating UK VAT too. HMRC have back-tracked on this by publicly endorsing the practice of splitting EU from UK revenue – despite revenue splitting normally being considered an illegal VAT-evasion practice)

Hip Hop is not only a style of music, but also the name of a virtual machine written by Facebook which compiles PHP Just In Time to make it go quickly.

Now we receive lots of unsolicited advice about how to run a not very popular wordpress blog and cope with the volume of traffic. Usually this involves ripping and replacing the entire infrastructure from a standard Linux/Apache/MySQL/PHP stack to something different (Nginx/MariaDB/PostgreSQL) which may not even be able to run WordPress at all (e.g. node.js).

At Mythic Beasts we like to understand what we’re doing, rather than blindly installing Magic Go Faster Solution Number 7. So we set up a test 2GB dual core virtual machine, that runs WordPress and a selection of popular plugins ( WordPress SEO, Akismet, Safe Report Comments, Liveblog, Facebook, Yet Another Related Posts Plugin, WordPress Supercache and Jetpack, no endorsement implied). Then we benchmarked with siege and managed the following results.

Apache/mod_php : 5.10 trans/sec

and when you turn supercache on and serve cached pages you get

Apache/mod_php/supercache : 873.50 trans/sec

So this gives us two scenarios, pages which we have to generate content for which can easily cause load issues, and pages served from supercache in which our VM is fast enough for all practical purposes and will easily weather even very big traffic spikes from news websites or television adverts.

Now, it’s very popular to tell us to use Ngnix as it’s faster than Apache. Is it though?

Nginx/php-fpm: 5.70 trans/sec
Nginx/php-fpm/supercache: 2230.58 trans/sec

Wow! Nginx is three times quicker than Apache at serving cached pages. This is amazing, but not very helpful. It means when our webserver is serving pages really quickly, we serve pages at three times really quickly, but when we’re generating pages on demand, it’s about 10% quicker. That’s not very special and doesn’t justify a rip and replace of the whole installation for a 10% performance improvement.

A quick look at the VM during the testing tells us that the bottleneck is executing the PHP code which creates WordPress pages. The choice of webserver is basically irrelevant; almost all the server time is spent executing PHP and reading data from the database.

Enter HipHop Virtual Machine.

This is nothing to do with the HipHop Virtual Machine. But we like tea and Banging Tunes

It has one focus, to execute PHP quickly for Facebook. Facebook have a lot of servers and spend hundreds of millions to billions per year on servers and data centres. A 50% performance improvement in PHP saves them huge sums of money in data centres and servers alone, so it’s clearly worth them trying to optimise as much as possible.

Here’s what happens with Apache/Nginx running HHVM.

Apache/HHVM :           35.93 trans/sec
Apache/HHVM/supercache: 928.70 trans/sec
Nginx/HHVM :            33.78 trans/sec
Nginx/HHVM/supercache : 2137.67 trans/sec

This is a huge improvement for non cached pages – seven times faster. Cached pages are bottlenecked in the webserver so it makes minimal difference, but they were already so fast we weren’t worried about them. Again Apache/Nginx are still pretty much the same speed for generated pages, we’re still dominated by the code execution time but a seven fold performance improvement is worth seriously considering.

Whilst we can reconfigure servers standing on our heads, we usually don’t.
Photo credit: Mark Dolby, Flickr, CC-BY.

All I need to do now is see if I can find someone with a very busy WordPress site and a million complaining users who would like to test it to see if it’s really as good as the lab tests suggest it might be.

Very sorry to hear the news that Big Bank Hank who co-wrote the first ever hit Rap track Rappers Delight died earlier this week from kidney complications related to cancer.

You see, he was six foot one, and he was tons of fun

At Mythic Beasts we try very hard to keep our customers happy, and to do our absolute best to meet their requirements in requests, even if they’re occasionally a little bit unusual.

One of our long standing customers is refreshing some of their hardware, and we had the following exchange to sort out the details

customer> The following 8 servers have been decommissioned and now need removing: 

mythic-beasts> We can sort that for you. Do you want to collect the servers or shall we recycle them for you?

customer> The drives can be kept for spares but you can ditch the servers or make a fort out of them or something..

a 1U server fort

Now it’s not really our field of expertise, but we think we’ve got a reasonable start on building a defensible concentric castle although we ran out of servers before we could start building the outer curtain wall.

Managed server customers receive as standard 24/7 monitoring of their servers, we check that the machines are up, that ssh is running, that the web-server is delivering the correct content amongst other checks. In the event a check fails our staff are alerted via SMS/pager to investigate the issue.

We’ve now enhanced this service for managed server customers, in the unlikely event you have a service affecting issue that the automated monitoring hasn’t caught, you can file an urgent ticket through our control panel which will create a new support ticket and alert our staff via SMS to deal with your issue.

This was a feature request from a customer in a meeting last Thursday and went into production as a service enhancement on Tuesday, we’re always receptive to suggestions from customers to make our offering better.

bzip2 is one of the great unix tools. It compresses and uncompresses data, and it does it very well. We’ve been using it within Mythic Beasts for years and it’s operated absolutely flawlessly.

We’re happy to report that we’re now hosting the main distribution site for bzip2

Thanks to Jonathan Wright who runs a very big website, for a nomination.

I’ve nominated Matt Smith, Rob McQueen and Neil McGovern.

Thanks to Ben Howe, our gap year student who’s adequately demonstrated to his colleagues the definition of a career limiting move by dunking a bucked of ice over his boss, The Haymakers for kindly providing the location for the company meeting, the chilled water and the ice, and the rest of my Mythic Beasts colleagues for filming and laughing.

Mythic Beasts have added paypal functionality to our billing system. You can now pay by credit or debit card, paypal, direct debit, BACS transfer or even cheque. Just don’t post us an envelope full of used fivers – save those for the sorts of services where you don’t get a VAT invoice.

If you’re a Debian Developer and you’re going to the annual Debian UK Barbeque we hope to see you there, and give you a beer to thank you for your hard work.

On Monday, one of our founders Pete is going to run the Boston Marathon which was the site of a terrible terrorist incident last year. You defeat terrorism by not being scared and not changing your plans, so despite the massive security inconvenience he’s gone anyway.

Last week Pete went for a run around the London Marathon course with Jeremy and Chris, as they’d always planned to turn up and together. Sadly Chris wasn’t able to make it because a fatal brain tumour in 2012 has set back his training quite badly – it’s not looking promising he’ll make the start line next year either.

So if you were feeling charitable, and wanted to win a Mythic Beasts Mug, you could make a donation to cancer research at Jeremy’s fund raising page, putting your time prediction in as a comment in the form

MB account-number 3:15:30

If you get the closest result to his finishing time we’ll send you a very limited edition Mythic Beasts Mug.

If you want some information to guide your mug winning entry, all of Pete’s training from January to March was curtailed with a foot injury. his first run back was the Cambridge Half Marathon (1:39), and last week in London did a final training run with a hang over and toilet break in 3:27:22. Last year when on form he ran 2:59:28 again in London. Boston has hills and the course is a straight line so wind conditions can make a big difference.

With DynDNS shutting down their free dynamic DNS service quite a few customers contacted us to ask for an upgrade to our DNS API so that it can effortlessly substitute in for DynDNS. For example we saw this request over twitter.

@Mythic_Beasts dyndns is closing its free dynamic dns service (*major internet trauma*). Could you write a blog post about what MB offers?

— David Earl (@frankshad) April 8, 2014

Several people offered a suggested implementation. They all asked for something along these lines,

Can you make it so that when you put an update for an A record through
it can substitute in the calling IP address instead of supplying one.

It’s clear that many of our customers still don’t instinctively get the implications of dual stack hosting yet.

In particular if you call the API from a v6 address and ask for an update to your A record it will fail because it doesn’t know your v4 address. Similarly if you call it from a v4 address and ask for an update to your AAAA record, it will also fail for precisely the same reason. You have to call the API to update your A or AAAA record over the correct v4/v6 channel in order to get the correct update, and if you’re dual stack you need to call it twice.

We decided that we should fasttrack this project, and gave it to Liam Fraser – who’s back with us during his Easter holidays. We’re proud to present our fully documented and enhanced Dynamic DNS API which sample code for single (v4 or v6) and dual (v4 and v6) stack hosts.

We include our DNS API free of charge for anyone who’s bought a domain name from us, so if you’d like to use our Dynamic DNS service, create an account at our control panel and follow the instructions.

If you’re bored of working for a company where you can’t get things done, and would like to work somewhere where you can implement, test, document and release a small enhancement in about six hours, you can apply for a job with us here. We’re especially interested in talented graduates and school leavers who know their Linux.