Happy Incorporation Day to Us
Happy Incorporation Day to Us
Happy Incorporation Day to Mythic Beasts
Happy Incorporation Day to Us

Fifteen years ago today someone with a boring job processed the paperwork and Mythic Beasts Ltd sprang into existence as a legal entity.

We had existed informally for a bit longer than that, we had registered the mythic-beasts.com that April, and our shell server, sphinx, had been running for a while, although it wouldn’t be until early 2001 that we sent our first invoice.

As we all work on t’internet, it’s difficult to all meet in the same pub this Friday for a celebratory drink. That will have to wait until our next full company meetup in September. Slamming the bedroom door, staying in and watching Brazil seems a more apt way for a teenage company to celebrate its bureaucratic anniversary.

One of the Mythic Beasts, Rhosyn has written an article on filtering for trefor.net, a widely read blog on technology and networking.

The part we particularly like this this quote,

As a long standing customer of Mythic Beasts 
(shameless plug; outstanding service and support, 
so good that I switched companies recently to work 
for them)

When you visit a web page, you’ll often see the URL change as it loads.  For example, if you attempt to visit http://mythic-beasts.com you’ll end up at https://www.mythic-beasts.com .   This is achieved using HTTP redirects, a response from a server that tells your browser that the page it is trying to load has moved.

HTTP redirects come in two flavours:

Permanent (301)

This tells the client that the page requested has moved permanently, and crucially, if it wants to load the page again, it needn’t bother checking the old URL to see if the situation has changed. This is a good way of redirecting something that you never want to undo, for example, if you’re permanently moving a website from one domain to another.

Temporary (302)

As the name suggests, this tells the client that the page has moved, but only temporarily, so the client should continue requesting the old URL if it wants to load the page again. This is a good way of telling users that your site is down for maintenance, that they they don’t have enough credit to access a site, or of some other issue that is likely to change.

#makeitwrong

Getting this wrong can be a massive pain for your users. For example, Three use a permanent redirect if you’ve run out of credit on your data plan, or you’re trying to use tethering in the wrong country, or some other temporary problem.

So imagine what happens when you run out of data on your plan. You attempt to visit your favourite website, say, http://www.xkcd.com . Three tell you that that page has been replaced by http://tethering.three.co.uk/TetherNoProductPost. Permanently.

Now find a working internet connection, attempt to load http://www.xkcd.com, and find that your browser quite reasonably takes you straight to the Three fail page, even if you’re no longer using a Three connection. Shift+Reload doesn’t help, even restarting your browser may not help.

Three have told your browser that every page you visited whilst out of credit has moved permanently to their fail page.

Expiring permanent redirects

The example given above is very obviously a place where a temporary 302 redirect should be used, but webmasters are often encouraged to prefer 301s in the name of improving search rankings. 301 redirects allow you to tell search engines that your .co.uk site really is the same site as your .com site, thus accumulating all your google juice in the right place. They also save a small amount of time in loading the page by avoiding an unnecessary HTTP request.

Even when used legitimately, 301 redirects are obviously hazardous, as there’s no way to undo a permanent redirect once it’s been cached by a client.

The safe way to do a 301 redirect is to specify that it will expire, even if you don’t expect to ever change it. This can be done using the Cache-Control header. For example, the redirect that we issue for http://mythic-beasts.com includes the following header:

Cache-Control: max-age=3600

This tells clients that they can remember the redirect for at most one hour, allowing us to change it relatively easily at some point in the future. We use the mod_expires Apache module to create this header, which also produces an equivalent “Expires” header (the old HTTP 1.0 equivalent of Cache-Control).

.htaccess example

The above can be implemented using a .htaccess file as follows:

ExpiresActive on
ExpiresDefault "access plus 1 hour"
Redirect 301 / https://www.mythic-beasts.com/

This example uses mod_alias and mod_expires which may need enabling globally in your web server. In Debian, Ubuntu and similar distributions, this is done by running the following command as root:

a2enmod alias expires

mod_rewrite example

Redirects are often implemented using Apache’s mod_rewrite. Unfortunately, mod_expires doesn’t apply headers to RewriteRules, but mod_headers can be used instead:

RewriteRule ^.* http://www.mythic-beasts.com/ [L,R=301,E=limitcache:1]
Header always set Cache-Control "max-age=3600" env=limitcache

The RewriteRule is used to sent an environment variable which is used to conditionally add a Cache-Control header. Thanks to Mark Kolich’s blog for the inspiration.

Again, you may need to enable mod_rewrite and mod_headers on your web server:

a2enmod rewrite headers

Escaping 301 hell

Fortunately, if you’re unlucky enough to get caught by a broken 301 redirect, such as the one issued by Three, there is an easy way to get to the page you actually wanted: simply append a query string to the end of the URL. For example, http://www.xkcd.com/?foo=bar. Browsers won’t assume that the cached redirect is valid for this new URL and websites will almost always ignore unexpected query parameters.

2015-07-03 – Updated to add mod_rewrite example
2020-03-16 – Updated to note that the relevant modules may need enabling

Here at Mythic Beasts it’s fair to say that our staff hold a wide spectrum of political beliefs, but I think one thing we can all agree on is that all the major political parties have at least some irredeemably stupid policies (and possibly also that some of the minor parties only have stupid policies).

This makes voting for a political party a pretty depressing prospect. So, what about voting for an elected representative who will look after our interests?

Our founders reside in two constituencies with notable MPs: Witney and Cambridge.

The MP for Witney is notable for being the Prime Minister. The MP for Cambridge, Julian Huppert, is notable for being a Liberal Democrat and yet still being highly regarded by a large number of his constituents.

Now, if you want good data on whether your MP is any good or not, you should head over to the excellent They Work For You and find out what they’ve been up to in Parliament on your behalf.

But who wants good data when you can have some anecdotes? Let’s look at two issues that have got us wound up recently.

Firstly, the EU VAT MESS, which causes us an administrative burden far in excess of the value of the affected revenue.

Julian Huppert was very active on behalf of the constituents who contacted him on this issue (Mythic only got as far as a tweet…), including submitting written questions in parliament, which received a predictably useless response.

On the other hand, Paul wrote to David Cameron twice (the first letter went AWOL), and received only a hopeless response which completely failed to address any of the issues raised.

Secondly, banning secure encryption. As a hosting company, the ability to undertake transactions securely online is quite important to our everyday business (see previous notes).

The appalling jeering by other MPs, and the pathetic response given by Theresa May, to Julian Huppert’s questions asked in Parliament demonstrated the he was clearly one of the few MPs who actually grasped the implications of the proposal, rather just resorting to rhetoric that fuels the fear that terrorism relies on.

As for David Cameron, well, it’s his idea.

So what can we conclude from this? Not a lot, except that we’d probably be in a far better place if parliament were full of representatives who listened to and understood their constituents, rather than those who get in on the strength of a party political vote.

At UKNOF31 we presented a talk entitled Catastrophic Unplanned Success, a slightly rushed history of how some of the rapid scale-up of RaspberryPi from the point of view of the hosting provider, detailing some of the issues we’ve dealt with during their extremely rapid scale up, and attempting to educate the teenagers into a proper DDoS rather than the half-hearted ones they’ve tried so far.

https://indico.uknof.org.uk/getFile.py/access?contribId=5&resId=0&materialId=slides&confId=33

We believe this talk was videoed, we’ll put the video up here too once it’s published.

Some time ago we were forwarded a plea by Liz Upton who’s sort of famous on the internet for some sort of cheap computer, on behalf of World Possible, which said

This brings us to good news / bad news.  Last month we pushed through 5TB of
FTP traffic, and over 20TB of FTP traffic on the year.  That's great, about
700 RACHEL downloads - but our web host isn't as excited with our success
and cut us off yesterday.

Liz thought this was the sort of thing we might be able to help with. So we got in contact and we’ve set them up with one of our older inexpensive servers to act as a new host. As it’s an educational project that we’d like to support; we thought we’d donate some bandwidth to help out. Since it nicely coincided with a substantial bandwidth upgrade in our Cambridge data centre we’d put the service there.

So far they seem pleased!

which is handy because some of their other suppliers who pay Amazon rates for bandwidth were a little bit annoyed with them.

Steven Allain

On Sunday night Pete was in the Hopbine and while buying some drinks the bartender asked him about his Raspberry Pi t-shirt and if he knew anything about it. One of the hazards of drinking in Cambridge is the barstaff are often considerably more knowledgable than you might expect at first.

Steven not only sells beer but is also a student at ARU studying zoology and has been using a Raspberry Pi and camera to look into monitoring and photographing things under water with motion detection. He commented that he’d just bought a Raspberry Pi model B+ and only a couple of weeks later the much faster model 2 B had come out and he wished he’d bought one of those instead, but as an impoverished student he couldn’t really justify replacing it.

Now we think taking photographs of fish and reptiles is pretty cool, so Pete took pity on him and gave him his model 2 Raspberry Pi in exchange for a future promise of some photographs of underwater things taken with his setup.

Ultimately this gets back to the real reason Mythic Beasts support Raspberry Pi. Not because it makes it cheap to run a formal curriculum for teaching in schools, but because it’s a catalyst for people to teach themselves. Steven may or may not have success in making a motion detecting under water camera but either way he’ll learn a lot in the process.

The mistake in all this? Not checking the Raspberry PI stock levels and Pete realising it’s going to take a few weeks before the replacement model 2 arrives – he’s back to his old much slower model B+ now and grumbling about it.

I would like to introduce our all new female GHOSTbusting team to tenuously tie in with a new Hollywood movie and gratuitously include a cool staff photo in this blog post, and for marketing reasons I’m going to ignore the reality that Toby did all the updates for GHOST.

Qualys found during a code audit a buffer overflow exploit for gethostbyname() in glibc which they’ve named GHOST. This means that any internet facing software that can be persuaded to do a DNS lookup is potentially vulnerable. To a first approximation that’s everything that’s listening on an internet socket.

The details are in CVE-2015-0235. Note this explains quite comprehensively how to exploit the vulnerability so we are expecting active exploitation to have already started.

The vulnerability was announced at 16:30 on Tuesday, at 16:40 the first ticket was opened in our queue automatically. We started reviewing the information shortly thereafter and deployed the updated packages to our shared hosting servers Tuesday evening. This gives a short window to discover any critical issues with the new packages before we start deploying updates to our managed hosting customers.

At 8:30am on Wednesday, we emailed every managed customer running vulnerable code (which is almost but not quite all of them) explaining the issue and indicating we’d be applying the patches immediately unless otherwise instructed not to. Giving customers a short window to reply before going ahead (some are automatically deploying via Puppet and don’t want us to update for them) we then applied the updates to the customer servers, which involved very brief interruptions to listening services as they restarted.

Subsequently spot auditing some customer machines indicates that the glibc update via the package manager may not have restarted every vulnerable process. We’re now writing some audit tools to check for missing service restarts. Tomorrow morning at 6am, our reporting package will report in lots of data about the status of all our managed customer machines including the complete process list and complete list of listening services, so on our reporting box we can do a complete audit for every listening process that hasn’t been restarted in the last 24 hours and investigate and fix where necessary.

If you aren’t a managed hosting customer of Mythic Beasts we implore you to update your systems as soon as possible, we strongly expect that someone is going to build a very big denial of service botnet very quickly from this vulnerability. If you have no idea how to update and audit your server please get in contact with us at support @ mythic-beasts.com even if you’re not hosted with Mythic Beasts.

8:30 : Wake up and get out of bed. Open the curtains to see the sun shining, put a dressing gown on and go downstairs to make some coffee.

8:40 : Take coffee to the home office and open up the laptop to start some work.

8:41 : Laptop does not ask for a password to decrypt the encrypted filesystem and refuses to work.

8:42 : Sip coffee and wait for desktop to boot.

8:43 : Log into desktop machine.
this wouldn’t actually work either, but we’re going to lie for narrative structure

8:45 : Open up web browser, default homepage is our support queue which displays message ‘I’m afraid this uses illegal encryption technology and you are not allowed to access this page’.

8:50 : Drink some more coffee.

8:55 : Realise there’s a copy of the customer support tickets in email, turn on email client.

8:56 : Wonder why email client gives strange connection errors that the mail server is refusing to allow it to connect with SSL turned on.

9:00 : Give up on email entirely, hurrah!

9:01 : Look at empty coffee cup, go downstairs to the kitchen to refill the coffee cup.

9:10 : Log on to company chat-room which fails to work with a connection error.

9:15 : Think this is all a bit bizarre so phone colleague on mobile, she answers to say that she’s having lots of problems too.

Spilled Coffee by Kenny Smith

9:20 : Conclude that the winning plan is clearly to spend the day updating some documentation while drinking coffee.

9:25 : Company wiki fails to load. Secure connection error.

9:30 : Decide to check the mrtg monitoring graphs to see if the network is working. Connection fails.

9:35 : Probably best to start fixing the mrtg monitoring server, first step, log into our bastion host which manages the access controls for servers on our network. Connection fails.

9:40 : This is getting really weird, probably best to go off and feed the cat who’s been miaowing for the last fifteen minutes demanding breakfast.

9:45 : Examine coffee carefully to check it’s not been tampered with and had hallucinogenic drugs added. Realise that if hallucinating could be hallicinating that no drugs were added when they were and how would you tell anyway. Conclude this is about to turn into a long, complex and ultimately nugatory philosophy problem.

10:00 : Return to desk, decide that the best plan is to audit our assets database and resolve some discrepancies between reality and the database by visiting the data centre.

10:05 : Unable to book visit to data centre, the data centre portal doesn’t work, connection errors.

10:10 : Unable to load the assets database, secure connection error.

10:11 : Unable to book car, Zipcar is down.

10:12 : Unable to look at map, Google Maps is down.

10:15 : Decide that the winning plan is to just give up, drink coffee and watch cat videos on youtube. Youtube fails to load with a secure connection error.

10:17 : Skim the news which has some article about a new government and some encryption technology. Click on a link in the forum which surprisingly fails to rick-roll.

10:20 : Now really very annoyed, going to have to waste time on facebook. Facebook refuses to load with a secure connection error.

10:30 : Phone company conference number for conference call to organise the day. Connection error.

10:35 : Really running out of ideas now of what to do. Go for a walk outside to a coffee shop. Mildly surprised that the sunshine is still working.

10:55 : Arrive in coffee shop to be greeted as Arthur Dent. Realise still wearing dressing gown, and for forms sake must now try and order a cup of tea.

11:00 : Order tea, coffee shop tells us that the credit card payment machine isn’t working and we’ll have to pay in cash. Observe that our wallet is empty. Leave coffee shop to go to cash machine.

11:10 : Cash machine is out of order.

11:30 : Return home and get dressed. Then collect cheque book, return to coffee shop. Persuade them that they can accept a cheque and order tea.

12:30 : Reflect that todays achievements so far consist of buying a cup of brown liquid that was almost but not quite entirely unlike tea. Go back home to face the afternoon.

13:00 : Decide that this is pointless and book tomorrow off. Holiday booking system doesn’t work, connection error.

13:10 : Decide this is lunacy and want to resign. Go to Linked In to update CV and find new job. Connection error.

13:20 : New job will probably be as crap as this one. Just resign. Fire up word processor, write resignation letter and email to boss.

13:30 : Email doesn’t work. Print it out.

13:40 : Printing doesn’t print either. Give up and copy it off the screen with a pen onto a piece of paper ready to post to boss. Realise there’s no stamps and with no cash it’s going to be hard to buy one.

13:45 : That’s it! Game over man! Game over! What the **** are we gonna do now? What are we gonna do? Maybe we could build a fire and sing a couple of songs? Why don’t we try that?

13:50 : Stop panicking and hit upon a cunning plan, steal all the money from the company and flee to a more sensible country than this one.

14:00 : Try to book a flight to Athens. Shopping cart fails with a connection error.

14:10 : Try to go to the bank website to withdraw all the money. Fails with a connection error.

14:20 : Visit the bank in person to steal all the money. Bank has a massive queue of people complaining because they can’t withdraw their money, apparently there’s ‘computer problems’.

14:30 : Give up on humanity entirely and go and find a park bench on which to live, in the vague hope that someone has a gold brick with which to wrap around a slice of lemon for brain smashing purposes.

The new Mythic Beasts Offices (public domain)

Those of you who follow us on Twitter are probably bored of us banging on about this, but the true lunacy of the EU VAT MESS has only just come to light. It turns out that the UK and other states are going to compensate the tax haven at the centre of this to the tune of €1.1bn for loss of tax revenue as a result of the rule change.

Let’s re-cap:

1. Large companies such as Amazon indulge in VAT tourism, by paying very low Luxembourg VAT when supplying to customers in the UK and other countries.

2. The EU declares this to be unfair tax avoidance, and decides to close the “loop hole” by making electronic services subject to VAT in the customer’s country rather than the seller’s.

3. Faced with the prospect of thousands of companies having to register for and operate VAT in 28 member states, HMRC sets up their VAT “one stop shop”, the VAT MOSS. This avoids filing separate returns to different states, but still requires sellers to track the multitude of different VAT rates in operation by different states, including obscure regional variations such as the Portuguese Azores.

4. The legislation does not include any thresholds for inter-state VAT, meaning if you sell a single item to a consumer in another EU state you must register for the VAT MOSS and charge EU VAT.

5. You can’t register for the VAT MOSS unless you are registered for UK VAT, meaning that if you make a single sale to another EU state, you’re obliged to start operating UK VAT on all your sales even if you’re well below the UK VAT threshold of £81k.

6. The guidance requires companies to collect an often impossible set of non-conflicting data to prove the consumer’s location, and then retain those records for 10 years.

7. HMRC’s original impact assessment recognised but dismissed this problem by vastly underestimating the number of businesses affected, and claiming that most small companies sell through online market places, giving as much as 70% of their revenue to companies such as… Amazon.

8. HMRC back-tracks on (5) by stating that you can avoid charging VAT on your UK sales by splitting your sales into two separate businesses – a technique known as revenue splitting which is normally considered illegal tax evasion.

9. Many companies (ourselves included) realise that the cost of compliance is greater than the affected revenue, and consider simply not supplying to consumers in other EU states, but are warned that this may be illegal under EU anti-discrimination laws.

10. Companies start to indulge in farcical discussions with HMRC about what constitutes an e-service. In some cases, by making the business less efficient, for example by manually attaching a PDF to an email rather sending it automatically, the service will no longer be considered an e-service.

11. Despite acknowledging that the change would impact businesses that are not currently registered for UK VAT, HMRC apparently did nothing to communicate the change to the rules to anyone other than VAT registered companies. Vince Cable then has the gall to tell companies just finding out about the change that he has done a lot to communicate the change.

12. Recognising that it’s about to lose a huge wedge of tax revenue, Luxembourg ups its VAT rate to 17%, a move which would probably have significantly curtailed VAT tourism on its own.

13. The UK and other member states agree to compensate Luxembourg to the tune of €1.1bn for the VAT revenue that they will lose as a result of companies ceasing to use Luxembourg as their tax avoidance state of choice.

You really couldn’t make it up.

As it stands, many micro-businesses are planning to simply shut up shop rather be killed by VAT bureaucracy.

For the record, we were already VAT registered, and did find out about the changes early this year, but the work of updating our billing system to cope with a plethora of different VAT rates, and the necessary “proof of residence” steps has been a massive and expensive distraction from doing useful things like upgrading our Virtual Servers.