Additional Managed Rack Capacity

March 14th, 2016 by

We’ve spent even more time than usual in data centres recently as we’ve been kitting out our new cage in the Meridian Gate data centre.

Much of the new capacity is being deployed as “managed racks”.  Racks are generally supplied with the bare essentials of electricity, cooling and locked doors.  At Mythic Beasts, we transform them into managed racks, including all the features you need to effectively administer your equipment remotely, including:

logging serial consoles

Logging serial consoles

  • Internet connectivity – we’ve got 10Gbps connections onto both LINX networks, connecting at different sites.  We’ve also got multiple transit providers, and are present on the LoNAP peering exchange.   Our network has native IPv6 support, and if you have your own address space, we can provide you with BGP feeds from our routers. We can also offer private LANs, both as VLANs or as physically separate networks.
  • Remote power management – power cycle your server immediately, at any time using our customer control panel.
  • Serial connectivity – a 115.2kbps serial connection may seem a bit old fashioned in an age when we’re wiring our switches together at 40Gbps, but they remain an extremely effective mechanism for out-of-band control of servers and other equipment, particularly when coupled with our logging serial console software.
  • On-site support – all of our London facilities have 24/7 access to the data centres’ on-site engineers.  We are also able to arrange for our own staff to carry out routine maintenance, such as replacing failed hard drives.

Meridian Gate is the third London data centre in which we have a presence, along with Sovereign House and Harbour Exchange, with the three sites connected by our own dark fibre ring.

Rebuilding Raspberry Pi

March 9th, 2016 by

After the Raspberry Pi 2 launch in February 2015, we had a review of how to improve and scale the hosting setup for Raspberry Pi. There were two components that caused us pain during the Pi 2 launch: the main site, running WordPress, and the forums, powered by phpBB.

The first question from our review was whether we should be putting effort into scaling a WordPress site. WordPress is estimated to be powering as many as a quarter of all websites, and it’s popular for a reason: it makes site development very easy. WordPress is easily extensible through themes and plugins, it’s supported by a vast array of existing third party plugins, and it provides a good built-in framework for delegating and moderating authoring roles.

Unfortunately, this ease of development is at least in part down to a very simplistic execution model, with each page being dynamically generated, executing code from every installed plugin, and typically resulting in multiple database queries. When the Raspberry Pi site gets busy, it’s usually down to a huge number of visitors hitting just a page which is essentially a static news story. WordPress provides no built-in mechanism for caching such content, so by default, we’re dynamically generating many copies of identical, or near identical pages.

Losing the flexibility and ease of development offered by WordPress just to cope with the handful of days when the site gets very busy would be unfortunate, so we decided to put effort into making the existing site scalable.

Caching

For pretty much every WordPress problem you can imagine, there’s at least one plugin offering to solve it for you. For site performance, there are a number of plugins such as WP Supercache, but as WordPress itself provides no framework for identifying cacheable parts of page, these can only take a very simple and typically over-cautious, page-based approach.

For example, if you’re a logged in user, you might get served a page that is in someway tailored to you, so Supercache bypasses its cache and serves you a dynamic page. Similarly, if Supercache sees a request that looks like a comment being posted, the cache is invalidated, and a dynamic page is served, and cached for future requests.

During the Pi 2 launch, we saw significant problems with load spikes when comments were posted. Clearly, small delays in comments being visible on the site is a minor annoyance compared to thousands of visitors being served an error page, so we set about making our caching more aggressive.

We wrote a small hack called staticify. This fetches the key pages from the blog every 60 seconds and renders them to static HTML. That way we always have a page in our static cache, and because we’re selecting the pages that we cache we can afford to be more brutal with our caching: we know that there’s no user-specific content on these pages, so we serve up the same cached page even if you’re logged in.

More virtualisation

An important goal after the Pi 2 launch review was to split out different parts of the site onto separate virtual servers. For example, having the WordPress blog and the forums software on different VMs made it much easier to experiment with using Hip Hop VM which offered a significant performance boost to the blog, but is incompatible with the forum software.

Although the Raspberry Pi setup runs as a private cloud on a single host machine, having different components split onto separate VMs makes it much easier to balance resources between them, and if necessary, spin up extra capacity quickly using our public cloud.

IPv6

When Raspberry Pi was hit by DDOS we built an IPv6-only backend network for the machines to communicate with each other. In the new setup all access to the back-end VMs comes from either one of four front-end load balancers, or a “gateway” VM. So we thought we’d remove IPv4 connectivity from the VMs entirely. For example this is ifconfig on one of the blog PHP VMs :

eth0      Link encap:Ethernet  HWaddr 52:54:00:3f:8a:5a  
          inet6 addr: 2a00:1098:0:82:1000:x:y:z/64 Scope:Global
          inet6 addr: fe80::5054:ff:fe3f:8a5a/64 Scope:Link

The VM needs to occasionally call out over IPv4. For example, akismet and Twitter don’t yet have full IPv6 support, so these requests go through a NAT64 gateway, provided by Mythic Beasts that proxies the connections so it appears almost seamless to the VM. This is part of the Mythic Beasts IPv6 education project, backward ISPs claim there is no demand for IPv6, whereas we provide multiple services from IPv6-only servers and give discounts if you use IPv6-only services.

SSL

Officially we enabled SSL because we wanted to improve our Google ranking but handy side effects include irritating the security services and preventing third party networks injecting adverts or corrupting downloads. The SSL decryption is done on the front-end load balancers and as they have lots of spare CPU incurs no performance issue. The only thing that isn’t is the image downloads because of incompatibilities with the current version of NOOBs. We hope to eventually resolve this.

Pi Zero Launch

November 26th at 7am, the Pi Zero is launched, a $5 computer given away on magazines. The bandwidth graph for the Raspberry Pi server does this:

Launch day bandwidth graph

Launch day bandwidth graph

It’s very exciting and quickly exceeds our previous records for the launch of Pi 2. The two VMs that generate all the webpages for the blog and deliver all the content are humming along at 10-25% capacity. The database VM is almost completely idle, we’ve successfully cached almost everything,our database server only sees load when a comment is being posted or the cache is being updated. Meanwhile we neatly exceed the 4500 users we had for the Pi2 breaking 10,000 simultaneous users at our peak.


A quick back-of-the-envelope calculation and we conclude that our staticify script avoided executing WordPress a large number of times and the following slightly dubious claim is mostly true:


The MagPi site was a bit more difficult, it hadn’t had the same level of optimisation and went through a number of changes throughout the day to accelerate it. However, the VM setup meant that the excess load was contained to specific virtual machines- under our original flat hosting setup the load from the MagPi would have taken everything offline and made identifying the underlying cause much harder.

Raspbian

We now run a full mirror of the main Raspbian site, and we’ve even done a test to make sure that the failover works.

The mirror director is a critical piece of infrastructure, without it package downloads will fail and updates can’t complete. So in the event of a failure we need to bring the mirror director back up much more quickly than we can restore 4TB+ of data from backup. As a result of this work we now have a hot spare, which has been fully tested.

Does it work and is WordPress still a good idea?

We weathered the PiZero and Christmas Day traffic peaks with ease and we think we can probably double or triple the number of people using the sites at peak times before we have to think much more or add hardware. The result is we’ve a really useful and very busy site, that supports our multiple contributors, moderators and users with a relatively minimal amount of engineering and administration time, on a comparatively small server setup.

One-click DNSSEC – public beta

March 4th, 2016 by

It’s been a long time coming, but we’re now pleased to announce that we’ve got DNSSEC support in public beta, and you can enable it for your domain at the click of a button.

What is DNSSEC?

DNSSEC is a set of extensions to the DNS protocol that ensures that you can trust the IP addresses that you get back from the DNS system. For example, if you visit www.yourbank.com, the first thing that happens is that your browser uses a DNS server to find out the IP address of your bank’s web server. But how do you know that you can trust the address that you get back? Your request will probably get bounced through multiple DNS servers, such as your home router, your ISPs servers, and finally the authoritative server for the domain. If any one of those gets compromised (and let’s face it, home routers have a terrible security record) it could easily insert a different IP address and direct your request to an entirely different server.

DNSSEC means that all responses are signed with encryption keys that have been lodged with the registry, so you can’t inject bogus responses just by compromising an intermediate server.  Of course, the system only works if the systems making the requests check the signatures of the responses that they receive, something which certainly doesn’t happen everywhere yet.

Sounds complicated?

Yes it is, particularly as it is recommended that the encryption keys that you use are changed (or “rotated”) regularly. Fortunately, we’ve now automated all the hard stuff, and if you’ve got your domain registration and DNS hosting with Mythic Beasts, you can make DNSSEC go just by hitting a big green button.  We’ll take care of the rest:

Screen Shot 2016-02-29 at 18.37.29

Unlike some people, we believe that the internet should be a safe place to do business by default, so this service is, and will continue to be, provided at no extra cost.

If you want to try it out, simply visit our control panel, find the domain under “My Domains” and follow the “DNSSEC” link.

Hosting the Raspberry Pi 3 launch, on a Raspberry Pi 3

February 29th, 2016 by

Four years ago we sat on the phone while Eben Upton pushed the button to launch his educational computer, the Raspberry Pi, and we joined them on a fairly remarkable journey.  “How do you sell and ship 10,000 Raspberry Pis?” turned into “how do you sell and ship 5,000,000 Raspberry Pis?” and “how do you contain the excitement of the internet when you put a computer on the front of a magazine?”

Today, we’re nervously watching all the server graphs as the new Raspberry Pi 3 launches and goes on sale. We’ve had one to play with for a while so we did what we do with any new shiny toy: benchmark it in a real world application.

Rasberry Pi 3

Our Raspberry Pi 3 next to a Raspberry Pi 2 serving requests for the Raspberry Pi 3 launch.

Our favourite application is rendering WordPress pages for the Raspberry Pi website, so we set up a testbed: Pi2 and Pi3 versus the virtual machines that run the blog. We picked a typical page and tried them out. Initial results weren’t promising – just one fifth the speed of the production VMs.  The VMs have the advantage of being on the same physical server as the VM that hosts the database.

Moving the Pis to the same switch as the database server, and upgrading from PHP 5.6 to PHP 7 brought Pi 3 page rendering times that were less than twice as long as the production servers.

Server Spec Seconds per page
Blog VM (PHP 5.6) 24 x 2.4Ghz Ivy Bridge 0.4
Pi 2 (PHP 7) 4 x 0.9Ghz A7 0.9
Pi 3 (PHP 7) 4 x 1.2Ghz A? 0.7

That’s fast enough to be usable. Parallelising requests across all cores, we can probably sustain about 4 hits/second from the Pi 2, 6 hits/second from the Pi 3 and around 50 hits/second for the main site.

These figures are for uncached pages.  As we’ve seen in the past, 50 hits/second isn’t even close to enough to cope with launch day traffic.  In reality, the vast majority of pages we serve are cached and both Pis can adequately serve 100Mbps of cached pages (versus 4Gbps for the main host).

So we’ve done what any sensible real world test would do, we’ve put them into the main hosting mix. If you read the headers you’ll see on some requests

  HTTP/1.1 200 OK
...
  X-Served-By: Raspberry Pi 3
...

indicating your page request came off a Raspberry Pi 3.

We’re aiming to serve about 1 in 12 requests from a Pi 2 or a Pi 3, but may adjust this up or down to keep the pi in action and not melting under the load.

How’s it done?

The backend for the Raspberry Pi site is built from virtual machines.  One VM  runs the database, and a pair that generate pages for the main, WordPress-based, website.  One of the pair is designated as primary, and also runs the admin backend for WordPress, which then synchronises files  to the other VM, now additionally, both the Raspberry Pis. All the backend servers exist on a pure IPv6 network.  We have a cluster of front-end servers that are dual stack, and load balance requests through to IPv6-only backends.

If you have IPv6 you can see the status of the two Pis here:

stats.pi2.raspberrypi.org
stats.pi3.raspberrypi.org

If you don’t have IPv6 complain to your ISP, then set up a tunnel at he.net.

The two Pis can tweet directly at @hostingpi3 and @hostingpi2. Sadly, Twitter doesn’t support IPv6 so traffic goes via our NAT 64 service that provides outbound connectivity for IPv6-only servers to legacy parts of the internet.

UK domain price rises

February 5th, 2016 by
You could save as much as this by renewing early

You could save as much as this by renewing early

On 1st March, Nominet (the UK domain registry) are raising their wholesale prices by 50% for all registration periods except 1 year registrations (which are rising by less, but are currently disproportionately expensive), and we will be applying corresponding increases to our prices for these domains.

If your domain is due for renewal in the next six months, you can save a few quid by renewing early.  You can renew your domain in advance for up to nine years.

Our new and old prices are as follows:

Period
(years)
Old price
(ex VAT)
New price
(ex VAT)
1 £5.42 £5.83
2 £7.50 £11.25
5 £17.92 £26.67
9 £30.00 £45.00
10 £32.50 £48.75

The problem with the youth of today is hard work and excessive talent

February 1st, 2016 by

Almost a decade ago in 2006 Eben showed off his Raspberry Pi protoype with the original aim being to improve the standards of computer science for Cambridge University Computer Science students. Four years ago the Raspberry Pi launched, and the first post-pi generation of students are now in their final year. Today I went back to the Computer Science department to deliver a lecture for their e-commerce course on web application architecture and scaling. Several of the examples were taken from scaling of the Raspberry PI website – a website literally built with the aim of educating the students sat directly before me.

One students in the room was Will Shackleton. Will came over after the lecture to show me one of his projects, he’d taken my original Upside-down-ternet project for messing with my neighbours who stole my wifi, and turned it into an android app, Network Spoofer that you can use on wifi networks.

His first implementation embedded a Debian chroot with my original terrible perl script and squid before he reimplemented it properly with a proper proxy and parser to enable many more features and cleaned it all up. We discussed some issues – e.g. evil hosting providers like us enabling free SSL which prevents intercepting connections, how a sense of ethics is important – Will has refused to implement SSL strip – and I gave my sketch idea of how to do an IPv6 gateway version that wouldn’t require arp spoofing.

So The Youth of Today are taking the things we built, cleaning up the messes we left behind and making things better.

Free SSL certificates for hosting accounts

January 29th, 2016 by

Customers with hosting accounts on either yali or onza can now get free SSL certificates for websites, allowing you to have an https version of your website. We’re using the Let’s Encrypt certificate authority to provide the certificates.

To get a certificate and enable https hosting for your site, simply press the button in the control panel, and within 5 minutes you should have a working https site.  You can find the option under “Web and Email Hosting“.

Free SSL at the press of a button

Free SSL at the press of a button

Let’s Encrypt certificates have a short expiry period, but we will take care of automatically renewing them for you.

Why use HTTPS/SSL?

Using SSL on your website means that traffic between our server and your user’s computers is encrypted and can’t be intercepted (despite David Cameron’s desires).  It allows browsers to guarantee that they are indeed talking to the website shown in the address bar, even if they are using an untrusted network connection.  Even if you don’t view the security aspects as a benefit, Google have previously announced that they will boost the page ranking of SSL-enabled sites.

Sphinx accounts

Unfortunately, this service is not yet available to customers on our sphinx server.  We are working on that, and will have it enabled in the near future.

Decimal points are important

January 5th, 2016 by

Ben at Raspberry Pi wanted to use his new vanity domain rpf.io, as a URL shortener rather than the URL of the common big services. The easy solution was to use an existing service on a paid account which gives us analytics and tracking. However, demonstrating the age old principle of if you have to ask you can’t afford it, his email reads…

$695/month for a .htaccess file

We like Open Source software, so instead of paying enough money to rent quite a nice car for a trivial .htaccess file we instead chose to install yourls on a little IPv6-only virtual machine behind our NAT64 and IPv6 Proxy services.

We’ve done some benchmarking, out of the box we could sustain 500 hits/second, adding in php-apc boosted this to well over 2000 hits/second which should be enough, even if Liz Upton gets very excited with the Raspberry Pi twitter account.

So you can test out the service here http://rpf.io/mythic before we start making these links public.

I did not go to the ball

December 11th, 2015 by


Last night was the annual TrefBash, a large party for people involved in the UK networking industry. Several people asked why Pete didn’t attend, so we’ve lent him the company blog to explain his answer.

 

Pete writes:

I donate a lot of time to Raspberry Pi because I believe that we need more competent people in the tech industry. One thing I see is at younger ages we have a mixed gender balance, at older ages it becomes increasingly male dominated. A quick look at the boards of the UK Internet Exchanges (100% male) and attendance lists for conferences will tell you that networking is no different from the rest of the tech industry with an extremely strong male bias.

It’s improved: it’s been a while since there has been an AGM for a Internet Exchange in the Bavarian Beerhouse where busty women with cleavage serve male geeks beer. We have a Respect policy for conferences so it’s now official that you can’t be discriminatory. However as with all things it’s two steps forward, one step back.

Not a photo of the burlesque dancer, our blog is safe for work even if the industry events aren't

Not a photo of the burlesque dancer, this blog is safe for work even if the industry events aren’t

The theme for last nights TrefBash15 was Bond meets Rocky Horror. Whilst I’m highly entertained about a theme where equal opportunities rapist Frank ‘n’ Furter explains to misogynist Bond that you should treat women as well as you treat men, and Bond suggests to Frank that maybe you should ask permission first, I can’t help feeling that wasn’t quite the way the event was planned.

To check I wasn’t being hyper-sensitive I did ask a number of people what they thought. Liz Upton generously allowed me to quote her in full.

Liz Upton, Director of Communications, Raspberry Pi

Liz Upton, Director of Communications, Raspberry Pi

James Bond – well, I SUPPOSE there’s the option of evening dresses for the ladies. That’s the very kindest spin I can put on it. Jesus, this industry sucks.

Liz Upton, Director of Communications, Raspberry Pi

 

 

 

 

So instead of going, I donated the cost of a night out in London to The Women In Engineering society instead.

PiZero won!

December 10th, 2015 by
Pi Zero, 2p for scale, not included with prize.

Pi Zero, 2p for scale, not included with prize.

Last week we started a competition to win a Pi Zero. We’ve had a small number of entries, half from school age people eligible to win, the other half from entertained techies. We’ve also been using this as a job filter for some time so we have a rich depth of existing answers.

The competition is very simple, our web-page generates a mathematical sum for you to work out the answer to, and in order to succeed you have to send us the answer within one second. It’s an anti-turing test – a person can’t do it but a computer can. When you’ve succeeded we ask you to send us the code. This gives two important things, a code sample that a candidate wrote and an idea of how long it took them to work out they should automate it.

A text-book answer from an experienced techie is about 15-30 minutes and delivers a short piece of code that fetches the web-page, finds the sum with a regexp or similar, works out the answer with eval, creates the response and sends it back to us. However, people are much more creative than that.

One excellent answer was a single line of shell script, which did the whole thing in bash, but even more cleverly searched the process list for the command that had been typed in, added the escaping back in and then mailed itself to us – a one liner that did the puzzle, recreated its source code and emailed itself to us.

Another excellent answer was someone who guessed that our code had a 32 bit roll-over bug in, tried options until we generated an answer bigger than 2^32 and fixed the otherwise text-book code to implement the bug on our side.

The absolute worst answer we’ve ever seen was someone whose CV listed them as a professional programmer with five years experience. After two whole days of typing answers into the website they finally worked out that the problem had to be automated. After three days of development they wrote a vast piece of Java code that was able to download the page and find the problem but it was only able to work out the answer if there were only three numbers and they all had to be added together. Instead of improving the code for the general case they put it in a loop and repeatedly called the page until by sheer luck they got a page their code could answer. Creative genius but not in a good way.

On to the entrants

So this is a difficult challenge for school age children and teenagers. Most of the entries came from older children 16 and up, and it’s clear that it was difficult and they had to learn new things specifically to solve this. PHP and Python were the preferred languages – the most novice friendly of all the tools available. We were very torn as to who should win. After lots of deliberation our runner up is this php entry from Nils in Germany who was also the first to submit,


// That was fun. There should be more conetsts like this.
// Sorry for the incredibly hacked together code...
<?php

$challenge =
file_get_contents(‘http://sphinx.mythic-beasts.com/cgi-bin/job.pl’);

$task = preg_replace(‘/.*

Evaluate /is’, ”, $challenge);
$task = preg_replace(‘/ \, .*/is’, ”, $task);

$id = preg_replace(‘/.*name=\”id\” value\=\”/is’, ”, $challenge);
$id = preg_replace(‘/\”.*/is’, ”, $id);

function calc_string( $mathString ){
$cf_DoCalc = create_function(“”, “return (” . $mathString . “);” );

return $cf_DoCalc();
};

$ans = calc_string($task);

$url = ‘http://sphinx.mythic-beasts.com/cgi-bin/job.pl’;
$data = array(‘answer’ => $ans, ‘id’ => $id);

$options = array(
‘http’ => array(
‘header’ => “Content-type: application/x-www-form-urlencoded\r\n”,
‘method’ => ‘POST’,
‘content’ => http_build_query($data),
),
);

$context = stream_context_create($options);
$result = file_get_contents($url, false, $context);

var_dump($result);

?>

Things we particularly like are that all the comments and code and email are in English to make it easier for us, even though it’s the authors second language.

Our winner though goes to 13 year old Nick. From a pure technical standpoint his code isn’t as good as Nils’ entry above, but through the comments it tells the story of slowly working his way towards the solution and appropriately credits the help he received – both personal and online.


#Mythic Beasts - Win a Pi Zero
#Written by Nick Lockhart from Chepstow, Wales, aged 13 3/4
#with help from an old fossil (his dad)
#You will need LXML for Python and Requests

from lxml import html
import requests
debugmode = 1

#get the page and parse all

elements into a Python list.
#For this purpose we should only get one element.
page = requests.get(‘http://sphinx.mythic-beasts.com/cgi-bin/job.pl’)
tree = html.fromstring(page.content)
paragraphs = tree.xpath(‘//p/text()’)

#Split out the question.
#First take out everything after the sum (space included)
#And then remove everything before the sum (space again included.)
#And convert to a string. Oddly, after the second time, we have to
reference the second element as there is a blank string in question[0]
#Finally, evaluate it.

question = paragraphs[0].split(‘ ,’)
question = question[0].split(‘Evaluate ‘)
question = question[1]
answer = eval(question)

if (debugmode == 1):
print(“The question was: ” + question)
print(“The answer should be: ” + str(answer))

#There’s a hidden input labeled “id”, which seems to be randomly generated.
#This is probably to track who’s submitting.
#We will also need to extract this.
#I learnt this piece of magic with help from StackOverflow. Thanks,
Mathias Muller!

secretkey = tree.xpath(‘//input[@name=”id”]/@value’)[0]

#That’s all we need to POST. Let’s generate a payload, send it off and
extract the response.
#The server expects the answer to be a string, so we convert it to a string.

payload = {“answer”:str(answer),”id”:secretkey}

if (debugmode == 1):
print payload

response =
requests.post(“http://sphinx.mythic-beasts.com/cgi-bin/job.pl”,
data=payload)
responsetree = html.fromstring(response.content)
print(responsetree.xpath(‘//p/text()’))

Of course the final comment to everyone who entered is if you ever need any kind of hosting, domain name or similar send us an email and include your entry number for a freebie / upgrade /discount. Secondly if you seek summer work or gap year employment, we’d invite you to get in touch and we guarantee that we’ll read your CV and take your application seriously.