Save the Black Horse

May 26th, 2017 by

The last pub in Dry Drayton has closed and is under threat of development. As a community, we’re working hard to save it.

It’s Beer Festival week in Cambridge. Suddenly official work takes a back seat compared to the importance of drinking, serving and appreciating fine beer in the sunshine. It’s great that the volunteers behind the bar include friends, colleagues, customers, suppliers and the occasional former MP.

However, for 51 weeks of the year the Cambridge Beer Festival isn’t operating and beer lovers among us have to go to a more humble establishment, the pub. Cambridge City is blessed with multiple excellent pubs, but occasionally it’s nice to take a visit to the outlying villages.

So we were very saddened to hear that the only pub in Dry Drayton, the Black Horse, was due to close. However, a community group has started assembling plans to turn it into a community pub on a similar model to the excellent Dykes End in Reach. They asked us to help with setting up their on-line presence. Mythic Beasts fully support the effort to have lovely pubs within walking and cycling distance so we’ve provided a Managed WordPress site to help their campaigning efforts. Today we’ll share a beer with them in the Beer Festival, and in the near future we hope to take a field trip to their re-opened countryside pub.

Update: They reported last night they’ve had a lot of signups to their newsletter and several interested investors. It looks like we’re going to be part of a successful pub rescue!


Ten years on, Chris Lightfoot looks more prescient than ever

February 13th, 2017 by

(gif from imgur via gify)

(title shamelessly stolen from Tom Steinberg, MySociety founder from his tribute to Chris, Mythic Beasts founder who died ten years ago).

Lots of people have been excited recently about this script, which allows you to remotely reinstall a Linux system with a different version of Linux by giving you a shell in a ramdisk and letting you reinstall the operating system from there.

Chris did this the hard way. Back in 2005 I remember being asked to code review ‘evil.c’, a script that allocated a lot of RAM (800MB!), compressed a filesystem into it, then uncompressed it back to the disk. On reboot it should come up with Debian instead of FreeBSD that it had earlier. It’s really very important not to swap during this process.

Amazingly it worked, and the first test was on a remote box and it saved us a data centre visit. Here’s the code in its full glory.

#include <sys/types.h>

#include <errno.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <zlib.h>

#include <sys/mman.h>
#include <sys/reboot.h>

#define SIZE        ((size_t)896058269)

#define E(...)      fprintf(stderr, __VA_ARGS__)
#define die()       do { fprintf(stderr, "%s; aborted\n", strerror(errno)); exit(1); } while (0)
#define zdie()      do { fprintf(stderr, "%s; aborted\n", Z.msg); exit(1); } while (0)

int main(void) {
    unsigned char *buf, *outbuf, *p;
    int fd;
    FILE *fp;
    z_stream Z = {0};
    unsigned int nin = 0, nout = 0;

    E("size = %lu\n", (unsigned long)SIZE);

    E("open /dev/amrd0... ");
    if (-1 == (fd = open("/dev/amrd0", O_RDWR | O_DIRECT)))

    E("allocate file buffer... ");
    if (!(buf = malloc(SIZE)))

    E("allocate write buffer... ");
    if (!(outbuf = malloc(1024 * 1024)))

    E("lock into memory... ");
    if (-1 == mlockall(MCL_CURRENT | MCL_FUTURE))

    E("open file... ");
    if (!(fp = fopen("/usr/bitter-first-2100M-of-sda.gz", "rb")))

    E("read file... ");
    p = buf;
    while (nin < SIZE) {
        size_t n;
        n = fread(p, 1, 262144, fp);
        if (n == 0)
        nin += n;
        p += n;
        E("\rread file... %.2f%% ", 100 * (float)nin / (float)SIZE);

    E("zlib version = \"%s\"\n", zlibVersion());

    /* Now we need to walk through the buffer decompressing it into the
     * write buffer, then writing the results to the device. */
    E("initialise inflate object... ");
    Z.next_in = buf;
    Z.avail_in = SIZE;
    if (Z_OK != inflateInit2(&Z, 15 + 32))

    while (nout < 2100) {
        int i;
        size_t N;

        Z.next_out = outbuf;
        Z.avail_out = 1024 * 1024;
        i = inflate(&Z, 0);
        if (i != Z_OK && i != Z_STREAM_END)
        if (Z.next_out != outbuf + 1024 * 1024) {
            fprintf(stderr, "\ndidn't get 1MB of output\n");

        /* this is where we'd write the data */
        N = 0;
        p = outbuf;
        while (N < 1024 * 1024) {
            ssize_t n;
                n = write(fd, p, 1024 * 1024 - N);
            while (n == -1 && errno == EINTR);
            if (n == -1)
            N += n;
            p += n;

        fprintf(stderr, "\r%d / 2100 MB", nout);

    fprintf(stderr, "\n");

    /* this is where we reboot */

    E("we should have rebooted by now -- probably best to assume we're completely\n"
      "screwed at this point\n");

    return 0;

Tax needn’t be taxing thanks to TaxCalc

February 1st, 2017 by

One of our customers has the lovely looking bandwidth graph on the right which plummeted to zero this morning. Normally a huge sudden drop in activity on a customer site would be cause for alarm but this is the excellent TaxCalc, they do software to calculate tax and it gets very busy in the run up to the deadline for self assessment at midnight last night.

As customers of our enhanced management services with a 24/7 SLA who run a fully mirrored setup across two of our data centres, we’re happy to report that everything went smoothly and their system scaled beautifully to handle the load.

Thankfully our elected overlords have decided to smooth out the load on our servers with new personal tax accounts and shortly we’ll all have to fill in four tax returns per year instead of one.

Don’t leave your laptop in the pub

December 9th, 2016 by

After about twenty pages of awesome beers, you discover they also have mead.

After about twenty pages of awesome beers, you discover they also have mead.

Last night we had our Christmas party. For a 24/7 operation, that means we have to have at least one laptop with us at the party. We had just one urgent customer issue which we dealt easily without ruining the night.

However, in addition to taking your laptop to the pub, Pete would like to remind everyone that it’s equally important to remember to take your laptop home from the pub too, as he didn’t. This means we have to have a brief security review to evaluate the risks of briefly losing a company laptop. Ten years ago when we had tens rather than thousands of servers, this would have resulted in a revocation and replacement of the company ssh key on every server under emergency conditions (and those of you with an unencrypted AWS key might worry about total company deletion).

Over the past decade we put more effort into improving our security. The laptop contains an encrypted filesystem, on that filesystem is an encrypted ssh key which will allow someone into our jump box. If they’ve worked out the password for the filesystem, and the password for the ssh key,they then also need to guess password on the jump box before they would be able to access customer or company systems. That’s three different passwords to guess, or two encryption breaks and one password to guess. The passwords are not chosen by the user, the come straight from pwgen and the random number generator. Whilst we’re not worried, we’ll do some extra monitoring the logs on the jump box for attempts on Pete’s account.

Of course there’s also a risk that someone physically tampered with the hardware to install a key-logger in between leaving it in the pub and recovering it the next day. The laptop passes a brief physical inspection. If it has been tampered with, it has been tampered with very well. If our attacker was sat in the pub with a spare key logger kit just in-case the laptop was left behind, it would have been easier and cheaper to stage a break-in at an employees house, or to have forced them to check their hand luggage on a flight, or to have installed the key logger before the laptop was bought, or maybe to have compromised the random number generator in any or all of our servers before they were bought. So our threat model remains relatively unchanged and we don’t think we’re under significantly more risk today than we were yesterday.

On the upside, the server room isn’t on fire.

December 8th, 2016 by
This is not the correct way to mix servers and water based fire suppressant.

This is not the correct way to mix servers and water based fire suppressant.

One of our customers does embedded development and have some custom servers in their office as a build platform. This is hardware specific to the embedded designs they’re working on and they can’t locate it in a data centre as they require regular human attention. Unstable development drivers cause crashes and the root flash filesystems need to be re-imaged and replaced.

Recently they’ve moved office and their new office has a ‘server room’, ideal for putting their very expensive custom kit in, and a handful of other machines that they keep locally in the office. While doing the fit out, they noticed that their ‘server room’ is attached to the main sprinkler system. A fire in the building and whilst the bread may be saved from being overly toasted, their expensive hand built development boards are drowned.

They raised this with their landlords who billed them the best part of a thousand pounds to resolve the problem, see the picture on the right.

I’m not sure if it’s the belief that the plastic roof will help, the combustible struts to hold it up or the lack of guttering that really emphasises the mismatch between what a landlord things a server room looks like and what a real data centre actually provides.

We’re in further discussions to see if we can host their custom kit too, because our server room has non computer damaging halon as a fire suppressant and we will return the servers to them unwashed. If your office server room looks like this, please get in touch at

I know that I know nothing

May 13th, 2016 by
Over a thousand people put together 43000 packages which forms the universal operating system.

Over a thousand people put together 43000 packages which forms the universal operating system.

One of the hazards of going to the pub in Cambridge is that very smart people will occasionally ask you difficult questions. Steve McIntyre, a former Debian Project leader asked our advice as to how Debian should specify a new central build server. Did we think that they’d be best off with lots of RAM or fast SSD, was PCI-E attached SSD better than SATA SSD or even sticking with cheap, slow but very large spinning hard disks.

On the unusual occasions we build software it completes very quickly, and for any big complicated package we’d just install the binary package from Debian. Advice that is almost always spot on, unless you are Debian attempting to make the binary packages in the first place.

We thought about this for a short time and proclaimed confidently that we didn’t know the answer.

However in Mythic Beasts we have a very strong science background. We suggested the right plan was to test it, take a big machine with multiple types of storage, disk, SATA attached SSD and PCI-E flash and try it out. Shortly afterwards our brain kicked in and realised that this looked just like the new VM hosts we were commissioning and with only a slight rearrangement to our plans we could lend one to Debian. Some weeks of work later and the answer is that an SSD makes a huge difference for the working filesystem, otherwise it doesn’t matter.

Without the hot air

April 15th, 2016 by

It’s with great sadness we learned of the death of Prof Sir David MacKay, FRS. He taught three of the Mythic Beasts founders information theory in 1999–2000, a fascinating and stunningly well-lectured course. The textbook Information Theory, Inference, and Learning Algorithms is freely available to download. Prof MacKay believed it was possible to make the world a better place.

Energy policy done by the numerate.

David’s other textbook Sustainable Energy, is not only fascinating, but vital reading for anyone interested in energy policy. It’s excellent, not because it provides answers, but because it teaches the tools to work things out. Can we power the UK only from wind energy? No. Can it make a substantial contribution? Yes, If we built lots and lots of wind how much can it contribute? Hopefully around 25% if you include a lot of offshore.

If more people read and understood the book we’d have fewer articles about the amazing solar panels that create power from rain, when the abstract tells us they create micro amps and milli-volts, throwing away 180W/m2 of solar to create 0.000000001W/m2 from rain.

David’s website is prepaid far in advance and will remain available for many years to come.

The problem with the youth of today is hard work and excessive talent

February 1st, 2016 by

Almost a decade ago in 2006 Eben showed off his Raspberry Pi protoype with the original aim being to improve the standards of computer science for Cambridge University Computer Science students. Four years ago the Raspberry Pi launched, and the first post-pi generation of students are now in their final year. Today I went back to the Computer Science department to deliver a lecture for their e-commerce course on web application architecture and scaling. Several of the examples were taken from scaling of the Raspberry PI website – a website literally built with the aim of educating the students sat directly before me.

One students in the room was Will Shackleton. Will came over after the lecture to show me one of his projects, he’d taken my original Upside-down-ternet project for messing with my neighbours who stole my wifi, and turned it into an android app, Network Spoofer that you can use on wifi networks.

His first implementation embedded a Debian chroot with my original terrible perl script and squid before he reimplemented it properly with a proper proxy and parser to enable many more features and cleaned it all up. We discussed some issues – e.g. evil hosting providers like us enabling free SSL which prevents intercepting connections, how a sense of ethics is important – Will has refused to implement SSL strip – and I gave my sketch idea of how to do an IPv6 gateway version that wouldn’t require arp spoofing.

So The Youth of Today are taking the things we built, cleaning up the messes we left behind and making things better.

PiZero won!

December 10th, 2015 by
Pi Zero, 2p for scale, not included with prize.

Pi Zero, 2p for scale, not included with prize.

Last week we started a competition to win a Pi Zero. We’ve had a small number of entries, half from school age people eligible to win, the other half from entertained techies. We’ve also been using this as a job filter for some time so we have a rich depth of existing answers.

The competition is very simple, our web-page generates a mathematical sum for you to work out the answer to, and in order to succeed you have to send us the answer within one second. It’s an anti-turing test – a person can’t do it but a computer can. When you’ve succeeded we ask you to send us the code. This gives two important things, a code sample that a candidate wrote and an idea of how long it took them to work out they should automate it.

A text-book answer from an experienced techie is about 15-30 minutes and delivers a short piece of code that fetches the web-page, finds the sum with a regexp or similar, works out the answer with eval, creates the response and sends it back to us. However, people are much more creative than that.

One excellent answer was a single line of shell script, which did the whole thing in bash, but even more cleverly searched the process list for the command that had been typed in, added the escaping back in and then mailed itself to us – a one liner that did the puzzle, recreated its source code and emailed itself to us.

Another excellent answer was someone who guessed that our code had a 32 bit roll-over bug in, tried options until we generated an answer bigger than 2^32 and fixed the otherwise text-book code to implement the bug on our side.

The absolute worst answer we’ve ever seen was someone whose CV listed them as a professional programmer with five years experience. After two whole days of typing answers into the website they finally worked out that the problem had to be automated. After three days of development they wrote a vast piece of Java code that was able to download the page and find the problem but it was only able to work out the answer if there were only three numbers and they all had to be added together. Instead of improving the code for the general case they put it in a loop and repeatedly called the page until by sheer luck they got a page their code could answer. Creative genius but not in a good way.

On to the entrants

So this is a difficult challenge for school age children and teenagers. Most of the entries came from older children 16 and up, and it’s clear that it was difficult and they had to learn new things specifically to solve this. PHP and Python were the preferred languages – the most novice friendly of all the tools available. We were very torn as to who should win. After lots of deliberation our runner up is this php entry from Nils in Germany who was also the first to submit,

// That was fun. There should be more conetsts like this.
// Sorry for the incredibly hacked together code...

$challenge =

$task = preg_replace(‘/.*

Evaluate /is’, ”, $challenge);
$task = preg_replace(‘/ \, .*/is’, ”, $task);

$id = preg_replace(‘/.*name=\”id\” value\=\”/is’, ”, $challenge);
$id = preg_replace(‘/\”.*/is’, ”, $id);

function calc_string( $mathString ){
$cf_DoCalc = create_function(“”, “return (” . $mathString . “);” );

return $cf_DoCalc();

$ans = calc_string($task);

$url = ‘’;
$data = array(‘answer’ => $ans, ‘id’ => $id);

$options = array(
‘http’ => array(
‘header’ => “Content-type: application/x-www-form-urlencoded\r\n”,
‘method’ => ‘POST’,
‘content’ => http_build_query($data),

$context = stream_context_create($options);
$result = file_get_contents($url, false, $context);



Things we particularly like are that all the comments and code and email are in English to make it easier for us, even though it’s the authors second language.

Our winner though goes to 13 year old Nick. From a pure technical standpoint his code isn’t as good as Nils’ entry above, but through the comments it tells the story of slowly working his way towards the solution and appropriately credits the help he received – both personal and online.

#Mythic Beasts - Win a Pi Zero
#Written by Nick Lockhart from Chepstow, Wales, aged 13 3/4
#with help from an old fossil (his dad)
#You will need LXML for Python and Requests

from lxml import html
import requests
debugmode = 1

#get the page and parse all

elements into a Python list.
#For this purpose we should only get one element.
page = requests.get(‘’)
tree = html.fromstring(page.content)
paragraphs = tree.xpath(‘//p/text()’)

#Split out the question.
#First take out everything after the sum (space included)
#And then remove everything before the sum (space again included.)
#And convert to a string. Oddly, after the second time, we have to
reference the second element as there is a blank string in question[0]
#Finally, evaluate it.

question = paragraphs[0].split(‘ ,’)
question = question[0].split(‘Evaluate ‘)
question = question[1]
answer = eval(question)

if (debugmode == 1):
print(“The question was: ” + question)
print(“The answer should be: ” + str(answer))

#There’s a hidden input labeled “id”, which seems to be randomly generated.
#This is probably to track who’s submitting.
#We will also need to extract this.
#I learnt this piece of magic with help from StackOverflow. Thanks,
Mathias Muller!

secretkey = tree.xpath(‘//input[@name=”id”]/@value’)[0]

#That’s all we need to POST. Let’s generate a payload, send it off and
extract the response.
#The server expects the answer to be a string, so we convert it to a string.

payload = {“answer”:str(answer),”id”:secretkey}

if (debugmode == 1):
print payload

response =“”,
responsetree = html.fromstring(response.content)

Of course the final comment to everyone who entered is if you ever need any kind of hosting, domain name or similar send us an email and include your entry number for a freebie / upgrade /discount. Secondly if you seek summer work or gap year employment, we’d invite you to get in touch and we guarantee that we’ll read your CV and take your application seriously.

Rebuilding Software RAID 1 refused to boot

October 30th, 2015 by

Dear LazyWeb,

Yesterday we did a routine disk replacement on a machine with software RAID. It has two mirrored disks, sda and sdb with a RAID 1 partition with software RAID, /dev/md1 mirrored across /dev/sda3 and /dev/sdb3. We took the machine offline and replaced /dev/sda. In netboot recovery mode we set up the partition table on /dev/sda, then set the array off rebuilding as normal:

mdadm --manage /dev/md1 --add /dev/sda3

This expects to take around three hours to complete, so we told the machine too boot up normally and rebuild in the background while being operational. This failed – during bootup in the initrd, the kernel (Debian 3.16) was bringing up the array with /dev/sda3, but not /dev/sdb3, claiming it didn’t have enough disks to start the array and refusing to boot.

Within the initrd if I did:

mdadm --assemble /dev/md1 /dev/sda3 /dev/sdb3

the array refused to start claiming that it didn’t have sufficient disks to bring itself online, but if I did:

mdadm --assemble /dev/md1 /dev/sdb3
mdadm --manage /dev/md1 --add /dev/sda3

within the initrd it would bring up the array and start it rebuilding.

Our netboot recovery environment (same kernel) meanwhile correctly identifies both disks, and leaves the array rebuilding happily.

To solve it we ended up leaving the machine to rebuild in the network recovery mode until the array was fully redundant at which point the machine booted without issue. This wasn’t an issue – it’s a member of a cluster so downtime wasn’t a problem – but in general it’s supposed to work better than that.

It’s the first time we’ve ever seen this happen and we’re short on suggestions as to why – we’ve done hundreds of software RAID1 disk swaps before and never seen this issue.

Answers or suggestions in an email or tweet.