Rebuilding Software RAID 1 refused to boot

October 30th, 2015 by

Dear LazyWeb,

Yesterday we did a routine disk replacement on a machine with software RAID. It has two mirrored disks, sda and sdb with a RAID 1 partition with software RAID, /dev/md1 mirrored across /dev/sda3 and /dev/sdb3. We took the machine offline and replaced /dev/sda. In netboot recovery mode we set up the partition table on /dev/sda, then set the array off rebuilding as normal:

mdadm --manage /dev/md1 --add /dev/sda3

This expects to take around three hours to complete, so we told the machine too boot up normally and rebuild in the background while being operational. This failed – during bootup in the initrd, the kernel (Debian 3.16) was bringing up the array with /dev/sda3, but not /dev/sdb3, claiming it didn’t have enough disks to start the array and refusing to boot.

Within the initrd if I did:

mdadm --assemble /dev/md1 /dev/sda3 /dev/sdb3

the array refused to start claiming that it didn’t have sufficient disks to bring itself online, but if I did:

mdadm --assemble /dev/md1 /dev/sdb3
mdadm --manage /dev/md1 --add /dev/sda3

within the initrd it would bring up the array and start it rebuilding.

Our netboot recovery environment (same kernel) meanwhile correctly identifies both disks, and leaves the array rebuilding happily.

To solve it we ended up leaving the machine to rebuild in the network recovery mode until the array was fully redundant at which point the machine booted without issue. This wasn’t an issue – it’s a member of a cluster so downtime wasn’t a problem – but in general it’s supposed to work better than that.

It’s the first time we’ve ever seen this happen and we’re short on suggestions as to why – we’ve done hundreds of software RAID1 disk swaps before and never seen this issue.

Answers or suggestions in an email or tweet.

If you put your mind to it

October 21st, 2015 by



With today being Back To The Future day, it’s worth reflecting on two pieces of advice I received in the mid 1980s. The best piece of advice was definitely from the film:

‘If you put your mind to you, you can accomplish anything’.

the worst was my mother:

‘Stop playing on your Spectrum and go and do your piano practice’

I’m not certain this generalises, I think that largely your parents do give better advice than Hollywood script writers.

Professor Cathie Clarke, Ada Lovelace day

October 13th, 2015 by

Unless you really like maths, you’re probably better off just looking at the pictures.

Today is Ada Lovelace day, where we celebrate the achievements of women in the traditionally male dominated fields of Science, Technology, Engineering and Maths.

Mythic Beasts came about as a side project of a bunch of students, most of whom studied at Clare College Cambridge. As a Cambridge student you receive supervisions, hour long tutorials in sets of two or three. You also live in the college for three years and some fellows of the college also have rooms in the same accommodation. As luck would have it, our director Pete was partnered with one of our other founders, Richard for supervisions in Physics, and in the second year they were jointly supervised by Professor Cathie Clarke and it turned out that her college room was directly opposite Pete’s in Clare Old Court.

This led to a slightly unusual arrangement, rather than everyone trekking into the department for supervisions they decided to hold them in Pete’s room at 8am, usually accompanied by a bacon sandwich and strong black coffee before heading off to lectures. Cathie was a superb teacher, neatly covering dynamics, orbits and effortlessly showing why practically the whole of spaceflight involves pointing your rocket motor in obviously the wrong direction in order to get to where you want to go. She also, neatly answered other questions including electro-magnetism that had stumped pretty much the whole year in Clare and all the supervisors in that subject, despite having had about sixty seconds notice and only half a cup of coffee.

Pete’s room was on the top right hand corner of this photograph, the bacon cooking kitchen roughly above the passageway.

There was a particularly memorable supervision, where Richard overslept and arrived very late to Pete’s room, anxious he’d missed the supervision. On waking Pete up, they jointly discovered the note on the door from Cathie apologising that she’d overslept and the supervision would need to be rescheduled. A co-incidence for which all parties were grateful.

So whilst her impressive CV gives away the huge publication list, professorship and that she’s the course coordinator for astrophysics; in person we had the privilege of knowing that she is also a superb prize-winning teacher, gifted researcher and somehow on top of all that, a lovely human being who occasionally oversleeps just like the rest of us.

Happy Incorporation Day to Us

August 14th, 2015 by

Happy Incorporation Day to Us
Happy Incorporation Day to Us
Happy Incorporation Day to Mythic Beasts
Happy Incorporation Day to Us

Fifteen years ago today someone with a boring job processed the paperwork and Mythic Beasts Ltd sprang into existence as a legal entity.

We had existed informally for a bit longer than that, we had registered the mythic-beasts.com that April, and our shell server, sphinx, had been running for a while, although it wouldn’t be until early 2001 that we sent our first invoice.

As we all work on t’internet, it’s difficult to all meet in the same pub this Friday for a celebratory drink. That will have to wait until our next full company meetup in September. Slamming the bedroom door, staying in and watching Brazil seems a more apt way for a teenage company to celebrate its bureaucratic anniversary.

Women in Technology and avoiding ISP filters

June 29th, 2015 by

One of the Mythic Beasts, Rhosyn has written an article on filtering for trefor.net, a widely read blog on technology and networking.

The part we particularly like this this quote,

As a long standing customer of Mythic Beasts 
(shameless plug; outstanding service and support, 
so good that I switched companies recently to work 
for them)

The hazards of 301 (permanent) redirects

June 15th, 2015 by

When you visit a web page, you’ll often see the URL change as it loads.  For example, if you attempt to visit http://mythic-beasts.com you’ll end up at https://www.mythic-beasts.com .   This is achieved using HTTP redirects, a response from a server that tells your browser that the page it is trying to load has moved.

HTTP redirects come in two flavours:

Permanent (301)
This tells the client that the page requested has moved permanently, and crucially, if it wants to load the page again, it needn’t bother checking the old URL to see if the situation has changed. This is a good way of redirecting something that you never want to undo, for example, if you’re permanently moving a website from one domain to another.
Temporary (302)
As the name suggests, this tells the client that the page has moved, but only temporarily, so the client should continue requesting the old URL if it wants to load the page again. This is a good way of telling users that your site is down for maintenance, that they they don’t have enough credit to access a site, or of some other issue that is likely to change.

#makeitwrong

three-301-fail

Getting this wrong can be a massive pain for your users. For example, Three use a permanent redirect if you’ve run out of credit on your data plan, or you’re trying to use tethering in the wrong country, or some other temporary problem.

So imagine what happens when you run out of data on your plan. You attempt to visit your favourite website, say, http://www.xkcd.com . Three tell you that that page has been replaced by http://tethering.three.co.uk/TetherNoProductPost. Permanently.

Now find a working internet connection, attempt to load http://www.xkcd.com, and find that your browser quite reasonably takes you straight to the Three fail page, even if you’re no longer using a Three connection. Shift+Reload doesn’t help, even restarting your browser may not help.

Three have told your browser that every page you visited whilst out of credit has moved permanently to their fail page.

Expiring permanent redirects

The example given above is very obviously a place where a temporary 302 redirect should be used, but webmasters are often encouraged to prefer 301s in the name of improving search rankings. 301 redirects allow you to tell search engines that your .co.uk site really is the same site as your .com site, thus accumulating all your google juice in the right place. They also save a small amount of time in loading the page by avoiding an unnecessary HTTP request.

Even when used legitimately, 301 redirects are obviously hazardous, as there’s no way to undo a permanent redirect once it’s been cached by a client.

The safe way to do a 301 redirect is to specify that it will expire, even if you don’t expect to ever change it. This can be done using the Cache-Control header. For example, the redirect that we issue for http://mythic-beasts.com includes the following header:

Cache-Control: max-age=3600

This tells clients that they can remember the redirect for at most one hour, allowing us to change it relatively easily at some point in the future. We use the mod_expires Apache module to create this header, which also produces an equivalent “Expires” header (the old HTTP 1.0 equivalent of Cache-Control).

.htaccess example

The above can be implemented using a .htaccess file as follows:

ExpiresActive on
ExpiresDefault "access plus 1 hour"
Redirect 301 / https://www.mythic-beasts.com/

This example uses mod_alias and mod_expires which may need enabling globally in your web server. In Debian, Ubuntu and similar distributions, this is done by running the following command as root:

a2enmod alias expires

mod_rewrite example

Redirects are often implemented using Apache’s mod_rewrite. Unfortunately, mod_expires doesn’t apply headers to RewriteRules, but mod_headers can be used instead:

RewriteRule ^.* http://www.mythic-beasts.com/ [L,R=301,E=limitcache:1]
Header always set Cache-Control "max-age=3600" env=limitcache

The RewriteRule is used to sent an environment variable which is used to conditionally add a Cache-Control header. Thanks to Mark Kolich’s blog for the inspiration.

Again, you may need to enable mod_rewrite and mod_headers on your web server:

a2enmod rewrite headers

Escaping 301 hell

Fortunately, if you’re unlucky enough to get caught by a broken 301 redirect, such as the one issued by Three, there is an easy way to get to the page you actually wanted: simply append a query string to the end of the URL. For example, http://www.xkcd.com/?foo=bar. Browsers won’t assume that the cached redirect is valid for this new URL and websites will almost always ignore unexpected query parameters.

2015-07-03 – Updated to add mod_rewrite example
2020-03-16 – Updated to note that the relevant modules may need enabling

A non-party political broadcast from Mythic Beasts

May 6th, 2015 by

Here at Mythic Beasts it’s fair to say that our staff hold a wide spectrum of political beliefs, but I think one thing we can all agree on is that all the major political parties have at least some irredeemably stupid policies (and possibly also that some of the minor parties only have stupid policies).

This makes voting for a political party a pretty depressing prospect. So, what about voting for an elected representative who will look after our interests?

Our founders reside in two constituencies with notable MPs: Witney and Cambridge.

The MP for Witney is notable for being the Prime Minister. The MP for Cambridge, Julian Huppert, is notable for being a Liberal Democrat and yet still being highly regarded by a large number of his constituents.

Now, if you want good data on whether your MP is any good or not, you should head over to the excellent They Work For You and find out what they’ve been up to in Parliament on your behalf.

But who wants good data when you can have some anecdotes? Let’s look at two issues that have got us wound up recently.

Firstly, the EU VAT MESS, which causes us an administrative burden far in excess of the value of the affected revenue.

Julian Huppert was very active on behalf of the constituents who contacted him on this issue (Mythic only got as far as a tweet…), including submitting written questions in parliament, which received a predictably useless response.

On the other hand, Paul wrote to David Cameron twice (the first letter went AWOL), and received only a hopeless response which completely failed to address any of the issues raised.

Secondly, banning secure encryption. As a hosting company, the ability to undertake transactions securely online is quite important to our everyday business (see previous notes).

The appalling jeering by other MPs, and the pathetic response given by Theresa May, to Julian Huppert’s questions asked in Parliament demonstrated the he was clearly one of the few MPs who actually grasped the implications of the proposal, rather just resorting to rhetoric that fuels the fear that terrorism relies on.

As for David Cameron, well, it’s his idea.

So what can we conclude from this? Not a lot, except that we’d probably be in a far better place if parliament were full of representatives who listened to and understood their constituents, rather than those who get in on the strength of a party political vote.

UKNOF31

April 22nd, 2015 by

At UKNOF31 we presented a talk entitled Catastrophic Unplanned Success, a slightly rushed history of how some of the rapid scale-up of RaspberryPi from the point of view of the hosting provider, detailing some of the issues we’ve dealt with during their extremely rapid scale up, and attempting to educate the teenagers into a proper DDoS rather than the half-hearted ones they’ve tried so far.

https://indico.uknof.org.uk/getFile.py/access?contribId=5&resId=0&materialId=slides&confId=33

We believe this talk was videoed, we’ll put the video up here too once it’s published.

Helping RachelPi

March 4th, 2015 by

Some time ago we were forwarded a plea by Liz Upton who’s sort of famous on the internet for some sort of cheap computer, on behalf of World Possible, which said

This brings us to good news / bad news.  Last month we pushed through 5TB of
FTP traffic, and over 20TB of FTP traffic on the year.  That's great, about
700 RACHEL downloads - but our web host isn't as excited with our success
and cut us off yesterday.

Liz thought this was the sort of thing we might be able to help with. So we got in contact and we’ve set them up with one of our older inexpensive servers to act as a new host. As it’s an educational project that we’d like to support; we thought we’d donate some bandwidth to help out. Since it nicely coincided with a substantial bandwidth upgrade in our Cambridge data centre we’d put the service there.

So far they seem pleased!

which is handy because some of their other suppliers who pay Amazon rates for bandwidth were a little bit annoyed with them.


Of Raspberries and Reptiles

February 17th, 2015 by

Steven Allain

On Sunday night Pete was in the Hopbine and while buying some drinks the bartender asked him about his Raspberry Pi t-shirt and if he knew anything about it. One of the hazards of drinking in Cambridge is the barstaff are often considerably more knowledgable than you might expect at first.

Steven not only sells beer but is also a student at ARU studying zoology and has been using a Raspberry Pi and camera to look into monitoring and photographing things under water with motion detection. He commented that he’d just bought a Raspberry Pi model B+ and only a couple of weeks later the much faster model 2 B had come out and he wished he’d bought one of those instead, but as an impoverished student he couldn’t really justify replacing it.

Now we think taking photographs of fish and reptiles is pretty cool, so Pete took pity on him and gave him his model 2 Raspberry Pi in exchange for a future promise of some photographs of underwater things taken with his setup.

Ultimately this gets back to the real reason Mythic Beasts support Raspberry Pi. Not because it makes it cheap to run a formal curriculum for teaching in schools, but because it’s a catalyst for people to teach themselves. Steven may or may not have success in making a motion detecting under water camera but either way he’ll learn a lot in the process.

The mistake in all this? Not checking the Raspberry PI stock levels and Pete realising it’s going to take a few weeks before the replacement model 2 arrives – he’s back to his old much slower model B+ now and grumbling about it.



We’ll settle for pictures of Sea Bass with frickin’ Laser Beams