Last night we quietly upgraded the disks in our Sphinx shell server to a pair of SSD drives. Sphinx has been suffering under heavy I/O load for a while now, and it’s safe to say that the SSDs have resolved that problem for the foreseeable future.

The upgrade was without downtime, using the magic of LVM’s pvmove command.

It’s been upgraded with a pair of fiendishly expensive server-grade SSDs. We’re not normally ones to pay too much attention to whether kit is designated as “server-grade” but in the case of SSDs it really matters due to the limited number of write cycles on SSDs. The new disks are good for 8TB of writes per day for 5 years, whereas the equivalent consumer grade version is only rated for 20GB/day, which wouldn’t last very long in Sphinx.

Sphinx has a special place in our hearts as it’s the machine on which the company was founded nearly 14 years ago, and it’s been in pretty much continuous service ever since. Of course, the current hardware has absolutely nothing in common with the dual Celeron BP6 that we deposited in a Fulham datacentre back in 2000, and it now lives in Docklands, but it’s still the same machine (right?) which is why it still says:

[pdw@sphinx ~]$ rpm -q redhat-release
redhat-release-6.1-1

(don’t worry, that’s probably the only package from RH 6.1 that we’re still using…)

We’ve just extended our range of Virtual Servers to include a 512MB “VS Lite” option for just £7 + VAT per month – or the equivalent of £5.83/month if you pay annually.

We’ve been doing Virtual Servers for 10 years now, and we’ve always hosted them on machines with hot-swappable hardware RAID, meaning that we can replace failed disks without customers even being aware that there was a problem. One of the great things about virtualisation is that reliability shares really well – but it still adds a bit to the cost

A large proporption of our Dedicated Server customers use machines without hot-swappable drives, and we thought we’d extend the option of using cheaper hardware to our Virtual Server range in the form of the VS Lite, our most affordable Virtual Server yet.

The VS Lite host hardware still has monitored RAID, so we can cope with disk failures without data loss^*, but it requires downtime in order to replace the drive.

Our standard Virtual Servers start at £12.50 + VAT per month for a 1GB server, and all run RAID-10 on hot-swappable drives, and offer a choice of data centres, allowing you to spread multiple servers across geographically diverse sites.

In the unlikely event that even £7/month sounds expensive, then we’ve got an even more affordable alternative with a bit of a twist up our sleeves which we’ll be unveiling shortly. The first person to guess what we’re planning can have one free for a year, and don’t worry, as with all our Virtual Servers you can upgrade and downgrade easily, so don’t be put off by the prospect of an even better offer in the pipeline!

On the other hand, there are no prizes for guessing what hardware we’re using for the VS Lite host servers…

* RAID is not backup.

Last year we announced some improvements to the Mythic Beasts DNS API, and I asserted that this made it good for implementing a Dynamic DNS service. Dynamic DNS is simply a mechanism for programmatically updating a DNS record, typically used to provide a consistent name for a computer that is at the end of an internet connection with a dynamically assigned IP address.

Well, last weekend I had the opportunity to try implementing a Dynamic DNS service with our API, and realised that it actually makes the task unduly difficult. It can be done, but in order to change a record, you need to remove the old record, and to remove the old record you need to know what it is currently. This meant that you had to use the LIST command, grep out the old record, and then issue the necessary DELETE and ADD commands. Aside from being hassle, it introduces an unavoidable race condiition between the LIST and DELETE commands.

We’ve now implemented the obvious fix: a REPLACE command, which replaces all existing records for the specified host and type, and replaces them with the one provided. Obviously this doesn’t work if for some reason you want multiple records for a single host, but for the obvious use case it means that Dynamic DNS can be handled in a single command:

curl --data "domain=MY_DOMAIN&password=MY_PASSWORD&command=REPLACE \
myhost 300 A 1.2.3.4" https://ctrlpanel.mythic-beasts.com/customer/primarydnsapi

The DNS API is a standard feature included with all Mythic Beasts domain registrations. Full documentation can be be found here.

Today we have made a number of changes Onza and Yali.

• Each user can now read their own web logs, which are stored in /home/user/www/log
• PHP scripts no longer need the execute bit set to run
• All scripts in cgi-bin directories have been moved to /home/user/cgi-bin to conform with Apache’s security recommendations

We hope that these changes will make it easier for people to run and debug their code.

A couple of weeks ago, Apple unveiled the latest incarnation of the Mac mini.  Naturally, we dashed out to buy a few to see if they’re going to be any good as servers.  Externally, this is the biggest revision of the Mac Mini yet, with a thinner all-aluminium case.  We always get a bit nervous when Apple unveil a new Mac mini as there’s a chance that they’ll ruin the formula that make it such a great server in the name of creating a fun toy for your living room.

The most noticeable change, aside from the new case, is the removal of the power brick.  The old minis relied on an external power supply that really was the size and shape of a brick.  Getting rid of these will make racking them a lot simpler, as well as saving space.  We reckon we should be able to get 24 machines and 3 APC Masterswitches into 6U of rackspace.  The C7 power connectors should be more secure too.

The next thing that we approve of is the easily accessible RAM slots.  Traditionally, Apple have charged silly money for ordering machines with extra RAM, so we’ve always done the upgrades ourselves.  Upgrading the old minis was a real pain, requiring the cover to be prised off with putty knives, so this is a welcome change.

The most important factor for us, and the thing that lead us to use the original PPC minis, is power consumption, as power is the primary cost when hosting a machine.  There’s good news here too: according to our power meter, power consumption is down from 26W to 12W (idle) and 44W to 40W (max).

Unfortunately, the reduction in hosting cost is offset somewhat by they usual price hike for the newer hardware, but we’re pleased to see that Apple have retained the “server” version of the mini with two hard drives, allowing us to continue to offer machines running software RAID.

Overall, the new Macs seem like a decent improvement on the old ones for our purposes.  Apple make a fuss over the great variety of ports now available on the back of the new mini, but inexplicably they haven’t provided the one thing we’d want: a good old-fashioned serial port.

We still need to make the minis work with our custom net-booting bootloader, but once that is done, we’ll be offering them as dedicated servers.

It’s been nearly six years since we first launched our Virtual Dedicated Servers.  At the time, the choice for virtualisation technology was easy: User Mode Linux.  Initially, UML was a well supported option, with UML patches being incorporated into the Linux kernel.  Over time, we’ve been following the development of other technologies such as Xen and KVM and at the end of last year we concluded that we should make the switch to Xen.

Getting Xen working reliably with our server management code has taken somewhat longer than expected, but we’re pleased to announce the the service is now live.

We’ve also taken the opportunity to roll out new hardware, allowing us to offer substantially higher specced VDSs for the same prices, with our base machine now coming in at 256MB RAM, 20GB, and an increased bandwidth allowance of 100GB/month for £15/month including VAT – less if you pay annually.

Although we’ve changed the virtualisation platform, we’ve retained the other key features of our virtual servers including:

Host servers with hot-swap hardware RAID.  Although these are significantly more expensive, we figure that reliability is something that can be shared particularly effectively through virtualisation: over the years, our VDS host servers have seen a fair few disk failures and replacements, but typically our customers don’t even know that they’ve happened.

Nightly backup to other host servers, allowing us to resume service quickly in the event of a serious hardware failure.

As part of the upgrade, we’ve also deployed a new approach to providing disk images which offers significantly better IO performance than the standard approach of storing the VDS filesystem as a file on the host filesystem.

The new VDSs are available now, and we’ll be contacting all existing customers in the near future to arrange migration to the new platform.