Capacity upgrades, cheaper bandwidth and new fibre

December 8th, 2017 by

We don’t need these Giant Scary Laser stickers yet.

We’ve recently upgraded both our LONAP connections to 10Gbps at our two London POPs bring our total external capacity to 62Gbps.

We’ve been a member of LONAP, the London Network Access Point, since we first started running our own network. LONAP is an internet exchange, mutually owned by several hundred members. Each member connects to LONAP’s switches and can arrange to exchange traffic directly with other members without passing through another internet provider. This makes our internet traffic more stable because we have more available routes, faster because our connections go direct between source and recipient with fewer hops and usually cheaper too.

Since we joined, both we and LONAP have grown. Initially we had two 1Gbps connections, one in each of our two core sites. If one failed the other could take over the traffic. Recently we’ve been running both connections near capacity much of the time and in the event of failure of either link we’d have to fall back to a less direct, slower and more expensive route. Time to upgrade.

The upgrade involved moving from a copper CAT5e connection to optic fibre. As a company run by physics graduates this is an excellent excuse to add yet more LASERs to our collection. Sadly the LASERs aren’t very exciting, being 1310nm they’re invisible to the naked eye and for safety reasons they’re very low powered (~1mW). Not only will they not set things on fire (bad) they also won’t blind you if you accidentally look down the fibre (good). This is not universally true for all optic fibre though; DWDM systems can have nearly 100 invisible laser beams in the same fibre at 100x the power output each. Do not look down optic fibre!

The first upgrade at Sovereign House went smoothly, bringing online the first 10Gbps LONAP link. In Harbour Exchange proved a little more problematic.  We initially had a problem with an incompatible optical transceiver. Once replaced, we then saw a further issue with the link being unstable which was resolved by changing the switch port and optical transceiver at LONAP’s end. We then had further low level bit errors resulting in packet loss for large packets. This was eventually traced to a marginal optical patch lead. Many thanks to Rob Lister of LONAP support for quickly resolving this for us.

With the upgrade completed, we now have two 10Gbps connections to LONAP, in addition to our two 10Gbps connections into the London Internet Exchange and multiple 10Gbps transit uplinks, as well as some 1Gbps private connections to some especially important peers.

To celebrate this we’re dropping our bandwidth excess pricing to 1p/GB for all London based services.  The upgrades leave us even better placed to offer very competitive quotes on high bandwidth servers, as well as IPv6 and IPv4 transit in Harbour Exchange, Meridian Gate and Sovereign House.  Please contact us at sales@mythic-beasts.com for more information.

Posted in IPv6, Network, Services

Sender Rewriting Scheme

October 30th, 2017 by

tl;dr: SRS changes the sender address when you forward email so it doesn’t get filed as spam.

We’ve just deployed an update to our hosting accounts that allows you to enable Sender Rewriting Scheme when forwarding mail for your domain.

We’ve previously mentioned how we’re seeing increased adoption of Sender Policy Framework (SPF), a system for ensuring that mail from a domain only comes from authorised servers. Whilst this may or may not reduce spam, it does very reliably break email forwarding.

If someone at sender.com sends you an email to you at yourdomain.com and you forward it on to your address at youremailprovider.com, the email that arrives at your final address will come from the mail server hosting yourdomain.com which almost certainly isn’t listed as a valid sender in the SPF record for sender.com.  Your email provider may reject the mail, or flag it as “untrusted”.

To fix this, we need a different TLA: SRS, or Sender Rewriting Scheme. As the name suggests, this rewrites the sender address of a forwarded email, from one in a domain that you don’t control (sender.com) to one that you do (yourdomain.com).

In the example above, the actual rewritten address would be something like:

SRS0-9oge=B5=sender.com=them@yourdomain.com

This includes an encoded version of the original address, and any email sent to it will be routed back to the sender.  This means that any bounces messages will end up in the right place.

The sender and recipient in these examples refer to the “envelope” sender and receiver.  The addresses that are normally visible to users are the “from” and “to” headers, which may be different and are unaffected by sender rewriting.  Applying SRS should be invisible to the end users.

SRS is now available as an option whenever you create or edit a forwarder using the customer control panel for email accounts hosted on our main hosting servers.  If your account is hosted on sphinx, we need to do a little extra magic to enable it, so please email support.

Posted in Email, Services

Education, and the teacher becomes the student.

October 6th, 2017 by

Learn more about XSS with Google

For a long time we’ve sponsored Gwiddle, a project that outgrew its hosting on Microsoft Azure, providing free hosting accounts for students. They’ve now become a fully fledged charity, The Gwiddle Foundation, and we’ve had to upgrade the servers we donated to accommodate their ever expanding user base.

Part of their security team is the very talented Aaron Esau (15), who recently applied his penetration testing skills to our website and picked up a difficult to exploit bug.

On our page that allows you to search for domain names, our code embedded the search terms in the results page without appropriately escaping the content. This is a classic cross site scripting bug. Exploiting this bug was far from trivial, as the search term had to be short and from a restricted character set.

Aaron managed to craft an exploit using an ingeniously short payload to extract a session cookie and has posted a full write-up of the vulnerability and exploit.

If you had recently logged into our control panel, not logged out, and then visited a malicious page with this exploit, then the attacker could steal a cookie which would, in theory, give the attacker access to your control panel pages. However, we practise defence in depth, and our cookies are tied to an IP address so simply stealing the session cookie doesn’t give you access unless you also share a source IP address. This is an example where NAT and IPv4 is less secure than having IPv6.

Once Aaron brought the bug to our attention we swiftly fixed the page, thanked him for notifying us and sent him an Amazon voucher to thank him for his time and responsible disclosure.

We should emphasise that we do not believe that anyone has ever attempted to exploit this bug, and that the IP restrictions on session cookies mean that the consequences were fully mitigated.

Nonetheless, it’s embarrassing for us to have such a stupid bug in our code and we’ve been investigating how it occurred. It seems that the reason it crept in is because the domain ordering pages use a different form framework from everything else. Most of our pages have HTML generated by a template, and wherever dynamic data is included, it’s run through a filter to escape any HTML characters. The domain ordering pages use a different approach with much of the HTML being generated by a form module which we then include verbatim into our output. Obviously the HTML in this data mustn’t be escaped, as it would break the form; the form module is responsible for escaping any user input. Unfortunately, there are some other parts of the page which don’t come from the form module, and so do need to be escaped. It’s not very clear from the template code which is which, leading to the bug of not escaping some fields.

Posted in Security

CAA records

September 1st, 2017 by

A handful of the hundreds of different organisations, all of whom must be trustworthy.

Everybody knows that SSL is a good idea. It secures communications. At the heart of SSL is a list of certificate authorities. These are organisations that the confirm the identity of the SSL certificate. For example, if GeoTrust says that Raspberry Pi is Raspberry Pi we know that we’re talking to the right site and our communications aren’t being sniffed.

However, the list of certificate authorities is large and growing and as it stands, you’ve got to trust all of them to only issue certificates to the right people. Of course, through incompetence or malice, they can make mistakes.

CAA records are a relatively new mechanism that aims to stop this happening, making it harder to impersonate secure organisations, execute bank robberies and steal peoples’ identities.



CAA records enable you to list in your domain’s DNS the certificate authorities that are allowed to issue certificates for your domain. So, Google has a record stating that only Google and Symantec are allowed to issue certificates for google.com. If someone manages to persuade Comodo they are Google and should be issued a google.com certificate, Comodo will be obliged to reject the request based on the CAA records.

Of course, in order to be of any use, you need to be able to trust the DNS records. Fortunately, these days we have DNSSEC (dns security).

How does it work?

A typical CAA record looks something like this:

example.com. IN CAA 3600 0 issue "letsencrypt.org"

This states that only Let’s Encrypt may issue certificates for example.com or its subdomains, such as www.example.com.

Going through each part in turn:

  • example.com – the name of the hostname to which the record apply. In our DNS interface, you can use a hostname of “@” to refer to your domain.
  • IN CAA – the record type.
  • 3600 – the “time to live” (TTL). The amount of time, in seconds, for which this record may be cached.
  • 0 – any CAA flags
  • issue– the type of property defined by this record (see below)
  • "letsencrypt.org" – the value of the property

At present, there are three defined property types:

  • issue – specifies which authorities may issue certificates of any type for this hostname
  • issuewild – specifies which authorities may issue wildcard certificates for this hostname
  • iodef – provides a URL for authorities to contact in the event of an attempt to issue an unauthorised certificate

CAA records can be added using the new section at the bottom of the DNS management page in our control panel:

The @ in the first field denotes a record that applies to the domain itself.

At Mythic Beasts, we’re a bit skeptical about the value of CAA records. In order to protect against the incompetence of CAs, they rely on CAs competently checking the CAA records before issuing certificates. That said, they do provide a straightforward check that CAs can build into their automated processes to detect and reject unauthorised requests, so publishing CAA records will raise the bar somewhat for anyone looking to fraudulently obtain a certificate for your domain.

Posted in Domains, Security, Services

Raspbian Stretch now available in the Raspberry Pi Cloud

August 31st, 2017 by

A very short service announcement.

Raspbian Stretch is now available for Raspberry Pis hosted in our Raspberry Pi Cloud. This joins Raspbian Jessie and Ubuntu Xenial as available images. With all of these you can upload an SSH key through our control panel and log in directly. Re-imaging and rebooting can both also be done directly from our control panel.

Re-imaging your Raspberry Pi will reset the image and delete all data on your Cloud Pi.

On the server side the most significant upgrade is PHP 7, which should double the performance of PHP-based applications running on the Raspberry Pi.

Posted in Raspberry Pi

rm -rf /var

August 10th, 2017 by

Within Mythic Beasts we have an internal chat room that uses IRC (this is like Slack but free and securely stores all the history on our servers). Our monitoring system is called Ankou, named after Death’s henchman that watches over the dead, and has an IRC bot that alerts through our chat room.

This story starts with Ankoubot, who was the first to notice something was wrong with the world.

15:25:31

15:25:31 ankoubot managed vds:abcdefg-ssh [NNNN-ssh]: 46.235.N.N => bad banner from `46.235.N.N’: [46.235.N.N – VDShost:vds-hex-f]
15:25:31 ankoubot managed vds:abcdefg-web [NNNN-web]: http://www.abcdefg.co.uk/ => Status 404 (<html> <head><title>404 Not Found</title></head> <body bgcolor=”white”> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.10.3</center> </body> </html…) [46.235.N.N www.abcdefg.co.uk VDShost:vds-hex-f]

15:31:42

15:31:42pete I can’t get ssh in, I’m on the console.

15:38:16

15:38:16pete This is an extremely broken install. ssh is blocked, none of the bind mounts work

Debugging is difficult because /var/log is missing. systemd appears completely unable to function and we have no functioning logging. Unable to get ssh to start and fighting multiple broken tools due to missing mounts, we restart the server and mail the customer explaining what we’ve discovered so far. This doesn’t help and it hangs attempting to configure NFS mounts.

15:53:53

15:56:35

Boot to recovery media completes ready for restore from backup.

15:58:34

16:05:07

16:08:36

16:08:36 ankoubot managed vds:abcdefg-ssh [NNNN-ssh]: back to normal
16:08:36 ankoubot managed vds:abcdefg-web [NNNN-web]: back to normal

16:14:22

16:31:42

Customer confirms everything is restored and functional and gives permission to anonymously write up the incident for our blog including the following quote.

Mythic Beasts had come highly recommended to me for the level of support provided, and when it came to crunch time they were reacting to the problem before I’d even raised a support ticket.
This is exactly what we were looking for in a managed hosting provider, and I’m really glad we made the choice. Hopefully however, I won’t be causing quiet the same sort of problem for a looooong while.

In total the customer was offline for slightly over 30 minutes, after what can best be described as a catastrophic administrator error.

Posted in Services

FRμIT: Federated RaspberryPi MicroInfrastructure Testbed

July 3rd, 2017 by

The participants of the FRμIT project, distributed Raspberry Pi cloud.

FRμIT is an academic project that looks at building and connecting micro-data-centres together, and what can be achieved with this kind of architecture. Currently they have hundreds of Raspberry Pis and they’re aiming for 10,000 by the project end. They invited us to join them, we’ve already solved the problem of building a centralised Raspberry Pi data centre and wanted to know if we could advise and assist their project.  We recently joined them in the Cambridge University Computer Lab for their first project meeting.

Currently we centralise computing in data centres as it’s cheaper to pick up the computers and move them to the heart of the internet than it is to bring extremely fast (10Gbps+) internet everywhere. This model works brilliantly for many applications because a central computing resource can support large numbers of users each connecting with their own smaller connections. It works less well when the source data is large and in somewhere with poor connectivity, for example a video stream from a nature reserve. There are also other types of application such as Seti@Home which have huge computational requirements on small datasets where distributing work over slow links works effectively.

Gbps per GHz

At a recent UK Network Operator Forum meeting, Google gave a presentation about their data centre networking where they built precisely the opposite architecture to the one proposed here. They have a flat LAN with the same bandwidth between any two points so that all CPUs are equivalent. This involves around 1Gbps of bandwidth per 1GHz of CPU. This simplifies your software stack as applications don’t have to try and place CPU close to the data but it involves an extremely expensive data centre build.

This isn’t an architecture you can build with the Raspberry Pi. Our Raspberry Pi cloud is as about as close as you can manage with 100Mbps per 4×1.2GHz cores. This is about 1/40th of the network capacity required to run Google architecture applications. But that’s okay, other applications are available. As FRμIT scales geographically, the bandwidth will become much more constrained – it’s easy to imagine a cluster of 100 Raspberry Pis sharing a single low bandwidth uplink back to the core.

This immediately leads to all sort of interesting and hard questions about how to write a scheduler as you need to know in advance the likely CPU/bandwidth mix of your distributed application in order to work out where it can run. Local data distribution becomes important – 100+ Pis downloading updates and applications may saturate the small backbone links. They also have a variety of hardware types, the original Pi model B to the newer and faster Pi 3, possibly even some Pi Zero W.

Our contribution

We took the members of the project through our Raspberry Pi Cloud is built, including how a Pi is provisioned, how the network and operating system are provisioned and the back-end for the entire process from clicking “order” to a booted Pi awaiting customer login.

In discussions of how to manage a large number of Federated Raspberry Pis we were pleased to find considerable agreement with our method of managing lots of servers: use OpenVPN to build a private network and route a /48 of IPv6 space to it.   This enables standard server management tools work, even where the Raspberry Pis are geographically distributed behind NAT firewalls and other creative network configurations.

Donate your old Pi

If you have an old Raspberry Pi, perhaps because you’ve upgraded to a new Pi 3, you can donate it directly to the project through PiCycle. They’ll then recycle your old Raspberry Pi into the distributed compute cluster.

We’re looking forward to their discoveries and enjoyed working with the researchers. When we build solutions for customers we’re aiming to minimise the number of unknowns to de-risk the solution. By contrast tackling difficult unsolved problems is the whole point of research. If they knew how to build the system already they wouldn’t bother trying.

Posted in IPv6, Network, Raspberry Pi

Encryption is vital

June 7th, 2017 by

We refuse to bid for government IT work because we can’t handle the incompetence.

At Mythic Beasts we make use of free secure encryption all the time. Like all powerful tools such as roads, trains, aeroplanes, GPS navigation, computers, kitchen knives, vans and Casio watches; things that are very useful for day to day life are also useful for criminals and terrorists. It’s very popular for our politicians and the Home Office, especially our current prime minister King Canute Theresa May and leader of The Thick Party, to suggest that fully secure encryption should be banned and replaced with a weaker version that will reveal all of your secrets but only to the UK security services.

We disagree and think this is a terrible idea. There’s the basic technical objection that a backdoor is a backdoor and keeping knowledge of the backdoor secret is essentially impossible. There’s a recent practical demonstration of this: the NSA knew of an accidental backdoor in Windows and kept it secret.  It was leaked, resulting in the thankfully not very effective WannaCry virus which disabled substantial fractions of the NHS. The government is very good at scope creep: the Food Standards Agency refused to disclose why it needs the power to demand your entire internet history. We think it fundamentally wrong that MPs excluded themselves from the Investigatory Powers act. Then there’s simple commercial objections: it’s a slight commercial disadvantage if every UK product has an ‘Insecure By Order of The Home Office’ sticker on the front when your foreign competitors products don’t.

However, Mathematics does not care what our politicians wish and refuses to change according to their desires. Strong cryptography is free, available on every computer, and can be given away on the front of a magazine. Taking away secure cryptography is going to involve dragging a playstation out of your teenagers’ hands, quite literally stealing from children. Of course secure communications will still be available to any criminal who can illegally access some dice and a pencil.

It’s a good job you can’t build encryption machines with childrens toys.

At Mythic Beasts we make extensive use of open source free cryptography. OpenSSH protects our administrative access to the servers we run and the customers we manage. OpenSSL protects all our secure web downloads, including last month’s million or so copies of Raspbian ensuring that children with a Raspberry Pi don’t have their computer compromised. We make extensive use of free certificates through Let’s Encrypt and we’ve deployed tens of thousands of upgraded packages to customers which are securely verified by GnuPG.

Without these projects, vast quantities of the internet would be insecure. So we’ve made donations to OpenSSH, GnuPG and Let’s Encrypt to support their ongoing work. We’d like to donate to OpenSSL but we can’t see easily how to pay from a UK bank account.

Posted in Misc, Security

Save the Black Horse

May 26th, 2017 by

The last pub in Dry Drayton has closed and is under threat of development. As a community, we’re working hard to save it.

It’s Beer Festival week in Cambridge. Suddenly official work takes a back seat compared to the importance of drinking, serving and appreciating fine beer in the sunshine. It’s great that the volunteers behind the bar include friends, colleagues, customers, suppliers and the occasional former MP.

However, for 51 weeks of the year the Cambridge Beer Festival isn’t operating and beer lovers among us have to go to a more humble establishment, the pub. Cambridge City is blessed with multiple excellent pubs, but occasionally it’s nice to take a visit to the outlying villages.

So we were very saddened to hear that the only pub in Dry Drayton, the Black Horse, was due to close. However, a community group has started assembling plans to turn it into a community pub on a similar model to the excellent Dykes End in Reach. They asked us to help with setting up their on-line presence. Mythic Beasts fully support the effort to have lovely pubs within walking and cycling distance so we’ve provided a Managed WordPress site to help their campaigning efforts. Today we’ll share a beer with them in the Beer Festival, and in the near future we hope to take a field trip to their re-opened countryside pub.

Update: They reported last night they’ve had a lot of signups to their newsletter and several interested investors. It looks like we’re going to be part of a successful pub rescue!

 

Posted in Misc, Wordpress

Cambridge Beer Festival, Raspberry Pi powered Apps for Beer

May 22nd, 2017 by

We drew the architecture diagram for the beer festival on a beer mat.

Today marks the first day of the Cambridge Beer Festival, the longest running CAMRA beer festival, one of the largest beer festivals in the UK and in our obviously correct opinion, by far the best. Not only have we run the web back-end for many Cambridge CAMRA websites for many years, this year we’ve been involved with Cambridge App Solutions who run the iPhone Beer Festival App. They’d been having some trouble with their existing hosting provider for the back-end. In frustration they moved it to their Cloud Raspberry Pi which worked rather better. They then suggested that we keep the production service on the Raspberry Pi, despite it being a beta service.

Preparing for production

We’ve set up all our management services for the hosted Pi in question, including 24/7 monitoring and performance graphing. We then met up with Craig, their director in the pub to discuss the app prior to launch. The Pi 3 is fronted by CloudFlare who provide SSL. However, the connection to the Pi3 from Cloudflare was initially unencrypted. We took Craig through our SSL on a Raspberry Pi hosting guide and about a minute later we had a free Let’s Encrypt certificate to enable full end-to-end data security.

 

 

The iPhone app that runs the Cambridge Beer Festival (also found at Belfast and Leestock)

 

The iPhone Beer Festival App tracks which beers are available and the ratings for how good they are. Availability is officially provided, ratings are crowd sourced.  The app is continuously talking to the back end to keep the in app data up to date. All this data is stored and served from the Raspberry Pi 3 in the cloud.

Proximity

The festival also has some Estimote beacons for proximity sensing which use Bluetooth Low Energy to provide precise location data to the phone. On entry to the beer festival the app wakes up and sends a hello message.

Posted in IPv6, Raspberry Pi, Security

« Older Entries
Newer Entries »