More DNS API fun: find an IP across all zones

September 21st, 2020 by

A customer was doing an IP address change on a server and wanted a quick way to find all references to the old IP address across all of their domains.

This seemed like a good job for our DNS API and a few UNIX utilities.

Finding matching records

Our DNS API makes it easy to find records with particular content:

curl -sn https://api.mythic-beasts.com/dns/v2/zones/example1.com/records?data=1.2.3.4

The -n assumes we’ve got a .netrc file with our API credentials. See our DNS API tutorial for more details.

This gives us a block of JSON with any matching records:

{
  "records": [
    {
      "data": "1.2.3.4",
      "host": "www",
      "ttl": 300,
      "type": "A"
    }
  ]
}

jq lets us turn the presence or absence of any matching records into an exit code that we can test with an if statement by piping into the following:

jq -e '.records | length > 0' 

This counts the number of members of the records array, and -e sets the exit code based on the output of the last expression.

Getting a list of zones

We want to check this across all zones, so let’s get a list of zones:

curl -sn https://api.mythic-beasts.com/dns/v2/zones

This gives us some JSON:

{
  "zones": [
    "example1.com",
    "example2.com"
  ]
}

What we really want is a flat list, so we can iterate over it in bash. jq to the rescue again. Simply pipe into:

jq -r '.zones[]'

and we get:

example1.com
example2.com

Putting it all together

Putting this all together with a for loop and an if:

IP=1.2.3.4
for zone in $(curl -sn https://api.mythic-beasts.com/dns/v2/zones | jq -r '.zones[]') ; do
  if curl -sn "https://api.mythic-beasts.com/dns/v2/zones/$zone/records?data=$IP" |\
      jq -e '.records | length > 0' >/dev/null ; then 
    echo "$IP found in $zone"
  fi
done

Gives:

1.2.3.4 found in example1.com

More than one way to do it

Another approach would be to use the zone file output format and check if the output is empty or not:

curl -sn -H 'Accept: text/dns' \
  "https://api.mythic-beasts.com/dns/v2/zones/$zone/records?data=$IP"

This give us matching records, one per line:

www         300 A 1.2.3.4

We can then test if we’ve got any matches using ifne (if-not-empty, part of the moreutils package in most distributions):

curl -sn -H 'Accept: text/dns' \
  "https://api.mythic-beasts.com/dns/v2/zones/$zone/records?data=$IP" \
  | ifne echo $IP found in $zone

Access to our DNS API is included with all domains registered with us. API credentials can be limited to individual zones or even records, can be either read/write or read-only.

ANAME records

Of course, it’s generally desirable to avoid including an IP address in lots of different DNS records in the first place. It’s preferable to assign the IP to a single hostname, and then point other records at that. Our DNS service supports ANAME records which allow the use of hostnames rather than IP addresses in places where CNAMEs cannot be used.

Automating DNS challenges

May 5th, 2020 by

We recently announced our new DNS API which we’ve just moved out of beta and into production. 

One of the goals of the new API was better support for automating DNS-based challenges, such as those used by Let’s Encrypt to authenticate certificate requests. 

DNS-based challenges are needed to obtain wildcard certificates from Let’s Encrypt, and can be a convenient way to get certificates for hostnames that don’t a have publicly accessible web server, but can be tricky to implement due to delays in updating DNS records, and automatic requires having credentials capable of DNS records for your domain stored on your server.

The new API has a number of features to address these issues.

Restricted credentials

The DNS API allows you to create API credentials that are restricted to editing specific records within your domain.  Credentials can be restricted by hostname, record type, or both.

For example, you can create credentials that can only edit the _acme-challenge TXT record needed for Let’s Encrypt challenges. Access to the DNS API is potentially very sensitive, so it makes sense to limit access as much as possible.

Restricted API key

Record verification

Updates made via the API do not become live immediately. There is a delay of up to a minute before they hit our master nameserver, and a potential further delay of a few seconds before the record propagates to our authoritative nameservers. When responding to a DNS-based challenge, you will typically want to ensure that the record is actually live before proceeding with verification.

Our DNS API provides a “verify” feature, that checks that records are live on all authoritative nameservers. For example, a GET request to the following URL would check that the nameservers have the latest update to the record:

https://api.mythic-beasts.com/dns/v2/zones/example.com/records/_acme-challenge/TXT?verify

This will return a 200 response if the nameservers are up-to-date, and 409 if they are not. This can be used to script a check after updating a record:

#!/bin/sh

ZONE=example.com
RECORD=_acme-challenge
TYPE=TXT

for i in $(seq 1 12); do
    RES=$(curl -n https://api.mythic-beasts.com/dns/v2/zones/$ZONE/records/$RECORD/$TYPE?verify -qs -w '%{http_code}' -o /dev/null)
    case $RES in
        200)    echo Records updated
                exit 0
                ;;
        409)    echo "Not yet updated ($i/12)"
                ;;
        *)      echo "Unexpected error: $RES"
                exit 1
                ;;
    esac
    sleep 10
done
echo Timed out
exit 2

Obtaining certificates the easy way

Our preferred Let’s Encrypt client is the excellent dehydrated, and we maintain a hook script for supporting DNS-based challenges in dehydrated. We haven’t yet updated the hook script to support our new API, but will be doing so soon and will post details here when it’s ready.

New DNS API

April 6th, 2020 by

We’ve just launched our new DNS API, which provides a much cleaner and more powerful API for automatic management of DNS records.

Key features of the new API include:

  • Configurable auth credentials – restrict access to individual records, or record types.  Ideal for Let’s Encrypt or other DNS-based challenges.
  • Choice of JSON or zone file format for input and output.
  • Atomic multi-record updates – update arbitrary sets of records in a single transaction.
  • Form parameters for record creation – record creation can be trivially scripted using curl.
  • Broad record type support – CAA, SSHFP, TLSA, SRV and many more.

For a walk through of the features of the new API, please see our DNS API tutorial , or for more details see the reference documentation.

To get started with the API, use the API keys section of the control panel to create some credentials.

Configurable API permits

Restricted API credentials for Let’s Encrypt challenges

The new API is currently in public beta, meaning that we reserve the right to make last minute breaking changes to the API, although we expect any such changes to be minor, and we would very much like to hear any feedback you may have.

Security in DNS, TLSA records now available in our control panel to support DANE

February 11th, 2020 by

The Internet is better when it’s secure. Finally, thanks to Let’s Encrypt it’s possible to automatically get SSL certificates free of charge and as a result the Internet is dramatically more secure than it used to be. If you’ve used our DNS API you may have discovered that you can verify Let’s Encrypt SSL certificate requests using DNS records, including issuing wildcard certificates.

We support secure DNS (DNSSEC) which prevents DNS records from being forged, making the process of authenticating your SSL certificate through DNS records far more secure than the email-based authentication that was typically used for certificates issued by commercial certificate authorities. We have implemented support for CAA records which uses DNS to restrict the certificate authorities that can issue your certificates. This is most useful if the DNS is trustworthy which, again, requires DNSSEC.

However, there seems to be an opportunity here to improve things further. Rather than relying on a 3rd party certificate authority to confirm that you have control of your own DNS, why can’t you just publish your certificate in DNS directly? If you can trust DNS this would seem to be an obvious improvement, and with DNSSEC, DNS becomes trustworthy. We’ve now added support for the additional record type TLSA which allows exactly that, as part of DNS-Based Authentication of Named Entities (DANE).

Adding a TLSA record through our control panel.

DANE is a flexible mechanism that can be used to add an additional layer of security to certificates issued by a 3rd party authority, or to enable the use of self-signed certificates.

Unfortunately at the moment few clients support TLSA so for the majority of interactions you’re still going to rely on the certificate authority to verify the certificate. But implementations exist for both Exim and Postfix. Step by step, email is becoming more secure.

Hosting made Sympl

May 21st, 2019 by

Sympl is so simple it’s even usable by Cambridge graduates

We’re pleased to announce that we are now supporting the Sympl open source project.  Sympl is a fork of Symbiosis, a platform that makes hosting websites and email on a virtual or dedicated server simple.  Once installed, configuring a new website, or creating a new email address and mailbox, is as simple as creating a new directory.  Web server, mail server and DNS configuration is all taken care of for you.

We’ve already taken the first steps towards integrating Sympl into our infrastructure by implementing support for our DNS API in OctoDNS.  For our next step, we will be adding support for OctoDNS to Sympl.  This means that it becomes possible to use Sympl with our DNS infrastructure, but equally you can use any other provider supported by OctoDNS (we don’t believe in lock in!)

We’re now very pleased to welcome Paul Cammish, the newest member of the Mythic Beasts team.  Paul has considerable experience, having worked at a number of different ISPs since 2000, most recently at Bytemark.  Paul created the Sympl project earlier this year, in order to provide ongoing support and enhancements for the platform.

We’re very excited by the possibilities that Sympl provides, and have some interesting ideas for future developments once we’ve dealt with the immediate priorities of DNS integration, and support for the upcoming Debian Buster release.

The source code for Sympl is now available in our self-hosted GitLab instance.

Moving to Mythic Beasts just got easier

April 9th, 2019 by

We’ve just rolled out a major overhaul of our DNS management interface. We hope that you’ll find the new interface faster and easier to use. As well as improvements to the user interface, we’ve also added the ability to import zone files. This means that if you’ve got a domain that is currently hosted with another provider, you can now easily transfer all of your DNS configuration to our servers in bulk (provided that you can get them to give you a copy of your current zone file).

Our DNS management interface is included with all domain registrations.  It’s also available for domains registered elsewhere for customers of our other services, including hosting accounts, virtual servers, dedicated servers and Raspberry Pi servers.

The DNS interface includes DNS API access, allowing you to support dynamic DNS and to automate other DNS management tasks.

We believe in retaining customers through good service rather than lock-in, so naturally there’s a corresponding zone file export feature.

Round-robin DNS – another use for ANAMEs

March 22nd, 2019 by

Sensible people don’t like to hard code IP addresses in lots of different places in DNS. Better to assign it a name, and then reference that name, as it makes it clearer what’s what and if you ever need to change that IP, you’ve only got to do it one place.

CNAME records can be a good way to do this, by aliasing a DNS name to an IP. Unfortunately, the DNS specs prevent you using CNAMEs in various places that you might want to, most commonly at the root-level of your domain (the dreaded “CNAME and other data” problem).

This is where ANAME pseudo-records come in. They look just like a CNAME record, but rather than being added to the DNS, our server converts them into A and AAAA records. This allows you to get the benefits of a CNAME in places where a CNAME is not legal.

This week a customer suggested another use for ANAME records that we’d not previously thought of: round robin DNS. That is, a single DNS name that points to multiple servers. As you can’t have multiple CNAME records for the same hostname, implementing round-robin DNS means hard-coding A and AAAA records into your zone file. Like this:

proxy.mythic-beasts.com. 3600	IN	A	93.93.129.174
proxy.mythic-beasts.com. 3600	IN	A	46.235.225.189
proxy.mythic-beasts.com. 3600	IN	AAAA	2a00:1098:0:80:1000:3b:1:1
proxy.mythic-beasts.com. 3600	IN	AAAA	2a00:1098:0:82:1000:3b:1:1

Which is messy. Wouldn’t it be nicer to use the names of the servers involved? Like this:

proxy.mythic-beasts.com. 3600	IN	CNAME	 rproxy46-sov-a.mythic-beasts.com.
proxy.mythic-beasts.com. 3600	IN	CNAME    rproxy46-hex-a.mythic-beasts.com.

Sadly, the spec says you can’t do that, but thanks to a minor tweak to our DNS control panel code, you can now do it with ANAME records. Simply specify multiple ANAME records for your host name, and we’ll go and find all A and AAAA records for all of the hosts that are referenced.

Thanks to @grayvsearth for the suggestion on this one.

ANAME records are available in our DNS management interface, which is included with all domain registrations, and available for free on other domains for customers of other services. Other features include a DNS API, allowing you to obtain Wildcard Let’s Encrypt certificates.

Let’s Encrypt wildcard certificates

February 15th, 2019 by

Wildcard… sounds a bit like wildcat… cat pics!
Photo by Peter Trimming, CC BY 2.0

We’ve just made some changes to our plugin for dehydrated in order to better support Let’s Encrypt wildcard certificates.

Unlike normal certificates, which can be obtained using a web-based challenge, Let’s Encrypt’s wildcard certificates require a DNS-based challenge. In other words, you need to prove that you can control the DNS for the domain for which you are requesting a wildcard certificate.

Mythic Beasts provides a simple API for controlling DNS, which makes it possible to automate the process of responding to these challenges, and we provide a plugin for the popular dehydrated client that does just this.

We’ve just deployed a minor change which means that it’s now possible to obtain a single certificate for a domain, and a wildcard under that domain.

Access to our DNS API is included with all domain registrations. For more information, please see our instructions on using DNS-based challenges wih Let’s Encrypt. Please note that in order to obtain wildcard certificates you need to be using dehydrated version 0.6.0 or later.

Retrosnub Acquisition

June 4th, 2018 by

A Mythic Beast eating a Retrosnub (artists impression)

Just before Christmas we were approached by Malcolm Scott, director of Retrosnub, a small cloud hosting provider in Cambridge. His existing connectivity provider had run out of IPv4 addresses. They’d decided to deal with this issue by adding charges of £2 per IPv4 address per month to encourage existing customers to return unused IPv4 addresses to them. As a cloud hosting provider with a substantial number of virtual machines (VMs) on a small number of hosts this had the result of tripling the monthly colocation bill of Retrosnub.

Aware of my presentation on IPv6-only hosting at UKNOF, Malcolm knew that opportunities for significant expansion were severely limited due to the difficulty of obtaining large amounts of IPv4 address space. Retrosnub faced a future of bankruptcy or remaining a very niche provider. His connectivity providers seemed strongly in favour of Retrosnub going bust so they could reclaim and re-sell the IPv4 space for higher margin services.

There are no expansion opportunities for new cloud hosting providers.

As a larger provider with our own address space, we had sufficient spare capacity in our virtual machine cloud to absorb the entire customer base of Retrosnub with no additional expenditure. Our work in supporting IPv6-only virtual machines will also make it easier to significantly reduce the number of IPv4 addresses required to support Retrosnub services. We formed a deal and agreed to buy the customer base of Retrosnub.

Combining operations

Since agreeing the deal, we’ve been working hard to merge our operations with minimum disruption.

The top priority was the domain name services because domains expire if you don’t renew them. Doing a bulk transfer of domain names between registrars is something which Nominet, the body responsible for UK domains, makes extremely easy, as it just requires changing the “tag” on all the domains.

Unfortunately, just about all other TLDs follow a standard ICANN process, which requires that a domain be renewed for a year at the time of transfer, and that the owner of the domain approves the process. If you were designing a process to destroy competition in a market by making it hard for resellers to move between registrars, it would look quite like this.

We’ve now got the bulk of domains transferred, and the next steps will be to migrate the DNS records from Retrosnub to Mythic Beasts so that our control panel can be used to change the records.

At the same time, we rapidly formulated a plan to migrate all the virtual machines in to stem the financial losses. Moving the VMs required an unavoidable change in IP address, and we also wanted to get them migrated from their current platform (Citrix Xenserver with para-virtualisation) to our own platform (KVM with full hardware virtualisation).

In order to ease the transition, we arranged for a pair of servers to do IP forwarding: a server in our cloud that forwarded the new IP to the VM in the Retrosnub cloud until it was migrated in, and another in the Retrosnub cloud that forwarded the old IP after the server had been moved. By doing this we were able to give customers a one week window in which to complete their IP migration, rather than forcing it to be done at the time that we actually moved the VM.

In the process of this migration, all customers received a significant bandwidth upgrade and majority received disk, RAM and CPU upgrades too.

We completed this on schedule before the quarterly colocation bill arrived, so instead of paying the much increased bill, we cancelled the contract and removed the servers from the facility.

Next steps

Our next step will be to migrate all the web and email hosting customers into our standard shared hosting environment. This has some time pressure as Google have plans for Chrome to start marking all non-HTTPS websites as insecure. We offer one click HTTPS hosting using Let’s Encrypt on all of our hosting accounts.

Domain price reductions

February 9th, 2018 by

Fortunately we don’t buy domains in Bitcoin

Most of our domains are billed to us in US Dollars, so our pricing is at the mercy of the GBP/USD exchange rate.  The pound has strengthened significantly against the dollar since we last reviewed our pricing, so we’ve just rolled out reductions on many of our domain registration prices.  For example, .com domains drop by £2 to £11 + VAT, and .cymru (which, confusingly, we buy in dollars) drops by £3.50 to £22.

We aim to offer straightforward, no-nonsense pricing with no unsustainable introductory discounts that punish customer loyalty with inflated prices in subsequent years.  We price our domains at a level that allows us to properly support our customers.