January 15, 2015

Last night's outage

Some of our users have noticed that YDNS was unavailable for almost 13 hours, starting on 12 p.m. this morning. This post lists all the details on what happened and what we will do to mitigate such issues in the future.

It all started with an unplanned maintenance in a data center, where we were moving some equipment and servers between racks. That also included the core infrastructure of YDNS, including the web site, database and the primary name server (ns1.ydns.eu). The problem here was that all secondary name servers (ns2.ydns.eu and ns3.ydns.eu) were using the database of the primary name server. While the secondary servers are on physically different locations around the globe, these servers were using the database located in the core infrastructure data center. No replication or similar techniques were used, so the loss of connectivity inevitably led to total failure of YDNS. No DNS queries - not even those to "ydns.eu" could be answered during the maintenance window. The maintenance took about 4 hours and all the servers were back up - except the storage system that hosts the virtual machines for YDNS. After some hours we finally found the cause of this and fixed it. YDNS came back to normal operation.

To avoid such problems in the future, we have changed some things. Each name server node got it's own read-only database server, which uses stream replication to get latest updates from the primary database server. That results in faster lookup speeds when querying DNS records and helps to keep YDNS answering DNS queries in case one or more servers are unavailable. If something like today would happen again, your DNS queries will still be answered, but you wouldn't be able to update any records until the primary database is back (which is linked with the web site and backend).

We currently offer three DNS servers at various locations:

  • ns1.ydns.eu (Hamburg, Germany)
  • ns2.ydns.eu (Frankfurt am Main, Germany)
  • ns3.ydns.eu (New York City, USA)

We will soon deliver a closed beta test of a brand new YDNS, which will also increase the availability and scalability of this service.

Again, we kindly apologise of any trouble caused by the outage last night.

In case you have any questions regarding this, please don't hesitate to leave us an reply or email us at support@ydns.eu.