The aforementioned fix has been implemented together with our affected customers, resolving this issue.
We are closing this incident now - the promised post-mortem on this issue will soon be published as a blog post.
Posted Sep 12, 2017 - 11:36 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Sep 11, 2017 - 13:35 UTC
Update
We have been informed that our DNS provider has silently changed their service portfolio, dropping a feature we previously relied upon: nameserver hosting without using the provider as a DNS registry.
To avoid further issues, we have decided to switch the DNS handling for the affected domains over to another DNS provider we have been using as an alternative. We have already contacted all customers and switched most of the domains that needed the change.
All other customers were not affected by this issue and won't be.
We are very sorry for the situation!
We will post a post-mortem report soon where we are going to describe the issue and consequences in detail.
Posted Sep 11, 2017 - 13:35 UTC
Investigating
We are currently experiencing issues with our Domain Name Service (DNS).
Over the weekend, a few domain zones have dropped from our DNS provider's service without warning or discernable cause.
Immediately after learning of this incident, we have restored full operation by recreating the missing entries manually.
We are communicating with our DNS provider while they are busy determining the root cause.
In the meantime, we are adding explicit external DNS checks to our monitoring system. While in this scenario the website HTTP checks we've already had in place only trigger after DNS caches expire, these new DNS checks will allow us to respond much faster a DNS zone vanishing.