HN Explorer

Ask HN: DNS host suddenly deleted us - fastest way to get your DNS back?

2 points by marquis ↗ HN
We had a terrible incident on Saturday where an engineer at our hosting provider deemed it relevant to remove our managed DNS service along with unrelated server removal we had requested, with no indication that they would do this. Fortunately I was able to change our name servers quickly, but it left some customers without service even until now, as some DNS caches haven't refreshed. We have server backups, load balancers, text alerts, server monitoring - everything we can do within our small budget to keep online 24/7. We never anticipated we might lose DNS. We're going to add more NS entries to another provider, but is there anything further we can do? Would love to hear any stories. It was a crazy weekend I don't want to repeat.

5 comments

[ 2.8 ms ] story [ 19.8 ms ] thread
cosjef
I would employ a secondary DNS service to replicate your zone files.
marquis
We're looking into this now. Is a secondary DNS service usually sufficient? Even Facebook seems to use a single domain NS.
baruch
If the concern is of a mistake causing the deletion of the zone a replicated zone file will be lost just as well. To avoid this you'll need two name-servers and update each of them independently.

Keeping not too long DNS caching times would help recovery, too short and it will be a drag on the site performance so a few (6?) hours should be fine to reduce recovery time.

reiz
What are the reasons to manage your own DNS? Why not using a service like http://www.zerigo.com/. Or just manage the Nameserver zonefiles on a provider like Hetzner.de?
marquis
Yes, this was a managed DNS service that we rented from our (now cancelled) hosting provider. They deleted it by mistake without telling us, when we asked them to discontinue an old server we were no longer using.