Been dealing with this all morning, finally got all of my zones migrated to Route53, but with the amount of time it takes for DNS changes to propagate we are going to be feeling this all day.
Any advice for the future on how to add redundancy to my DNS setup? Is it as simple as maintaining Nameservers on two different providers and pointing to them both on my domain?
RAX has a 6.3 billion market cap. If an org this size -- specialising in hosting -- cannot field a DDOS-resistant DNS server, who can?
I'd like to migrate or even perhaps add a secondary DNS but RS DNS doesn't seem to even offer zone transfers (the best you can do, I guess, is to use the API to get your records out).
This is surprising. The Rackspace DNS infrastructure is large and robust. It was believed just the bandwidth to the masters alone should be sufficient for even the largest DDOS. So this is either something different or of a vastly larger scale than has been seen before.
Yeah, no zone transfers, but you can always grab a complete Bind 9 export via the API or just call support and they can fetch it for you.
>cannot field a DDOS-resistant DNS server, who can?
Its tough to do. DNS is a dumb and ancient protocol from the "lets all be friends" days of the internet. Its a bit more complex than installing mod-evasive and calling it a day.
DDOS is currently an unsolved problem for many popular protocols and services. Blaming just Rackspace seems unfair. Last week in was Namecheap. The week before it was someone else, etc. DNS DDOS is just a non-trivial problem to solve.
This should also be a reminder to have more than one DNS provider in your domain record. A backup nameserver from a different provider saves you and your customers a lot of heartache.
True, I was thinking of this earlier. When making changes you have to do it in 2 places, which is really the only downside, but it does protect you a bit. Of course, it's still not foolproof - when the first or second DNS server gets used is implementation dependent. We got bit by this, but not too badly - RAX filtered out some residential ISPs whose IPs were part of the DDoS attack but also sent legitimate queries so it was only some people who couldn't resolve. So I might still be too lazy to implement it for all our clients' zones. I'll have to read their post-mortem and see what steps they are taking to avoid this issue in the future.
Hm. I always thought DDoS protection for DNS was rather simple problem to solve, since anycast works with DNS traffic pretty much by design, so you can have hundreds of machines spread all over the world, handling hundreds of gigabits of bandwidth. And filtering of DNS packets is also relatively simple thing to do at line rate.
It's simple except that you need to have capacity greater than what is achievable by people controlling the DDOS.
I'm not up to date on what that number is these days, but a few years ago 100Gbit/s was the target. For an Anycast DNS service, you don't need that in every site -- if you're willing to take small sites offline and degrade your service's end-user performance in this scenario. You will still need it in 2-3 sites, probably located in US west / US east / EU west because of the cost of datacenter space and the availability of cheap connectivity.
Serving 100Gbit/s requires 20 10G circuits in each site (~$15-30k/mo), 80 servers (2-3 racks, so $5-10k/mo), and a not-so-cheap router.
So your monthly opex is $20-40k per site, $60-120k worldwide. Your capex is $450k per site, $1.35mm worldwide.
It's not an insane amount of money, but it is kind of a lot to spend on DNS. It's particularly tough to justify at a lot of companies that you need this kind of buildout, when normally you can operate on 1/20th that amount of hardware.
Good post. I would argue some of your locations and how many servers are actually needed, but that's splitting hairs.
The real point I want to make is that the only companies really putting this amount of money into DNS infrastructure are DNS providers themselves. I know of no hosting provider/ISP/etc. who puts nearly this much thought or effort into DNS.
Even for myself - and I consider myself a DNS junkie - my anycast cluster was a dozen or so servers worldwide co-located w/ a mid-market CDN provider (plus a few locations from my own ASN). That setup was greater than 99% of DNS installs out there - but it would have fallen over under a major attack. It truly takes some skill and financial commitment to build out correctly.
I agree with you, there are only two kinds of companies with the kind of DNS infrastructure that could withstand a large scale DDOS:
1. DNS hosting companies (Dyn, Amazon, etc.)
2. Large Internet companies (Yahoo, Google, etc.)
And for sure not all of them! A former employer's public zone was hosted by a DNS hosting company that didn't use Anycast, didn't have a footprint outside US/EU, and seemed to have little to no idea when or why traffic spikes happened.
So to do DNS really right you need a fair amount of money, some specialized expertise and knowledge of some non-DNSy things (Anycast, among others). I'm really not interested in hosting my own (public) DNS these days at work, it's just not worth the time and effort until you get gigantic. As with e-mail, there are specialists who can do it better than (most) places can.
I do kind of worry about depending on one DNS provider, so I've tried to use two when possible. That's a whole other world of hurt though.
It's not simply having enough aggregate capacity though either. The way anycast works is that a host will take the shortest network path to the closest server.
So for example (very high-level) you might have a server in the US, and one in Europe. If your are eating a DDoS attack maybe the botnet has tons of US-based IPs, but very few euro. In that scenario, you will have all your US customers down but the EU guys are just fine.
That's pretty simplistic though. It's less by country, more by ASN. And it's also a case of automation killing you. What happens when the DDoS attack takes out a single PoP worth of DNS servers? Usually you have some automated tooling that will take down that PoP entirely, and withdraw the announcement. You want this behaviour in almost all cases. However, now that attack traffic just got sent to the next-closest server (in this example EU - taking down the whole worldwide cluster).
So it's not as simple as having "100gbps worth of inbound DNS capacity" - you need to spread it out properly, and of course deal with attacks like you would otherwise.
The other problem is it's UDP - so spoofed packets and the like are a much bigger problem.
You're right, you definitely need significant single site capacity (2-3 sites with 100Gbit/s, not just the entire system, for instance), and you may even have to play routing games to move traffic around if that's not enough. Any other sites may have to get shut off during massive abuse incidents.
It's a tough call to say if it's better to protect the rest of the network by trying to isolate the attack, even though that means you are totally down in one region. I'm inclined to keep 1/2 running than to have nothing, but if I have this release valve of moving traffic to the farms designed to soak up abuse traffic then it makes my decision easier. I should be able to get back up worldwide, if I've done my provisioning properly.
One of the nice things about this kind of attack is that your attacker has no choice but to follow your routing, whereas if you use your DNS GSLB to re-route an attacker away from your HTTP server farm (to handle a different sort of attack), they may not stop hammering on its IP address. After all, it's pretty unlikely that they respect DNS TTLs.
This approach is simplistic, but it's basically the only solution. The nature of a DDOS attack is that you cannot predict its origin, beyond the distribution of compromised client machines around the world. E.g. your attackers are pretty unlikely to come en masse from Africa.
Spoofing is an issue. Not much to do about that at your farm level, but you can start working with your upstreams to get ACLs in place to prevent some portion of that. And if the attack is prolonged and sufficiently large, the only way you're going to stop it is to identify sources and get help from involved networks and ISPs.
While the discussion you want to have is a valid one, you're opening on a complete logical fallacy.
Market cap and technical sophistication, esp at large scale, tend to become divergent. In other words, it is generally assumed that big companies do things poorly. That's the entire notion of startups.
I will say this: Dyn can probably do a better job of hosting your DNS than Rackspace, Rackspace is not focused on DNS, and I'm not aware that they even really do anything special except for running lots of big DNS servers. Even when I worked at Rackspace 10y ago, the size of their customer base made them a constant target for DoS attacks. Do you need Dyn? Also probably not.
Rackspace has patented DDos prevention technology called RiverGuard, but since being patented only illustrates that it is novel, and not that it is of any quality, that is also meaningless.
I will say that, even though I prefer as a past employee to work with other companies, because the things Rackspace is bad at are things they don't give a fuck about solving, they run a damned fine network.
Further, as a replacement for / alternative to a colo, it is irrelevant whether a company can run a good DNS infrastructure, because that's not what I'm asking them to do. Rackspace DNS is there for convenience.
Cloud providers like AWS use their own cloud infrastructure (we think) to host services like Route53, which are also not infallible, but tend not to _entirely_ go down like this. Rackspace, while being somewhat competetive in the cloud space, doesn't operate anything like Amazon, doesn't do anything in a very distributed manner, and is backing those cloud offerings with what I expect to be vertically scaled DNS servers on top-end Dell hardware behind some kind of load balancers.
Money doesn't make you good at things, rather it helps you to not have to worry about the details, so it is inversely true that a company with a substantial market share should be expected not to make the same mistakes as the little guys. They're operating all day, all week, all year under the fallacy that their scale is proof of the quality of their work, and following this logic, they don't have to do any work at all. ;)
Are you a disgruntled employee? You sure sound like one. That, plus that you worked there over ten years ago - and that Rackspace didn't even offer a DNS api until a few years ago[1] - makes it very hard to take seriously any of your claims about the nature of their dns infrastructure.
All we know is that Rackspace has dns servers in three out of its six datacenters[2], and that they appear not to have always-on DDoS protection in place.
Market cap and technical sophistication, esp at large scale, tend to become divergent.
You say this, but go on to compare Rackspace unfavorably to Amazon who, at 141.93B (of which AWS is just a part), has over 20x Rackspace's market cap and probably 10x the infrastructure[2][3]. Big companies do things poorly, but then it's hard to have highly-available dns without having lots of infrastructure.
Namecheap, Route53, Dyn, DNSimple, and now Rackspace have suffered DDoS attacks with outages of varying severity. Don't act like DDoS prevention is a solved problem and claim technical incompetence.
All of my websites using Rackspace DNS where down for around 9 hours.
It was not a good start on a Monday. And it was even harder to explain that your websites are up, but not up to a client lol, and there is basically nothing I can do.
So yah this marks the point at which I will be using R53 and RS DNS servers.
Tip: If you're using >=2 providers to survive DDOS attacks, it's best to use no more than two name servers from any one provider. That's because resolvers will generally try up to 3 different name servers before giving up.
The "no more than two nameservers" rule means that those attempts will always span at least two providers.
Sites on Rackspace servers, DNS through NetSol (not that they are good or bad, but it wasn't RS). Didn't realize anything was going on until seeing this article.
Why don't DNS clients fail back to use the most recent good IP address as a default?
Indeed how can I set up my Kubuntu desktop to have this behaviour. Most domains probably point to the same IP for years at a time I'd imagine. If that's true it seems very strange that just because example.com's DNS server was offline this one day out of the last 600 that my browser can't "guess" that "123.45.67.89 example.com" is going to work.
You'd need to add some security around that to avoid abuse I'm sure but doesn't it seem reasonable? Shouldn't DNS failures only prevent you from getting to a server if the IP has changed??
It's nice having cheap/free DNS from people like Rackspace and Amazon, but situations like these make you realise that it's sensible to use a company like Dyn (http://dyn.com/) that are experts in highly-available DNS, rather than something that's a small part of a hosting provider's services.
It's easy to forget that you can have redundancy in your load balancers, web servers and databases (replication, multiple data centres, etc), but DNS is how you're found by the rest of the Internet.
No DNS resolution = no one reaches your expensive, lovingly-crafted infrastructure.
I think Rackspace invests a lot of money into their DNS infrastructure. Yea it's free but its mostly because you're paying so much extra for the servers and support.
No DNS doesn't always mean no one reaches your expensive infrastructure. That may be true for websites relying on lots of random traffic, but most if not all of our customers have been to our site before so there's a strong chance that the DNS has already been cached on their computer or router.
But that's no use if you use the default TTL of 2-3 hours that I always see whenever I configure new dns entries. Sensible TTL should be at least 24-48 hours, maybe even more for your returning users to not be affected by those kind of outages.
A 24-48 hour TTL means you're hosed for 24-48 hours if your hosting service has an extended outage because you can't point any of those visitors at the IP addreses at another host. With a short TTL on the other hand, the worst-case scenario is that your DNS provider is offline and you have to change nameservers at the registrar. With a hosting outage, you can just point the domains at the new IPs and be back up in minutes to hours.
Four regions have "emergenecy maintanance" right now at Dyn. Of course, we don't know whats the cause of that. I imagine if its also a DDOS they'll probably not reveal that. I have heard from other sysadmins that they've had problems with them as well.
I have DNSmadeeasy and they seem to do okay, but only because they're such a small player and avoid being targetted too often.
Sorry, but there's no silver bullet here. Cloud providers get hacked and DDOS'd all the time. Kiddies and morons find it amusing to do so. Roll your own if you want to avoid being attached to such a big target and have a cloud provider just be your secondary namesever.
What they do unfortunately is charge you for the queries but there is no way of knowing which host has exceeded the limits query wise. When we first started using them they stated that even though we were exceeding the limits according to how they measure "not to worry you won't get charged". Of course perhaps 1 to 1.5 years later they now do charge (email out of the blue one day) and constantly try to bump us to a higher level service. [1] All this for a few zones that use to run comfortably on a 2 servers that we had many other things going on as well as DNS. (Now they claim we get 2.5 million queries per month but there is no way to determine exactly if that is true or for what host, ie foo.domain.com vs. www.domain.com is causing the excess queries).
[1] Which is more than paying the overage charges which is what we do every month.
It's easy to blame 'large hosting provider' and suggest going with 'specialized provider', but isn't stuff like this really leapfrog between hackers and the good guys?
Sure, service X might be able to block a 50 foobit attack but what about when the next vulnerability is found and they can launch 500 foobits of DDoS?
Nice to see you mention a repository - I love storing DNS details in git, and setup a site to push that on to Amazon's route53 infrastructure (https://dns-api.com/).
Having revision-control is wonderful for history-tracking.
This is a really cool idea. I'd like it even more if I could add "remotes" for different providers. DNSimple, Route53, etc. and push my changes to each one. I suppose the service would need to support a number of providers that themselves support APIs.
Hey I wanted to thank you for showing this to me. I am using it on my personal domain and plan to use it on some customers as well. Really cool tool, thanks!
I did think of that, actually hosting the git repositories, and having a hook to make the updates on pushs.
But it seems like I can be significantly simpler and more likely to be reliable if I just get triggered by a webhook. I think I'd be willing to do it for a couple of people as a trial but I'd not expect so much interest.
For anyone else worried about this, the best way to mitigate this going forward is to have secondary DNS servers.
Your primary DNS provider should allow automatic zone transfers. This makes it so that any changes you do to your primary service gets propagated to the secondary service within seconds.
Once setup you'll automatically have redundancy incase the primary provider starts timing out.
Who launches these massive DDoS attacks against DNS infrastructure? It would require a substantial botnet to pull off, so they must have some compelling reason. Maybe there's something obvious I'm missing, but I don't see one, except perhaps for a government or large organization which had many competitors using the DNS service they attack.
There is no requirement in a substantial botnet for this, even a small one will do. Also internet is full of cheap amplified ddos offerings. Attacker could use multiple offerings for more bandwidth if he feels like it. Maybe this is something obvious you are missing.
Until more orgs implement BCP 38, it only takes a handful of servers with an open resolver to multiply the attackers bandwidth by a factor of 30 or more. UDP does not verify the source IP is the sender. Most of the internet will drop spoofed packets already, but there are still enough netblocks passing spoofed traffic, and vulnerable DNS servers for this to be a concern.
I'm not savvy to this stuff so pardon the unsolicited conspiracy theory, does this DDoS have anything to do with the NK internet outage, or is it just coincidence?
Anyone with an idea how well Google Cloud DNS handles DDoS? I would image it just sucks it up, because they already saw any DDoS volume (would it be any or every, I'm not a native speaker?) before.
I'll make the answer simple. Market cap and size do not matter. All it takes is _competent staff_ and management support.
Actually DNS is not that hard to mitigate. You just need optimized compute and the bandwidth to take it to the clean up equipment.
60 comments
[ 2.9 ms ] story [ 133 ms ] threadAny advice for the future on how to add redundancy to my DNS setup? Is it as simple as maintaining Nameservers on two different providers and pointing to them both on my domain?
https://news.ycombinator.com/item?id=8716662
Seems namecheap and DNsimple have suffered from these attacks lately. I had some sites affected by namecheaps DDOS.
I'd like to migrate or even perhaps add a secondary DNS but RS DNS doesn't seem to even offer zone transfers (the best you can do, I guess, is to use the API to get your records out).
Yeah, no zone transfers, but you can always grab a complete Bind 9 export via the API or just call support and they can fetch it for you.
Its tough to do. DNS is a dumb and ancient protocol from the "lets all be friends" days of the internet. Its a bit more complex than installing mod-evasive and calling it a day.
DDOS is currently an unsolved problem for many popular protocols and services. Blaming just Rackspace seems unfair. Last week in was Namecheap. The week before it was someone else, etc. DNS DDOS is just a non-trivial problem to solve.
This should also be a reminder to have more than one DNS provider in your domain record. A backup nameserver from a different provider saves you and your customers a lot of heartache.
I'm not up to date on what that number is these days, but a few years ago 100Gbit/s was the target. For an Anycast DNS service, you don't need that in every site -- if you're willing to take small sites offline and degrade your service's end-user performance in this scenario. You will still need it in 2-3 sites, probably located in US west / US east / EU west because of the cost of datacenter space and the availability of cheap connectivity.
Serving 100Gbit/s requires 20 10G circuits in each site (~$15-30k/mo), 80 servers (2-3 racks, so $5-10k/mo), and a not-so-cheap router.
So your monthly opex is $20-40k per site, $60-120k worldwide. Your capex is $450k per site, $1.35mm worldwide.
It's not an insane amount of money, but it is kind of a lot to spend on DNS. It's particularly tough to justify at a lot of companies that you need this kind of buildout, when normally you can operate on 1/20th that amount of hardware.
The real point I want to make is that the only companies really putting this amount of money into DNS infrastructure are DNS providers themselves. I know of no hosting provider/ISP/etc. who puts nearly this much thought or effort into DNS.
Even for myself - and I consider myself a DNS junkie - my anycast cluster was a dozen or so servers worldwide co-located w/ a mid-market CDN provider (plus a few locations from my own ASN). That setup was greater than 99% of DNS installs out there - but it would have fallen over under a major attack. It truly takes some skill and financial commitment to build out correctly.
1. DNS hosting companies (Dyn, Amazon, etc.) 2. Large Internet companies (Yahoo, Google, etc.)
And for sure not all of them! A former employer's public zone was hosted by a DNS hosting company that didn't use Anycast, didn't have a footprint outside US/EU, and seemed to have little to no idea when or why traffic spikes happened.
So to do DNS really right you need a fair amount of money, some specialized expertise and knowledge of some non-DNSy things (Anycast, among others). I'm really not interested in hosting my own (public) DNS these days at work, it's just not worth the time and effort until you get gigantic. As with e-mail, there are specialists who can do it better than (most) places can.
I do kind of worry about depending on one DNS provider, so I've tried to use two when possible. That's a whole other world of hurt though.
So for example (very high-level) you might have a server in the US, and one in Europe. If your are eating a DDoS attack maybe the botnet has tons of US-based IPs, but very few euro. In that scenario, you will have all your US customers down but the EU guys are just fine.
That's pretty simplistic though. It's less by country, more by ASN. And it's also a case of automation killing you. What happens when the DDoS attack takes out a single PoP worth of DNS servers? Usually you have some automated tooling that will take down that PoP entirely, and withdraw the announcement. You want this behaviour in almost all cases. However, now that attack traffic just got sent to the next-closest server (in this example EU - taking down the whole worldwide cluster).
So it's not as simple as having "100gbps worth of inbound DNS capacity" - you need to spread it out properly, and of course deal with attacks like you would otherwise.
The other problem is it's UDP - so spoofed packets and the like are a much bigger problem.
It's a tough call to say if it's better to protect the rest of the network by trying to isolate the attack, even though that means you are totally down in one region. I'm inclined to keep 1/2 running than to have nothing, but if I have this release valve of moving traffic to the farms designed to soak up abuse traffic then it makes my decision easier. I should be able to get back up worldwide, if I've done my provisioning properly.
One of the nice things about this kind of attack is that your attacker has no choice but to follow your routing, whereas if you use your DNS GSLB to re-route an attacker away from your HTTP server farm (to handle a different sort of attack), they may not stop hammering on its IP address. After all, it's pretty unlikely that they respect DNS TTLs.
This approach is simplistic, but it's basically the only solution. The nature of a DDOS attack is that you cannot predict its origin, beyond the distribution of compromised client machines around the world. E.g. your attackers are pretty unlikely to come en masse from Africa.
Spoofing is an issue. Not much to do about that at your farm level, but you can start working with your upstreams to get ACLs in place to prevent some portion of that. And if the attack is prolonged and sufficiently large, the only way you're going to stop it is to identify sources and get help from involved networks and ISPs.
Market cap and technical sophistication, esp at large scale, tend to become divergent. In other words, it is generally assumed that big companies do things poorly. That's the entire notion of startups.
I will say this: Dyn can probably do a better job of hosting your DNS than Rackspace, Rackspace is not focused on DNS, and I'm not aware that they even really do anything special except for running lots of big DNS servers. Even when I worked at Rackspace 10y ago, the size of their customer base made them a constant target for DoS attacks. Do you need Dyn? Also probably not.
Rackspace has patented DDos prevention technology called RiverGuard, but since being patented only illustrates that it is novel, and not that it is of any quality, that is also meaningless.
I will say that, even though I prefer as a past employee to work with other companies, because the things Rackspace is bad at are things they don't give a fuck about solving, they run a damned fine network.
Further, as a replacement for / alternative to a colo, it is irrelevant whether a company can run a good DNS infrastructure, because that's not what I'm asking them to do. Rackspace DNS is there for convenience.
Cloud providers like AWS use their own cloud infrastructure (we think) to host services like Route53, which are also not infallible, but tend not to _entirely_ go down like this. Rackspace, while being somewhat competetive in the cloud space, doesn't operate anything like Amazon, doesn't do anything in a very distributed manner, and is backing those cloud offerings with what I expect to be vertically scaled DNS servers on top-end Dell hardware behind some kind of load balancers.
Money doesn't make you good at things, rather it helps you to not have to worry about the details, so it is inversely true that a company with a substantial market share should be expected not to make the same mistakes as the little guys. They're operating all day, all week, all year under the fallacy that their scale is proof of the quality of their work, and following this logic, they don't have to do any work at all. ;)
All we know is that Rackspace has dns servers in three out of its six datacenters[2], and that they appear not to have always-on DDoS protection in place.
Market cap and technical sophistication, esp at large scale, tend to become divergent.
You say this, but go on to compare Rackspace unfavorably to Amazon who, at 141.93B (of which AWS is just a part), has over 20x Rackspace's market cap and probably 10x the infrastructure[2][3]. Big companies do things poorly, but then it's hard to have highly-available dns without having lots of infrastructure.
Namecheap, Route53, Dyn, DNSimple, and now Rackspace have suffered DDoS attacks with outages of varying severity. Don't act like DDoS prevention is a solved problem and claim technical incompetence.
1: http://www.thewhir.com/web-hosting-news/web-host-rackspace-o...
2: http://www.rackspace.com/about/datacenters
3: http://aws.amazon.com/about-aws/global-infrastructure/
http://aws.amazon.com/route53/
It was not a good start on a Monday. And it was even harder to explain that your websites are up, but not up to a client lol, and there is basically nothing I can do.
So yah this marks the point at which I will be using R53 and RS DNS servers.
The "no more than two nameservers" rule means that those attempts will always span at least two providers.
Why don't DNS clients fail back to use the most recent good IP address as a default?
Indeed how can I set up my Kubuntu desktop to have this behaviour. Most domains probably point to the same IP for years at a time I'd imagine. If that's true it seems very strange that just because example.com's DNS server was offline this one day out of the last 600 that my browser can't "guess" that "123.45.67.89 example.com" is going to work.
You'd need to add some security around that to avoid abuse I'm sure but doesn't it seem reasonable? Shouldn't DNS failures only prevent you from getting to a server if the IP has changed??
It's easy to forget that you can have redundancy in your load balancers, web servers and databases (replication, multiple data centres, etc), but DNS is how you're found by the rest of the Internet.
No DNS resolution = no one reaches your expensive, lovingly-crafted infrastructure.
No DNS doesn't always mean no one reaches your expensive infrastructure. That may be true for websites relying on lots of random traffic, but most if not all of our customers have been to our site before so there's a strong chance that the DNS has already been cached on their computer or router.
"you have to change nameservers at the registrar"
Would add that that this is also one of the reasons not to use your registrar for DNS, as a generality.
This affected everyone who uses CloudFront (which is a lot), including Amazon.com itself. There, all product images were timing out.
https://news.ycombinator.com/item?id=8665367
http://www.forbes.com/sites/benkepes/2014/11/26/in-response-...
I have DNSmadeeasy and they seem to do okay, but only because they're such a small player and avoid being targetted too often.
Sorry, but there's no silver bullet here. Cloud providers get hacked and DDOS'd all the time. Kiddies and morons find it amusing to do so. Roll your own if you want to avoid being attached to such a big target and have a cloud provider just be your secondary namesever.
What they do unfortunately is charge you for the queries but there is no way of knowing which host has exceeded the limits query wise. When we first started using them they stated that even though we were exceeding the limits according to how they measure "not to worry you won't get charged". Of course perhaps 1 to 1.5 years later they now do charge (email out of the blue one day) and constantly try to bump us to a higher level service. [1] All this for a few zones that use to run comfortably on a 2 servers that we had many other things going on as well as DNS. (Now they claim we get 2.5 million queries per month but there is no way to determine exactly if that is true or for what host, ie foo.domain.com vs. www.domain.com is causing the excess queries).
[1] Which is more than paying the overage charges which is what we do every month.
We recently went through this with another provider and wrote about it (http://blog.papertrailapp.com/dns-outage-on-monday-december-... the gist: "Relying on one DNS infrastructure, no matter how large or distributed, is an unnecessary risk"
https://github.com/rackspace/pyrax/blob/master/docs/cloud_dn...
You could also do it via CURL: https://community.rackspace.com/products/f/25/p/1743/4945#49...
Sure, service X might be able to block a 50 foobit attack but what about when the next vulnerability is found and they can launch 500 foobits of DDoS?
Said the guy on the Hacker News website.
Having revision-control is wonderful for history-tracking.
But it seems like I can be significantly simpler and more likely to be reliable if I just get triggered by a webhook. I think I'd be willing to do it for a couple of people as a trial but I'd not expect so much interest.
Your primary DNS provider should allow automatic zone transfers. This makes it so that any changes you do to your primary service gets propagated to the secondary service within seconds.
Once setup you'll automatically have redundancy incase the primary provider starts timing out.
https://status.rackspace.com/