13 comments

[ 3.1 ms ] story [ 38.6 ms ] thread
Thanks for the detailed report, good lessons learned herein.
"Any request to .js file resulted in a valid javascript. This script was loaded and executed instead of Qbaka tracking script."

Who would do that?

"The TTL value of the corrupted DNS record was one week."

For a parking page? Who would do that?

I am almost sure that the reasoning for the redirect script was something like this: since they have ads on that "parking" page, whoever tries to access the server, should see the ads. That's probably why they have this terrible redirection script in place. Which still keeps the original question "who would do that", because this is terrible.

The TTL value has no direct connection to the parking page. It's probably by the coincidence that DNS record had a high TTL and pointed to that IP address. But no one can be sure here, because there is always a posibility that DNS was hacked, although the DNS hosting provider sais that DNS error was not a result of a hack.

This is exactly the sort of issue that's supposed to be eliminated by DNSSEC validation.

The registrar's database gets corrupted? They are not hosting my DS records, anyway. That's in the top-level domain. You will be the least of anybody's worries if the top-level domain dies. And as long as the DS record is live, a validating nameserver can tell that the faulty DNS server is sending incorrect resource records, and should be disregarded.

Side note: My DNS server is a hidden master, that publishes zones via secondary servers, run by separate companies.

The corruption would still be disruptive, but it would not be such a disaster.

This is exactly the sort of issue that's supposed to be eliminated by DNSSEC validation.

Emphasis on "supposed to be," because what DNSSEC brings to the table in practice is headaches and outages. You won't see stories on HN about the IETF, NASA or entire TLDs going down due to DNSSEC because it's normal, expected.

I was thinking of trying to set up DNSSEC for my domains. Are you saying that it isn't worth the hassle? What benefits / drawbacks does it bring?
It's yet another example of security technology that has too many adjustable knobs, is poorly documented, and fails subtly.

The main benefit is that it makes it much harder to spoof DNS responses. Qbaka just had an invalid zone published for their domain. The Great Firewall of China has been leaking fraudulent DNS responses around the world. There's the Kaminsky attack. This is a real-world problem. Also, some people hope that such verified DNS records will allow them to sidestep the certificate authority problem, but I think that's futile.

A side benefit I've been exploiting is that DNSSEC allows me to publish my domains on free secondary servers, confident that the secondary is not able to publish valid DNSSEC signatures on fraudulent records. I don't have a lot of domains, though.

The main drawback is that it has so many moving parts that can break if improperly used. Which signing algorithm and key size to use? Should you use NSEC or NSEC3? How and when do you rotate the keys? (Default: no rotate.) Which key rotation technique should you use? Where do you generate and store your signing keys? If Microsoft can't consistently remember to renew their DNS registrations, then how will they remember to update their DNSSEC keys?

Which brings me to a huge drawback: Now your DNS server and your DNS registrar are part of your security infrastructure. Do you trust your registrar to hold on to your DNS signing key? GoDaddy is happy to do so. If not, then you'll have to maintain your zone separately. And even if you maintain your own zone, both your registrar and the TLD administrator are free to publish an updated DS record claiming that your DNSKEY is invalid and theirs is the true DNSKEY for your domain.

Another drawback is that DNSSEC zones are huge. Dan Bernstein calls DNSSEC a DDoS bandwidth amplifier, because adding DNSSEC signatures to DNS queries makes the responses so much bigger. DNS servers should have response rate limiting, anyway.

So, there are very important benefits, and very big drawbacks. The way I see it, DNSSEC is horribly flawed, but it's the most widely deployed solution to fix certain real-world security problems, so we're stuck with it. At least it's designed to be backwards compatible and incrementally deployed, unlike the IPv6 fiasco.

Anyone know who the DNS provider was? This seems like a perfectly appropriate time to name and shame.
No point in shaming, anyone can experience this kind of problem. Even Google once marked all websites as "contains virus", and Amazon cloud was unavailable due to a storm. It was not the well known domain name registrar's DNS.
we cannot call every single internet provider around the world and ask them to drop DNS record from cache

Google allows users to flush a record from their cache https://developers.google.com/speed/public-dns/cache which is better than nothing, especially if you use Google's resolvers on your servers.

The Google DNS was not affected, or more likely it was quickly flushed (I guess, the guys who made this error were the first to flush Google DNS).
One semi-solution to that (which would have made the problem almost invisible to any webpage visitors) would be to only serve the script over HTTPS. Presumably the domain parking page was not using a cert that matched "cdn.qbaka.net", so a browser would fail to download anything due to a cert mismatch.
That was my original thought as well but since cdn.qbaka.net was just a cname to their real cdn it wouldn't have worked. Unless they gave their private ssl key to the cdn provider, which is really not recommended.