"As changes accumulate in the root zone, the difference between the versions begin to matter more. Typical changes in the root zone are changing delegations (NS records), rotating DS records for DNSSEC updates and with all this comes updates to DNSSEC signatures. If the differences remain, then the outdated root server will see the DNSSEC signatures expire and that will begin to have more marked consequences."
I mean, the specific "more marked consequences" are unspecified, but I think "could affect stability" is a reasonable interpretation thereof, and I do personally care that they don't know wtf caused it yet, so...
But Cogent do know what happened. The article ends with a quote from them:
> On May 21 at 15:30 UTC the c-root team at Cogent Communications was informed that the root zone as served by c-root had ceased to track changes from the root zone publication server after May 18. Analysis showed this to have been caused by an unrelated routing policy change whose side effect was to silence the relevant monitoring systems. No production DNS queries went unanswered by c-root as a result of this outage, and the only impact was on root zone freshness. Root zone freshness as served by c-root was fully restored on May 22 at 16:00 UTC.
"2024‑05‑23 - On May 21 at 15:30 UTC the c-root team at Cogent Communications was informed that the root zone as served by c-root had ceased to track changes from the root zone publication server after May 18. Analysis showed this to have been caused by an unrelated routing policy change whose side effect was to silence the relevant monitoring systems. No production DNS queries went unanswered by c-root as a result of this outage, and the only impact was on root zone freshness. Root zone freshness as served by c-root was fully restored on May 22 at 16:00 UTC."
Edit to add:
This was mentioned on Tuesday on the dns-operations list:
Internet stability is not threatened by the loss of a single root server, it’s not even a blip on the radar. In fact most people wouldn’t even notice if they all went down for a few minutes, DNS operators do not actually use these servers directly. You can even run your own if you are crazy enough.
Everyone likes to meme on DNS but it’s literally designed to make these servers as unimportant as possible. It’s a really well designed system that will probably be around as long as humans exist.
> that will probably be around as long as humans exist.
My goodness, that's a strong claim to make, about a glue layer in a computing technology that's been around for just 50 years. I mean, arguably the internet as we knew it is already dying. The original design of DNS was flawed (no security), and the patches to fix those flaws are far from perfect.
Replacing Internet infrastructure is incredibly difficult. We'll all be switching over to IPv6... any decade now.
DNS was flawed, but it was very good for the time, and it still is the keystone around which the entire Internet functions. That's pretty incredible for a system that has indeed received a lot of patches. And DNSSEC might turn out to be just as well designed as the original DNS. Besides, what are we going to replace it with,
namechain? I'll put you on hold while all the ISPs switch over.
Essentially, it generated 3 random, single-label (so, "TLD") queries, and ran them through DNS. Whatever recurser that lands at obviously wouldn't have random junk in its cache (unlike probably any other root record) and thus had to go to the root servers for it.
They did this to try to not display an omnibox result for single words (the example in the article is "marketing") on what I'm going to just call "broken networks"; see TFA if you're curious.
Note that running your own mirror of the root on the recursive nameserver(s) on your network will keep the traffic generated from these sorts of queries off the rest of the internet.
If you're already running your own recursive resolver on reasonably up-to-date software it can be surprisingly easy to set this up; see rfc8806 for some config examples.
Per rfc8806, because the data in the root zone has long TTL values, most queries to the root servers are for nonexistent top-level domains due to typos and search paths gone wrong, so in some sense it's community service to reduce useless traffic to the roots -- you do a single burst download of about 2MB daily instead of all of your query traffic.
Other reasons why you might do it are covered in rfc8806 - slightly reduced latency on cold-cache DNS queries, added resilience against DoS attacks against root DNS servers.
Honestly the issue here isn't that they let the zone get stale. The root zone doesn't change that much. So a couple of TLDs had to hold off on making changes, who cares.
The issue here is that they didn't notice until someone with a direct contact at Cogent emailed them to tell them.
A corollary to that issue is the fact that the last news item is the same thing: they failed to get updates and their monitoring failed to tell them. Supposedly their monitoring was fixed then, in 2019. Why should we believe it's any more fixed now?
The glitch began on the 18th and is now in the situation where some users can and others cannot access the site. I, myself, cannot access the site six days later, when connected via my network wifi router (ISP is Cox Communications), but can, if instead I connect to the Hotspot on my IPhone(AT&T). Traceroutes fail in either case, continually, at the server 198.207.200.70 (which does not appear to be owned by cogent).
What, if anything, can an end-user such as myself do in such a situation. Where can such an end-user send a report that might lead to a fix? I assume the owners of the indilib.org site are trying whatever they can, but I have no idea what their reporting chain might be.
What's your evidence that this is related to c.root-servers.net serving a stale version of the root zone?
If you can get the correct IP address for a site out of the DNS but can't reach the site, it's not a DNS problem, and not a problem with the root DNS servers. This is likely correlated only in time with the c.root-servers.net issue.
That said, your difficulty reaching indilib.org may be related to the Cogent-TATA peering dispute or one of Cogent's other peering disputes - from where I sit, a traceroute to indilib.org goes through cogent's network. But that's not what this submission is about.
The biggest issue here is a total lack of monitoring of root zone sync among all 1840+ root server instance, where an issue is detected accidentally by a random dude.
Huge oversight from ICAAN and ARIN. I know they are behind anycast, but each server must also have an unicast address for management.
35 comments
[ 3.4 ms ] story [ 64.0 ms ] thread"As changes accumulate in the root zone, the difference between the versions begin to matter more. Typical changes in the root zone are changing delegations (NS records), rotating DS records for DNSSEC updates and with all this comes updates to DNSSEC signatures. If the differences remain, then the outdated root server will see the DNSSEC signatures expire and that will begin to have more marked consequences."
I mean, the specific "more marked consequences" are unspecified, but I think "could affect stability" is a reasonable interpretation thereof, and I do personally care that they don't know wtf caused it yet, so...
> On May 21 at 15:30 UTC the c-root team at Cogent Communications was informed that the root zone as served by c-root had ceased to track changes from the root zone publication server after May 18. Analysis showed this to have been caused by an unrelated routing policy change whose side effect was to silence the relevant monitoring systems. No production DNS queries went unanswered by c-root as a result of this outage, and the only impact was on root zone freshness. Root zone freshness as served by c-root was fully restored on May 22 at 16:00 UTC.
"2024‑05‑23 - On May 21 at 15:30 UTC the c-root team at Cogent Communications was informed that the root zone as served by c-root had ceased to track changes from the root zone publication server after May 18. Analysis showed this to have been caused by an unrelated routing policy change whose side effect was to silence the relevant monitoring systems. No production DNS queries went unanswered by c-root as a result of this outage, and the only impact was on root zone freshness. Root zone freshness as served by c-root was fully restored on May 22 at 16:00 UTC."
Edit to add:
This was mentioned on Tuesday on the dns-operations list:
https://lists.dns-oarc.net/pipermail/dns-operations/2024-May...
dig @${server} . soa
c.root-servers.net showed an older serial number than all the others yesterday.
This seems like a major hole in their monitoring systems.
Everyone likes to meme on DNS but it’s literally designed to make these servers as unimportant as possible. It’s a really well designed system that will probably be around as long as humans exist.
My goodness, that's a strong claim to make, about a glue layer in a computing technology that's been around for just 50 years. I mean, arguably the internet as we knew it is already dying. The original design of DNS was flawed (no security), and the patches to fix those flaws are far from perfect.
DNS was flawed, but it was very good for the time, and it still is the keystone around which the entire Internet functions. That's pretty incredible for a system that has indeed received a lot of patches. And DNSSEC might turn out to be just as well designed as the original DNS. Besides, what are we going to replace it with, namechain? I'll put you on hold while all the ISPs switch over.
It's still very good. People keep finding new ways to apply DNS.
I was just grumbling about the OTT claim that the DNS will still be there as the last human becomes extinct.
I wouldn't mind some alien technology taking over for TCP/IP but that's squarely within the "unforseeable" IMO
No, but a lot of servers do. In 2020 the root servers saw 120 billion queries a day, which was double what they saw in 2018:
https://blog.apnic.net/2020/09/28/scaling-the-root-of-the-dn...
It's probably closer to half a trillion queries a day now.
For those that are wondering like me: https://blog.apnic.net/2020/08/21/chromiums-impact-on-root-d...
Essentially, it generated 3 random, single-label (so, "TLD") queries, and ran them through DNS. Whatever recurser that lands at obviously wouldn't have random junk in its cache (unlike probably any other root record) and thus had to go to the root servers for it.
They did this to try to not display an omnibox result for single words (the example in the article is "marketing") on what I'm going to just call "broken networks"; see TFA if you're curious.
If you're already running your own recursive resolver on reasonably up-to-date software it can be surprisingly easy to set this up; see rfc8806 for some config examples.
Other reasons why you might do it are covered in rfc8806 - slightly reduced latency on cold-cache DNS queries, added resilience against DoS attacks against root DNS servers.
The issue here is that they didn't notice until someone with a direct contact at Cogent emailed them to tell them.
A corollary to that issue is the fact that the last news item is the same thing: they failed to get updates and their monitoring failed to tell them. Supposedly their monitoring was fixed then, in 2019. Why should we believe it's any more fixed now?
I know this similar scenario happened to our lab setup of root servers.
https://www.indilib.org/forum/general/14691-indilib-org-site...
The glitch began on the 18th and is now in the situation where some users can and others cannot access the site. I, myself, cannot access the site six days later, when connected via my network wifi router (ISP is Cox Communications), but can, if instead I connect to the Hotspot on my IPhone(AT&T). Traceroutes fail in either case, continually, at the server 198.207.200.70 (which does not appear to be owned by cogent).
What, if anything, can an end-user such as myself do in such a situation. Where can such an end-user send a report that might lead to a fix? I assume the owners of the indilib.org site are trying whatever they can, but I have no idea what their reporting chain might be.
If you can get the correct IP address for a site out of the DNS but can't reach the site, it's not a DNS problem, and not a problem with the root DNS servers. This is likely correlated only in time with the c.root-servers.net issue.
That said, your difficulty reaching indilib.org may be related to the Cogent-TATA peering dispute or one of Cogent's other peering disputes - from where I sit, a traceroute to indilib.org goes through cogent's network. But that's not what this submission is about.
Huge oversight from ICAAN and ARIN. I know they are behind anycast, but each server must also have an unicast address for management.