HN Explorer

Some .io nameservers are returning wrong results again

192 points by JelteF ↗ HN
At my company (getstream.io) we've been having issues when resolving our domains randomly. After some more investigation it seems some .io nameservers are returning bad results. Specifically ns-a2.io and ns-a4.io. A correct result for crates.io looks like this:

  $ dig crates.io @ns-a1.io +norecurse

  ; <<>> DiG 9.11.2 <<>> crates.io @ns-a1.io +norecurse
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18874
  ;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 3, 
  ADDITIONAL: 1

  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 4096
  ;; QUESTION SECTION:
  ;crates.io.			IN	A

  ;; AUTHORITY SECTION:
  crates.io.		86400	IN	NS	dns3.easydns.ca.
  crates.io.		86400	IN	NS	dns1.easydns.com.
  crates.io.		86400	IN	NS	dns2.easydns.net.

  ;; Query time: 83 msec
  ;; SERVER: 194.0.1.1#53(194.0.1.1)
  ;; WHEN: Wed Sep 20 15:33:13 CEST 2017
  ;; MSG SIZE  rcvd: 127
A bad one looks like the following:

  dig crates.io @ns-a4.io +norecurse

  ; <<>> DiG 9.11.2 <<>> crates.io @ns-a4.io +norecurse
  ;; global options: +cmd
  ;; Got answer:
  ;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 43223
  ;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 1, 
  ADDITIONAL: 1

  ;; OPT PSEUDOSECTION:
  ; EDNS: version: 0, flags:; udp: 4096
  ;; QUESTION SECTION:
  ;crates.io.			IN	A

  ;; AUTHORITY SECTION:
  io.			900	IN	SOA	a0.nic.io. noc.afilias-nst.info. 1497256201 10800 3600 2764800 900

  ;; Query time: 10 msec
  ;; SERVER: 74.116.179.1#53(74.116.179.1)
  ;; WHEN: Wed Sep 20 15:34:59 CEST 2017
  ;; MSG SIZE  rcvd: 105
This has happened last year as well, so I wouldn't recommend running something important on a .io domain: https://news.ycombinator.com/item?id=12813065

80 comments

[ 3.1 ms ] story [ 159 ms ] thread
cjsuk
We're seeing this as well.

Edit: it has now taken out our CI infrastructure and dev environments as it's caching NXDOMAIN records.

a13n
We're getting this as well.
dmgawel
It affects also our website.

Is there any example of other TLDs having similar issues in past?

JelteF
If this is affecting you it's best to put a long TTL (at least 1 hour) on your domains. That way a good lookup will be cached longer and it will affect less people.
cjsuk
Won't this also cache a bad record for longer like NXDOMAIN which is what we're getting?
JelteF
No, the TTL for caching NXDOMAIN is decided by the nameservers SOA record. In this case 900 (15 minutes). The reason for this is that if there's no domain there's also no way to get it's TTL.
cjsuk
Thanks for the clarification - much appreciated!
nodesocket
All of my .io DNS records have a TTL of 24 hours and 3600 (refresh-time) 900 (retry-time) 1814400 (expire-time) 60 (negative caching TTL). My PagerDuty is going bat nuts.
sandstrom
If the domain/zone records are handled by someone other than nic-io (say Route53), does that really help?

If I change the TTL for an A-record in Route53, how will that help if there is an NS-issue with nic-io?

(awesome if you could explain!)

Also, does the SOA and NS records have any effect here?

aeden
If their servers were unresponsive, then a long TTL might help as caching resolvers would retry and eventually get a successful result. Based on the output of the OPs tests, they are responding with incorrect NXDOMAIN results. This means that resolvers will cache the NXDOMAIN result using the SOA negative result TTL.

You could remove the delegation entries for the name servers that are returning results though, that might help.

X-Istence
nic-io would need to remove the delegation entries, since it is the .io servers that are returning the wrong response. This has nothing to do with any DNS servers actually hosting the <domain>.io zones.
JelteF
There's a random chance that a DNS request resolves correctly because not all nic-io domain servers are returning incorrect results. If you provide a high TTL on your own records they will be cached for a longer time. Which has as a result that there will be less requests going to the nic-io servers.

This still doesn't rule out bad responses. As far as I know there's no way to achieve that. However, it will make sure correct ones stay active longer.

A high TTL on your NS records would be good as well for the same reason. On the SOA it shouldn't really matter.

benmmurphy
.io seems to always have problems. its really sad that lots of people use this domain because it looks cool :( [at work we use this as well]
leftnode
Yup, seeing this as well for one of our .io domains.
jlukic
Wanted to confirm same issue for my .io domains. This has gotten a lot of people up this morning.
obeattie
Also seeing this affect several of our .io domains.
[deleted]
(comment deleted)
msipenko
same issue here
[deleted]
(comment deleted)
msipenko
.io had a similar issue beginning this year.
mittermayr
Customers are already filling up our support inbox, so same here. They can access nic.io

Any quick fix ideas I can get to a customer without technical command-line experience? Changing DNS or modifying hosts file is not really an option.

abofh
They boned :-/

In this case, think of it as if the guy who made your maps is giving you the wrong country's maps. It's well made, clear, concise and wrong. There's no 'greater' authority then the map maker, so unless you can reconfigure your users to believe in a different map (change hosts/dns), there's nothing that can be done.

Patience :(

dedene
Same issue here. Does NIC.io have a status page for their root servers somewhere?
inka
Same here. Started 14:20 UK time (13:20 UTC).
nodesocket
Confirmed ns-a2.io and ns-a4.io don't seem to be responding with DNS results.
dmuld
This affected my service as well. Where does the root of the problem lie, with the .IO service itself or somewhere else?
bhouston
We are getting some client reports of connectivity issues with clara.io. I suspect it is this issue but haven't been able to confirm.
[deleted]
(comment deleted)
[deleted]
(comment deleted)
adrienjarthon
Indeed I noticed the same issue for my customers using .io domains (and me unfortunately): https://twitter.com/updownio/status/910508435720065024

Seems to be getting better now. High TTL helps but mostly if it was there before so do it now for the next downtime (it's frequent with .io)

JelteF
The problem is not actually downtime, because DNS handles that automatically if at least one nameserver is up. The problem is nameservers returning incorrect results. But yeah, .io has quite frequent problems.
adrienjarthon
Yeah for a DNS server returning wrong results is basically a downtime to me
codegeek
I tried loading redis.io and got the network error. I thought it was just their site but looks like a bigger issue with .io.
tbarbugli
Seems to be fixed now
a13n
For us too, but not everyone. Seems like they're rolling out some kind of fix.
JelteF
The nameservers seem to be returning correct results for our domain again :) (getstream.io). However, the issue is still there for crates.io, so this seems the outage is still partially ongoing.
duellsy
We're copping the same at https://elev.io as well...

Timing couldn't be worse, we did a product release on ProductHunt earlier today :(

the_common_man
I am curious why you need a product hunt launch when you have over 500 companies using you? Are they all paying customers?
riffic
You should be aware how sketch some ccTLDs are and it would do you some good at this early stage to have a fallback domain.