IO domain DNS failure

I have been seeing the same failures for many IO TLDs

https://www.whatsmydns.net/#A/gitlab.io

Example: https://i.imgur.com/SU4BAHK.png

I have noticed the failures occurring moreso in Asia and Russia than in other regions.

There are 7 IO nameservers (ns-d1.io, ns-l1.io ,ns-y1.io, ns-a1.io, ns-a2.io, ns-a3.io). Only ns-a4.io and ns-a3.io respond.

We are experiencing the same from Vietnam

We are also having issues resolving some .io domains from AWS Sydney (ap-southeast-2)

We've been seeing this for exactly five hours now, all across East Asia.

Edit: Looks like .io is managed in London, and their customer service off hours match the downtime.

Having the same issue in Singapore. got nothing to do, helpless.

So, "ns-a4.io" is the culprit! On ns-a4.io, xxx.io can be resolved, but yyy.xxx.io cannot be resolved(delegated). Ref: http://qiita.com/MasahitoShinoda/items/40cd312fabc6db604b39

Same here in South Korea. Still not fixed.

Getting significantly worse now.

.sh and .ac are also run by the same folks - looks like there are issues with those tlds as well.

Two of their servers were sending out NXDOMAINs which are then cached by downstream forwarders. It seems they have turned those off now.

It looks, the the problematic nameserver "ns-a4.io" is down. So, the problem should be disappeared...

At the same time, it looks, there is only one "ns-a3.io" nameserver working for IO domain, at now...

I recieved this reply from nic.io support a few minutes ago: http://imgur.com/a/wV2Kk

This is still affecting us. Our customers on Heroku seem to be the worst affected as Heroku is unable to resolve the DNS

AWS response:

Hello,

Some customers have reported intermittent resolution issues with .io domains. We can confirm that Route 53 DNS services are operating normally at this time and these issues seem to be related to the .io top-level domain provider.

We've investigated the issue and are primarily seeing incorrect responses from two of the .io nameservers: ns-a4.io and ns-a2.io. These nameservers are returning NXDOMAIN intermittently for domains that do exist. As a result, once a resolver receives the erroneous response, it will cache the non-existence for the negative TTL, which for .io is set to 3600 seconds (1 hour).

One suggestion we have is to increase the TTLs for your domains in Route 53 so that the resolvers cache DNS answers for a longer duration.

Best regards,

(Customer support rep name redacted)

Amazon Web Services

We value your feedback. Please rate my response using the link below.

For Planio (https://plan.io/), all accounts are available via yourname.planio.net as an alternative while this outage lasts.

For those interested, we wrote a blog post about single points of failure with DNS last Friday when the Dyn attack happened. Our strategy ensures our customers are mostly unaffected by issues like this. See https://blog.ably.io/routing-around-single-point-of-failure-...