HN Explorer

Massive Dyn DNS outage

391 points by jtmarmon ↗ HN
Sites down:

- DYN

- Twitter

- Etsy

- Github

- soundcloud

- spotify

- heroku

- pagerduty

- shopify

- intercom (app, not landing page)

Note that if these sites seem to be up to you, it's likely that your machine has cached the DNS response for these sites.

Some of these sites seem to work when using a UK VPN

301 comments

[ 3.7 ms ] story [ 442 ms ] thread
scrollaway
github.com itself isn't resolving in some parts of the world. status.github.com is a beautiful default nginx 404 page.
[deleted]
(comment deleted)
kchoudhu
Reddit is out -- my productivity is up, and I'm not happy about it.
[deleted]
(comment deleted)
cloudaphant
Working here. Try a non-US VPN if you want to stop doing work again.
kchoudhu
Ran my VPN ansible script on a London-based VPS. I'm back to not working!
zhemao
Reddit is up for me, but Github is down. Goodbye productivity.
num3thod
No parts of Github are up for me (Ottawa). Over to you, Bitbucket.
mikemcquaid
GitHub employee here. We're monitoring an incident with our upstream DNS provider:

https://twitter.com/githubstatus/status/789433336083001344

jtmarmon
not very useful to post a status update on a site that is experiencing the same issues with the same DNS provider.

maybe post a github gist? oh wait...

cylinder
Should there be a global internet status page at an easily memorizable "vanity" IP address?
planteen
Seriously, just like the 8.8.8.8 name server. I used to work at a place where the firewall blocked accessing pages by IP though.
alfalfasprout
Hahahahaha you do realize Twitter is one of the affected sites, right?
mohsinr
Exactly, I see tweet in Google SERPS for GitHub , I try to access and twitter too is down like Github.
benhoyt
Can we add a line to our /etc/hosts file with a hard-coded DNS entry to fix it in the meantime? What's the IP address?
eternalban
I looked up an ip for twitter (via a google search) and still timeout on pings.

[edit: twitter and github are both accessible again.]

DeGi
For me it resolves to 192.30.253.113
askmike
For me (EU) github.com resolves to 192.30.253.112.
[deleted]
(comment deleted)
kempbellt
Added these to my hosts file. Working fine as a temporary workaround.

199.16.156.70 twitter.com

104.244.43.231 abs.twimg.com

104.244.43.231 pbs.twimg.com

192.30.253.113 github.com

151.101.24.133 assets-cdn.github.com

be5invis
github homepage is unicorn now
facorreia
> our upstream DNS provider

Maybe you should have more than one. Then I could actually carry out my work...

drinchev
CircleCI is also down. As well as github hooks to slack.
tair
Intercom as well :(
[deleted]
(comment deleted)
num3thod
I just realised how online adult entertainment has the most redundancy of any Internet service category, bar none.
deno
You’re right :)

  $ dig @8.8.8.8 ns +short pornhub.com
  ns2.p44.dynect.net.
  ns3.p44.dynect.net.
  ns4.p44.dynect.net.
  sdns3.ultradns.net.
  sdns3.ultradns.com.
  sdns3.ultradns.org.
  sdns3.ultradns.biz.
  ns1.p44.dynect.net.
PornHub has better DNS redundancy than Github and Twitter.
drmunklung
I'm sure this is to circumvent blacklisting rather than for this scenario.
deno
I can’t think of any type of blacklisting this would prevent. Care you elaborate?
dtoombs
Amazon does too:

  C:\>nslookup

  > set type=ns
  > amazon.com

  Non-authoritative answer:
  amazon.com      nameserver = ns3.p31.dynect.net
  amazon.com      nameserver = pdns1.ultradns.net
  amazon.com      nameserver = ns4.p31.dynect.net
  amazon.com      nameserver = pdns6.ultradns.co.uk
  amazon.com      nameserver = ns1.p31.dynect.net
  amazon.com      nameserver = ns2.p31.dynect.net
gamache
Important: PagerDuty.com seems affected by this outage. So keep a real good eye on your graphs today -- you might not receive the alert.
sn
pagerduty has switched to AWS + 4 different root TLDS:

$ host -t NS pagerduty.com

pagerduty.com name server ns-219.awsdns-27.com.

pagerduty.com name server ns-739.awsdns-28.net.

pagerduty.com name server ns-1198.awsdns-21.org.

pagerduty.com name server ns-1569.awsdns-04.co.uk.

I think I'm satisfied.

gamache
I'm not trying to single out PagerDuty in being vulnerable. They run a particularly crucial service, and I'm sure they are doing everything they can to get out from under this.

That said, I still can't load www.pagerduty.com in a browser right now. :/

james_a_craig
Seems to be fine from the UK, so those of you with suitable VPNs might like to try that.
liveoneggs
a bunch of services using fastly are also impacted
CarVac
launchpad.net seems to be down.

edit: Using Google public DNS fixes things.

45h34jh53k4j
This is going to be a fun day. This little DNS outage is likely to cause millions of lost revenue for many industries.

Im dead in the water and I cant complain on twitter :-(

6d6b73
change your name server at your registrar to something else, add all necessary entries in the new DNS and be up before DDoS is stopped.
drinchev
Sadly we're too interconnected. Every company that relies on that DNS should do what you suggest, but the control is definitely not in our ( users ) hands.
6d6b73
Yeah, users are screwed. Unless they have a little more experience with how unreliable cloud can be, and they made a local copy of everything* their work depends on, just in case.

*Everything that can be local.

45h34jh53k4j
nah our corp dns is fine, its all the cloud services we and everyone else uses. Thank Sergey and Larry their stuff still works
exolymph
I found out Twitter was down because I wanted to go complain about PayPal, lol.
ljosa
We switched from Dyn to Route53 a few weeks ago … lucky.
6d6b73
All this talk about redundancy, real-time apps, scalable architecture and and a "simple" DDOS against DNS architecture brings half of the internet down. Honestly did nobody think about having a spare dns at some other company? or even backup dns server exactly for a situation like that?
ljosa
The TTL for the glue records of a .com domain is 48 hours, so even if you have Route53 set up and ready to go, it takes a long time to switch the zone away from Dyn.

We switched from Dyn to Rout53 a few weeks ago. It took about 12 hours before half of the traffic had shifted over.

6d6b73
That's the reason to have your DNS at at least two different companies, working in tandem. In a case where one is down, your Unicorn Corp doesn't go down with it.
St-Clock
How does that work in practice? Even if I set NS records pointing to two different DNS providers, I don't think a DNS client would automatically switch and retry if one is too slow to respond/times out.
6d6b73
Why not? That's the whole idea behind having more than one NS server isn't it?
acdha
Most DNS resolvers will automatically try each NS record until they get a response. That might be your ISP rather than your iPhone but that's an old practice going back to when the internet was even less reliable because some random Sun box under someone's desk failed.

Modern web browsers will also do the same thing if a query returns multiple A records and they get a connection error.

acdha
Exactly: there's nothing wrong with only using one provider if you're not willing to pay for two services but if you can't afford downtime you really need active diversity all the way down.

Route53 uses a bunch of different top-level domains for the same reason – if someone does manage to take the .com servers offline you'll be glad .co.uk is run by a separate organization.

tuke
N.b., unless things have changed, I don't think Dyn (for one) allows secondary DNS. Maybe that has changed. {?}
bhaak
From where I'm looking at the internet (central Europe), I don't notice anything.

Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-)

Edit: Looks like the eastern part of the USA is affected: https://cloudharmony.com/status-for-dyn

tzaman
Not really, I can't access our production servers which are in US east. Can't access Intercom with which we provide customer support. Our clients are mailing us that payment provider doesn't work either. So we're losing money while being in central EU.
a3n
> Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-)

Works on my continent.

551199
I'm in Europe now and worked fine something like 30 min ago.

Now facing similar issues reported in original post.

ibjhbrown
Things are down here on the west coast too...
pamvegas
I'm in Las Vegas and I haven't had access to several sites all day. I don't think this is limited to eastern U.S.
xXCrusherXxFTW
Same Im in Illinios and I can't play destiny which is made by bungie which is down for some fucking reason
rotauserch
they need a round robin dns that is geographically dispersed
mancerayder
If ONLY they had 10X developers ...

EDIT: joking aside, the issue with multiple DNS providers is primarily (in my experience at the company I'm at having investigated this in the past) intelligent DNS entries. Example, 'return these A records, in this order, based on the number of requests, roughly balanced'.

There's no universal standard, just common aspects. DNS Provider A has one set of features, names for returning A records in some way, based on some weighted averaging, provider B has a different mechanism. So as an infrastructure person you have to:

1. Investigate provider A and B features for intelligent DNS (it's not even universally called intelligent DNS!), and mentally parse the commonalities and differences

2. Create a custom mechanism to keep them in sync internally. So you hope that A and B have an API for maintaining the records.

3. Ensure that when someone in your org wants to make an update, A and B update at the same time in the same way

4. APIs don't change.

dsl
In my experience Dyn pushes this functionality very hard during customer on boarding, effectively as a lock in for their platform.
mancerayder
Well I really think those features are useful, though. They let you get some powerful de-facto load balancing and HA and other complex 'features' without any new hardware or much complexity. They give you some easy creative freedom in designing a redundant infrastructure, give you some geo-aware DNS the likes of which I remember paying Akamai a lot of money for years go, etc. But the lock-in problem is absolutely right.

I'm also familiar with the aggressiveness of that sales team. I prefer another provider and they were trying to solicit our business by specifically calling out our provider as amenable to a DDOS attack which had occurred.

Someone with a simpler setup with standard DNS features will find it much easier to use multiple providers, of course.

BlackGuyCoding
agreed 100%. This kind of thing shouldn't happen and this widespread. It's like no one is preparing for worse case scenarios until AFTER something tragic happens.

On the West Coast and I just lost twitter/soundcloud/github - 9:40 am PST

karmajunkie
loss of twitter, soundcloud, and github is what passes for tragic these days?
exolymph
Try PayPal. That's gonna be a big issue for everyone who uses their payment processing.
oxtoacart
As a programmer, the loss of GitHub is a pretty big deal in terms of work productivity, and as a maintainer of production systems that rely on assets from GitHub, it's a pretty big availability issue.
karmajunkie
I mean, i get it, its a big deal to us. But tragic? Syria is tragic. Github is just above annoying.
waterhouse
Let's say that it's tragic for the people whose job it is to maintain those services.
sphinx65
Yeah, as if a "spare DNS" has anything to do with solving the problem. You could get into your own infrastructure, but your customers are still screwed.
xtracto
Maybe it is time to widely adopt namecoin ?
blazinDC
Yes!! For sure, after such an attack it will force the internet into rethinking their DNS strategies. Why commit to having a single point of weakness?

For those interested: https://namecoin.org/

123456789101112
..and they thought war happens only with guns and tanks. don't the know, digital is the new platform to kill each other?
johansch
Works fine from Asia.

(When asking e.g. ns1.p34.dynect.net directly.)

leabdalla
zendesk admin panel is down here in brazil too
mugsie
From Ireland all of the above are resolving (with no cache).