Weird, works for me - from Italy (not sure if there isn't just some caching going somewhere down the line and I can see it because of that)
edit: nevermind, it's almost certain i've got it cached
I can query the authoritative ns*.p16.dynect.com DNS servers from Europe (Germany in my case), and the traceroute looks like it's near Frankfurt. So the anycasted copies here seem fine.
If anycast routing is in play, which is not unlikely with a DNS service like that, then it may also be that specific servers are being attacked so the outages don't affect users in all locations as some will be routed to infrastructure that is not affected.
I'm surprised; I would have thought such large sites would use more than one DNS provider? I mean:
$ host -t NS twitter.com
twitter.com name server ns4.p34.dynect.net.
twitter.com name server ns3.p34.dynect.net.
twitter.com name server ns2.p34.dynect.net.
twitter.com name server ns1.p34.dynect.net.
I would have expected at least one of those to be somewhere else. What is the reason they would not have a backup provider?
I'd guess the reasoning is that DNS providers these days are all anycast-style DNS. A DDoS would usually just be a blip on a few servers around the world depending on where the attacks originate.
I'm not saying it's a good reason but it's a reason.
I know a lot about some things, but almost nothing about networking, so excuse me if this is a really dumb question but - would your physical location determine what hosts you returned from that query? Like if you were in Asia would you get different ones back?
All of these work for me from Germany, and querying their authorative nameservers works just fine (so definitely no caching effect). Anycast for the win!
Isn't it the whole idea of git to make user independent from place like github? After all if you have the repository on your local machine you can continue working as if nothing happened, right?
I'm so damn tired of the "host it locally" mentality. Not everyone has the resources to host all of that locally.
For example, most open source projects.
But even outside of that, we use github for issue tracking, the new project management kanban stuff, a CI server, reading documentation (which is offline, but the online versions are nicer on the eyes), and a ton more. Not to mention that StackOverflow and other discussion forums tend to be used by many.
Yes you can host all of that locally, but we don't have a few hundred thousand a year to spend on some sysadmins to maintain all of that, and we don't have the time or money to run the machines, vet the software, and keep it up more reliably than github does for next to nothing.
And I'm so damn tired of people complaining about cost to run stuff locally. The true cost of not having some basic stuff setup locally, even for backup purposes is when situation like this happens. It does not take long time or resources to download all of the libraries, with corresponding docs to a local server, or even your laptop. It is not complicated to have all of the new issues sent to an email to have a version of them available at all times. And you don't need a sysadmin to administer all of that.
>And you don't need a sysadmin to administer all of that.
I disagree with that. If you have a server, you need a sysadmin. End of story.
Who is going to secure the system and setup ssh keys? who is going to run updates? who is going to monitor for security issues? who is going to run backups? who is going to secure those backups? who is going to oversee the installation of the network, the battery backups, the racks, the server hardware, etc... Who will swap out bad disks? Who will recover the system when it goes down? Who is going to double the hardware and setup high availability (remember, you are competing with github for uptime here)? And god help you if you have one guy that does all of this. What happens if he gets hit by a bus?
An on-prem server isn't a "backup", it's a liability. And without the resources to maintain it, it's going to become a nightmare. I've been there, and I won't ever do it again.
I'm either going to pay to do it right, or give it to someone who will. And if that means a few hours of downtime every year or so, then that's a wonderful tradeoff for me.
> It does not take long time or resources to download all of the libraries, with corresponding docs to a local server, or even your laptop. It is not complicated to have all of the new issues sent to an email to have a version of them available at all times.
Luckily github (and alternatives) provide all of that. It sends us emails and slack messages on everything, so if it's down, we can still read, and we all have our local repos. But reading is different than working.
If you have any substantial business, you already have a sysadmin on your team. He's not doing his job if he has no local versions of almost everything that is online. He should be staging everything locally, before deploying to the cloud. The currently very popular way of deploying everything live, without any testing, or staging is one of the reasons behind current crappy state of the internet.
I disagree, with very large companies, you have no "local" sysadmin, and no local versions of anything. Especially if your IT department is actually its own company.
If self-hosted, somewhere, you could still be screwed by having Dyn as your DNS provider.
If dev-machine-hosted, then uh, your issue tracker is no longer an issue tracker. Your build server is not a build server. All the services besides Git are not meant to operate offline in a decentralized/distributed fashion.
Library documentation, sure, that could be local. Otherwise, your assertion that all of this tools infrastructure can somehow be replicated, easily, in a way that makes the difference between working online or offline effectively zero, is nonsense.
First of all I'm not saying that the whole infrastructure could be replicated, only critical parts, and parts that can be easily hosted locally.
Second, pointing to a new machine is as simple as updating IP in your hosts file, or dns server.
Third, you can use vmware or any other virtualization stack to replicate your infrastructure locally. In fact that's the best way to build things - create virtual network, use it for testing, troubleshooting and debugging, and deploy only when everything is working.
All I'm saying is that if you're company is making any kind of money, and your development environment depends 100% on online services, you're doing it wrong.
Not that I'm saying that you're completely wrong, but you're oversimplifying the problem and the solution to it. Your last statement is not necessarily true; you have to balance the cost and how much of a PITA it is to set up and maintain vs how much money would be lost by disgruntled customers for a single rare outage. Granted, it depends on the kind of business you are running, but not every business is so fragile. In fact, I'd say that most(as in >50%) are not that fragile. Even better when you can simply put the blame on someone else, which everyone can do in our case right now.
I'm under the impression that you say self-hosted ~= dev-machine-hosted ?
If that's the case, i think you're misguided : imho, the internet as it was designed was conceived so that everyone has its little self-hosted thing, with dev-machine just for the purpose of, well, test and dev, with the latter goal of it being self-hosted.
Just look how email is technically designed and how it was meant to work, and we use it now, relying mostly on Gmail or Outlook, or worse, using Facebook for emails : we put all our eggs in the same basket.
If I were running a business, here are the options as I'd see them:
Option A) Spend no money and experience an outage maybe once a year, if that. And the problem works itself out.
Option B) Spend money and gain technical debt to avoid a problem that happens maybe once a year, if that.
Which one would you pick? I mean, maybe if everything you have is closed-source or you are guaranteeing 99.9% uptime to your customers, perhaps option B makes sense. Otherwise, the choice seems fairly obvious to me.
What's your source that you "experience an outage maybe once a year" ??
I work in IT infrastructure and I see attacks literally every day, moreover, most people just setup a quick LAMP or MEAN stack to prove their concept, and then they leave it like that, so most of the time, no, the problem don't just "work itself out".
I had several, sporadic 'secure connection could not be established' yesterday while trying to open HN, amongst others. Painfully slow page load times across the board, too(Craigslist, Monoprice,weather.gov, etc) Still may be my buggy phone SIM...
Sorta. When I changed phones I cut my micro SIM down to nano size. Cut a wee bit too much off and it now can slide off contacts if jarred... gotta get a new SIM.
Since many (all?) of Dyn's authoritative server IPs are anycast, attack traffic is probably not well distributed either. If you're routed to a server that's getting a lot of attack traffic, you're likely to have problems, but a server without much attack traffic will work fine.
Because it's being down voted? I think any down votes are more likely because the comment doesn't add substantively to the conversation and is distracting given other recent threads.
um. if you are basing your logic on orwelianness then the England should be your your top candidate. massive monitoring of the populace, severe restrictions on the ability to carry anything that is vaguely pointy or goes bang, poor freedom of speech rules (relative to the US).... I love England but as far as orwellian societies go, you can do notably worse than the US.
As someone living in England, I can confirm that a) we are an extreme surveillance society, which the general population neither really understands, nor cares about b) the vast majority of us are very grateful we don't have (legalised) guns on the streets, and we suffer from a much lower homicide rate as a result
It doesn't make a whole lot of sense for the USA to take down the internet, as they benefit the most from it. A significant fraction of that economy is based on it, much larger than in the cases of China and Russia. It would be like the owner of a coal mine campaigning for a carbon emissions tax: maybe there's something we don't know, but from the information we have it seems unlikely.
Note that this wouldn't rule out the USA as such. First, it could be a longshot preparedness thing, with no expectation that it would ever be used. Second, they could be red-teaming the thing (looking for weaknesses so that they can arrange for them to be shored up).
In either of these scenarios, it's no less likely that the USA would be doing it than anyone else. If you assume that whoever is doing this is planning to use their knowledge, however, the economic argument makes the USA less likely to be involved.
For either of your cases, I can't imagine the value of having it last this long, or even be this severe. The impact on the economy and trust in a infrastructure company is too high.
I'd be more willing to put my money on someone attacking an entity downstream who is normally immune to DDOS attacks of this size.
There are many types of actors even within the USA nation-state / government.
For example, if a particular part of the government got wind of a data dump about to be released by another nation-state or independent actor (for example, a leak of some kind) - I think some parts of the USA government that possesses the ability to do so wouldn't hesitate to take down dns to the entire internet to avoid another similar data leak to the Snowden dump.
Be really wary of attributing intent: you do not know who will benefit the most from taking down certain services. To claim that the US benefits from the internet so much that it wouldn't do certain actions to protect itself from certain types of harm is shortsighted.
Even my example could be really wrong, but the idea is that nobody really can say - "oh the internet is too important to xyz, they'll never do anything!"
Or they could take it down for enough time for other parts of the government and/or international diplomatic system to do their work.
Remember, a data leak is not just a technical issue. They can resolve it in any number of ways - get a small team incursion into another state's territory for extraction, etc. All the outage needs to do is to hold open that window for enough time for all the different parts of the entire threat response chain to do each part's job.
A lot of technical people think tech is the end, but no - if you get a small team to go knock on the person's door, and get your internet response team to shut down dns, or to get someone on site at the telco to perform certain actions at the router/switch level, etc all portions working together is a powerful way to resolve or to accomplish certain goals.
Think bigger, especially with state actors - the resources are there, and this line of thought is probably really basic stuff that people came up with in the 1960's or 70's (even when the arpanet was being created, there was probably already a team tasked with taking such actions - it only make sense to have 2 teams working on such goals in tandem - one to create the network, the other to take it down)
I'm probably wrong, but this is how I see it (not sure about the OP).
News cycles happen fairly rapidly, so if you could take down a number of sites that might be friendly to the dissemination of potentially damaging information just long enough such that it's forgotten about, or the attack is so large the media talks about the attack instead, then you might be able to successfully avoid widespread public knowledge of such information. Though, this would be best aided with collusion or cooperation (intentional or otherwise) from the media. Toss in a few unrelated services as a bonus for collateral damage, and you might be able to avoid scrutiny or, at the very least, shift the blame to an unrelated state actor. It won't prevent the release of information, but that's not the point--you want to prevent the dissemination and analysis of that information by the public at large.
This is all hypothetical, of course, and not likely to work. It also comes with the associated risk that if you were discovered or implicated, public outrage might be even worse than if you allowed the release of the information you hoped to distract from in the first place! As such, I can't imagine anyone would be stupid enough to try.
If you take them offline before they've managed to disseminate the info, then it can't be forgotten because nobody knew about it in the first place. Which means when the sites come back on, the info is still newsworthy.
What kind of information would be so sensitive as to risk crashing the economy over, yet so trivial that people would forget about it because they couldn't access Twitter for a day? I get that it's more nuanced than that, but I'm really struggling with this scenario; sensitive information tends to get out if it's important enough, even if you're willing to kill a bunch of people.
It renders the server that's hosting a leak unable to broadcast the leak temporarily, while they arrange more conventional measures to seize it. It's a more rapid response than getting a warrant and a police team on location. The broad nature of the attack also avoids tipping off the server owners.
I still give it less than a 5% probability, though.
Honestly, that's fairly thin. WL uses torrents and other means of disseminating data that don't rely on central control structures. Plus, presumably, WL has the ability to quickly shift data into secure hands who are willing to release it when things quiet down.
So, sure, the USA could go send someone to sieze the hard drives of someone who has confidential information. But, I have to imagine one of the first steps when getting that kind of information is to disseminate it to others (at least some of whom are unknown to the states). If they were hit, these people would very quickly take that as a signal to indiscriminately release all the information.
Total and absolute speculation follows: If the US wanted reliable take-down capability, they might want to test it first, and it would be least provocative if they tested here in the US.
As for the length of the "test," they might want to see how the US would react to such attacks in the future, and shake out anything critical. "Oh, these two agencies can't talk to each other. Good to know."
The usual thinking goes something like; well, the US created the internet so why would they want to take it down? Yes, NSA spies and all that, but they need the internet up to do that and also as bad as NSA is, it's nowhere near as bad as China or Russia where they ... (ranges from censorship to eating babies alive)
And the ratio of false flag operations to false accusations of false flag operations is about 1:99. Lots of unlikely things "do happen," but if that's the immediate explanation you reach for you're going to be wrong most of the time.
You might want to take this up with that "rdtsc" guy who wrote "As someone else pointed out this is a classic false flag operation." He seems to disagree with you on those points.
Oh right, I know him! He is a decent fellar. I think he was saying if US is attacking its own infrastructure, then a false flag operation is a plausible explanation. Talking to him a bit revealed he didn't say this US attacking itself.
As someone else pointed out this is a classic false flag operation. Look up Gleiwitz incident and Operation Northwood. It can be very effective. With good opsec and anonymity online it can be even easier.
> kind of conspiratorial thinking?
You mean like lizard aliens infiltrating our planet? -No. But in the realm of "shooting down of passenger and military planes, sinking a U.S. ship in the vicinity of Cuba, burning crops, sinking a boat filled with Cuban refugees, attacks by alleged Cuban infiltrators inside the United States, and harassment of U.S. aircraft and shipping and the destruction of aerial drones by aircraft disguised as Cuban MiGs", yes.
It was mostly a reply to "US would have absolutely no reason for doing this" and the reply is there cold be a plausible reason.
Who is "we"? The u.s. government is a conglomerate of interests, organizations, individuals... Many of whom are quite indiscriminate. I'm not at all proposing that the U.S. was involved here. I'm questioning the simple identification of the U.S. government with the word "we", and the corresponding assumption that this institution is integrated in a carefully discriminating way...
From the context of the paper, because the USA could just send a three-letter-agency agent of some sort to Dyn, a US-based company, and ask what their infrastructure looks like? (Presuming of course some weird scenario where they weren't already tracking it, which seems unlikely.)
It's okay. James Comey, the FBI chief, said the US electoral system is such a mess, it would be too hard for an attacker to hack it or damage its integrity in any way. It's all good.
This is terrifying. Thankfully I don't think much actual voting infra is network reliant. But it could probably delay the results from being finalized for days, and allow Trump to spew further allegations of rigging.
Though if they targeted electric grid, water, and public transport, starting early in the day and choosing the regions by their populations political leaning, it could easily have an effect on the result itself.
Hahahaha. For sure it's not supposed to be network reliant. But from my experience working on critical infra, even things like power grids and rail systems, this is almost never the case.
You don't need to target voting infrastructure. You target media infrastructure (DNS, streaming, web media) in order to either reduce or shift voter turnout. A candidate ahead in a battleground state? You stomp on media reporting to ensure their opponent's voters aren't dissuaded from heading to the polls.
Control the message, and through that the actual votes cast.
Yeah it just needs to be "The internet was broken so your votes were lost" and then some made up post-hoc explanations that 90% of people don't understand so they can't dispute
I think you're right, not much of the voting infra is network reliant, but the more I think about it the more it seems that the "fear" factor of such outages could influence the election. Or, perhaps a curated working set of information sources, thanks to selective DDoS. Regardless, terrifying to be sure.
I'm working at the polls in CA, and can verify this; all critical information is moved by sneakernet with a two-person rule on its handling.
Of course, I have no information on the security model of the pre-election preparations and post-election tabulation, but luckily results for each polling place are also posted for the public to inspect - media outlets and campaigns can verify the tabulation themselves with a slight delay.
Assuming you're referring specifically to targeting media companies reporting on the results and not the electric grid like someone else mentioned, wouldn't they have to DDoS Google itself for that to work? I don't really see a DDoS of Google being effective.
That was exactly my thought. This may be unrelated, or it may be a test run. But a large scale attack on Election Day that crippled communications would stir up unrest for a variety of reasons. Although I think that's highly unlikely to change the outcome, unrest after such a contentious election is not good.
What can they do? It's not Twitter themselves being DDOS'd, it's a DNS provider. This propagates up the chain to impact both a Tier 1 network and cloud providers, which hits tons of stuff on top of that.
if you utilize geoip routing features of one provider, it can be difficult to impossible to then ensure repeatable/deterministic behavior on a second provider.
Is it really important who wins it there are only two candidates that share common view on many problems? And you don't need Internet to count votes anyway.
Probably not a great idea. If the internet went down at my work, none of us would be able to do anything, so we'd probably all head out to the polls just because we have nothing better to do. Unintentionally increased turnout.
From what I know of the situation (don't trust me, I'm not going to offer citations or sources), this attack wasn't particularly large in terms of gigabits/second. It was, however, very large in terms of economic impact.
I would assume that when a large number of big enterprise-y things go down, HSI takes notice. When other providers get attacks that are 20x larger (gbit/sec), but have much less widespread impact and impact on less enterprise-y things, they don't care so much.
- Hillary Clinton's personal email server was hacked a while ago.
- A lone hacker published a document obtained by hacking the DNC servers. The document includes opposition research on Donald Trump and how Hillary can attack him in the election.
- Wikileaks published emails obtained by hacking the DNC
- US intelligence agencies confirmed that Russia was behind the DNC hack
- This is happening while the war in Syria and Iraq is growing. The Russians are there to "fight ISIS" but they have deployed an air defense system even though ISIS doesn't have any air force.
- Russia's only air craft carrier is trespassing through UK waters to get to Syria in a show of force that doesn't really add anything to their military capabilities there.
- Finland (yes, Finland) is increasingly worried about Russia. They violated their air space, and they're questioning Finland's independence. Finland shares a long boarder with Russia.
- US election is in 3 weeks and Donald Trump is openly in love with Putin. Trump questioned the benefit of NATO which is the basis for Europe stability after the 2nd world war.
I just want to sell my software, why does everyone have to fight?!
Thank you for these links. I'm trying not to get wrapped up in conspiracies but am increasingly worried by the mounting conflict. I'd love to hear a calm, reasoned response from someone more knowledgable than me on these topics.
Yeah I've gotten a little wrapped up as well looking into all of this mounting tension between the US and Russia. Protip, stay away from /r/the_donald.
My personal opinion is that it is mostly political and I think (hope) that what is happening in Syria won't escalate to direct conflict between the US and Russia.
I stumbled across this little blog article the other day and it helped relieve some of my anxieties.
I don't actually think this will lead to open conflict. My comment at the end was just saying that this is what a world war would probably look like now, and that this back and forth might continue for a while.
I'd like to hear from an expert as well, rather than rely on piecing news items together.
The global powers are negotiating with themselves, and consolidating state power at home. Proxy wars for the former, and fear mongering at home for the latter.
The players are the 0.001% who control these states and the rest (we) are the captive (and propagandized) audience. They are being super kind as to at least make it entertaining for us.
> - Russia's only air craft carrier is trespassing through UK waters to get to Syria in a show of force that doesn't really add anything to their military capabilities there.
If they were really gearing up for war why would they move their only carrier away from the mother land. Your article even says it is more of a "show of force" than start of war.
I'm sure that carrier is being followed by multiple NATO submarines as well.
It seems incredibly unlikely that a global war would start over Syria when we've had 60 years of proxy conflict instead. Russia or NATO have absolutely nothing to gain from an open military conflict.
> US election is in 3 weeks and Donald Trump is openly in love with Putin.
He states that he's never met Putin nor has any holdings in Russia. He has stated that he is open to positive relationships with the Russian government.
> Trump questioned the benefit of NATO which is the basis for Europe stability after the 2nd world war.
I believe he stated that he wants NATO to "pay their fare share" in the costs of maintaining the organization.
I'm not a Trump supporter but we shouldn't believe everything we read.
Trump says he never met Putin, now. In the past, he said he did. I just did a search for "trump met putin" and found a bunch of news sites reporting that in a GOP debate a while ago Trump said
“I got to know him very well because we were both on ‘60 Minutes,’ we were stablemates, and we did very well that night.”
Trump was boasting in that debate about nothing, they were on the same episode of 60 minutes but they were not even on the same continent for that episode.
That's not really the point. The point is Trump is now saying he hasn't, but in the past he said he has. Not only is he contradicting his earlier statement, but it also makes him not trustworthy. And of course, if he was boasting about having supposedly met Putin in the past, that means he thought it was a good thing to boast about, which suggests that he is sympathetic to Putin and to Russian interests.
> - Finland (yes, Finland) is increasingly worried about Russia. They violated their air space, and they're questioning Finland's independence. Finland shares a long boarder with Russia.
The Finns actually have still quite good relationship with Russia (better than other neighbors) and nobody's actually questioning Finland's independence. Baltic countries is a different story.
It's good to know things are clam, I was just quoting the article. Not sure where they got that from
> Finland is becoming increasingly worried about what it sees as Russian propaganda against it, including Russian questioning about the legality of its 1917 independence
Is this sabre rattling or the prelude to a global conflict? Surely at worst it will (continue to) be a proxy war between NATO and Russia in Syria and nothing more? What motive is there for Russia or NATO to engage in open warfare? I'm not sure that a slow and prolonged lead up to an open war would even be effective in this situation.
Perhaps it should be "Say hello to Cold War v2.2017"?
> - Russia's only air craft carrier is trespassing through UK waters to get to Syria in a show of force that doesn't really add anything to their military capabilities there.
I read they were passing in international waters. Is that not the case? It's clearly a show of force, but no need for the hyperbole if it is not true.
> - Finland (yes, Finland) is increasingly worried about Russia. They violated their air space, and they're questioning Finland's independence. Finland shares a long boarder with Russia.
Finland isn't worried, they have had stable relations for half a century as both sides agreed to not mess with each other. They have even refused to join NATO because it is actually safer for Finland and vice versa.
Cowboys with missiles stationed on Russia's border making hyperbole statements (like you do) - now that would be a real threat. (the same was also true the other way around with the Soviets stationing missiles in Cuba)
If this is supposed to be "taking down the internet", then I'm not impressed. Using cached DNS still gives access to any service. I'm even typing here on HackerNews.
If this is another practice run, then I'm still not impressed. Taking down one provider is not that hard. Good luck finding the resources to do this DDoS to ALL large DNS providers out there.
Maybe it's not really fair to link to that post every time a DDoS with more than average payload happens. Especially since the post doesn't mention any specifics, because well, "protect my sources". It's like the "buy gold now" guy starring in the 2 AM infomercial predicting an economic recession within the next 5 years, without adding what the exact cause is going to be. He is probably going to be right, but that doesn't make him a visionary.
I work as a freelancer and today I didn't get paid and that's just me. Companies probably lost millions today. By just one DDoS to one DNS provider.
Yeah it's not the whole Internet, but how do you define "taking down the Internet" anyway. Is it every connected computer or just a huge amount of interconnected big websites? Because the latter is happening right now.
Schneier:
"I have received a gazillion press requests, but I am traveling in Australia and Asia and have had to decline most of them. That's okay, really, because we don't know anything much of anything about the attacks.
If I had to guess, though, I don't think it's China. I think it's more likely related to the DDoS attacks against Brian Krebs than the probing attacks against the Internet infrastructure, despite how prescient that essay seems right now. And, no, I don't think China is going to launch a preemptive attack on the Internet."
I'm using Google Public DNS too. I don't really know if there is a relation with DynDNS but I'm still experiencing issues in GitHub and Twitter, like partial loading of images.
Perhaps Google had old (but valid) records still in their cache for a while. Google DNS was working for me for a while, and then stopped. Apparently Dyn has the problem fixed, but maybe there is some TTL based propagation delays still. I updated my internal network to use Dyn's internet guide/public DNS and the problem is fixed.
Maybe this is their strategy: we break it, you buy it ;)
Seems to be impacting POPs in US East most severly. We use Ripe Atlas to assess the impact of DNS outages, and in the past hour have measured about 50-60% recursive query failure from a few hundred probes in that region: https://cloudharmony.com/status-for-dyn
I'm a Verizon FIOS customer in NYC and was unable to reach nytimes.com and several other sites this morning. Switching my DNS to Google's (8.8.8.8 and 8.8.4.4) seemed to fix the problem, but I don't understand why yet.
that's not going to help much if the authoritative name servers (which is what dyn is, btw) go down for more than a day.
Max record cache time is 86400s (24h), so if the attackers can keep it down for 24h then google will have to have custom instructions in place (or cache more aggressively than the RFC allows)
Since the attacked dyndns DNS servers are evidently anycast, the google server you are reaching might connect to a different dyndns server than you do. If google has luck to reach a less overloaded server, they might get an answer where you get none.
In addition, Google Public DNS engineers have proposed a technical solution called EDNS Client Subnet. This proposal allows resolvers to pass in part of the client's IP address (the first 24/64 bits or less for IPv4/IPv6 respectively) as the source IP in the DNS message, so that name servers can return optimized results based on the user's location rather than that of the resolver. To date, we have deployed an implementation of the proposal for many large CDNs (including Akamai) and Google properties. The majority of geo-sensitive domain names are already covered.
Is there any reason why Dyn has to be "down" from Google's perspective? Is it possible that the large DNS providers maintain private network between each other, such that DDoS attacks that are effective against the public are ineffective against the private network?
They sell premium services, have a large sales team, and are very aggressive. I get emails from them weekly discussing millisecond savings of their DNS solutions and the value increase in customers and sales.
Squeaky wheels get grease and their sales team squeaks a lot.
Realistically they compete with Neustar which is shockingly expensive and has less features and is harder to use.
I chose Dyn over Neustar (UltraDNS) when it was time to renew contracts because it was 60% cheaper, had a better latency, their support was great and the interfaces were clear.
Not a fanboy or anything, I really don't like how aggressively they hound me now (even though I have nothing to do with DNS for my current employer), but it's cheap and effective so it's not surprising people use them.
I had a NS1 demo account. And then they stopped doing that, but it still worked. And then I lost the credentials, and now my account is invalid for a password reset :(
Way, way back in time, they offered lifetime DNS hosting for a relatively low price.
I bought that, and they've honored the deal. Admittedly it comes with limits that would make it useless for any large site, but it's just great for individuals.
They're widely used because they were one of the few providers of geo-aware DNS service for a long time. (These days there are other, cheaper options, including Amazon Route53.)
They offer Anycast and have POPs around the globe. They also have some other nice features such as intelligent failover and extra GEO IP features. Things that you would otherwise have to build yourself.
They have been around a long time. For years they had a free product called DynDNS that would allow you get an A record for your dynamic IP at home.
Exhibit A of MEN having zero consequences for their sexual assault on the English language! Especially in the tech industry, where English sexual assault is unlike other industries!
Well, the parts that relied out outside services hooked up via SSO were not demoed, but majority of it worked fine because demo server was misconfigured to not actually rely on the external services. It is pretty funny.
I am a bit paranoid about disclosing details, but basically our SAML IDP was down, so the sales person couldn't log in at all. I was messing with the demo server to convince myself that it is 100% IDPs fault and we can't do anything about it, and discovered to my surprise that the form-based authentication was not disabled on it (normally our servers are in one mode or the other, but not both, even though this is an artificial separation). So I gave them the direct link to the form based entry point and most of the demo could be done.
We had a demo at the exact same time. (Internal weekly product demo, not that critical).
We did it on local host, the only one reliable 100% of the time.
The internet is resilient against being completely taken down (as demonstrated by everything working just fine for me from Germany). It's explicitly not resilient against taking parts of it down. You can take some continents off the internet entirely by cutting half a dozend cables, but it's extremely hard to make the internet unusable for everyone.
AWS says "We are investigating elevated errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region." Is that coincidental, or are they being DDoSed also?
That might explain why we are down - most of our EC2 instances are in us-east-1. Looks like Amazon SQS is impacted too. We are getting a stream of undeliverable messages, and our 'dead letter' queue is filling up!
Apparently us-east-1 is backed by Dyn (and only Dyn) as well?
$ host -t NS us-east-1.amazonaws.com
us-east-1.amazonaws.com name server ns3.p31.dynect.net.
us-east-1.amazonaws.com name server ns1.p31.dynect.net.
us-east-1.amazonaws.com name server ns2.p31.dynect.net.
us-east-1.amazonaws.com name server ns4.p31.dynect.net.
That's… utterly bizarre to me. us-east-2 has a more diverse selection:
$ host -t NS us-east-2.amazonaws.com
us-east-2.amazonaws.com name server u4.amazonaws.com.
us-east-2.amazonaws.com name server u6.amazonaws.com.
us-east-2.amazonaws.com name server u3.amazonaws.com.
us-east-2.amazonaws.com name server u2.amazonaws.com.
us-east-2.amazonaws.com name server u1.amazonaws.com.
us-east-2.amazonaws.com name server u5.amazonaws.com.
us-east-2.amazonaws.com name server ns2.p31.dynect.net.
us-east-2.amazonaws.com name server ns1.p31.dynect.net.
us-east-2.amazonaws.com name server pdns1.ultradns.net.
us-east-2.amazonaws.com name server pdns5.ultradns.info.
us-east-2.amazonaws.com name server ns3.p31.dynect.net.
us-east-2.amazonaws.com name server ns4.p31.dynect.net.
us-east-2.amazonaws.com name server pdns3.ultradns.org.
Not that anyone should be running a service whose availability they care about solely in us-east-1 anyway…
us-east-1 is the oldest region and predates Route 53. Not adding extra DNS providers to the older regions is probably an oversight.
(The EC2 API team requests load balancers from a separate load balancer team. The load balancer team probably didn't exist as a separate team when some of these regions were created.)
$ host -t NS us-east-1.amazonaws.com
us-east-1.amazonaws.com name server pdns5.ultradns.info.
us-east-1.amazonaws.com name server ns3.p31.dynect.net.
us-east-1.amazonaws.com name server pdns1.ultradns.net.
us-east-1.amazonaws.com name server pdns3.ultradns.org.
us-east-1.amazonaws.com name server ns4.p31.dynect.net.
us-east-1.amazonaws.com name server ns1.p31.dynect.net.
us-east-1.amazonaws.com name server ns2.p31.dynect.net.
us-east-1.amazonaws.com name server u1.amazonaws.com.
us-east-1.amazonaws.com name server u2.amazonaws.com.
us-east-1.amazonaws.com name server u3.amazonaws.com.
us-east-1.amazonaws.com name server u4.amazonaws.com.
us-east-1.amazonaws.com name server u5.amazonaws.com.
us-east-1.amazonaws.com name server u6.amazonaws.com.
If that were the reason I wouldn't expect this update:
6:36 AM PDT [RESOLVED] Between 4:31 AM and 6:10 AM PDT, we experienced errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region. During the issue, customers may have experienced failures indicating "hostname unknown" or "unknown host exception" when attempting to resolve the hostnames for AWS services and EC2 instances. This issue has been resolved and the service is operating normally.
720 comments
[ 2.8 ms ] story [ 356 ms ] threadSo far twitter, etsy, soundcloud, spotify, github, pagerduty...crazy that this can even happen
I'm not saying it's a good reason but it's a reason.
No GitHub, well, it's gonna be a fun Friday...
For example, most open source projects.
But even outside of that, we use github for issue tracking, the new project management kanban stuff, a CI server, reading documentation (which is offline, but the online versions are nicer on the eyes), and a ton more. Not to mention that StackOverflow and other discussion forums tend to be used by many.
Yes you can host all of that locally, but we don't have a few hundred thousand a year to spend on some sysadmins to maintain all of that, and we don't have the time or money to run the machines, vet the software, and keep it up more reliably than github does for next to nothing.
I disagree with that. If you have a server, you need a sysadmin. End of story.
Who is going to secure the system and setup ssh keys? who is going to run updates? who is going to monitor for security issues? who is going to run backups? who is going to secure those backups? who is going to oversee the installation of the network, the battery backups, the racks, the server hardware, etc... Who will swap out bad disks? Who will recover the system when it goes down? Who is going to double the hardware and setup high availability (remember, you are competing with github for uptime here)? And god help you if you have one guy that does all of this. What happens if he gets hit by a bus?
An on-prem server isn't a "backup", it's a liability. And without the resources to maintain it, it's going to become a nightmare. I've been there, and I won't ever do it again.
I'm either going to pay to do it right, or give it to someone who will. And if that means a few hours of downtime every year or so, then that's a wonderful tradeoff for me.
> It does not take long time or resources to download all of the libraries, with corresponding docs to a local server, or even your laptop. It is not complicated to have all of the new issues sent to an email to have a version of them available at all times.
Luckily github (and alternatives) provide all of that. It sends us emails and slack messages on everything, so if it's down, we can still read, and we all have our local repos. But reading is different than working.
If self-hosted, somewhere, you could still be screwed by having Dyn as your DNS provider.
If dev-machine-hosted, then uh, your issue tracker is no longer an issue tracker. Your build server is not a build server. All the services besides Git are not meant to operate offline in a decentralized/distributed fashion.
Library documentation, sure, that could be local. Otherwise, your assertion that all of this tools infrastructure can somehow be replicated, easily, in a way that makes the difference between working online or offline effectively zero, is nonsense.
Second, pointing to a new machine is as simple as updating IP in your hosts file, or dns server.
Third, you can use vmware or any other virtualization stack to replicate your infrastructure locally. In fact that's the best way to build things - create virtual network, use it for testing, troubleshooting and debugging, and deploy only when everything is working.
All I'm saying is that if you're company is making any kind of money, and your development environment depends 100% on online services, you're doing it wrong.
If that's the case, i think you're misguided : imho, the internet as it was designed was conceived so that everyone has its little self-hosted thing, with dev-machine just for the purpose of, well, test and dev, with the latter goal of it being self-hosted.
Just look how email is technically designed and how it was meant to work, and we use it now, relying mostly on Gmail or Outlook, or worse, using Facebook for emails : we put all our eggs in the same basket.
Option A) Spend no money and experience an outage maybe once a year, if that. And the problem works itself out.
Option B) Spend money and gain technical debt to avoid a problem that happens maybe once a year, if that.
Which one would you pick? I mean, maybe if everything you have is closed-source or you are guaranteeing 99.9% uptime to your customers, perhaps option B makes sense. Otherwise, the choice seems fairly obvious to me.
I work in IT infrastructure and I see attacks literally every day, moreover, most people just setup a quick LAMP or MEAN stack to prove their concept, and then they leave it like that, so most of the time, no, the problem don't just "work itself out".
Yup. But the whole point of GitHub is to make you dependent on GitHub.
They've been slightly successful.
>> We are seeing a widespread DNS issue affecting connections to our services both internally and externally.
Only 2 of the points in the US are affected on https://www.whatsmydns.net/ for the domains we've got on Dyn - same for Twitter etc
https://www.schneier.com/blog/archives/2016/09/someone_is_le...
Edit: And to be clear: I don't mean to imply there's any connection :)
Why not the USA?
Note that this wouldn't rule out the USA as such. First, it could be a longshot preparedness thing, with no expectation that it would ever be used. Second, they could be red-teaming the thing (looking for weaknesses so that they can arrange for them to be shored up).
In either of these scenarios, it's no less likely that the USA would be doing it than anyone else. If you assume that whoever is doing this is planning to use their knowledge, however, the economic argument makes the USA less likely to be involved.
I'd be more willing to put my money on someone attacking an entity downstream who is normally immune to DDOS attacks of this size.
For example, if a particular part of the government got wind of a data dump about to be released by another nation-state or independent actor (for example, a leak of some kind) - I think some parts of the USA government that possesses the ability to do so wouldn't hesitate to take down dns to the entire internet to avoid another similar data leak to the Snowden dump.
Be really wary of attributing intent: you do not know who will benefit the most from taking down certain services. To claim that the US benefits from the internet so much that it wouldn't do certain actions to protect itself from certain types of harm is shortsighted.
Even my example could be really wrong, but the idea is that nobody really can say - "oh the internet is too important to xyz, they'll never do anything!"
I don't understand how this would change anything unless you're assuming they would take down the Internet permanently
Remember, a data leak is not just a technical issue. They can resolve it in any number of ways - get a small team incursion into another state's territory for extraction, etc. All the outage needs to do is to hold open that window for enough time for all the different parts of the entire threat response chain to do each part's job.
A lot of technical people think tech is the end, but no - if you get a small team to go knock on the person's door, and get your internet response team to shut down dns, or to get someone on site at the telco to perform certain actions at the router/switch level, etc all portions working together is a powerful way to resolve or to accomplish certain goals.
Think bigger, especially with state actors - the resources are there, and this line of thought is probably really basic stuff that people came up with in the 1960's or 70's (even when the arpanet was being created, there was probably already a team tasked with taking such actions - it only make sense to have 2 teams working on such goals in tandem - one to create the network, the other to take it down)
News cycles happen fairly rapidly, so if you could take down a number of sites that might be friendly to the dissemination of potentially damaging information just long enough such that it's forgotten about, or the attack is so large the media talks about the attack instead, then you might be able to successfully avoid widespread public knowledge of such information. Though, this would be best aided with collusion or cooperation (intentional or otherwise) from the media. Toss in a few unrelated services as a bonus for collateral damage, and you might be able to avoid scrutiny or, at the very least, shift the blame to an unrelated state actor. It won't prevent the release of information, but that's not the point--you want to prevent the dissemination and analysis of that information by the public at large.
This is all hypothetical, of course, and not likely to work. It also comes with the associated risk that if you were discovered or implicated, public outrage might be even worse than if you allowed the release of the information you hoped to distract from in the first place! As such, I can't imagine anyone would be stupid enough to try.
I'll take my tinfoil hat off now.
I still give it less than a 5% probability, though.
Honestly, that's fairly thin. WL uses torrents and other means of disseminating data that don't rely on central control structures. Plus, presumably, WL has the ability to quickly shift data into secure hands who are willing to release it when things quiet down.
So, sure, the USA could go send someone to sieze the hard drives of someone who has confidential information. But, I have to imagine one of the first steps when getting that kind of information is to disseminate it to others (at least some of whom are unknown to the states). If they were hit, these people would very quickly take that as a signal to indiscriminately release all the information.
Not necessarily so strange. See https://en.wikipedia.org/wiki/Bootleggers_and_Baptists for instance.
As for the length of the "test," they might want to see how the US would react to such attacks in the future, and shake out anything critical. "Oh, these two agencies can't talk to each other. Good to know."
I hate the way modern times makes me look.
To pin it on someone else?
"17 Intelligence agencies told me Russia hacked our DNC thing" (Clinton).
So maybe it is now "Oh look they took down the whole internet as well".
That's... kind of conspiratorial thinking? Would you cut off your own hand so you could blame it on someone else?
Where did I say it was my immediate explanation and this is _likely_ what is happening?
> kind of conspiratorial thinking?
You mean like lizard aliens infiltrating our planet? -No. But in the realm of "shooting down of passenger and military planes, sinking a U.S. ship in the vicinity of Cuba, burning crops, sinking a boat filled with Cuban refugees, attacks by alleged Cuban infiltrators inside the United States, and harassment of U.S. aircraft and shipping and the destruction of aerial drones by aircraft disguised as Cuban MiGs", yes.
It was mostly a reply to "US would have absolutely no reason for doing this" and the reply is there cold be a plausible reason.
Then I felt horrible.
Some NPR story about US Cybercommand responding to Russian cyber attacks, 'at place and time of our choosing.'
'Some you might hear about. Some you might not.'
FFWD to a couple days ago, NPR story about a botched European and Russian lander.
Today, US Eastern Seaboard is seeing connectivity disruption due to DDoS attacks.
*
Unwinding the stack, the latest news is these DDoS attacks are not likely state sponsored.
Russia pulling off a coordinated attack that soon after and in response to my theorized US retaliation seems unlikely.
US attacking a joint partnership between Russia and Europe civilian space program seems unlikely.
https://www.techdirt.com/articles/20160912/16553435504/fbi-d...
Of course, he said nothing about internal rigging:
https://twitter.com/TweetBrettMac/status/789372518436052992
Though if they targeted electric grid, water, and public transport, starting early in the day and choosing the regions by their populations political leaning, it could easily have an effect on the result itself.
Control the message, and through that the actual votes cast.
Of course, I have no information on the security model of the pre-election preparations and post-election tabulation, but luckily results for each polling place are also posted for the public to inspect - media outlets and campaigns can verify the tabulation themselves with a slight delay.
host -t ns twitter.com: ns3.p34.dynect.net, ns4.p34.dynect.net, ns1.p34.dynect.net, ns2.p34.dynect.net.
host -t ns amazon.com: ns3.p31.dynect.net, ns4.p31.dynect.net, ns2.p31.dynect.net, pdns6.ultradns.co.uk, pdns1.ultradns.net, ns1.p31.dynect.net.
"How do all these major players have singly-homed DNS"?
http://www.cnbc.com/2016/10/21/major-websites-across-east-co...
Is this par for course for all large DDOS attacks or did something tip them off?
I would assume that when a large number of big enterprise-y things go down, HSI takes notice. When other providers get attacks that are 20x larger (gbit/sec), but have much less widespread impact and impact on less enterprise-y things, they don't care so much.
As @scrollaway mentioned, 6 weeks ago, Bruce Schneier posted that several companies told him that they're detecting attempts to probe their networks and find ways to bring it down https://www.schneier.com/blog/archives/2016/09/someone_is_le...
Now let's look at the progress of events:
- Hillary Clinton's personal email server was hacked a while ago.
- A lone hacker published a document obtained by hacking the DNC servers. The document includes opposition research on Donald Trump and how Hillary can attack him in the election.
- Wikileaks published emails obtained by hacking the DNC
- US intelligence agencies confirmed that Russia was behind the DNC hack
- It was reported that the CIA is starting a cyber attack against Russian targets. http://www.nbcnews.com/news/us-news/cia-prepping-possible-cy...
- This is happening while the war in Syria and Iraq is growing. The Russians are there to "fight ISIS" but they have deployed an air defense system even though ISIS doesn't have any air force.
- Russia's only air craft carrier is trespassing through UK waters to get to Syria in a show of force that doesn't really add anything to their military capabilities there.
https://www.theguardian.com/world/2016/oct/20/russian-fleet-...
https://www.theguardian.com/world/2016/oct/19/convoy-of-russ...
- Finland (yes, Finland) is increasingly worried about Russia. They violated their air space, and they're questioning Finland's independence. Finland shares a long boarder with Russia.
http://www.businessinsider.com/r-finland-sees-propaganda-att...
- US ships were attacked near Yemen after they're bombed some targets the belong to the rebels. https://www.theguardian.com/us-news/2016/oct/13/us-enters-ye...
- US election is in 3 weeks and Donald Trump is openly in love with Putin. Trump questioned the benefit of NATO which is the basis for Europe stability after the 2nd world war.
Say Hello to World War III, everybody!
Thank you for these links. I'm trying not to get wrapped up in conspiracies but am increasingly worried by the mounting conflict. I'd love to hear a calm, reasoned response from someone more knowledgable than me on these topics.
My personal opinion is that it is mostly political and I think (hope) that what is happening in Syria won't escalate to direct conflict between the US and Russia.
I stumbled across this little blog article the other day and it helped relieve some of my anxieties.
https://cluborlov.blogspot.com/2016/10/oopsa-world-war.html
The players are the 0.001% who control these states and the rest (we) are the captive (and propagandized) audience. They are being super kind as to at least make it entertaining for us.
If they were really gearing up for war why would they move their only carrier away from the mother land. Your article even says it is more of a "show of force" than start of war.
So how did you jump to WW3?
It seems incredibly unlikely that a global war would start over Syria when we've had 60 years of proxy conflict instead. Russia or NATO have absolutely nothing to gain from an open military conflict.
He states that he's never met Putin nor has any holdings in Russia. He has stated that he is open to positive relationships with the Russian government.
> Trump questioned the benefit of NATO which is the basis for Europe stability after the 2nd world war.
I believe he stated that he wants NATO to "pay their fare share" in the costs of maintaining the organization.
I'm not a Trump supporter but we shouldn't believe everything we read.
“I got to know him very well because we were both on ‘60 Minutes,’ we were stablemates, and we did very well that night.”
http://time.com/4108198/donald-trump-60-minutes-putin/
The Finns actually have still quite good relationship with Russia (better than other neighbors) and nobody's actually questioning Finland's independence. Baltic countries is a different story.
Source: A Finn here.
> Finland is becoming increasingly worried about what it sees as Russian propaganda against it, including Russian questioning about the legality of its 1917 independence
Is this sabre rattling or the prelude to a global conflict? Surely at worst it will (continue to) be a proxy war between NATO and Russia in Syria and nothing more? What motive is there for Russia or NATO to engage in open warfare? I'm not sure that a slow and prolonged lead up to an open war would even be effective in this situation.
Perhaps it should be "Say hello to Cold War v2.2017"?
I read they were passing in international waters. Is that not the case? It's clearly a show of force, but no need for the hyperbole if it is not true.
Finland isn't worried, they have had stable relations for half a century as both sides agreed to not mess with each other. They have even refused to join NATO because it is actually safer for Finland and vice versa.
Cowboys with missiles stationed on Russia's border making hyperbole statements (like you do) - now that would be a real threat. (the same was also true the other way around with the Soviets stationing missiles in Cuba)
If this is another practice run, then I'm still not impressed. Taking down one provider is not that hard. Good luck finding the resources to do this DDoS to ALL large DNS providers out there.
Maybe it's not really fair to link to that post every time a DDoS with more than average payload happens. Especially since the post doesn't mention any specifics, because well, "protect my sources". It's like the "buy gold now" guy starring in the 2 AM infomercial predicting an economic recession within the next 5 years, without adding what the exact cause is going to be. He is probably going to be right, but that doesn't make him a visionary.
Yeah it's not the whole Internet, but how do you define "taking down the Internet" anyway. Is it every connected computer or just a huge amount of interconnected big websites? Because the latter is happening right now.
If I had to guess, though, I don't think it's China. I think it's more likely related to the DDoS attacks against Brian Krebs than the probing attacks against the Internet infrastructure, despite how prescient that essay seems right now. And, no, I don't think China is going to launch a preemptive attack on the Internet."
[1] https://www.schneier.com/blog/archives/2016/10/ddos_attacks_...
Maybe this is their strategy: we break it, you buy it ;)
https://help.dyn.com/internet-guide-setup/
If you can't load that page, the public DNS servers are: 216.146.35.35, 216.146.36.36
Max record cache time is 86400s (24h), so if the attackers can keep it down for 24h then google will have to have custom instructions in place (or cache more aggressively than the RFC allows)
In addition, Google Public DNS engineers have proposed a technical solution called EDNS Client Subnet. This proposal allows resolvers to pass in part of the client's IP address (the first 24/64 bits or less for IPv4/IPv6 respectively) as the source IP in the DNS message, so that name servers can return optimized results based on the user's location rather than that of the resolver. To date, we have deployed an implementation of the proposal for many large CDNs (including Akamai) and Google properties. The majority of geo-sensitive domain names are already covered.
from https://developers.google.com/speed/public-dns/faq
"Oh, maybe its our shitty ISP screwing up everything again."
No, it's in a bigger scale.
Squeaky wheels get grease and their sales team squeaks a lot.
I chose Dyn over Neustar (UltraDNS) when it was time to renew contracts because it was 60% cheaper, had a better latency, their support was great and the interfaces were clear.
Not a fanboy or anything, I really don't like how aggressively they hound me now (even though I have nothing to do with DNS for my current employer), but it's cheap and effective so it's not surprising people use them.
I bought that, and they've honored the deal. Admittedly it comes with limits that would make it useless for any large site, but it's just great for individuals.
Ironically, a quick search of my Gmail mailbox came up with this gem in the subject line from Dyn.
"Did you know the average cost of a single DDoS outage is $882K?"
$ dig +short ns amazon.com
ns1.p31.dynect.net. pdns1.ultradns.net. ns4.p31.dynect.net. pdns6.ultradns.co.uk. ns3.p31.dynect.net. ns2.p31.dynect.net.
They have been around a long time. For years they had a free product called DynDNS that would allow you get an A record for your dynamic IP at home.
us-east-1 is the oldest region and predates Route 53. Not adding extra DNS providers to the older regions is probably an oversight.
(The EC2 API team requests load balancers from a separate load balancer team. The load balancer team probably didn't exist as a separate team when some of these regions were created.)
Don't confuse regions with availability zones. (Though in this case, the availability zones don't help...)
6:36 AM PDT [RESOLVED] Between 4:31 AM and 6:10 AM PDT, we experienced errors resolving the DNS hostnames used to access some AWS services in the US-EAST-1 Region. During the issue, customers may have experienced failures indicating "hostname unknown" or "unknown host exception" when attempting to resolve the hostnames for AWS services and EC2 instances. This issue has been resolved and the service is operating normally.