CloudFlare recently posted about a 400 Gbps attack they saw. I can't imagine any reason Meetup would have that sort of bandwidth capacity lying around.
Pre-CloudFlare, those were expensive. I can't speak to the technical work needed to hook something like MeetUp with CloudFlare, but if this is the first DDOS they've suffered it might not have been particularly worth it to them until now.
>> but if this is the first DDOS they've suffered it might not have been particularly worth it to them until now.
Isn't that the whole point of using a preventative approach when it comes to your site's security? So you don't learn the hard way?
Of course knowing a lot of InfoSec guys, they can only tell them what they should have, it's up the owner's to make the call to get it in place. I don't know how many times one of them complained about doing a pen test on someone's site, making the recommendations and then one year later, they haven't fixed anything yet.
plan ahead and work with DDoS mitigation services before you're attacked
Where can I find more information about this? I've been thinking about exactly this issue with regards to running an MMO. Pretty much none of my functionality can be cached, so I don't think CloudFlare can help me. The best solution I've come up with is to run my MMO as a few servers on NFO, which hosts game servers and has a good reputation for "not just nullrouting you."
The NTP attack is a simple example. No legitimate NTP traffic would be aimed at that site, so it can all be blocked at the firewall with no load on your servers. The problem with that attack was simply the bandwidth. Cloudflare's network helped to spread the load across many datacenters around the world.
Game servers are tough with their latency requirements. Generic web traffic is much easier to mitigate with offsite services (Prolexic, Defense.net, Cloudflare, etc).
If you're running game servers, your only good choice is to host with someone that has a really good onsite mitigation system. The offsite solutions add too much latency there.
"On Monday we mitigated a large DDoS that targeted one of our customers. The attack peaked just shy of 400Gbps."
And yes, it did affect other parts of their network, but they were able to successfully mitigate it.
"We saw attack traffic hitting every one of CloudFlare's data centers. While we were generally able to mitigate the attack, it was large enough that it caused network congestion in parts of Europe"
Not particularly. It just requires money, planning, and someone on staff who's old enough to remember the early 2000's when popular sites were routinely attacked. Seems like Meetup is lacking in at least one of those.
The are using CF or at least they were at one point during the weekend. It's not going to matter though if the attackers know the IP range the servers are hosted at. They'll just bypass CF and attack the servers directly.
Shouldn't it be easy though to configure the servers/load balancers to reject any request except for CloudFlare's?
Then again, if the attacker spoofs their IP address, I'm not sure how this would work (and I'm not sure how Cloudflare prevents against this as it is).
We've been dealing with DDOS attacks affecting mainstream citizens and businesses of 1st world countries for about a decade now. Why isn't there law enforcement dedicated to catching people who do this? We're at a point where CEOs of companies can engage in such behavior and the companies can stay in business.
If there is someone reading this who works with a US congressman or senator, this may be a chance to introduce legislation that can make a positive impact, make it easier to establish Internet-facing companies, and instead of eliciting dismay from the community of technical people, gain their gratitude instead.
Yes, and then NSA will say "Sure, we'll protect you. You just have to give us access and control to your whole network and servers, because that's the only way to 'properly protect you'. Deal?"
This is exactly what these agencies want. To give up on your responsibility for the security of your users, so they can take over. If your company does that, please tell me now so I know to avoid it forever.
The solution is to better design anti-DDoS systems, not to let the government "handle it".
Actualy, the NSA - just like the CIA, is known to carry out false flag acttacks to achieve a certain outcome. This point should not be overlooked.
In this case Meetup is a perfect NSA 'false-flag' target because it's not a financial institution (NSA ally), and yet is used by millions of people. Thus millions of people starting to ask if GCHQ/NSA will keep them 'safe' online.
The idea being they could be ticketed for using their hardware unsafely on public infrastructure, much like a speeding ticket - but I'd suggest lower amounts corresponding to the lower risk to lives.
Unlike speeders, I assume they are mostly unwitting and unwilling participants in the attack. Seems a bit unfair. Why not also sue Adobe for allowing the zero-day that led to the PC getting infected and added to a botnet?
"Unlike speeders, I assume they are mostly unwitting and unwilling participants in the attack."
If you don't check your vehicle and something falls off of it, you're responsible. For that matter, if you do check your vehicle and something falls off of it, you're responsible. You don't have to have "wittingly" or "willingly" done something for a ticket - negligence (even minor) is plenty. You're responsible for hardware you deploy.
Edited to add: Also, plenty of speeding tickets are given to people who never said to themselves "I'm going to speed," but simply neglected to pay enough attention to either posted limits or their speedometer. I know that's been the case with the couple of speeding tickets I've received. So I'm not really sure it's "unlike speeders" at all.
"Why not also sue Adobe for allowing the zero-day that led to the PC getting infected and added to a botnet?"
In general, I don't think it makes sense for liability to start with the vendors because the risks and downside is determined primarily by the particular deployment. I've no problem with allowing the companies to offer indemnification for tickets caused by their failures, and if we went that route that's something users should demand (... while still allowing people to publish GPL software for those willing to accept the responsibility, without taking a huge financial risk).
I get what you're saying, I just don't really think it's a good idea and I definitely don't think it will work. You would need a massive number of people to suddenly start caring about network/PC security AND you would need them to have the ability to actually do something about it AND it does nothing for anyone not under US law.
"You would need a massive number of people to suddenly start caring about network/PC security"
"You can get a ticket" seems to get people to care, up to a point.
"you would need them to have the ability to actually do something about it"
Certainly the case. I lean toward "change the incentives and the market will provide", but this could be better fleshed out.
"it does nothing for anyone not under US law."
It does plenty for people not under US law - they won't be DDoSed from as many bots in the US.
It doesn't do as much about people under jurisdictions not covered, but:
1) A lot of the software that is developed is developed with the US in mind as a market, especially those things that are sufficiently common to really let malware spread by targeting a couple exploits. Making that software more secure might decrease the rate of infections abroad.
2) Other jurisdictions might very well follow suit - particularly with international pressure.
Setting law enforcement priorities is typically an Executive Branch function, no? I guess Congress could earmark money specifically to fighting DDoS-based crime.
Turn the appeal to fear that causes extortion to work back on itself: Make paying extortionists cause the payer to be an accessory. Allow future victims of the same extortionist to collect joint-and-several civil damages from the collaborator.
When are we going to hold individuals and companies legally responsible for the hardware they control? We already do this for other types of crimes such as remailing scams. If you're going to have hardware on the internet you should be responsible for it. There should be penalties for being irresponsible. Ignorance is never an excuse.
And then set the precedence that a person in Indonesia controlling a server in Iceland can be legally responsible in the US? How do we determine jurisdiction?
The problem here isn't whether companies are using DDoS mitigation services, it's whether they have ISPs null routing them. The weak point here at the top tier ISPs, not necessarily the individual companies being targeted. You can have the best DDoS mitigation service ever, but if the top-tier ISPs black hole you, you're in a very bad spot.
Why would a "tier 1" (I hate that outdated term) transit provider ever null-route an IP that's not in their block nor in a customer's block? I haven't heard of them doing that. That would be a very shady thing to do.
I'm sorry I used a term you don't like -- let's get past that though -- What about if the customer you speak of is the DDoS mitigation service?
Even without doing this though, if they remove a BGP route and other ISPs cannot route through them, that's a problem for whoever lives at the destination AS number.
I'm really confused here. Why would a DDoS mitigation service announce routes for a network they don't serve for a customer? That would simply be an old-fashioned BGP hijacking.
Transit providers don't simply send traffic to Prolexic and Defense.net because they think they should. They send traffic there for routes that the mitigators are announcing. They'll only announce client routes (and only when clients announce to the mitigators).
Transit providers will null route traffic that's inbound to you if you request it. This lets you stop the traffic at their core, instead of overwhelming your edge gear.
You are the customer after all; you have a say in what traffic makes it to your edge from your upstream provider.
He was referring to being nullrouted by transit providers that he's not a customer of. Short of something like a replay of the Morris worm, I can't see that happening.
Transit providers get paid to provide transit. They don't filter traffic that isn't bound for their customers.
No, they don't. Transit providers acting in a pure transit manner do not null route destination networks that they're not responsible for. Provider B, providing transit from AS A to AS C, will not block traffic bound for C or beyond. They might block some things bound for AS B or a customer of AS B, but they're not acting in a pure transit capacity there.
Content and eyeball networks are free to do whatever they want with regards to routing and blocking. Transit providers? No, they just provide transit. That's the business they want to be in. They're not in the blocking business.
Thanks for the NANOG tip though. I'm a member and active participant. See you in Seattle.
The main NANOG threads are about detecting the NTP traffic and blocking malicious requests in content and eyeball networks -- not transit. There's also the OpenNTP project discussion -- http://openntpproject.org/
At first I almost laughed when I saw the sum - they're asking $300? For a company that's been around for almost 15 years?
That said, Meetup's reasons for not paying it are very solid. I'm glad they're spending far more than $300 of their own resources[0] to fight this attack, because other websites would be the ones paying the price if they decided to start this precedent.
I use meetup.com for several meetups. This is not good. I'm really looking forward to finding out how they plan to mitigate this kind of issue in the future, b/c there has to be a way.
61 comments
[ 3.3 ms ] story [ 111 ms ] threadIt's akin to setting up an IDS/IPS before you get pwnt through sloppy PHP/MySQL injections.
Isn't that the whole point of using a preventative approach when it comes to your site's security? So you don't learn the hard way?
Of course knowing a lot of InfoSec guys, they can only tell them what they should have, it's up the owner's to make the call to get it in place. I don't know how many times one of them complained about doing a pen test on someone's site, making the recommendations and then one year later, they haven't fixed anything yet.
I'm not sure I'd classify being subject to DDOS as a security hole.
Where can I find more information about this? I've been thinking about exactly this issue with regards to running an MMO. Pretty much none of my functionality can be cached, so I don't think CloudFlare can help me. The best solution I've come up with is to run my MMO as a few servers on NFO, which hosts game servers and has a good reputation for "not just nullrouting you."
We have plenty of clients like that. They are using us for a variety of reasons, DDoS protection is one.
Curious: who is "we?"
If you're running game servers, your only good choice is to host with someone that has a really good onsite mitigation system. The offsite solutions add too much latency there.
http://blog.cloudflare.com/technical-details-behind-a-400gbp...
"On Monday we mitigated a large DDoS that targeted one of our customers. The attack peaked just shy of 400Gbps."
And yes, it did affect other parts of their network, but they were able to successfully mitigate it.
"We saw attack traffic hitting every one of CloudFlare's data centers. While we were generally able to mitigate the attack, it was large enough that it caused network congestion in parts of Europe"
Cue George Santayana quote.
Then again, if the attacker spoofs their IP address, I'm not sure how this would work (and I'm not sure how Cloudflare prevents against this as it is).
If there is someone reading this who works with a US congressman or senator, this may be a chance to introduce legislation that can make a positive impact, make it easier to establish Internet-facing companies, and instead of eliciting dismay from the community of technical people, gain their gratitude instead.
This is exactly what these agencies want. To give up on your responsibility for the security of your users, so they can take over. If your company does that, please tell me now so I know to avoid it forever.
The solution is to better design anti-DDoS systems, not to let the government "handle it".
In this case Meetup is a perfect NSA 'false-flag' target because it's not a financial institution (NSA ally), and yet is used by millions of people. Thus millions of people starting to ask if GCHQ/NSA will keep them 'safe' online.
The FBI is responsible for "cyber" crimes.
If you don't check your vehicle and something falls off of it, you're responsible. For that matter, if you do check your vehicle and something falls off of it, you're responsible. You don't have to have "wittingly" or "willingly" done something for a ticket - negligence (even minor) is plenty. You're responsible for hardware you deploy.
Edited to add: Also, plenty of speeding tickets are given to people who never said to themselves "I'm going to speed," but simply neglected to pay enough attention to either posted limits or their speedometer. I know that's been the case with the couple of speeding tickets I've received. So I'm not really sure it's "unlike speeders" at all.
"Why not also sue Adobe for allowing the zero-day that led to the PC getting infected and added to a botnet?"
In general, I don't think it makes sense for liability to start with the vendors because the risks and downside is determined primarily by the particular deployment. I've no problem with allowing the companies to offer indemnification for tickets caused by their failures, and if we went that route that's something users should demand (... while still allowing people to publish GPL software for those willing to accept the responsibility, without taking a huge financial risk).
"You can get a ticket" seems to get people to care, up to a point.
"you would need them to have the ability to actually do something about it"
Certainly the case. I lean toward "change the incentives and the market will provide", but this could be better fleshed out.
"it does nothing for anyone not under US law."
It does plenty for people not under US law - they won't be DDoSed from as many bots in the US.
It doesn't do as much about people under jurisdictions not covered, but:
1) A lot of the software that is developed is developed with the US in mind as a market, especially those things that are sufficiently common to really let malware spread by targeting a couple exploits. Making that software more secure might decrease the rate of infections abroad.
2) Other jurisdictions might very well follow suit - particularly with international pressure.
Laws enacting agencies' active enforcement of those laws.
This. It would need the resources of a small agency all by itself.
https://en.wikipedia.org/wiki/Computer_Fraud_and_Abuse_Act
The more likely problem is shady networks who either explicitly allow criminal activity or are just negligent.
A good number of DoS attacks can be stopped just by origin networks implementing BCP38.
Even without doing this though, if they remove a BGP route and other ISPs cannot route through them, that's a problem for whoever lives at the destination AS number.
Transit providers don't simply send traffic to Prolexic and Defense.net because they think they should. They send traffic there for routes that the mitigators are announcing. They'll only announce client routes (and only when clients announce to the mitigators).
You are the customer after all; you have a say in what traffic makes it to your edge from your upstream provider.
Transit providers get paid to provide transit. They don't filter traffic that isn't bound for their customers.
Content and eyeball networks are free to do whatever they want with regards to routing and blocking. Transit providers? No, they just provide transit. That's the business they want to be in. They're not in the blocking business.
Thanks for the NANOG tip though. I'm a member and active participant. See you in Seattle.
The main NANOG threads are about detecting the NTP traffic and blocking malicious requests in content and eyeball networks -- not transit. There's also the OpenNTP project discussion -- http://openntpproject.org/
That said, Meetup's reasons for not paying it are very solid. I'm glad they're spending far more than $300 of their own resources[0] to fight this attack, because other websites would be the ones paying the price if they decided to start this precedent.
Also, somewhat relevant: https://en.wikipedia.org/wiki/Danegeld
[0] I wonder how much this downtime actually represents to them (plus the engineering time)