And Cloudflare's cited packet rate from the Moobot attack is a bit higher than Google is citing in this article. Attacks approaching 1 Gpps are pretty serious and both of these companies are fighting the problem by spending tons of money. As Google says here, "sufficient capacity to absorb the largest attacks". No magic bullet.
The capacity of networking gear isn't unidimentional - most network equipment has both a maximum number of packets per second and a maximum total throughput. I'm not in the business, but I suspect that sophisticated network filtering equipment even at the high end can have its packets-per-second capacity eaten up before it hits line speed, just like low end routing equipment can.
As a result, there are some people whose ability to withstand a DDoS has more to do with packets per second than bits per second, so it becomes a useful metric.
Yes, a 1-byte UDP packet is just about as costly as a 1000-byte UDP packet. Considering that an ordinary PC can drive a 100gbps line interface with large frames, 1tbps is really nothing at all. But 1 Gpps is orders of magnitude more than one machine can normally send or receive.
> As a result, there are some people whose ability to withstand a DDoS has more to do with packets per second than bits per second, so it becomes a useful metric.
Right, definitely. A large sports streaming event might (easily!) do 10Tbps across a country, even more, but each GB served may only represent ~5k requests - 10e3/8 * 5e3 = 6.25M RPS, with each request being a single packet (small GET requests). That's big, but workable across enough machines for a big provider.
500M Pps is 80x that figure - you're basically DoS'ing header processing filters in your networking hardware.
People are not switching. Serious practitioners always distinguished between pps, bps and rps(or qps), as they measure different dimensions as explained in the article.
If people keep installing Windows Simple TCP/IP Services and activating the chargen server[1] on cloud machines with faster network interfaces, it's going to keep getting worse. I mean, BCP38 would be nice too, but not installing more chargen reflectors would be good enough for me.
[1] I wish I were joking, but this was verifiably the source of most of the volumetric DDoS I fought at my last job.
Hell if I know, but you send a 0 payload UDP, and it sends a 64k payload UDP back, IP fragmented obviously. Oh, and their firewall helpfully blocks sending the first fragment... cause that's helpful.
I want to believe you’re joking about a service I last installed during the Clinton administration - and didn’t make public even then - but we never get a break on that.
Right behind this, I’ll put cloud providers who helpfully put default 0.0.0.0/0 rules in their firewalls.
https://en.wikipedia.org/wiki/Character_Generator_Protocol in case anyone else had never heard of this... um... unfortunate protocol. Works on TCP and UDP, basically "DDoS vector as a service" if I ever saw one (seriously, could you design a better protocol for DDoS?). Just... Who? How? Why!?
Just to add a use case, but I know that my Mikrotik routers / switches have something called a bandwidth test, and it works fairly similar. It’s very useful for determining bottlenecks and/or tuning settings.
spoofers who generate a ton of syns to legit destinations which result in a lot of syn+ack to the victim. Bcp38 would help here.
Botnets generating a ton of UDP to destinations. Hosters, cloud providers (especially those with vulnerable/open EMR clusters) and broadband ISPs with easily compromised customers are the problem here. Kudos to those who take down the botnet command and controls.
Memached/ntp/cldap amplifiers. Still out there, still a problem. Thankfully a few of these services are policed at large peering interconnection points.
Exactly right. Security has been on a path of “layer more complexity over inherently insecure protocols” and it’s not going to stop any time soon because selling more products/services is worth too much money.
There’s a lot of work and very little money in replacing outdated networking protocols with smarter/more secure protocols. And there’s very little punishment (from customers or regulators) for suffering from massive cyber attacks, even if you expose the sensitive information of people who never even chose to give you their information (like Equifax). There’s no punishment or reputational damages from having your network traffic rerouted from a BGP attack. The consumer pays all the cost and they don’t even change their behavior after paying that cost, so why would companies bother?
Normally this is a situation where governments would step in and protect their citizens, but... well there’s just too much money on the line. So the programmers get the blame and the customers pay the price and security vendors make another $100 million and we do it all over again next month.
What's your proposed idea at the protocol level which solves the "I'll just saturate your pipe with garbage packets" case? That was the Mirai problem and I can't imagine a protocol solution. These are real, fully controlled endpoints.
Even when we have solutions to some issues (authenticated BGP), people don't implement it and there's nothing Google can really do about that. (Apart from not accepting traffic from those ASes?)
Make it harder to generate garbage packets? BCP38 on the ISP side would solve IP spoofing and UDP reflection attacks. Stronger liability laws for both ISPs and end-users who deliberately expose vulnerable systems to the internet would help too.
2 issues: I mentioned Mirai which was not spoofed or reflected - just lots of devices and real sources. Garbage traffic as in something you didn't request.
And the liability may work in one country, but you can't put a liability on ISPs internationally. Unless you want to end up in a situation where the US tells China their ISP is liable for spam traffic and hear that a US ISP is liable for traffic deemed illegal by the party.
Anything but dumb pipes is going to be extremely complicated internationally.
Lots of devices deployed by someone, set up by someone, and someone is paying the internet bill for those devices to be online. Maybe the ISP should be talking to that someone, even in an automated way where vulnerability scanners cut off their access and they need to manually re-enable it after having disconnected the offending device?
> US tells China their ISP is liable for spam traffic
I think most countries would agree that certain behaviors are undesirable. The damage & waste of time caused by spam and malware transcends political and religious opinions for example.
The solution doesn't have to be a legal liability per-se, simply a code of conduct that all ISPs agree to and anyone found to be in breach will get cut off by the others. They're not necessarily on the wrong side of the law, but for all intents and purposes their business is done because nobody will peer with them.
Looking at that graph, there is way too much scatter / too little data in order to plot clean exponential fits like that. A linear model would likrly fit the data just as good. At least throw on some bands indicating uncertainty in the fit.
There are three distinct series of data (indicated by color) there.
Looking at each individually I have to disagree. They look like perfectly kosher (and in fact good) exponential fits. Bands would still be nice though.
I agree. It's better to cut off the line past the point where it no longer has any data to support it.
In this case it's about 2.5 years of missing data that is substantiating a claim of 2x+ increase during the same time. It's reasonable to say it had an exponential function up until two years ago, and we don't know what it was since then.
R squared doesn't capture that inaccuracy because there is no data to compare it against during the scary time period in question. It's just a line in a void.
The trend-lines were really just to guide the eye (the text gives this context), but if you really must know: the R^2 for all three fits was above 0.9, suggesting the exponential growth model is reasonable.
As you say, the pps and bps don't show much curvature, so a linear fit could indeed work for them for the displayed portion of the graph. But it's non-sensical when you look further back in time... predicting negative attack volumes prior to mid-2011. ;)
The criticism that it would go negative in the past is meaningless. Of course it’s unphysical, but that doesn’t mean a linear model isn’t appropriate for today’s data.
You would just choose a point of time in the past where you believe the model is inapplicable. Maybe it was a different linear model then, or maybe mostly constant, who cares, we’re modeling today’s range of data, which might be well explained by a linear model, doesn’t really matter what 2011’s data was doing unless we separately believe the same growth law had to apply across all the years, and I see no reason why.
Another alternative model besides linear would be quadratic (or another X^(1+epsilon) polynomial where epsilon is small). This would avoid the problem of negative data and likely fit the data better than an exponential.
I think the question is really about the growth of volume of connected, compromised devices. Growth curves are often sigmoid shaped, meaning, exponential until they're not. The exponential is often great for modeling growth trends up until the plateau, but it's hard to know when the corner will turn.
Exponentials are also well motivated by differential equations... (Say, if you're modeling growth of IOT devices based on word of mouth.) Polynomials with degree 1+epsilon, less so.
> The trend-lines were really just to guide the eye
That's the problem, no? You can draw a line/curve over any data and convince a lot of people that it's relevant, but that's more cognitive bias than anything.
> the R^2 for all three fits was above 0.9, suggesting the exponential growth model is reasonable
What is the R^2 for other fits? Say linear or quadratic (the most obvious alternative choices)?
> it's non-sensical when you look further back in time
Of course, because the factors that are at work in producing the data are not constant in time. So there is no reason to expect a single curve fit with a single set of parameters to be applicable for all times.
Take a site offline then tell the owners that if they pay you 5 bitcoin you'll bring it back. Google wouldn't ever pay but a medium/large retail company may if they are losing 100k/hour in sales
For a similar price you can pay ddos defense to a real non-criminal company and like Google ignore their threats. This is a good investment for everyone as it lowers the rewards of crime and thus gets criminals out.
What prevents the criminals from either increasing their firepower to the point here DDoS defense providers can't handle, or outright run/be affiliated with those defense providers?
It's hard and expensive controlling enough machines to ddos Google. And Google wouldn't want to work with someone who was ddosing sites for the same reason or doesn't work with the Mafia.
I would regularly see attacks of exactly 300 second duration. I imagine people were kicking the tires on DDoS services for the lolz. Almost always targetting our www servers, rather than anything useful.
I don't think big tech is all that eager to actually solve this. Mitigate - yes, but not solve. What is their motivation to really fight this? They have capacity to spare and insecurity is a wonderful driver of cloud service adoption. Also, they are in a much better position than smaller companies to deal with security-related regulation if something of that sort will pass congress.
IoT is a huge contributor to the DDoS problem right now, but I don't see any big names campaigning against it. It's not really seen as a threat, but as an opportunity. Many problems in security are, in fact, opportunities for big tech to consolidate power, squash smaller competitors and charge more for basic services.
Poorly implemented IoT devices are a big contributor, but it's possible to design high quality IoT devices that are a net win. That's why big tech companies aren't campaigning against IoT.
For the rps (requests per second) it doesn't seem to be exponential in their graph (as the line indicates), but actually more linear [1]. That's interesting because it suggests that most attacks are increasingly network based rather than application based.
I guess we're now at the point where we can just spin up more machines to absorb attacks and use various forms of caching, but it's not really possible to do the same for networking.
I suspect packets per second has been scaling slower than bandwidth. For instance consumer routers can push 1 gigabit but only at rather large packet sizes. This is a major selling point for commercial networking gear
Something I would like to see is the attacks normalized over the number of customers - are they also increasing their customer base exponentially? If so, maybe this data really isn't so interesting...
You can defend on your own HW, using several methods:
-A proxy, e.g. Cloudflare
- A fat link, e.g. 100Gbps Cogent + a good hardware firewall (completely independent scenario)
- Use a GPRE tunnel based mitigation, if you own your own AS number, e.g. by Verisign
Cloud providers beat the DDoS drum to convince on prem companies that they're only safe on the big cloud.
I used to run something like 100 blogs for our clients. None ever DDoSed. I surmise getting DDoSed is as rare as getting hit by lightning.
You can protect web apps against smaller attacks easily by just doing a SPA on a CDN and verifying tokens on your load balancer. If you use SHA token verification (signature based JWT verification is slow) your load balancers can probably reject traffic at line rate.
Run a pile of servers behind 40 gig interfaces and a DDoS attack will be the data center's problem. You'll saturate their line before your service goes down. Combine that with DNS or JS client side load balancing and multiple data centers will go down before your app does.
If you've got a SPA using static files hosted on a CDN it's probably the best way to load balance. Like DNS but you have control over backend selection.
You can set retry interval a lot shorter than DNS. No problems with DNS caching. And you can try secondary on HTTP errors, not just connection timeout. In my experience clouds like to fail in ways that return 500/400 instead of timing out
> In March 2014, malicious javascript injected into thousands of websites via a network man-in-the-middle attack caused hundreds of thousands of browsers to flood YouTube with requests, peaking at 2.7 Mrps (millions of requests per second).
What would be the reason for leveraging this kind of attack against a target like YouTube? It seems like an odd target
I'm starting to think that the proliferation of DDoS protection from big cloud providers is actually driving an arms race in the scale of these attacks, and making in the end things worse for everyone except the biggest providers.
78 comments
[ 2.6 ms ] story [ 145 ms ] threadEach packet has a different length (due to different payloads), hence now we are switching to an inconsistent unit of measurement?
As a result, there are some people whose ability to withstand a DDoS has more to do with packets per second than bits per second, so it becomes a useful metric.
Right, definitely. A large sports streaming event might (easily!) do 10Tbps across a country, even more, but each GB served may only represent ~5k requests - 10e3/8 * 5e3 = 6.25M RPS, with each request being a single packet (small GET requests). That's big, but workable across enough machines for a big provider.
500M Pps is 80x that figure - you're basically DoS'ing header processing filters in your networking hardware.
[1] I wish I were joking, but this was verifiably the source of most of the volumetric DDoS I fought at my last job.
It's not needed for any application that I've ever heard of.
Why would anyone install this? On an Internet-facing server no less!?
Right behind this, I’ll put cloud providers who helpfully put default 0.0.0.0/0 rules in their firewalls.
https://en.wikipedia.org/wiki/Character_Generator_Protocol in case anyone else had never heard of this... um... unfortunate protocol. Works on TCP and UDP, basically "DDoS vector as a service" if I ever saw one (seriously, could you design a better protocol for DDoS?). Just... Who? How? Why!?
The big things these days are:
spoofers who generate a ton of syns to legit destinations which result in a lot of syn+ack to the victim. Bcp38 would help here.
Botnets generating a ton of UDP to destinations. Hosters, cloud providers (especially those with vulnerable/open EMR clusters) and broadband ISPs with easily compromised customers are the problem here. Kudos to those who take down the botnet command and controls.
Memached/ntp/cldap amplifiers. Still out there, still a problem. Thankfully a few of these services are policed at large peering interconnection points.
Reading. ... Yep. The article even includes the obligatory "try harder" bit about security.
There’s a lot of work and very little money in replacing outdated networking protocols with smarter/more secure protocols. And there’s very little punishment (from customers or regulators) for suffering from massive cyber attacks, even if you expose the sensitive information of people who never even chose to give you their information (like Equifax). There’s no punishment or reputational damages from having your network traffic rerouted from a BGP attack. The consumer pays all the cost and they don’t even change their behavior after paying that cost, so why would companies bother?
Normally this is a situation where governments would step in and protect their citizens, but... well there’s just too much money on the line. So the programmers get the blame and the customers pay the price and security vendors make another $100 million and we do it all over again next month.
https://www.atlanticcouncil.org/programs/scowcroft-center-fo...
Even when we have solutions to some issues (authenticated BGP), people don't implement it and there's nothing Google can really do about that. (Apart from not accepting traffic from those ASes?)
And the liability may work in one country, but you can't put a liability on ISPs internationally. Unless you want to end up in a situation where the US tells China their ISP is liable for spam traffic and hear that a US ISP is liable for traffic deemed illegal by the party.
Anything but dumb pipes is going to be extremely complicated internationally.
Lots of devices deployed by someone, set up by someone, and someone is paying the internet bill for those devices to be online. Maybe the ISP should be talking to that someone, even in an automated way where vulnerability scanners cut off their access and they need to manually re-enable it after having disconnected the offending device?
> US tells China their ISP is liable for spam traffic
I think most countries would agree that certain behaviors are undesirable. The damage & waste of time caused by spam and malware transcends political and religious opinions for example.
The solution doesn't have to be a legal liability per-se, simply a code of conduct that all ISPs agree to and anyone found to be in breach will get cut off by the others. They're not necessarily on the wrong side of the law, but for all intents and purposes their business is done because nobody will peer with them.
Sure. That's a possible meatspace solution and was used by some providers. It's not a protocol level solution though.
> simply a code of conduct that all ISPs agree to and anyone found to be in breach will get cut off by the others
Totally. Same thing could be applied to phone spam. It seems that nobody's that interested in it though.
Looking at each individually I have to disagree. They look like perfectly kosher (and in fact good) exponential fits. Bands would still be nice though.
In this case it's about 2.5 years of missing data that is substantiating a claim of 2x+ increase during the same time. It's reasonable to say it had an exponential function up until two years ago, and we don't know what it was since then.
R squared doesn't capture that inaccuracy because there is no data to compare it against during the scary time period in question. It's just a line in a void.
Or change to thin dashed line when no more data
As you say, the pps and bps don't show much curvature, so a linear fit could indeed work for them for the displayed portion of the graph. But it's non-sensical when you look further back in time... predicting negative attack volumes prior to mid-2011. ;)
You would just choose a point of time in the past where you believe the model is inapplicable. Maybe it was a different linear model then, or maybe mostly constant, who cares, we’re modeling today’s range of data, which might be well explained by a linear model, doesn’t really matter what 2011’s data was doing unless we separately believe the same growth law had to apply across all the years, and I see no reason why.
I think the question is really about the growth of volume of connected, compromised devices. Growth curves are often sigmoid shaped, meaning, exponential until they're not. The exponential is often great for modeling growth trends up until the plateau, but it's hard to know when the corner will turn.
Exponentials are also well motivated by differential equations... (Say, if you're modeling growth of IOT devices based on word of mouth.) Polynomials with degree 1+epsilon, less so.
That's the problem, no? You can draw a line/curve over any data and convince a lot of people that it's relevant, but that's more cognitive bias than anything.
Prediction is hard, especially about the future ;)
What is the R^2 for other fits? Say linear or quadratic (the most obvious alternative choices)?
> it's non-sensical when you look further back in time
Of course, because the factors that are at work in producing the data are not constant in time. So there is no reason to expect a single curve fit with a single set of parameters to be applicable for all times.
https://imgur.com/ksYbkEv
IoT is a huge contributor to the DDoS problem right now, but I don't see any big names campaigning against it. It's not really seen as a threat, but as an opportunity. Many problems in security are, in fact, opportunities for big tech to consolidate power, squash smaller competitors and charge more for basic services.
I guess we're now at the point where we can just spin up more machines to absorb attacks and use various forms of caching, but it's not really possible to do the same for networking.
[1] https://storage.googleapis.com/gweb-cloudblog-publish/images...
ie. "you can pay the muggers or build yourself a castle with a drawbridge"
I used to run something like 100 blogs for our clients. None ever DDoSed. I surmise getting DDoSed is as rare as getting hit by lightning.
You can protect web apps against smaller attacks easily by just doing a SPA on a CDN and verifying tokens on your load balancer. If you use SHA token verification (signature based JWT verification is slow) your load balancers can probably reject traffic at line rate.
Run a pile of servers behind 40 gig interfaces and a DDoS attack will be the data center's problem. You'll saturate their line before your service goes down. Combine that with DNS or JS client side load balancing and multiple data centers will go down before your app does.
If you've got a SPA using static files hosted on a CDN it's probably the best way to load balance. Like DNS but you have control over backend selection.
You can set retry interval a lot shorter than DNS. No problems with DNS caching. And you can try secondary on HTTP errors, not just connection timeout. In my experience clouds like to fail in ways that return 500/400 instead of timing out
What would be the reason for leveraging this kind of attack against a target like YouTube? It seems like an odd target