I believe it. Very insignificant data point but - I've worked for small scale startups and larger scale enterprises, and in both cases you can see bots and crawlers trying to do random stuff. I don't think it was actually %70+ of the traffic, but can see specific types of businesses getting even more hits from them.
Would this be solved by just providing APIs? Scraping is a pain and people only do it when there's no API. (Though these days a lot of businesses offer unofficial 3rd party APIs by scraping.)
It's also stupid and massively wastes computation and traffic on both ends.
Then people would think "but then I'd have to pay money to offer a free API", but they're already doing that in a more expensive way via the web interface anyway.
Legitimate scrapers, maybe. Everyone else does it to circumvent the API limitations, by posing as real traffic. APIs imply API keys which can be traced and banned.
Well, that's what happens when you make the API more limited than what people can do with scraping (or with using the public "our own web client" API key).
No, there’s a huge number of bots out there just doing stupid hacky stuff like trying for wp/admin/. Vast numbers of what I’d characterise as script-kiddy type attempts. Looking at the raw logs of a webserver not behind something like CloudFlare really opened my eyes to this. There’s an extraordinary number of garbage requests.
Yea, there was an HN thread complaining about WAF's a few days ago, but damn, if you're on the public internet having some kind of WAF cuts down the bullshit responses by a massive amount.
Same thing goes for email. I had one particular domain I managed spam filtering on and for every 10,000 messages we received, 1 was delivered to users. The amount of crap requests on the internet is insane.
And that's just requests on 80/443. Hook up an IDS to the WAN interface and, you'll see thousands of attempts per day to exploit vulnerabilities across other services too.
My low-traffic personal projects get 90%+ malicious traffic.
I think a more efficient approach would be to prosecute those who do it - bandwidth costs money - and to implement mechanisms to deter them. A not insignificant number of ai workers think are entitled to using people’s bandwidth, servers and intellectual property for free. Until a couple of such people are taken to task this wont stop.
I manage a network of several hundred websites. About 2 years ago we started blocking bot/crawler user-agents. Some websites see thousands of these requests every day, skewing analytics and eating bandwidth.
You're better off using a combination of IP blocklisting and ratelimiting, both of which use some sort of binary tree to speed up lookups, rather than using a mega-regex which your solution proposes, which, with tons of backtracking, is very likely slower than serving the request would have been.
Not to mention that user agents are easily spoofable.
I do not believe. I do not believe that bots account for more traffic than all the streaming services, all the filesharing and all the porn crisscrossing the internet. I would believe 73% of requests, even 73% of page loads/scrapes, but 73% of all traffic is too huge a number.
Since the payouts are not tied to the individual watcher or listener, scams run rampant. I guess the temptation to manipulate the money flow is way higher than having a "fair" distribution of each subscribers money to the artists.
Most bots operate in the gray area of just getting the view count up and not messing with the ads. They can operate publicly and not have law enforcement raid their offices.
Yt definitely makes an effort to prevent that. You can read up on how botguard's virtual machine works in some older writeups.
I'm not defending the validity of this article but it is very rare that Netflix, Youtube, or any other stuff backed by the big CDNs(Content Delivery Networks) cross actual metered transit connections(for the networks, not including what the end user has to pay for their servivces.)
I would definitely believe that Bots are responsible for %73 of the traffic. Not sure how one would necessarily characterize them as "bad" but definitely wasteful. Worked at site that published science. They collected page view metrics. Doing something simple like excluding page views that where done by IP addresses that continuously cycled through all articles resulted in over a %70 reduction in page views. Needless to say it was never deployed because the high view numbers made the authors feel good.
bots are still a subset of the traffic for streaming, porn, etc. Whether its for ad fraud or faking popularity on streaming those bots are running 24/7.
However I agree, on its face 73% sound ridiculous.
I feel like this is trying very hard to push a particular narrative but not actually bothering to substantiate anything.
From the top...
> Bad Bots are increasing dramatically — Arkose estimates that 73% of all internet traffic currently (Q3, 2023) comprises Bad Bots and related fraud farm traffic.
Note that "Bad Bots" are never defined, and that 73% will never be explained, either to say how they mean it (73% of connections is plausible, 73% of packets is not, not in a world with Netflix and YouTube), or how they would measure such a thing.
Then we get into little gems like
> Scraping, it must be said, is a legally murky area. It is not specifically illegal; but if it defies a website’s published terms of use, it is certainly immoral.
Which... that's a claim you can make, but if you just throw out terms like "certainly immoral" without bothering to substantiate them I get to laugh at you.
The tendency to cast aspersions without substantiating them gets substantially more pronounced later:
> This is a website you can use to make sure your bots aren’t getting prevented by a website,” [...] “You can purchase this software. It has enterprise support and so on. But it is purpose built to commit crime. That is what it does. And there are many other different websites like this, but they look like legitimate businesses. It is a good example of a product purpose built to commit fraud.”
I'll grant that businesses don't like people bypassing their protections, but that doesn't make it automatically illegal or fraud to scrape or even to programmatically interact with sites. I'm not naive enough to think that it's not being used for illegal stuff, but that doesn't mean you get to paint the tools with a single brush.
That's a description, not a definition. If they said that they classified a bot as bad if it engaged in one of five described behaviors, that would be a definition, but that's not what they've done. (And if that was the definition they were using, then I'm going to call out scraping and account management as not sounding like malicious behavior)
The actual report from Arkose (linked elsewhere in the comments) is much more detailed in what they’re measuring and reporting.
That said, the language around the “73% of traffic” figure is still a little ambiguous but most plausibly means “73% of requests/sessions to the significant commercial properties we studied”
Assuming that’s what they mean, it’s reasonable that bots are banging on major sites three times more often than legitimate users.
I think the linked article is doing the sloppy sensationalizing and the report appears to be much less so.
To build on the above, it's worth thinking about the transition that happens between those two quotes:
> Scraping, it must be said, is a legally murky area. It is not specifically illegal
There's already a jump happening here from "it's not specifically illegal" to "it's a gray area because it's immoral." Even if it was certainly and unambiguously immoral, that does not make it automatically a legally murky area. In fact, scraping seems to be reasonably established as specifically legal.
So the article is already kind of edging away from reality, where they acknowledge "this is not a crime" but they want to blur that and say that it's legally dubious, even though what they really mean is that they believe it's immoral.
Which, fine, I don't respect it but whatever. But then we jump to:
> But it is purpose built to commit crime.
Wait, no. Scraping is not specifically illegal. This is the emulator debate again; we have something that really is strongly implied to be legal and has won court cases before. It's not like it's an untested gray area, we know that there are scenarios where scraping websites against the permission of the website owner is not illegal.
So the article pretends it's more of a gray area than it actually is, and then once it has thrown doubt on it, drops the pretense entirely and calls it illegal. Not all scraping is fraud and not all scraping is illegal, and it's pretty galling for the article to admit that scraping is not illegal and then to turn around and pretend that they didn't just admit that.
> It is also a good example of crime-as-a-service.
No, it's not. The article admitted that scraping is not specifically illegal. Tools to help scrape websites are not "crime-as-a-service" because objectively (and there is legal precedent backing this up) scraping is not a crime.
----
The propaganda strategy being employed here is to admit a piece of factual information (scraping is not illegal), however hesitantly, and then to pretend that it was never admitted.
If anyone calls the propagandist out, they can point to their admission and say, "no, I acknowledged that it's not illegal, see". But then for every other part of the conversation, they just ignore that and act like it is illegal, even though at best all they've put forward is that they think it's more dubious than courts currently recognize.
They concede the point only to the degree they're forced to and only to the degree that would prevent someone from saying that they're lying or ignoring reality. But as soon as the context changes, they go back to acting like the point was never conceded.
This has the benefit of not only allowing them to make the same arguments they wanted to make before, but also sort of trains listeners to think of the original conceded point as more murky than it actually is. "Sure," the reader thinks, "it's technically not illegal, but at the point we're talking about crime-as-a-service and illegal tools, surely there's something legally dubious going on, this must be more of a gray area" -- even though not only has the article not backed up that idea, it's actually admitted that the idea is wrong.
> There's already a jump happening here from "it's not specifically illegal" to "it's a gray area because it's immoral. Even if it was certainly immoral, that does not make it automatically a legally murky area. In fact, scraping seems to be reasonably established as specifically legal.
Yeah, I didn't want to get distracted from my main arguments, but that stuck out to me too - in my non-lawyer amateur understanding, the LinkedIn case[0] means that in the US scraping very much is explicitly legal.
Yeah, IANAL but that is my understanding as well. You have to love the boldness of reframing "a court has specifically ruled this legal" as "no one has ruled it illegal". It's technically true while also heavily implying that it's just untested and it could be ruled illegal at any time.
It's very: "technically speaking, the Constitution doesn't explicitly require people to quarter troops in their own homes, but..."
Not outright wrong, just kind of missing a lot of context and quietly implying a lot of stuff that is wrong.
> Note that "Bad Bots" are never defined, and that 73% will never be explained, either to say how they mean it
Arkose runs a bot detection service, OpenAI being one of their customers (check your requests on chat.openai.com). I’m guessing the 73% are the number of requests/sessions that fail their JavaScript fingerprinting.
Not sure how reliable these numbers are as I have a Go library specifically for passing their fingerprinting in pure Go (No JS) and it has been working for a while without much change
And in the opposite direction, I disbelieve stats about bots because of the number of web sites that treat my real browser driven by an actual human as a bot:)
(My personal favorite was being blocked by cloudflare, and then getting an email from cloudflare telling me how great they are because they block so many (alleged) bots)
And this is why we can't have nice things.
Now that we will/are starting to have AI-created content, and visits from bots, comments from AI bots.. we need some new metric I reckon
I think "offline" means mocking your ethernet driver to produce packets confabulated by a locally-running copy of ChatGPT, so you can doomscroll the internet even without needing to pay Comcast for the privilege.
Cloudflare refused to serve me the linked page after I enabled JS and passed the captcha, "The action you just performed triggered the security solution". Each time it happens, I wonder whether website owners then look at the statistics, see the reported blocks, and are reinforced in their belief that Cloudflare does something useful. But in this case I also wonder whether that number was derived from such blocked requests as well; whether they count me as a bad bot or just a bad person.
> The biggest increases in attacks from Q2 to Q3 are SMS toll fraud (up 2,141%),
Toll Fraud is rampant lately. I can tell because people have attempted it on nearly every consumer-facing project I've worked on. Additionally, the ill-advised "text the app to your phone" gimmicks have disappeared (fraudsters have cleaned up on direct, public endpoints to send SMS).
Shameless plug, but I've written a little about Toll Fraud and how to prevent it [0]
I would confess that I had written a bad bot in the past, fetching data every time I runs it multiple times a day when the data source only updates once or twice a week. I didn’t know better back then
The article mention sms tool fraud as the biggest increase..
We faced the same problem at Zenly and had to build our own anti-spam strategies to prevent it. We used multiple providers to improve our conversion rate and reduce cost. We are now building this as a service https://www.ding.live/ and are seeing huge improvements for our customers in term of cost savings and conversion rate.
I help run some websites with many links. I estimate that more than 90%, possibly close to 99% of our traffic is crawler bots, even after blocking some of them based on behavior.
62 comments
[ 2.8 ms ] story [ 115 ms ] threadIt's also stupid and massively wastes computation and traffic on both ends.
Then people would think "but then I'd have to pay money to offer a free API", but they're already doing that in a more expensive way via the web interface anyway.
Legitimate scrapers, maybe. Everyone else does it to circumvent the API limitations, by posing as real traffic. APIs imply API keys which can be traced and banned.
Same thing goes for email. I had one particular domain I managed spam filtering on and for every 10,000 messages we received, 1 was delivered to users. The amount of crap requests on the internet is insane.
My low-traffic personal projects get 90%+ malicious traffic.
Many sites intentionally make scraping as painful as possible because they don't want anyone scraping them.
Tools like this help: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blo...
Not to mention that user agents are easily spoofable.
Anyone malicious who knows what they're doing will just send you a statistically common user agent string.
Under most http libraries this is a one-liner:
I mean, it would not surprise me at all if bot traffic on Youtube is larger as in bytes/s of video, than actual visitors.
Remember that Google has no interest in disclosing how many bot views there are.
There was this news article about Spotify banning some musicians accounts due to listen farming.
https://www.musicbusinessworldwide.com/great-big-spotify-sca...
Since the payouts are not tied to the individual watcher or listener, scams run rampant. I guess the temptation to manipulate the money flow is way higher than having a "fair" distribution of each subscribers money to the artists.
Yt definitely makes an effort to prevent that. You can read up on how botguard's virtual machine works in some older writeups.
I watched today a couple hours of Youtube videos. How many bots are needed to match that much traffic?
if we're talking about requests it makes sense. I just have to open my apache/nginx logs to see how many bots are attempting to get in.
However I agree, on its face 73% sound ridiculous.
https://www.arkoselabs.com/wp-content/uploads/Breaking-Bad-B...
From the top...
> Bad Bots are increasing dramatically — Arkose estimates that 73% of all internet traffic currently (Q3, 2023) comprises Bad Bots and related fraud farm traffic.
Note that "Bad Bots" are never defined, and that 73% will never be explained, either to say how they mean it (73% of connections is plausible, 73% of packets is not, not in a world with Netflix and YouTube), or how they would measure such a thing.
Then we get into little gems like
> Scraping, it must be said, is a legally murky area. It is not specifically illegal; but if it defies a website’s published terms of use, it is certainly immoral.
Which... that's a claim you can make, but if you just throw out terms like "certainly immoral" without bothering to substantiate them I get to laugh at you.
The tendency to cast aspersions without substantiating them gets substantially more pronounced later:
> This is a website you can use to make sure your bots aren’t getting prevented by a website,” [...] “You can purchase this software. It has enterprise support and so on. But it is purpose built to commit crime. That is what it does. And there are many other different websites like this, but they look like legitimate businesses. It is a good example of a product purpose built to commit fraud.”
I'll grant that businesses don't like people bypassing their protections, but that doesn't make it automatically illegal or fraud to scrape or even to programmatically interact with sites. I'm not naive enough to think that it's not being used for illegal stuff, but that doesn't mean you get to paint the tools with a single brush.
This stuck out to me as well. What are "bad bots" and how do they differ from "good bots", and how are the authors determining which is which?
Not in a world where pron exists.
- second paragraph, first sentence
That said, the language around the “73% of traffic” figure is still a little ambiguous but most plausibly means “73% of requests/sessions to the significant commercial properties we studied”
Assuming that’s what they mean, it’s reasonable that bots are banging on major sites three times more often than legitimate users.
I think the linked article is doing the sloppy sensationalizing and the report appears to be much less so.
> Scraping, it must be said, is a legally murky area. It is not specifically illegal
There's already a jump happening here from "it's not specifically illegal" to "it's a gray area because it's immoral." Even if it was certainly and unambiguously immoral, that does not make it automatically a legally murky area. In fact, scraping seems to be reasonably established as specifically legal.
So the article is already kind of edging away from reality, where they acknowledge "this is not a crime" but they want to blur that and say that it's legally dubious, even though what they really mean is that they believe it's immoral.
Which, fine, I don't respect it but whatever. But then we jump to:
> But it is purpose built to commit crime.
Wait, no. Scraping is not specifically illegal. This is the emulator debate again; we have something that really is strongly implied to be legal and has won court cases before. It's not like it's an untested gray area, we know that there are scenarios where scraping websites against the permission of the website owner is not illegal.
So the article pretends it's more of a gray area than it actually is, and then once it has thrown doubt on it, drops the pretense entirely and calls it illegal. Not all scraping is fraud and not all scraping is illegal, and it's pretty galling for the article to admit that scraping is not illegal and then to turn around and pretend that they didn't just admit that.
> It is also a good example of crime-as-a-service.
No, it's not. The article admitted that scraping is not specifically illegal. Tools to help scrape websites are not "crime-as-a-service" because objectively (and there is legal precedent backing this up) scraping is not a crime.
----
The propaganda strategy being employed here is to admit a piece of factual information (scraping is not illegal), however hesitantly, and then to pretend that it was never admitted.
If anyone calls the propagandist out, they can point to their admission and say, "no, I acknowledged that it's not illegal, see". But then for every other part of the conversation, they just ignore that and act like it is illegal, even though at best all they've put forward is that they think it's more dubious than courts currently recognize.
They concede the point only to the degree they're forced to and only to the degree that would prevent someone from saying that they're lying or ignoring reality. But as soon as the context changes, they go back to acting like the point was never conceded.
This has the benefit of not only allowing them to make the same arguments they wanted to make before, but also sort of trains listeners to think of the original conceded point as more murky than it actually is. "Sure," the reader thinks, "it's technically not illegal, but at the point we're talking about crime-as-a-service and illegal tools, surely there's something legally dubious going on, this must be more of a gray area" -- even though not only has the article not backed up that idea, it's actually admitted that the idea is wrong.
> There's already a jump happening here from "it's not specifically illegal" to "it's a gray area because it's immoral. Even if it was certainly immoral, that does not make it automatically a legally murky area. In fact, scraping seems to be reasonably established as specifically legal.
Yeah, I didn't want to get distracted from my main arguments, but that stuck out to me too - in my non-lawyer amateur understanding, the LinkedIn case[0] means that in the US scraping very much is explicitly legal.
[0] https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
It's very: "technically speaking, the Constitution doesn't explicitly require people to quarter troops in their own homes, but..."
Not outright wrong, just kind of missing a lot of context and quietly implying a lot of stuff that is wrong.
Arkose runs a bot detection service, OpenAI being one of their customers (check your requests on chat.openai.com). I’m guessing the 73% are the number of requests/sessions that fail their JavaScript fingerprinting.
Not sure how reliable these numbers are as I have a Go library specifically for passing their fingerprinting in pure Go (No JS) and it has been working for a while without much change
(My personal favorite was being blocked by cloudflare, and then getting an email from cloudflare telling me how great they are because they block so many (alleged) bots)
You mean like wifi?
its kinda expensive and dangerously addictive
the resolution and refresh rate are amazing
We analyzed tens of billions of sessions worldwide across industries, [...]
I think they mean: 73% of all commercial sessions are malicious.
Very sensationalist, at the end of the day they are selling something
Toll Fraud is rampant lately. I can tell because people have attempted it on nearly every consumer-facing project I've worked on. Additionally, the ill-advised "text the app to your phone" gimmicks have disappeared (fraudsters have cleaned up on direct, public endpoints to send SMS).
Shameless plug, but I've written a little about Toll Fraud and how to prevent it [0]
[0] https://koptional.com/article/how-to-stop-twilio-toll-fraud/
Producing a RSS feed and clean scrapable page is always a good thing to do to boost your page recognition and popularity.
You data does not have to be scrapable, but you page should always be bot friendly.
When Netflix was a cheap stock but it made up a significant part of internet traffic, I assumed it could be the next big thing.
So how do you invest in "bad bots", whatever this is supposed to mean. Honestly, I don't trust the numbers.