mmm, in many cases these residential proxies are media boxes, and they consent as much as anyone else consents to what amazon, or google or facebook does; it's buried somewhere in the recesses of the TOS.
The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
Because this isn't clearly against the law, nor should it be. If websites want to ban based on IP address lots of innocent users get caught in the cross-fire.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
> The question is more about why the US and others can't properly enforce the bullshit all this amounts to.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
The excessive scraping and ignoring robots.txt only breaks the informal social contract established over the past decades of the open internet.
The real problem is the companies offering money to developers if they include unrelated SDKs in their calculator or flashlight (for example) applications. Those SDKs add functionality to incorporate those devices into a network that can be used for scraping. The traffic is little, but is distributed over millions of residential devices all over the world, making it difficult to categorize or block. That should be illegal, and that's what Google et al can be expected to be policing on their app stores.
On what grounds would it be illegal though? Things don't become illegal just because you don't like them. They may become illegal just because the president doesn't like them, but I don't think you're him, and in the absence of that, there has to be a majority of Congress and most of them want a reason.
It's not illegal to run code on a device without informed consent to everything the code does. The CFAA may be excessively broad but it isn't that broad.
ask any of the people who installed it if they understand what they're doing and especially what the consequences of that are. i doubt you could get a dozen people who do of the millions and millions who installed it.
The comments are not showing up for me now, but when they were still showing for anonymous users, there was a link to https://commoncrawl.org. I've been sort of worried about letting agents hit websites, I wonder if a fetch_url agent tool could be made to look in common crawl first before hitting the web for it?
just their smallest dataset looks to be 6 TB _compressed_. not a thing you can really ship as part of the agent. but if somebody made a fetch_url tool that sharded that across all users of it, i'd give it a try. could probably just layer that on top of bittorrent or IPFS or something.
People think this is causing issues for data collection for LLMs, but in reality it's not and there are several very trivial mechanisms to employ in data collection to bypass the "poison data" issue.
The internet landscape was already poisoned with fake data, fringe conspiracies, and text before this Poison Fountain initiative.
exactly i took a look at that subreddit and doesnt look like theres any professionals just bunch of anti-AI users who thinks they are smarter
its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective
also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities
I've banned this account because we don't allow single-purpose accounts on HN, and your account has been doing that for quite some time now.
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
It's just a sign that single-agenda accounts aren't allowed here—no more, no less. As I said above, "We ban such accounts regardless of what the single purpose happens to be".
In such cases, someone usually accuses us of making the moderation call because we secretly agree with or oppose one side of the topic. I'm not sure if your comment is meant that way or not. But I had never heard of this particular project and have no opinion about it one way or the other.
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
I think the reasoning is about having alt accounts for different purposes. He intention is to map one human to one account and have all of their thoughts from that one account, instead of one human having one account to discuss scraping on, and a different account to discuss crypto on.
The most recent 70 comments, not to mention every submission of the last 6 months, were all about the same thing. That's extreme. Not all mentioned that specific project, but there is only one topic there and the posts were extremely repetitive. This is not a close call.
Jon's been around a while and some of the piss and vinegar of youth may have subsided. He does tend to show up with LWN comes up, whether as a topic of discussion (or more often) from submitted articles. That's his baliwick, and again, I don't fault him at all for it.
Our other friend here is a more recent participant to HN, at least under this handle. (I don't know that there are others, only what I can see from this one.)
I understand how it can be confusing. The key factors in doing it this way are (1) the community regards older accounts, especially ones that have significant posting history, as more credible; and (2) doing it publicly rather than silently has transparency value.
I feel like the solution is a better common crawl. As nice as it would be to block the frontier AI labs from getting access to information, we should reset the baseline of information accessibility so there's less marginal advantage on these labs.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
I agree, if up-to-data data was available somewhere else and free, there would be no reason to pay hackers and scrape.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
Well this is not what is happening in practice, Wikipedia / Wikidata, OpenStreetMap, OpenFoodFacts... All provide APIs and even a full dump of their database available to download for free, but no, the stupid bots still DDoS them 24h/24.
What really confuses me is ... people always say, it's because companies are gathering data for AI training. Then why would they need to scrape the same page thousands of times per day?
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
That's really a huge issue right now (to some extent even before the AI hype) that almost everywhere google is effectively the only entity explicitly allowed to scrape.
Hah. I have a homelab with a couple of sites, including a personal Forgejo installation.
Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.
However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
The paradox of them selling "intelligence on demand" or "coding agents rivaling the best developers" and yet having dumb as fart bots/scrapers is lost on many. But not all.
There's no paradox, just deception. "AI" is dumb as a bag of rocks, but good at convincing people that it isn't. However, it's also a really good semantic vector search that can find and combine existing answers to many pre-asked questions.
Maybe they have just too much money at hand, would not surprise me, people are still investing into gen AI like there is no tomorrow. Also, for the completely criminal operations, you only have to find a way to infect and distribute your bot to, e.g. some common internet of shit device. Scaling is basically free afterwards as you don't need to ask anyone. The article also hints that those are actually the biggest problem.
Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.
Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s.
Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk.
It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.
It's not AI companies scraping these websites, it's AI companies creating a massively profitable need for data, and every random Joe with vibe coded scrapers tries to make a buck out of it.
It should really just be called DDoS, at a certain level of incompetence intent doesn’t matter. You’re right that there’s zero (information gathering) benefit over reasonable scraping which wouldn’t cripple the site.
750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour
Millions per hour is tens per second though; perhaps the fix is performance improvements
We have put in a number of performance improvements, yes. The nice thing about those is that they also make the site snappier when it's not under load. Right now we have just over a million items in our CMS, plus our publicly available mailing list archives, which are much larger, even if they're less frequently referenced.
It isn't. AI scraping has nothing to do with it, for the reasons you said. Someone wants the web to go offline, they are DDoSing the entire web, and it's working. For some reason we are tackling the symptom instead of finding out who that person is. Come on, it can't be that hard to subpoena Bright Data. The law enforcement system knows how to track down someone who's trying to be anonymous on the internet.
I had meta's crawler hitting the pi searcher at something like 5qps for days on end, just ... querying for substrings of pi, ignoring robots.txt, etc. it wasn't enough to break anything but it triggered a lot of alerts.
I can imagine that sites with dynamic content and potentially unbounded query types or pathnames are in danger from particularly stupid crawlers.
Grok actually shows a number of sources used for an answer. Once I asked it something simple and it apparently scanned 200 different websites. And it was just a short prompt. Now imagine millions of users asking for something multiple times a day.
Ironically in early 2023 a lot of websites went out of their way to block Common Crawl. Unsurprisingly that shifted scraping toward individual actors whereas the previous solution in research was to download CC dumps and process them.
We aren't sure if that really made a significant difference in Common Crawl's data quality. It does hurt our dataset from a humanities point of view, alas.
I'm sure there are those who would participate, either because they want their data to be captured by AI labs or as a form of compromise.
That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).
If you're referring to Common Crawl, which has existed since 2008, indeed your predictions are somewhat accurate. It's easy to opt out or limit what is collected. The crawling itself is inexpensive to us and the hosting is from the AWS Open Dataset Sponsorship Program. And there's no charge for downloading it.
Appreciate your kind words! Many opeople have worked at Common Crawl over the years, and it's been a labor of love fueled by positive comments like yours and the large list of PhD theses helped by our public web dataset.
Very little of it. When you see a million IPs systematically working their way through your URL space, it's pretty clear that there's a central control node behind it all.
Your earlier article suggests you aren't using a CDN. Might be well worth looking into - not for any bot detection so much as just having a good old fashioned cache in front of you.
Caches only help for pages that have been requested recently. The behavior of crawlers - going from one page to the next across the whole site - will probably not be mitigated significantly by a cache.
I've seen some logs where a bunch of random ips were hitting a client's search endpoint feeding what looked like user questions to it. Of course none of them returned anything useful but it was causing a lot of strain and even causing the site to go down (gotta love wordpress's stock search).
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
Most well-known/large agentic web tools I've seen are actually super honest about who they are -- even when they write out scripts they're very keen to identify themselves using user-agents. Most of the time those tools are fine - it's the ones that happen to have a random choice of the 5 most common Chrome/Firefox user-agents making sequential scrapes but cycling through IPs on African and South American residential IPs that are the problem!
At least those bots are easy to block though. I run a niche stats website for an esport and I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data - most of what they scrape is directly available from Valve's APIs.
> I have no idea why there's loads of residential trawlers/botnets with 10k+ IPs trying to get that data
Probably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.
On my sites, I see ClaudeBot consistently (and has been like this for over a year) ask for "${SITE}.com/base_dir1" and then get the redirect (Caddy does this automatically) to get "${SITE}.com/base1/base_dir1/" (trailing slash).
The hrefs on my sites include the trailing slashes for directories, so looks like ClaudeBot's internal code is stripping them off before requesting them, and therefore essentially makes almost 2x the requests to my sites for directories, half of them ending up being redirects back to the same url but with the trailing slash.
The issue with scrapping is the intensity and volume of bots.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
A lot of websites want "bot defense" due to high volume scrapers, and that "bot defense" often also ends up blocking low-volume wget/curl and polite crawlers like Common Crawl's CCBot.
Cloudflare can verify certain bots when they come from known ip addresses. So if your site is using cloudflare it can let CCBot if it has done the verification.
Residential Proxies are the most emblematic technology of our era- a group of people looked at something that used to be considered a crime (botnets) and realized that if they just did it openly, no one would ever punish them.
You're one with the lower case shibboleth so I have no doubt you surround yourself with dishonest founders, but faking users is pretty damn low on the usecases for residential proxies.
I said they're unethical because they tend to be hidden in innocuous seeming apps or sprung on unwitting individuals via clickwraps on their smart devices.
ive seen unscrupulous founders fake traction during diligence, which is my day job
but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks
then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection
How strange, my fantasy was totally on point and literally describes your day job! But does the forced lowercase thing just erode one's ability to make a coherent point?
"ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks"
You've never seen this, making this non-sequitur? Or this "directionally correct by my priors" speak for something else?
-
Either way it has nothing to do with why the proxy is unethical (I already covered that) and everything to do with why their startup is unethical
(or not? if they're not being assholes about the scraping more power to them, I've used proxies + agents to let my users get their own data off other sites that felt it was their right to block people from accessing their own data easily)
No one's firing up a residential proxy to read your blog, and the corporate AI scrapers have all the resources in the world even without residential proxies
They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
I actually read a really interesting article from a relatively small blog explaining that they're receiving a massive amount of scraper traffic from residential proxies.
The article was called "The one we're commenting on"
> The LWN content-management system contains over 750,000 items (articles, comments, security alerts, etc) dating back to the adoption of the "new" site code in 2002. We still have, in our archives, everything we did in the over four years we operated prior to the change as well. In addition, the mailing-list archives contain many hundreds of thousands of emails.
Does that sound like your typical self-hosted blog?
The Bright Data mentioned in the article, as well as other similarly malicious but even harder to identify parties, most certainly do fire up residential proxies, no matter what the site, no matter how useless or duplicate the data which they're trying to get. Not at first - they start by trying to get your content from cheap data center connections - but as soon as some kind of bot-mitigation appears, they move to residential proxies to try and evade that, with a first tier coming from "global south" residential proxies, and then scaling up to (presumably more expensive / less widely available) proxies from the USA (I've seen some from Europe, but very few in relative terms). Each tier also appears to have the option to run JavaScript.
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
Mmm... your quote (IDK where it's from) mentions them having consent from device owners, but your FBI link cautions on how to avoid getting infected by malware.
If they have consent, they're not really botnets. Botnets involve infecting devices without the owners knowing.
It wouldn't be much different from e.g. open WiFis at restaurants and hotels and most VPN services.
And even if you spent a minute explaining the proposition to a user off the street, it still wouldn't be fair unless you laid out the drawbacks. Which leads me to a question.
There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?
> There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP.
Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.
Thank you. Gmail must not be like our fellow HN users we see here, quoting a couple:
“I'm tiny and only run little personal stuff. I just block vast IP address blocks.”
“Apologies. :( Since you say you've never visited the website before, then that means you're either in one of the countries or in one of the residential IP ranges that I've had to block.”
Although Google isn’t afraid to completely block iCloud private relay from Google scholar. Other sites may reject the first iCloud private relay visit, then reopen site in new tab and often automatically assigns new unblocked IP. Anyway, I would’ve thought it’s an acceptable cost from e.g. Google’s perspective to block ranges that did something bad once in spite of collateral damage.
> Anyway, I would’ve thought it’s an acceptable cost from e.g. Google’s perspective to block ranges that did something bad once in spite of collateral damage.
Almost nobody is on a static IP. ISPs have an interest to keep their customers connected and not affected by other customers, so they'd probably use as large address pools as possible for their dynamic IP rotations.
Tech companies like Google are interested in being global monopolies to dominate markets and also having as good and large-sampled statistical data as possible from everyone. Blocking thousands of users for each bad user that can likely easily switch to another block if they really want to doesn't seem effective. It would also affect their reputation. Who wants to rely on an email service that may block you because of other users on your IP block? Reliability is very important for businesses.
With small personal sites/blogs like the ones you mentioned, they can just block most of the planet without a problem. There's not as much reason to be as reachable as possible, and they're also more budget conscious.
This is a super dishonest characterization. Running software on a bunch of machines, even machines in other peoples' homes has never been a crime. Folding@home isn't a crime (obviously). It's controlling those machines without consent via malware that is criminal. And if it is open and consensual in exchange for something a person wants, it is unreasonable to compare it to botnets.
On people not reading agreements, that's a somewhat valid, but kinda weak argument. It's specially valid because software agreements presented on a window, with text that can't even be copied, with a "click here to agree" button tend to change over time and some don't even notify the users while pretending they apply. (I'm still waiting on online agreements requiring non-forgeable signatures to be enforceable, rather than just the word of somebody saying that a `true` on some row in a DB they control means they are.) OTOH, you can't just say, "It's too long. I didn't bother reading it, so it doesn't apply to me even though I benefited from it."
On "always covert and never consensual", it seems some IPs come from partnerships with ISPs. Beyond some ISPs being possibly regulated by their government on the matter of only offering their IPs to residents to control local prices, I don't see why ISPs shouldn't be able to offer some of their IPs for proxying. It would be a valid business model.
And it's made necessary because another group of people thought that selling IP blocking services would be a good idea. One party sells walls, another party sells ladders.
Well, one party gives away free walls if you agree to fill your castle with surveillance cameras you don't control.
Can BitTorrent’s architecture contribute anything useful here?
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
as well as the bot owners could would never believe that the torrent has been kept up to date. the only way to do that would compare to the actual site, so why not just scrape the actual site and be done with it?
We do. First off we have a public parquet-format index of all of the urls we crawl every month. And then that also lives in a HDFS table that determines when we want to recrawl a page we've crawled before.
Sorry, I wasn't clear. In order to know you have an up to date copy of the page you need two pieces of information:
1. When the page itself was last updated
2. When the crawled copy was last updated
In order to get an accurate date for 1, you have to crawl it, and if you are crawling it you might as well use that copy you just crawled.
The missing here is that for pretraining AI models should accept a cut off date and not worry about being perfectly up to date. Keeping things up to date is more useful developing internet search engines for grounding.
I maintain a long robots.txt and also 403 the user agents there when they request anything but robots.txt. I've blocked traffic from a few countries altogether and also block anything on SpamHaus' DNSBL.
>There are ways to tell the difference — the bots usually do not fetch images or CSS, for example — but, by the time that determination is made, the address in question will not be used again. Blocking the address at that point is just a waste of time.
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
> do you want random IoT devices talking to your web server?
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
Blocking in ipv6 works roughly the same way as in ipv4, just that the scale is different. Instead of blocking something like a company's /24 or an ISP's /16 when they don't respond to abuse messages, you block the company's /48 or the ISP's /32. It'll vary per organisation how large a range they got exactly but you can see that in WHOIS. End users are no longer at a /32 (v4) but at /64 (v6), or some prosumers might have a /29 (v4) and /56 (v6). Same concept, just a different prefix length
Various ISPs give out either a /64, a /56 or a /48. Anything else is very unusual. A normal approach is to limit by /64 at first, limit a /56 to 3-4 times the rate limit of a /64 and a /48 to 3-4 times that again. Someone who has a /48 gets to enjoy 16 times the rate limit of someone who has a /64 but that's not too bad, and you don't have to tune anything per ISP.
Anyone who has bigger than /48 is no longer an individual user. They could be an ISP set up solely for scraping, though that comes with some fees and requirements.
> ...we have tried to minimize the impact on real readers as much as possible. We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a significant improvement.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
You don't have to wait for the flood. You can add rules to add weight based on system load, and add more difficulty levels beyond the ones in the default config.
I've only just started using it, and the existing config DSL doesn't let you do this, but it's just a patch away (in principle): instead of using system load as a signal to increase or decrease weights, use a combination of finalized weight and system load (or a load-equivalent or net-traffic-equivalent signal from some openmetrics agent) to select between difficulty buckets.
So, as an example, difficulty could have an arbitrary scale independent of load, and regardless of that scale, it could increase by 1 at 50% cpu, 2 at 80% cpu, etc. I'm also not sure currently if a challenge will re-appear if anubis determines that a client should get a difficulty N challenge, but they only have a token for an N-1 or N-2 challenge. Easy enough to implement in principle, I just haven't looked at the code at all. I shall test it to see if it already does that.
From my understanding this is also how cloudflare bot protection has worked for a long time, and then they look for entropy in user input to confirm the user is human. Also how recaptcha without images works.
Google and Cloudflare both are not just looking at entropy of mouse movements, that was cracked years ago, they are fingerprinting you and correlating your session with all your activity cross domains to score your botlike behavior.
I doubt they are doing it. You just have to get on a VPN to and see yourself being flooded with captchas despite browsing the web like a normal human and solving dozens of captchas along the way.
Profiling visitor traffic to protect against abuse is a legitimate use case, even for GDPR. It's only a violation if that usage is not disclosed in the ToS.
Supposedly, but not really. I regularly encounter sites where cloudflare serves me with an ambiguous ban notice rather than a proof of work. What's worse is that these apparent IP bans take effect even if I already had a valid active session (ie previously passed the check).
Yes, a VPN involved. That doesn't make it okay and notice that anubis by default works without issue (though possibly with a more difficult challenge) in the exact same scenario.
Regardless of the precise logic it's no excuse for the policy. Simply hand out a sufficiently difficult PoW to prevent widespread abuse.
I'm quite certain it isn't a generic "datacenter" list though because a given VPN exit that was working will suddenly stop. Meanwhile I have a valid cookie yet that is disregarded.
That is an entirely different matter. They only block (AFAIK) editing which is a scenario where PoW would not be expected to solve the problem. What I was talking about here is IP banning as a (boneheaded) mitigation against high volume scrapers.
Except when it throws you into a reload loop. It's pretty buggy, and trivial to bypass.
And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.
When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.
Putting aside the question of whether it will continue to work, even somewhat, against botnets, I find your first paragraph staggering.
Reload loops, or being able to "bypass" anubis (unless you merely mean bypassing it for the token validity period by solving a challenge), sound like misconfigurations. There's no reason for anubis itself to cause reload loops; it's tricky to configure a webserver to use it in some scenarios. Similarly, any ability to bypass anubis probably means the site is using it in auth/challenge mode only, and then misconfigured their webserver's auth checking.
I looked this up and realised it’s the page I’d seen briefly on a range of websites lately. It’s not annoyed me at all. Not nearly as much as having to complete captchas with slow refreshing tiles.
The captcha itself (matching pictures to text) is mostly for ML training data. I think pass/fail is mostly based on heuristics like how you moved your mouse which could get analyzed before you complete the captcha. https://www.techradar.com/news/captcha-if-you-can-how-youve-...
reason why is 1. Google and others really needed the training data, and 2. it probably helped justify the cost of providing the captcha service for free worldwide (old free tier was 1M/mo)
I implemented something similar for my bot defences. If headless chrome is detected you still get the same anubis-style PoW but even if you submit the right answer you get rejected.
Imo the worst is recaptcha. At least with cloudflare the work you have to provide is minimal. With recaptcha it can take me much longer than 5 seconds, and lately I have trouble even completing their challenges correctly. Nowadays if I see a (recaptcha) captcha I drop the site unless I must visit it for some reason, it is not worth the time, the effort or the annoyance.
Most CF / Recaptcha problems are users going "off the golden path", and not realizing that their config changes are at fault.
If you're on a consumer router, using a mainstream stock browser with stock settings (maybe plus uBlock Origin), with your Google account logged in, it's very, very likely to just work. If you're part of the .01% of users with opinions about that sort of thing... you're not worth optimizing for.
A lot of users also run cheap cracked fire sticks and other low reputation hardware that's proxying their residential traffic for nefarious reasons which makes all the big providers put up their guard.
At least for me, CF is fine; recaptcha is the only one I really have problems with.
I dont care what recaptcha wants to optimize for. I dont think that using a vpn is that a rare thing anyway. If others have figured out how to do it without requiring spending 30 seconds to solve a captcha, I dont see why websites still use recaptcha/captchas for that.
And that it is "my fault" not being logged into google I was least expecting to see here.
A few months ago there was a story posted here about someone who completely eliminated crawlers on their website with Anubis.
I think it was getting upvoted before users were clicking the article because if you did, you had to leave the Anubis PoW page open for several minutes before you could get into the site. The Anubis difficulty scale is unintuitive and the difference between a small delay and becoming unusable is easy to cross.
That’s a terrible justification, and I would question if it’s even true. Is there anything to support that conclusion or is it just a guess you’ve made?
Because I don’t buy it. There is no time limit for discussions. One can take as much time as they like to enter text on their phone. Also if they’re motivated by the discussion they may take the time and make the effort to write thoughtful replies.
Finally, there’s no reason that users visiting on desktop computers won’t make equally shallow remarks. Shallow online discussions have existed long before smartphones existed.
I think you just described most of the reason I like it. Plain text sites tend to scale text, allow pinch to zoom, and are just generally faster.
Post deletion we've seen used for manipulation on other sites, it's a sticky topic. HN seems to want you to stand by your word, so it makes sense here.
I don't think PoW scales, because if the bot authors get serious they'll start using native implementations that are much more efficient than the web ones real users are running. In theory maybe Anubis could start using WebGPU to help close that gap, but then anyone without WebGPU support is out of luck.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
It's not about traffic identification at all, but rather a hashing algorithm that is deliberately resistant to parallelization and GPU/ASIC acceleration, which shrinks the gap in solving speed between the fastest systems (i.e. datacenter-class compute resources) and typical systems (e.g. the CPU in your smartphone or laptop).
Uh, is it resistant to parallelization across multiple sites? Because that's the situation for the scrapers. They're not trying to solve a single PoW challenge across many cores.
Typically, the machine doing the content processing, including solving PoW, is the centralized "control" node described in the article, not the machines who's IP addresses are being used. In typical residential proxy networks, the residential proxies are exposed to the customer (the person paying for and using the proxies) as just SOCKS5 addresses, and no computational power from those compromised devices is made available for the scraper besides that used to power the SOCKS5 server itself, the customer is just paying for the transport and address (and indeed, is often billed on either a per-GB or per-IP basis).
In effect, if the customer (the entity paying for and using the proxies) wants to solve PoW challenges through those connections, it is indeed the customer who must pay that compute cost, not the compromised devices.
Note that this is the case for a majority of, but not all, residential proxy networks, which often are built through quasi-voluntary distribution channels, including SDKs included in otherwise legitimate mobile applications distributed through Apple's App Store and Google Play.
These distribution channels tend to be categorically unavailable (or at least unreliable) for true RAT-style malware that enables remote operators to dynamically assign arbitrary computational workloads to client devices.
This isn't to say that true botnets built with actual malware delivered through either software exploits or phishing attacks don't also perform as residential proxy networks, but such categories are a relatively small subset of all residential proxy networks, and there are much higher ROI malicious activities to be performed on these devices rather than serving as relatively mundane traffic networks for scraping.
Ah, I see what you're getting at. Yes. You can think of any given computer aa having a fixed amount of compute budget for these types of acceleration-resistant hashing algorithms. Let's say the scraper can perform 10,000 hashing operations per second total on their machine, and needs an average of 1,000 hashing operations to solve the PoW. It's a minor detail, but note that these PoW challenges non-deterministically vary in the number of hashing operations needed to produce a valid hash, not dissimilar to bitcoin mining, where a hash with a certain number of 0s prefixed is sought, and the scraper essentially has to brute force through all possible inputs until an input that produces a valid hash is found.
In a well-designed PoW systems, there is a per-site prefix or suffix that is required to be prepended or appended to these random inputs, and it may change not only between websites, but even between PoW sessions on the same website, and should not be predictable - only being disclosed to the client at the time the PoW challenge is issued. In such a case, the scraper cannot simply precompute a bunch of valid hashes that work across multiple sites, nor a bunch of valid hashes that will always be good for even one site, the scraper operator will need to compute these hashes (with a limited budget to do so) upon initiating each PoW session.
So, in your example, each request needs 0.1 core-seconds of PoW. Is that right? Or, since you're saying 10000 hashes across the entire machine and 1000 needed, then on a typical 96-core server you need 9.6 core-seconds of PoW? The former means you pay about $0.0000005 per request at standard cloud rates; the latter means you pay about $0.00005 per request _and_ your site is totally unusable by legitimate clients. Both are _easily_ worth it for someone backed by VC billions and hungry for data. The network and storage fees are likely to be more significant than that already, not to mention actually training a model; they don't balk at downloading terabytes of crap already.
Note that none of this assumes any sort of acceleration from parallelization (be it through GPUs or reusing work across servers) or precomputation relative to what a normal client does. Compute is just really cheap in dollars compared to the cost of having a user wait, and these companies _also_ have a lot of appetite for spending dollars compared to that of a normal user. As others have pointed out, the only reason why Anubis works (sort-of; not for everyone) right now is that it is uncommon enough, essentially “proof that you bothered to have your crawler run JavaScript at all”. It's a confusion measure.
You raise some good points here, but PoW is meant to be one tool in the toolbox, not the only line of defense. You can still maintain blocklists of known scrapers (or better yet, have your PoW system be aware of them and silently adjust the difficulty to an impossible level, such that the scraper gets stuck trying to solve your PoW challenge until it hits a timeout configured by the scraper, if they were wise enough to configure one). It's also courteous to not only build and maintain your own blocklists, but to share them with e.g. vtotal and spamhaus, to help protect others.
Similarly, you have tarpits, which generate infinite mazes of garbage data, or even deliberately poisoning training data (should the scrapers be training LLMs) though this don't entirely eliminate the deleterious effects of scraping on the host's web server (more info: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-...).
If you were thinking or hoping that PoW would be a magic silver bullet that stops scrapers all by itself, then correct, it does not stop all scrapers. Scraping and anti-scraping is fundamentally a constantly evolving cat and mouse game that demands adaptability and punishes complacency from all participants trying not to lose.
If that happens the browser engines (all what, 3 of them?) can add a PoW API to call into native code. Or a pathologically scalar algorithm can be adopted so that wasm is good enough. RandomX or something close to it probably qualifies.
> The anubis author has stated they recognize it's an arms race, but PoW scales.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
Anubis's default 1-week token lifetime may not be enough to dissuade enough scraper networks to make a difference, particularly with the default weight->difficulty level hierarchy, but that's for individual site admins to determine.
We can all argue based on how we envision "ideal" scraper networks being run, but what matters is how they're actually run. For now, anubis is helping a lot of sites cope with misbehaving bot scrapers written by the script kiddies you mention, who don't care if the internet burns as long as they finish their scrape 1 hour faster. If anubis motivates them to devote a few brain cells to make their scrapers smarter, they may also fix the scrapers to not take down the sites they're scraping.
Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).
> That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.
Except that doing hundreds of requests from the same IP makes it pretty trivial to detect scraping, opening you up to being banned, or fun stuff like a slowloris or being fed poisoned data.
There are major websites that take more than 2-3 seconds to load every new page. Google, for example. Reddit. Facebook. Instagram. Doesn't seem to have hurt them.
PoW barely affects the "residential proxies" aka. malware botfarms. The IPs are free for them and siphoning additional system resources for PoW doesn't matter at all for them. PoW only affects the large centralised scraping by the AI providers, which are not operating behind "residential proxies".
Most users of residential proxies just get a SOCKS5 address and routing, they don't actually get computational resources of the infected systems beyond that. The user of the proxies, the operator of what the article describes as a control node, would be the device responsible for the PoW.
Do you have any evidence that AI providers aren't using residential proxies?
If you pop my machine and use it to route 100 MBit/s, I might not notice for months.
If I hear the fan spinning at night, you're probably getting caught immediately.
If you pop my mom's TV box and use it to route data within the connection's capabilities, you're getting away with it. If you consume a little bit of resources, still. If you consume enough to be useful for these kind of challenges, chances are her TV playback will start to stutter, which will be resolved by taking the compromised TV box, and removing the malware using advanced mechanical means called "a trash compactor".
>chances are her TV playback will start to stutter
video decoding is hardware accelerated, and there's probably enough excess compute to be able to do some sort of PoW challenge. Besides, unlike humans, bots aren't in a hurry, so they can spread out the work across a long time to minimize disruption.
PoW can theoretically scale effectively infinite because it can mine cryptocurrency. Millions of compromised IoT devices hitting your server? Now you have enough money for a faster server.
It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.
But why? Obviously an unjustified cryptominer is bad, like unnecessarily slow JavaScript, but this one has a good purpose and to the user is no different than PoW.
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
Can you not design a PoW that is most efficient in a browser? Don't brute force hashes like Hashcash/Bitcoin, do something similar to RandomX instead but in JS. Browsers ought to run the fastest JS interpreters already so if interpreting JS becomes the bulk of the work, that attack might not work. Maybe even involve the DOM or whatever else makes sense.
Proof of work does not scale. It trades something fungible and incredibly cheap (CPU) for something incredibly expensive (user-visible latency). There is no set of parameters where the cost is going to be a meaningful deterrent to any kind of abuse (even something as low-yield as scraping) without adding crippling amounts of latency to real users.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.
Sure, but end users as a group still have a significant amount of RAM.
Even on a low-specced machine you can afford to have the currently-active website tab consume a few hundreds of megabytes of RAM. It was mostly sitting idle anyways, so most people aren't even going to notice. Multiply that by a few thousand concurrent visitors and you're burning hundreds of gigabytes without anyone caring.
Some scraper being forced to install hundreds of gigabytes of extra RAM in their crawling node? They will notice having to go from a $0.50 / h instance to a $15.00 / h one, solely for the extra RAM requirement.
It doesn't make the economics any different. In a browser environment, you're maybe looking at the acceptable lease being 100MB for 1 second. Much more than that, and you start hitting limits of what browsers will let you do on low-end phones. Longer than that, and we're back to the user-observable latencies being too long.
100MB for 1 second just is not much of a deterrent.
The success of the ongoing Anubis rollout proves the opposite. People are used to slowly-loading websites - the rise of garbage SPAs has seen to that. Staring at a spinner for a second every once in a while is not an issue for genuine users.
On the other hand, the additional CPU usage rises the compute cost of scraping by several orders of magnitude. If you don't have to scrape this specific website, you'd be stupid not to move on and hit someone else.
Ideally the general cost of scraping would be high enough that it isn't affordable any more - especially anonymized - but considering the amount of money brainlessly being pumped into AI I doubt that'll happen any time soon.
You could of course also make the argument that the user's time is worth something as well and should be included as part of the cost, but that ship sailed a loooong time ago. If you care about that, you should be calling for the death of client-side Javascript and any form of advertising.
Scrapers generally aren't looking for random websites to scrape. They have a specific URL in mind. Only if the goal was DDoS would they not care which URL was accessed.
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
You can't do that any more. Too many ISPs, especially mobile carriers, don't hand out anything resembling a fixed IP address any more. It's CGNAT and constantly changing IP addresses alllll the time now.
On a small enough site (even LWN might qualify) the chance of two random sets of client IPs intersecting can be quite low.
Private trackers do this. If they ban a user that geolocates to a certain city and ISP, they'll ban new signups from that city and ISP because there's probably only a few users from the same city and ISP. And then report to their friends at other trackers, that a user with that city and ISP is trying to evade a ban.
Well, we don't use a captcha either. If it were a choice between a captcha and a proof of work system, we'd have to reevaluate things. Luckily, for now, we're able to get away with a much lighter touch.
Anubis appears to be a largely ineffectual stopgap that has been cargo culted into prominence, for reasons I don't fully understand.
The cost of solving the default Anubis PoW is negligible on cloud servers, and it's even lower if you use native code rather than JavaScript to solve it, which Travis Ormandy helpfully demonstrated last year (https://lock.cmpxchg8b.com/anubis.html). If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.
So Anubis doesn't do much to stop bots, but it makes otherwise lightweight websites (little JavaScript or interactivity) almost unusable on low-resource systems like my old phone or an old Atom-based nettop.
> when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape
This "IP-bound proof-of-work" thing is gonna kill multipath TCP and bring down IPv6 with it. Uffff.
> If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.
Then anubis adopts it itself, increases the amount of work that needs to be done and the bar stays the same again for everyone? Seems like mostly a non-issue unless there is an arms race towards ever more optimized solvers which I don't believe is possible.
> > If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.
> Then anubis adopts it itself, increases the amount of work that needs to be done and the bar stays the same again for everyone?
No, the bar doesn't "stay the same" for everyone interacting with Anubis.
My otherwise-perfectly-usable 8-year-old phone, which can't be patched to run a native solver, becomes even more unusable on sites gates with proof-of-work challenges like Anubis.
This is the whole problem with PoW. It forces thousands or millions or billions of client devices to do increasing amounts of useless work which is relatively easy for cloud-based attackers to adapt to, but very difficult for hardware-constrained and software-ossified mobile clients to adapt to.
In other words, it asymmetrically punishes the clients that it's not intending to punish.
I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are redundant and computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
What’s more, they will scale up with increased resources on the site.
If you redline at 20 searches a sec, and put in 4 more workers, suddenly you’re serving 100r/sec to the bots, paying 5x for it, and your users are still seeing shit qos. I've seen multiple cores of nginx saturated just dealing with one dos/crawl run on a somewhat high profile site.
I don't think people sit around going "Grrrr who can I ban next?". Instead this stuff gets noticed because you see the webserver at 99% CPU utilization for 2 days straight, check the logs, and see you are somehow getting crawled by half the IPs in New York City.
>We have not gone with tools like Anubis, partly because it causes annoying delays for those trying to get to the site, but also partly because it seems inevitable that the scrapers will eventually find their way around it. Indeed, there are some indications that is already happening. A proof-of-work requirement is not a huge obstacle when you have millions of other people's machines to do the work on.
The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
> The moment it does you'll be able to tell because Bitcoin will crash to 0.
The "workaround" for PoW is running the PoW computation on hardware that's better suited for the task. Bitcoin mining has been using ASIC for many years now.
Let's say a legitimate user is willing to wait for one minute on a budget phone. Then your PoW is limited to what that phone can compute in one minute. But on the attacker's specialized hardware this computation only costs fractions of a penny, so they are barely hindered by it.
The SHA256 based PoW scheme has a very heavy ASIC advantage. People have tried to design PoW scheme that minimize the custom hardware advantage, but I'm not sure if they managed to close the gap far enough to make PoW feasible for this application.
True, reminds me of the failure of litecoin to make an algorithm that was asic resistant and benefitted normal user hardware, but it didn't last long.
PoW tends towards ASIC capitalization. Theoretically users would ideally pay some cents for a cloud asic to browse in this scheme. Not the ideal early web, but maybe better than dead internet.
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
> Tracking which devices still send lots of data when no one else is using the network?
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
Opnsense has a traffic capture feature in the interface diagnostics menu, if you want to spot check what servers the devices are currently talking to.
Should be pretty obvious: client devices and internal services will have no traffic 98% of the time, just NTP for timekeeping, DHCP lease renewal, and associated ARP (running total: two dozen packets if you monitor them for a full 24h), then any system updaters (readily identifiable by the initial DNS requests), and finally of course you'll see the traffic of the service that the device hosts, if any, which can be easily dismissed by not looking at incoming connections (scraping uses outgoing connections)
> Apps are not infected with NetNut. This is just Google abusing their monopoly position to hurt its competitors.
If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network.
It doesn't seem like monopoly abuse to exclude such malware from application stores, just like key loggers or apps intercepting other apps network traffic without the user being aware of it (say the banking app's network traffic and password entry).
>sell access to the user's internal residential network
That is not what the SDK was doing. The actual code in the SDK does this (simplified to take less space):
if (addr.isSiteLocalAddress() || addr.isLoopbackAddress()) {
LogUtils.e("PopaTunnelAsyncThread", "Hacking? The Host Resolved Ip is " + addr + " on tunnel id:" + tunnelId);
throw new IllegalArgumentException("Hacking? The tunnel host resolved ip is internal");
}
Local and loopback addresses like 10.0.0.0, 172.16.0.0, 192.168.0.0, fc00::/7, 127.0.0.0, ::1 do not work. It will not connect to people's private file servers on their network.
It'll still connect to IPv6 addresses and bypass any firewalls.
Also users might become part (victim?) of a police investigation because of illegal actions that seem to originate from their local residential connection.
So still good to take down such backdoors. Would be nice to go after the botnet operators as well...
Instead require app developers to explicitly call out this monetization method. This is neither designed to be a backdoor, nor a botnet. These are harmful classifications which can encourage people to try and target them like malware, when they are essential tools for privacy, bypassing geoblocking, and being able to collect public information from sites. The average person is not going to be involved with a police investigation. That has a tiny probability of happen and trying to blow it up to such proportions is dangerous to the existing of this important tool.
355 comments
[ 5.4 ms ] story [ 122 ms ] threadThe question is more about why the US and others can't properly enforce the bullshit all this amounts to.
I'm not sure what the solution would look like - maybe Cloudflare's payment required for requests beyond a certain limit? But I think that the world needs user freedoms now more than ever.
It would cost too much money, either for police to raid all the physical shops and ebay sellers selling dodgy IPTV boxes, or for ISPs to hire enough competent support staff to monitor and respond to abuse@ email addresses and follow through.
The real problem is the companies offering money to developers if they include unrelated SDKs in their calculator or flashlight (for example) applications. Those SDKs add functionality to incorporate those devices into a network that can be used for scraping. The traffic is little, but is distributed over millions of residential devices all over the world, making it difficult to categorize or block. That should be illegal, and that's what Google et al can be expected to be policing on their app stores.
* no, small print in a click-through agreement doesn't count.
its very easy to detect and bypass poison type of tools largely because of the fact that there are far more outlets for truthful info so unless you can get everyone to buy in (with real legal liabilities) its not effective
also its possible to poison the poisoners with a certain pill that would have very real consequences for those maintaining whatever github repo/communities
Really makes you think, what we're feeding them...
We ban such accounts regardless of what the single purpose happens to be. Pre-existing agendas are not what HN is for and destroy the curious conversation that it is supposed to be for.
https://news.ycombinator.com/newsguidelines.html
I wasn't aware of this project. Thanks for the heads up.
In such cases, someone usually accuses us of making the moderation call because we secretly agree with or oppose one side of the topic. I'm not sure if your comment is meant that way or not. But I had never heard of this particular project and have no opinion about it one way or the other.
10 comments (excluding subsequent in-thread replies) over four months, always in contexts in which either the topic of LLM scraping or Poison Fountain itself has already been mentioned.
This strikes me as contextually informational, and is no different from other project representatives appearing in threads discussing their own subjects or posts. Such as, say, Jon Corbet (@corbet), of LWN, whose activity on HN shows a similar pattern and roughly equivalent frequency.
I hope it goes without saying I'm not suggesting corbet's handle be banned, anything but.
atomic128's comments are predictable, but apposite, informative, non-disruptive, and address an increasingly urgent issue. Whether or not it's an effective mitigation is of course another discussion, but it seems plausible at first blush.
As dang should well know but others may not, I often contact mods directly for HN issues, including numerous "one-note flute" alerts. atomic128's account should be un-banned, though perhaps they might communicate with HN's mods over what would be a more acceptable mode of interaction.
I finally made it back
[editing - bear with me...]
I took a look at the most recent comments from both accounts and they don't look similar to me in this respect.
I think there are two questions here though:
1. Was the violation egregious?
2. Did it deserve an immediate ban, or did they deserve a warning etc.?
Seems to me the answer to (1) is yes, but the answer to (2) I'm less sure about.
Our other friend here is a more recent participant to HN, at least under this handle. (I don't know that there are others, only what I can see from this one.)
Im not against the ban perse (single purpose accounts are bad), just curious if they had a chance to change their contribution style.
I worry a lot of the anti scraping rhetoric will just injure the open web and put somebody like cloudflare in charge.
You could perhaps even get website operators to "push" new data to a common crawl database. The scrapers would learn there is no value on scraping X domain because the data is available elsewhere more easily.
Edit: the article says millions of times per hour? (!?)
The article is also astonished by this, and speculates it might be some kind of underground AI labs but... millions of them? Or does it only take one with too much money and a badly configured scraping setup?
Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.
However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.
Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.
Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s.
Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk.
It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.
Who’s doing it, are they even using the data?
Millions per hour is tens per second though; perhaps the fix is performance improvements
That'll be great until.. they rewrite the scrapers in Rust! Then we're really hosed!
Which powerful entities have historically hated a free and open internet?
...all of them??
I can imagine that sites with dynamic content and potentially unbounded query types or pathnames are in danger from particularly stupid crawlers.
Grok actually shows a number of sources used for an answer. Once I asked it something simple and it apparently scanned 200 different websites. And it was just a short prompt. Now imagine millions of users asking for something multiple times a day.
That said, the approach is flawed. It looks like the people doing the scraping want everything. There are some people who do not want their data to be captured by LLMs. A common crawl would make it easier to those people to opt out, limit what is captured, or to poison the data. (I'm assuming the only way to avoid fragmentation is for the crawl to be done in the open and by consent.) Then there is the question of who would pay for the crawling and hosting. You could try charging for access to the dataset, but that would only encourage others to develop and sell their own dataset (especially since there are likely many who would want their interest in such a dataset to be confidential).
I'm guessing the training companies are taking real/synthesized user queries and trying to distill what they can from site searches.
Probably as simple as the fact that there are unmetered residential proxy plans, which means once you're already paying for one, there's no reason not to use it for everything.
On my sites, I see ClaudeBot consistently (and has been like this for over a year) ask for "${SITE}.com/base_dir1" and then get the redirect (Caddy does this automatically) to get "${SITE}.com/base1/base_dir1/" (trailing slash).
The hrefs on my sites include the trailing slashes for directories, so looks like ClaudeBot's internal code is stripping them off before requesting them, and therefore essentially makes almost 2x the requests to my sites for directories, half of them ending up being redirects back to the same url but with the trailing slash.
I think that nobody would care if I use wget or curl for few pages, e.g. because I would like to read a site as offline or archive it.
Btw average age of any page is 10 years. Deletion or structural change after acquisition is common, Signal vs Noise site recent wipe out could serve as an example why we need to archive sites.
If only you were the only one doing it...
Highly unethical but the way the internet is going they're the last anti-hero of a somewhat open internet
unethical yes but really raises the question as to what we see is real or not
No they really don't, dishonest founders do that.
You're one with the lower case shibboleth so I have no doubt you surround yourself with dishonest founders, but faking users is pretty damn low on the usecases for residential proxies.
I said they're unethical because they tend to be hidden in innocuous seeming apps or sprung on unwitting individuals via clickwraps on their smart devices.
but ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks
then inventing a fantasy about who i associate with instead of answering that conflict is an unusually loud form of projection
"ive never seen one raise $4.5m for an ai agent startup built around pulling fresh web data, then openly cheer the unethical proxy infrastructure used to evade consent and blocks"
You've never seen this, making this non-sequitur? Or this "directionally correct by my priors" speak for something else?
-
Either way it has nothing to do with why the proxy is unethical (I already covered that) and everything to do with why their startup is unethical
(or not? if they're not being assholes about the scraping more power to them, I've used proxies + agents to let my users get their own data off other sites that felt it was their right to block people from accessing their own data easily)
??? shift is an extra key to press
most shibboleths are subtle like that
They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
The article was called "The one we're commenting on"
Does that sound like your typical self-hosted blog?
When compared to
> They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
yes, absolutely.
> Many providers build their proxy pools by partnering with device owners who agree to share their bandwidth, while others use embedded SDKs in free apps or VPNs.
WTF. That's just botnets.
Source: https://www.fbi.gov/investigate/cyber/alerts/2026/evading-re...
If they have consent, they're not really botnets. Botnets involve infecting devices without the owners knowing.
It wouldn't be much different from e.g. open WiFis at restaurants and hotels and most VPN services.
There must be countless individuals all over the world who suddenly can't log into their Gmail or create any new accounts because a fraudster sent spam from their IP. I wonder: has anyone has tried to quantify that problem?
Places with open WiFi like hotels and restaurants would be having the same problem. People on CGNATs would be having the same problem. An IP doesn't correspond with a single user.
Almost nobody is on a static IP. ISPs have an interest to keep their customers connected and not affected by other customers, so they'd probably use as large address pools as possible for their dynamic IP rotations.
Tech companies like Google are interested in being global monopolies to dominate markets and also having as good and large-sampled statistical data as possible from everyone. Blocking thousands of users for each bad user that can likely easily switch to another block if they really want to doesn't seem effective. It would also affect their reputation. Who wants to rely on an email service that may block you because of other users on your IP block? Reliability is very important for businesses.
With small personal sites/blogs like the ones you mentioned, they can just block most of the planet without a problem. There's not as much reason to be as reachable as possible, and they're also more budget conscious.
It doesn't matter if page 42 of the Terms and Condition "clearly states it." This does not constitute informed consent.
On "always covert and never consensual", it seems some IPs come from partnerships with ISPs. Beyond some ISPs being possibly regulated by their government on the matter of only offering their IPs to residents to control local prices, I don't see why ISPs shouldn't be able to offer some of their IPs for proxying. It would be a valid business model.
Well, one party gives away free walls if you agree to fill your castle with surveillance cameras you don't control.
I admit this is a naive question. I have no idea how applicable bt is to web requests. This problem just seems to have a similar “too many people want this resource” shape.
1. When the page itself was last updated
2. When the crawled copy was last updated
In order to get an accurate date for 1, you have to crawl it, and if you are crawling it you might as well use that copy you just crawled.
The missing here is that for pretraining AI models should accept a cut off date and not worry about being perfectly up to date. Keeping things up to date is more useful developing internet search engines for grounding.
We agree that it would be great if it was even more widely used.
Backbone operators should not be allowed to knowingly maintain connections to networks that allow connections from China or Russia.
Disrupting the largest residential proxy network - https://news.ycombinator.com/item?id=46802748 - Jan 2026 (221 comments)
I don't get it. Don't we keep blacklists of this stuff? And if they hammer thousands of requests per site per second and never reuse an IP, they'd run out of addresses in a few weeks.
Then they'd switch to IPv6, and... well, are we using IPv6 for anything important?
Like we need it for IoT, but do you want random IoT devices talking to your web server? (IPv4 handled mobile phones just fine not that long ago, right?)
Probably not, but since IoT manufacturers did zero to lock down their devices, those devices are doing a lot more than their owners think they are doing
Anyone who has bigger than /48 is no longer an individual user. They could be an ISP set up solely for scraping, though that comes with some fees and requirements.
This is a good thing, thanks to this we have powerful open source LLMs.
> This activity overwhelms sites with traffic.
When LLMs get good enough, we won't need those sites anymore :)
[not satire, this is what I think, without self-censorship]
It's massively less annoying than a captcha, which is both a longer delay (typically, at present) and a massive cognitive distraction/roadblock.
The anubis author has stated they recognize it's an arms race, but PoW scales. Captchas and other signals are already at the end of the road; any additional difficulty increases false bot-positives, which are already unacceptably high.
For websites running dynamic languages, a binary (anubis is in go) sentry that operates before[1] the website is forced to expend any resources, is usually a significant improvement.
[1] this is true regardless of whether anubis is in reverse proxy mode or auth mode.
I've only just started using it, and the existing config DSL doesn't let you do this, but it's just a patch away (in principle): instead of using system load as a signal to increase or decrease weights, use a combination of finalized weight and system load (or a load-equivalent or net-traffic-equivalent signal from some openmetrics agent) to select between difficulty buckets.
So, as an example, difficulty could have an arbitrary scale independent of load, and regardless of that scale, it could increase by 1 at 50% cpu, 2 at 80% cpu, etc. I'm also not sure currently if a challenge will re-appear if anubis determines that a client should get a difficulty N challenge, but they only have a token for an N-1 or N-2 challenge. Easy enough to implement in principle, I just haven't looked at the code at all. I shall test it to see if it already does that.
https://www.kaspersky.com/blog/what-is-wrong-with-free-vpn-s...
Yes, a VPN involved. That doesn't make it okay and notice that anubis by default works without issue (though possibly with a more difficult challenge) in the exact same scenario.
I'm quite certain it isn't a generic "datacenter" list though because a given VPN exit that was working will suddenly stop. Meanwhile I have a valid cookie yet that is disregarded.
And contrary to grandparent, PoW only worked because it was a novel thing to work around, a simple "type human" prompt would've worked as well.
When anubis gets widespread enough users will still run the PoW in javascript or whatever while the scrapers will run much more optimized native code, so no, it doesn't scale.
Reload loops, or being able to "bypass" anubis (unless you merely mean bypassing it for the token validity period by solving a challenge), sound like misconfigurations. There's no reason for anubis itself to cause reload loops; it's tricky to configure a webserver to use it in some scenarios. Similarly, any ability to bypass anubis probably means the site is using it in auth/challenge mode only, and then misconfigured their webserver's auth checking.
reason why is 1. Google and others really needed the training data, and 2. it probably helped justify the cost of providing the captcha service for free worldwide (old free tier was 1M/mo)
funny with all the IP information they have, cloudflare cannot do a better job. (I am on IPv6)
and most of the time, its on marketing product pages like in framework main site, which can be cached.
If you're on a consumer router, using a mainstream stock browser with stock settings (maybe plus uBlock Origin), with your Google account logged in, it's very, very likely to just work. If you're part of the .01% of users with opinions about that sort of thing... you're not worth optimizing for.
I dont care what recaptcha wants to optimize for. I dont think that using a vpn is that a rare thing anyway. If others have figured out how to do it without requiring spending 30 seconds to solve a captcha, I dont see why websites still use recaptcha/captchas for that.
And that it is "my fault" not being logged into google I was least expecting to see here.
Parent was just starting a fact of how our digital overlords determine the probability of your browser being a bot. Why take it personal?
A few months ago there was a story posted here about someone who completely eliminated crawlers on their website with Anubis.
I think it was getting upvoted before users were clicking the article because if you did, you had to leave the Anubis PoW page open for several minutes before you could get into the site. The Anubis difficulty scale is unintuitive and the difference between a small delay and becoming unusable is easy to cross.
Unfortunately whatever HN is using routinely blocks my login with "Sorry."
some websites just always give me 403.
I believe that's the HN application itself, not a WAF in front of it.
Poor accessibility, bad mobile support, no options to delete content beyond a narrow window.
Because I don’t buy it. There is no time limit for discussions. One can take as much time as they like to enter text on their phone. Also if they’re motivated by the discussion they may take the time and make the effort to write thoughtful replies.
Finally, there’s no reason that users visiting on desktop computers won’t make equally shallow remarks. Shallow online discussions have existed long before smartphones existed.
Sent from my iPhone.
Post deletion we've seen used for manipulation on other sites, it's a sticky topic. HN seems to want you to stand by your word, so it makes sense here.
Then again, a large portion of the problem seems to be bots making way too many requests and in general not being optimized in the first place, and this does help filter those out.
In effect, if the customer (the entity paying for and using the proxies) wants to solve PoW challenges through those connections, it is indeed the customer who must pay that compute cost, not the compromised devices.
Note that this is the case for a majority of, but not all, residential proxy networks, which often are built through quasi-voluntary distribution channels, including SDKs included in otherwise legitimate mobile applications distributed through Apple's App Store and Google Play.
These distribution channels tend to be categorically unavailable (or at least unreliable) for true RAT-style malware that enables remote operators to dynamically assign arbitrary computational workloads to client devices.
This isn't to say that true botnets built with actual malware delivered through either software exploits or phishing attacks don't also perform as residential proxy networks, but such categories are a relatively small subset of all residential proxy networks, and there are much higher ROI malicious activities to be performed on these devices rather than serving as relatively mundane traffic networks for scraping.
In a well-designed PoW systems, there is a per-site prefix or suffix that is required to be prepended or appended to these random inputs, and it may change not only between websites, but even between PoW sessions on the same website, and should not be predictable - only being disclosed to the client at the time the PoW challenge is issued. In such a case, the scraper cannot simply precompute a bunch of valid hashes that work across multiple sites, nor a bunch of valid hashes that will always be good for even one site, the scraper operator will need to compute these hashes (with a limited budget to do so) upon initiating each PoW session.
Note that none of this assumes any sort of acceleration from parallelization (be it through GPUs or reusing work across servers) or precomputation relative to what a normal client does. Compute is just really cheap in dollars compared to the cost of having a user wait, and these companies _also_ have a lot of appetite for spending dollars compared to that of a normal user. As others have pointed out, the only reason why Anubis works (sort-of; not for everyone) right now is that it is uncommon enough, essentially “proof that you bothered to have your crawler run JavaScript at all”. It's a confusion measure.
Proof of work does not work.
Similarly, you have tarpits, which generate infinite mazes of garbage data, or even deliberately poisoning training data (should the scrapers be training LLMs) though this don't entirely eliminate the deleterious effects of scraping on the host's web server (more info: https://arstechnica.com/tech-policy/2025/01/ai-haters-build-...).
If you were thinking or hoping that PoW would be a magic silver bullet that stops scrapers all by itself, then correct, it does not stop all scrapers. Scraping and anti-scraping is fundamentally a constantly evolving cat and mouse game that demands adaptability and punishes complacency from all participants trying not to lose.
Your sibling, roommate, neighbor that uses your internet, previous IP owner, posts too much? You get blocked too.
Using VPN? Blocked.
Your iPhone is too old, blocked.
Your screen brightness too low? Believe or not, blocked.
... What?!
I've ended up putting only IPv6 on the domain. It's running this way for 2 years already.
The scraper wars are largely between script kiddies and people with both deep intimate networking and DOM knowledge. Yes greyhairs, I’m looking at you.
The problem is, you can’t PoW every page load and resource request because the user experience will suck and people will run away. And that window - the gap between what people will tolerate vs draconian enforcement - is exactly what the scrapers exploit.
And looking at the PoW options out there - I’ve seen at least one PoW WAF (honestly can’t remember if azure or amazon) have their PoW boil down to repeated trigonometric functions, ie very optimisable.
It’s a neat concept, but the answer and future to my eyes look bleak.
We can all argue based on how we envision "ideal" scraper networks being run, but what matters is how they're actually run. For now, anubis is helping a lot of sites cope with misbehaving bot scrapers written by the script kiddies you mention, who don't care if the internet burns as long as they finish their scrape 1 hour faster. If anubis motivates them to devote a few brain cells to make their scrapers smarter, they may also fix the scrapers to not take down the sites they're scraping.
Your typical end user doesn't switch IPs that often, so it's fine to Anubis them again when they do. A scraper, on the other hand, has a tradeoff to make between rotating ips often (requiring a challenge on every request) or keeping only a few IPs (making cross-request identification much more valuable and reliable).
They meant you can’t PoW every page transition.
If clicking every link on your website throws you back to another Anubis page for 2-3 seconds, users will bounce.
That’s why Anubis does an up front challenge and then you’re good for a while. It’s a really low cost for the scrapers.
Except that doing hundreds of requests from the same IP makes it pretty trivial to detect scraping, opening you up to being banned, or fun stuff like a slowloris or being fed poisoned data.
Do you have any evidence that AI providers aren't using residential proxies?
If I hear the fan spinning at night, you're probably getting caught immediately.
If you pop my mom's TV box and use it to route data within the connection's capabilities, you're getting away with it. If you consume a little bit of resources, still. If you consume enough to be useful for these kind of challenges, chances are her TV playback will start to stutter, which will be resolved by taking the compromised TV box, and removing the malware using advanced mechanical means called "a trash compactor".
video decoding is hardware accelerated, and there's probably enough excess compute to be able to do some sort of PoW challenge. Besides, unlike humans, bots aren't in a hurry, so they can spread out the work across a long time to minimize disruption.
It doesn’t matter that the challenge must be verified: present multiple challenges, some are verified while others mine crypto.
But why? Obviously an unjustified cryptominer is bad, like unnecessarily slow JavaScript, but this one has a good purpose and to the user is no different than PoW.
Not if the honest party is doing it in a browser: The same computer can so any POW so much faster in C than any amount jf JS and WASM that it will never ever ever be a contest.
> becoming much more obvious and easy to block, or they have to use massive amounts of compute.
If you believe this, please contact me: I think compute is free[1] and can probably help you out.
[1]: https://news.ycombinator.com/item?id=30175269
> The dilemma for bots: when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape, becoming much more obvious and easy to block, or they have to use massive amounts of compute.
There is no dilemma. They get a token, they maybe do some automated multi-armed bandit per-site to figure out how to maximize the extraction rate they get from a single token, and then they use an IP for that many requests / that amount of time before ditching it.
it could be RAM-bound, which is very much NOT cheap nowadays :)
Even on a low-specced machine you can afford to have the currently-active website tab consume a few hundreds of megabytes of RAM. It was mostly sitting idle anyways, so most people aren't even going to notice. Multiply that by a few thousand concurrent visitors and you're burning hundreds of gigabytes without anyone caring.
Some scraper being forced to install hundreds of gigabytes of extra RAM in their crawling node? They will notice having to go from a $0.50 / h instance to a $15.00 / h one, solely for the extra RAM requirement.
100MB for 1 second just is not much of a deterrent.
On the other hand, the additional CPU usage rises the compute cost of scraping by several orders of magnitude. If you don't have to scrape this specific website, you'd be stupid not to move on and hit someone else.
Ideally the general cost of scraping would be high enough that it isn't affordable any more - especially anonymized - but considering the amount of money brainlessly being pumped into AI I doubt that'll happen any time soon.
You could of course also make the argument that the user's time is worth something as well and should be included as part of the cost, but that ship sailed a loooong time ago. If you care about that, you should be calling for the death of client-side Javascript and any form of advertising.
You can't do that any more. Too many ISPs, especially mobile carriers, don't hand out anything resembling a fixed IP address any more. It's CGNAT and constantly changing IP addresses alllll the time now.
Private trackers do this. If they ban a user that geolocates to a certain city and ISP, they'll ban new signups from that city and ISP because there's probably only a few users from the same city and ISP. And then report to their friends at other trackers, that a user with that city and ISP is trying to evade a ban.
The cost of solving the default Anubis PoW is negligible on cloud servers, and it's even lower if you use native code rather than JavaScript to solve it, which Travis Ormandy helpfully demonstrated last year (https://lock.cmpxchg8b.com/anubis.html). If Anubis were to be even more widely adopted, botnet operators would surely adopt and optimize native code solvers en masse.
So Anubis doesn't do much to stop bots, but it makes otherwise lightweight websites (little JavaScript or interactivity) almost unusable on low-resource systems like my old phone or an old Atom-based nettop.
> when tokens are bound to the connecting ip, scrapers must limit the connecting IP pool for each site they want to scrape
This "IP-bound proof-of-work" thing is gonna kill multipath TCP and bring down IPv6 with it. Uffff.
Then anubis adopts it itself, increases the amount of work that needs to be done and the bar stays the same again for everyone? Seems like mostly a non-issue unless there is an arms race towards ever more optimized solvers which I don't believe is possible.
> Then anubis adopts it itself, increases the amount of work that needs to be done and the bar stays the same again for everyone?
No, the bar doesn't "stay the same" for everyone interacting with Anubis.
My otherwise-perfectly-usable 8-year-old phone, which can't be patched to run a native solver, becomes even more unusable on sites gates with proof-of-work challenges like Anubis.
This is the whole problem with PoW. It forces thousands or millions or billions of client devices to do increasing amounts of useless work which is relatively easy for cloud-based attackers to adapt to, but very difficult for hardware-constrained and software-ossified mobile clients to adapt to.
In other words, it asymmetrically punishes the clients that it's not intending to punish.
Sometimes it feels like what people want is to only serve websites and content to good normal users but not evil bad “scrapers” (because maybe maybe your content will be monetized in some nebulous way) but … you put your content up publicly on the web! That should be part of reasonable use!
Not necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are redundant and computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
If you redline at 20 searches a sec, and put in 4 more workers, suddenly you’re serving 100r/sec to the bots, paying 5x for it, and your users are still seeing shit qos. I've seen multiple cores of nginx saturated just dealing with one dos/crawl run on a somewhat high profile site.
The first argument that it introduces delays to users is solid, but I would advise reconsidering on the second one that a PoW workaround will be found. The moment it does you'll be able to tell because Bitcoin will crash to 0.
Will bots use infected computers to do compute to work around it? Maybe, but it requires a CPU in addition to a network reputation, 2 mechanisms are stronger than one.
https://salad.com/earn
The "workaround" for PoW is running the PoW computation on hardware that's better suited for the task. Bitcoin mining has been using ASIC for many years now.
Let's say a legitimate user is willing to wait for one minute on a budget phone. Then your PoW is limited to what that phone can compute in one minute. But on the attacker's specialized hardware this computation only costs fractions of a penny, so they are barely hindered by it.
The SHA256 based PoW scheme has a very heavy ASIC advantage. People have tried to design PoW scheme that minimize the custom hardware advantage, but I'm not sure if they managed to close the gap far enough to make PoW feasible for this application.
PoW tends towards ASIC capitalization. Theoretically users would ideally pay some cents for a cloud asic to browse in this scheme. Not the ideal early web, but maybe better than dead internet.
> More recently, media-streaming devices have been identified as a major carrier of malicious scraping software. Sometimes the devices are compromised at the source; other times, they are just poorly secured and easily compromised after the fact.
I run an OPNsense firewall at home and the OpenWRT router at a hackerspace. Are there ways of auditing that devices aren't compromised? Tracking which devices still send lots of data when no one else is using the network?
That's what I personally do at least: I have nlbwmon [0] installed on my OpenWRT router to track data usage per device, then I scrape it every minute with Prometheus and plot it in Grafana [1]. This helps me see if any IoT devices are compromised, but it probably won't help much if people are using sketchy free VPNs on their phones. I also adblocking enabled on my router [2], which helps block a few malicious domains (but certainly isn't a panacea).
[0]: https://github.com/jow-/nlbwmon
[1]: https://www.maxchernoff.ca/files/grafana-network-bandwidth.p...
[2]: https://docs.mossdef.org/adblock-fast/
Should be pretty obvious: client devices and internal services will have no traffic 98% of the time, just NTP for timekeeping, DHCP lease renewal, and associated ARP (running total: two dozen packets if you monitor them for a full 24h), then any system updaters (readily identifiable by the initial DNS requests), and finally of course you'll see the traffic of the service that the device hosts, if any, which can be easily dismissed by not looking at incoming connections (scraping uses outgoing connections)
This is such a malicious interpretation. Do you think VPN operating are also trying to attack websites? Both offer the same kind of product.
>paid for hijacking their users' network connections
Nothing is being hijacked. Again the author is using wording to try and paint these people as malicious actors.
If apps ship with stealth backdoors to sell access to the user's internal residential network, that's malware. I doubt any users want app providers to sell access to their private file server and anything else on their local network.
It doesn't seem like monopoly abuse to exclude such malware from application stores, just like key loggers or apps intercepting other apps network traffic without the user being aware of it (say the banking app's network traffic and password entry).
That is not what the SDK was doing. The actual code in the SDK does this (simplified to take less space):
Local and loopback addresses like 10.0.0.0, 172.16.0.0, 192.168.0.0, fc00::/7, 127.0.0.0, ::1 do not work. It will not connect to people's private file servers on their network.Also users might become part (victim?) of a police investigation because of illegal actions that seem to originate from their local residential connection.
So still good to take down such backdoors. Would be nice to go after the botnet operators as well...