I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do on a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying ORC puzzles).
Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.
It's sad, but this train has left the station a quarter of a century ago imo. Many people have since become billionaires scraping the web without anyone agreeing (Google, Yahoo, and now possibly Anthropic and OpenAI).
There is a huge difference between a clearly identifiable, robots.txt-respecting, largely symbiotic search engine spider versus this new AI-driven casual sociopathy of firing shotguns of deliberately masked bots at sites in ways which benefit no one except for, maybe, the bot-owner.
I agree with you, though, that the future has almost certainly already been decided, and see the eventual outcome of all this selfishness to be mandatory device attestation to access most services on the internet—which will of course never be allowed on any open platform. AI engineers who believed in freedom to compute (or just individual freedom more generally) have committed perhaps the greatest self-own in the history of our species to date.
There is no detection method that will prevent AI from accessing systems without also blocking humans. The only thing we can do at this point is throttling.
Agreed on inevitable collateral blockage of real people using real "headed" browser. I'm getting a ton of that already, personally.
Throttling is poor help though. Mass scrapers are using "residential proxy" loophole + rotating UA and other attributes. You can't throttle somebody without identifying them. Unless you're talking about a global rate-limit.
I'm becoming more and more convinced that a big source of outrage on the internet is caused by people assuming that all other people are a homogenous blob.
It's not that "we" are going from one thing to another. It's that these are two different people, with different ethical boundaries.
You're framing this as if people are deliberately making decisions that they believe are unethical. The reality is that people have different ethical frameworks. For example, I believe that there is no ethical distinction between whether a web request originates from a browser or from an LLM on my behalf.
Another person replied already and I agree with them, but also I think you need to see this from the perspective of the people operating the service you are accessing.
As an example, imagine a really small community maintaining a small site/wiki/cms/forum that has the ultimate goal of promoting human relationships around a common interest (let's say retro computing as an example, but it could be anything). There are many many many such communities on the internet.
It's very improbable you will specifically instruct your LLM to access their site, it's way more probable it will happen without you even knowing, as a result of you doing some /deep-research or something. And not only that, but your LLM will probably spawn a ton of agents to gather as much information as possible in as little time as possible. A torrent of requests will go at this community's site, effectively killing it. They are a small community, they use their spare time and money to maintain something to serve them, they don't have the resources to serve your LLM and until you showed up they probably never even had to think about Cloudflare. They are certainly not against you getting the content, but they don't want you causing them issues either and you just did.
End result? Your LLM (effectively you) DoSed a small community's site. You caused harm. Could you have caused similar harm if you were doing it on your own? Sure. But you would have done it on purpose, not accidentally while instructing your LLM to do something else.
This isn't a made up story, it has happened already more than 1 times.
So the question is, now that you know your LLM can cause harm without you even knowing it, how does this change your stance?
> The reality is that people have different ethical frameworks.
Sure, Nazi, Hitler or Stalin all considered themselves to be the good guys. Kushner, Trump, etc are also acting within their ethical system (if I am getting money or power or fame it is ok to do it). Just about the only exception is Thiel who openly frames himself as evil, but is proud of it.
That does not mean we cant criticize their crappy actions or "ethical frameworks".
And yes, all the above are making deliberate decisions to be unethical assholes.
The internet mixes those people, and an Agent basically pushes the signal to noise ratio that businesses have relied on the intenet to basically zero.
If the internet just allowed indescriminate traffic, neither #1 nor #2 survives, and while it's interesting to think the value is, it certainly isn't anything we understand.
Maybe you think the value will still exist, but some how agents will replace all value with equivelents. That's an argument, but I don't think it will.
anyone can spin these kind of side projects and do easy talk, but the moment you actually try to use this on signed-in Linkedin or Amazon its going to fail
the only solution is to drive your regular browser with all your sessions/cookies via an extension
Yep, it's super similar to this! I think his is a bit overengineered, but tbh I haven't built Firefox plugins in forever so that might be the way you have to do stuff on FF. All you need is the websocket plugin to have "full access" to all visited webpages, and then your driver just hooks into the DOM like normal (with the extra ws layer—and even the websocket layer can be simplified away I think).
It is, because `--remote-debugging-port` is detected via the root DOM object (and that can't be changed unless you want to recompile Chromium), so for example, if you try doing a Google search with debugging enabled, you'll get blocked (usually just by being served a blank page).
And the worst part is humans just getting more cognitively demanding captchas.
Some days I'm literally fatigued enough I can't clear the overly complex hCaptchas these days I keep getting forced upon me and bots sail right through this crap.
I think it's inevitable that eventually one of the big AI companies makes something like this. It's such an obvious consumer win. "I am not a bot" checkboxes are a string and peg tethering the elephant.
Most residential proxies are already far more blocked and rate limited than any Meta IP. The internet is becoming a very weird place, where individual and "trusted" personal IPs are becoming a kind of commodity. Some sites are already scoring IPs based on usage activity - like a credit score. It's only a matter of time until this data is collated and commoditised. AI analysis is turning this up to 11.
> Some sites are already scoring IPs based on usage activity - like a credit score. It's only a matter of time until this data is collated and commoditised.
Spamhaus is nearly thirty years old and the notion of electronic distribution of IP and domain "reputation" lists is at least that old.
I'll bet my hat that the Internet "advertising" [0] industry has been calculating and determining the reputation of individual households (if not individual users) for at least a decade.
[0] The scare quotes are because its primary purpose these days is for dragnet private-sector surveillance.
Every time a new tool launches there's a good window where it can act as a scraping proxy. Back in the 2010s I used Google Translate for years to scrape hard targets like LinkedIn but these windows are much shorter these days as scraping is so much bigger.
One thing with Muse though is that you can scrape Meta's own sites which are currently all going under login walls and restricting discovery/search entirely.
I think this is the end of sites where it is expected that only humans may interact with them. It's been cat and mouse for a while, and some places like Reddit sell API access, but these agents are for all intents and purposes, humans interacting with the site. They're going to need to figure out new business models.
As someone having to fight Meta's bots everyday to keep websites accessible to actual customers, I'm not surprised to read that it can be seen as something positive on the other side of the fence.
The end game will be that either your site is fully open to bots and humans alike, or your site is open to humans only(×) and requires Airport security style identity verification.
There is at least one very large derivatives marketplace that will require basic recent price information (OHLC) to be behind user registration by next year. Enjoy what you have done scrapers.
"I'd sure hate if someone did this to my site, but at least the agent did it for me. I don't have to feel guilty now."
51 comments
[ 0.19 ms ] story [ 37.5 ms ] threadNot sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.
I agree with you, though, that the future has almost certainly already been decided, and see the eventual outcome of all this selfishness to be mandatory device attestation to access most services on the internet—which will of course never be allowed on any open platform. AI engineers who believed in freedom to compute (or just individual freedom more generally) have committed perhaps the greatest self-own in the history of our species to date.
Also it’s great(!) to see that we’re going from “but ethics” to “I got mine, who cares”.
Humans are interesting creatures.
Throttling is poor help though. Mass scrapers are using "residential proxy" loophole + rotating UA and other attributes. You can't throttle somebody without identifying them. Unless you're talking about a global rate-limit.
This predates AI as a _capitalism_ problem.
It's not that "we" are going from one thing to another. It's that these are two different people, with different ethical boundaries.
Is defeating captchas unethical? I don’t think so. Not on its own.
Is scraping unethical? I don’t think so. Not on its own.
Are there tons of uses for both of those that are sketchy or outright wrong? Yeah, absolutely.
As an example, imagine a really small community maintaining a small site/wiki/cms/forum that has the ultimate goal of promoting human relationships around a common interest (let's say retro computing as an example, but it could be anything). There are many many many such communities on the internet.
It's very improbable you will specifically instruct your LLM to access their site, it's way more probable it will happen without you even knowing, as a result of you doing some /deep-research or something. And not only that, but your LLM will probably spawn a ton of agents to gather as much information as possible in as little time as possible. A torrent of requests will go at this community's site, effectively killing it. They are a small community, they use their spare time and money to maintain something to serve them, they don't have the resources to serve your LLM and until you showed up they probably never even had to think about Cloudflare. They are certainly not against you getting the content, but they don't want you causing them issues either and you just did.
End result? Your LLM (effectively you) DoSed a small community's site. You caused harm. Could you have caused similar harm if you were doing it on your own? Sure. But you would have done it on purpose, not accidentally while instructing your LLM to do something else.
This isn't a made up story, it has happened already more than 1 times.
So the question is, now that you know your LLM can cause harm without you even knowing it, how does this change your stance?
Sure, Nazi, Hitler or Stalin all considered themselves to be the good guys. Kushner, Trump, etc are also acting within their ethical system (if I am getting money or power or fame it is ok to do it). Just about the only exception is Thiel who openly frames himself as evil, but is proud of it.
That does not mean we cant criticize their crappy actions or "ethical frameworks".
And yes, all the above are making deliberate decisions to be unethical assholes.
1. People like other people.
2. Businesses need to sell product
The internet mixes those people, and an Agent basically pushes the signal to noise ratio that businesses have relied on the intenet to basically zero.
If the internet just allowed indescriminate traffic, neither #1 nor #2 survives, and while it's interesting to think the value is, it certainly isn't anything we understand.
Maybe you think the value will still exist, but some how agents will replace all value with equivelents. That's an argument, but I don't think it will.
in batman, bruce wayne shuts does the massive surveillance network, seeing it was problematic (2008)
in spiderman, peter parker just shrugs off being able to know where anything is happening anytime (2026)
Yes, that's what I'm referring to. Citing myself:
> Also it’s great(!) to see that we’re going from “but ethics” to “I got mine, who cares”.
This is exactly how I observe a shift in general.
the only solution is to drive your regular browser with all your sessions/cookies via an extension
Is that necessary? You could start Chromium with an open debug port and use Chrome Devtools Protocol to send commands.
I've had success launching the browser and using dumb dumb methods to get around the captcha before attaching playwright.
Dumb dumb methods = wmctrl, xdotool, bash (work fine for Cloudflare's "are you human" check)
Assuming "more people catching onto" this, expect Cloudflare and most everyone else to follow the suite.
Some days I'm literally fatigued enough I can't clear the overly complex hCaptchas these days I keep getting forced upon me and bots sail right through this crap.
It can be used for testing software and using it as a human support.
Muse's utility has significantly dropped with the blockages.
Spamhaus is nearly thirty years old and the notion of electronic distribution of IP and domain "reputation" lists is at least that old.
I'll bet my hat that the Internet "advertising" [0] industry has been calculating and determining the reputation of individual households (if not individual users) for at least a decade.
[0] The scare quotes are because its primary purpose these days is for dragnet private-sector surveillance.
One thing with Muse though is that you can scrape Meta's own sites which are currently all going under login walls and restricting discovery/search entirely.
Good one, I can't stop laughing. Thanks!
So of course I clicked yes and it dutifully convinced the site that it was not a bot.
But I'm wondering: at what cost?
(×) and their AI delegates
Multiply it by 1,000,000 access attempts - daily.
It is question of time when even normal browsers would need some allowed fingerprint to go to sites otherwise caught by clouflare or similar wall.
"I'd sure hate if someone did this to my site, but at least the agent did it for me. I don't have to feel guilty now."