For a while I worked in ads, and the specific team I was in put us in a position of being on both sides of this. We were tasked with monitoring advertisements for fraud, which meant we both had to catch bot traffic, but also we had to check ads for scams/phishing/etc. So on the one hand we were trying to plug holes in our own bot detection for botnets trying to drive up impression numbers to get website operators paid more, but on the other hand we were trying to bypass bot detecting on the part of phishing page authors (who would do things like display a totally plausible e-commerce page if they think the ad viewer is a bot, but a bank of america phishing page if they think it's a real user).
> other hand we were trying to bypass bot detecting on the part of phishing page authors
An approach I recently saw gated the phishing page behind a Google Account login page. User that was logged in wouldn't even notice the brief redirect. Scanners would just get stuck. I hope they've patched it by now though...
A coincidence of how I work with my browser profiles is that the browser (profile) that does most of my web browsing isn't logged into Google. It's nice to see that this has extra benefits:) Now if you'll excuse me, I need to go move my entire computing environment into a VM so that viruses will think they're being analyzed and refuse to run... (Kidding but also not)
You can use Firefox nightly and create a webextension using their internal APIs. Gives you access to injecting non-synthetic or faked events (they look indistinguishable since they are emitted in the same code path that the real input device uses) and scraping the DOM with the added advantage of not exposing the automation flags that Firefox has when you use Puppeteer and the like in headless, automated mode.
edit: Firefox Nightly because it gives you much deeper access to Firefox internals.
I can't give examples (I did the work for a company so I don't own it) but starting at [0] [1] should get you to the internals. It definitely carries some risk of Firefox Nightly just randomly breaking when it updates but this solution got us beyond quite a number of bot detectors so if you're willing to risk Firefox Nightly it's possible. We didn't need nearly the flexibility something like Puppeteer has since we were injecting rrweb into the target page to record the DOM mutations and sending those to other clients.
That will give you a good start on where to even begin looking. web-ext is insanely handy for launching Firefox with a WebExtension already installed (but I did a lot of other work around this to productize the offering).
Kasada seems to catch every off-the-shelf stealth plugin for puppeteer/selenium and the like, used famously by various shoe retail sites and Twitch. It isn't scraping per se, but you could try making a Twitch account and see if you get a "Unsupported Browser" error by the integrity API secured by the Kasada VM.
There's actually an (admittedly small) data scraping industry.
Let's say you are Wal-Mart, and you'd like to know which of the products you sell are available cheaper at Target. Or which neighbourhoods they deliver to that you don't, or which stores have longer opening hours than yours, or whatever.
You can't legally exchange data with Target directly, that would risk making you an illegal price-fixing cartel. You can legally visit their website, but you don't feel like matching up 10,000+ products.
Instead, you call up a business intelligence firm who has already scraped your site and theirs and matched the products up. They'll send you a CSV, for a price.
I can't help but feel a little conflicted about things like this in the current climate. On one hand, as a developer trying to just get some updated stuff into a spreadsheet, this looks extremely useful.
On the other hand, there are a lot of people right now that want to keep stuff accessible to humans and not have it scraped for models. I know the lid's completely off the box on that front so it's probably useless to fight it, but seeing products explicitly designed to circumvent bot prevention feels kinda bad.
Agreed, we've seen a lot of governing bodies _now_ start scraping to fight scam sites and "fake" sites, so there's a lot of legitimate use-cases popping up. It's not necessarily just about gathering e-commerce data or backfilling lack of an API anymore.
no services remain "good" so why ask the question. it gets ignored or answered but then a decade or two down the line the philosophy gets scrapped, a story as old as time.
I still don't understand why Google publicly removed "Do No Evil" from their company. It's one thing to not follow it but to actively remove it feels weird. Like they wanted to say "HEY Y'ALL GET READY FOR EVIL!"
Interesting I remember a bunch of articles and stuff talking about them removing the do no evil bit. Guess I didn't do enough research. Appreciate the correction!
I know this is slightly off topic, but I have to ask: Why not create a new account rather than use the same self-described throwaway account you used to ask about raising prices (for browserless?)[1]? For what it's worth, I initially assumed you likely weren't legitimate and didn't engage, so if you want to keep contributing to threads as yourself you might want to consider using a new/different username.
the problem isn't scraping, the problem is the "content" that is being "created".
A site I worked on changed our captcha, and in the 24 hours after, we had a 90% reduction in new accounts. We had a full API including webhooks, the only private data was on the account page, there was no reason to scrape, but even if you did, there was no reason to be logged in... these accounts were used to harass, intimidate and scam legitimate users.
Every single time I see these scraping discussions I get the same thoughts:
Businesses use data from the user. The Business does additional crunching on that data to derive new interesting data for the user. Who owns the data? The user or the app?
At the very least the user partially owns the data and as such, I'd argue that the user should have the right to share the data between different applications however they see fit. However, businesses tend to think that they somehow have the legal (moral even?) right to keep that data in their walled gardens. For as long as this (imo unfair) stance is common, I think that data extraction by use of these anti-bot-bypassing technologies is fair game.
If the data is public or semi-public then chances are I consented to display that data there. I consented to it being used on that site, for the purpose of that site. Not for random other companies.
And most scraping isn't done by users. It's done by companies. For profit. Often for less than enlightened reasons.
LinkedIn is a good example: I want my data displayed to people on that job site. I don't want it harvested by every recruiter under the sun who will then spam me. I certainly don't want that data sold between those recruiters long after I deleted my account on LinkedIn. Tinder and sites like that are also an obvious example: yes it's (semi-)public, but I also wouldn't want it to be scraped and harvested by some company – I just want it to be shown temporarily to a limited set of people.
In general, I don't think people should have a moral right to decide where and how the data that they made public is used, or to decide if it can get scraped or not.
And, in general, I take the fact that you published something on the Internet as a tacit moral consentment for the rest of the world to use it how they want.
This comes with a couple of big asterisks, because (1) Copyright law exists, and I generally try to not break the law, even if I don't agree with it. But the discussion in this thread is mostly separate from copyright: for instance, I don't think a court would see someone scraping and redistributing data from someone's LinkedIn profile as a copyright infringement case.
And (2) because I think that in some specific cases, using published data can be morally wrong, but not as a general rule.
i somewhat agree; people volunteer it when posting anything online. but they also volunteer their advertising id on their phones (even if they dont know it) - just as they dont know (and dont care) they are the product when on websites like facebook
i feel the 'antibot' stuff is more related to the adtech industry vs site-scrapers - remember getting a dedicated server and having friends click on links just to pay for it? Geocities and all these free websites, the biggest costs were bandwidth and storage (not that its not now)
since the AI Boom, there's just more hype over people wanting 'credit' (or money) for something they posted on a forum X-units of time ago.
its called the World Wide Web for a reason.. keep it open, even if it is to 'a bot' - never know when somebody's 'bot software' is reading your webpage for somebody who has some disadvantage and needs assistance
That's incorrect. The fact that the internet is not a community is not a problem - it's objectively correct and morally desirable. It's neither possible, nor desirable, nor good, to treat the internet as a community. It isn't, and we wouldn't want it to be in the first place, just like we wouldn't want all the nations of the world to be fused together.
That was only possible because the internet was far smaller. There are now billions of people with internet access - the cat is out of the bag.
The best you can do is try to form your own little community in a corner of the internet - which is actually a great idea. Mass social media has so many problems - trying to make a little nook for yourself and a small group of enthusiasts/friends seems like the best way to go.
On the other hand, I'm not sure if the whole idea of people visiting a website where they ought to behave and play by the house-rules is doing the world much good. And I'm not just saying this because one of my pet projects was crippled by an overzealous bot-prevention system (yes I was using a program I wrote myself to access your system, no that doesn't make me a bot).
Quite a few things would be vastly better when the information that is on websites is not just publicly readable but publicised in the full sense of the word. Several industries are paying 20~40% revenue just to be findable.
Now maybe scrapers aren't the best solution, but making stuff less accessible is usually not doing much good. Especially not for accessibility.
> keep stuff accessible to humans and not have it scraped for models
I suspect that over the next 100 years, as AI and biotech come together, the boundaries between human and bot will start to blur. At some point it will be unethical to differentiate between human, bot, and everything in-between.
I argue that that time is now, if at the very least for the sake of AI-based assistive technologies to be developed to turn a disabled {100% human} into a highly functional {80% human + 20% bot}.
Bot detection should, for this reason, be doomed as an ADA violation.
Perhaps the "conflict" lies in the notion that a "developer trying to just get some stuff updated into a spreadsheet" is somehow different from "people right now that want to keep stuff acceessible to humans and not have it scraped for models [for use by developers]."
What if all users of the www were treated equally. There is in practice a conflict of interest between www users calling themselves "developers" and the rest of www users. Yet these "developers" purport to act 100% on behalf of the rest of www users, not themselves. The failure to acknowledge any self-interest is dishonest.
I agree, being a dev myself hosting few website, and a data enthusiast, it is very hard to balance both, except if all data was open source and released for all it's owners. All online platforms, especially social media relay on users input (reddit, HN, Twitter, etc ...) yet the trend seems to be heading the other way (closing down apis, etc ...), imo this is quite unfair. Not to mention awful practices by shopping giants like Amazon and Aliexpress, Airlines, Booking.com (et al.) to profile people and "scam them". The data ethic question is lost on both sides, only the few decent ones that remain are the true victims, you mom'n'pop website.
The problem of scraping content for models should be solved on a legal level. Trying to do it on a technical level is mostly futile, and wrong, because it actively makes legitimate uses worse.
If you really really really want a technical mitigation in addition to a legal one, then paywalls are vastly better than incredibly messy and problematic heuristic bot-detection. There's a huge payment problem for content anyway, so paywalls would mitigate or solve many other problems in addition to the AI-training-data one.
In the end, the best way to avoid being blocked is to be a good actor. All of these hacks won't stop someone who's determined to prevent access (ie: LinkedIn).
That's actually one of the reasons why I started https://browserbase.com/. Maintaining headless browser infrastructure can be such a pain. I've spent a lot of time managing headless chrome fleets at scale, so happy to answer any questions.
If I understand correctly, a lot of the issues you can run into with regards to blocking come from the fact that you're using a headless browser. Past a certain point, wouldn't it be less work to use a regular browser and drive with Selenium or similar solutions? Or does that not address the kind of problems you're facing?
I used to semi-automate access to some sites by using Selenium with a non-headless browser. These were sites where there were just one or two pages where I wanted some automation to fill out a form or scrape some data, and they frequently made changes to the home page that made it hard to automate navigating from the home page to the pages I wanted to automate.
The idea was to have a script use Selenium to launch non-headless Chrome and then wait:
driver = Chrome()
driver.get("https://example.org")
input("Press enter when ready")
I could then manually deal with logging in, answering any CAPTCHA that came up, and navigate to the page I wanted to run my automation. Then I could press "enter" in my terminal and my script would continue.
That used to work fine, but then on sites using Cloudflare's CAPTCHA it stopped working. Solving the CAPTCHA would just result in another CAPTCHA.
I tried an alternative Selenium Chrome driver that was supposed to be more stealthy, and tried setting various flags that were supposed to make it so JavaScript could not tell that Selenium was there, and those worked for a while, but then they stopped working.
The results were similar using Selenium with Firefox.
I also tried Puppeteer, with Chromium and Firefox, and they too could not get past the CAPTCHA loops.
I then tried Playwright. With Chromium and Webkit that got the CAPTCHA loops. With Firefox it actually worked. I didn't even see the CAPTCHA. The non-interactive check for not being a bot passed.
Still, the whole approach seems fragile. I don't know if Firefox/Playwright working was due to some fundamental difference between Firefox and the others or just Cloudflare having not yet gotten around to dealing with it.
Are there any stories you're willing to share, any tough nuts you've had to crack to improve some aspect of operations, whether it be reliability, performance, bot detection evasion, or something else completely?
I've only dealt with scraping on a small scale and I quickly realized that running "browsers as a service" is a pain in the ass, they're not exactly lightweight, they like to get "stuck", balloon in memory or some such.
I imagine your business will be quite successful if reliability is good and the price is right!
Trying to defend against malicious bots is tedious at best, impossible at the worst. I don't really see how lowering the bar would be a net positive. People will start using methods that will cause more collateral damage and just reduce user freedoms.
My guess is that this only increases the push towards attestation and attestation-like approaches. Login walls and PAT/Privacy Pass are just the start.
Even if you or I might find attestation against user freedoms nobody is going to legislate attestation away. The alternatives to the vast majority are way worse.
If it's too difficult to guarantee end-users' devices are legitimate using other means then the alternative would just be more invasive fingerprinting and login/paywalls.
In addition to that any access restrictions would probably end up being less precise and thus cause more collateral damage.
Which would also mean an increased amount of false negatives, leading to an increase in spammy content.
It will, but only on Android recent, locked-down Android devices. The first step in device attestation will probably allow the Android click farms to continue, but once it's in place, restrictions will just become stricter over time.
Not only click farms, these devices are also used for SMS toll fraud and other similar attacks. SafetyNet's attestation only helps to the extent that you can ban a specific model/maker.
Yes that's basically it, you can ban a specific device but that's like removing water from the ocean with a bucket.
Betting on device attestation really means betting that computing devices will become more and more expensive in the future since the device cost is the only blocker created by the attestation.
And if there's one thing I'm expecting, it's that's not going to happen, device prices will continue to decline.
It could, but not in a way any one of us would want it. This specific attested device isn't marked as legit by any of the big players? Into the blocked list you go. This is very similar to what we see today with email providers for example. If you start running a new server, you start with a negative reputation and have to climb out, you don't start from neutral.
> If you start running a new server, you start with a negative reputation and have to climb out, you don't start from neutral.
Not really. If you start a new server from an IP range with known bad actors, sure. Many have tried and failed to run their own mail servers from Digitalocean, vultr, or even more dedicated-esque server hosts and pretty quickly see how much of a hassle doing so is.
But if you buy a v4 /24 from some reputable or old company and get it assigned to your own AS, you won't have negative reputation and will be fairly successful as long as you have spf/dkim/dmarc set up properly.
> But if you buy a v4 /24 from some reputable or old company and get it assigned to your own AS
Doesn't that have an extremely high barrier to entry? A legal entity, significant ongoing fees, and some infrastructure?
I'm under the impression that this is entirely out of reach for anyone just trying to run an email server at home for personal use. I would love to be wrong!
Services like these are why massive blocks of IP addresses end up firewalled off, and why Cloudflare/Google CAPTCHAs are absolutely everywhere.
Normally, you can just block non-consumer ISPs, but this site offers "residential proxy" services (basically a botnet-as-a-service), which means that now consumer IP ranges need to be selectively blocked as well.
I think PAT/Privacy Pass will solve this problem as far as normal users are concerned ("normal" meaning "running Windows, macOS, iOS, or Android, on devices with hardware attestation capabilities"), but soon enough we'll arrive in an age where you can only visit so many web pages before you've exceeded your daily internet allowance.
Attestation creates all sorts of challenges absolutely, but it actually doesn't really help here - there's plenty of services doing this on real devices, and no real challenges to doing so at the prices people will pay. The overhead is the one-off cost of the cheapest legitimate device that will attest, split between the many users to get near 100% utilisation over a large period. Once you look at cheap androids & PCs, this gets very cheap indeed.
The only solution is actual paid services, or real-user verification (and deduplication, which means little privacy, which means legal problems in much of the world) for free accounts.
If you publish something on the internet for free, you _must_ accept that people can read it automatically. You can make it difficult, make it a bit more expensive, but at the end of the day paid data means paywalls.
What are the legitimate (i.e. legal) use cases for a product such as this?
I agree with another comment that called this "Abuse as a Service". It seems to me this product's design goal is nothing more than to circumvent measures site owners take to prevent abuse of their site and run a sustainable business.
I used their previously available bot detection defeat to add an import feature to my website: Users could link to their creation on another site and my site would scrape the publicly available content so they wouldn't need to re-enter all their data
I've used their product many times actually, and I'm shocked on Hacker News of all places no one's thinking of anything besides abuse. How often is it useful to get information from a webpage and apply it in a new context? Then think of how often said webpage is behind a Cloudflare bot detector.
If it's the user's data, then under GDPR the other site is obligated to provide a way for them to download/transfer it, specifically with this use case in mind.
They are completely in the right to block you though, you're not the owner of that data, you might be breaking their TOS.
> If it's the user's data, then under GDPR the other site is obligated to provide a way for them to download/transfer it, specifically with this use case in mind.
In Europe, if the company is actually following the law, in theory yes.
> They are completely in the right to block you though, you're not the owner of that data, you might be breaking their TOS.
There are a litany of posts on this very site that detail why HiQ vs LinkedIn is more nuanced than you're making it out to be. HiQ didn't ultimately have the slam dunk win that people think they did.
- GDPR doesn't require it be a convenient export. Users want to paste a link on my site, a click a button, and have it magically appear. Not fill out a form, dump their entire account and sift through that.
- I never opined on the validity of blocking bots
- I never opined on if it's breaking their TOS
Abuse implies a harmfulness. Giving users a quick import option from already public data isn't harmful.
“ In exercising his or her right to data portability pursuant to paragraph 1, the data subject shall have the right to have the personal data transmitted directly from one controller to another, where technically feasible.”
They’re not necessarily in the right to block you, if you’re the data subject or acting on their behalf.
That's a bad example, "programmatically unsubscribing" means giving spammers information that this address is alive. A much better solution is to report the unwanted email as SPAM, so the sender's reputation takes a hit.
(and for that 1% of the cases where the address is not a spammer and user knows it, they can just hit "unsubscribe" manually)
What about something like Nitter? Archiving? Adversarial bridging between different platforms? Automation?
How will well-behaved scrapers undermine the sustainability of a business? I guess adblocking is one, but we can already do that with uBlock and that's legal. Or adversarial bridging, but that only serves to boost competition.
In other words, the question is flipped; why would well-behaved (i.e. non-DDoSing) scrapers be illegal?
Legality isn't the question here. If you want to speak to the legality, anyone circumventing a robots txt that explicitly has your bot's user-agent and 'disallow: *' is unauthorized access (I imagine it's more nuanced for 'user-agent: *'). No website is required to allow anyone to visit and can discriminate against any client or software any way they want.
I think you're conflating automation and intentional avoidance of bot detection as part of automation. The issue I have is not that this service allows users to automate browsing activities. The issue is that this service deliberately tries to circumvent being detected as automating browser activities because websites are trying to prevent bots. There are LOTS of services that allow users to create automations without disguising themselves. If you are using well-behaved scrapers that respect TOS then you shouldn't have to use a service like this.
Nitter is an example of a service that explicitly disrupts Twitter/X's way to make money. If they can't make money then they can't provide the service, there would be no Twitter/X, and hence no Nitter. Of course they would try to prevent that kind of behavior and it should be obvious why. Resorting to using a service like this in order to continue using Nitter should raise some alarm bells. Sure you can still do it and rationalize it however you want, but you have to acknowledge you're trying to get the value of the service without paying for it.
Perhaps there are cases where there is a dissonance between a website's TOS and how they are blocking bot traffic? That sounds like a valid gripe. Otherwise, I don't buy the argument.
That's fair enough. I think that falls under similar arguments to adblocking; it's against ToS, and affects the revenues of ad-supported businesses, but it seems like the popular view is to use it regardless.
for stuff I use similar self-hosted solution:
detecting when kid lessons are available on local portal.
but to be fair cheapest option here ($200) isn't usable for non-business usage
Ps: context why I need automation for such thing: those lessons are really popular and are announced at unpredictable time / there might be another spot when someone resigns
I've got a couple of things I've used browser automation tools for:
• I want to automate (or at least semi-automate) downloading bank statements. I've got ~14 accounts (checking, savings, credit card, IRA, investment, HSA) across 7 financial institutions.
It's tedious to go download statements from all of them manually.
• I want to save stories from FanFiction.net (FFN) for offline reading. FFN's terms allow automation as long as it doesn't operate faster than a human [1].
[1] From their TOS:
> You agree not to use or launch any automated system, including without limitation, "robots," "spiders," or "offline readers," that accesses the Service in a manner that sends more request messages to the FanFiction.Net servers in a given period of time than a human can reasonably produce in the same period by using a conventional on-line web browser.
> I want to automate (or at least semi-automate) downloading bank statements. I've got ~14 accounts (checking, savings, credit card, IRA, investment, HSA) across 7 financial institutions.
Could you not shoot an email to those institutions asking for a copy of the documents?
Not OP, but I do the same for ~7 accounts across 5 institutions. There's no need to contact them since you can manually download the statements, but it's a chore if you're doing it frequently. I usually run my script a few times a week.
Scanning for malicious and phishing websites. These types of sites are just enjoying the ease of free services like Cloudflare to block automated analysis tools and tailor their phishing campaigns to very specific geographical locations and user groups.
I think they should introduce request rate limits per IP/domain, for example max 1 parallel request. In this case there will be no significant load, but the data can be scraped.
Scraping is important for example, to monitor competitors' prices to see the opportunity to raise your own prices.
And let's not forget that Google does a lot more scraping than anyone else and has ridiculous profits from it.
I'm not a customer, but I have a use case that in my opinion should be legal.
For years I've used my own terminal UI player (di-tui) for di.fm. At some point in the not-so-recent past, di.fm added Cloudflare's WAF, which prevents me from using one of my app's features: managing channel favorites within the app.
To be clear, I'm a paying di.fm customer, and my app only works for paying customers. But now my preferred method of listening to di.fm is slightly hamstrung because Cloudflare's WAF sits between me and little string token available to every browser that accesses di.fm (even non-paying customers).
You sell jastingo™ brand widgets, but you notice fakes are being sold on eBay, Amazon, AliExpress. You set up a scraper to search for jastingo widgets every day on every marketplace site, but you get blocked. So now you need an unblocker to enforce your copyright/trademark/patent.
I'm genuinely scraping a certain social network that doesn't have an accessible API to do what I need. My user is logged in and I just automate the logged-in browser to go to the pages and get the data I need into a console so I can get the data I require.
If there was an accessible API to do what I need, I wouldn't do this because scraping sucks. I have to write 100 JavaScript edge cases to handle all the times the host's servers fail in very weird ways. Plus, walking DOMs on these shitty sites with 10,000 nested divs is not fun. GPT helps with this.
It's net-positive for the host though, as I upload a lot of valuable content that their users genuinely like, but it sucks that I have to be sneaky to get the data I need.
Yeah that’s my reading. No way this is passing Akamai bot detection.
There are lots of signals like timings, user tapping and scrolling behaviour, signed sessions cookies that represent browsing flows which may be legitimate or not. And that’s all assuming you’re on a good looking IP. To do this you need a large supply of residential IPs which then leads to the dodgy underworld of botnets.
I’d be surprised if this works for anything but the most basic bot protection, this is an advanced space.
If it does work for those cases, they should be either keeping it quiet and making bank, or boasting about having a secret sauce, not basic stuff like this.
Edit: for apps, Akamai provides an SDK that uses things like your motion data to create a signature that suggests that you're a real user. This signature is either injected into API requests or into a webview session. I'm sure it's crackable if you dedicate significant reverse engineering resources to it, but then you've got to crack every version, crack every other implementation from other companies, etc. Non-starter.
When it comes to "motion data", are you referring to the client's movement around the website, i.e. which URLs they access and in which order? Their taps, clicks, scroll events, and other inputs?
Motion data (especially in the context of an SDK, which I assume means they're talking about in-app environments rather than browser environments) usually refers to gyroscope readouts on a phone or tablet.
A device that stays rock-steady throughout an entire browsing session isn't necessarily suspicious on its own—for example, you could have your phone laid flat on the table while your browse with your pointer finger—but it can be a useful tell in combination with other suspicious factors.
It does for the browser APIs, however Akamai provide an SDK to embed in your app, and apps do not require permission for gyro data. Then the standard practice is to a) pull a token out of the SDK for any API requests you make, and b) connect the SDK to any web view instances via the JS bridge so that any requests made in the embedded browser also get the same tokens.
For web-only, I believe they have a JS only bundle that your site can include which I would imagine does different things, but which would also bring a higher risk profile associated with it. Sites use these risk profiles to determine things like whether to offer specific services, whether to ask for more authentication, etc.
Ah! That makes more sense, I thought we were talking about the web.
That said, since I wrote that comment, I found out that on Android, both Firefox and Chrome grant access to gyroscope data without a permission dialog, which is extremely surprising. I don't have an iOS device to verify, but apparently Safari gates the API behind a permission dialog.
Asking out of almost total ignorance of this field, what prevents someone from running a script that sets their agent to a phone browser and then sending fake gyro data? Surely there's a way to emulate enough to make it look like a phone that's being held by someone, right? We can do realistic camera shake in blender to the point where something looks like it's being held by a person, why couldn't we fake minute movements like the device is held?
Why do we even need an actual device? We can emulate if we even need to and set our headers to look like we're coming from a device browser.
I'm sure that happens, but I haven't done any work with gyro data myself. There's similar logic for mouse cursor movements, there are libraries out there that will generate a natural looking curve that moves the cursor from one position to another, with imperfections that emulate human hand movement.
> Why do we even need an actual device? We can emulate if we even need to and set our headers to look like we're coming from a device browser.
This one is much harder, your browser, OS, and hardware leave a uniquely identifiable fingerprint (with Javascript enabled). A website can render some graphical pattern on a <canvas> or audio in an audio context, and the resulting output will have minute differences that originate from your rendering and audio pipelines.
You can try to fake these, but it all depends on the sophistication of the target website. You can quickly end up in really deep rabbit holes: https://www.nullpt.rs/devirtualizing-nike-vm-1
It's a never ending battle. Lots of tools aren't as sophisticated as they claim to be, and the current mechanisms inject a lot of "other stuff" that can be easily found. We're trying to do this in a more novel way that's faster, less prone to needing frequent updates, and is more akin to how actual users interact with browsers. Definitely work to be done, but it's exciting to see, and I appreciate the good luck!
I've been fighting tailor made bots for over 15 years, and I can't say I'm a fan of people trying to circumvent it. In some cases it might not matter, but in others it can actually ruin a lot of things.
Let's say I have a few 100 Gbps connections that is mostly idle, is it fine if I direct them at browserless? No? Exactly, that kind of traffic is not wanted.
Browserless could implement limits on their side, for example, allow only 1 parallel request to IP/domain. This way there will be no significant load, but the data will be scraped.
As if IPs matter in the golden age of IoT botnets and residential app malware.
You can't even block the amount of subnets that's coming for you in a DDoS attack, thinking a human is able to keep up with something like this is pretty blindsighted and naive. The differentiation of network protocols and relay attacks alone is way too slow to be mitigated in most systems.
I'm sorry if this offends you, but it's companies like you that I despise. WAFs are a good thing. Your bot is not authorized. We'll be blocking all IP ciders from you, your referrers, and your service. Thanks.
90% of the problems solved with puppeteer could be instead solved by replaying a few API requests. Cheaper, less wasteful, faster, in some cases, harder to detect (WAF is rarely deployed on APIs because there are much less heuristics)
APIs change all the time. I work with plenty of undocumented APIs from a service we actually pay for, and it's a hassle of playing whack-a-mole when something changes without notice.
I do to. Almost every project I've worked on involves an undocumented API. It's still cheaper to devote developer time to reacting to changes than paying to run headless chrome and evade WAFs
But you can't fully detect the changes without the front-end. When an API changes, so does the official front-end that is interacting with the API. We need to track changes to the HTTP API requests coming from the front-end. You can't detect those changes just by hitting the APIs directly. If the official front-end changes how it sends data to the back-end, we need to change to match it.
We also need to detect and react to UI layout changes, and the headless browser is the only real way to do that.
> If smaller indie hackers are going to build useful competitors
Any useful competitor will have to scale well beyond the point of indie hacker, where someone is (eventually) getting rich. That doesn't make for a great argument for bypassing consent.
Depends on how you describe "useful." Kagi is a profitable search company that satisfies its customers while competing directly with Google. They aren't useful in the sense that they're taking marketshare from Google, but they are useful to their user base. I hope they continue being able to scrape sites with the same freedom as other search engines.
Edit: As mmcclure points out Kagi actually doesn't index much. So that's a bad example that might invalidate my point.
I'm a happy Kagi user, but they might be somewhat a bad example. I suspect their actual indexing work is pretty limited, and they only seem to claim that it's for indexing "niche" stuff.
> Our data includes anonymized API calls to traditional search indexes like Google, Yandex, Mojeek and Brave, specialized search engines like Marginalia, and sources of vertical information like Apple, Wikipedia, Open Meteo, Yelp, TripAdvisor and other APIs. Typically every search query on Kagi will call a number of different sources at the same time, all with the purpose of bringing the best possible search results to the user.
Scraping itself isn't a problem. Scraping content where people are actively trying to stop it from being scraped is where things get questionable.
both OpenAI and Google rely on scraping as their foundation for…everything.
OpenAI and Google also (claim to) respect the most basic bot deterrent, robots.txt, with their crawlers. If someone doesn't want their content scraped to be included by search indexes or LLMs/models, then that certainly includes "smaller indie hackers."
I want to try Browserless but it doesn’t appear to be possible to do so without signing up for a free trial of a $200/month plan - have they especially hidden this or am I just blind?
They recently increased their pricing quite a bit. We're looking to offer a much more affordable pay-as-you-go pricing model at https://browserbase.com/
Feel free to shoot me an email if you're interested in trying it out! paul@browserbase.com
I'm surprised that puppeteer uses that default window size. Last time I did a scraping project, I assumed randomizing the window size would be table stakes.
Puppeteer is not created for scraping per se right. I mean it could be, but isn't puppeteer created/maintained by google? So it's not meant to fly under the radar and avoid detection, it's just a naive tool for automating websites that would never try to block it.
Personally I see a legit use of this as avoiding detection with small amounts of traffic. I just want a browser bot to do things in my name, and not DDOS, but just spare me from having to personally go and tap my phone screen.
I'm thoroughly confused on their pricing model. What is a unit? Am I reading this correctly that $500/mo comes with 25GB? So $20 per GB transferred?
Which we're talking about running a real browser downloading all the bloat of a target website, 25GB doesn't go very far. Is this just absurdly expensive?
The first link shows the tests and is the most educational.
Does anyone know of more recent tools that show why bot detection was triggered? Versus just testing against Cloudflare bot detection and getting pass/fail.
So basically is this Evil as a Service, or is there a legitimate case for this that is not identical to "stress testing is a legitimate case for me selling botnet DDos"?
IFTTT is a legitimate use. There are an unlimited number of valid non-evil use cases for automated tasks looking at websites and doing stuff based on the data scraped.
IFTTT is a legitimate use of scraping public data from websites. But forget about IFTTT, it was just used as an example. It's just an example of automation looking at public website content and then taking some action. You're too focused on IFTTT and not the problem with blockading publicly accessible data.
For example, Zillow goes to great lengths to restrict bots, but I want to get updates if a property changes somehow on Zillow. It's publicly accessible data if you're not using a bot. "IFTTT" is just an example of a service that automates these things. I can write my own automation service, but if I'm using a bot, Zillow will block me (I've actually done this before).
I'm surprised Cloudflare hasn't tried to create a market for this yet. They could charge some sort of fee between the scrapers and the scrapees and create additional revenue streams for sites being scraped. Hell they could even give tools to sites that lets them manage who can scrape them and who can't.
I'm not saying I like this, I'm just saying I'm surprised it hasn't happened yet.
Cloudflare's WAF breaks the web. The web should be open, anonymous, and automatable on behalf of users. Any web service that doesn't like this idea should not get my business. Period.
What we really need is a browser that absolutely cannot access sites "protected" by Cloudflare, and this browser needs to become standard. Sites that put themselves behind Cloudflare need to pay a price for their hostility, and that price is the loss of our business.
If I want to program a bot to get around the horrendous lossage of the terrible UI and multi-megabyte bloat you have created by farming out your web server programming to the lowest bidder, I have the right to do so. In turn my bot will respect robots.txt and will throttle itself appropriately.
This is the deal you signed up for when you built a web server. If you don't like it, GTF off the web. And take your attestation and WAF with you.
I tend to agree. I'm working on a web scraping project that allows an ordinary person to spot misbehaviour by well-resourced corporations. Cloudflare is absolutely getting in the way, and helping the bigger guys escape accountability by making them hard to track.
im not sure what the legal liabilities are for offering this type of service but couldn't somebody potentially use this to scrape a highly litigious website and end up bringing heat to browserless's company and its founders I wonder
"we just provide the tool" defence no longer works in court
like there's even a 7 day trial opening this up to anyone with bad intentions
160 comments
[ 2.9 ms ] story [ 220 ms ] threadAn approach I recently saw gated the phishing page behind a Google Account login page. User that was logged in wouldn't even notice the brief redirect. Scanners would just get stuck. I hope they've patched it by now though...
edit: Firefox Nightly because it gives you much deeper access to Firefox internals.
That will give you a good start on where to even begin looking. web-ext is insanely handy for launching Firefox with a WebExtension already installed (but I did a lot of other work around this to productize the offering).
[0] https://firefox-source-docs.mozilla.org/toolkit/components/e... [1] https://firefox-source-docs.mozilla.org/overview/gecko.html
Let's say you are Wal-Mart, and you'd like to know which of the products you sell are available cheaper at Target. Or which neighbourhoods they deliver to that you don't, or which stores have longer opening hours than yours, or whatever.
You can't legally exchange data with Target directly, that would risk making you an illegal price-fixing cartel. You can legally visit their website, but you don't feel like matching up 10,000+ products.
Instead, you call up a business intelligence firm who has already scraped your site and theirs and matched the products up. They'll send you a CSV, for a price.
On the other hand, there are a lot of people right now that want to keep stuff accessible to humans and not have it scraped for models. I know the lid's completely off the box on that front so it's probably useless to fight it, but seeing products explicitly designed to circumvent bot prevention feels kinda bad.
Source: Founder@browserless.io
[1] https://news.ycombinator.com/item?id=21271224
A site I worked on changed our captcha, and in the 24 hours after, we had a 90% reduction in new accounts. We had a full API including webhooks, the only private data was on the account page, there was no reason to scrape, but even if you did, there was no reason to be logged in... these accounts were used to harass, intimidate and scam legitimate users.
Businesses use data from the user. The Business does additional crunching on that data to derive new interesting data for the user. Who owns the data? The user or the app?
At the very least the user partially owns the data and as such, I'd argue that the user should have the right to share the data between different applications however they see fit. However, businesses tend to think that they somehow have the legal (moral even?) right to keep that data in their walled gardens. For as long as this (imo unfair) stance is common, I think that data extraction by use of these anti-bot-bypassing technologies is fair game.
And most scraping isn't done by users. It's done by companies. For profit. Often for less than enlightened reasons.
LinkedIn is a good example: I want my data displayed to people on that job site. I don't want it harvested by every recruiter under the sun who will then spam me. I certainly don't want that data sold between those recruiters long after I deleted my account on LinkedIn. Tinder and sites like that are also an obvious example: yes it's (semi-)public, but I also wouldn't want it to be scraped and harvested by some company – I just want it to be shown temporarily to a limited set of people.
And, in general, I take the fact that you published something on the Internet as a tacit moral consentment for the rest of the world to use it how they want.
This comes with a couple of big asterisks, because (1) Copyright law exists, and I generally try to not break the law, even if I don't agree with it. But the discussion in this thread is mostly separate from copyright: for instance, I don't think a court would see someone scraping and redistributing data from someone's LinkedIn profile as a copyright infringement case.
And (2) because I think that in some specific cases, using published data can be morally wrong, but not as a general rule.
i feel the 'antibot' stuff is more related to the adtech industry vs site-scrapers - remember getting a dedicated server and having friends click on links just to pay for it? Geocities and all these free websites, the biggest costs were bandwidth and storage (not that its not now)
since the AI Boom, there's just more hype over people wanting 'credit' (or money) for something they posted on a forum X-units of time ago.
its called the World Wide Web for a reason.. keep it open, even if it is to 'a bot' - never know when somebody's 'bot software' is reading your webpage for somebody who has some disadvantage and needs assistance
The best you can do is try to form your own little community in a corner of the internet - which is actually a great idea. Mass social media has so many problems - trying to make a little nook for yourself and a small group of enthusiasts/friends seems like the best way to go.
Quite a few things would be vastly better when the information that is on websites is not just publicly readable but publicised in the full sense of the word. Several industries are paying 20~40% revenue just to be findable.
Now maybe scrapers aren't the best solution, but making stuff less accessible is usually not doing much good. Especially not for accessibility.
I suspect that over the next 100 years, as AI and biotech come together, the boundaries between human and bot will start to blur. At some point it will be unethical to differentiate between human, bot, and everything in-between.
I argue that that time is now, if at the very least for the sake of AI-based assistive technologies to be developed to turn a disabled {100% human} into a highly functional {80% human + 20% bot}.
Bot detection should, for this reason, be doomed as an ADA violation.
What if all users of the www were treated equally. There is in practice a conflict of interest between www users calling themselves "developers" and the rest of www users. Yet these "developers" purport to act 100% on behalf of the rest of www users, not themselves. The failure to acknowledge any self-interest is dishonest.
If you really really really want a technical mitigation in addition to a legal one, then paywalls are vastly better than incredibly messy and problematic heuristic bot-detection. There's a huge payment problem for content anyway, so paywalls would mitigate or solve many other problems in addition to the AI-training-data one.
That's actually one of the reasons why I started https://browserbase.com/. Maintaining headless browser infrastructure can be such a pain. I've spent a lot of time managing headless chrome fleets at scale, so happy to answer any questions.
Does all my crawling, it goes very slow, it's never trigger the bot detectors.
The idea was to have a script use Selenium to launch non-headless Chrome and then wait:
I could then manually deal with logging in, answering any CAPTCHA that came up, and navigate to the page I wanted to run my automation. Then I could press "enter" in my terminal and my script would continue.That used to work fine, but then on sites using Cloudflare's CAPTCHA it stopped working. Solving the CAPTCHA would just result in another CAPTCHA.
I tried an alternative Selenium Chrome driver that was supposed to be more stealthy, and tried setting various flags that were supposed to make it so JavaScript could not tell that Selenium was there, and those worked for a while, but then they stopped working.
The results were similar using Selenium with Firefox.
I also tried Puppeteer, with Chromium and Firefox, and they too could not get past the CAPTCHA loops.
I then tried Playwright. With Chromium and Webkit that got the CAPTCHA loops. With Firefox it actually worked. I didn't even see the CAPTCHA. The non-interactive check for not being a bot passed.
Still, the whole approach seems fragile. I don't know if Firefox/Playwright working was due to some fundamental difference between Firefox and the others or just Cloudflare having not yet gotten around to dealing with it.
I've only dealt with scraping on a small scale and I quickly realized that running "browsers as a service" is a pain in the ass, they're not exactly lightweight, they like to get "stuck", balloon in memory or some such.
I imagine your business will be quite successful if reliability is good and the price is right!
https://www.youtube.com/watch?v=vs-qzlW9M50&t=726s
Trying to defend against malicious bots is tedious at best, impossible at the worst. I don't really see how lowering the bar would be a net positive. People will start using methods that will cause more collateral damage and just reduce user freedoms.
My guess is that this only increases the push towards attestation and attestation-like approaches. Login walls and PAT/Privacy Pass are just the start.
If you can't bypass bot detection now because in response they'll make the bot detection harder then there's really no winning is there?
In addition to that any access restrictions would probably end up being less precise and thus cause more collateral damage.
Which would also mean an increased amount of false negatives, leading to an increase in spammy content.
It just annoys browsers and OS with a smaller marketshare to the point that I'm wondering if it's even legal with antitrust legislation.
Devices are way too cheap for an attestation system to work to counter bots
Betting on device attestation really means betting that computing devices will become more and more expensive in the future since the device cost is the only blocker created by the attestation.
And if there's one thing I'm expecting, it's that's not going to happen, device prices will continue to decline.
Not really. If you start a new server from an IP range with known bad actors, sure. Many have tried and failed to run their own mail servers from Digitalocean, vultr, or even more dedicated-esque server hosts and pretty quickly see how much of a hassle doing so is.
But if you buy a v4 /24 from some reputable or old company and get it assigned to your own AS, you won't have negative reputation and will be fairly successful as long as you have spf/dkim/dmarc set up properly.
Doesn't that have an extremely high barrier to entry? A legal entity, significant ongoing fees, and some infrastructure?
I'm under the impression that this is entirely out of reach for anyone just trying to run an email server at home for personal use. I would love to be wrong!
until you give yourself negative reputation by running abuse bots from that range.
Normally, you can just block non-consumer ISPs, but this site offers "residential proxy" services (basically a botnet-as-a-service), which means that now consumer IP ranges need to be selectively blocked as well.
I think PAT/Privacy Pass will solve this problem as far as normal users are concerned ("normal" meaning "running Windows, macOS, iOS, or Android, on devices with hardware attestation capabilities"), but soon enough we'll arrive in an age where you can only visit so many web pages before you've exceeded your daily internet allowance.
The only solution is actual paid services, or real-user verification (and deduplication, which means little privacy, which means legal problems in much of the world) for free accounts.
If you publish something on the internet for free, you _must_ accept that people can read it automatically. You can make it difficult, make it a bit more expensive, but at the end of the day paid data means paywalls.
I agree with another comment that called this "Abuse as a Service". It seems to me this product's design goal is nothing more than to circumvent measures site owners take to prevent abuse of their site and run a sustainable business.
I've used their product many times actually, and I'm shocked on Hacker News of all places no one's thinking of anything besides abuse. How often is it useful to get information from a webpage and apply it in a new context? Then think of how often said webpage is behind a Cloudflare bot detector.
They are completely in the right to block you though, you're not the owner of that data, you might be breaking their TOS.
In Europe, if the company is actually following the law, in theory yes.
> They are completely in the right to block you though, you're not the owner of that data, you might be breaking their TOS.
IANAL, but AIUI that's definitely not true in the United States and I suspect similar ideas hold elsewhere: https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
- GDPR doesn't require it be a convenient export. Users want to paste a link on my site, a click a button, and have it magically appear. Not fill out a form, dump their entire account and sift through that.
- I never opined on the validity of blocking bots
- I never opined on if it's breaking their TOS
Abuse implies a harmfulness. Giving users a quick import option from already public data isn't harmful.
They’re not necessarily in the right to block you, if you’re the data subject or acting on their behalf.
Like if I want to programmatically unsubscribe from a subscription, why should I have to do it myself?
(and for that 1% of the cases where the address is not a spammer and user knows it, they can just hit "unsubscribe" manually)
How will well-behaved scrapers undermine the sustainability of a business? I guess adblocking is one, but we can already do that with uBlock and that's legal. Or adversarial bridging, but that only serves to boost competition.
In other words, the question is flipped; why would well-behaved (i.e. non-DDoSing) scrapers be illegal?
The question was literally,
> What are the legitimate (i.e. legal) use cases for a product such as this?
Nitter is an example of a service that explicitly disrupts Twitter/X's way to make money. If they can't make money then they can't provide the service, there would be no Twitter/X, and hence no Nitter. Of course they would try to prevent that kind of behavior and it should be obvious why. Resorting to using a service like this in order to continue using Nitter should raise some alarm bells. Sure you can still do it and rationalize it however you want, but you have to acknowledge you're trying to get the value of the service without paying for it.
Perhaps there are cases where there is a dissonance between a website's TOS and how they are blocking bot traffic? That sounds like a valid gripe. Otherwise, I don't buy the argument.
Ps: context why I need automation for such thing: those lessons are really popular and are announced at unpredictable time / there might be another spot when someone resigns
• I want to automate (or at least semi-automate) downloading bank statements. I've got ~14 accounts (checking, savings, credit card, IRA, investment, HSA) across 7 financial institutions.
It's tedious to go download statements from all of them manually.
• I want to save stories from FanFiction.net (FFN) for offline reading. FFN's terms allow automation as long as it doesn't operate faster than a human [1].
[1] From their TOS:
> You agree not to use or launch any automated system, including without limitation, "robots," "spiders," or "offline readers," that accesses the Service in a manner that sends more request messages to the FanFiction.Net servers in a given period of time than a human can reasonably produce in the same period by using a conventional on-line web browser.
Could you not shoot an email to those institutions asking for a copy of the documents?
They’ll respond within a few days, asking me to log into some web portal to prove that I am me, and then we’re back where we started
Scraping is important for example, to monitor competitors' prices to see the opportunity to raise your own prices.
And let's not forget that Google does a lot more scraping than anyone else and has ridiculous profits from it.
For years I've used my own terminal UI player (di-tui) for di.fm. At some point in the not-so-recent past, di.fm added Cloudflare's WAF, which prevents me from using one of my app's features: managing channel favorites within the app.
To be clear, I'm a paying di.fm customer, and my app only works for paying customers. But now my preferred method of listening to di.fm is slightly hamstrung because Cloudflare's WAF sits between me and little string token available to every browser that accesses di.fm (even non-paying customers).
If there was an accessible API to do what I need, I wouldn't do this because scraping sucks. I have to write 100 JavaScript edge cases to handle all the times the host's servers fail in very weird ways. Plus, walking DOMs on these shitty sites with 10,000 nested divs is not fun. GPT helps with this.
It's net-positive for the host though, as I upload a lot of valuable content that their users genuinely like, but it sucks that I have to be sneaky to get the data I need.
Data portability! Tools like this can be used to allow individuals to export their data from hostile web services trying to hold it hostage.
Legal in the EU, with GDPR.
This will get you past some very mundane bot detections, but really this is like, the very first baby step of a long rabbit hole.
The people who are taking this game seriously are 5-10 years ahead of this step. Good luck ¯\_(ツ)_/¯
There are lots of signals like timings, user tapping and scrolling behaviour, signed sessions cookies that represent browsing flows which may be legitimate or not. And that’s all assuming you’re on a good looking IP. To do this you need a large supply of residential IPs which then leads to the dodgy underworld of botnets.
I’d be surprised if this works for anything but the most basic bot protection, this is an advanced space.
If it does work for those cases, they should be either keeping it quiet and making bank, or boasting about having a secret sauce, not basic stuff like this.
Edit: for apps, Akamai provides an SDK that uses things like your motion data to create a signature that suggests that you're a real user. This signature is either injected into API requests or into a webview session. I'm sure it's crackable if you dedicate significant reverse engineering resources to it, but then you've got to crack every version, crack every other implementation from other companies, etc. Non-starter.
A device that stays rock-steady throughout an entire browsing session isn't necessarily suspicious on its own—for example, you could have your phone laid flat on the table while your browse with your pointer finger—but it can be a useful tell in combination with other suspicious factors.
For web-only, I believe they have a JS only bundle that your site can include which I would imagine does different things, but which would also bring a higher risk profile associated with it. Sites use these risk profiles to determine things like whether to offer specific services, whether to ask for more authentication, etc.
That said, since I wrote that comment, I found out that on Android, both Firefox and Chrome grant access to gyroscope data without a permission dialog, which is extremely surprising. I don't have an iOS device to verify, but apparently Safari gates the API behind a permission dialog.
Why do we even need an actual device? We can emulate if we even need to and set our headers to look like we're coming from a device browser.
> Why do we even need an actual device? We can emulate if we even need to and set our headers to look like we're coming from a device browser.
This one is much harder, your browser, OS, and hardware leave a uniquely identifiable fingerprint (with Javascript enabled). A website can render some graphical pattern on a <canvas> or audio in an audio context, and the resulting output will have minute differences that originate from your rendering and audio pipelines.
Check out: https://amiunique.org/fingerprint https://browserleaks.com/ https://fingerprint.com/ https://coveryourtracks.eff.org/
You can try to fake these, but it all depends on the sophistication of the target website. You can quickly end up in really deep rabbit holes: https://www.nullpt.rs/devirtualizing-nike-vm-1
Humans move their devices even when typing.
https://sensor-js.xyz/demo.html
I think (though am uncertain) that it’s similar for App Store apps too.
My assumption about an SDK like that is that it's getting that access through some assumed mean(s) that the user just clicks past without thinking.
Source: Founder@browserless.io
Let's say I have a few 100 Gbps connections that is mostly idle, is it fine if I direct them at browserless? No? Exactly, that kind of traffic is not wanted.
You can't even block the amount of subnets that's coming for you in a DDoS attack, thinking a human is able to keep up with something like this is pretty blindsighted and naive. The differentiation of network protocols and relay attacks alone is way too slow to be mitigated in most systems.
We also need to detect and react to UI layout changes, and the headless browser is the only real way to do that.
If smaller indie hackers are going to build useful competitors, they’re going to need to scrape.
I understand that a lot of scraping is from spammers, but not all.
Any useful competitor will have to scale well beyond the point of indie hacker, where someone is (eventually) getting rich. That doesn't make for a great argument for bypassing consent.
Edit: As mmcclure points out Kagi actually doesn't index much. So that's a bad example that might invalidate my point.
> Our data includes anonymized API calls to traditional search indexes like Google, Yandex, Mojeek and Brave, specialized search engines like Marginalia, and sources of vertical information like Apple, Wikipedia, Open Meteo, Yelp, TripAdvisor and other APIs. Typically every search query on Kagi will call a number of different sources at the same time, all with the purpose of bringing the best possible search results to the user.
source: https://help.kagi.com/kagi/search-details/search-sources.htm...
Feel free to shoot me an email if you're interested in trying it out! paul@browserbase.com
Which we're talking about running a real browser downloading all the bloat of a target website, 25GB doesn't go very far. Is this just absurdly expensive?
https://bot.sannysoft.com/
https://antoinevastel.com/bots/datadome
The first link shows the tests and is the most educational.
Does anyone know of more recent tools that show why bot detection was triggered? Versus just testing against Cloudflare bot detection and getting pass/fail.
https://www.browserscan.net/en/bot-detection
https://bot.incolumitas.com/
For example, Zillow goes to great lengths to restrict bots, but I want to get updates if a property changes somehow on Zillow. It's publicly accessible data if you're not using a bot. "IFTTT" is just an example of a service that automates these things. I can write my own automation service, but if I'm using a bot, Zillow will block me (I've actually done this before).
I'm not saying I like this, I'm just saying I'm surprised it hasn't happened yet.
What we really need is a browser that absolutely cannot access sites "protected" by Cloudflare, and this browser needs to become standard. Sites that put themselves behind Cloudflare need to pay a price for their hostility, and that price is the loss of our business.
If I want to program a bot to get around the horrendous lossage of the terrible UI and multi-megabyte bloat you have created by farming out your web server programming to the lowest bidder, I have the right to do so. In turn my bot will respect robots.txt and will throttle itself appropriately.
This is the deal you signed up for when you built a web server. If you don't like it, GTF off the web. And take your attestation and WAF with you.
"we just provide the tool" defence no longer works in court
like there's even a 7 day trial opening this up to anyone with bad intentions