It's wild to me that PoW systems are how we sort the bot problem... the bummer is that all this "work" is just wasted cycles, at least in crypto there is a token you can sell.
Scraping would be costly in this world: scrapers would have to spend tokens in order to get the webpage.
But, in this world, the website owner would receive tokens that they can then use to do whatever they want, including paying for servers and bandwidth. This is the sense in which the cycles aren’t wasted: the website owner now has cash to spend.
Effectively, both scrapers and ordinary users would be paying for the privilege of getting website bytes.
This also solves the problem of having to wait for your phone to solve the challenge while you’re browsing: you can buy or mine some tokens ahead of time and pay them as soon as challenged. So can the scrapers, but because they’re accessing enormous numbers of pages it’s hopefully prohibitively expensive for them.
It would be cool if the POW could be done ahead of time. That way I don't get stuck waiting while I'm working. Some type of credit/tokens that my browser could then spend.
Then we could implement an exchange, so that if you generate too much Anubis POW, you could exchange it with others.
We could link it to a site, you generate for HN, I for Reddit, but it so happens that you visited Reddit more and I HN, so we depleted our Anubis POW, so we could exchange some Reddit Anubis with some HN Anubis.
Yes, I also strongly suspect that this is only going to move more parts of scrapers onto consumer devices. The egress proxies are already there, why not use a little bit of the compute as well?
The malware already results in noticeable slowdown just running the scraper. Solving anubis challenges is significantly more difficult than simply requesting the page, so it is not feasible on-device without making the device totally unusable.
The goal isn't to eliminate scrapers, its to prevent a distributed scraping network from requesting 10000 pages a second each from 10000 different websites.
It's still very unclear to me how Anubis is supposed to work in the long term or even the medium term. The assumption is that scrappers, a.k.a. AI companies, a.k.a. those who hog all the RAM, are short of memory...?
The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
>The assumption is that requiring 1 second of compute before allowing a page load will have more impact on the people requesting 1000 pages/sec than it will on consumers requesting 1 page every minute.
Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages.
There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.
> requiring 1 second of compute before allowing a page load
After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves
Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing
The issue is that scrapers are rotating their IP and essentially performing a DDoS attack. Anubis is part of a defense-in-depth solution. If scrapers reuse a cookie, traditional anti-abuse mechanisms will work.
I don't know what you mean by traditional, but switching IP addresses rapidly is not normal user behavior so this would be trivial to filter out for something like Anubis and require another solve..
I would disagree with the Anubis effectiveness falling off, but the llm bros are throwing life itself into training to defeat any “perceived” challenge.
There was a lot of discussion (715 comments) about this question a couple days back, if anyone missed it. It was the thread about kernel.org sharing their anubis stats.
> In my experience the kinds of people who run this exact combination of circumstances also tend to be the kind of people that have a wide variance in the level of kindness they display to the authors of open source programs that happen to be in their way.
Love this. There’s been past discussion on HN re how OSS maintainers are treated, and this is such a wry sentence. I really appreciate the tone / attitude to the problem.
Well a little translation would be that some are very kind, some are total assholes, and everything in between. But your reading is probably more accurate.
I think the literal interpretation is correct here. It's a way of saying "you, dear reader who runs this combination of things, might be very kind, but some of your compatriots will be unusually unkind and will turn up in my issue tracker and on my socials to complain".
Anubis is fine if you want to block bots that are bothering you, but I never understood why they don't have robots.txt at the root level too. I happily obey that for my private crawler. I suppose you don't want to duplicate implementations. Fair enough. I guess I should add an Anubis detector so I can just blacklist because that person clearly doesn't want bots.
and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.
Is there a place where I can try out if my browser is compatible? Easier to find out now than when I'm trying to get work done and a million websites now have it deployed
On https://wasm-feature-detect.surma.technology it shows that I don't have 3 of all these features but I'm not sure if Anubis needs any of them to not kick me back to the pure JS solution
Which would apparently be bad because
> The WebAssembly that's shipped with this flow is ridiculously performant. This may mean you need to adjust the difficulty [upward, to avoid that bots solve it trivially when they support wasm, I assume this means]
combined with
> The wasm2js flow doesn't currently have a way to update the progress bar [so you have no clue about remaining time]
(wouldn't be the first time that I gave up on a page because it was stuck on 0 hashes per second)
If you don't disable browser features for fun and/or sport, Chrome as of version 120 or later (or another browser of that vintage) will use the optimized flow.
Bit sad that the default assumption is that everyone should use Google Chrome, even when explicitly asking for a compatibility tester. I'm asking because I don't use Google's browser
Hats off to Xe for spending so much time on backwards compatibility, especially the tidbit about targeting Chrome 66.
I have a Mac from 2014 running Yosemite that I occasionally use to test for backwards compatibility in my own frontend code (for fun!). But IMO the best way to ensure compatibility is to use period-correct toolchains or toolchains where the pace of change is slower, like ClojureScript.
The interesting part is that bots mitigation is actually an economic problem. You don't need to make scraping impossible, you only need to just make it expensive enough so that abusive traffic stops being worth it.
The git host example is probably the one which Anubis is the dumbest defense for: the main reason to have those links is for easy machine interaction. So that's dead when you implement it.
So why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.
Not to mention it is the least efficient representation of git commits. It seems a lot of people are unaware that git generates the diff representation you see with "git show" on the fly -- internally, commits are Merkle trees representing a snapshot of the repo file contents.
As you said, if you actually want to interact with a remote git repo you'd do so with packfiles not diffs.
That's exactly what's happening to the Linux git repository. The server constantly has mutltiple CPU cores dedicated to rendering git of various old, dead branches in html for scrapers.
I think the Rust feature you’re looking for regarding recompiling the standard library is called “build-std”, that should be enough for you to search for it. (For similar reasons you also need that flag if you are trying to use Rust to build multithreaded wasm binaries, so it might come up for you!)
Great article, WebAssembly is such a good technology when you have use case for it.
I've just finished creating a WASM 1.0 implementation (simple JIT on x86 and x86_64) for plugin usage and I'm looking on how to support more languages beyond C (for which I have "micro-libc" to create really small binaries). In particular C++ and Rust so the article contains useful info for that. C++ that would be more problematic I think, would have to resort to full Emscripten or something... I've tried to compile standalone C++ stdlib but wasn't much sucessful yet.
BTW, I see in the screenshots you also test Firefox, I think it would make sense to test the latest two ESR versions (these are often used by Linux distributions).
> It also means that the "hey Claude vibeslop me a CUDA Anubis solver" route is on its way to being fundamentally dead.
I assume that the previous challenges will still be available for circumstances where WASM is not available (unless we are just working on the basis that it is available if JS is?) or perhaps as a fallback. TFA explicitly mentions “smart” TVs, and I'm sure there are plenty of those out there that run browsers old enough to not know what WASM is. TBH these would not be a priority for me, but they obviously are for this project given they (well, TVs generally, not specifically elderly ones) get specific mention in this write-up.
If so, what is to stop scrapers from just forcing the fallback POW option instead of trying to run the new memory-hard algorithm at all?
Ah I must have skipped by that bit. And presumably the memory-hard algorithm is CPU-easy enough that this won't be a problem. That leaves me wondering "why not just use a JS memory-hard routine in the first place, but maybe I've shouted over that too. I'll give TFA a reread later.
I think it is still compute-heavy if not compute-hard, and stuff like WASM SIMD makes it less compute heavy (not really available in JS). And also, for memory bound problems, you probably don't want the JS GC needing to clean up afterwards when WASM gives you that control.
I assume that SIMD acceleration is usable with WebCrypto, but I have no evidence either way. It's also not part of WebAssembly so there's that to consider too lol
I have every copy of Firefox here configured with webassembly disabled...because I don't tend to do what Webassembly was designed for, i.e. online games, video/audio editing, emulation, etc. [1] and because I dislike things running in the background without my knowledge.
So this is going to be interesting.
My plea (in a small voice) to the OP: just as many sites still do for JavaScript, please add a "This captcha required Webassembly to continue" message to your code when Webassembly is disabled. And, incidentally, for smaller platforms/browsers that don't offer a Webassembly engine at all.
You should really be running the most popular browser in its bone stock configuration. If you want to "customize" your web experience, you shouldn't be surprised when things don't work right.
Many website owners who are looking to block bots don’t really mind also blocking people with unconventional configurations if that means the bots are more likely to be blocked.
From a statistical standpoint I imagine that traffic that can’t run JavaScript and can’t run WebAssembly is way more likely to be a bot than a human, maybe by a ratio of 10:1 if I was to guess.
Obviously, as a user, do what you want. Doesn’t bother me if someone else disables a bunch of stuff on your browser. For me, I think at some point I accept the modern web for what it is, with the exception of blocking ads. The web without JavaScript isn’t the web at all just like MLB baseball without the pitch clock isn’t MLB baseball.
> You should really be running the most popular browser
This amounts to giving Google effectively complete control over the web. I'm not sure that's a good idea.
I'm somewhat sympathetic to "If you want to 'customize' your web experience, you shouldn't be surprised when things don't work right." in terms of turning off commonly-used features, but the request isn't to make it work right. OP just wants a helpful error message.
The second half of this post chronicles the author's Herculean efforts to make everything work well in the cases you describe.
> One of the big things that blocked this shipping for so long was not having an escape hatch of some kind to allow clients that disable WebAssembly by policy to get through the gate. [...] This combination of factors means that there would need to be another implementation of the proof of work code in JavaScript that would actually execute the number crunching.
Thanks. You're right; it was a very thorough article and I trailed off about halfway through. I still say there are many improper uses of Webassembly out there, but I credit the developer for a highly thoughtful approach to the project. I really wonder if Cloudflare et al. does as much due diligence?
I really don't see why you'd disable WebAssembly and allow JavaScript instead. Expecting developers to make a fallback for a baseline-supported feature for any reason but backwards compatibility with runtimes that do not so support WASM is extremely silly.
I also have JS disabled in Firefox on my phone. It's on by default on my laptop, as I apparently visit different sites when my butt is planted in a chair.
My wish is actually for the days of NPAPI where Webassembly would be just another plugin (like Flash) and browsers clung to mainly HTML, JS, and CSS. Then developers wouldn't be quite so keen to rely on it and the web would be much less bloated. Yes, those days are long gone... As a tradeoff, I'd settle for a simple toolbar icon that lit up when Webassembly is running, and/or a high level UI (just like Location, Camera, and Microphone) so I can enable it for just those sites that need it. I tried selling Mozilla on the idea once but they apparently weren't interested.
Baselines aren't entirely universal. I remember the days when the Mozilla (or early Firefox) browser was the only viable web client for Linux, but some sites still opted to run behind ActiveX. Today I don't think you'll find a Haiku or Amiga browser with Webassembly support, nor Firefox for Linux on IBM Power motherboards. It's easy to forget that just because most any 'nix or BSD OS has a plethora of choices, that others aren't still dealing with the "ActiveX" situation and feeling like everybody's out to get them.
Two different kinds of "background". GP is talking about what indications they get a page is running arbitrary code as they load it, (some of) web workers is about persisting running that code after it's loaded. It can be in the "background" for both or neither, depending on the context, and both meanings are valid.
> GP is talking about what indications they get a page is running arbitrary code as they load it
I mean, you get the same indicators as when the page runs arbitrary Javascript on load? WASM assembly may be (slightly) harder to read than typescript compiled to JS, but the source is all there, it calls the same APIs, etc.
Correct, but that's not related to their complaint. They just want a fallback warning message to display when WASM is disabled like it used to when JS was disabled.
Right, but the base assumption here is unfounded. Disabling WASM isn't in any way equivalent to disabling Javascript - you are just kneecapping your own browser performance, rather than closing off a large attack surface.
It's amusing (although disheartening to me as someone who builds online games) that wasm is now being treated like a flash plugin by sysadmins. You're not entirely wrong, but it's just funny to me that we took a full ten years to get back to roughly the same place. Now with Ruffle running Flash in wasm, no less! We build shit that doesn't work in the DOM, search engines hate it, walled gardens try to block it, admins fear it, they shut it down and the whole thing starts over.
I haven't disabled web workers, but am playing with the concept. T-Mobile's payment portal screams and then bans my IP (something like that) if try. I have disabled service workers globally, and so far no fallout.
The anubis docs website is used as an example of how anubis works.
Also, if you are not using a big CDN (e.g. Cloudflare), you still need to pay for the traffic (even if you don't pay as much for the CPU). This is doubly true for smaller CDNs like Bunny.
I'm not a fan of Anubis for various reasons but the idea that bot traffic in only harmful with dynamic pages must die. CPU (yes, even to serve static pages) is not free, bandwidth is definitely not free. There's an idea that serving a static page to a bot has a marginal cost tending to zero, but it's never really zero and serving them by millions definitely has a cost.
Also, while some pages may look like static HTML pages, they may be generated on the fly by an expensive and/or slow backend, which adds to the cost. I happen to maintain servers for academics and some content management systems are slow and have an expensive CPU cost. While it's OK for the low number of humans interested in the subjects they deal with, it's definitely not fit for massive bot scrapping. And before you ask, no, it's not always practical to have cache upfront or to pre-generate all pages.
There are numerous services that will let you host static pages for free or nearly free. There are also numerous services that sit in front of your website that can block bots and reduce load on your origin server, many of which are also free, or very low cost relative to the service they provide.
The situation you are in is far less dire sounding when you consider that you have these options available to you.
So your employer is having a problem, and prevents you from using any of the available options to solve it? And you have asked them about all of them/told them about the problem? They don't like saving money?
Anubis is not free either, it is a matter of how much it costs to run Anubis vs to let bots in.
When I see LKML using Anubis when the pages it serves are tens of kB, all presumably static, I wonder if they do it out of spite rather than to protect their servers.
As a user/reader/viewer I absolutely hate Anubis and usually turn around when I see it pop up (at least on my phone where it takes ages to compute), but with stats like that, I get why a site operator would resort to using it.
> But no, let's in fact choose the stupidest possible way of doing it — by rendering everything as HTML commit by commit and then parsing it.
This drives me crazy with so-called SOTA LLMs that have "achieved AGI".
Fable, Sol, Astra, will start by trying to reverse engineer a binary to figure out how something works when software is open source and one search query away.
You let them know it's open source, and they will start using github API instead of just cloning and grepping.
I think the kernel.org post proves the parent point rather than contradicts it.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
14 CPU cores total for running a website like kernel.org is laughable. This is not worth deploying Anubis on client's devices, this is not worth the time of the engineer who worked on it. Provisioning more hardware would have been better for everyone.
Shitty scrapers hitting images over and over again is also a problem. Not just /pic/1.webp - /pic/2.webp, but in practice I've seen three AI scrapers that would request things like /pic/1.webp?version=webp&foo=bar and other random query parameters that none of my domains even use. It's just stuck brute-forcing files or something, sucking up loads of bandwidth downloading the same image for no reason.
It also depends on if you want your content to be scraped. Any bot ignoring robots.txt should be considered potentially malicious and worth considering such mechanisms for.
> Shitty scrapers hitting images over and over again is also a problem. Not just /pic/1.webp - /pic/2.webp, but in practice I've seen three AI scrapers that would request things like /pic/1.webp?version=webp&foo=bar and other random query parameters that none of my domains even use
That's a trivially solved problem - have the web server refuse (or redirect if you're feeling nice) requests with unexpected query parameters. No reason to make things worse for real users before optimizing your site.
And even if there are actual parametric resources you want to serve, apply limits only to them and let users visit the rest without unnecessary bot checks.
Sure, there are loads of easy solutions to this problem. Putting everything behind Cloudflare is the most common one, Anubis and its competitors are another. I could also block all data centers, countries with loads of CGNAT + botnets like Brazil, China, and probably India, and refuse any user agent not in the top 5 modern user agents with a little TLS fingerprinting to make sure.
I could probably also analyze all the paths across my servers and write some bespoke Nginx config files to filter out all requests with illegal queries, but that's harder, more fragile, and just leaves the servers open for the next time a scraper finds a new way to screw me over.
The bots I'm fighting have gone for the nuclear option, relying on botnets and spoofing to bypass basic filtering, so I don't see why I shouldn't do the same.
I only hear bad things about bots lately, are there any benevolent bots?
I've done a tiny bit of scraping myself (usually to archive old websites before they die), and I've gotten banned a few times even though I wasn't hammering anything.
Looks like it's getting harder and harder to do scraping for legitimate purposes. Although I suppose legitimate is subjective :) Everything becomes valuable when it's old enough...
Others have answered your question. A lot of the sites listed here https://anubis.techaro.lol/docs/user/known-instances/ have dynamic content; Git web interfaces in particular (Codeberg, the Linux kernel, FFMPEG, and more are on the list) are vulnerable to poorly or maliciously configured scrapers.
153 comments
[ 1.9 ms ] story [ 8.5 ms ] threadThat would defeat the purpose. The goal is to make scraping costly, not profitable.
But, in this world, the website owner would receive tokens that they can then use to do whatever they want, including paying for servers and bandwidth. This is the sense in which the cycles aren’t wasted: the website owner now has cash to spend.
Effectively, both scrapers and ordinary users would be paying for the privilege of getting website bytes.
This also solves the problem of having to wait for your phone to solve the challenge while you’re browsing: you can buy or mine some tokens ahead of time and pay them as soon as challenged. So can the scrapers, but because they’re accessing enormous numbers of pages it’s hopefully prohibitively expensive for them.
https://github.com/eternal-flame-AD/pow-buster - Browser extension that already used WASM to accelerate the anubis solver, among others.
https://github.com/kasper93/anubis_webgpu - Browser extension that uses webgpu, for another order-of-magnitude speedup (depending on your GPU of course).
We could link it to a site, you generate for HN, I for Reddit, but it so happens that you visited Reddit more and I HN, so we depleted our Anubis POW, so we could exchange some Reddit Anubis with some HN Anubis.
I feel like Anubis is ironically speedrunning a lot of discoveries the crypto folks have already made several years ago...
Lmao yeah no. I don't think a little argon2 is going to change shit all.
I mean the thesis of Anubis itself is "scrappers are compute limited (in ways that consumer devices are not)" which has its own massive flaws.
Personally I don’t understand where this confusion comes from - it’s a simple economic tactic. Yes the large scale scrapers won’t run out of compute but putting anubis in front of a website demonstrably reduces the amount of crawling and that is the entire goal.
Which doesn't hold. A 1s delay, despite being inperceptible is still a cost to human users. After all, 3600 x 1s delay adds up to an hour. Worse yet, the typical human visitor will always have to solve a challenge, because chances are it's the first time they're visiting your blog or whatever. Someone scraping reuse a valid session for at least a few more pages.
There's also the problem of time. It might be tempting to think the cost of a challenge is 1M cpu cycles or whatever, but the cost is better modeled as two parts, a time cost (for the human) and a cpu cost (for the computer). A scraper might have to pay the cpu cost, but not the time cost, because they can have their scraper solve challenges in their sleep. Unless the user is utterly destitute, their time is worth far more than whatever the compute cost is.
After 1 solve, you get a cookie and can load tons of pages with it. Scrapers and user agents alike will need to spend the compute only once to get a valid session for themselves
Seems to be working so far though. Fwiw, lots of silly things stop bots until someone puts their mind (or tokens) to bypassing it, and then the blocking mechanism has to adapt. We'll see how it plays out, for now it's a lot better than begging big brother (cloudflare, google, or amazon captcha) for access, so I'm quite appreciative of what Techaro is doing
https://news.ycombinator.com/item?id=49491791 ("Creepy Crawlies (kernel.org)")
https://people.kernel.org/monsieuricon/creepy-crawlies
I was particularly struck by the ASIC estimates 'semiquaver shared.
Love this. There’s been past discussion on HN re how OSS maintainers are treated, and this is such a wry sentence. I really appreciate the tone / attitude to the problem.
I'd say "wide variance in the level of kindness" is a rather British way to describe some of the opinions that have been voiced.
- https://techaro.lol/robots.txt
- https://anubis.techaro.lol/robots.txt
and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.
On https://wasm-feature-detect.surma.technology it shows that I don't have 3 of all these features but I'm not sure if Anubis needs any of them to not kick me back to the pure JS solution
Which would apparently be bad because
> The WebAssembly that's shipped with this flow is ridiculously performant. This may mean you need to adjust the difficulty [upward, to avoid that bots solve it trivially when they support wasm, I assume this means]
combined with
> The wasm2js flow doesn't currently have a way to update the progress bar [so you have no clue about remaining time]
(wouldn't be the first time that I gave up on a page because it was stuck on 0 hashes per second)
…<img not found/>…
I don’t know what this picture was supposed to be, but a 404 demonstrates your point perfectly well.
I have a Mac from 2014 running Yosemite that I occasionally use to test for backwards compatibility in my own frontend code (for fun!). But IMO the best way to ensure compatibility is to use period-correct toolchains or toolchains where the pace of change is slower, like ClojureScript.
I don't understand the units here, wouldn't a bit versus nibble difference make the multiplier 16, instead of 1024?
The git host example is probably the one which Anubis is the dumbest defense for: the main reason to have those links is for easy machine interaction. So that's dead when you implement it.
So why even host the links? It's a git repo: send them a local got client and let them clone the repo in browser or something.
As you said, if you actually want to interact with a remote git repo you'd do so with packfiles not diffs.
I document my code, in issue and elsewhere on the Web, by referring to specific commits so having a URL for each is useful to me at least.
I've just finished creating a WASM 1.0 implementation (simple JIT on x86 and x86_64) for plugin usage and I'm looking on how to support more languages beyond C (for which I have "micro-libc" to create really small binaries). In particular C++ and Rust so the article contains useful info for that. C++ that would be more problematic I think, would have to resort to full Emscripten or something... I've tried to compile standalone C++ stdlib but wasn't much sucessful yet.
BTW, I see in the screenshots you also test Firefox, I think it would make sense to test the latest two ESR versions (these are often used by Linux distributions).
I assume that the previous challenges will still be available for circumstances where WASM is not available (unless we are just working on the basis that it is available if JS is?) or perhaps as a fallback. TFA explicitly mentions “smart” TVs, and I'm sure there are plenty of those out there that run browsers old enough to not know what WASM is. TBH these would not be a priority for me, but they obviously are for this project given they (well, TVs generally, not specifically elderly ones) get specific mention in this write-up.
If so, what is to stop scrapers from just forcing the fallback POW option instead of trying to run the new memory-hard algorithm at all?
So this is going to be interesting.
My plea (in a small voice) to the OP: just as many sites still do for JavaScript, please add a "This captcha required Webassembly to continue" message to your code when Webassembly is disabled. And, incidentally, for smaller platforms/browsers that don't offer a Webassembly engine at all.
[1] https://webassembly.org/docs/use-cases/
From a statistical standpoint I imagine that traffic that can’t run JavaScript and can’t run WebAssembly is way more likely to be a bot than a human, maybe by a ratio of 10:1 if I was to guess.
Obviously, as a user, do what you want. Doesn’t bother me if someone else disables a bunch of stuff on your browser. For me, I think at some point I accept the modern web for what it is, with the exception of blocking ads. The web without JavaScript isn’t the web at all just like MLB baseball without the pitch clock isn’t MLB baseball.
This amounts to giving Google effectively complete control over the web. I'm not sure that's a good idea.
I'm somewhat sympathetic to "If you want to 'customize' your web experience, you shouldn't be surprised when things don't work right." in terms of turning off commonly-used features, but the request isn't to make it work right. OP just wants a helpful error message.
> One of the big things that blocked this shipping for so long was not having an escape hatch of some kind to allow clients that disable WebAssembly by policy to get through the gate. [...] This combination of factors means that there would need to be another implementation of the proof of work code in JavaScript that would actually execute the number crunching.
My wish is actually for the days of NPAPI where Webassembly would be just another plugin (like Flash) and browsers clung to mainly HTML, JS, and CSS. Then developers wouldn't be quite so keen to rely on it and the web would be much less bloated. Yes, those days are long gone... As a tradeoff, I'd settle for a simple toolbar icon that lit up when Webassembly is running, and/or a high level UI (just like Location, Camera, and Microphone) so I can enable it for just those sites that need it. I tried selling Mozilla on the idea once but they apparently weren't interested.
Baselines aren't entirely universal. I remember the days when the Mozilla (or early Firefox) browser was the only viable web client for Linux, but some sites still opted to run behind ActiveX. Today I don't think you'll find a Haiku or Amiga browser with Webassembly support, nor Firefox for Linux on IBM Power motherboards. It's easy to forget that just because most any 'nix or BSD OS has a plethora of choices, that others aren't still dealing with the "ActiveX" situation and feeling like everybody's out to get them.
You do see how selfish it is, right? While the author did introduce such a solution, it's such a massive waste of time.
Point of order: WASM doesn't let things run in the background. To do that you need WebWorkers (a javascript feature, not WASM-specific).
I mean, you get the same indicators as when the page runs arbitrary Javascript on load? WASM assembly may be (slightly) harder to read than typescript compiled to JS, but the source is all there, it calls the same APIs, etc.
I forgot to do so on my phone. I noticed Firefox had slowed to a friggeb crawl when opening pages. Disabling web workers fixed that.
Also, if you are not using a big CDN (e.g. Cloudflare), you still need to pay for the traffic (even if you don't pay as much for the CPU). This is doubly true for smaller CDNs like Bunny.
Also, while some pages may look like static HTML pages, they may be generated on the fly by an expensive and/or slow backend, which adds to the cost. I happen to maintain servers for academics and some content management systems are slow and have an expensive CPU cost. While it's OK for the low number of humans interested in the subjects they deal with, it's definitely not fit for massive bot scrapping. And before you ask, no, it's not always practical to have cache upfront or to pre-generate all pages.
The situation you are in is far less dire sounding when you consider that you have these options available to you.
Well, sounds like it's not your problem then.
When I see LKML using Anubis when the pages it serves are tens of kB, all presumably static, I wonder if they do it out of spite rather than to protect their servers.
As a user/reader/viewer I absolutely hate Anubis and usually turn around when I see it pop up (at least on my phone where it takes ages to compute), but with stats like that, I get why a site operator would resort to using it.
This drives me crazy with so-called SOTA LLMs that have "achieved AGI".
Fable, Sol, Astra, will start by trying to reverse engineer a binary to figure out how something works when software is open source and one search query away.
You let them know it's open source, and they will start using github API instead of just cloning and grepping.
> At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing but rendering git commits as html.
14 CPU cores total for running a website like kernel.org is laughable. This is not worth deploying Anubis on client's devices, this is not worth the time of the engineer who worked on it. Provisioning more hardware would have been better for everyone.
Yes and that should be fixed before you subject real users to resource-wasting scripts.
And to answer your intended demand: this is in some cases impossible or unreasonable. And things were working fine before LLM DDoS.
It also depends on if you want your content to be scraped. Any bot ignoring robots.txt should be considered potentially malicious and worth considering such mechanisms for.
That's a trivially solved problem - have the web server refuse (or redirect if you're feeling nice) requests with unexpected query parameters. No reason to make things worse for real users before optimizing your site.
And even if there are actual parametric resources you want to serve, apply limits only to them and let users visit the rest without unnecessary bot checks.
I could probably also analyze all the paths across my servers and write some bespoke Nginx config files to filter out all requests with illegal queries, but that's harder, more fragile, and just leaves the servers open for the next time a scraper finds a new way to screw me over.
The bots I'm fighting have gone for the nuclear option, relying on botnets and spoofing to bypass basic filtering, so I don't see why I shouldn't do the same.
I only hear bad things about bots lately, are there any benevolent bots?
I've done a tiny bit of scraping myself (usually to archive old websites before they die), and I've gotten banned a few times even though I wasn't hammering anything.
Looks like it's getting harder and harder to do scraping for legitimate purposes. Although I suppose legitimate is subjective :) Everything becomes valuable when it's old enough...