Since the Firefox installation is an .xpi file, I assume it's also blocked by Mozilla? Not saying I'm surprised, though, giving the nature of the extension.
Not sure, perhaps the author of that one just hasn't tried to get it approved. There's also https://addons.mozilla.org/en-US/firefox/addon/bypass-paywal... and a number of other ones for Firefox that don't look to be blocked, so I don't think Mozilla is taking the same stance.
For just flipping the switch, no I don't think there's any implication.
For installing an unpacked extension:
- Obviously you don't have the benefit of the Chrome store checking for abuse.
- You'll need to read the manifest.json file yourself to see what permissions you're granting, because the warning popup doesn't show up when installing this way.
You will bypass the permission auth dialog, which is your last official chance to see what the code you're about to execute has access to.
The extension developer could add a malicious permission + new code to exploit it and it would look the same as using developer mode to add a Hello World extension
They suffer from the same problem. They can't update the extension because google changed the policies. Now any attempt to remove paywalls is not allowed
Just a recommendation: the site calls itself "Extensions Hub" despite the domain being "extensionhub.site". This is confusing and makes the site less trustworthy. Additionally "extensionhub.site" doesn't sound super trustworthy to me to begin with (maybe because the TLD is rare and "extensionhub" sounds a lot bigger than how it presents itself).
If the idea is to host other extensions on it as well, I'd suggest putting a little more effort into it so it feels like something that actually has extensions on it rather than blog posts. The page for the extension itself for example has none of the usual links or info about the author of the extension (the date and author feel like they're part of a blog post ABOUT the extension, not info about the extension itself because of the layout) and the actual link for the extension itself is a direct download link in the prose of the article itself.
This may be because I'm European but the complete lack of info about who operates the site (no privacy policy, just a twitter link and a copyright statement linking back to the site itself) screams "scam" to me on top of the impression that this is a blog trying to present itself as an extension store.
Please take this as constructive feedback but if I saw this site in the wild I'd assume it's malware.
Great name, TLD has zero impact from my perspective. I wouldn't fret about packaging - leave that to those who buy overpriced lattes because of the branding.
Let me just say -- and I'll get voted down for this -- that I think breaching paywalls is generally immoral. While I can imagine some moral reasons, I think most people are doing it even though they can pay. They are taking someone's work and using it without permission while refusing to pay.
Of course I may be biased because I make some of my living creating content behind paywalls. If free access and ad support were possible, I would choose them first. But they aren't. To me, this extension and the people who support it aren't any different from those who go into restaurants, order food, eat it and then leave without paying. And there will be those who trot out some extreme rhetoric about world hunger or something to justify their actions.
Google is blocking this extension because it's bad for the web as a whole. Don't empower people who destroy the information sources and turn the web into a fact desert. Support those who create knowledge and share it fairly and equally, albeit at a price.
The issue with paywalls is the bait and switch - you push the whole content for SEO then force users to pay money.
Nobody cares if you want to sell the secrets of the universe for $5000/mo. We care about the bait and switch.
By setting up this bait and switch, you hurt your own paywall - if the search engine can get a full copy, so can I. Want these extensions to stop working? Stop the bait and switch.
"FREE VACATIONS! (you have to sit through a timeshare presentation and there are catches"
We widely regard the above as unethical.
Adding content to a search engine literally says "you can come read this content!". That is the purpose of search engines. Even Google penalizes paywall behavior and will downrank them - which forces the paywall people to get more clever.
Sorry but don't abuse search engines and users to sell your content. Buy ads like a big boy.
I wouldn't say it is necessarily immature to use a paywall as opposed to ads. I for one sometimes prefer paying for a site just so i dont have to see ads.
Though I do agree, the bait and switch aspect of finessing seo and such leaves a bad taste in your mouth. Glad google penalizes such behavior
They are being gamed by their own game. Nobody took their content - they submitted it to search engines and are upset the search engine provides a cached copy.
If they don't like it they can not submit to search engines.
> Just because something's available on request from a server does not mean it's up for grabs.
When you give the content out for free to the indexer, you have given a copy to the world. Google search is not your advertising machine - you can pay for that priviledge if you would care for it.
Publishers are subverting users' reasonable expectations that the information they search and receive hits for on Google is publicly accessible—why else would it appear in search?
It's actually everyone's business because this creates a web that is less usable for everyone. If publishers were willing to commit and make their paid content server-side for customers only, they would have a stronger case against infringers.
Since when are Google results 100% clearnet? This has never been the case, ever. Search "login" in Google and you'll get a lot of content that is not publicly accessible.
They provide users with an expectation of what information they will receive if they pay for the content.
The fact that this "glimpse" of the content pollutes your web searches is a search engine problem.
It would be trivial to filter sites with paywalled content. But Google refuses to let you do that. Hope someone else will come along and help with that.
The content is available to search engines, and they are highly optimized for SEO, so they show up high in search results. So users expect by clicking the link in the search result they can see the rest of the content. Then you get hit with a paywall.
The news sites return different content to search engines than they do to browsers. Which means you get a different site than what is shown in the search preview, specifically a paywall page.
I would be a lot less annoyed with these paywalls if they didn't rank so highly on search results, or at the very least indicated they were paywalled on the search page and were easy to filter out.
> By setting up this bait and switch, you hurt your own paywall - if the search engine can get a full copy, so can I.
I don’t get this. They do whatever they want with their content. Editors give free book copies to journalists; that doesn’t gives you the right to have one as well.
...and if they post their content to the public, the public can read it. Giving content to journalists is not the same as giving your content to Google to index in their search engine and caches. The later is not private.
Journalists post quotes and snippets, just as Google does. Not that it matters. Just because I gave something to someone else for free, doesn't mean I have to give it to you.
The idea that a single public exhibition of a work is enough to invalidate any future sale of that work is totally silly. Are you gonna bust out the window of a Barnes & Noble because libraries exist?
> Giving content to journalists is not the same as giving your content to Google to index in their search engine and caches.
It’s exactly the same. You give access to an entity that serves to promote your content. The fact that Google’s cache is accessible is a technical implementation detail; Google could remove access to that cache tomorrow and that wouldn’t change anything.
> The fact that Google’s cache is accessible is a technical implementation detail;
It's intentional to fix the kind of abusive behavior you are engaging in. If you serve content to a search engine, that search engine will reproduce the content.
Search engines are for publically available content.
Publishers give books to journalists with the expectation that the journalists will immediately put an electronic copy of the book in a publicly-available CDN and then spend nearly-unlimited resources promoting the free copy?
The fact that you can do something that you are not authorized to do doesn't mitigate the fact that you don't have consent to do so. Just because we expect criminals to break the internet without consent doesn't mean everyone is morally absolved to do the same.
Sites that serve content to Google's crawlers, but then slap up a paywall are spamming users, and unfairly drowning out content producers that are playing by the rules. Last time I checked, they were doing this without Google's consent.
As for the read path, the sites consented to Google indexing their stuff, and Google consents to letting people read the crawler cache. I don't see the issue.
Is it immoral to use an adblocker? Many sites present extensive UI interruptions in the hopes that you will allow their ads.
Is it immoral to browse with Javascript disabled? Does morality require foreign code to run on your local system in order for these intrusive elements to actively manipulate (and potentially exploit) you?
In the days of newsprint, ads might have been garish, but they were bound into the media and they could not track you. Advertising now involves so much more surveillance, that I think the majority of the immorality is not with the end user.
I'm going through another round of google reviewing my extension right now and it does suck, but I can't imagine deciding that getting people to enable developer mode, manually download and load my extension (every time there's an update, too) is the more likely path to adoption.
If you call the script main.user.js instead of main.tampermonkey.js tampermonkey and other userscript tools will pick up the script automatically instead of needing to paste the URL in. For example I put my userscripts at https://userscripts.kevincox.ca and just clicking any of them should offer to install if you have a userscript extension set up.
I've had several YouTube ads telling me I should use a "hybrid" approach to self host some of my services, on hardware in my building, to keep cost down. I had to check a calendar to see what year it was!
My first webserver was the most frankenputer I have used to date. I had a friend that was at one of the large colo companies in town. He had a bunch of computer parts that he cobbled together with no case. They were hanging on the wall so that the mobo was flat against a dry erase wall, the PSU was on the desk with cables up to the hanging mobo, the HDD was velcroed to the dry erase wall next to the mobo. Talking about keeping costs down.
If I'm reading this correctly, the extension sends every URL you use it on to some random service hosted on Cloudflare workers [1]. That doesn't sound good...
> If you're interested I can show you the source code
This doesn’t really solve the old problem that there’s no proof the source code you show is the one that runs on Cloudflare. Couldn’t that code run locally?
Looking at the source, it seems like the page is proxied through a CloudFlare worker black box called “cfworker-beatthatwall.jayass.workers.dev”. It would be easier to trust if all the work was done locally.
Just for some context, Google did block this one, but there's a ton of extensions on the store that do similar things and are not blocked. Google's review process for chrome extensions is weak.
If you don't know what bookmarklets are: Edit any old bookmark and put the above line into the url field. Next time you click it, it will bring you to the Google cache of the current page you are on.
That's not the only thing it does. But it's the starting point. After that ir removes some scripts so the paywall don't show again after you load the page
So, OP must be using some other means to retrieve the page.
In the case of NY Times, they're likely just grabbing the non-archived version and performing an operation similar to 12ft ladder.
Google cache fetching seems like it might be an effective strategy for a site like Washington Post that have extremely effective paywall enforcement (till your turn off JS), but also allow Google cache.
Installing extensions in developer mode is not a good solution.
It’s not good to train users to do this because this is what malicious extensions do. Also, this will produce popup warnings and/or they can be automatically disabled by the browser.
I don’t even run extensions that I have developed myself in developer mode because it’s a PITA.
replace https:// of a paywall site with archive.is/
no extension needed.
And if you want to use Google cache (and Google is your main search engine) just add ? in front of URL and with two clicks (dots->cached) you get to the cached version of Google.
104 comments
[ 3.4 ms ] story [ 174 ms ] threadAccording to the original link the URL for the script is https://raw.githubusercontent.com/jorgefsilva/beatthatwall/m....
As long you are careful with the extension you install manually you should be fine.
e.g. from https://windirstat.net/download.html
For installing an unpacked extension:
- Obviously you don't have the benefit of the Chrome store checking for abuse.
- You'll need to read the manifest.json file yourself to see what permissions you're granting, because the warning popup doesn't show up when installing this way.
- There's a few attacks that unpacked extensions can do because they can spoof their extension ID, and Chrome doesn't consider it a bug. See: https://bugs.chromium.org/p/chromium/issues/detail?id=130196...
The extension developer could add a malicious permission + new code to exploit it and it would look the same as using developer mode to add a Hello World extension
Edit: Unfortunately https://12ft.io seems to be down right now, I hope it's temporary.
https://web.archive.org/web/20220824175226/https://12ft.io/
If the idea is to host other extensions on it as well, I'd suggest putting a little more effort into it so it feels like something that actually has extensions on it rather than blog posts. The page for the extension itself for example has none of the usual links or info about the author of the extension (the date and author feel like they're part of a blog post ABOUT the extension, not info about the extension itself because of the layout) and the actual link for the extension itself is a direct download link in the prose of the article itself.
This may be because I'm European but the complete lack of info about who operates the site (no privacy policy, just a twitter link and a copyright statement linking back to the site itself) screams "scam" to me on top of the impression that this is a blog trying to present itself as an extension store.
Please take this as constructive feedback but if I saw this site in the wild I'd assume it's malware.
If this grow enough in the next few months I will change some things including the TLD.
Adding privacy policy is the next step. Again thanks for your constructive feedback.
I could use some help with that
Of course I may be biased because I make some of my living creating content behind paywalls. If free access and ad support were possible, I would choose them first. But they aren't. To me, this extension and the people who support it aren't any different from those who go into restaurants, order food, eat it and then leave without paying. And there will be those who trot out some extreme rhetoric about world hunger or something to justify their actions.
Google is blocking this extension because it's bad for the web as a whole. Don't empower people who destroy the information sources and turn the web into a fact desert. Support those who create knowledge and share it fairly and equally, albeit at a price.
Nobody cares if you want to sell the secrets of the universe for $5000/mo. We care about the bait and switch.
By setting up this bait and switch, you hurt your own paywall - if the search engine can get a full copy, so can I. Want these extensions to stop working? Stop the bait and switch.
The technology does not allow them to do this without sharing it with you as well. But that is ethically irrelevant.
(Of course, we also deserve a good search engine allowing us to remove paywalled sites and seo spam from our results.)
We widely regard the above as unethical.
Adding content to a search engine literally says "you can come read this content!". That is the purpose of search engines. Even Google penalizes paywall behavior and will downrank them - which forces the paywall people to get more clever.
Sorry but don't abuse search engines and users to sell your content. Buy ads like a big boy.
Though I do agree, the bait and switch aspect of finessing seo and such leaves a bad taste in your mouth. Glad google penalizes such behavior
If they don't like it they can not submit to search engines.
According to the law, someone did.
Just because something's available on request from a server does not mean it's up for grabs.
When you give the content out for free to the indexer, you have given a copy to the world. Google search is not your advertising machine - you can pay for that priviledge if you would care for it.
It's actually everyone's business because this creates a web that is less usable for everyone. If publishers were willing to commit and make their paid content server-side for customers only, they would have a stronger case against infringers.
They provide users with an expectation of what information they will receive if they pay for the content.
The fact that this "glimpse" of the content pollutes your web searches is a search engine problem.
It would be trivial to filter sites with paywalled content. But Google refuses to let you do that. Hope someone else will come along and help with that.
I would be a lot less annoyed with these paywalls if they didn't rank so highly on search results, or at the very least indicated they were paywalled on the search page and were easy to filter out.
I don’t get this. They do whatever they want with their content. Editors give free book copies to journalists; that doesn’t gives you the right to have one as well.
The difference is consent.
Google is obeying robots.txt. Google caches a copy of what you serve and gives that out. That is part of the deal.
Nobody does that. The comparison doesn’t hold.
The idea that a single public exhibition of a work is enough to invalidate any future sale of that work is totally silly. Are you gonna bust out the window of a Barnes & Noble because libraries exist?
It’s exactly the same. You give access to an entity that serves to promote your content. The fact that Google’s cache is accessible is a technical implementation detail; Google could remove access to that cache tomorrow and that wouldn’t change anything.
It's intentional to fix the kind of abusive behavior you are engaging in. If you serve content to a search engine, that search engine will reproduce the content.
Search engines are for publically available content.
If you want to advertise, pay for it.
As for the read path, the sites consented to Google indexing their stuff, and Google consents to letting people read the crawler cache. I don't see the issue.
Is it immoral to browse with Javascript disabled? Does morality require foreign code to run on your local system in order for these intrusive elements to actively manipulate (and potentially exploit) you?
Is it immoral to rewrite a url in such a way that access is allowed? See: https://news.ycombinator.com/item?id=16906571
In the days of newsprint, ads might have been garish, but they were bound into the media and they could not track you. Advertising now involves so much more surveillance, that I think the majority of the immorality is not with the end user.
ps I have upvoted you.
Dedicated websites that are independent of a centralised controlling body? What a brilliant idea! Those ol' greybeards were on to something
[1] https://raw.githubusercontent.com/jorgefsilva/beatthatwall/m...
If you're interested I can show you the source code
I encourage developers here and elsewhere to reward each other for the fruits of their labor, instead of expecting a free lunch.
This doesn’t really solve the old problem that there’s no proof the source code you show is the one that runs on Cloudflare. Couldn’t that code run locally?
If so, you can also do this via a simple bookmarklet:
If you don't know what bookmarklets are: Edit any old bookmark and put the above line into the url field. Next time you click it, it will bring you to the Google cache of the current page you are on.For example, take this NYT article https://www.nytimes.com/2022/08/31/health/life-expectancy-co...
Google cache:
https://webcache.googleusercontent.com/search?q=cache:https:... -- 404
OP's service
https://cfworker-beatthatwall.jayass.workers.dev/?url=https:... -- works.
However, one can adopt your bookmarklet to use OP's service when needed instead of installing extension/userscript that seem to match all the sites.
> <meta data-rh="true" name="robots" content="noarchive, max-image-preview:large"/>
So, OP must be using some other means to retrieve the page.
In the case of NY Times, they're likely just grabbing the non-archived version and performing an operation similar to 12ft ladder.
Google cache fetching seems like it might be an effective strategy for a site like Washington Post that have extremely effective paywall enforcement (till your turn off JS), but also allow Google cache.
It’s not good to train users to do this because this is what malicious extensions do. Also, this will produce popup warnings and/or they can be automatically disabled by the browser.
I don’t even run extensions that I have developed myself in developer mode because it’s a PITA.
no extension needed.
And if you want to use Google cache (and Google is your main search engine) just add ? in front of URL and with two clicks (dots->cached) you get to the cached version of Google.
again no extension needed.
Stay safe.
Like, what if I want to run a different marketplace for chrome/firefox/safari extensions?