Usually on these sites, there'll be an `overflow: hidden` element that's holding all the content. If you can find and disable that CSS line, it'll work as normal. Or just save it to the Wayback Machine and read it through that.
I am going to add this to the post if that is alright. Please let me know if I shouldn't cite your HN username on the post. You can reply here and I'll see it. Thank you, this is super helpful.
In the case of Washington Post it just has a "position: fixed" style on the <body> element. That's usually the case with most of these scroll locking sites, one of the root parent elements will have some CSS style that you can click off.
Some sites now are literally not loading the actual paywalled content until after you sign in, so not matter what you do you aren't going to be able to access it unless someone with a paid subscription shares that content and it is then uploaded to a third party paywall bypasser.
> I'm not sure of how it works (does it subscribed to them all?) but https://archive.ph/ is a good way to see the content in those cases.
I think for search engine crawlers there are versions without a paywall so these articles can get fully indexed. Archive.ph, and similar services, might get the full content this way somehow. But I am just guessing.
you're spot on, this is exactly how sites like archive.org, archive.ph, or even if you click on "view cached version" on Google get the non-paywalled versions.
bloomberg.com for instance, hides pay walled lines in empty <div>s.
the other method is to disable javascript and cookies (works on nytimes.com), or press ESC key to stop page loading before paywall kicks in (works on telegraph.co.uk) :)
> But really, if you are regularly reading content on a site you should subscribe to support the journalists employed there
I'd go further than your statement: I try not to read paywalled contents. Actually I don't get all these workarounds about paywalls. I'm like "they don't want me to read it? I'm not going to read it then".
Sure, they want me to read if I think they are worth to have my subscription. But I don't think they are worth to have my subscription, otherwise I would subscribe them. So I don't read.
Caching is parroting what the original poster publishes. If the OP asks not to index, you cease indexing. However, if you are purposeful "hosting," you may not accept and continue hosting, knowing all the legal consequences that may follow.
Caching has nothing to do with indexing. If I set up an HTTP cache to serve cached websites on my network, it would be difficult for anyone to request to not cache the HTTP responses considering it is most likely they would not be aware of them at all. It is, however, exactly equivalent to what websites such as archive.ph are doing.
Archive.ph et al is run by a russian fellow so probably the third one, especially now that Russia doesn't seem to care about something like this at all
What about Internet Archive wayback machine archive?
Is deleting your cookies legal? How about spoofing your user-agent?
How about a browser plugin that automates what OP describes?
In the case of caches like Google, Internet Archive, or `archive.today` (same thing as `archive.ph`)... probably, in the USA? If it winds up in court, we will find out, eventually.
Simply reading anything on the web technically involves "making a copy" already, which is one reason it gets and has remained somewhat confusing and complex to determine what is or is not legal with regard to copying web content. You can't simply say "making a copy is not allowed".
> But really, if you are regularly reading content on a site you should subscribe to support the journalists employed there.
While there are some paywalled websites that allow you to read _n_ articles per period for "free", there are many that don't. How do I know in this case whether it's worth the cost?
There are also times where I'll see a link to something behind a paywall with an interesting headline (frequently on HN), but from a publication I don't regularly read, so have no intention of a subscription. It would be nice in this case to be able to pay a one-time, small contribution.
Worth stating I don't disagree necessarily with the sentiment, there are just a few "edge cases" that make it impractical.
I subscribe to a major newspaper. But I'm not going to subscribe to all major newspapers. The individual subscription model doesn't really fit a world where you can go to one site, and click links to articles from lots of different publications.
If they had a common subscription, where you pay one reasonable fee and they divide it up according to whose articles you read, I'd subscribe to that. Since they don't, I subscribe to one paper and do workarounds on the others. I feel this is ethical because if everyone did it, with a decently random distribution, then the newspapers would survive just fine. They'd make the same overall revenue as when everyone had one newspaper, showing up at their doorstep each morning.
How about getting a paid membership for certain higher education institutions’ libraries, which will give you access to not only books, but also a plethora of periodicals. And that will be fully legal and ethical.
> If they had a common subscription, where you pay one reasonable fee and they divide it up according to whose articles you read
So some extent that's what Apple News+ (included in Apple One) is. But it doesn't do multi region stuff I don't think, and misses some major publications.
There is a common subscription mostly overlooked. Your local library.
What most of the press subscription services make the mistake of is trying to simplify the billing process to the individual. Ripe opportunity for someone to come along and make an all in one paid subscription service at the county level and make it easy to log into all sites with your library membership.
The current one is per site model is way too much from too not be worth it.
Syndication used to be the way this idea worked. You'd buy your local paper that wrote about the cat stuck in the tree down the street, but it also republished national articles from AP. If you really cared about another locale, you might subscribe to two papers, which meant you might see the same national article twice.
This is also quickly happening with social media now.
Twitter $8, FB $12 (web) / $15 ios)
You're asking for the cable tv model where they aggregate premium channels, imho that doesn't work either... you end up paying for a lot of stuff that you're not interested in.
That is basically the Apple News model. If you have Apple News (part of the Apple One bundle) you can get articles from a number of publishers that would normally require a separate prescription. I’m not sure if the publishers really like this model but so far they seem to be willing to play along.
I stumbled upon a link to an article with an interesting headline. I would like to read it, so I click the link, but there's a paywall. I have no idea what site that even is. This is the first and probably last time I'm seeing it. No way it gets a single cent from me. This just can't work when there are so many news websites competing for subscriptions.
Yes, archive.ph works most of the time, can't recommend it enough.
I’d be more willing to subscribe if publications didn’t pull the dark pattern of making signing up fast and easy, but then requiring a call to a rep to cancel.
I used to subscribe to the nytimes but a few years ago I needed a break from news. My plan was to come back in 6-12 months, but they made me wait on the phone for 25 mins for something that should have taken a couple of minutes on their site. I cancelled and never went back.
Move to the Netherlands. The medium used to sign up, should be available to cancel. Cancellation can't be harder than signing up, and (barring a few exceptions), you can't have an autorenew period of longer than a month, after an initial year.
These may actually be EU rules BTW, haven't checked.
This website almost succeed every time I run out of my tricks, like:
1) ESC to interrupt the page load
2) quickly hit "view mode" before the wall appears
3) add a "." behind the .com, so like .com./
4) visit in incognito window when the tokens run out (e.g. Medium)
5) Check Google cache of this page, (you can quickly add cache: URL to visit the cache page)
6) Check archive.org cache of some lost pages
7) maybe some extensions but I seldom use them nowadays
8) before, there are some cool sites like, sorry I forgot the names, all stopped working, those websites can remove paypall
I also have one to kill all running javascript and remove all event listeners, it works wonders when you are redirected to a paywall / login page after a few seconds.
I recommend the browser extension "Bypass Paywalls Clean". I sometimes think about the morality of using it, but I just don't find it viable to pay all the websites where I read just a single article.
> but I just don't find it viable to pay all the websites where I read just a single article.
This.
In the print days, you'd buy a newspaper; you'd have access to all the articles in that edition. I used to read a daily paper.
In the modern world, these papers expect you to pay for a newspaper just to read a single article. I dunno, perhaps they could form a "Paywall Consortium", so that I could pay a one-day fee to the consortium, and have access to Washpo, Telegraph, NYT etc. for 24 hours. Let the consortium figure out how to distribute the fees - it's not my concern.
But if you want me to buy the whole paper to read a single article, well, ain't gonna happen.
> But if you want me to buy the whole paper to read a single article, well, ain't gonna happen.
This was common for non-subscribers in the print days. Newspapers would print a number of enticing headlines and images on the front page above the fold, and display those folded newspapers for sale at dispensers, newsstands, and stores. Many people who bought a one-off paper would buy for a single article that interested them.
A quarter? For a printed newspaper? The cover price of The Guardian is £2.50 weekdays, £3.50 weekends (about 5 quarters and 8 quarters respectively). Apparently Washpo is $2.50 for the daily edition, which is about two quid.
Can't speak for OP but I think I would be annoyed if I was paying a subscription and I found out I could get around it with a simple `.` because of developer incompetence.
Why would this work? I know FQDNs sometimes require specifying a period at the end, and browsers seem to accept it as well, but I wonder why this would affect some frameworks' displayed content. Normally a site would rely on a cookie to maintain logged-in status, and that shouldn't be affected by the request URL.
Maybe some backend concession to allow viewing the article on hostnames/uris that are meant for internal use, partner sharing sites, etc? Like an if/then that checks the origin hostname?
10 years ago paywalled sites contained the content just hidden. Today I haven't seen a site in a long time that renders the content hidden (why would it do that? There is no reason to do it based on indexing/SEO as far as I'm aware).
Even cached/archived versions these days tend to not include the whole text. Basically: they figured out how to make a paywall, which frankly isn't that surprising.
What I find annoying about paywalled sites is that they provide the full content to Google. And Google is OK with indexing the full content, even though it is not available on the internet, and even though they explicitly forbid the practice of showing different content to a search engine from what is available publicly.
Paywalled sites are just fine, but they are not part of the open Internet, and should not pretend to be.
Yeah, 100% agree with this. It's like these sites want to have their cake and eat it, and both get the traffic the 'open web' provides while not having to actually share any of their work there.
It's like if you needed an app to view a page, yet Google had all its content indexed. Why is that (rightly) seen as unreasonable while charging users for content you provide to bots for free isn't?
Sometimes the full content is available - but only if you navigate to it via a Google results page, and you don't have existing cookies implementing a "free article limit".
Title should be : "getting around very poorly implemented paywall (eg WaPo) with devtools alone".
As soon as your site sends the whole content of the article to the browser, you're not even trying seriously. (And Firefox "reading mode" is just much better ux than the devtools.)
Another option no one has menitoned is AMP. Many sites that try to use "paywalls" have AMP URLs which point to pages that have all the full text of the article in <p> tags. These AMP sites generally look great in a text-only browser that does not run Javascript. Popular example is WSJ. In the URL, add /amp before /article/.
Paywalls are insidious because they target non-subscribers. Why let non-subscribers view articles. Why not password protect all subscriber content. Paywalls are a way to make money from (the attention of) non-subscribers, targeting them with ads and tracking. The strategy is apparently to annoy people to the point of subscribing. Yet even if they subscribe they will still be subjected to advertising. One potential advantage is that a paying subscriber has an enforceable contract. In theory the contract could contain enforceable privacy protections. "Tech" companies would never agree to give people enforceable privacy protections; it would destroy their "business".
The way to save journalism, especially local news, is to regulate "Big Tech" middlemen, who generally do not employ journalists and produce zero content.1 The quality of journalism in general has taken a nosedive, but placing the blame for that on web users not purchasing subscriptions is conveniently ignoring the true culprit.
1. Arguably that's a prerequisite to maintaining their Section 230 protection. In the recent Supreme Court oral arguments, Google's counsel argued Google is not a publisher. Then minutes later she argued Google has to make design decisions "like any publisher", therefore Google gets a free pass to reorganise information in annoying and perhaps harmful ways to maximise ad services revenue, like inserting "popular" videos into YouTube search results that have nothing to do with the query string.
Yep. And the browser vendor can make changes to DevTools, archive.ph is already privacy-unfriendly and blocked in some countries, it could disappear, 12ft.io could disappear, and so on.
The fact that seems least likely to change is that most "paywalls" rely on Javascript, CSS or some other "feature" of so-called "modern" web browsers.
I have lost count of how many times an HN comment alleges "paywalled" in a thread and I am reading the article just like any other because I am not using a graphical web browser. I would not even know there was a "paywall". Paywalls have dependencies.
My lynx is always at the ready for paywalls and other annoyances. My guess is that showing up using such an archaic tool implies you might also be using a screen reader setting like blynx [1]. The more thoughtful sites will even give you a link to jump directly to content from the top...
Lynx was a program I used on the early www. In the early 2000s I moved on to something better. Without much effort anyone can have a static binary with better HTML table rendering that is significantly smaller than today's lynx.
readelf -d links
There is no dynamic section in this file.
stat -c '%s' /usr/bin/lynx links
1894400
1276712
"Other proposals would have more broadly exposed providers to liability for hosting unlawful content, if the provider is aware of that content.318 For example, the Platform Accountability and Consumer Transparency Act (PACT Act) would have amended Section 230 so that certain providers lose immunity under subsection (c)(1) if the provider is notified about illegal content or activity occurring on its service and does not remove the illegal content or stop the illegal activity within 24 hours."
"Proposals like the PACT Act would state that if a provider is aware of an unlawful post, it will lose immunity for a lawsuit premised on that specific post.334"
"The CASE-IT Act would have taken a similar approach, providing that service providers and users lose Section 230(c)(1) immunity for a year if they engage in certain activities, including permitting harmful content to be distributed to minors, if the harmful content is made readily accessible to minors by the failure of such provider or user to implement a system designed to effectively screen users who are minors from accessing such content.337"
"In the same vein, other bills would have caused providers to lose Section 230 immunity if they use algorithms to distribute content to users or display behavioral advertising.338"
Yeah on the system most of the local papers use over here, the full content is not even loaded for non-subscribers. It used to be, so removing the blur div worked, but now only the headline, byline and lead text are visible :(
122 comments
[ 3.3 ms ] story [ 190 ms ] thread* { overflow: visible !important; }
Not sure why this 'hack' is on the front-page.
No idea how it works but it looks like actual content is loaded separately once the gates are open?
https://www.theinformation.com
For those cases try something like Google Cache or Wayback Machine! It still won't always work, but it's nevertheless got a pretty good success rate.
I'm not sure of how it works (does it subscribed to them all?) but https://archive.ph/ is a good way to see the content in those cases.
But really, if you are regularly reading content on a site you should subscribe to support the journalists employed there.
I think for search engine crawlers there are versions without a paywall so these articles can get fully indexed. Archive.ph, and similar services, might get the full content this way somehow. But I am just guessing.
https://blog.archive.today/post/678202832257794048/why-cant-...
While pretending to be GoogleBot used to get you full articles (or grabbing them from cache) this doesn't seem to be the case for some sites anymore.
They just give the first part of the article without the paywall, as that's usually enough for SEO purposes.
Many consumers often wouldn't read more text anyway. About one paragraph might even be too much to fill the modern attention span.
https://searchengineland.com/google-no-longer-recommends-usi...
bloomberg.com for instance, hides pay walled lines in empty <div>s.
the other method is to disable javascript and cookies (works on nytimes.com), or press ESC key to stop page loading before paywall kicks in (works on telegraph.co.uk) :)
I'd go further than your statement: I try not to read paywalled contents. Actually I don't get all these workarounds about paywalls. I'm like "they don't want me to read it? I'm not going to read it then".
- Is it fair use because it's "archiving" the web?
- Is it because it's on the open web and it's public domain?
- Or is it illegal, and people do it because they can ¯\_(ツ)_/¯
Technically it is illegal. Do I use it? Yes.
Is Google cache legal?
What about Internet Archive wayback machine archive?
Is deleting your cookies legal? How about spoofing your user-agent?
How about a browser plugin that automates what OP describes?
In the case of caches like Google, Internet Archive, or `archive.today` (same thing as `archive.ph`)... probably, in the USA? If it winds up in court, we will find out, eventually.
Simply reading anything on the web technically involves "making a copy" already, which is one reason it gets and has remained somewhat confusing and complex to determine what is or is not legal with regard to copying web content. You can't simply say "making a copy is not allowed".
While there are some paywalled websites that allow you to read _n_ articles per period for "free", there are many that don't. How do I know in this case whether it's worth the cost?
There are also times where I'll see a link to something behind a paywall with an interesting headline (frequently on HN), but from a publication I don't regularly read, so have no intention of a subscription. It would be nice in this case to be able to pay a one-time, small contribution.
Worth stating I don't disagree necessarily with the sentiment, there are just a few "edge cases" that make it impractical.
If they had a common subscription, where you pay one reasonable fee and they divide it up according to whose articles you read, I'd subscribe to that. Since they don't, I subscribe to one paper and do workarounds on the others. I feel this is ethical because if everyone did it, with a decently random distribution, then the newspapers would survive just fine. They'd make the same overall revenue as when everyone had one newspaper, showing up at their doorstep each morning.
So some extent that's what Apple News+ (included in Apple One) is. But it doesn't do multi region stuff I don't think, and misses some major publications.
What most of the press subscription services make the mistake of is trying to simplify the billing process to the individual. Ripe opportunity for someone to come along and make an all in one paid subscription service at the county level and make it easy to log into all sites with your library membership.
The current one is per site model is way too much from too not be worth it.
Twitter $8, FB $12 (web) / $15 ios)
You're asking for the cable tv model where they aggregate premium channels, imho that doesn't work either... you end up paying for a lot of stuff that you're not interested in.
Yes, archive.ph works most of the time, can't recommend it enough.
I used to subscribe to the nytimes but a few years ago I needed a break from news. My plan was to come back in 6-12 months, but they made me wait on the phone for 25 mins for something that should have taken a couple of minutes on their site. I cancelled and never went back.
These may actually be EU rules BTW, haven't checked.
"May I ask why you want to cancel your account today?"
"No, you may not."
For some insane reason they ALL phrase the question this way, enabling this little grammatical gem of a response. Enjoy.
"... calls may be recorded for training purposes."
"Thanks for giving me permission, I am recording this call ..."
This website almost succeed every time I run out of my tricks, like:
1) ESC to interrupt the page load 2) quickly hit "view mode" before the wall appears 3) add a "." behind the .com, so like .com./ 4) visit in incognito window when the tokens run out (e.g. Medium) 5) Check Google cache of this page, (you can quickly add cache: URL to visit the cache page) 6) Check archive.org cache of some lost pages 7) maybe some extensions but I seldom use them nowadays 8) before, there are some cool sites like, sorry I forgot the names, all stopped working, those websites can remove paypall
9) console tricks though I dunno.
https://pastebin.com/qBjJHkMv
I also have one to kill all running javascript and remove all event listeners, it works wonders when you are redirected to a paywall / login page after a few seconds.
This is supposed to be saved as a Javascript Bookmarklet?
Yes, I'm using it as a bookmarklet. I'm using firefox but I think it should work the same for other browsers.
This.
In the print days, you'd buy a newspaper; you'd have access to all the articles in that edition. I used to read a daily paper.
In the modern world, these papers expect you to pay for a newspaper just to read a single article. I dunno, perhaps they could form a "Paywall Consortium", so that I could pay a one-day fee to the consortium, and have access to Washpo, Telegraph, NYT etc. for 24 hours. Let the consortium figure out how to distribute the fees - it's not my concern.
But if you want me to buy the whole paper to read a single article, well, ain't gonna happen.
This was common for non-subscribers in the print days. Newspapers would print a number of enticing headlines and images on the front page above the fold, and display those folded newspapers for sale at dispensers, newsstands, and stores. Many people who bought a one-off paper would buy for a single article that interested them.
https://site.com./1235/article
Those behind Cloudflare don't seem to be vulnerable to this though.
I've emailed the sites I've found where this works and none of them have fixed it after a year.
Why?
Unfortunately they kind of go hand in hand these days!
Guess it's a feature then and not a bug.
Even cached/archived versions these days tend to not include the whole text. Basically: they figured out how to make a paywall, which frankly isn't that surprising.
There are so many ways to do a paywall and you’ll still see all sorts of flavors across the web today.
Paywalled sites are just fine, but they are not part of the open Internet, and should not pretend to be.
It's like if you needed an app to view a page, yet Google had all its content indexed. Why is that (rightly) seen as unreasonable while charging users for content you provide to bots for free isn't?
This isn’t true. This paywall treatment is something they do allow and have worked to accommodate.
https://youtu.be/QHtnfOgp65Q
Google could easily add an option to filter out paywalled content but that would reduce clicks.
Just press that, select the paywall (and any other junk backdrops/opaque divs), and press delete.
Sometimes the site also sets an `overflow: hidden` in the css, and you need to remove that to see the content..
As soon as your site sends the whole content of the article to the browser, you're not even trying seriously. (And Firefox "reading mode" is just much better ux than the devtools.)
Paywalls are insidious because they target non-subscribers. Why let non-subscribers view articles. Why not password protect all subscriber content. Paywalls are a way to make money from (the attention of) non-subscribers, targeting them with ads and tracking. The strategy is apparently to annoy people to the point of subscribing. Yet even if they subscribe they will still be subjected to advertising. One potential advantage is that a paying subscriber has an enforceable contract. In theory the contract could contain enforceable privacy protections. "Tech" companies would never agree to give people enforceable privacy protections; it would destroy their "business".
The way to save journalism, especially local news, is to regulate "Big Tech" middlemen, who generally do not employ journalists and produce zero content.1 The quality of journalism in general has taken a nosedive, but placing the blame for that on web users not purchasing subscriptions is conveniently ignoring the true culprit.
1. Arguably that's a prerequisite to maintaining their Section 230 protection. In the recent Supreme Court oral arguments, Google's counsel argued Google is not a publisher. Then minutes later she argued Google has to make design decisions "like any publisher", therefore Google gets a free pass to reorganise information in annoying and perhaps harmful ways to maximise ad services revenue, like inserting "popular" videos into YouTube search results that have nothing to do with the query string.
The fact that seems least likely to change is that most "paywalls" rely on Javascript, CSS or some other "feature" of so-called "modern" web browsers.
I have lost count of how many times an HN comment alleges "paywalled" in a thread and I am reading the article just like any other because I am not using a graphical web browser. I would not even know there was a "paywall". Paywalls have dependencies.
While I’m sure some combination of plugins can do it, Tor is another go-to for me for paywall avoision.
Too bad violoncello is no longer available in the iOS App Store.
[1] http://leb.net/blinux/blynx/
There’s no such thing as maintaining protection. It’s not something you “lose.”
"Proposals like the PACT Act would state that if a provider is aware of an unlawful post, it will lose immunity for a lawsuit premised on that specific post.334"
"The CASE-IT Act would have taken a similar approach, providing that service providers and users lose Section 230(c)(1) immunity for a year if they engage in certain activities, including permitting harmful content to be distributed to minors, if the harmful content is made readily accessible to minors by the failure of such provider or user to implement a system designed to effectively screen users who are minors from accessing such content.337"
"In the same vein, other bills would have caused providers to lose Section 230 immunity if they use algorithms to distribute content to users or display behavioral advertising.338"
https://crsreports.congress.gov/product/pdf/R/R46751
Guess they caught on to the "cheaters."