HN Explorer

Ask HN: Why does Google index pages I cannot access?

76 points by FreeHugs ↗ HN
Today I googled "follow famous readers online" and this is the top result:

https://www.nytimes.com/programs/better-reader/day-1

Google shows that the term is on the page. But when I try to access it, it redirects me to a signup page.

What's the deal here?

Does Google not notice that? Does it turn a blind eye? Does the NYT trick Google by displaying something else to Google then to me?

Or are pages that only members can read now part of the Google index?

37 comments

[ 7.4 ms ] story [ 90.7 ms ] thread
luckylion
> Does it turn a blind eye? Does the NYT trick Google by displaying something else to Google then to me?

Pretty much this. Google used to have a policy called "first click free" that basically said "you can have a paywall, but a user must be able to see the first content they click on from a search result. for more content, you can make them sign up". They dropped that policy at some point (and it was never really enforced, though many/most did follow it), so cloaking for paywall-purposes is okay now.

frou_dh
Notably, that quirk was the impetus for the link called "web" under every HN submission. Which, in my experience, still does work more often than not to get around paywalls.
ebj73
Do you know anything about how they're doing this? Is it special content in the http headers of the requests from Google? Or do they explicitly know which IP-addresses Google will be doing it's web crawling from?
[deleted]
(comment deleted)
tus88
At some point google entered into an entreaty with corporate news organizations to drive users towards subscription sites and paying for news. It is also probably influenced by more recent attempts to steer people towards establishment media sources rather than "fake news" websites.
moksly
I think it’s really hard to balance without giving us better personal search settings. This is anecdotal of course, but I rarely find anything on free “as in beer” media that’s worth my time. If Google excluded paywalled content its search functionality would be a lot less useful to me as a result.

On the other hand, you don’t subscribe to everything, and it might be useful to tune some of it out. Like I have a subscription to two Danish news papers, Information and Weekendavisen, and I like when google provides me with articles for them, because those articles are often going to be the height of what I want from my search results on subjects they cover. I don’t have a subscription to other Danish news papers, however, and maybe google would be better if it let me filter them. Maybe not though, it would certainly increase my personal bubble, but banning paywalls outright would really break google for me.

pbhjpbhj
> If Google excluded paywalled content its search functionality would be a lot less useful to me as a result.//

Almost all companies would stop paywalling - well almost all those who get traffic to their site from Google [I'm being somewhat facetious, that's practically every site].

mijamo
You'd be surprised... There is a surprisingly high number of sites that do not rely at all on Google to get their audience. I would even dare to say it is the majority of sites. Google is a great way to get some random folk looking for something, but if you really KNOW your audience there are much better ways to reach it.
pbhjpbhj
I've done a little user training, every user (!) - including my own children - has typed in to the Google (or very occasionally Bing) search box in order to navigate to a website. Even if they know to type an address they don't use the address bar, they use Google. Drives me crazy.
jpalomaki
Would help if you could easily filter out non-free results. Or highlight the free ones.
m-p-3
Or put a little dollar sign beside non-free results, like they do (or did) for ads.
blihp
A number of paywalled sites allow Google through either because it's in their self interest to do so or because they have a business arrangement with them. This has been going on for a long time. It used to be that Google would penalize a site in its search results for showing their web crawler something other than what a user would see but that appears to have been relaxed in recent years, at least for the larger paywall sites.[1]

[1] I believe that's the case... I don't recall in earlier years paywall pages ranking so high in the search results but my memory could be faulty on this.

ronreiter
Google must give a paywall filter in my opinion. I totally agree that a lot of times there is no intent for the Google user to see paywalled content when looking for information.
FreeHugs
I now noticed that you can read the page if you hit escape fast enough:

https://www.nytimes.com/programs/better-reader/day-1

When you visit it, it shows the page for a split seconds. Then redirects. By hitting escape in that split second, the page stays on screen and the redirect does not take place.

Modern developers :-) Implement everything clientside in Javascript.

It's kind of a fun game. Especially since the page then has no ads at all. All newspaper pages should look like this.

weinzierl
Spiegel Online used (or uses, idk) a blurring overlay to hide their payed content. My first thought was "That'll be easy!" and I removed the overlay only to find that the underlying text was obfuscated in a way that made it look like the original text through the blur. Well played Spiegel!

Still, I suspect the substitution could at least be partially reversed but I never tried. What makes it more complicated than a usual substitution cipher is that it can be arbitrarily lossy because it isn't meant to be deciphered. As an extreme case they could just replace all uppercase letters with X and all lowercase letter with x - for example - and no information could be gained while still looking similar when blurred. Luckily it's not what they did, the substitution looked more complicated. Another reason to make me believe that it's reversible is that I think I remember David Kriesel (the "Lies, damned lies and scans" guy) once hinted that he did it. Anyway, deciphering it would be a nice Sunday afternoon entertainment, I guess...

pintxo
They did in fact use a Caesar cipher [1], a quick calculation on letter frequencies gave a strong hint. The rest was looking up the shift value and coding a js bookmarklet [2] to "decrypt" the text. It had some issues with certain German Umlauts, but was not enough of an issue to fix it.

Looks like they now longer include the full text publicly. So they seem to have improved.

[1] (German) https://andreas-zeller.blogspot.com/2016/06/spiegel-online-n...

[2] javascript: document.querySelectorAll('div.obfuscated-content')[0].parentNode.classList = [];var cc = (s, c) => s.split('').map(s => /[^\s]/.test(s) ? String.fromCharCode(s.charCodeAt(0)+c) : s).join('');var dn = (n) => { if (n.hasChildNodes()) {Array.from(n.childNodes).map(dn);} else if (n.parentNode.nodeName !== 'A') {n.textContent=cc(n.textContent, -1) }};document.querySelectorAll('p.obfuscated').forEach(dn);document.querySelector('.lp_mwi_payment-method-wrapper').parentNode.parentNode.remove();

ThrustVectoring
I usually just download the URL in the terminal and open it locally when pages do that nonsense.
digitalengineer
Even worse for google image searches. Every search is invested with Pinterest results you can’t see without signing up...
ZoomStop
Google seems intent on ruining their decent image search engine. The removal of features like search by exact size is making it worthless for certain tasks, without a good alternative.
LocalH
Probably a bullshit concession to the copyright cartel
xnyan
Use bing. I prefer the results (MUCH less censorship than google) and they let you use it like a tool.
dplgk
I don't know why but I know it used to be against Google rules to show one thing to the crawler and different content to the user. Apparently they don't care anymore.
[dead] nairobi222
I think it’s really hard to balance without giving us better personal search settings. This is anecdotal of course, but I rarely find anything on free “as in beer” media that’s worth my time. If Google excluded paywalled content its search functionality would be a lot less useful to me as a result. On the other hand, you don’t subscribe to everything, and it might be useful to tune some of it out. Like I have a subscription to two Danish news papers, Information and Weekendavisen, and I like when google provides me with articles for them, because those articles are often going to be the height of what I want from my search results on subjects they cover. I don’t have a subscription to other Danish news papers, however, and maybe google would be better if it let me filter them. Maybe not though, it would certainly increase my personal bubble, but banning paywalls outright would really break google for me.
HocusLocus
Accessing hidden pay content is WRONG.

And I dussent think you wuz brought up to do WRONG.

Cyder
Sometimes you can bypass signup/restricted walls by using developer tools in Firefox. My son does this when doing homework and looking for answers.