Ask HN: What's Up with Google?

350 points by emsy ↗ HN
Recently, I notice more and more search results Google are locked behind registration gates, and not only does Google list them at all, they're often the top results (Think Quora, Pinterest). I never had issues accessing the content in the results but recently, it's bad. Does no one at Google care? Are the people in charge investors at Pinterest/Quora?

220 comments

[ 2.6 ms ] story [ 226 ms ] thread
It's not just pinterest/quora. Bloomberg News is also often up there, and a few others.
Google should just outright delist these sites, they are incredibly annoying and worthless
Google has "kinda" allowed this to happen for a number of years. They used to push sites to have a "first read free" policy so that their bot can still access semi-gated content.

Now they suggest "flexible sampling" [0] to let "users" view a limited amount of content per time period.

This is a game Google has been playing for a while that goes beyond SEO and seems to really be some issue with publishers. But to be fair there are a number of instances where publishers want free access to Google's search pages and users while requesting a fee for writing the content they indexed in the first place.

[0] https://developers.google.com/search/docs/advanced/appearanc...

There's a number of ways that you can expose paywalled content to Google without showing it to the user. The obvious one is paywalling content on the client side and hiding it from the user. I actually think this probably explains why client-side paywalling is so prevalent even though it's obviously far less effective. You can also serve your content in structured data that google's bots read (e.g. json+ld). I also think it's a tricky question because people often want to see authoritative sites high up in the search results even if they're paywalled, so "delisting these sites" isn't a solution that satisfies most people.
Hiding content is supposed to be against Google's stated policies.
Try searching for a news article, using the exact same title, as it's published on let's say... Breitbart. Now tell me that Google do not "hide" sites from it's users.
Google's policies explicitly differentiate between paywalled content from what they call cloaking. See the GP's link to the Google developer documentation.
Indexing flexible sampling makes no sense to me. I realize GoogleBot sees Quora differently than a human sees it, but they could easily make back end penalties. Your site loads too slowly? Organic search penalty. Broken rendering on mobile? Penalty. So why is human inaccessible content OK?

If content is blocked to me 70% of the time, shouldn’t there be a commensurate penalty? These publishers are shitting up Google results while harvesting free traffic to goose their new user acquisition numbers.

It’s the degradation of the organic Google product experience, I’ve always been baffled why they allow it.

Why would they? Few will stop using Google over that problem, and who's to say they aren't getting paid to keep Pinterest in the results?
They'd be staring at a bunch of lawsuits if they did that. Remember Google itself pretty much forces/nags you to login if you're on Chrome.
(comment deleted)
An era has finished, that's all. Google's corpse will be decomposing for quite a while though. Hopefully they will find their Nadella before they get completely IBM'd.
They're leading the field in AI/ML innovation which is the future. Wouldn't count them out when their stock performed best of any FAAMG this past year and revs are growing faster than the rest
IBM made Deep Blue, Watson (the Jeopardy player), two of the 10 most powerful computers in service today, and outperformed the S&P 500 from 2001 to 2014.

What's your point? A gigacorp has tons of inertia?

I recently stopped using google and switched my default to DuckDuckGo. So far I haven't needed to go back to google for anything.
Me too. DDG is still not as good on certain searches, but most of the time it's perfectly adequate.
One day I hope either ddg becomes good, or a good alternative pops up.
(comment deleted)
DDG is pretty good, it just requires a bit more context on average. As using a search engine is a lot about knowing the right choice of keywords it takes some time to adjust to the differences. Waiting for another random search engine that works exactly like Google is gonna take a while.

I still find what I need, that's all I care about. And I can't remember seeing a pinterest link on the first page.

Frequently, I search ddg, fail to find what I'm looking for, and then search Google. Or don't bother searching ddg on a certain kind of search because I know it won't find it. Both Google and DDG are mediocre versions of 2010 Google.
You are correct. I tried DDG before but it just isn’t good enough. I have been trying the new you.com search engine for a couple of months and it is really good, more so if you are in tech and your searches are work related.
Same, I've found you.com much more useful than DDG, especially for code related searches. Been using it for a month and haven't gone back.
So DDG (bing) doesn't list these sites OP mentioned?
It’s less the DDG/Bing is good and more that Google is declining.
How does switching to DDG / Bing help? You would only get worse results.
All I’m saying is I switched my default search on my phone and browser to ddg (just to try it out) and the results have been good enough to not make me switch back to google.
Golden...for their AMP!
For about the past month I feel like I have had a ton of searches that returned irrelevant results. I use to be able to have a typo and other wrong things and it always did a good job until recently.
It isn't so much about whether "Google cares" or otherwise. There are a lot of sites and a lot of vested interests trying to game their search rankings, and a huge number of keywords that the game is playing out in.

Sometimes low quality sites will have the advantage and sometimes high quality sites will have the advantage. Google is working with a bias to high quality, but they are fighting a broad fight against persistent opponents. Every so often Google does a big algorithm refactor to try and make life hard for these sort of sites.

The google scraping bot gets past the pay walls in many cases
Can I change my mobile browser to use the same user agent?
Kiwi Browser on Android, then install User Agent Switcher?
Won't work if they care, as there are other verifications: IP addresses https://developers.google.com/search/apis/ipranges/googlebot..., domain name https://developers.google.com/search/docs/advanced/crawling/...
>Won't work if they care,

Indeed. Many have lazy useragent implementations though

And if Google cared then neither of those pages would be public. At some point SEO became something that Google condoned and actively supported instead of seeing it as people trying to game the system.
I have a user-agent switcher and switching to googlebot definitely puts you beyond many paywalls.
It's a trend I've noticed with other services, notably Drive and Calendar - they no longer innovate, just get a UI tweak every few years. It's very reminiscent of cash cows, which typically are resourced to just keep going at minimal expense for as long as possible. Another possibility is that the product isn't "sexy" anymore, so they are having trouble finding the great staff they need to stay at the forefront. This would be fine if the rest of the world stood still, but clearly SEO has moved on in the meantime, and so we end up with a quagmire. It's basically another form of bit rot.

Unfortunately DDG is much worse in a different way - it seems incapable of dealing with synonyms. I can't recall a specific example since I switched back to Google, but it's on the order of searching for Python and getting results for Java (wrong subtree of the taxonomy), searching for Wellington and getting results for the whole of New Zealand (ignoring specifiers), or searching for double glazed windows and getting results for Microsoft Windows (no understanding of the semantics of the words being related).

DDG is also much worse if you Google things like “1 crore inr in usd.”
Googler, opinions are my own.

For products that are core, large UI changes tend to just annoy a lot of users, unless there's something really amazing about the change. Even then, people are hesitant to like new UIs. Engineers and designers definitely want to try new things, but balancing those against messing with UIs that a billion+ people use is hard.

My general take on drive and calendar, if they are trying to improve these cases for enterprise users. There's a lot of very non-sexy work going on to make these improvements, and they aren't always visible.

It's not "large" UI changes that are needed. (I mean, they are needed, the current google UI for most of their products sucks).

But it's small improvements that make the big difference. A small user interface element added or removed to speed up an interaction. An updated user experience so that data can be found or organized in a slightly different way.

I could pick on almost everyone of Google's products and have gripes about some UI or UX choice. Missing functionality here, overly complex there, etc. Every product has obvious low hanging fruit that could be improved.

That's what parent is referring to (I assume). There's no more micro-innovation happening. If it's not a change that makes 100 bazallion dollars, then it's just not a change worth doing, apparently.

I don’t think it’s that per se, for instance Meet has been innovating in small ways to the point where I believe it’s way better than Zoom. But not all Google tools are innovating at the same rate.
Every product has obvious low-hanging fruit that could be improved in the UX — yet the company persists in spending effort on sweeping yet entirely cosmetic changes that deliver zero value.

A good example is changing the typeface of text labels in every product to match the new Google logo, which Google seem to have been continuously doing for years on end now. I would argue this even has negative value, as it creates a mix of two typefaces (Roboto and Product Sans) in every product that used to have one, which causes every future UI design decision to be saddled with the extra cost of deciding which of the two to use.

One such example is the ability to copy my own email address. Gmail makes it so damn difficult it's mind-boggling. Whenever I try to copy my email address other elements are included in the selection, right-click is disabled and probably other annoying things happen: https://i.imgur.com/Ds0ROuT.png

Protonmail understood this and catered to this need without much fuss: https://i.imgur.com/1ZhWOsC.png

Calendar is great; but I wish it would let me edit an event the same way I can quick-enter one; from a pop-up. I'm half-inclined to write a browser ext to do it.

Also the latest updates to Sheets in the menubars have been welcome

Plus it's more fun to make a new chat / meet or payment app. Apparently.
I'm sure a marketing firm specifically tailored to such startups looking for a quick exit would be rather profitable.
I don't think it came across the way I intended, because I am one of those users who don't really care for most large UI changes. It feels like UI designers are the only people in large organisations who are actually allowed to throw old code away. My point was more that non-UI improvements are extremely rare, probably because they are both difficult to get right and not "sexy" in the sense of being nicely demo-able to less technically savvy managers.
Please. No.

The stuff just works.

From the experience of having several critical business SaaS tools go thru constant UX changes that disrespect the user and make the product less stable, I'm actually glad Google still have some respect for the user.

That's why i will gladly buy gsuite for the businesses i provide advice to, yet actively considering replacements of core business infra of which im a power user....for several years.

Its hard to keep something running with no bugs, or downtime.

Its even harder to change something, and keep it bug free, and with no downtime.

I think innovation != backwards-incompatible UX changes. There's quite a lot that Excel can do that GSuite can't, for instance. There's probably room for additional features that don't affect current workflows.
You're right, however there are spots where the tools/products have just the 'right amount' of features.

e.g. grep, vim.

Those are both massively bloated in terms of features, IMO. One curse of most open source projects is that features are almost never removed, even when they are obsolete, dangerous or plain stupid.
Oh, I don't want them to "innovate" like they've done elsewhere, just that they need to continue actually improving their services. I've had lots of interoperability, standards compliance and currency conversion issues with their services and practices, such as that time when I spend something like a year getting false promises from their support about being able to pay them in a different currency from the one with which I'd set up the account. In the end they exhausted the grace period for payment, and just shut down access to my account. Luckily by then I'd shopped around and found (to my surprise) a cheaper alternative which I'm still using.
Just works? Have you tried moving a file in g drive?
I need to do this regularly. The UI for move file is atrocious!
I wonder if they ever got Google docs to print/convert to pdf without disrupting the formatting.
Thanks to this cursed god forsaken parasite "Pinterest" I just completely stopped using google's image search option. I'm using bing/ddg now and it feels soooo good compared to how awful google search experience became
Bing is best for image search but it suck on mobile.
Both Firefox and Chrome have an 'unpinterested' add-on.
I think the utility of generic search engines is coming to an end. Services like Google, Bing and DDG are numbered in usefulness.

Instead, I am guessing (maybe hoping) that we see a return of the moderated directories, like Yahoo or DMOZ of old. A 2.0 spin on these directories, with lessons learned from all the years.

Imagine that you just go to StackOverflow and search there directly for your answer. Want a funny laugh, go to your favorite comedy website (like facebook.com) and search directly. Want some news, go to CNN or Fox (depending on your persuasion) and fill your echo chamber.

Not kidding. I think search as we know it is dead. And there is a new paradigm just sitting out there ready for the next generation to make.

A directory of well known lists of websites with specialty search engines available on each.

I've been coming to a similar conclusion.

Increasingly I've had to use `site:<domain name>` syntax with DDG, but DDG has clearly become significantly worse in recent months. This was confirmed by others in another HN thread recently. Even if I use the site-syntax or use quotes to specify exact match (which DDG used to honor), I'll often get nonsense results whereas Yandex somehow manages to find results that aren't on DDG or Google.

And yeah, I've been having to use search engines on specific sites like Wikipedia because I know there's gonna be less shenanigans.

Sometimes I wonder if I'm living in the past. Do normies actually use search engines anymore?

I think there's a market for sites that have optimized search around a specific category of information. For example, imagine you had a hybrid directory+search model, where you could drill down into the directory: Top -> Programming -> Lisp -> Scheme

And from there, search a repertoire of Scheme related articles, tips, blogs, etc.

This model already exists, somewhat, in the form of scholarly articles, medical journals, etc. But there seems to be some innovation needed.

If there were such a "site" that focused on and aggregated Scheme related documentation, you'd think that being listed on that search-directory would be totally acceptable to the website owner. After all, a web article or blog generally wants to be read, especially by an interested reader.

Somehow we need to get back to the web being like a library, with a librarian and a well organized categorical index for references.

Getting humans back into the search loop I think is where we're going to circle to. Like asking your local librarian for information about a topic that is in their special collection.

this is known as faceted search https://en.wikipedia.org/wiki/Faceted_search

on edit: I remember Carrot2 had the ability to derive categories from an index dynamically at one point, so when we put all the Danish laws into it and you searched for murder it generated categories such as Knife Murder etc. anyway here is the online Carrot2 with a search on Scheme https://search.carrot2.org/#/search/web/scheme/folders

on edit2: obviously this Carrot2 online thing is just a demo, they don't have a particularly big or up to date index. I'm also not sure how well Carrot2 would scale as it was, IIRC, an academic project to investigate different ways of searching / organizing search results. Check out the treemap.

Hey - thanks for giving a name for this and the link to Wikipedia. I know it's not a "new" idea, but a faceted search organized and usable broadly across the www would be really useful.
In the past Google appeared to better honour advanced searches using boolean operators. You could add operators to narrow a search quite effectively.

From my experience, they have diluted that functionality in favour of keyword and context matching against popular searches. Often the thing I want to find is not the most popular thing. Breaking out of that and the search bubbles you get associated with is becoming increasingly difficult. I find myself more and more clicking past the first page if results hoping what I want is there just with less SEO. Rarely is there anything relevant beyond page 3. I definitely remember sometimes digging multiple pages deep into the search results and still finding useful information.

In the early days of Google the speed at which you got the search results was a major factor, I remember being surprised at the speed of response and the quality of the results the first time I was introduced to Google. They still include the search time at the top of the results.

I would prefer to regain strict boolean logic in search at the cost of increased search time. I often care more about actually finding what I want than how fast the result can be returned.

Maybe something like the default search works as it does today as it gives a fast result that has a high likelihood will contain what most people want to see. Then allow for advanced search that strictly adheres to the search terms but could take a few minutes to return the results.

Yandex searches the long tail better and doesn't have really awful politicized censorship like Google does for some searches.
I've been using a browser extension aptly named "Search the current site". Just click the icon and type a search term, while being on that website. It will append site:domain to your search.
I tried that, but I try to search in wide world, sometimes about topics that I don't even understand. So I don't know where to look. And many of my searches are like that (examples: how merchant fees work, how QC affects stock prices, effects of vitamin D3, how price parity is determined, how to focus on something boring) All these are varied terms and most of search results will be like that, so I wonder if moderated directories would ever serve that.
I hear you. But arguably, imagine you could search a "collection" after you drilled down into your category.

  Top -> Finance -> Credit Card -> Merchant Fees + Search
  Top -> Finance -> Stock Market -> Algorithms + Search
  Top -> Health -> Vitamins + Search
And of course, the Wikipedia is a good jumping off point for all of that information. But imagine a wikipedia-like place that then allowed you to find the top level information that you're looking for with additional search capabilities in its supporting collection.

Maybe the start of this is taking a Wikipedia article on Price Parity and then enumerating the referenced articles in the footnotes and putting a search engine over those perhaps.

I don't know. But there's got to be something better than what we have right now. I feel another leap of search innovation happening soon.

You'll have to think for yourself a bit. Which platform and country are the merchant fees about? Do you want the official source and legal text, a professionals opinion from a magazine or newspaper, or a blog post by someone detailing what they experienced?

A good search engine can and should ask you about this and present you with your choices. Or, of course, they can choose what to show you to optimise for something, be it response time, resource consumption or ad profit.

How many of your searches lead directly to Wikipedia or stackoverflow or GitHub. Often we know where we should go but the default search engine in the address bar leads us through a Google intermediary. I myself use keyword searches to go directly to sites, but I doubt that is very common.
i just use google to search reddit these days.
Funny, Google search is so cluttered these days I almost automatically append "reddit" to any product or technique search. I think a lot of people are doing this, because the "People also ask" block on search results will often have questions like "best lawnmower reddit?" (a fine example of that Google quality we've all heard so much about)
Which is why reddit is starting to flood with spam. It won't last long.
Try DuckDuckGo. It has shortcuts for searching - at last count - 13,565 sites.
But is this driven by the supply side or demand side?

Seems to me that users would still benefit from a cross-site search capability (eg to follow your example, what if you want something funny not from a specific site or you don't know which sites offer something)

It's subjective but most of the pressure seems to be because the search quality has dropped off recently, not that I'm no longer willing/eager to search.

isn't that pretty much what you.com tries to do? Sort of a combination of both ordinary search results and categories based on sources.
Such optimism!

Don't you think it's more likely in a post keyboard/computer-literate world Search web sites are displaced by vendor-provided App search boxes?

You wanted news? Here's the CNN proprietary app in this beautiful walled garden! Enjoy the ads and total loss of privacy! [This app requires the following permissions: ALL YOUR BASE ARE BELONG TO US]

I think that's a fair point. The world of "apps" has certainly changed the meaning and usefulness of search. There's no exploring and random discovery when you're in the walled garden, unlike what it used to be like when you could really "surf" the web.
I hope to see the rise of distributed search. It's not that generic search is the problem, just that companies have different, and often counter, priorities than users.
Internal search can often be scammy though. If you search Google for products, you'll land on an Amazon page, but if you do your search on Amazon instead, they will sometimes only show you more expensive versions of the general product, as if the cheap one that you were looking for doesn't exist.
I don't buy it. The technology to fight SEO gaming and filter bad results has been around for a while and is only getting better, but google is getting greedy and sloppy. Google has mind-blowing AI/machine learning capabilities and I didn't see shit about cleaning up search results in their last update. They could fix this if they wanted, and an upstart company that figures it out will eat google's lunch. Google makes an insane amount of profit, and a search engine that actually works may make less but would still be profitable.
> Instead, I am guessing (maybe hoping) that we see a return of the moderated directories, like Yahoo or DMOZ of old. A 2.0 spin on these directories, with lessons learned from all the years.

Not impossible, but given the proliferation of content on the web and the endless possibilities for competing organizing schemas and editorial choices, it would pretty much have to be a directory of directories.

So, each directory could be it's own self-organizing community, like a subreddit.

You should try Neeva[0], made by ex-Google Search folks.

[0]: https://neeva.com/p/signup?o=s&c=c3af6jnfmc2dehs8tq5g

"coming soon to your region"

Ugh, that triggers me hard. I haven't lived in Germany nor Sweden for ~7 years now and Google never really figured out (despite me telling it so) that I don't need gmaps results in Stockholm and Berlin, and that yes, all my g products should be showing the English UI.

If these folks are ex-google they might consider not doing shit like this in the first place.

Damn, that sucks! No idea they did things like that.
If you can get a beta invite for https://kagi.com get it. Not only are their defaults really good you also have the option to customize your searches by blacklisting, creating groups with sites like stackoverflow, github, hackernews etc for a dev search and so much more. It feels like google in the 00s with better customization.
Tried getting one. Told me to fill up a survey. Closed.
You won't be a good beta tester without providing feedback so good for them.
It's a paid search engine, they're going to get your name, address, credit card details and everything you search for and click on. If a survey puts you off, that's probably a good early warning.
> Customize Results

I feel like Google could be 10 times better if it allowed the user to block/boost pages.

For those who aren't aware: Google is a lesser-known competitor of the recommended search engine DuckDuckGo.

You can Duck for 'google' to find out more about it.

DuckDuckGo is just Microsoft's highly successful attempt at getting you to use Bing
I'm p sure my ad-blocker removes these results all. I dont use google that often but never ever without ad-blocker. Whats the point of a page full or ads when I hardly ever search for anything I want to buy.
I can pay for a search engine service if it blocks paywalls, bloated sites, fake bullshit clickbait contents. It’s time consuming to find what you’re looking for with Google. Unfortuanetely, there is no alternative.
gated results are clearly cloaking right? displaying different content for googlebot vs the search user?

https://developers.google.com/search/docs/advanced/guideline...

Kind of. When NYTimes or Medium only show the first bit of the article it's because you've exceeded your free allotment. Tracking number of articles read uses cookies, which Googlebot does not send.

Still, some sites may condition on User-Agent or otherwise present different content than a clean browser and I think those should be de-listed for cloaking.

I think it's probably tied to the rise in gated content. Especially news. Unless they want to delist every for-profit news site, they can't carte blanche remove walled content. The algo is fundamentally based on popularity. If enough users are logging in to those sites and clicking on them, they'll keep being ranked.
It blows my mind that Google penalizes sites for being a fraction of a second slower than they imagine they should be, but a site I can’t actually view- right to the top baby.
But is that still the case? I recall reading about this as well, but that was years ago. In the meantime Google throws up a gigantic disclaimer popup every time you want to use any of their services, slowing things down by seconds.

(Happens every time because I don't whitelist Google cookies, because are you kidding.)

I just discovered how poor the search results are if you're logged in with your Google account. Have to log out or view in incognito mode just to see what you were searching for. Also I didn't realize how censored Google's results were until I started comparing them side by side with DDG.
Can you search as if you're from a different country? My incognito searches are useless because they are region-specific.

Edit: Yes, you can do this if you click on settings -> show more settings -> show more countries and pick the country you want. Which seems like a pain.

You should be able to do this by navigating to google.com/?gl=us, google.com/?gl=sg, etc.
It's also that free sites suck these days. Quora is still free for vast majority of content, just need to register, and has very interesting facts and nuanced opinions. The Economist is a place to read real news rather than talk shows that CNN and Fox have become. Ad supported doesn't seem to work for quality content. So do you want to at least be aware of the quality stuff or just browse junk? Serious question, there should be a search engine to cater to "free only" crowd.
I certainly wouldn't want Google to exclude or even automatically downrank paid sites for exactly the reason you bring up here. If Google didn't show the paid stuff, I'd be unlikely to know it exists. And there's nothing wrong with having to pay for high quality information. I wonder if a payment model that provides the option to pay per article might make sense to allow people who know they're not going to regularly use the site to still benefit from the information. Or maybe not per article, but instead the option to pay for a day of access instead of a month.
I’m all for paying for high quality information, but wasn’t the point of the Web that anyone could participate? For example, I’d rather pay an independent blogger in Ukraine to tell me about the border crisis from their lens than go through a centralized gatekeeper like NYT to have some office worker in NYC opine on the crisis.

Unfortunately Google downranks primary sources and sends secondary sources to the top. Essentially the Web has become cable television instead of a global system for the open exchange of knowledge and information and something about that is super depressing and rubs me the wrong way.

This is one of the many problems we're trying to tackle at you.com (full disclosure, I work there).

We felt like it's important that you can select your preferred sources and downvote sources you don't like.

For this and other reasons we separate the experience into a private and personalized mode. So you get exactly what you want based on your explicit settings (like StackOverflow code snippets or arxiv papers) and can have hardcore privacy when you want that (where your location isn't used, not even your searches are stored like on DDG).

What's the business model, currently and in the long run? Also, is this built on top of another search engine's results?
Hey. We will work on it together with the community this year.

Private mode is even more private than DDG so we will likely have to do private ads just like them.

But for the personalized experience we have a few ideas that I think are more creative and more aligned with users and privacy but not fully fleshed out yet. We're still a very small team but hoping to ship them this quarter.

Care to explain how is the private mode superior to DDG and what is a purpose of a non-private mode?
It's not. When OP first launched his search engine a few months ago, it was pointed out to him multiple times that DDG doesn't seem to communicate with third-parties, but You does even when in private mode. OP avoided that question - and a few others - then and, it appears he's still avoiding it now (he hasn't responded to you in 2 days).

https://news.ycombinator.com/item?id=29161545#29194124

I’ve had pretty good success with dev related searches on you.com. I like the idea of being able to downvote garbage sources. Who owns the company?
(comment deleted)
(comment deleted)