Soft paywalls aren't, like many nouns that need an introducing qualifier (think homeopathic medicine for the most common example).
More specifically, a soft paywall isn't a starving artist trying to get their daily bread from valuable content (If they were I'd be biased to support them, because I'm among the smaller crowd around here that rather likes IP law and procing things, and kind of enjoy poking the gibs me dat for free beehive. I even probably support, horror of horrors, what I heard was Musk hard-paywalling twitter to make it profitable, and he's neither starving nor an artist), it's a malicious trick to hijack when you're trying to find free information on a subject by bait-and-switching a, say, promising quoted section from the middle of the article on a search result. It's like a digital used car salesman who promised a great deal or free benefit in an ad but is all out of non-full-price inventory once you drive all the way out to the lot.
Until these sorts of sites stop shipping their whole catalog to be indexed as available while hiding behind ever-more-sophisticated ui barriers (or browser drm soon, maybe?) and firmly take an honest and upfront stance of either freely available to read or benefiting from paying customers I think the right stance is that silver beats gold; You only survive if you do unto others as they do unto you, so for as long as they're putting out bait we should take the worm and skip the hook: https://archive.ph/X3gHI
> it's a malicious trick to hijack when you're trying to find free information on a subject by bait-and-switching a, say, promising quoted section from the middle of the article on a search result
Already feels that way with login walls like on Xitter or Instagram tbh.
Independent of the indexing, just because they are quoted and linked to everwhere, as if they were a public billboard.
Especially twitter seems to toggle theirs on/off on a random basis or based on referer, heuristics, etc.
Or Quad 9... or potentially any DNS server that refuses to provide EDNS. I don't think there's an "official list" of DNS services on the archive.today shitlist, but it's been growing lately. These days you have to select recursive DNS on a balance of "archive.* works" vs. privacy preservation.
> PS I was reading (sceptically) 'google search is broken' already for a year or so in HN comments and have to admit I believe it now.
Even ignoring results, mainstream started to notice a while ago. Perhaps the biggest social proof at large is when it is shown distilled as comedy and people can laugh about it:
The "mainstream" accepts many claims as true that can be objectively demonstrated to be false. Why should I have such confidence they're right about a theory like this that seems to be unfalsifiable?
Seems like the complaints that have been showing up here a for a while have reached the mainstream. Any chance of an upstart competitor taking the opportunity?
It certainly seems like Google Search prefers to return ad-ridden monetized sites rather than more usable sites, with the obvious explanation being that they earn more money from sites that show ads.
But this feels like too obvious a conspiracy theory and it’s probably not directly designed to act that way - more likely it is an unintended consequence of something else. Unintended, but tolerated all the way to the bank.
They must have considered some possibility of a "Subscriber Results" option for Google One subscribers or something along these lines? I'm guessing that would be too risky to make their non-subscriber results look like garbage?
They serve such a large number of users who would never pay, I cannot imagine they would ever consider admitting that they could serve better results. That would be a disaster. They make money off of users from surveillance and advertising. They would never make enough from subscriptions to offset the losses in perception from all the people they are surveilling for advertisers.
There is a huge difference between "Youtube without ads or with ads" and "good google searches or worse google searches". Everyone understands that with premium you can pay to remove ads. But paying for better google searches would upend the entire image of google as the premier search engine.
This is a good analogy, but maybe doesn't put enough emphasis that for search users are often actively seeking information and ads can (not always) even contribute to the usefulness of results. On YouTube the user is more often seeking entertainment and the ads are blocking them from that.
Probably more like selection bias. Sites that are heavily monetized with Google have relationships with them and can work directly with them to improve their SEO and page rankings. Plus if they have all those ads they almost certainly also use Analytics, which can help them optimize too.
But also, if someone claimed to be a Google engineer and told me they purposely rank up pages that have Adsense on them, I wouldn't call them a liar immediately.
Unfortunately, the sad truth is likely that Google has hard telemetry evidence that users in aggregate prefer this SEO junk, engage with it, and move on feeling that Google successfully satisfied their query.
If Google could drive higher levels of engagement and satisfaction by screening out the dross I think they would, it’s clearly within their abilities.
Hopefully AI gets good enough and scales efficiently enough so they can disaggregate this kind of ranking decision.
The recipe problem is literally this: users who go to a site check the recipe, realize it isn't a fit and return to Google are considered failures lowering your result status.
By putting fluff and ads you extend the time on the page making your site seem more valuable.
It is hard to tell "that didn't answer my question" from "that quickly answered my question wrong" after all.
(Note it isn't uncommon to always bounce off a recipe site as you check multiple to make sure the one you found makes sense)
> hard telemetry evidence that users in aggregate prefer this SEO junk, engage with it, and move on feeling that Google successfully satisfied their query.
How in the world can telemetry tell you any of that? The only metrics I've heard from Google that relate to it is whether or not they return to do the same search again within a short window, but I don't think that tells you users prefer SEO junk or that they're satisfied with the search results.
A more likely explanation is that ad-ridden monetized sites are able to invest more in SEO since a visitor is worth more to them than an ad-free or low-ad site.
That wouldn’t explain why Bing has the same problem as Google. Bing isn’t making money off of serving pages with Google ads. If anything, they are handing money to a competitor.
Google can allow us to block sites from search results, but Google don't want to. For example I will easy live without results from pinterest, expert-exchange, medium, and list of other sites..
As the article mentions, the web has changed. It's very hard to return a list of good results when there are none. I've yet to see side-by-side comparisons where another search engine is doing a better job.
I've had "Search Generative Experience" (AI/LLM synthesized result/summary above the standard web results) turned on for awhile. It is definitely the future. To survive the flood of revenue optimized content (recipes being a familiar example), you need an active [software] agent working on your behalf to read content and make one straight thing out of the crooked timber of the web.
The problem isn't just that the internet has become more SEO-oriented and encumbered with low-value clickbait websites -- it's also that Google doesn't listen to instructions the way it used to.
Google started slipping when it began assuming that it knows better than you do. For instance, when the exact search operator (" ") stopped working.
I'd take the Google of 2014 over the Google of 2023 without thinking twice, and I think that I'm not alone in this.
Do you use it often? It doesn't work the way it used to. I can't count the number of times I've searched for a technical term -- in quotes -- and got results back for a subtly (sometimes not so subtly) different term.
I don't even bother with quotes anymore. When I'm searching for something technical from the normal interface, I'm almost always going to Tools -> All results dropdown -> Verbatim.
I think Frank Zappa would chortle with irony that a Russian search engine returns his discographies uncensored in the here and now versus an American one.
All search engines are inherently biased, since they perform a ranking task (which is a value judgement). They all encode (possibly uncessfully) the values of their creator. This is why having multiple search engines is important, as diversity in search options is the closes you'll get to objectivity.
(said the independent search engine creator with the same air of righteousness as when Plato argued philosophers should be kings)
Yes but they’re also politically biased. Not to mention, the enormous amount of web content that is effectively banned from seArch engine results due to low prestige
Absolutely. i so often do a search and it presumes to think it knows what i want rather than giving me what I ask for, sometimes even when I'm being quite specific. I will say it does a great job for programming stuff but a lousy job whenever I'm researching economics topics.
Even programming stuff it does a lousy job. I use R a lot. I plot with the base graphics package because it works fine. Finding R information that does not involve the tidyverse library is so damn difficult on google search. Its bests to just go straight to the documentation since stackoverflow and other sites are pretty worthless with pretty much all answers requiring ggplot or dpylr even if they can be implemented with zero dependencies with the base tooling.
Same situation with python honestly. No I don’t want to use pandas, if I wanted better data wrangling I’d just wrangle in base R then import the wrangled data into python if need be instead of adding another dependency to my python code. Apparently this makes me a heretic based on google search results.
My most recent complaint is the disappearance of the pagination numbers.
Since I know the top results are mostly useless spams on some searches, I used to go to page 4 or 5 to find the actual results. Now to achieve the same result I have to click and scroll 5 times.
It works, but it does something different than it used to. It used to only include results that actually contained the exact phrase you put within quotes. That is no longer the case.
This is not my experience. I search for exact phrases the same way I always have. Three to result does not contain the word I emphasized. This never happened before a few years ago. Now it happens daily.
Are you sure the page doesn't contain it? It is not a simple as just visiting the page and searching for the thing you quoted as there may be punctuation between words or the text may be in the alt text of an image or somewhere else that is not obvious.
I am quite sure. The kinds of words in question would not even really make sense for "SEO gaming" to come into play. Typically, it presents with trying to match a word vs a phrase(I don't ever try to match phrases). I try matching smaller and smaller fragments of the word and still... nothing. I remember this working back when it was common for pages on gray websites to stuff giant blobs of text with juicy keywords out of view with janky html. Yet, somehow those pages were ranked very low and results were good. There is a clear degradation in quality.
Actually, it has happened with results that include hacker news discussions. I scrape through and expand EVERY comment on a 6 month old post and still find no partial or exact matches for my search keyword. It is very clear to me that Google has switched behavior. It used to flex on how awesome it was to bring you to an esoteric corner of the internet matching your exact query. Now, it tries to converge your queries into what it thinks you want(not what you asked for) and funnel you into large websites at the expense of utility.
This is a good point. I searched for "rename" today and the top result switched it to "name" instead. The TOP result completely changed the query. Infuriating.
I also love when my search is narrow enough that Google instead starts to return more generic pages for totally different programming languages instead of the smaller result set that matches exactly. It is very easy to observe as you explore a topic. At some point, you got a wall where any increase in search specificity seems to yield worse results than queries that helped you get to that point.
Yeah; that's a good point. Google's desire to provide popular/well known information sources is outweighing the ability for people to be specific. It's infuriating to be ignored as a user. Hence using verbatim constantly, which feels like a checkbox for "stop failing me".
Which... Why isn't Google learning that some users are constantly seeking out verbatim for results they'll click on and adjusting their search results to be more verbatim?
I have no idea; but I speculate this is one area challengers have it easier because
a) there is no giant industry optimizing for ranking on Kagi. There are whole armies of people trying to game google search, meaning whatever heuristics they have would probably be rendered useless shortly if google were to adopt them.
And
b) Kagi has a much smaller, more technical userbase. So like the early web they can get away with a search experience tailored for those kinds of people and not millions of people powering on a smartphone as their first computer.
The biggest power of Kagi is “you” are the algorithm. A page is optimizing for Kagi and providing bad results? Block them. You found a good website burrowed down? Raise/promote them.
Kagi works for you, with your input. It doesn’t do “advanced guessing” by implicitly and opaquely processing your interaction with it.
Explicit control can work for ranking but can’t scale to the retrieval step of search - you aren’t going to sift through thousands of thousands of pages of potential results to give feedback on a meaningful portion of them, there must be some method of pulling pages to be ranked. This is where the gaming happens, with basically limitless spam domains that need to be filtered out. There just is no way Kagi is 100% relying on self reports for this.
The whole idea of the model is to fit it on some reference so you can project it onto new data you haven't classified yet. So your first few weeks, you might get some spam sure. But eventually if you are optimizing the system by e.g. only ranking technical blogs with no seo bullshit, then technical blogs with no seo bullshit is what would fit best with that model among the new data and what you see on your results. That can't be gamed without knowing the sort of model fit you are generating and this fit is defined by the user.
A lot of people on HN are very happy with Kagi, but I haven't seen examples of searches to understand where it is better. In a blind "taste test", I would be surprised if Kagi or Bing is preferred to Google for most searches.
Well, all my daily searches, whether it be technical or daily mundane things return better results with less cruft than Google. Also, it notifies you if a page has excessive number of trackers.
Combine it with personal website ranking and lenses, it surpasses Google easily.
Also since it has no ads, there’s no noise.
I don’t use Microsoft products, so I can’t comment on Bing.
I did side by side for a while and Kagi was consistently superior to Google and bing. I think it boils down to they don’t try to cross sell advertisements so they’re more oriented towards actually providing better results rather than better paying results.
Kagi's killer feature can't be blind tested, because Google and Bing don't have it. The ability to pin, raise or block sites in your results is a game changer.
Interestingly, Google had that in 2008 (SearchWiki), but it got extremely low usage - enough that at a time when Google was serving 30K QPS, you could read the queries with SearchWiki interaction as they came in on a screen.
I suspect this is a niche feature for power-users only. HN is filled with power users, so they love Kagi, but it doesn't necessarily translate to the masses.
Do you need examples of searches? If people get more useful results from one search engine over another, that's all that matters.
I can't provide you with side-by-side examples because I don't compile lists of such examples. I'm searching to find stuff, not researching search quality. I just know that I have more problems getting good search results from Google over a couple of the other engines.
Yes. I would love to see some actual examples. People have said that verbatim search doesn't work for them and I've never observed this, so I'm also skeptical of claims of dramatically better search results.
I can almost guarantee you that you'd find better results if you could block all results from spammy domains (without using an extension). It's practically a tautology.
It looks like this uses some sort of automated system controlled by Google -- I am skeptical we'll see any improvements, given how bad results have been for so long (see all the SO and GH clones returned by Google for many examples). Who knows, though; I've certainly been wrong before.
No one can provide examples. But it doesn't mean they are lying. They may think it's better but actually isn't. Maybe it is worse but people are protective over it because their identity is connected to it's success.
If a user of service thinks a different service is better, then it is better for them by definition. "Better" is a subjective thing. If X seems better for me than Y, then it is.
"Are US drivers licenses forgery-proof?" and "Can US Real-ID drivers licenses be forged?"
gave utterly irrelevant #1 or #2 hits: nothing to do with US or Real-ID (zero mentions), it's about New South Wales, Australia (lots of mentions of "NSW"). (although from US magazine Wired.com, but Google can still tell this article is totally irrelevant).
2) I previously in 2022 posted "Google search relevance fail: result for “Africa longitude” [https://news.ycombinator.com/item?id=30337563] where the #1 site hit and text result were the grossly misleading "Africa/Coordinates 8.7832° S, 34.5085° E", which is not a range, or even a centroid, just a junk location in southern Tanzania, where a (Russian?) SEO spam person seemed to have dropped a bogus pin, mislabeled "оф. 3, 27 ул. Хабаровская, Bila Tserkva, Russia". In Tanzania. 17 months later and same gloriously unfixed garbage results.
Other bizarre top hits were the small Ottawa e-commerce digital-map site https://www.mapsofworld.com/lat_long/africa.html,
with the incomplete and inaccurate non-sentence "Africa's latitude and longitude lies between 9.1021° N, 18.2812° E."
(That e-ecommerce store seems to have an autogenerated page with incorrect information on latitude and missing longitude for each country).
These are cases where Google itself 5 years ago would have done better. Or compare to Kagi.
I’ve thought about making an account and spending the free quota on a side by side comparison. The problem is you need a representative sample of queries. The things I search might not be what you search or what my mom searches.
I don't think the problem, in general, is that no good results exist for any given query, more often the problem is that the good results are rarely well search engine optimized, and thus stand zero chance of making the search results.
Some people admittedly expect pretty weird websites to exist. Like who would even publish a good objective comparison between products in a specific category. But people make those queries looking for those types of results, and spammers generate such content to make bucks off the traffic. 20 years ago you'd have gotten nothing in most cases.
Or simply just passionate people talking about a niche.
Consider coffee. You'll find hundreds of articles comparing the most high-end grinders. You even have PhDs writing white papers on the physics of coffee and how particle size distribution achieved by different grinders affects extraction quality. Those are genuine opinions from aficionados.
The same is true for all kinds of other things. The internet has a lot more than the auto-generated marketing crap you give it credit for.
This is true for things that are related to a hobby. There are many things where it basically doesn't exist. Nobody is writing articles comparing server colocation offerings near ${YOUR_CITY}, yet people search for them expecting them to exist.
Finance markets are spread throughout the world and people spend a fair amount of money building efficient networks to route data between them faster than the competition, yes.
I'll also had that a huge portion of genuinely good content is now inaccessible to Google in walled gardens like Discord, Facebook, Twitter, and TikTok.
"To survive the flood of revenue optimized content (recipes being a familiar example), you need an active [software] agent working on your behalf to read content and make one straight thing out of the crooked timber of the web."
So what do you think, will be the goal of all advertisers?
To get their recommendations baked into that AI. Direct bribing/revenue sharing with the manufactur of that AI. Or sneaking data into training sets via bribed employes and so on.
Right. That's why the agent should live under your control, and receive training from your search history.
Paid search is the future. Either you pay a subscription fee to something like Kagi, or you pay for hardware and electricity turn run your own "AI agent" locally.
I don't believe that anything Google does now - summarization included - will avoid being gamed just as effectively as their current search results have.
It's a negative-sum arms race, but if Google (or whoever) wasn't spending the money to fuel the arms race by passing on ad money, who would pay for Google to exist in the first place?
I am convinced that a lot of the reason why Google search went downhill is that they started using more AI. Specifically, using it to try to interpret what I'm searching for rather than taking my word for it, which leads to irrelevant search results.
That's just my hypothesis, of course. I don't know for certain. But it does make me very skeptical about using AI to help search the web.
I liked StartPage/Ixquick results much better than what Google used to show. Maybe they still do but wouldn't know as I stopped using it after they got acquired. Didn't/don't they rely on Google?
> To survive the flood of revenue optimized content..., you need an active [software] agent working on your behalf
How long do you think it will be before that software agent's output will be just as riddled with commercialized content? I think we might have only a short window where we get to enjoy the fairly unadulterated output of the base LLMs by the large providers (yes, I'm aware that they're screening for safety already, but there could be a lot more coming). Imagine a dialogue like this: Q: What should I watch out for when buying new car tires?
A current answer might be something like: Here's a list of things to watch out for when buying car tires: A, B, C, D, E. Remember to always consult with specialists. Shall I explain more about the topic?
A future answer might look something like: According to experts like the professionals at [tire shop near you], here are the things you need to watch out for when buying car tires: A, B, C. You can use this coupon to get a 10% discount off your next purchase at [tire shop near you]. Shall I book an appointment for a free consultation at [tire shop near you] for you?
> you need an active [software] agent working on your behalf
I'd love to have one, but do any exist? SGE works on google's behalf, collecting data for google who will use what they learn from it against me. I'm guessing that until we can use common hardware to train/host an AI agent ourselves, we're stuck with terrible search results unless we're willing to hand over more and more data to others who want to exploit us, while pretending we can trust them to return results that are in our best interest instead of theirs. That's not a great future to look forward to.
When google first appeared it was a revolution compared to everything else. If you expect this kind of difference, none of the alternative search engines come close.
If you are looking for big search engines, there aren't many really, I strongly believe you should give them all a shot and just alternate between them.
The biggest takeaway from this is that google isn't massively better than the worst alternative anymore and it sticks mostly due to recognition.
> When google first appeared it was a revolution compared to everything else.
At first it wasn't even because their search was that much better than anyone else's. What made google popular was that it wasn't a massive, messy, ad-filled page of things you didn't care about. Other search engines were bloated "portals" that tried to do everything: email, news, shopping mall, weather, etc
Google had a simple clean interface that loaded instantly on a typical dial up connection and that's really what got people to make it their homepage. Today, google has slowly turned into exactly what it was created to solve. Ad-filled bloated pages that try to be everything by pushing things you don't care about in your face.
I'm using brave search and digging it. I _really_ love the goggles feature. They have popular ones such as "Hacker News 1k short". Really makes the old web come alive. Also a fan of the "lk short" one.
I've been using kagi the last 1.5 years after seeing it recommended here. It's been fantastic for me. I use it on all my devices and never use Google or DDG any more. It feels like old Google with more features. Unfortunately it's not free but they just today announced unlimited search for $10/month which for me is quite reasonable. I sometimes hit over 1000 searches per month (1532 is my record so far). Although it costs money I like knowing that kagi is the product, not me or my data m
I'm a little tired of all these "Google search used to be better" feel-pieces that don't do anything to substantiate the claim beyond relying on the author's vague sentiment.
Let me try restating it for the literal-minded: I see many articles making a vague claim that Google search "used to be better." I do not know how one could even measure this and I have a strong suspicion it's based more on the influence of hearing other people parrot it or some vague dissatisfaction with the present than on any real degradation.
Technical searches on Google have become a pain in my ass. I'm constantly hit with low effort articles about topics that Google has guessed are similar enough but definitely aren't.
10-15 years ago, Google didn't have the same internet to crawl -- but it also didn't fucking _ignore_ what I asked for.
The literal minded of you perhaps were not there when Google arrived at the scene. And as you admit: you don't really know if there is a difference. Yet hearing about others that perceive difference makes you very tired.
I was there. We used Altavista and Yahoo and they rocked. Until we tried Google.
I was there too and I don't think it was so much better. I find it very useful right now. Have any better arguments?
You are also misconstruing what I said. I don't say I "don't really know" if it was better. I don't think it was better and seeing people baldly assert the opposite without proof hasn't convinced me my memory is wrong. I'd like to hear something better than "trust me bro." I'm tired of people getting published with evidence-free arguments (though I guess it beats someone condescending to me based on a wrong guess of how old I am).
Buyer beware: Sensation sells (as is portrayed in this news article). Competitors are one click or one setting away. Last I tried switching to DuckDuckGo things didn't go so well -- just overall subpar experience. Bing arguably is a bit better, but that too didn't satisfy, despite the super cool chat feature (deserves kudos for that!).
We're building a new search engine. Of course it's AI driven, but at its core, it's all about human dignity. When you use it, you'll have a voice. If you'd like to try it, please email me: josh@sirch.org
Thanks!
Josh
PS. If you'd like to join our team, we have 16 people now (mostly Snapchat/Instagram/Twitter folks), and we'd like to go up to 50 before we raise capital. It's going to be a wild ride.
The other day I asked Google "Are US drivers licenses forgery-proof?" and "Can US Real-ID drivers licenses be forged?"
The #1 or #2 hit is utterly irrelevant: nothing to do with US or Real-ID (zero mentions), it's about New South Wales, Australia (lots of mentions of "NSW"):
That's still a relevance fail: "Real ID" driver's license [0] refers to the 2005 US improved standard in driver's license, which was intended to be harder to forge. Which is the entire point of the Google query. (We all know pre-Real-ID driver's licenses could be forged, that's not at issue.)
But that link is a 2002 article at least about the US, but not Real ID.
(any random hit for ""real ID" forgery" in the last 18(!) years would be better)
What were the other links on Kagi? As far as this query goes, neither Google nor Kagi seem to get it right. Google gets the country wrong and Kagi misses the “Real ID” part. The difference is, you’re paying for the second mistake.
I will also add that the particular link you showed is a 7300 word document from 2002, most of which has nothing to do with ID cards at all. It’s at a college graduate reading level and would take 26 minutes for someone to read according to wordcounter.net.
I was going to say I don't personally get all this Google is broken stuff. It works ok for me for most things and I like a lot of the extras like integrating with maps. I'm a 'level 4 local guide' on it now for what it's worth.
The one thing I have noticed recently though is chatgtp is way better for certain queries. Like I wanted to figure which brand of fillings I'd had in my teeth the other month. Googling "brands of dental fillings" was hopeless - it gives all results like "here are a number of different fillings, including: amalgam, composite fillings". No brands. Chatgtp though, first try listed all the brands. I think it was sonicfill by the way - quite cool tooth tech.
For those who are lazy like me—or maybe you'd call it efficient—here's the key points summary from Kagi:
- Google Search has evolved significantly from its early days as a simple list of blue links to becoming an encyclopedia, predictive engine, image repository, shopping mall and more that is overloaded with information.
- It has become more difficult to find authoritative answers on Google Search due to an overabundance of sponsored content, prompts, and low-quality keyword-stuffed pages.
- Google is currently undergoing an antitrust trial regarding whether it maintains its search engine monopoly through anticompetitive means such as exclusivity deals rather than having truly superior technology.
- Access to large amounts of user data is very important for powering search engine algorithms through personalization and improving the user experience.
- While Google argues diminishing returns to scale for user data, internal emails show Google engineers acknowledging that scale remains highly important for the company.
- Some feel Google has lost its way and become conservative due to its success, with its corporate bureaucracy stifling acquired companies.
- Google Search has become bloated with advertisements and prioritization of its own services over organic results.
- Search quality may be declining as its algorithms are gamed by low-quality sites and search engine optimization techniques.
- Google's trajectory of scaling up its mission to organize the world's information through acquisitions and exclusivity deals has put pressure on it to keep growing.
- No company can likely grow indefinitely to keep up with the ever-expanding internet without being overwhelmed, as Google Search now demonstrates.
Thanks for the excellent summary. Though I think the article and summary miss the root cause: Google Search has two masters, users and advertisers. These stakeholders’ interests are not aligned, and by taking one middle road after another, Google has deteriorated into near-uselessness for end users, relying on scale rather than quality.
> Google Search has two masters, users and advertisers
No, it doesn't. It has one master: advertisers. Advertisers are the customer. Users are the product. Search results are the bait.
This is self-evidently true. Your customer is the one who pays for your services. Google's users don't pay, never have, never will, because search is not a product you can sell.
User attention, on the other hand, is a very valuable commodity for which people will pay handsomely.
No. Users are engaging in a barter transaction. It's not different currencies, it's the complete absence of currency. Barter transactions are symmetric. You could as easily say that Google is the user's customer.
Users are Google customers in the same way that (say) fish are a fisherman's customer. Fisherman engage in barter transaction with fish, offering them bait (food, or at least the prospect of food) in exchange for allowing themselves to be sold as a product to a third party who gives the fisherman actual money. That is what makes the third party the fisherman's customer and not the fish. Likewise, Google engages in barter transactions with users, offering them bait (useful information, or at least the prospect of useful information) in exchange for allowing their attention to be sold to a third party for actual money. The money is what makes the advertiser Google's actual customer and not the user.
Apologies, thought we were talking economics, where barter is every bit as legitimate as cash. But you’re talking polemics where there’s moral failing in Google’s and user’s parts. Fair enough.
I didn't say barter was illegitimate, I said it was symmetric. There are salient differences between barter transactions and currency transactions even from a purely economic perspective. It has nothing to do with morality or polemics. There is a reason money is a thing.
That’s a pretty great summary. Can you elaborate a bit on how this works with Kagi? Is it just a wrapper over gpt-4, do you feed it an article and ask for a summary, or does it summarize in the search results?
Good question. I've been using Kagi a long time but have never tried this feature. I just tried, and it does bypass the paywall. No idea which sites that works on, but I'd imagine it correlates with ons archive.is can bypass - I assume crawlers are just allowed to read the whole thing.
I would add, as advertiser this time, that you don't know what you are really paying in AdWords. I used AdWords from the beginning and there were business leads for super niche spaces, now it is very difficult to get leads with a moderated budget and you cannot explain clearly why.
> Google is currently undergoing an antitrust trial regarding whether it maintains its search engine monopoly through anticompetitive means such as exclusivity deals rather than having truly superior technology.
I'm not really sure how they lose that case -- Firefox switched to Yahoo after MS outbid them, and switched back to Google the second they could legally do so and keep the money.
Apparently (since this is the top-ranked comment) I'm alone in thinking that this sort of "summary" is dumb and pointless. Just skim over the article. You cannot separate form and content. What's next -- a PowerPoint summary?
I'm still not sold on the idea that "personalization" improves the "user experience" of search.
When I think about what I actually use search for... it's highly situational. It depends as much on day-to-day and moment-to-moment context as it does any long-term trend specific to myself.
Whatever "personalization" could be realized over search results, would it really offer that much benefit over improving the search function to sift through and find results that are just more useful generally?
What I mean is there seems to far, far more to be gained from making search results better fit the query than to fit me personally.
Why is it that when I use Google search, the links are always some intermediary google.com/** links which when clicked takes me to the original site. Is this some kind of user tracking?
There is a logical issue here, people are accusing Google with the line of attack that large scale gives them benefits. I've seen variants of this before:
> “It’s absolutely not true that scale is not important,” Manber wrote. “We make very good use of everything we get.”
I think that is true, but then the obvious conclusion - if one party doesn't have 90% of the market then the quality of search results will be lower. Successful anti-trust prosecutions against the party with 90% of the market will have a likely outcome of:
1) Reducing the quality of service for everyone.
2) Pushing the 90% market company to be one domiciled somewhere with more reasonable monopoly laws.
Successfully proving that more data -> more scale -> better results is actually implies that one company with 90% market share is desirable. Although I'm still not going to use Google search, I don't think that argument is correct. But the line of attack is not clever.
That’s called a natural monopoly which is a well studied phenomenon. Most commonly they are regulated as utilities with guaranteed low returns on capital and limited ability to set prices.
That isn't a natural monopoly, that is the quality of the service increasing with market share. The cost of competitors entering the market could be anything (it could easily be quite small, there are a lot of search engines, many small 1-man projects). The issue here is the quality of the product scales with market share.
With a natural monopoly, if you built 2 of them in defiance of the economics of the situation they'd have similar quality of service to the single operator.
I think you’re right on a narrow definition of price and quantity like what Wikipedia gives:
> A natural monopoly is a monopoly in an industry in which high infrastructural costs and other barriers to entry relative to the size of the market give the largest supplier in an industry, often the first supplier in a market, an overwhelming advantage over potential competitors. Specifically, an industry is a natural monopoly if the total cost of one firm, producing the total output, is lower than the total cost of two or more firms producing the entire production.
In theory pricing/output in economics is always “quality adjusted” (which is one of the hard things in inflation series)
Yes, traditionally a natural monopoly is one where if you split them, costs go up. With Google, GP is saying if you split them up quality goes down. Either way there’s “consumer harm”. (I guess very directly if the time to search goes up, the economic price of using Google goes up too)
185 comments
[ 2.8 ms ] story [ 373 ms ] threadMore specifically, a soft paywall isn't a starving artist trying to get their daily bread from valuable content (If they were I'd be biased to support them, because I'm among the smaller crowd around here that rather likes IP law and procing things, and kind of enjoy poking the gibs me dat for free beehive. I even probably support, horror of horrors, what I heard was Musk hard-paywalling twitter to make it profitable, and he's neither starving nor an artist), it's a malicious trick to hijack when you're trying to find free information on a subject by bait-and-switching a, say, promising quoted section from the middle of the article on a search result. It's like a digital used car salesman who promised a great deal or free benefit in an ad but is all out of non-full-price inventory once you drive all the way out to the lot.
Until these sorts of sites stop shipping their whole catalog to be indexed as available while hiding behind ever-more-sophisticated ui barriers (or browser drm soon, maybe?) and firmly take an honest and upfront stance of either freely available to read or benefiting from paying customers I think the right stance is that silver beats gold; You only survive if you do unto others as they do unto you, so for as long as they're putting out bait we should take the worm and skip the hook: https://archive.ph/X3gHI
Already feels that way with login walls like on Xitter or Instagram tbh. Independent of the indexing, just because they are quoted and linked to everwhere, as if they were a public billboard.
Especially twitter seems to toggle theirs on/off on a random basis or based on referer, heuristics, etc.
Even not Google. I wonder if they can still turn things around, but I have the feeling MS caught up finally at high speed and passed them.
PS I was reading (sceptically) 'google search is broken' already for a year or so in HN comments and have to admit I believe it now.
Even ignoring results, mainstream started to notice a while ago. Perhaps the biggest social proof at large is when it is shown distilled as comedy and people can laugh about it:
https://www.youtube.com/watch?v=NT7_SxJ3oSI
https://news.ycombinator.com/item?id=37606618
Also, searching for that quote on HN via Algolia gives many hits.
Disclaimer: developer
I am bootstrapped, don't advertise, and need no investment
But this feels like too obvious a conspiracy theory and it’s probably not directly designed to act that way - more likely it is an unintended consequence of something else. Unintended, but tolerated all the way to the bank.
It kind of depends what the ad value per user is, and then charge X% above that. Though maybe it’s so high that it would be an enterprise price level.
But also, if someone claimed to be a Google engineer and told me they purposely rank up pages that have Adsense on them, I wouldn't call them a liar immediately.
If Google could drive higher levels of engagement and satisfaction by screening out the dross I think they would, it’s clearly within their abilities.
Hopefully AI gets good enough and scales efficiently enough so they can disaggregate this kind of ranking decision.
By putting fluff and ads you extend the time on the page making your site seem more valuable.
It is hard to tell "that didn't answer my question" from "that quickly answered my question wrong" after all.
(Note it isn't uncommon to always bounce off a recipe site as you check multiple to make sure the one you found makes sense)
How in the world can telemetry tell you any of that? The only metrics I've heard from Google that relate to it is whether or not they return to do the same search again within a short window, but I don't think that tells you users prefer SEO junk or that they're satisfied with the search results.
I've had "Search Generative Experience" (AI/LLM synthesized result/summary above the standard web results) turned on for awhile. It is definitely the future. To survive the flood of revenue optimized content (recipes being a familiar example), you need an active [software] agent working on your behalf to read content and make one straight thing out of the crooked timber of the web.
Google started slipping when it began assuming that it knows better than you do. For instance, when the exact search operator (" ") stopped working.
I'd take the Google of 2014 over the Google of 2023 without thinking twice, and I think that I'm not alone in this.
https://www.google.com/advanced_search
[verbatim mode] https://www.google.com/search?q=unexpected+newline+in+compos...
[quotes] https://www.google.com/search?q=%22unexpected+newline+in+com...
The top two results are the same, but besides that verbatim mode seems way better to me.
(said the independent search engine creator with the same air of righteousness as when Plato argued philosophers should be kings)
Same situation with python honestly. No I don’t want to use pandas, if I wanted better data wrangling I’d just wrangle in base R then import the wrangled data into python if need be instead of adding another dependency to my python code. Apparently this makes me a heretic based on google search results.
Since I know the top results are mostly useless spams on some searches, I used to go to page 4 or 5 to find the actual results. Now to achieve the same result I have to click and scroll 5 times.
It still works.
Works exactly how quotes used to.
Actually, it has happened with results that include hacker news discussions. I scrape through and expand EVERY comment on a 6 month old post and still find no partial or exact matches for my search keyword. It is very clear to me that Google has switched behavior. It used to flex on how awesome it was to bring you to an esoteric corner of the internet matching your exact query. Now, it tries to converge your queries into what it thinks you want(not what you asked for) and funnel you into large websites at the expense of utility.
Which can be incredibly frustrating if you know the problem well enough to need a different answer.
I also love when my search is narrow enough that Google instead starts to return more generic pages for totally different programming languages instead of the smaller result set that matches exactly. It is very easy to observe as you explore a topic. At some point, you got a wall where any increase in search specificity seems to yield worse results than queries that helped you get to that point.
Which... Why isn't Google learning that some users are constantly seeking out verbatim for results they'll click on and adjusting their search results to be more verbatim?
Its results are superior, with way less spam and SEO optimized content.
a) there is no giant industry optimizing for ranking on Kagi. There are whole armies of people trying to game google search, meaning whatever heuristics they have would probably be rendered useless shortly if google were to adopt them.
And
b) Kagi has a much smaller, more technical userbase. So like the early web they can get away with a search experience tailored for those kinds of people and not millions of people powering on a smartphone as their first computer.
Kagi works for you, with your input. It doesn’t do “advanced guessing” by implicitly and opaquely processing your interaction with it.
I guess their "Universal Summarizer" is a part of their indexer, so they try to understand what you want and provide you more relevant results.
They probably have other signals and systems to detect spam and other unwanted things.
Combine it with personal website ranking and lenses, it surpasses Google easily.
Also since it has no ads, there’s no noise.
I don’t use Microsoft products, so I can’t comment on Bing.
I suspect this is a niche feature for power-users only. HN is filled with power users, so they love Kagi, but it doesn't necessarily translate to the masses.
I think it’s important to measure and quantify these things.
I can't provide you with side-by-side examples because I don't compile lists of such examples. I'm searching to find stuff, not researching search quality. I just know that I have more problems getting good search results from Google over a couple of the other engines.
If a user of service thinks a different service is better, then it is better for them by definition. "Better" is a subjective thing. If X seems better for me than Y, then it is.
"Are US drivers licenses forgery-proof?" and "Can US Real-ID drivers licenses be forged?"
gave utterly irrelevant #1 or #2 hits: nothing to do with US or Real-ID (zero mentions), it's about New South Wales, Australia (lots of mentions of "NSW"). (although from US magazine Wired.com, but Google can still tell this article is totally irrelevant).
2) I previously in 2022 posted "Google search relevance fail: result for “Africa longitude” [https://news.ycombinator.com/item?id=30337563] where the #1 site hit and text result were the grossly misleading "Africa/Coordinates 8.7832° S, 34.5085° E", which is not a range, or even a centroid, just a junk location in southern Tanzania, where a (Russian?) SEO spam person seemed to have dropped a bogus pin, mislabeled "оф. 3, 27 ул. Хабаровская, Bila Tserkva, Russia". In Tanzania. 17 months later and same gloriously unfixed garbage results.
Other bizarre top hits were the small Ottawa e-commerce digital-map site https://www.mapsofworld.com/lat_long/africa.html, with the incomplete and inaccurate non-sentence "Africa's latitude and longitude lies between 9.1021° N, 18.2812° E." (That e-ecommerce store seems to have an autogenerated page with incorrect information on latitude and missing longitude for each country).
These are cases where Google itself 5 years ago would have done better. Or compare to Kagi.
[1] https://kagi.com/privacy
Some people admittedly expect pretty weird websites to exist. Like who would even publish a good objective comparison between products in a specific category. But people make those queries looking for those types of results, and spammers generate such content to make bucks off the traffic. 20 years ago you'd have gotten nothing in most cases.
Consider coffee. You'll find hundreds of articles comparing the most high-end grinders. You even have PhDs writing white papers on the physics of coffee and how particle size distribution achieved by different grinders affects extraction quality. Those are genuine opinions from aficionados.
The same is true for all kinds of other things. The internet has a lot more than the auto-generated marketing crap you give it credit for.
Sniper in mahwah for example will discuss some of this for example from an HFT angle.
So what do you think, will be the goal of all advertisers?
To get their recommendations baked into that AI. Direct bribing/revenue sharing with the manufactur of that AI. Or sneaking data into training sets via bribed employes and so on.
I don't trust black boxes and never will.
Paid search is the future. Either you pay a subscription fee to something like Kagi, or you pay for hardware and electricity turn run your own "AI agent" locally.
It's a negative-sum arms race, but if Google (or whoever) wasn't spending the money to fuel the arms race by passing on ad money, who would pay for Google to exist in the first place?
That's just my hypothesis, of course. I don't know for certain. But it does make me very skeptical about using AI to help search the web.
I want a robust set of search modifiers instead.
How long do you think it will be before that software agent's output will be just as riddled with commercialized content? I think we might have only a short window where we get to enjoy the fairly unadulterated output of the base LLMs by the large providers (yes, I'm aware that they're screening for safety already, but there could be a lot more coming). Imagine a dialogue like this: Q: What should I watch out for when buying new car tires?
A current answer might be something like: Here's a list of things to watch out for when buying car tires: A, B, C, D, E. Remember to always consult with specialists. Shall I explain more about the topic?
A future answer might look something like: According to experts like the professionals at [tire shop near you], here are the things you need to watch out for when buying car tires: A, B, C. You can use this coupon to get a 10% discount off your next purchase at [tire shop near you]. Shall I book an appointment for a free consultation at [tire shop near you] for you?
I'd love to have one, but do any exist? SGE works on google's behalf, collecting data for google who will use what they learn from it against me. I'm guessing that until we can use common hardware to train/host an AI agent ourselves, we're stuck with terrible search results unless we're willing to hand over more and more data to others who want to exploit us, while pretending we can trust them to return results that are in our best interest instead of theirs. That's not a great future to look forward to.
If you are looking for big search engines, there aren't many really, I strongly believe you should give them all a shot and just alternate between them.
The biggest takeaway from this is that google isn't massively better than the worst alternative anymore and it sticks mostly due to recognition.
At first it wasn't even because their search was that much better than anyone else's. What made google popular was that it wasn't a massive, messy, ad-filled page of things you didn't care about. Other search engines were bloated "portals" that tried to do everything: email, news, shopping mall, weather, etc
Google had a simple clean interface that loaded instantly on a typical dial up connection and that's really what got people to make it their homepage. Today, google has slowly turned into exactly what it was created to solve. Ad-filled bloated pages that try to be everything by pushing things you don't care about in your face.
https://search.brave.com/goggles/discover?goggles_id=https%3...
Technical searches on Google have become a pain in my ass. I'm constantly hit with low effort articles about topics that Google has guessed are similar enough but definitely aren't.
10-15 years ago, Google didn't have the same internet to crawl -- but it also didn't fucking _ignore_ what I asked for.
I was there. We used Altavista and Yahoo and they rocked. Until we tried Google.
You are also misconstruing what I said. I don't say I "don't really know" if it was better. I don't think it was better and seeing people baldly assert the opposite without proof hasn't convinced me my memory is wrong. I'd like to hear something better than "trust me bro." I'm tired of people getting published with evidence-free arguments (though I guess it beats someone condescending to me based on a wrong guess of how old I am).
We're building a new search engine. Of course it's AI driven, but at its core, it's all about human dignity. When you use it, you'll have a voice. If you'd like to try it, please email me: josh@sirch.org
Thanks! Josh
PS. If you'd like to join our team, we have 16 people now (mostly Snapchat/Instagram/Twitter folks), and we'd like to go up to 50 before we raise capital. It's going to be a wild ride.
Just turned off its servers and said "we're closing".
Then some would realise what a huge impact it serves on a daily basis.
Pop culture will never last.
The #1 or #2 hit is utterly irrelevant: nothing to do with US or Real-ID (zero mentions), it's about New South Wales, Australia (lots of mentions of "NSW"):
> ‘Tough to Forge’ Digital Driver’s Licenses Are—Yep—Easy to Forge. Researchers found a litany of security flaws that allow simple, quick, and cheap forgeries in Australia. https://www.wired.com/story/digital-drivers-license-forgery-...
https://cis.org/Report/Americas-Identity-Crisis-Document-Fra...
But that link is a 2002 article at least about the US, but not Real ID.
(any random hit for ""real ID" forgery" in the last 18(!) years would be better)
[0]: https://en.wikipedia.org/wiki/Real_ID_Act
I will also add that the particular link you showed is a 7300 word document from 2002, most of which has nothing to do with ID cards at all. It’s at a college graduate reading level and would take 26 minutes for someone to read according to wordcounter.net.
Google remains the best search engine, period.
Nothing is even close.
The one thing I have noticed recently though is chatgtp is way better for certain queries. Like I wanted to figure which brand of fillings I'd had in my teeth the other month. Googling "brands of dental fillings" was hopeless - it gives all results like "here are a number of different fillings, including: amalgam, composite fillings". No brands. Chatgtp though, first try listed all the brands. I think it was sonicfill by the way - quite cool tooth tech.
- Google Search has evolved significantly from its early days as a simple list of blue links to becoming an encyclopedia, predictive engine, image repository, shopping mall and more that is overloaded with information.
- It has become more difficult to find authoritative answers on Google Search due to an overabundance of sponsored content, prompts, and low-quality keyword-stuffed pages.
- Google is currently undergoing an antitrust trial regarding whether it maintains its search engine monopoly through anticompetitive means such as exclusivity deals rather than having truly superior technology.
- Access to large amounts of user data is very important for powering search engine algorithms through personalization and improving the user experience.
- While Google argues diminishing returns to scale for user data, internal emails show Google engineers acknowledging that scale remains highly important for the company.
- Some feel Google has lost its way and become conservative due to its success, with its corporate bureaucracy stifling acquired companies.
- Google Search has become bloated with advertisements and prioritization of its own services over organic results.
- Search quality may be declining as its algorithms are gamed by low-quality sites and search engine optimization techniques.
- Google's trajectory of scaling up its mission to organize the world's information through acquisitions and exclusivity deals has put pressure on it to keep growing.
- No company can likely grow indefinitely to keep up with the ever-expanding internet without being overwhelmed, as Google Search now demonstrates.
No, it doesn't. It has one master: advertisers. Advertisers are the customer. Users are the product. Search results are the bait.
This is self-evidently true. Your customer is the one who pays for your services. Google's users don't pay, never have, never will, because search is not a product you can sell.
User attention, on the other hand, is a very valuable commodity for which people will pay handsomely.
> No, it doesn't. It has one master: advertisers. Advertisers are the customer. Users are the product. Search results are the bait.
I wouldn't put it that strictly. Users are still masters in that they can continue to use Google or not.
The currencies are different but Google absolutely needs to satisfice two disparate customers.
Users are Google customers in the same way that (say) fish are a fisherman's customer. Fisherman engage in barter transaction with fish, offering them bait (food, or at least the prospect of food) in exchange for allowing themselves to be sold as a product to a third party who gives the fisherman actual money. That is what makes the third party the fisherman's customer and not the fish. Likewise, Google engages in barter transactions with users, offering them bait (useful information, or at least the prospect of useful information) in exchange for allowing their attention to be sold to a third party for actual money. The money is what makes the advertiser Google's actual customer and not the user.
https://kagi.com/summarizer/
When I was using Bing AI or Google Bard to summarize an article, they often refuse to do so because of paywall or some other reasons.
I'm not really sure how they lose that case -- Firefox switched to Yahoo after MS outbid them, and switched back to Google the second they could legally do so and keep the money.
When I think about what I actually use search for... it's highly situational. It depends as much on day-to-day and moment-to-moment context as it does any long-term trend specific to myself.
Whatever "personalization" could be realized over search results, would it really offer that much benefit over improving the search function to sift through and find results that are just more useful generally?
What I mean is there seems to far, far more to be gained from making search results better fit the query than to fit me personally.
even if its only 10 searches a day, it needs something. I am never going to use it unless I can try it out for a while.
> “It’s absolutely not true that scale is not important,” Manber wrote. “We make very good use of everything we get.”
I think that is true, but then the obvious conclusion - if one party doesn't have 90% of the market then the quality of search results will be lower. Successful anti-trust prosecutions against the party with 90% of the market will have a likely outcome of:
1) Reducing the quality of service for everyone.
2) Pushing the 90% market company to be one domiciled somewhere with more reasonable monopoly laws.
Successfully proving that more data -> more scale -> better results is actually implies that one company with 90% market share is desirable. Although I'm still not going to use Google search, I don't think that argument is correct. But the line of attack is not clever.
With a natural monopoly, if you built 2 of them in defiance of the economics of the situation they'd have similar quality of service to the single operator.
> A natural monopoly is a monopoly in an industry in which high infrastructural costs and other barriers to entry relative to the size of the market give the largest supplier in an industry, often the first supplier in a market, an overwhelming advantage over potential competitors. Specifically, an industry is a natural monopoly if the total cost of one firm, producing the total output, is lower than the total cost of two or more firms producing the entire production.
In theory pricing/output in economics is always “quality adjusted” (which is one of the hard things in inflation series)
Yes, traditionally a natural monopoly is one where if you split them, costs go up. With Google, GP is saying if you split them up quality goes down. Either way there’s “consumer harm”. (I guess very directly if the time to search goes up, the economic price of using Google goes up too)
What do?