Fellow kagi user here. I have been very happy with the service so far. Switched over to it when they announced their new pricing model a little while ago. It is more than worth the cost. Even without tweaking, the quality of results is far higher than what I had gotten used to getting from DDG or DDG with !g. The ability to rank domains is amazing though.
Every time I see someone talk about this feature on HN (almost every day) I get jealous, it's such a good idea. When I'm searching (on google) I'm always thinking; when I get Kagi I'm gonna downrank this site, uprank that one etc.
Bitcoin is such a bad payment method, even with Lightning network, because you need to buy a volatile asset and payment processors typically charge huge fees to receive BTC payments.
I didn't mean it in an ironic way if that's what you're referring to. I interpreted your statement to say "I don't want my search history connected to my credit card.", sorry if I misunderstood what you said.
If what you mean is you don't believe they are not connected to your account, I can point you to the FAQ entry on their website.
Kagi has improved immensely and I no longer ever feel the need to check Google against it. It's a great product that I am more than happy to pay $10/mo
I'm so happy that their new pricing scheme has done away with search quotas. Their claim that the average user does 100 searches a month seems absurdly low. I would blow through my 300 search quota in a few days.
That is an interesting list; I'm not at all surprised to see pinterest holding the top ~10 spots, but things like facebook, instagram and twitter being so high does surprise me; perhaps it's indicative of the type of user they have (I too block those domains).
I could take that list, as-is and use it as a block list on my entire network.
I mean when you look at all the raised and pinned domains it's almost entirely programming related topics so "programmers dislike sites that have terrible browser experiences for unauth'd users" is like the least surprising thing.
It's a paid search engine, and not a cheap one. Not exactly mass market material ;) and I'd assume the percentage of HN users is rather high, hence a lot of comments from us, here.
I get the reason why Google does not do domain level filtering due to antitrust concerns, but at least they should be able to adopt this kind of user preference approach. Why aren't they doing this years ago?
Also a kagi user since they changed their pricing. The results have been decent for me for the most part, except for shopping which I still use google for. I do wish they’d step up their css game though. Maybe it’s just safari on iOS, but I get style problems constantly. The address bar doesn’t behave properly with the cursor if you type in a decently long query, and so often the search results will overflow the container horizontally. It’s a small price to pay for good search results, but stuff like that should be table stakes for a tool like this.
They own the search market anyway, and the more time you waste on their platform searching for what you want to find, the more ads you see and the more money Google makes.
What else are those 97% people gonna do, "Google on Bing" instead?
So for them, being bad is actually more profitable than being good, meaning there's a conflict of interest between what Google provides and what their users want, but since there's basically no equivalent competition, they get away with it laughing all the way to the bank.
Where will they go when Android, Chrome, Safari and Firefox all default to Google and plenty of Edge users also switch to Google?
Due to the defaults it's the only search engine most people ever heard of. Think of Plato's cave allegory.
If all your life you've only used Google and never anything else, making it the ground truth for you, how would you know it's bad in order to motivate you to look elsewhere?
LLMs are still far away from the daily mainstream usage Google gets, and since all devices default to Google, and people never change those, they'll stay on Google and only occasionally switch to LLMs for more advanced tasks.
Honestly, I hate to jump on this bandwagon, but I was after some information to help my wife put together some notes on a topic; at first I was coming up with questions we could answer and I'd search for the answers/sources online;
I decided to first try ask an LLM, so I asked Bard some targeted questions and asked for sources, and had all of the answers I needed, conveniently bullet pointed within 3-4 minutes, and all I had to do was go and verify the sources, job done.
GPT4 is the only one I've used much but it really can be good as a Google alternative for certain kinds of questions. I wouldn't trust it for anything obscure and complicated though - if you ask it about something like competitive Pokemon, you'll just get a stream of confidently incorrect junk that any beginner could disprove. (Asking for sources does help this sometimes though.)
For sure, given Google being who they are, Bard is quite good at things like Kubernetes, Golang, things you'd expect a search engine to have indexed, etc.
I haven't used ChatGPT extensively so I can't make a comparison, but with the right questions, I've been able to get great answers to technical things I can't be bothered to look up by hand.
I've done the same for a handful of things I use to default to a search engine for.
My favorite is recipes.
The blog spam around recipes is notoriously bad and is a direct result of Google Search.
Using these LLMs, all I need to do is tell it what I'm looking for in general and it will give me an entire recipe with no extraneous information.
It can handle adding or removing ingredients, substitutions. It can adjust servings. It can flip the recipe to work in a slow cooker or pressure cooker.
I've even had some limited success where I list restrictions based on picky eaters in the house for creating longer meal plans. No search engine can compete with that.
There were apps designed around these use cases. I foresee these sort of nuanced and personalized interactions being a key to drawing people away from search engines.
Absolutely, these sort of things which have varied but somewhat straightforward solutions will excel.
I'm going to give it a go for recipes, because as you say, blog spam is a nightmare. My wife has a lot of cookery books which she likes for her style of cooking, but I'm a little more "ad-hoc".
I get what you're saying, but why hasn't a better search engine emerged yet?
Bing's AI shows some promise, but I can't find anything better than Google. DuckDuckGo is shit. Classic Bing is shit. Brave Search showed promise, but it's also full of spam and with a smaller index. Marginalia showed some promise for smaller websites, but it's small, too.
All of them are unusable for local searches, except for Google, which is where I need search most. That and searching for obscure programming-related error messages.
Kagi doesn't even use it's own index, you're basically paying for a UI making API calls to Bing and Google. How well does changing the ranking work anyway, if you don't have your own index?
I want to believe that if Google were more aligned with users, search would be better. But where is that better search engine to showcase it?
Do a poll on HN. I bet that the vast majority of users here still use Google's search, and you can't blame HN users of not knowing of alternatives.
If Google really keeps its market share due to their monopoly, where is that better option that's being ignored? Tell me and I'll jump on it.
I know it's been mentioned in this thread but Kagi is a gods damn breath of fresh air compared to the current public search engines. They had a short outage recently and having to go back to Google for 30 minutes was painful. I have no idea what their secret sauce is (likely just being different from Google's algo that's SSO'd to death is enough) but it's now the easiest $10 I spend a month.
>I get what you're saying, but why hasn't a better search engine emerged yet?
Although Moore's law brought down the price of hardware and information processing dramatically in the last 20 years, it is still fairly expensive to crawl the Web, index it and rank it. Hardware cost + engineering cost can escalate pretty quickly, unless you decide to have smaller index than major search engines but then users will complain that search results are not good enough.
That's what Microsoft was thinking when they, in practice, stopped developing Internet Explorer (after version 6). That didn't work out too well for them...
> the more time you waste on their platform searching for what you want to find, the more ads you see and the more money Google makes.
That's not the only way - it's in their best interest to rank ad-laden pages higher. That's more ad-impressions, which is how they really make their money.
Lets say a user searches for $FOO. Why on earth would google return the most relevant result if that result is ad-free? They can return the second-most relevant result, and get impressions on both the search-result page and the page that the user sees when they navigate to the first result.
I would also assume they're largely not competitive with each other. Adwords ads are mostly valuable when they've got intent. I.E. I search "5g home internet" and it passes me off to Verizon or AT&T.
Searches that end you up on AdSense (or pages that could have AdSense) probably are much less likely to have intent. So instead they pass you off to something you will read so they can hit you with retargeted ads from the last time you searched with the intent of buying you something.
I am sure there's overlap, but probably not a lot.
I thought the same thing and recently ran across perplexity mentioned in some random tweet. (I know I am late to the party) It blew me away, I use it as my primary search engine now for most things.
Today I searched for a product, google found a local site with the item, I clicked on the link and I was redirected to aliexpress. How they can not catch this?
Internet-scale web crawlers are hard to hide. You can just set up a bunch of domains, and flag any IPs that repeatedly visit them as likely crawlers; then serve malicious content to the rest.
Try going to the URL directly, and try curling the URL with a referrer of Google.
There's a thing that's been happening the last year or two where compromised web servers serve the intended content unless there's a referrer of Google (and maybe Bing?). The site looks normal to the owner, or to crawlers, but when a user clicks through from Google it redirects them to a spam site.
I mostly see it with restaurants getting hijacked by herbal quackery.
You need to learn more about cloaking. This is a fascinating topic. There are many techniques to show different results to Google and a real web user, like checking the user agent, checking the ASN for the originating IP, checking for indications of a real browser, etc.
> We find that only a small portion of product reviews on the web uses affiliate marketing, but the majority of all search results do. [...] We further observe an
inverse relationship between affiliate marketing use and content complexity, and that all search engines fall victim to large-scale affiliate link spam
campaigns.
I think this is an excellent methodology for testing the quality of search results. I would love to see a standard search engine test and scoring system based on this, maybe similar to some of the LLM scoring systems.
Forgive my naivety, but wouldn't a simple way for a search engine (like Kagi) to avoid falling victim here to detect affiliate link programs? There's got to be a small handful of patterns for affiliate link tracking:
1. Domain Interception & HTTP redirects
2. Tracking codes embedded in the URL directly
> Kagi surfaces shopping results featuring unbiased reviews and no affiliate links to help you identify the best product across categories. Top results include discussions focused on helping you find the best item to purchase - you are not bombarded with affiliate links and ads. Continue to scroll and you will see product comparisons across multiple vendors so you can pick what best suites you. Kagi's shopping search will always return a detailed discussion of which product to buy not a competition amongst advertisers promoting where you should buy. Kagi is focused on providing you the best results to make an informed decision not polluted by affiliate links and advertisements.
Kagi is better than Google, but just yesterday it returned the top link to Amazon-linking “check the price for this item on Amazon right now” affiliate spam. So it’s not perfect.
I think it’s pretty good. I understand sometimes we root for the good guys like Kagi and look at their work with rose tinted glasses and even think they’re perfect, like you did, but it’s not really feasible.
A pattern I’ve noticed from people not working in tech, or more junior people in tech, is that they have a thinking that software can ever be perfect. If either of those categories describe you, I can tell you from experience no software is ever perfect!
In adversarial environments, this is true. In non-adversarial environments, software of a certain defined scope can at least get very close to perfection.
Ultimately a site can get around affiliate / referral detection with internal links that redirect though I suppose. So the crawler would need to follow and track all links to detect this. Also sites will surface different links / content based on user agents, so I don't know how effective this is.
Kagi is a paid search engine to avoid the manipulation that Google welcomes. To avoid every trick and targeted tricks it may be cost prohibitive. Fortunately Kagi isn't popular enough yet for targeted manipulations to have a noticeable impact.
On second read, I think what the Kagi doc means is that they don't provide first-party affiliate links (i.e. if you click on an Amazon.com search result it's not using their affiliate). They will still show you search results that have affiliate links embedded (but presumably don't affect or negatively affect ranking).
I am working on a search engine that checks a site for affiliate links from known providers and demotes sites that have a large amount of them.
In addition:
* it demotes sites with popups (think newsletter sign ups)
* it demotes sites that block (or complain about) ad blockers
* it demotes sites with a high number of ads and favors sites with no ads
* it demotes sites using certain sketchy ad companies.
* It demotes sites that have paywalls
* It detects possible link networks and flags them for human review/removal.
* sites with RSS feeds get promoted.
* There is a toggle to hide all sites with ads or external trackers, but it is still WIP (The whole project is).
There are many other features. No idea if I am going to make it public, I created it to update my skillset. I actually thought about setting up a nonprofit and making it open source, but I haven’t decided.
What would be the difference between an affiliate link program and a web ring?
I don't know if discovery is actually a bottleneck to be automated away. It might be the fun part. I'm thinking back to the Napster approach where you could browse other people's libraries for music ideas.
Currently, Kagi has (if you hover/click the shield icon to the right of a result) an indication of the information it knows about a website (as well as a way for you to rank it higher or lower for yourself).
One of these is "ads/trackers". I imagine that it would be feasible for this to include some of the more common affiliate URL types, or third party lead/affiliate tracking bounce hops like awin.
Clearly there will always be some amount of ability to "defeat" this kind of measure by obfuscating links, but eventually the user needs forwarded to a link that has a referral parameter or a site that sets an affiliate cookie or similar.
The "tracker category" also can give a bit of extra information - things like "invasive fingerprinting, advertising"
I don't know if this would be a very long-term solution if the big ones (ok, google) did it. Advertisers would catch on very quickly, and some legitimate review sites which might get funding through affiliate links unrelated to the product being reviewed would lose out to straight-up paid for "reviews" that are funded wholly by the manufacturer and just don't use affiliate links.
Their entire business is gaming metrics. Most of those metrics are ones they invented for themselves, but if there's any advantage at all, they'll game it
this is one way to do it, but I wouldn't say its sufficient. If I search for 'things to do in Seattle', you get many 'blogs' and such that a writer gets paid by sources to insert their place into the things to do list. I didn't word that well, so for example: I own a coffee shop, I pay them moneys, and the '25 things to do in Seattle' writer puts my coffee shop in the list.
If I do an image search for the word 'strawberry', how many of those results are not stock images, images from a store, etc. of a strawberry? can you find an actual picture of a strawberry sitting in the wild? or just some picture of a strawberry a person uploaded without trying to sell you something?
This doesn't apply to the content complexity finding, but the finding that "product reviews which are in top search results are more likely to contain affiliate links than product reviews which are not" can also be explained by the fact that if and only I am getting a bunch of hits on my product reviews, I'm incentivized to monetize that with affiliate links.
We should also ask ourselves if affiliate links are really that bad. Someone could be making honest complete reviews and monetizing those with affiliate links, does that inherently mean that the search results are lower quality?
That approach also misses all the copied-a-github-issue low-effort content that seem to crop up on Google.
It's bad because most of the reviews will be for items you can only get on Amazon or another retailer with an affiliate link program and not for items available elsewhere.
I've seen so many "Best $WHATEVER of $YEAR" and it's really "Best $WHATEVER $PRODUCED_BY_WEIRDLY_NAME_FLY_BY_NIGHT_COMPANY_ON_AMAZON $YEAR"
In theory, if we had truly open marketplaces that offered equal monetization of all links to all similar products in the same category regardless of the price offered to the consumer, sure.
In practice, that is never the case. At best you're getting reviews that only compare products available in the same marketplace (say, for example, all the table saws you can buy on Amazon). At worst, you're getting reviews from vendors who offer the highest payout.
Also, a heck of a lot of consumer products are absolute garbage. (IMHO, most of them, but others may feel differently.) Who is going to write an honest, scathing review of a product and then monetize the link to it? Why even bother?
> that inherently mean that the search results are lower quality?
Almost? There doesn't need to be anything nefarious going on, but human beings align to incentives, basically always. So the trick with reviews is going to be to avoid any incentives coming from the product side, and embrace those coming from the consumer side.
Anecdotally, I can say that I’ve definitely noticed a negative correlation between affiliate marketing and content quality, though I would be very interested in seeing a formal study of it.
Affiliate links create misaligned incentives between content creators and consumers. This includes on the products listed themselves, where the affiliate is incentivized to select the products that give the most kickback rather than the ‘best’ for the consumer. But it also includes the content itself. Affiliate links create incentive to (a) write reviews, where a site wouldn’t have bothered before; (b) churn out lots of content with lots of links, that can be picked up by search engines; and (c) to not invest much in the actual reviews, and instead generate quick, low-quality content, since the content creator doesn’t actually care about finding the “best” product (which takes lots of time and money to do correctly), only on having readers click their links.
This is why sites with names like Celeb Rumor Central have dozens of articles like “top 10 coffee makers 2024”. They just hire a freelancer to do a few quick Google searches for coffee makers, then churn out 10,000 word articles with sections like “the history of coffee” and “why do people drink coffee?” Then the actual review is “this machine is purported to boil water and drip it through coffee grounds, and has high ratings on Amazon (we may earn a small commission when you click a link on our site)”. Increasingly, they’re just using AI to do it, cutting out the cost of a freelancer.
In my personal opinion, affiliate marketing is one of the worst things to happen to the modern web, and the source of a ton of content farm SEO spam pages.
Sure but I feel like you are taking game theory and applying it to real life. See for example https://www.rtings.com/monitor/reviews/best/by-usage/gaming they do a lot of actual reviews with tests and all their links are affiliated. If as a Google user I want a review of Gaming Monitor, having this as the first link would likely be an acceptable-to-good scenario.
There is a lot of money to be captured in paid search with these sites. There are some pretty large players in the space who just built these types of top 5 or top 10 pages for tons of verticals and then bid on keywords related to those verticals. They can go to brands and demand significantly higher fees than most affiliates would get and the brands will usually pay because that acquisition is usually more efficient than try to bid on the terms the affiliate is bidding on.
Back when I was still involved in acquisition marketing, I did a test where I killed all paid search and built relationships with the affiliates to negotiate our "ranking" higher up the page. It was a huge boom for business. We scaled paid acquisition at a profitable CAC, which was very difficult to maintain, let alone scale, in bidding directly and it was significantly less work to manage.
Consumers value this type of content quite a bit, even if many are skeptical of the quality. Sometimes it's nice to just see a pared down list of things with even a cursory rundown of features/differences.
I don't know, feels like a paper titled "Is Google Getting Worse" could have benefited from actually looking at Google results rather than only results of other search engines.
Edit: This got downvoted to hell, so let me be more explicit. This study did not look at Google results, the title is pure clickbait. They used Startpage results as a proxy for Google results. I don't think that's a valid assumption, even if Startpage is using Google's index.
Using a reasonable proxy is not "pure clickbait", though it may be misleading. It adds an additional assumption about the Google results not being tampered with by Startpage, but it seems like a reasonable one, compared to the alternative of Google doctoring the search results when detecting a scraper and/or doing personalization.
If the authors have done their due diligence and confirmed the results from Startpage are actually Google results, then I don't see why they couldn't claim so in their title.
The authors did not do any work to establish it as a reasonable proxy. We know that's the case, because they don't describe any of that work in the paper. they just state that they're using this proxy measure, and then seem to be using Startpage and Google interchangably for the rest of the paper. Sure, if they'd actually done the searches on Google, the title would not be outright dishonest. But they didn't, so this defense of yours is just absurd.
It's pretty obvious why the results won't be the same: the feature sets of the search engines are different despite being based on the same index. Startpage's own documentation even has a page on how and why the results are different!
(Personalization is a part of the Google feature set, and has to be taken into account when considering the results. It's also a part of the Bing feature set. This didn't stop them from reporting the results of their testing on Bing in detail.)
Hi, I'm lead author of the study, so I can give some background on this. This is certainly valid criticism and we could have addressed our reasoning for using Startpage in more detail, but it just didn't fit in the paper anymore.
The reason we used Startpage is simply that it's much easier to scrape. We started off checking only Startpage (as proxy for Google) and DuckDuckGo (as proxy for Bing), since they are both simple to scrape and produce stable rankings. Plain Bing, on the other hand, is a lot trickier. You often get different results for the same query and you get blocked much more easily if you send too many. The only stable way to use Bing is via the (certainly not very cheap) API, though even that wouldn't necessarily guarantee the same user experience as the web frontend.
We did notice, however, that DDG (despite being mostly Bing) did deviate quite a bit in their results, so we started scraping Bing as well for a fairer comparison. As for Startpage, we did check it initially and we found the results to be virtually identical, except for a few minor rank differences here and there (which are probably just geo personalisation). The differences may have become larger now that Startpage also taps Bing to a certain amount, though when I do spot checks, the results are still sufficiently similar and they are also sufficiently different from what actual Bing gives you. Most prominently, Startpage/Google give you a lot more YouTube results, which we did another small spin-off study on (to be published at CHIIR this year https://downloads.webis.de/publications/papers/bevendorff_20...). Moreover, we could also measure certain immediate effects of Google's ranker updates in Startpage, which weren't as apparent in Bing. So we are confident that Startpage is a reasonable proxy for Google, though it's certainly something we will keep in mind for follow-up studies.
I'm sure that using these sites was the thing for your research for all kinds of practical reasons, and you don't need to justify that :) My actual complaint was about the misleading title, which you did not address.
Note that the clickbait title actually hijacked your research; there's like three comments out of 240 here that are about your paper. So while it probably feels good to have received all this attention, it's just bogus engagement. I guess it's pretty meta for a paper about search result spam to do blackhat social media engagement optimization though.
I wouldn't say it's misleading or wrong. It's certainly a bit catchy, but it also aligns with a greater discussion that's going on at the moment. Regardless of the title, most people probably wouldn't or couldn't discuss scientific literature in-depth anyway, but search engine spam is something many can relate to, be it from Google or otherwise. It's certainly a topic we will be hearing about a lot more in the future. Besides, we didn't even anticipate this to be picked up by non-scientific journals that quickly, though the title may have helped. ;-)
I recently caught myself that I unconsciously retry my google searches by adding more keywords. I also noticed that small sites, blogs, forums are gone from google searches.
Now I am using 3 search engines.
Bing for Normal searches.
Google for local(country) searches.
Yandex for small sites, blogs, forums.
Just now I was searching for flares,and after 5 attempts with different keywords I could not even get the Wikipedia page, just ads and blog spam for boating flares. Went to ddg and got informative links and the wiki entry first try.
DDG uses bing search behind the scenes. In terms of the quality of the results, those two should be the same. Bing has more "related" links sprinkled in with its results, but the main results are the same between the two.
It is accurate to say DDG uses bing behind the scenes, but that is not its only source. They have their own crawler (DuckDuckBot[1]), and pull results from many sources[2].
Probably because DDG is shit compared to google. It consistently misinterprets context, it’s shy about domain-specific searches, especially involving pornography, and struggles to ‘correct’ poorly-written search strings.
It doesn’t come with the baggage that google does, ovviously, but let’s not pretend it’s any kind of real competition in terms of functionality.
Funny how I get the opposite result with google, which ignores my quoted text and tries to feed me whatever it believes I want to look up, instead of the exact matches. Especially annoying for code searches.
Have you tried DuckDuckGo? I find it fantastic -- it does both general queries and local country queries well. Anytime I go back to Google (e.g. when I'm setting up a new computer) I am surprised that anyone still uses Google -- the result I am looking for is almost always half way down the page on Google yet it is #1 or nearby on DuckDuckGo. Apparently it's based on Bing's index but with their own tech on top.
Are there stats* of Google Search across the years? I felt I don't use Google as much as I used to. And it isn't because "I know more stuff", but mainly because the way we use internet has changed. I wonder if kids or teens (who most of them don't know how to use an email inbox) would use Google... (I guess yeah?)
* Of course, the stats should include the total amount of internet users globally, or normalize the amount of searches based on that...
Long-term stats are tricky because of how much of the landscape has changed. There's the desktop/mobile split, developing countries increasing their internet use, heavier use of apps, growth and decline of results getting indexed, and change in what we search for.
Definitely I think the way we use the internet has changed profoundly. There are a lot of apps that provide useful information but they may not be made indexable by search engines. Much less useful information is simply out there on the open web, and much of it are locked behind logins. There were previous deals like Twitter sending a completely copy of all new tweets to Google, but these are basically dying.
It's especially interesting since you mentioned normalizing searches by the number of internet users. The country with the largest number of internet users is China, with more than 1 billion of them. And they don't have access to Google. And their local copycat, Baidu, is years behind Google in terms of technological sophistication and simultaneously years ahead of Google in terms of user hostility. So what do Internet users in China do in a post-search world? They simply open various apps and use the full text search feature of different apps. For general knowledge they might open ZhiHu and search there; for something resembling the old-time personal blogs by individual users they might open XiaoHongShu and search there; for short videos they might open Douyin and for long ones Bilibili. For reaching an organization be it a store or a museum or a hospital or a government department they might open WeChat and search there for an official account or mini program (a mini program is a website that uses WeChat APIs and can only be opened in WeChat).
I made these observations on a recent trip to China and it's clear to me what a post-search world looks like because China is already there.
> And their local copycat, Baidu, is years behind Google in terms of technological sophistication and simultaneously years ahead of Google in terms of user hostility.
Baidu search was fine during early days, issue as you hinted was PRC internet went mobile first and content got locked behind various platforms and made deliberately hard to scrape - crawling/indexing got locked much earlier than west. Hence now as more gets locked in west behind logins, western behaviours also shifting towards that model, how much of default search is query + wiki/reddit/youtube or straight into short video services like looking up recipes on XiaoHongShu. Reddit especially, simply because reddit app has horrible search experience. Also technically, Baidu rankdex predated Google PageRank which Larry Page referenced for Pagerank patents. Eitherway, depending on how ChatGPT copyright drama plays out, imo more people will just go the lazy route and have AI generate good enough summaries for most queries.
>I made these observations on a recent trip to China and it's clear to me what a post-search world looks like because China is already there.
You are talking like open web is dead but it's not. There are millions of blogs and personal sites out there. Walled gardens are user hostile and hungry for money, that's why enshittification[0] happens.
The open web is not dead. Unfortunately it will be soon, where "soon" is roughly a decade or so. If you haven't seen the signs of it dying a slow death, you've been hiding under a rock. I totally agree with you about walled gardens and enshittification; but I see no way to stop them.
I've been in web development and SEO for almost 20 years now.
When I first started out all the veterans of SEO kept telling me not to do this, don't do that with things that could get your site buried in the SERP's. At the time Google's algorithm was really good at ferreting out affiliate links, link farms and other nefarious black hat techniques SEO's used to game Google.
Now? Complete opposite. I have several freelance clients and I've used every dirty SEO trick in the book and all of them have worked like magic to get my clients sites ranked on page 1 or 2 of the SERP's.
I have no idea what changed, but Google is super easy to manipulate now to get your site or specific pages ranking really high. I haven't heard or seen any of the horror stories I read and people blogged about constantly when I first started out for years - which tells me they're all probably doing the same thing I am and not seeing any repercussions.
Maybe Google doesn't care because users have become so savvy, they can filter through a ton of garbage in minutes to find what they really want?
I recently googled "rubbish removal mysuburb" and found a pretty good looking website. Let's say it was myrubbishremoval.com/mysuburb. The page text repeatedly named the suburb.
It turns out they appear to have a folder for basically every suburb in the country, thousands of subdirectories yielding the same looking page with that suburb name inserted into the text, making them look local no matter where you are. I can similarly find this with many other industries.
This sort of thing twenty years ago was a huge no no. Google talked about detecting mostly duplicate text and would bury you for it. Like a very basic rule was that if two pages were mostly the same only one would really rank.
Nowadays the "lead funnel" is an entire industry on websites following this pattern, and it clearly works.
Interesting because I've been under the assumption that this kind of "keyword stuffing" was cause issues with ranking so I've always wondered what some of these sites were doing "right"
It's not worse when you append site:reddit.com to every single search but this is only a function of the fact that reddit can't figure out how to build their own search. Outside of maybe programming stuff where I'll still click on links, I don't think google has driven me organically to a new site in years.
> It's not worse when you append site:reddit.com to every single search
Could you give some examples of search queries that would benefit from filtering by reddit?
(My own example: I've been looking for recommendations for a solid Linux laptop. A good result would be a list of reviews written from personal experience of owning such laptops. Reddit was useless for that.)
I've noticed a pattern with 'free' services like Facebook and now Google. With simple tasks like checking emails or finding phone numbers in contacts, there's no friction. However, Facebook lost my trust long ago as it seemed more focused on capturing my attention for ad revenue, leading to time wastage.
I initially trusted Google for its efficient and seemingly fair service, where smart ad-targeting was the price for speed. But now, Google feels similar to Facebook; it's harder to switch to alternatives like Kagi on iPhone due to financial ties with Apple.
This shift in Google's approach, prioritizing trapping attention over genuine service, is frustrating. I'd rather pay for a search service that values my time and provides real utility, than endure the hidden costs of 'free' services.
In last 5 years I have noticed a clear decline in the usability of Google Maps as ... a Map.
Increasingly you have to pick a destination and blindly follow directions, your ability to use it as an informational tool, but exercise your own judgement seems to have been intentionally crippled.
I can’t even blindly follow it. The last two or three times I’ve tried to use Google maps for directions, it has redirected me through walking trails and bike paths. Each time I closed it and just used Apple Maps to get where I needed to go.
> I'd rather pay for a search service that values my time and provides real utility, than endure the hidden costs of 'free' services.
How much would you pay for Google search without ads? Or more importantly, how much would the average Google user be willing to pay?
If you live in the US and click 3-4 ads per month, you're generating ~$10-20/mo in revenue for Google from advertisers.
(Like you, I would also love to pay for an ad free Google, but sadly advertisers are willing to pay Google a lot more money than consumers are willing to)
I do pay $10/mo for search. Also lol "and click 3-4 ads per month" what? This is HN, who clicks ads? I've never clicked an ad. I spend all day trying to teach people to never click ads. At this point I actually couldn't click ads even if I wanted to because uBlock hard-blocks all the domains they redirect through for conversion tracking.
I occasionally click on Sponsored search results, if they're what I'm looking for and I don't much care for the company on the other end of the link. Take that, Big Corporate!
I am not sure if Google has empirically gotten worse or not, but I do know that I very often have to use multiple search engines more often than not now in order to find the information I am looking for.
For me, the biggest quality drop literally happened overnight. It was when they removed the PLUS operator and any other changes that they made that same day. There was many other small changes that made it worst, but that's nothing in comparison. I think that was the day Google became evil.
Oh, what a methodology. They ran with the meme of "best pants" having bad results, when the problem with "best pants" is that it's a nonsense query. I personally cannot recall ever doing a "best pants" query, and the results of such queries don't factor into whether I think a search engine is good.
Google is better than Bing and anything else for my day today work. Few things I miss though :
1. Spam is more than in past. The outrage porn, clickbait headlines etc. are lot more than in past.
2. Dominance of few domains despite poor quality content. For lot of coding related queries, dev.to, hashnode etc. appear in top results despite being clearly spammy.
3. Paywalled content. Most irritating part is sites like medium which appear in top results, have high value content and yet are behind paywall.
Internet is growing and so are Google's problems but I think they are still on top of things.
Google search for topics I'm unfamiliar with/wanting to learn about all lead to low quality, SEO-optimized to hell, clik-baity sites that are just riddled with ads. I have to add "reddit" to most searches just to find semi-relevant content.
But Google search for topics i'm super familiar with and just need a transactional search to look something up tend to be much better and generally the fastest way to accomplish a task.
Right, adding "discussion", "forum" and even "reddit" to the search term increases quality dramatically. Also, Google is still great to search websites. I gave up on Stackoverflow's own search but add "site:stackoverflow.com" to my searches on google.
BTW, maybe someone wants to create a very simple webpage with a search mask that allows adding a few (customizable) terms and options and simply forwards that to Google's search when pressing enter.
I've started using uBlockList to block those sites and it does clean it up quite a bit. I assume those sites also work for some, but they're usually of low quality to me so I block them.
The second type of searches you describe seem to be better so far, but I've stumbled upon a bunch of obviously generated garbage recently. So not sure how long it'll hold.
I've tried things like DDG, YaCy, Bing and others, but often Google is just significantly better (but not necessarily good).
Google Images is filling up with AI slop to the point of unusability for some purposes, even content that isn't actively engaging in SEO shenanigans is getting pushed to the top. In some cases even the key image that Google displays first on a normal search is fake, like when searching for the "tank man" prioritized showing a fake AI selfie from the mans perspective. It's still there in fact, just not the first result anymore, but Google might have demoted it manually after the backlash.
Looking to browse works by a specific artist? Good luck, Google evidently can't tell the difference between a genuine J. C. Leyendecker piece and anything shat out by a "Leyendecker style" image generation model. Search for "Yoji Shinkawa" and this https://i.imgur.com/RYghaoY.png is currently the first thing you see, which isn't even close to his style, but Google has somehow determined it's the image that best represents him. The full images page shows his actual work but interspersed about half-and-half with AI imitations.
My speculation is that Google prioritizes showing recent results, presumed to be the freshest most up-to-date information, but of course for a historical event like Tiananmen Square or an artist who died in 1951, nearly all of the fresh results past a certain point are AI simulacra.
I find DuckDuckGo's[0] (DDG) images feature to be a brilliant replacement for Google Images/Lens. Google Images doesn't let you view the actual resource anymore and disabled that feature years ago. DDG's image feature is crucial since it doesn't bring you to the site itself, it just points to the raw URL of where the asset resides and you can easily download it, without hunting the graphics down on the site itself, although DDG does offer a link to the page where the image was found. The important bit is pressing the 'View File' button.
I doubt a legal case would succeed, and Google would know that. But the types of companies who threaten to sue over deep image linking are also the most likely to threaten not to sign contracts for content sharing with YouTube etc or for ad services. In other words, Google's other businesses create a liability for their search effectiveness.
If I recall right, it was over Getty Images, who were upset that people were right-clicking their images to save them from the image search (with the watermark) instead of clicking through to the page to be prompted to buy it
So rather than just delist Getty and solve the problem, they decided to make their product objectively worse
1. Domain-specific engines is too complicated. Let's have a general purpose search engine.
2. Too many buttons is too complicated. Let's have a single search box.
3. Guessing what you want when you are anonymous is too complicated. We need your search history to tailor your search for you.
Please, search engines, hear me: you can not simplify the needs and wants of 7 billion people and zettabytes of online content into a single little shitty search box! It's too complicated!
At this point the only thing these search engines are good for is for finding the URL of something you already know the name of. It's a phone book. I type Hacker News on google, it tells me its phone number: https://news.ycombinator.com/
#3 is certainly bullshit, but Yahoo failed because of #1. Forcing a user to drill down through hundreds or thousands of categories to do a search is absurd, don't you think?
What other buttons do you want?
But for sure Google has crippled its core product. First they removed the + option, which forced the inclusion of words (their excuse was that it "interfered" with their stupid "Google Plus" product which is now gone). Yes you can use "allintext:" but come on. I'm not even sure that's honored anyway.
And the removal of the ability to exclude certain sites from results.
I think the search box was always there, but for some reason it seemed that you needed to select a category...
OK, I just looked up their old layout, and they buried the search bar amongst so much crap that it wasn't clear whether it pertained to the ad or the thing above it, or what... if you even noticed it: https://helios-i.mashable.com/imagery/longforms/04ILIeAX3JAF...
For output to be accurate, input has to be precise. But there's a trend of dumbing down and minifying interfaces and then tacking AI to guess what the user really means from the little input he is allowed.
It will never work. It can't work. Anyone who thinks this can work is delusional. And with Google it's particularly obvious how stupid the trend is.
Look at the "tabs" Google has for search: all, images, videos, shopping, news, etc. This is things users can input. But wait! What if an user wants news about something? And they have to reach all the way out to the news tab. That's too much for our bubbling moronic users to manage! They can't into computers. They have room temperature IQ. They have never used Google before, so they don't know where the tab is. They probably don't know what tabs or links are either. I know what I'll do. I'll put an AI to reorder the TABS of my users based on their search history, query input, season of the year, and their zodiac sign based on what birth day they used when they signed up for a Google account. That should solve it.
And now the order of the tabs is all over the place and when you want to click the "images" tab it's sometimes not the second tab and when you want to click the "videos" tab it's sometimes not the third tab.
I think this is very interesting because you have to think. If Google can fail this hard at tabs, which is not really a complicated thing to program, imagine how hard they are failing at indexing the entire interweb. Imagine if they are doing to search results the same nonsense bullshit they are doing to the tabs. Just imagine it. It's clear they have absolutely no idea what they're doing with the tabs.
>And now the order of the tabs is all over the place and when you want to click the "images" tab it's sometimes not the second tab and when you want to click the "videos" tab it's sometimes not the third tab.
I also noticed this; they redesigned their search tabs few months ago and now it's sort of bad and user unfriendly. Sometimes there is "News" tab, sometimes there is no news tab and tabs look all the same and generic(white rectangles with black text).
>I think this is very interesting because you have to think. If Google can fail this hard at tabs, which is not really a complicated thing to program, imagine how hard they are failing at indexing the entire interweb. Imagine if they are doing to search results the same nonsense bullshit they are doing to the tabs.
God knows what is in their index and what is not in their index. I think every search engine needs to make its index open and transparent. And yea Google can fail with all sort of things, they are not ubermensch or something like that.
Google's ranking algorithms and search technology are millions of LOC and not even Google engineers know how exactly Google Search works.
I wonder if "affiliate links" is a reasonable proxy for page quality? I wouldn't use it directly in a ranking function but it might be a nice automated way to estimate whether results are good.
In a good way or bad way? Affiliate links might bias reviews, but their presence might mean the review was high enough quality that enough people see it and click on affiliate links.
Google created this problem by slurping up all advertising money from open web activity for themselves.
The more market share of online display advertising they gained, the worse their results got.
Why? Because the only way to have a large volume of authentic content being produced at scale is to have a healthy ecosystem of independent sites that are profitable based on display ads.
As much as HN-types hate advertising, it was literally the only thing that made the web of yesteryear so special. Things like Adsense enabled blogs on tons of niche topics to be monetized and thus we had better open web content.
When Google decided there was more money to be made off ads before you even clicked on a search result, that ironically was what ended up killing search.
Now the only way to monetize content from search is via shill company blogs, affiliate marketing listicles (10 best dog toys), etc. So that’s what we get.
For example, if there was a passionate person creating authentic, amazing content about dogs, they wouldn’t even crack page 1 on any search for dog toys no matter how good their content is.
>For example, if there was a passionate person creating authentic, amazing content about dogs, they wouldn’t even crack page 1 on any search for dog toys no matter how good their content is.
So basically the problem of search engines and Google in particular is discovery. All early Google adapters say that they loved Google because it gave them relevant results and because they discovered new websites on Google. Nowadays they "discover" SEO spam, ads and the usual suspects like the most popular sites in that search category.
That's why we need "discovery engine" for Web, something like TikTok but adapted to Web. If search engines were the evolution of web directories, we need to think about how we can evolve search engines.
I visited a small country in my last vacation. I ended up bringing a bit of money back because I was in a hurry I thought it would be easy to exchange back at home, even at worse rates. I live in a big city after all. Of course, I was very wrong.
I spent an afternoon Googling every possible incantation only to get useless AI generated text, travel agency sites or simply unrelated content.
I was about to accept my loss when I tried Kagi. The first page showed an exchange that accepted the currency. Very far from me and with terrible rates, but still.
Anecdata and all, but the fact is that I'm using Kagi more and more and it's winning my trust and good will fast.
It would have been an expensive souvenir. And I don’t travel all that much. Also, inflation and even outright currency replacements are somewhat common in smaller economies.
You mean my comment? It’s not. I have no relation with them other than a free account.
I think 2005-2010 Google was peak web search. It would have given me an obscure blog with the top 5 currency exchange offices. Kagi gave me one single crappy result, but it was still, sadly, better than what 2024 Google can do.
Why would Kagi pay someone to promote them on HN when there's already dozens of praise everyday here?
Even as a happy customer, I can see how suspect this praise can be, but remember that HN is exactly who Kagi was created for.
And I don't remember having spend $10 a month on something that had that much impact on my daily life.
It's _just_ a search engine, but I do about 50 search a day, and having great results each time while the other one is giving me more and more garbage results is huge.
So I completely understand why people satisfied with Kagi can be noisy about how great they think it is.
As an aside, many full service banks can exchange foreign currency for you. For example, Bank of America does (https://www.bankofamerica.com/foreign-exchange/exchange-rate...). You can also order foreign currency prior to your trip and they'll mail it to you.
Interestingly in my country many full service banks have dropped foriegn currency services. They did that during the panademic but never returned back to offering that again. I suspect they make so much money from Visa/Mastercard commissions that they'd prefer people paid with card rather than cash.
Right, it’s the same here. I learned that the hard way. If I ever need cash for traveling again, this is the way. I much rather use credit card if I can, however. But this place didn’t even had cell phone signal, so accepting anything other than cash was impossible.
I don't know if the overall search quality has degraded or not, but SEO has been become definitely a much more severe issue than ever before. Google is the main target for this attack for obvious reasons but no other search engines are really immune to this. I'm skeptical if this can be tackled by any technical solutions; the problem is not just a specific type of SEO spamming but the structure where the enemies are constantly optimizing against your fundamental goal.
I recommend not just bookmarking, but downloading a copy. Either a webarchive file or manually extract the main content from there (Reader mode in browsers is great help). Link rots happen. More than half of my bookmarks don't work.
Yes!!! I've been playing around with SingleFile and other similar software to try and hoard as much useful data as I can. And I've been mocking preppers for all these years.
An acquaintance of my second cousin has experimented with SingleFile CLI (https://github.com/gildas-lormeau/single-file-cli) and she reports it works great (it is rather heavyweight because it does invoke headless Chromium or even Firefox if you set it up right).
Google search went from critical for me to irrelevant overnight with ChatGPT 4.
Like, literally irrelevant. I'm still watching YouTube, using Gmail, and occasionally checking out something on Google Shopping if I want to find something locally instead of on Amazon. But I use Google search about 90% less now than I did a year ago.
The worst thing Google allow phishing sites in sponsors. There are many cases of people searching for a website and lands on the phishing site instead, getting credit card details stolen.
296 comments
[ 0.21 ms ] story [ 249 ms ] threadThen I also use DEVONagent from time to time.
https://www.devontechnologies.com/apps/devonagent
But often I just follow links from „trustworthy“ sources like HN for example.
[0] https://blog.kagi.com/kagi-features
Pay with Bitcoin [1].
[1] https://blog.kagi.com/accepting-paypal-bitcoin
If Kagi supported USDC I would definitely use it.
I’m not letting perfect be the enemy of good.
If what you mean is you don't believe they are not connected to your account, I can point you to the FAQ entry on their website.
https://help.kagi.com/kagi/faq/faq.html#what-data-does-kagi-...
I'm so happy that their new pricing scheme has done away with search quotas. Their claim that the average user does 100 searches a month seems absurdly low. I would blow through my 300 search quota in a few days.
I could take that list, as-is and use it as a block list on my entire network.
They own the search market anyway, and the more time you waste on their platform searching for what you want to find, the more ads you see and the more money Google makes.
What else are those 97% people gonna do, "Google on Bing" instead?
So for them, being bad is actually more profitable than being good, meaning there's a conflict of interest between what Google provides and what their users want, but since there's basically no equivalent competition, they get away with it laughing all the way to the bank.
Due to the defaults it's the only search engine most people ever heard of. Think of Plato's cave allegory.
If all your life you've only used Google and never anything else, making it the ground truth for you, how would you know it's bad in order to motivate you to look elsewhere?
LLMs
I decided to first try ask an LLM, so I asked Bard some targeted questions and asked for sources, and had all of the answers I needed, conveniently bullet pointed within 3-4 minutes, and all I had to do was go and verify the sources, job done.
I haven't used ChatGPT extensively so I can't make a comparison, but with the right questions, I've been able to get great answers to technical things I can't be bothered to look up by hand.
My favorite is recipes.
The blog spam around recipes is notoriously bad and is a direct result of Google Search.
Using these LLMs, all I need to do is tell it what I'm looking for in general and it will give me an entire recipe with no extraneous information.
It can handle adding or removing ingredients, substitutions. It can adjust servings. It can flip the recipe to work in a slow cooker or pressure cooker.
I've even had some limited success where I list restrictions based on picky eaters in the house for creating longer meal plans. No search engine can compete with that.
There were apps designed around these use cases. I foresee these sort of nuanced and personalized interactions being a key to drawing people away from search engines.
I'm going to give it a go for recipes, because as you say, blog spam is a nightmare. My wife has a lot of cookery books which she likes for her style of cooking, but I'm a little more "ad-hoc".
When you have +95% of the market, you can merely be good enough that users don't leave.
In this case, until Google literally starts serving ads-only and links to other google products (Hi Chrome!), they are not going to lose any money.
Bing's AI shows some promise, but I can't find anything better than Google. DuckDuckGo is shit. Classic Bing is shit. Brave Search showed promise, but it's also full of spam and with a smaller index. Marginalia showed some promise for smaller websites, but it's small, too.
All of them are unusable for local searches, except for Google, which is where I need search most. That and searching for obscure programming-related error messages.
Kagi doesn't even use it's own index, you're basically paying for a UI making API calls to Bing and Google. How well does changing the ranking work anyway, if you don't have your own index?
I want to believe that if Google were more aligned with users, search would be better. But where is that better search engine to showcase it?
Do a poll on HN. I bet that the vast majority of users here still use Google's search, and you can't blame HN users of not knowing of alternatives.
If Google really keeps its market share due to their monopoly, where is that better option that's being ignored? Tell me and I'll jump on it.
Although Moore's law brought down the price of hardware and information processing dramatically in the last 20 years, it is still fairly expensive to crawl the Web, index it and rank it. Hardware cost + engineering cost can escalate pretty quickly, unless you decide to have smaller index than major search engines but then users will complain that search results are not good enough.
That's not the only way - it's in their best interest to rank ad-laden pages higher. That's more ad-impressions, which is how they really make their money.
Lets say a user searches for $FOO. Why on earth would google return the most relevant result if that result is ad-free? They can return the second-most relevant result, and get impressions on both the search-result page and the page that the user sees when they navigate to the first result.
Searches that end you up on AdSense (or pages that could have AdSense) probably are much less likely to have intent. So instead they pass you off to something you will read so they can hit you with retargeted ads from the last time you searched with the intent of buying you something.
I am sure there's overlap, but probably not a lot.
There's a thing that's been happening the last year or two where compromised web servers serve the intended content unless there's a referrer of Google (and maybe Bing?). The site looks normal to the owner, or to crawlers, but when a user clicks through from Google it redirects them to a spam site.
I mostly see it with restaurants getting hijacked by herbal quackery.
I think this is an excellent methodology for testing the quality of search results. I would love to see a standard search engine test and scoring system based on this, maybe similar to some of the LLM scoring systems.
1. Domain Interception & HTTP redirects 2. Tracking codes embedded in the URL directly
> Kagi surfaces shopping results featuring unbiased reviews and no affiliate links to help you identify the best product across categories. Top results include discussions focused on helping you find the best item to purchase - you are not bombarded with affiliate links and ads. Continue to scroll and you will see product comparisons across multiple vendors so you can pick what best suites you. Kagi's shopping search will always return a detailed discussion of which product to buy not a competition amongst advertisers promoting where you should buy. Kagi is focused on providing you the best results to make an informed decision not polluted by affiliate links and advertisements.
1. https://help.kagi.com/kagi/features/shopping.html
A pattern I’ve noticed from people not working in tech, or more junior people in tech, is that they have a thinking that software can ever be perfect. If either of those categories describe you, I can tell you from experience no software is ever perfect!
https://i.imgur.com/vPRpBYJ.mp4 (yes the recording is very broken but you can still see)
When I hover over the "i" icon it says:
> Information provided by Looria, Amazon, Reddit and other sources. Shop confidently with Kagi - our results have no affiliate links.
The sites featured in the "Shopping" widget were:
- Reddit
- RTINGS: has 8 affiliate links (just from one article I picked, it was featured several times)
- NYTimes: has 47 affiliate links (just from one article I picked, it was featured several times)
- Techgearlab: has 9 affiliate links
Even if you ignored the widget, the first result was a site with 7 amazon affiliate links.
In addition:
* it demotes sites with popups (think newsletter sign ups)
* it demotes sites that block (or complain about) ad blockers
* it demotes sites with a high number of ads and favors sites with no ads
* it demotes sites using certain sketchy ad companies.
* It demotes sites that have paywalls
* It detects possible link networks and flags them for human review/removal.
* sites with RSS feeds get promoted.
* There is a toggle to hide all sites with ads or external trackers, but it is still WIP (The whole project is).
There are many other features. No idea if I am going to make it public, I created it to update my skillset. I actually thought about setting up a nonprofit and making it open source, but I haven’t decided.
I don't know if discovery is actually a bottleneck to be automated away. It might be the fun part. I'm thinking back to the Napster approach where you could browse other people's libraries for music ideas.
One of these is "ads/trackers". I imagine that it would be feasible for this to include some of the more common affiliate URL types, or third party lead/affiliate tracking bounce hops like awin.
Clearly there will always be some amount of ability to "defeat" this kind of measure by obfuscating links, but eventually the user needs forwarded to a link that has a referral parameter or a site that sets an affiliate cookie or similar.
The "tracker category" also can give a bit of extra information - things like "invasive fingerprinting, advertising"
If I do an image search for the word 'strawberry', how many of those results are not stock images, images from a store, etc. of a strawberry? can you find an actual picture of a strawberry sitting in the wild? or just some picture of a strawberry a person uploaded without trying to sell you something?
That approach also misses all the copied-a-github-issue low-effort content that seem to crop up on Google.
I've seen so many "Best $WHATEVER of $YEAR" and it's really "Best $WHATEVER $PRODUCED_BY_WEIRDLY_NAME_FLY_BY_NIGHT_COMPANY_ON_AMAZON $YEAR"
In practice, that is never the case. At best you're getting reviews that only compare products available in the same marketplace (say, for example, all the table saws you can buy on Amazon). At worst, you're getting reviews from vendors who offer the highest payout.
Also, a heck of a lot of consumer products are absolute garbage. (IMHO, most of them, but others may feel differently.) Who is going to write an honest, scathing review of a product and then monetize the link to it? Why even bother?
Almost? There doesn't need to be anything nefarious going on, but human beings align to incentives, basically always. So the trick with reviews is going to be to avoid any incentives coming from the product side, and embrace those coming from the consumer side.
Affiliate links create misaligned incentives between content creators and consumers. This includes on the products listed themselves, where the affiliate is incentivized to select the products that give the most kickback rather than the ‘best’ for the consumer. But it also includes the content itself. Affiliate links create incentive to (a) write reviews, where a site wouldn’t have bothered before; (b) churn out lots of content with lots of links, that can be picked up by search engines; and (c) to not invest much in the actual reviews, and instead generate quick, low-quality content, since the content creator doesn’t actually care about finding the “best” product (which takes lots of time and money to do correctly), only on having readers click their links.
This is why sites with names like Celeb Rumor Central have dozens of articles like “top 10 coffee makers 2024”. They just hire a freelancer to do a few quick Google searches for coffee makers, then churn out 10,000 word articles with sections like “the history of coffee” and “why do people drink coffee?” Then the actual review is “this machine is purported to boil water and drip it through coffee grounds, and has high ratings on Amazon (we may earn a small commission when you click a link on our site)”. Increasingly, they’re just using AI to do it, cutting out the cost of a freelancer.
In my personal opinion, affiliate marketing is one of the worst things to happen to the modern web, and the source of a ton of content farm SEO spam pages.
Back when I was still involved in acquisition marketing, I did a test where I killed all paid search and built relationships with the affiliates to negotiate our "ranking" higher up the page. It was a huge boom for business. We scaled paid acquisition at a profitable CAC, which was very difficult to maintain, let alone scale, in bidding directly and it was significantly less work to manage.
Consumers value this type of content quite a bit, even if many are skeptical of the quality. Sometimes it's nice to just see a pared down list of things with even a cursory rundown of features/differences.
Edit: This got downvoted to hell, so let me be more explicit. This study did not look at Google results, the title is pure clickbait. They used Startpage results as a proxy for Google results. I don't think that's a valid assumption, even if Startpage is using Google's index.
If the authors have done their due diligence and confirmed the results from Startpage are actually Google results, then I don't see why they couldn't claim so in their title.
It's pretty obvious why the results won't be the same: the feature sets of the search engines are different despite being based on the same index. Startpage's own documentation even has a page on how and why the results are different!
(Personalization is a part of the Google feature set, and has to be taken into account when considering the results. It's also a part of the Bing feature set. This didn't stop them from reporting the results of their testing on Bing in detail.)
The reason we used Startpage is simply that it's much easier to scrape. We started off checking only Startpage (as proxy for Google) and DuckDuckGo (as proxy for Bing), since they are both simple to scrape and produce stable rankings. Plain Bing, on the other hand, is a lot trickier. You often get different results for the same query and you get blocked much more easily if you send too many. The only stable way to use Bing is via the (certainly not very cheap) API, though even that wouldn't necessarily guarantee the same user experience as the web frontend.
We did notice, however, that DDG (despite being mostly Bing) did deviate quite a bit in their results, so we started scraping Bing as well for a fairer comparison. As for Startpage, we did check it initially and we found the results to be virtually identical, except for a few minor rank differences here and there (which are probably just geo personalisation). The differences may have become larger now that Startpage also taps Bing to a certain amount, though when I do spot checks, the results are still sufficiently similar and they are also sufficiently different from what actual Bing gives you. Most prominently, Startpage/Google give you a lot more YouTube results, which we did another small spin-off study on (to be published at CHIIR this year https://downloads.webis.de/publications/papers/bevendorff_20...). Moreover, we could also measure certain immediate effects of Google's ranker updates in Startpage, which weren't as apparent in Bing. So we are confident that Startpage is a reasonable proxy for Google, though it's certainly something we will keep in mind for follow-up studies.
I'm sure that using these sites was the thing for your research for all kinds of practical reasons, and you don't need to justify that :) My actual complaint was about the misleading title, which you did not address.
Note that the clickbait title actually hijacked your research; there's like three comments out of 240 here that are about your paper. So while it probably feels good to have received all this attention, it's just bogus engagement. I guess it's pretty meta for a paper about search result spam to do blackhat social media engagement optimization though.
Now I am using 3 search engines.
Bing for Normal searches. Google for local(country) searches. Yandex for small sites, blogs, forums.
Google is garbage, has been for quite some time.
1. https://duckduckgo.com/duckduckgo-help-pages/results/duckduc...
2. https://duckduckgo.com/duckduckgo-help-pages/results/sources...
It doesn’t come with the baggage that google does, ovviously, but let’s not pretend it’s any kind of real competition in terms of functionality.
Edit: They aren't likely to add it. https://kagifeedback.org/d/687-implement-regional-pricing
* Of course, the stats should include the total amount of internet users globally, or normalize the amount of searches based on that...
It's especially interesting since you mentioned normalizing searches by the number of internet users. The country with the largest number of internet users is China, with more than 1 billion of them. And they don't have access to Google. And their local copycat, Baidu, is years behind Google in terms of technological sophistication and simultaneously years ahead of Google in terms of user hostility. So what do Internet users in China do in a post-search world? They simply open various apps and use the full text search feature of different apps. For general knowledge they might open ZhiHu and search there; for something resembling the old-time personal blogs by individual users they might open XiaoHongShu and search there; for short videos they might open Douyin and for long ones Bilibili. For reaching an organization be it a store or a museum or a hospital or a government department they might open WeChat and search there for an official account or mini program (a mini program is a website that uses WeChat APIs and can only be opened in WeChat).
I made these observations on a recent trip to China and it's clear to me what a post-search world looks like because China is already there.
Baidu search was fine during early days, issue as you hinted was PRC internet went mobile first and content got locked behind various platforms and made deliberately hard to scrape - crawling/indexing got locked much earlier than west. Hence now as more gets locked in west behind logins, western behaviours also shifting towards that model, how much of default search is query + wiki/reddit/youtube or straight into short video services like looking up recipes on XiaoHongShu. Reddit especially, simply because reddit app has horrible search experience. Also technically, Baidu rankdex predated Google PageRank which Larry Page referenced for Pagerank patents. Eitherway, depending on how ChatGPT copyright drama plays out, imo more people will just go the lazy route and have AI generate good enough summaries for most queries.
You are talking like open web is dead but it's not. There are millions of blogs and personal sites out there. Walled gardens are user hostile and hungry for money, that's why enshittification[0] happens.
[0] https://en.wikipedia.org/wiki/Enshittification
When I first started out all the veterans of SEO kept telling me not to do this, don't do that with things that could get your site buried in the SERP's. At the time Google's algorithm was really good at ferreting out affiliate links, link farms and other nefarious black hat techniques SEO's used to game Google.
Now? Complete opposite. I have several freelance clients and I've used every dirty SEO trick in the book and all of them have worked like magic to get my clients sites ranked on page 1 or 2 of the SERP's.
I have no idea what changed, but Google is super easy to manipulate now to get your site or specific pages ranking really high. I haven't heard or seen any of the horror stories I read and people blogged about constantly when I first started out for years - which tells me they're all probably doing the same thing I am and not seeing any repercussions.
Maybe Google doesn't care because users have become so savvy, they can filter through a ton of garbage in minutes to find what they really want?
It turns out they appear to have a folder for basically every suburb in the country, thousands of subdirectories yielding the same looking page with that suburb name inserted into the text, making them look local no matter where you are. I can similarly find this with many other industries.
This sort of thing twenty years ago was a huge no no. Google talked about detecting mostly duplicate text and would bury you for it. Like a very basic rule was that if two pages were mostly the same only one would really rank.
Nowadays the "lead funnel" is an entire industry on websites following this pattern, and it clearly works.
Quick: If you want to search on what is the best toaster to buy, what URL do you type?
How do you find out the weather?
How do you find a local brewery in your area?
For me the first answer is Amazon. And the second is ask Siri. The third is Apple Maps.
Google is now below 50% of my search terms. Heck, I’m more likely to search Reddit for some queries.
Could you give some examples of search queries that would benefit from filtering by reddit?
(My own example: I've been looking for recommendations for a solid Linux laptop. A good result would be a list of reviews written from personal experience of owning such laptops. Reddit was useless for that.)
People actually discussing how they did this is the only real answer, generic websites with pictures and descriptions of his weapons never help.
I initially trusted Google for its efficient and seemingly fair service, where smart ad-targeting was the price for speed. But now, Google feels similar to Facebook; it's harder to switch to alternatives like Kagi on iPhone due to financial ties with Apple.
This shift in Google's approach, prioritizing trapping attention over genuine service, is frustrating. I'd rather pay for a search service that values my time and provides real utility, than endure the hidden costs of 'free' services.
Increasingly you have to pick a destination and blindly follow directions, your ability to use it as an informational tool, but exercise your own judgement seems to have been intentionally crippled.
How much would you pay for Google search without ads? Or more importantly, how much would the average Google user be willing to pay?
If you live in the US and click 3-4 ads per month, you're generating ~$10-20/mo in revenue for Google from advertisers.
(Like you, I would also love to pay for an ad free Google, but sadly advertisers are willing to pay Google a lot more money than consumers are willing to)
Today? Nah. Search has become bad enough that I don't believe it's worth any cash.
1. Spam is more than in past. The outrage porn, clickbait headlines etc. are lot more than in past.
2. Dominance of few domains despite poor quality content. For lot of coding related queries, dev.to, hashnode etc. appear in top results despite being clearly spammy.
3. Paywalled content. Most irritating part is sites like medium which appear in top results, have high value content and yet are behind paywall.
Internet is growing and so are Google's problems but I think they are still on top of things.
Google search for topics I'm unfamiliar with/wanting to learn about all lead to low quality, SEO-optimized to hell, clik-baity sites that are just riddled with ads. I have to add "reddit" to most searches just to find semi-relevant content.
But Google search for topics i'm super familiar with and just need a transactional search to look something up tend to be much better and generally the fastest way to accomplish a task.
BTW, maybe someone wants to create a very simple webpage with a search mask that allows adding a few (customizable) terms and options and simply forwards that to Google's search when pressing enter.
The second type of searches you describe seem to be better so far, but I've stumbled upon a bunch of obviously generated garbage recently. So not sure how long it'll hold.
I've tried things like DDG, YaCy, Bing and others, but often Google is just significantly better (but not necessarily good).
Looking to browse works by a specific artist? Good luck, Google evidently can't tell the difference between a genuine J. C. Leyendecker piece and anything shat out by a "Leyendecker style" image generation model. Search for "Yoji Shinkawa" and this https://i.imgur.com/RYghaoY.png is currently the first thing you see, which isn't even close to his style, but Google has somehow determined it's the image that best represents him. The full images page shows his actual work but interspersed about half-and-half with AI imitations.
My speculation is that Google prioritizes showing recent results, presumed to be the freshest most up-to-date information, but of course for a historical event like Tiananmen Square or an artist who died in 1951, nearly all of the fresh results past a certain point are AI simulacra.
Another search to try is "hand reference", tons of AI garbage
[0] https://duckduckgo.com/
Or they were at least threatened with legal action. I remember when it happened many of my friends were annoyed overnight
So rather than just delist Getty and solve the problem, they decided to make their product objectively worse
1. Domain-specific engines is too complicated. Let's have a general purpose search engine.
2. Too many buttons is too complicated. Let's have a single search box.
3. Guessing what you want when you are anonymous is too complicated. We need your search history to tailor your search for you.
Please, search engines, hear me: you can not simplify the needs and wants of 7 billion people and zettabytes of online content into a single little shitty search box! It's too complicated!
At this point the only thing these search engines are good for is for finding the URL of something you already know the name of. It's a phone book. I type Hacker News on google, it tells me its phone number: https://news.ycombinator.com/
Finding my preferred UI https://hckrnews.com/ that's not even in the first page of results. Rajat's version https://hckrnws.com/... nowhere to be seen.
Google is becoming a mediocre phone book.
What other buttons do you want?
But for sure Google has crippled its core product. First they removed the + option, which forced the inclusion of words (their excuse was that it "interfered" with their stupid "Google Plus" product which is now gone). Yes you can use "allintext:" but come on. I'm not even sure that's honored anyway.
And the removal of the ability to exclude certain sites from results.
Are you saying that's how Yahoo worked? IIRC, that was optional, they always had the search box:
https://web.archive.org/web/19961023235123/http://www10.yaho...
OK, I just looked up their old layout, and they buried the search bar amongst so much crap that it wasn't clear whether it pertained to the ad or the thing above it, or what... if you even noticed it: https://helios-i.mashable.com/imagery/longforms/04ILIeAX3JAF...
It will never work. It can't work. Anyone who thinks this can work is delusional. And with Google it's particularly obvious how stupid the trend is.
Look at the "tabs" Google has for search: all, images, videos, shopping, news, etc. This is things users can input. But wait! What if an user wants news about something? And they have to reach all the way out to the news tab. That's too much for our bubbling moronic users to manage! They can't into computers. They have room temperature IQ. They have never used Google before, so they don't know where the tab is. They probably don't know what tabs or links are either. I know what I'll do. I'll put an AI to reorder the TABS of my users based on their search history, query input, season of the year, and their zodiac sign based on what birth day they used when they signed up for a Google account. That should solve it.
And now the order of the tabs is all over the place and when you want to click the "images" tab it's sometimes not the second tab and when you want to click the "videos" tab it's sometimes not the third tab.
I think this is very interesting because you have to think. If Google can fail this hard at tabs, which is not really a complicated thing to program, imagine how hard they are failing at indexing the entire interweb. Imagine if they are doing to search results the same nonsense bullshit they are doing to the tabs. Just imagine it. It's clear they have absolutely no idea what they're doing with the tabs.
I also noticed this; they redesigned their search tabs few months ago and now it's sort of bad and user unfriendly. Sometimes there is "News" tab, sometimes there is no news tab and tabs look all the same and generic(white rectangles with black text).
>I think this is very interesting because you have to think. If Google can fail this hard at tabs, which is not really a complicated thing to program, imagine how hard they are failing at indexing the entire interweb. Imagine if they are doing to search results the same nonsense bullshit they are doing to the tabs.
God knows what is in their index and what is not in their index. I think every search engine needs to make its index open and transparent. And yea Google can fail with all sort of things, they are not ubermensch or something like that.
Google's ranking algorithms and search technology are millions of LOC and not even Google engineers know how exactly Google Search works.
> like when searching for the "tank man" prioritized showing a fake AI selfie from the mans perspective
In this case the label is that it's a AI-generated and the source is an article talking about that AI-generated picture.
The more market share of online display advertising they gained, the worse their results got.
Why? Because the only way to have a large volume of authentic content being produced at scale is to have a healthy ecosystem of independent sites that are profitable based on display ads.
As much as HN-types hate advertising, it was literally the only thing that made the web of yesteryear so special. Things like Adsense enabled blogs on tons of niche topics to be monetized and thus we had better open web content.
When Google decided there was more money to be made off ads before you even clicked on a search result, that ironically was what ended up killing search.
Now the only way to monetize content from search is via shill company blogs, affiliate marketing listicles (10 best dog toys), etc. So that’s what we get.
For example, if there was a passionate person creating authentic, amazing content about dogs, they wouldn’t even crack page 1 on any search for dog toys no matter how good their content is.
So basically the problem of search engines and Google in particular is discovery. All early Google adapters say that they loved Google because it gave them relevant results and because they discovered new websites on Google. Nowadays they "discover" SEO spam, ads and the usual suspects like the most popular sites in that search category.
That's why we need "discovery engine" for Web, something like TikTok but adapted to Web. If search engines were the evolution of web directories, we need to think about how we can evolve search engines.
I spent an afternoon Googling every possible incantation only to get useless AI generated text, travel agency sites or simply unrelated content.
I was about to accept my loss when I tried Kagi. The first page showed an exchange that accepted the currency. Very far from me and with terrible rates, but still.
Anecdata and all, but the fact is that I'm using Kagi more and more and it's winning my trust and good will fast.
I think 2005-2010 Google was peak web search. It would have given me an obscure blog with the top 5 currency exchange offices. Kagi gave me one single crappy result, but it was still, sadly, better than what 2024 Google can do.
Even as a happy customer, I can see how suspect this praise can be, but remember that HN is exactly who Kagi was created for. And I don't remember having spend $10 a month on something that had that much impact on my daily life. It's _just_ a search engine, but I do about 50 search a day, and having great results each time while the other one is giving me more and more garbage results is huge. So I completely understand why people satisfied with Kagi can be noisy about how great they think it is.
I am able to run it in attick on raspberry pi. We do not have to rely so heavily on google.
https://github.com/rumca-js/Django-link-archive
It is true that it does not serve me as google, or kagi replacement. It is a very nice addition though.
With a little bit off determination I do not have to be so dependent on google.
Here is also a dump of known domains. Some are personal.
https://github.com/rumca-js/Internet-Places-Database
...and my bookmarks
https://github.com/rumca-js/RSS-Link-Database
Some more years, and google can go to hell.
Like, literally irrelevant. I'm still watching YouTube, using Gmail, and occasionally checking out something on Google Shopping if I want to find something locally instead of on Amazon. But I use Google search about 90% less now than I did a year ago.