Okay that's sort of what the !bang in DDG is for, and why it's a meta-engine. What's the blue sky ideal for a real, no-bullshit, everything search engine that doesn't fall prey to the constant flood of garbage?
I have an exterminator who comes to my house every couple months, and sets up traps here, poison there. I don't have any rats in my house. I do see rats running across the yard sometimes. The exterminator explains it like this: Rat pressure. The rats overpopulate and there's "pressure" (like, uh, "memory pressure", which is also a fluid concept) so they try to get into your house more, through smaller holes, as a function of how much outside drama is going on, how many they are and how overpopulated, how scarce their food supply is, how cold it is outside, and whatever else drives rats into your house. (I love the dude who's my exterminator).
Anyway, this is the same problem every search engine faces. The more surface area they expose, the more pressure they have building, the more ways people have to fake out their systems.
We have to go back to the 1990s Yahoo! model. Curated content. A list of websites that are reputable. 1990s Yahoo is the future.
Creating a "trustless" search crawler, where anybody can participate, and then applying an algorithm to determine trust or value feels like it'd be a never-ending arms race - that'd require AI and extensive/expensive resources that is likely better invested in developing real trust networks and curation; curators are corruptible and regulatory capture of policy is possible if the organization is infiltrated or poorly overseen.
Carte blanche opening your system up to anyone to inject data seems like the wrong foot to start off on, whereas my curating a moderator, someone I personally know and feel good about, trust to whatever level, and hiring them - ideally making sure they're someone you respect and you're someone they respect, pay them well, and at scale will be able to pay for itself; this did just bring to mind however big pharma and pharmaceutical trials structure and how that system can be/is/has been captured - and so perhaps the pressures when dealing with multi-billion dollar market categories will always lead to shenanigans if ever trying to centralize too much, not allowing for de-risking and broader resource distribution via sales/profits to more parties than the "5-star" rated products.
In a thread on HN, I think it was yesterday, a few people posted about review sites where some product reviews are free - but others you had to pay for. A system to facilitate such organizations could allow a highly competitive environment, where organizations develop/build a brand - build trust for their brand as being competent and thorough - and so then over the lifetime of a reader/customer, perhaps they'll spend $1,000 buying reviews (say 333 big purchases over 40 years that you're willing to pay $3 a hit for) to make sure they're ; mind you there will be organized that could be captured to say promote one conglomerate of products over another, perhaps even regionally, but I'm beginning to think it's a necessary layer to combat the shit show that is Amazon (et al) reviews. Ideally these systems and how the reviews present the information, and how thorough - the technical depth and breadth and testing done - will help educate those who dive into using this system, which will sharpen themselves while keeping reviewers on their toes and arguably strengthening their organizations and competency as well.
Legitimate question; if this is a real business model (and I believe it could be) then why the fuck does Yahoo.com look like a dead clickbait aggregator instead of, yknow, what it used to look like? i.e. FINANCE ___ [Stocks] [ etc] ENTERTAINMENT ___ [Movies] [TV]
Where's that site?
Visionary leadership left the company? And arguably it lost its soul and excitement. I genuinely thought when Marissa Mayer was brought in as CEO and announced they bought Tumblr for $1.1 billion in cash, I thought she could actually turn Yahoo! around - that she understood platforms and holistic systems; perhaps she did but her hands were tied, and then they made terrible decisions like banning porn on Tumblr - so bureaucracy, politics, and arguably the ad industrial complex and "mainstream" pressures (perhaps like billing/financial system being used as a tool to suppress freedom/sexuality/porn because they've not been successful politically) were pressures her or the Board of Directors couldn't counter.
Then Google came along and was a better search engine, for a time, that was a traffic leak for Yahoo! - and then Google has now devolved; I also thought Google had a good shot at competing with Facebook, but whomever's pulling the strings there, the launch of various platforms, they don't seem to understand it can take 5-10 years after the MVP of a product is launched for it to mature - but for whatever reason their executives or managers haven't been comfortable pulling the trigger, arguably because anyone with that entrepreneurial spirit just takes their idea and gets funding and owns a large portion of whatever they've done; but then you can never develop a full breadth, holistic ecosystem, that can grow into every crevice, nor as broadly, or nuanced as possible - so they're stuck being Search, Gmail, Calendar, etc.
I'm quite certain I've figured out the foundational MVP facilitating an "infinitely" growing system and that would allow 3rd parties to integrate, however I have severe chronic pain that messes up my executive function, so it's difficult for me to actually self-direct and execute - I'm stuck mostly in a low activity, stream of consciousness and go-with-the-flow life of routine - otherwise I would try to launch my plans, which I've done plenty of UX/UI for, as that is simple enough that somehow bypasses higher executive function (moving a pixel and then responding via visual feeling of it isn't complex) - but organizing to turn that into adequate specs to get solid estimates or fixed price quotes for work is extremely difficult to me.
On January 11th I do have a surgery that may or may not reduce my pain by 50%+, may or may not improve my executive function, ... I've even attempted to write draft "Show HN:" posts to explain what I am doing, the starting feature sets, the reasoning behind the design decisions I've made - but it just gets too complicated too quickly for me mentally then to be able to organize further or polish it. I think I have the perfect domain name for it too: ENGN (engine), what makes me smile every time I notice it in my layout/mockup of it is in the search input box it says "Search ENGN". My username on HN actually is an older incarnation of a plan I had, loceng being a short form of "local engine" - and ENGN being from engine, a name I brainstormed after Tumblr sold for $1.1 billion - and I realized that eventually I'd want to try to do my "local engine" idea but that that was too long for a brand name. Fortunately engn.com was for sale at the time, I can't remember if it was $2,000 USD or $4,000 USD - either way, not a bad price for a 4-letter .com that's pronounceable to something with meaning.
I've wanted to write a book too - on health, health systems, and on these systems we're talking about here. I'm 38 now and I taught myself to program when I was 11, learned SEO at 15, evolved to design as I'm more creative and programming became mindnumbing to me, and eventually thought I'd need (or want) VC money - so I started engaging on Fred Wilson's of USV.com's blog - AVC.com - so I have plenty of self-taught experience. The problem is even going back to my shorter or...
Listen, first of all, do not consider ending your life. Seriously, you're way too smart for that. I'm sure you've got it worse, but I've had enormous sciatic problems in my life, I've had 3 herniated discs; they're behaving for the moment after massive doses of cortisone and without any painkillers, but I know what it feels like to cough them all out of my back at the same time. Not to be able to put a foot in front of another or turn your neck for weeks. (I'm a huuuuge fan of intramuscular cortisone injections, though. Like 5 or 6 large cortisone over a week, with some B-12. Every couple years. Not in the spine... fuck that. Alternating butt cheeks. You won't feel any benefit until the third day at least. If you can convince a doctor to give you that for a week, you will be fucking superman. They won't do this in America unless you know a doctor personally, but they'll do it in Mexico or Spain. I had it the last time my discs went out and it's been 6 years and the inflammation has not come back. They thought it would).
Anyway, before you off yourself, do try a fuckton of intramuscular steroids. The fifth day I levitated off a bed in the hospital; I hadn't walked in a week; I felt so good I went to a club; I got drunk and spent the night on a beach drinking and making out with an 18 year old model from Denmark. Seriously. There were wild cats walking around; it was winter on the Spanish coast. If you do one thing before you die, go get five cortisone shots in your ass, in a week.
I also got the hiccups for 24 hours and couldn't sleep, but that's neither here nor there. And I got temporary blindness in my left eye from fluid behind the retina, caused probably by too much testosterone. But. Goddamn it, I'm ok. You can be okay.
Enough about that.
About Yahoo and Google. That entrepreneurial spirit is, in my experience, way too often just about getting the funding and fucking off. We all know why these companies go downhill, but somehow it's always such a shock when they actually deteriorate in front of our eyes, huh? Google's search results, for instance. I would have expected their cofre business to stay more or less fine, not collapse a couple years after all the competition was eliminated.
It would be fine if they didn't grow into every crevice. Get search right, that's all we ask. I don't want Google to be my chat room or my shopping site. Why do they need to? Search is huge. They own 90% of the market.
>> but organizing to turn that into adequate specs to get solid estimates or fixed price quotes for work is extremely difficult to me.
That's always the worst. The business side. I've always just built things and hoped for the best. It sounds like you've got something interesting going there, although I have no damn clue what you're building, that's an exciting feeling. ENGN is killer. If you own ENGN.com, hell, money well spent.
I don't understand what you mean about "executive function", since you obviously have the capacity to write well-crafted email and think pretty clearly; perhaps I lack the executive function to discern your lack of executive function (I'm a brutally self-punishing alcoholic, but otherwise a damn good programmer)
Anyway I don't know if you're trying to ask for pointers to workers for this concept, I'm probably not it; I'm $200/hr and I'm already covered for the next year. This, however, should be your symphony. And I think you know how to do it.
Another frustrating thing for me: well-intentioned people like yourself will chime in and offer advice or suggestions for things they've found highly helpful - and instead of me being able to compile/organize a list of things I've tried and their outcomes, explaining why they didn't help, why I didn't continue them, why there wasn't a net benefit, I instead get reminded that that kind of writing up a complete list is an extremely difficult task for me. I'm only replying to you this morning because I couldn't actually begin responding to you yesterday even though I saw your reply because 1) I had opened my right eye already which allowed the post LASIK eye pain and symptoms to trigger fully (central sensitization/hyperalgesia), and 2) I had begun to do some activity yesterday - namely driving a short distance to my mother's for a late lunch and then take my puppy to an off-leash dog park for an hour - that relatively small amount of movement being enough to really compound/amplify with the seemingly exponentially growing pain/sensitization from the eye pain.
Central sensitization and hyperalgesia is really a different kind of beast when it comes to pain, people including most doctors really seem to have a hard time understanding it. That added stress to my nervous system from opening my right eye and triggering/ramping up the sensitization/pain from just low activity and careful movement was enough to lock up my thinking, and arguably my emotions, but more specifically the locking/the eye pain and increased pain from movement makes fluid thinking have much more friction added to it; one good or as relatable as possible thought experiment for a "normal"/unaffected person may be to imagine the feeling when you get something in your eye and you desperately react to get it out because it's so painful: now imagine that foreign object feeling is immense because it's permanent and broad, it's "a lot of objects" in your eye(s) because your cornea and nerves were sliced across 90%+ of the cornea (and so abnormally/constantly signaling as such), and imagine how your brain/nervous system may try to cope from such an overwhelming/overriding system constantly firing to draw your full attention to what your eye is telling your body is an active/present moment object in your eye - potentially as if you just walked into a sharp object and your eye is signalling for you to literally freeze still because it thinks you're in the moment just done something to critically wound your eye and moving another millimetre or less would threaten your survival (as the evolutionary strength of the reaction has come to dictate).
Well, the answer is some people after LASIK get this severe reaction, their nervous system gets overwhelmed - arguably the more naturally sensitivity, creative, healthy, and grounded people will have stronger/more detrimental symptoms/reaction to the eye damage; central sensitization and hyperalgesia that LASIK for years completely swept under the rug as being possible and only recently admitting to being a potential "side" effect. In fact they purposefully mislabeled what should be called corneal neuralgia syndrome as "dry eye syndrome" to mislead people away from learning that the "dry eye" part is actually on a spectrum of symptoms caused by the damaged cornea/nerves that happens in 100% of their surgeries; non-LASIK done research done has shown up to 40% of people have permanent problems after LASIK, and in 2011 one of the expert FDA advisors, who votes yes to approving LASIK, published a letter to the FDA asking them to immediately recall LASIK because there was data they were ignoring that they should have never been ignoring, and that it should have never been approved in the first place. Part of the medical industrial complex where arguably regulatory and institutional capture has occurred in the name of profits.
> Creating a "trustless" search crawler, where anybody can participate, and then applying an algorithm to determine trust or value feels like it'd be a never-ending arms race - that'd require AI and extensive/expensive resources
This is very true. How many times have I clicked on a site met with ads so bad that the browser slows down, and after 10 seconds the page gets covered up by more and more crap and then a paywall shows up sometimes too. Now here's the thing - a competitor to Google might detect you clicking back and then pop-up a special set of controls near the search result that lets you say: "too many ads" or "paywall".
However, if such an engine were to start beating Google, I'm sure Google would implement it in their own way: automatically detect why you clicked back in such a short timespan.
> However, if such an engine were to start beating Google, I'm sure Google would implement it in their own way: automatically detect why you clicked back in such a short timespan.
Do you seriously believe that Google doesn't use that as a datapoint already?
Google already detects immediate returns and knock off that link for you. What's problematic to me is I tend to reactively mash back and forward and link just goes from it.
The funny thing is that if the people who worked on spam at Google were free to talk about it, I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence. It's a ridiculously hard problem, especially when people are targeting you directly. But they aren't free to talk about it, because if they did it would just give more assistance to the spammers, and make the problem worse.
I'm not saying that curated search results for particular verticals is a terrible idea (though I'm sure like anything the devil is in the details), but on the whole Google search is very, very good considering the constant assault they are under from spammers (which most other search engines are not, at least directly).
There is no scenario - none - where thousands of engineers at Google working on search wake up in the morning and say "we sure have made it good enough wr2 SPAM. I think I'll have another Danish."
I sure wish I had problems that were totally unsolvable, they are so easy to measure progress on. /sarcasm
I think it’s more likely that because they are just building hundreds of tiny tweak experiments and it’s someone else who desides what to build and if it even worked. Search quality is such a meta-problem that it goes beyond any real hope of simply working on it in anything beyond piecemeal trial and error fashion on their dataset.
I agree with this and the grandparent comment wholeheartedly. That said, there's a kind of institutional blindness that can build up in companies—especially ones that dominate their sector. It may have roots in intransigent upper management, ossified and inflexible process, wide-scale burnout, a culture of passing the buck, or any number of other pathologies.
I don't claim that Google has any of these and certainly have no insight into their search group. But I've personally been at powerful companies with best-of-the-best talent that were blind to the decay in their own living room, so I would caution against immediate dismissal of PG's take.
Especially since "made a change that improved search result relevance by X%" is an extremely compelling story for promotions. If indeed there is a launch-driven culture for promos at Google then there'd be extra incentive for new mechanisms to reduce low quality search results.
Also, i'd be very surprised if they didn't have tens of thousands of workers aiding in spam review already.
The hard part in all of this isn't finding and stopping spam - it's defining what spam is. Are all the pie recipes where there's a 2000 word essay about their grandma at the top 'spam'? They still have the recipe, and Google Home devices pick up the recipe instructions just fine so people end up not reading it, but many people would still consider that spam since it adds such an obstacle to getting the information you want. Same for cnet articles like "Best smart home devices to buy in 2022" - it's a reputable brand with a list of smart home devices, but it's hardly a review and exists to funnel people to their Amazon affiliate link.
AFAIK the 2000 word essays in recipes are Google's fault - it prioritizes pages with a lot of content, so you have to add that junk to the top in order to rank highly. While I'm sure there's more going on behind the scenes than I'm aware of, it does seem like the rules could be altered on a category-specific basis where a lot of text isn't necessarily a positive.
This is largely because micro-blogging means less content, and less content means you could write five 300-word blog posts instead of one 1,500-word post.
I've done blogging for the last 10+ years, and many of those I spent as a freelancer working with startups/brands/editorials. Everyone is after "word count" and I absolutely hate it.
Whenever I work on articles for my own blog, I just don't consider word-count at all. I think if your content is great and informative, then readership will be natural.
I collect post views, but not using Google Analytics or anything like that. I built a pretty substantial developer blog (tips, resources, etc,.) back in 2014. I think it peaked at around 350,000 monthly visitors after 12 months.
Later on, I sold it because I needed the money. Not so much that I didn't want to keep working on it. Unfortunately, the new owners didn't have any idea how to maintain a "healthy" content blog, and it has plummeted down to around 30,000 monthly visitors. All the content they're publishing now is some thin headline-clickbait bullshit.
I even gave them free advice on how to fix it, but I think that for a lot of people, they just don't care and will mindlessly pump out as many pieces of content as possible. And such blogs can be identified from a mile away.
And therein lies the problem with Google SEO at the moment. Even myself, someone who has done SEO work for more than a decade, I can see that results are getting worse. In some niches, the same crappy articles that dominated 6-7 years ago are still dominant today.
Could it also be due to reduction of public interest in blogs over the past few years? Most stuff are now published in the form of vlogs instead of blogs. I do miss the good old blogs era, tho, and I wish there were still high quality blogs around.
It's two-fold. If Google priortizes pages with a lot of content that's one thing, but longer content also means more space for ads, or more scroll events to trigger ads, etc.
Incidentally, prioritizing long content seems odd to me, in my experience the best pages are short and get right to the point, at least in the context of something like a recipe or other "how to" resources.
This one is hard because it does actually seem to be the case that the cruft around the recipe is valuable if the content is right. Most of the recipe blog stuff is garbage, but if you look at youtube it is clear that creators who add extra flair around the recipe are a powerful force.
Recipe intro text is useful for contextualizing the recipe and copyright purposes. In RSS days, it was a way to get readers to click through, so the author got the ad views. Also people who write recipes like to write about food.
This reminds me of the page inflation that struck tech books during the late 1990s / early aughts. The Marketing Wisdom was that fat books sold (or took up more shelf space), so texts got padded with weak writing, gratuitous puffery, and other elements, which (much as the recipie essays) simply got in the way of delivering actual informative content.
(The fact that many of these books were rushed out with very poor quality control also didn't help.)
While I think the essays are excessive, I appreciate that some of them document that the blogger actually made the recipe with progress pictures. With the more basic recipes websites, I wonder if anyone's actually made it before or if the recipe is from some scrapped database of unknown origin and quality.
Yeah, the newest nuisance seems to be sites that clone Github Issues and StackOverflow with a crapper interface. Somehow they rank higher than the original sources. I'd say it's spam but it's definitely not traditional spam.
And the strange Wikipedia mirrors that are shown in Google Verbatim searches instead of the original. If I disable Verbatim, they disappear and I get regular Wikipedia instead.
I'm not going to say solving spam programmatically is easy, but the gitmemory garbage site (for one example) has been around long enough that there's no excuse for not downranking or removing it. How hard could it possibly be for humans to spot these few sites and nuke em? I'm sure Google engineers see them all the time.
> and Google Home devices pick up the recipe instructions just fine so people end up not reading it
I think this isn't entirely related, but that's perhaps the beginning of a bias you might end up having that everyone experiences technology in the same way as it marches on. I've yet to encounter a Google Home in the wild, I imagine far more people are consuming recipes on phones, tablets and PCs.
Let's have niches where the content is hand curated by human beings instead of pure statistics by machines.
Hmm why stop there let's actually make the users do the curating and even the content creation by rewarding them with social validation. Let’s have hard working moderators who work on the community full time.
Then we could just build a search engine over it. We could call it Reddit. Or HackerNews.
Maybe the users aren't all as good as professionals at curating the information. Let's hire professionally trained curators pay them well and we could call them newspapers. Then we can come in disrupt them and replace them with an algorithmic marketplace that eventually becomes infested with click bait.
> The hard part in all of this isn't finding and stopping spam - it's defining what spam is.
This is one area where Google could use personalised results to provide a better experience for the user. Let me decide what spam is for me. Let me mark results as good or bad, so that the algorithm knows what kind of pages should be prioritised or filtered out the next time. Google SearchWiki was a step towards this but they killed it off.
Is conservative leaning info spam or not spam? What about liberal leaning info?
We have seen what this leads to inside the social networks as well as YouTube, and at a macro scale I think we might want to have a shared concept of what constitutes a good search result for a given query.
At micro scale, it can seem more optimized to get exactly the type of result you want, but if we take an absurd example like an Apple Pie recipe shouldn't we all have shared understanding of what types of ingredients would make for an Apple Pie?
The shared understanding, I believe, is core to communication. If all of us have our own specific ideas of Apple Pie, then who is actually right on what an Apple Pie really is? What happens when your search results insist that an Apple Pie doesn't actually have apples in it, but instead pears?
Compounding the problem, the 2000 word essay is sometimes really useful if it's describing a technique used in the recipe (cf Stella Parks' recipe for homemade bagels on Serious Eats: https://www.seriouseats.com/homemade-bagels-recipe). But somehow only spammy blogs with plagiarized recipes, AI-generated "essays," and affiliate links for every ingredient and tool used make it into the first page of results on Google (or DDG, for that matter).
At some point, Google must have moved away from using site-level reputation in search rankings, as I almost never see recipes from reputable sources like King Arthur Baking, Serious Eats, or Food52 in the first page of results.
This 100%. In travel, we see Google constantly tweaking its algorithms, and compared to Bing, Google surfaces a ton more small, well-written travel blogs [1]
Not only that, Paul and Michael have seen plenty of startups, and at least in recent memory, the number of vertical search and consumer startups that Y Combinator has funded hasn't been that high
As a consumer startup, I know this issue firsthand. Paul and Michael assume that if you build a better product, they will come! That's simply not true these days.
Instead, you need to:
- Build a better product
- Option 1: Figure out a channel with enough growth on an existing platform. This likely means you're doing SEO for your new search engine
- Option 2: Get your customer lifetime value high enough so you can pay for ads. This is tough, since it's a bit of a chicken and the egg problem since most search engines are monetized with ads
As the founder of Wanderlog (YC W19; https://wanderlog.com), a consumer vacation planning app [1], I definitely remember the idealistic days when I thought the best consumer product on its own would win! But growth doesn't just come, and the same can be said of vertical-specific search engines.
[1] Try searching "[your city] itinerary" on Google vs. Bing: it's much more likely you'll find a small blog rather than Lonely Planet or the local travel bureau as the top result
>> - Option 2: Get your customer lifetime value high enough so you can pay for ads. This is tough, since it's a bit of a chicken and the egg problem since most search engines are monetized with ads
nonoonononooonono. No.
Don't monetize anything for the first 10 years. That's the only way it can work.
Then you can go monetize it and buy an island and not give a shit if you destroy what you created.
Interestingly Google didn't have a top-result ad and the google.com/travel carousel is 4th from the bottom.
For the actual results, both thefearlessforeigner.com and paigemindsthegap.com seem to be actual travel blogs (the pictures didn't appear in a reverse image search, so they are probably organic), but they're clearly geared towards being a 'faq' for visiting the city and have affiliate links where appropriate. Bing went straight for discoveratlanta.com, and frommers.com is well-thought-out but not a personal travel blog.
Hi! I used Wanderlog to plan a recent month-long group trip, which was definitely the most complex vacation I've had to plan. For context I am very active when traveling (e.g. multiple activities each day); so not sure how my experiences map to others.
The best part of it was (going to a foreign country) being able to find / identify all the attractions relative to each other, so I could go to cluster A on Monday, cluster B, on Tuesday, etc.
The hardest part of it (and why I needed to create a separate google sheets anyways) was--once I figured out opening hours of different locations, hard-to-book activities with limited reservations--the ease of moving things around more fluidly e.g. cluster B on Monday, cluster A on Tuesday, etc. and having a more information-dense view so I could see larger portions of the itinerary at once.
It would be cool to have an "input everything" --> "input time restrictions / unmovable things" --> output planned activity cluster type workflow.
I understood PG's point differently. My understanding is that he is suggesting an angle of attack in which carefully crafted manual reviews (that do dot scale) can be used to bootstrap a product that does scale thanks to something else (e.g. collaborative filtering). All of this being on a niche domain where you can drive a wedge into the mediocre performance of Google (online shopping probably being the worse possible choice, but there are many others).
Your point is good but I'm not sure I'd say very good given how easily the same SEO spam domains can stay at the top of search results for ages simply by scraping someone else's content. What I'd be most interested in knowing is what their success metrics are defined as — for example, how much of a problem does Google's management consider it if someone searches, finds the answer they were looking for on someone's Stack Overflow rip-off, and stops searching? I could easily believe that a significant amount of what we're seeing here is that they're focused on some kind of user frustration metric which doesn't include things like damage to other businesses.
Yes, I've noticed this particularly with technical results. A lot of sites seem to have scraped StackOverflow and GitHub issues, put a crappy ad-loaded interface around them, and somehow out-rank the original SO/GitHub content.
It's like the bad-old-days of ExpertsExchange, which somehow was never delisted by Google for its shady SEO tactics.
You just have to look at Google's profit motive here. Their motive isn't to provide quality search results. Their motive is to show users ads, either in the search results themselves or on the destination sites via their ad network. The SEO spam sites aren't a bug, they are a feature of Google's profit algorithm. Google's search quality will never improve so long as their motivation is to show you ads. Why should it? Competition may help here, either by an outsider like the OP suggests, or via breaking Google up with anti-trust enforcement, or both (my preference).
As a user, your best personal and ethical move is to install an ad-blocker, to make ad-based business models less viable, which will help promote business models that don't abuse the customer.
>" The core problem, I guess, is that search engines view all their results as ads. That’s why they got into the ad business in the first place. "
This seems a bit overly cynical. Some search engines only served ads, but they're long gone. The survivors are those who dedicated themselves to finding links which were responsive to people's search intent. They seem to have gotten into ads because it was the best business model in this market.
> A lot of sites seem to have scraped StackOverflow and GitHub issues, put a crappy ad-loaded interface around them, and somehow out-rank the original SO/GitHub content.
Some even made slideshows of SO screen captures and put that on Youtube, with a fake video or spoken intro to make believe an actual content will be discussed... A number of shameless people would go any length to grab bits of money anywhere and anyhow, and I've hit those links a couple of times.
> It's like the bad-old-days of ExpertsExchange, which somehow was never delisted by Google for its shady SEO tactics.
This is really what made me suspect that Google was teetering on the edge of the MBA death spiral: these problems run for years when they'd be easy to block, which suggests to me that whatever metric gets you a bonus / promoted doesn't include things like that which are long-term threats to their core business even if it's selling a lot of ads short-term.
I agree. You can say a lot of bad things about Google, but they definitely have some of the smartest and highest paid engineers working on their search. Plus there are already a lot of people trying to compete with Google and so far, no one seems to provide consistently better results.
The only advantage a startup might have is that they could do completely new concepts, such as specifying what area you search in, allow you to modify their classification of your query and/or moderating sites you include - which is probably necessary anyway, since you'll hardly have the budget to fully index the web. I'm not saying it's impossible, but it's not going to be easy at all.
And after all of that, you still need a way to make some money.
> But they aren't free to talk about it, because if they did it would just give more assistance to the spammers, and make the problem worse.
The reality is more that some Google engineer will come up with an algorithm change that makes the result 40% better, but it will come at the expense of making that search 3ms slower so the change won't get merged. Or it will make the results worse for some niche set of queries that the business team really cares about, so again it won't get merged.
There are lots of consumers who would gladly pay $1 a month or whatever in order to use a couple extra milliseconds of compute power per per search in exchange for drastically better results, so there is lots of room for a startup to compete.
> There are lots of consumers who would gladly pay $1 a month or whatever in order to use a couple extra milliseconds of compute power per per search in exchange for drastically better results
Google has a paid-for Search API, so they could do that if they chose to pursue it. And then they could let Google One users opt-in to the same thing via ordinary Search. I'm not sure whether Bing has anything equivalent.
A lot of people forget that one of the inputs to the Google ranking algorithm is input from human quality raters who work off of an extensive, 172 page, guide that Google publishes and updates for anyone to read: https://static.googleusercontent.com/media/guidelines.raterh...
Most comments focus on the technical side of things, whereas I'm sure there are also legal restrictions involved in this. If Google delists a website on the grounds that it's a copycat of stack overflow, or because they have low quality content according to Google's taste, there might be lawsuits filed against Google, claiming that the company is discriminating.
In which countr[y|ies] does Google not have the discretion to decide that certain sites / pages / etc. "belong considerably further down" in Google's search results pages? Seems to me that sorting the search results to #1, #2, #3, etc. is pretty well baked into their basic product.
How hard is spam, really. If you're Google?
Here's what I would do as a heuristic (uh, not evil?):
We know everything about you and everywhere you've visited and everyone you've talked to in the last 60 days. We know all their phone numbers and email addresses. We even know the girl's phone number you met at the bar, who didn't give you her phone number. So if any of those people email you, we'll categorize that as "not spam".
Also, if it's your boss or a coworker, "not spam".
If it's a major company that's existed for more than ten years, not spam.
Everyone else, spam.
Done.
This is hyperbolic, right? But they can solve spam in a split second, if they just admit they're watching you all the time.
I said this in a similar thread yesterday, but I think this is an unsolvable problem because much of the content either no longer exists in website form or is old.
To put it simply, a new generation of the people who used to make the reliable niche websites that not just answered your questions but also helped you learn a particular topic have moved to youtube instead.
Google search is hollowing out as a result with the meat going and the SEO'd fluff that kinda answers the question but ONLY the direct question being asked with none of the wider expertise that more educated people in what they were searching for.
Of course google owns youtube as well.. so perhaps they just see it as an inevitable transition.
Just a note on that, youtube search is finally getting better, yesterday I noticed it was able to find key words in the middle of a lecture that had nothing in the title or comments. I always wonder about their AI transcription service, it's gotten so good, if they're storing all that audio as text, I guess their search is going to get excellent?
Really? I don't think the bureaucratic bloat at Google cares and the original authors of the search engine in its current incarnation are probably long gone. It is maintenance mode and I don't think they dare touch too much.
It takes time and effort to build up a spam site's ranking but it is trivial to blacklist those who get to the top.
The curated search results business model doesn't work. Google gives "aggregators" and other search engines the death sentence for organic search traffic from economically meaningful queries, so you'd get no free traffic. This is one of the major antitrust complaints against Google in the EU. Since you get no organic search traffic, you need to build a brand using advertising, and once you start down that road you need to monetize the first click which compromises the quality of your site.
> I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence
Really?
I can point you to Hard Problems that have been solved better at little startups than at Google - or, indeed, at any other bigco. That's why acquisitions happen.
Why does Google having 1000 engineers working on a problem automatically mean they are the smartest?
The problem isn't that Google doesn't employ these people or invest in their activities.
It's that Google has destroyed their own search results in order to continue to expand their revenue opportunities.
If Google:
- Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT)
- Allowed you to block certain domains from your search results, like YT videos. (If they added some kind of "coordinated network detection" and down-ranked domains coordinating with ones you've blocked, that'd be pretty cool).
- Allowed you to create your own custom search engines, like "Programmable Search Engine".
That would be incredibly valuable. They already have most of the tech. They could even create a subscription service around custom search engines if they really wanted. Plenty of people would find something like that incredibly valuable.
Anyhow, buried in there is your startup idea. Remember: your startup doesn't have to generate the same revenue or profit as the incumbent on day one to be successful.
Not really that simple, I see a lot of potential for abuse - using bots and brigading to mass downvote your competitors or political opponents.
Couple of positions up or down in google results for somewhat popular and valuable keywords can mean the difference in thousands of dollars per day of ad or affiliate revenue. I suspect it would get pretty wild if google launched something like this. There already are black-hat seo methods and services, but something so simple and direct would turn it up to 11.
They already have the tech to fight this on YT. They, in theory, are supposed to be doing the same thing to detect inauthentic behavior on ad placement and click abuse.
> That would be incredibly valuable. They already have most of the tech. They could even create a subscription service around custom search engines if they really wanted. Plenty of people would find something like that incredibly valuable.
Why would they do this? Google's customers are the advertisers, not the end-users. And no one is going to pay for a search engine, it's been tried and has failed.
You only need ~1/10000th of Google's revenue to be a financially successful startup. 1/1000th and you'll have a great business, and at 1/100th you'll be somewhere between a unicorn and a decacorn.
I don’t know if that follows. Google has been maximizing revenue at the expense of the search result quality.
An “objectively superior” search engine, from an end user’s perspective, might have to make engineering choices that come at the expense of ad revenue.
But we’re all just talking hypothesis, it’d be cool to see someone launch a startup to get some answers.
there is a smaller niche market. SEMrush is a tool used in the digital marketing industry that is now public and has a multi-billion dollar market cap. It originally started as a search engine. When they didnt gain traction they used the tech to monitor Google and interface it for customers who are tracking their performance in search results (and much more).
> And no one is going to pay for a search engine, it's been tried and has failed.
Always curious about things like this. I certainly would pay for this; it sounds like many other people here as well would. I'm curious if the constraint is that there aren't enough people to actually pay for the investment required for the service, or if there aren't enough people willing to pay to meet the standard VC notions of success. We seem to have a problem with building and supplying services for niche (read: "not expressable as an integer percent of the world's population") customer bases, and I'm never sure if that's a business problem or a cultural problem.
The people most able to pay for a service like this are the people that advertisers most want because they’re the people with enough discretional budget to spend on things like better Google search results. Allowing someone to buy something like this also reduces your attractiveness to your advertising clients.
if you think about it, Google provides advertisers a customized search engine to find customers. So it is not you searching the web, it is web's advertisers searching leads
I think you have to look at it more like Amazon Prime.
Nobody is going to pay for /just/ a search engine. But they might pay for, say, a /better/ search engine, plus additional features around gmail/gcal/gdrive.
Think of it more as subscribing "to google" and less as subscribing to "google search".
Regardless, the point isn't to "fix" google. It's to highlight a possible path for a new market entrant.
... If an existing player wanted to make a move here, I would say that both Mozilla and Apple are well positioned to add "personalized search" to a subscription service. Same with Microsoft. DDG could also make moves here if they expanded beyond search.
The problem with all of this is it would help us greatly, but it would be useless to the 99% that the internet is increasingly being designed for. Modern UI trends are becoming obsessed with removing as many options and features as possible so the dumbest humans bordering on smartest vegetables can still use the service.
It does not if there are common interests and characteristics among users. Let's say for example I'm a young African-American girl who wants to learn how to code and I query "how can African-American girl learn coding?" and Google shows me Black Girls Code a non-profit organization that focuses on providing technology education for African-American girls. Considering that Google knows that I'm African-American girl and that I want to learn coding, how many other African-American girls want to learn coding? Probably many so caching doesn't break customization and personalization as long as Google knows my characteristics and interests and characteristics and interests of other people that are similar to mine.
It doesn’t have to actually, there are some pretty advanced caching mechanisms that allow you to combine cached elements together. For the web at least, you could do this back in the day with Server Side Include, and a place I worked at used it to cache logged in content.
How do you fight brigading, the organization of groups elsewhere to collectively vote on something? Eg white supremacist groups get together and vote down everything by people of color, and vote up their pages about how great they are?
This is a great point. I would think Google could rank users from low quality to high quality in terms of the quality of the websites which they recommend or downvote. Tricky business and could be difficult to control, but basically the same thing they currently do for websites, but extended to humans.
easy, voting blocs, you assign yourself to the results of people who vote similarly to you. additionally there'd be local and regional blocs too. I can't think of a reason that the naive everyone sees everything everyone else is doing would work in the long run. That's Twitter, and it's garbage.
Randomly select votes that are actually recorded. Then add in metavoting that votes on the votes with random sampling. At Google's scale with a sufficiently random sampling you'd be extremely hard pressed to successfully brigade or spam the voting.
Google could easily use its current fingerprinting to constrain (to an extent) multiple votes. Even knowing only a portion of the population will participate in the voting they can use a Wilson confidence interval[0] or similar to properly weight votes.
Random sampling works here since you're not guaranteed one vote per user per page and the outcome in binomial, seen and downvoted or seen and not downvoted.
Those shitty SEO spam sites exist only to serve ads, and Google has a monopoly on internet ads. So there is no real incentive for them to solve the problem.
Real review sites serve ads too. I don't think Google has any incentive to make things worse, and they still want people to google reviews instead of just asking friends or people on reddit for reviews.
Google has 28.9% share, Facebook 25.2%, and Amazon has 10% and growing fast. Not a monopoly, and the incentive is there: if search results are consistently bad, people will stop searching as much, and revenue and market share decline.
The biggest perverse incentive for Google is that making better search results can mean less clicks to ads (clicking an ad because results are crap, going thru more pages of results means more ads). Clicks are revenue which is much easier to optimise for.
Internal Search owners can push for better algos, but what if the algo causes revenue to fall? Are there strong forces strong enough within the organisation to ensure that search quality prevails?
If this is the case, the problem is existential. It can only be arrested at the very top
If I had to pay $10/month for good search results, I absolutely would. I think most people would. Get rid of the ads and spam, and you have a service worth a premium. The solution is to make it user-centric instead of advertiser(spammer)-centric.
The biggest perverse incentive for Google is that making better search results can mean less clicks to ads
This gets close to the real root of the issue -- attention is monetizable independently of the quality of content. There would be much less incentive to create SEO spam if search engines negatively weighted pages with ads and affiliate links, and if manufacturers were barred (e.g. by the FTC) from owning or imitating reviewers.
> The biggest perverse incentive for Google is that making better search results can mean less clicks to ads
This is also something that Google can control if competitors come along.
i.e. If a reasonable competitor comes along that is willing to sacrifice ad revenue for better search result quality than Google, google can just adjust their search quality upwards to knock them out (and then adjust it back once the competitive threat is gone).
Perverse incentives from Google are all over the place - Searching for the delivery business "Just Eat" in the UK for instance returns an ad for their competitor Deliveroo above the legitimate organic search result for me - and I can also see that JustEat are trying to pay for their own brand name just to compete - and IMO this sort of behaviour is anti-competitive, borderline extortion considering Google is the de-facto way of searching for a business, and shocking from a search-quality perspective (where the wrong result is intentionally shown at the top because they paid more money).
Things have changed in the past few years, now that Google has developed advanced transformer models, but for a long time Google's question answering facility has been: "let spammers make 10^8 pages where the title is the question and the answer is in the page".
The trouble is that there's a fine line between "answer is in the page" and "word salad!"
> - Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT)
They're going the OPPOSITE DIRECTION from this!! They recently removed all downvote visibility of YouTube videos from the user, so now downvotes only feed into their algorithm. So in the last line of defense of me ending up watching a shitty video, one of the most valuable tools has been removed by my betters. It's preposterous that people think that Google is doing a good job. They're actively getting worse, and ignoring everyone saying so.
They're doing a great job. I'm so happy dislike visibility have been removed. It removes the effectiveness of pile on coordinated harassment which many youtubers have fallen victim too.
Except the ratio is visible for the video creator, so they know nonetheless. And there's the possibility of disabling the the like/dislike for each video.
All the big channels I follow are mad at this change, and there's a coordinated effort to bring it back.
The custom search engine is harder than you'd think.
Google's search algorithm is tuned up for searching the whole web. It turns out the heuristics you need are very different depending on the size of the collection.
When Gerard Salton was doing IR experiments with punched cards he was working with collections of as little as 70 documents and in that case you are going to be very concerned about recall and not precision. Maybe there is 1 relevant document and if you miss it you failed.
If you had 70 billion documents you might have 10,000 relevant documents and if you lost 60% of them you still have 4,000 documents. The end user gets more results than they can sift through.
Thus I always groan when I see a site is using "Google Site Search" because the relevance is usually worse than you'd get with the alternatives.
Connected with that is the tuning work: Google has sufficient data to tune up a big model for everybody but true personalized search eludes them because they don't have enough data from you to tune up a model for you.
I agree with you that "true personalized search eludes them because they don't have enough data from you to tune up a model for you". That's what Larry Page said as well "Google doesn't know what you know". His ultimate goal is Answer Machine powered by AI but that's not happening anytime soon. I think internet search engines that we are using today are primitive compared to what we will have in the future.
you mean the dislike counter they just disabled to force people to sit through more low quality content and pre-roll ads to claim increase in platform engagement and viewership?
The only thing matters is revenue and Google had increases in acquisition costs in prior revenue reports. Expect to see the data points for the latter metrics to be highlighted on the earnings announcement, and a record quarter for YT coming out of the change.
> - Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT)
Not convinced this would help. The spammers would just hire people to dislike competitors
> - Allowed you to block certain domains from your search results
This I would use. Never show me results form collider, watchmojo, ranker,
> - Allowed you to create your own custom search engines, like "Programmable Search Engine".
I think this would lead to people writing highly polarized engines. The Red Pill engine for example and we'd have a new problem, the proliferation of popular highly biased results. Of course that's not to say Google's results aren't already biased but they certainly are trying to cover everyone.
> Allowed you to block certain domains from your search results
I would love for Google to build this in. Until they do, there is a WebExtension that does this: https://addons.mozilla.org/en-US/firefox/addon/hohser/ ("Block or Highlight Search Engine Results"). I use it to block stuff like W3Schools so when I search for something, MDN is always #1. Saves me a lot of time having to add "MDN" to the end of every query.
The search results markedly worsened in the last 5 years. Why could they keep up with SEO spam until 5 years ago, and now they can't? Their revenue has been growing dramatically, so they could proportionally increase the allocation. It's probably because the focus of their HR/changing workforce is now elsewhere: maybe fighting "disinformation": both COVID and political. Those efforts were non-existent 5 years ago.
I think it is also no longer in their interest. If you look at their mobile results now, there are sometimes no search results for webpages, just ads, and their automatically extracted data. So, it is in their interest now to have the search for non-advertisers to be bad. Eventually people will consider those results junk and just use the google extracted data/people who paid to go up.
I agree it’s a hard problem. I don’t agree it’s “really really good”. I regularly encounter obviously scammy websites. With Google’s js execution capabilities I’d assume they can detect that. I’m talking about the VPN install pop ups and so on. Right now there’s a whole bunch of GitHub.Io hosted sites that’s doing that. It’s not even porn. It’s home decoration stuff.
I don’t doubt it is hard but I’m forced to sign into Google now pretty much, just let me rate results and ban domains again etc. You will solve the seo problem really quick and start giving me results I want.
Spam, yes, but Google has also made meaningful shifts that are clearly directed from the top-down. It's much harder now (imo) to get specific results, they've overall started looping SERPs into broad answers.
This is def a user-engagement strategy -- but it has cons as well.
Part of the complaints in the thread were spam related, other were something deeper
Legitimate sites could help a lot by adding machine-readable descriptions of their content, per the schema.org spec. The richness of these descriptions means that this is effectively a "hard", non-forgeable claim to being a worthwhile, non-spam source (quite unlike the old META tags that got abused to death pre-Google). Of course spam sites could simply lie in their schema.org tags, but the lies are easy to spot (with combined machine- and human-review) and then they just get banned. It makes it a lot harder (and hopefully infeasible) to SEO-spam by just copying random content.
A lot of what counts as spam these days isn't something like "I search for bicycle reviews and get penis enlargement pills", it's more like "I search for bicycle reviews and get some blog who searched Amazon for the 5 most popular bikes and posted links to them with a little blurb and called it a 'Review'".
These sort of things are easy to spot, but only if you actually have a basic amount of familiarly with the topic. It's hard to spot with "AI" or super-cheap labor.
Highly OT, but if a technical person (not at a managerial level) involved in tackling spam at Google were to leave the team, are they allowed to work on the similar problem space at a different company?
The problem, IMO, might be the monoculture we have around search. Because Google is soo big, it's enough for spammers to target it and they have the vast majority of the search visibility. If we had better, more diverse competition, that might manifest as a tradeoff, presumably, they would have competing and diverse criteria so you would probably not be the top result on _all_ dominant search engines. SEO spam needs upkeep and attention to latest algos, else it decays. Competing algos would yeld better results for everyone. Maybe Google is just ripe for a shakeup.
Well... Yes, it should. But, no, it seems it does not. I thought about this when typing it but did it anyway, maybe because I thought there is still something worthwhile there.
I still think the model could work if the algorithm is sufficiently different than Google's. Ideally, people would go "I did not find anything I cared about on Google, I know, I'll use Bing!" - but nobody does this, because the results are consistently worse.
Don't get me wrong, I like G as a company, I think they do worthwhile things! But they have left things slip and need competition into this field, I mean real competition, then maybe they would actually address issues.
Maybe the issue is also on the incentive level as well. I mean more searches means more eyeballs and more money for Google. If someone searches one thing and they are done that is less interaction! I hope they don't work like this, but it's possible.
And another possible problem is the opposite. Maybe Google is optimizing search for what it thinks people want, but it uses the wrong metric. Or it gives people what they want but not what they need.
I think the problem is just that the solution isn't in Google's wheelhouse: There is no algorithmic ranking system that can't be gamed. Human moderation and curation is the only way to provide true quality, and Google is allergic to solutions that don't automate and scale.
I think a really good search engine would still algorithmically search it's index, but the content library should be human-curated with a goal of ingesting content via author, not via platform. Once a given author was human-approved as a quality source of information, content they produce could be automatically ingested going forwards, and conditionally re-reviewed by a human if there were reports the quality had decreased.
This was Yahoo in late 90s early 2000s. They had a human curated directory search where one could look up something like "kayaking" and find a bunch of sites on kayaking. Then if you wanted to search on keyword it was outsourced to AltaVista and later Google. Altavista results were terrible and were almost nothing more than a keyword search (IE the word you were searching appeared on this page). Google got much better at the general search and this was history.
I think the death of the directory search dramatically dropped the number of self-curated, informative sites from a domain expert that were common in the early internet. Now instead of making a website, many people are on content silos like Reddit/FB
I do still think we could adapt this model on top of content silos... assuming we can index them! Consider that one could also, rather than just ingesting Reddit content, we ingest new posts from particular users who write quality posts on Reddit.
Assuming a method also existed for an author to authenticate themselves with the search engine, one could also enable an author to help identify their content across multiple platforms, as well as suggest other quality authors to consider.
> The funny thing is that if the people who worked on spam at Google were free to talk about it, I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence.
That may be true, but I think one of the good points made on the OP is that it might actually be cultural constraints that keep them from solving the problem:
> You might need to do a lot of manual spam fighting initially. That could be both the thing-that-doesn't-scale, and the thing that differentiates you by being alien to Google's DNA. (They must hate manual interventions; so inelegant).
Google has some very smart and knowledgeable people, but the things they do have to fit into certain boxes, which means there are some problems they just can't fix, e.g.
* Everything has to be automated at scale, which leads to consistent poor user experience (unappealable account closures initiated by inscrutable algorithms, SEO spam).
* You get promoted by building new products, not maintaining existing ones, which leads to self-defeating churn outside of core areas (e.g. abandoning Google Talk and squandering their position in the messenger market).
But why is Google even dealing with spam? What if they (or someone else) curated top websites for a given category? For instance, when I search for a programming-related term, I already know that I want to see the answer on either Stack Overflow or one of a few reference documentation sites. It is possible that some other site could have the answer instead, but in practice the random sites that often show up at the top of the results are usually SEO spam. A search engine that figured out or let you select the semantic space you are in and then promoted known websites - maybe ones you curate yourself! - would be a big improvement.
Of course you can always hardcode the site you want in the Google search results but this is hacky and not very expressive.
I'm sure they know all about it, but are prevented from doing anything by the business model. Pinterest has been spamming up my search results for years. Maybe other people find it helpful, but I do not. It's obvious I am never going to get value from Pinterest. Let me click a button to add it to my block list. One single click would have given me years of massively improved results.
The fact that this feature does not exist shows that there is something deep within Google's core that is preventing them from addressing SEO spam, just like there is something deep within Airbnb that makes it difficult to filter out Airbnbs with problem reviews.
Google has been coasting for a good long time and now major players are realizing they are wide open for disruption.
You talk about this "constant assault from spammers" like it's not Google's fault and it's an intractable problem. That is not a correct characterization. There are plenty of low hanging fruit that could easily be detected and deranked, for instance scraped stack overflow spam. But google chooses not to deprioritize these results. The reason they don't is that they make money on ad clicks, which many responses have already elaborated on.
I think the issue is that these crappy results are kind of good for revenue. It’s not just organic results impacted but all the affiliate ads.
Google is smart so I assume they crunched the numbers and figured out they make more money from people filtering through crappy results that include viewing and clicking ads than by surfacing good content.
I think Google is optimizing for ad revenue, not for good search.
I believe that the only moat protecting $100B of AdWords revenue is the quality of the Google Search results. There is no meaningful switching cost to using a new search engine, and the spend inertia in ad spend is not very significant (i.e. any online marketing manager will happily spend 5% of their budget in a different search engine adwords-like program if they get better ROI, there is no incentive the be a "Google Ads only shop").
On the other hand, Google needs to maintain the ballistic trajectory of its revenue growth. So how can they fix search quality when they've minmax'd themselves into this situation in the first place? If they were to make the ads background yellow again, that would have negative short term effects that I doubt any career exec can stomach.
The other moats are lock-in to advertising networks, website metrics, and effective control over Web standards through the Chrome browser.
An alternative search platform might provide better search. It would be fighting Google on at least three other fronts. It might have some success, but it would be challenging. (As history largely demonstrates.)
Even a rival tech monopolist, Microsoft, barely holds even with its own search offering (I use that indirectly via DDG), and scrapped its own web-browser development
I mean, Google is big and it has that advantage like any big enterprise, but the search engine market is very permeable compared to what people traditionally refer to as a moat (say, trying to compete with YouTube as a video hosting platform or with Salesforce as a CRM).
If you have a good search engine, people will flock to it, and search ads will be valuable. That's it. That's how Google became Google.
The fact that Microsoft couldn't do it honestly doesn't mean much. Microsoft also couldn't do a phone OS, a portable music player, and many other things. They have a complex web of conflicting interests that $SEARCH_ENGINE_STARTUP does not.
Just for the record, I'm in at least mild agreement that search is looking increasingly vulnerable. Google are falling down here.
It's just that "search" is really a web of interrelated services, capabilities, and revenue streams, and they tend to reinforce each other strongly. I'd like to see the monopoly disrupted.[1] But I don't think it's just a matter of "build a better mousetrap^Wsearch engine." Attack one corner, and Google will snipe at you from the others.
And with the AdWords cash cow, they've got an immense revenue stream.
________________________________
Notes:
1. Well, mostly. Google's acquired so goddamned much personal data that the premise is frankly kind of terrifying as well --- a weakened Google with neither the revenues nor talent to defend that pile.... And I'd really like to see the toppling occur without simply raising a new monopoly in its place.
“ There is no meaningful switching cost to using a new search engine.”
Unfortunately, defaults matter and Google is spending billions of dollars yearly to make sure they are the default search engine wherever they can. Most people don’t switch from default.
The way I see it that's only a problem if you want to be bigger than Google. But you can get to $1B in revenue with just the deliberate adopters (i.e. under 1% of the market).
>This may not just be a problem with Google but possibly also the recipe for beating Google. A startup usually has to start with a niche market. Why not try writing a search engine specifically for some category dominated by SEO spam?
>You might need to do a lot of manual spam fighting initially. That could be both the thing-that-doesn't-scale, and the thing that differentiates you by being alien to Google's DNA. (They must hate manual interventions; so inelegant).
Is he describing...Yahoo circa 1994? A manually curated directory service.
I wouldn't just complain about Google. Google search results mostly reflect a deeper problem with the web today. I do miss the simplicity of the 2000s.
And makes me think that StumbleUpon had a similar curation ability, in that the value qualifier is how often [hopefully] real people interact with content - tracked by who's using SU and agreed to allow tracking; can't remember if sharing that was optional or not?
The gamification of the system then would have to come through onboarding fake users, pretending/mimicking real user behaviour to send that signal into the system; not sure if SU ever ran into that problem or was actively paying attention to trying to identify and removing fake or suspicious signals from their output?
I feel a much better system is easily within reach, it's simply getting the right structure to it, the right foundation, and then it will quickly take off due to the quality difference. I've already figured out a design pattern that Twitter and Facebook has indoctrinated us with, making us think it is normal - and keeping us blind to an actual normal way or organizing or communicating, but that isn't conducive to control or ad revenues - and so extending my future plans to include a better search-directory system would fit snugly into my efforts.
Some of the examples used in the Twitter thread Paul was referring to would be better served by a manually curated directory service with a possible addition of a search engine only surfacing content from the sites in the directory.
For health information and recipes in particular there are only a handful of really high quality sites that have quality content for 95% of the information most people need. I bet if you wanted to increase the coverage to 99%, that list would expand to less than a thousand sites. At those numbers manually curating the information would be easily achievable.
How to get people to use your top notch Google replacement instead of Google, however. That's the hard problem.
It's really hard to get people to use something other than Google. If you were to launch such a product, it would have to be so much better that people recommend it organically to other people.
That was what Google was to Yahoo/Altavista back in the day, a 10x improvement. Reading this thread, people feel pain enough do all sorts of hacky stuff - appending 'reddit 'or 'forum' to queries, blacklisting spam domains, switching search engines depending on topic. If G keeps declining and a new product does things better, the penny will drop and people will swap.
Siebel and PG see blood in the water no doubt, they see G's market share and want to fund companies to take some of this.
That's all fine and dandy, but the goal isn't to just make some good sites a bit easier to find, it's to keep the top of your search results from being interspersed or superseded by SEO spam. Unless I misunderstood your suggestion.
Okay. I was gonna respond with something snarky about what a crappy mod he was, or how he was a Saint and you don't deserve to worship at his functional feet. But I'll tell you what he was wrong about: He was, as a leader and a human and a mod, petty. He pied pipered himself into a sweet spot and no one would deny he's a good coder, but there the ego took off and forever left behind a skidmark. The cool exterior, the sense of self-importance, the punching down, above all the love of spreading one's revelatory wisdom to the poor little guy; you can love that sort of thing too much, and he did. Perhaps you weren't here or didn't interact with him directly. In my view he became dismissive and derogatory toward people who worshiped him (like you) once he acquired a small degree of fame.
Good idea. You could start with fitness. Lots of high-quality information out there that’s entirely, 100% inaccessible via google.
Over COVID, I did the whole fitness thing from a few different angles (overhauled diet, trained for a marathon, now lifting weights a lot). I found I could only find good info by going directly to a trusted source - literally, typing http://www. like I’m in the 90s or something. This is the exact issue a search engine should solve, but Google doesn’t.
Is your trusted source the same as my trusted source which is the same as my neighbor’s trusted source which is the same as my Australian cousin’s trusted source?
If not, at least one of us is going to hate this new search engine.
Social media and trial and error. I knew a bit, and asked some friends for advice. I used that to find people who said stuff I thought made sense on Quora, Tik Tok, and Instagram (ie they agreed with what I knew to be true and false, so I could assume the other stuff they said was likely to be true as well). I tried what they said, found what worked, and went all in when I saw results.
Importantly, this was bottom up: it was largely recommendation engines suggesting people I then filtered through for what I wanted (running, bodybuilding) vs didn’t (traditional weight loss). I couldn’t specify what I wanted, or it would be garbage SEO spam.
That's a new angle to me. I rely on pseudonymous content like Reddit and HN. It does make sense to look for people or groups focused on a larger topic like fitness.
What are some search categories that are so dominated by spam that they are unusable?
I'll start: "how to rent a car" [0]
[0] worth noting that I personally get somewhat reasonable results for this, with a 3rd result from nerdwallet.com and a 4th from wikihow.com, both of which seem to answer the question in an unbiased way
I think a lot of this is due to Google both owning search and the ads on the websites (AdSense). There’s an incentive for them to prioritize click farms (and other sites filled with their ads). I think in general there may be a correlation between the number of ads on a site and its usefulness to me, which is inverse to its usefulness to google.
I’m curious what would happen if those products were split up into 2 separate companies.
I also can't help but wonder this. I'm certain people at google search want to provide the best quality search results and do this with integrity. But at some point in the business hierarchy you are at a level where people set objectives for both these departments ( search & ads ) and are trying to optimise for things like total revenue/profit.
> Maybe ultimately you open up spam fighting to your users. If you managed this well, you could harness a lot of energy.
Doesn't Google already consider that if a user returns to the results page (or clicks a second link) then the first link visited was not satisfactory. Seems like a pretty elegant solution.
> What would a paid version of Google Search results look like - where Google can just try to give me the best possible results and not be worried about generating revenue?
God please no. YouTube premium shows what Google would do, i.e., they would further ruin the free experience by ramping up the amount of ads you see to "incentivize" the premium search.
Premium offerings like that are amazing simply for the fact that you can 'return' to the days where you obtained services by paying for them directly, not by looking at ads and paying with your mindshare. Google and YT aren't free services, and it's a miracle they continue to be accessible with ad blockers enabled.
One example, the website called gitmemory which crawls github data regularly and have better SEO than github that usually you will find results above original github links.
Several people mention DuckDuckGo in that Twitter thread. I use DuckDuckGo for my main search engine, and it's not obviously any better than Google regarding SEO spam.
I'm honestly not trying to take a potshot against PG or YC here, but it's kinda funny to see him saying this after I worked for a YC-backed startup years ago that built its core revenue streams around generating SEO spam, we just marketed it as something else. Just to be clear, I don't think PG or YC are responsible for all or even most SEO spam, but I know firsthand that they've profited from it through at least one of their incubated companies.
I never considered the possibility that an incubator would support a specific product, then later on call for alternatives that would essentially freeze out the original product that they supported. I'm sure this very rarely happens, but it's interesting to see a real-world example in action.
I don't think it's as counterintuitive as it sounds - just because they're playing the game doesn't mean that they think it's right. If your options are to not be successful at SEO or to do the SEO spam thing, I don't think it's necessarily wrong to do the latter - it's your job to make your startup successful, not to make a stand against the way Google does things.
I view it as something like the rich folks who call for additional taxation of the rich. They're not going to just pay extra money that they don't have to under the current tax rules, both because it's not particularly fair and because one person paying extra taxes, even if they're very wealthy, isn't going to make a big impact. That doesn't mean they can't lobby to change the rules and be totally fine with it if everyone is paying additional taxes.
Don't know how rarely it happens. After all, weapons dealers frequently arm both sides of a conflict. They don't really need to care who wins - they're just making twice the amount of money.
Interestingly, Google Maps doesn't suffer as much from the issues with Google Search. Maybe because it has those community-driven curation features that PG is talking about? Google Maps is fantastic at finding places to go to (and getting you there).
Also, why hasn't Apple built a search engine yet? It baffles me that they chose to go head-to-head with Google on Maps, yet outsourced their search engine. I would've liked it the other way around: Google Maps and Apple Search.
It’s much harder to get SEO spam style content to maps given the geographic region limits involved… But it happens my favourite example is searching for suppliers of something basic, say structural aluminium extrusions, big and heavy and you ideally don’t want to ship it far so it’s an ideal thing to search for a local supplier of. In Australian results is basically a given that I’ll get results for my city because they will list it as a delivery area they supply to however when you actually try to find them on the map as a pin, nope it’s either not their or just is a sales office not a warehouse or workshop, so they have tricked the system into listing them as local to my area in a way that pollutes my maps search results.
But this only works for certain industries. It’s much less common to see this kind of tactic if your searching for say a coffee shop because they sheer number of local results let’s Google be “hyper local” with these kinds of results.
Google Maps is spammed by fake locksmith or other trades that make it look like they're local but all route to the same boiler room (probably right next to the tech support or IRS scammers) from where they dispatch a crooked & most likely unlicensed tradesman that will do a poor job and overcharge you (destroying the lock so they can sell you an overpriced replacement instead of picking it, etc).
For licensed trades the solution is to go to your official trade licensing body (for the UK it's the NICEIC for electricians and the Gas Safe Register for gas/HVAC technicians), for unlicensed ones it's more difficult. There are "review" sites that claim to provide good results but their business incentives & vulnerability to spam/fake reviews are unknown.
> Why not try writing a search engine specifically for some category dominated by SEO spam?
Back in the olden days, there were lots of organizations that collated high quality content from the best writers. They nurtured expert writers and paid them well. They fact-checked the content and employed diligent editors and proofreaders so it was accurate and well-written. Over the years, they'd build a reputation for reliability and trustworthiness that kept people coming back for more. If you wanted to learn about fitness, or cars, or cooking, or science, you'd find a reputable author and publisher and buy their magazines or books.
But then, in the early 2000s, the geniuses from SV "disrupted" the publishing industry and its financial model. They brought us a much better way to find content, the search engine. Because they were so much better than the old-fashioned publishers, search engines gobbled up the advertising money and became the dominant gateway to content. Publishers had to abandon expensive high-quality writing because rankings and eyeballs now mattered more than quality and trustworthiness. Instead of investing in writers, they invested in marketers and SEO specialists.
The result: worthless content, writers banging out garbage for peanuts, and useless search engines.
Two decades later, looking at the barren wasteland they had created, the SV geniuses thought: I know what we need, more search engines, but smaller ones that collate high-quality content from the best writers. There must be money in that, right?
I think you're unfairly putting blame on Silicon Valley. Publishers were only able to produce high-quality content because, with no conversion metrics, advertisers were willing to overpay for placement. Tech undermined publishers' revenue, but what it revealed was that people don't actually want high-quality journalism, they want entertainment, and they're definitely not going to pay a premium for it. This was hidden behind publishers' business model.
>Publishers were only able to produce high-quality content because, with no conversion metrics, advertisers were willing to overpay for placement.
This implies that big budget advertisers (the CPGs, like Coke and P&G), are buying Google/FB because they have better conversion metrics. That isn't true today; only SMBs and gaming companies care about conversion metrics. There are interns in LA/NY probably collectively spending millions on FB for P&G and only reporting the number of likes back to their bosses. Google and FB has never meaningfully delivered on conversions past anything like app downloads.
Tech undermined publisher's revenue because the internet cratered distribution costs. Advertising revenues for big media crashed because the eyeballs moved away, not because it was any less efficient.
Did any of these heavily buy newspaper ads before 2000? Definitely TV, possibly magazine, but newspaper? I just don't remember seeing ads for Tide in newspapers.
I thought for a moment about your reply. Do you remember Saturday / Sunday coupon sections? These are essentially adverts. Those are 100% consumables -- like branded health, food, and cleaning products.
That's a fair point; I forgot about those. That said, the earlier comment was on conversion metrics. Unlike brand advertising, brands track conversion metrics on coupons closely.
I stand by my point that tech just exposed a bad business model. Newspapers were only viable because they were the ~only game in town.
Now what actually killed newspapers wasn't search, it was Craigslist killing the classifieds...at least according to a study Google paid for.
It's been really sad to grow up and watch the cool techie optimism of the 2000's internet get sucked dry by profit motives and left to rot. The change has occurred pretty much entirely within my adult lifetime (I'm only 27 and I still remember when Google was the cool new thing on the internet).
It went from "search engines and the web will usher in a new era of wisdom and democracy" to "useful content is dying at the hands of monetization schemes, and also the internet will be the death of liberal democracy, woe unto us all" in about 15 years.
I tried creating a search engine for recipes. It works well and people like it, but the struggle is no one remembers that it exists and Google is just their default for search.
So from an individual developer perspective, it's very hard to get people to change their habits. And Google/duck/Bing is the one stop shop.
It's still out there, but I haven't worked on it much lately. I always think that if I had some good advertisers, a better UI, and a salary coming in, maybe it could take over some of Google's usage!
> I tried creating a search engine for recipes. It works well and people like it, but the struggle is no one remembers that it exists and Google is just their default for search.
One piece of feedback: It seems to somewhat liberally fuzz the term, doesn't tell me about it and not toggle it in the UI. I searched for "Natto" and got some great results at the top (Spicy Kimchi Natto!). However, a few results down the recipes start including "Nattu" which seems to be a Indian chicken dish and then quickly have nothing to do with the term at all and give me stuff like Tafelspitz.
Yeah, that's the typo detection in search. The key thing is that "Natto" is prioritized, so that part is working correctly!
Tweaking search can be a whole full time thing. I'll write it down and maybe look into if I can tweak the search so if there are exact matches to not show alternatives.
Yeah, that was part of the long term monetization plan. Paid for user accounts ($3ish/month) that allow people to clip recipes, view previews, discuss, modify, and share.
Never quite got around to it unfortunately, as I wanted more users before new features.
I think the name may be part of the problem there. Most things people want recipes for probably don't contain or go with garlic. E.g. if I wanted to make a cake, garlic is one of the last things I'd think of.
Something else that has largely disappeared is that there used to be a fair amount of organization of content, whereas now a lot of content is just thrown into a big pile and the user is left to go fishing on their own with search engines, whose ability to search seems to be declining (ie: Google often seems to no longer support mandatory include/exclude search parameters). Generally speaking, the result seems to be decreasing order and increasing chaos.
Of course, the massive volume of content creates a fundamental problem, but user curation & categorization on sites like Youtube would be possible, were Google to provide the software support so people could do that. Whether this and similar decisions are deliberate or accidental is likely one of those things that we will never know.
Oh, they were just usually wrong on everything else. Fortunately, these days we have individuals debunking the nonsense. Back then, people just uncritically believed total horseshit.
The invariant has always been: find people who make falsifiable predictions and improve. Back then the pool was small and you had no choice. Now, fortunately we have a choice.
I guess there's also the rise of influencers in the mix here. The commoditization of publishing means content creators can more easily work independently.
>Because they were so much better than the old-fashioned publishers, search engines gobbled up the advertising money
No, what happened is that the publishing industry lost their monopoly. They could no longer extract monopoly rents from advertisers.
One problem with search engines is affiliate marketing. If Google de-indexed the junk affiliate sites, the web would be much less polluted with affiliate spam.
> No, what happened is that the publishing industry lost their monopoly. They could no longer extract monopoly rents from advertisers.
Publishing was not a monopoly. Google/FB are a duopoly. If publishers capture rent from advertisers, they plow it back into content, aka the thing consumers actually want. If Google/FB capture rent, they don’t provide a living wage to content creators and plow the money into buying other startups and whatever “metaverse” is.
I cannot find the link now, but there is great one page graph / chart that shows all the categories of Craigslist that have been cloned/displaced by start-ups. Does anyone have a link to that report?
554 comments
[ 0.24 ms ] story [ 313 ms ] threadhttps://www.gnod.com/search
There in fact are many vertical search engines. You can click on "more engines" to see the whole list.
I have an exterminator who comes to my house every couple months, and sets up traps here, poison there. I don't have any rats in my house. I do see rats running across the yard sometimes. The exterminator explains it like this: Rat pressure. The rats overpopulate and there's "pressure" (like, uh, "memory pressure", which is also a fluid concept) so they try to get into your house more, through smaller holes, as a function of how much outside drama is going on, how many they are and how overpopulated, how scarce their food supply is, how cold it is outside, and whatever else drives rats into your house. (I love the dude who's my exterminator).
Anyway, this is the same problem every search engine faces. The more surface area they expose, the more pressure they have building, the more ways people have to fake out their systems.
We have to go back to the 1990s Yahoo! model. Curated content. A list of websites that are reputable. 1990s Yahoo is the future.
Carte blanche opening your system up to anyone to inject data seems like the wrong foot to start off on, whereas my curating a moderator, someone I personally know and feel good about, trust to whatever level, and hiring them - ideally making sure they're someone you respect and you're someone they respect, pay them well, and at scale will be able to pay for itself; this did just bring to mind however big pharma and pharmaceutical trials structure and how that system can be/is/has been captured - and so perhaps the pressures when dealing with multi-billion dollar market categories will always lead to shenanigans if ever trying to centralize too much, not allowing for de-risking and broader resource distribution via sales/profits to more parties than the "5-star" rated products.
In a thread on HN, I think it was yesterday, a few people posted about review sites where some product reviews are free - but others you had to pay for. A system to facilitate such organizations could allow a highly competitive environment, where organizations develop/build a brand - build trust for their brand as being competent and thorough - and so then over the lifetime of a reader/customer, perhaps they'll spend $1,000 buying reviews (say 333 big purchases over 40 years that you're willing to pay $3 a hit for) to make sure they're ; mind you there will be organized that could be captured to say promote one conglomerate of products over another, perhaps even regionally, but I'm beginning to think it's a necessary layer to combat the shit show that is Amazon (et al) reviews. Ideally these systems and how the reviews present the information, and how thorough - the technical depth and breadth and testing done - will help educate those who dive into using this system, which will sharpen themselves while keeping reviewers on their toes and arguably strengthening their organizations and competency as well.
Then Google came along and was a better search engine, for a time, that was a traffic leak for Yahoo! - and then Google has now devolved; I also thought Google had a good shot at competing with Facebook, but whomever's pulling the strings there, the launch of various platforms, they don't seem to understand it can take 5-10 years after the MVP of a product is launched for it to mature - but for whatever reason their executives or managers haven't been comfortable pulling the trigger, arguably because anyone with that entrepreneurial spirit just takes their idea and gets funding and owns a large portion of whatever they've done; but then you can never develop a full breadth, holistic ecosystem, that can grow into every crevice, nor as broadly, or nuanced as possible - so they're stuck being Search, Gmail, Calendar, etc.
I'm quite certain I've figured out the foundational MVP facilitating an "infinitely" growing system and that would allow 3rd parties to integrate, however I have severe chronic pain that messes up my executive function, so it's difficult for me to actually self-direct and execute - I'm stuck mostly in a low activity, stream of consciousness and go-with-the-flow life of routine - otherwise I would try to launch my plans, which I've done plenty of UX/UI for, as that is simple enough that somehow bypasses higher executive function (moving a pixel and then responding via visual feeling of it isn't complex) - but organizing to turn that into adequate specs to get solid estimates or fixed price quotes for work is extremely difficult to me.
On January 11th I do have a surgery that may or may not reduce my pain by 50%+, may or may not improve my executive function, ... I've even attempted to write draft "Show HN:" posts to explain what I am doing, the starting feature sets, the reasoning behind the design decisions I've made - but it just gets too complicated too quickly for me mentally then to be able to organize further or polish it. I think I have the perfect domain name for it too: ENGN (engine), what makes me smile every time I notice it in my layout/mockup of it is in the search input box it says "Search ENGN". My username on HN actually is an older incarnation of a plan I had, loceng being a short form of "local engine" - and ENGN being from engine, a name I brainstormed after Tumblr sold for $1.1 billion - and I realized that eventually I'd want to try to do my "local engine" idea but that that was too long for a brand name. Fortunately engn.com was for sale at the time, I can't remember if it was $2,000 USD or $4,000 USD - either way, not a bad price for a 4-letter .com that's pronounceable to something with meaning.
I've wanted to write a book too - on health, health systems, and on these systems we're talking about here. I'm 38 now and I taught myself to program when I was 11, learned SEO at 15, evolved to design as I'm more creative and programming became mindnumbing to me, and eventually thought I'd need (or want) VC money - so I started engaging on Fred Wilson's of USV.com's blog - AVC.com - so I have plenty of self-taught experience. The problem is even going back to my shorter or...
Listen, first of all, do not consider ending your life. Seriously, you're way too smart for that. I'm sure you've got it worse, but I've had enormous sciatic problems in my life, I've had 3 herniated discs; they're behaving for the moment after massive doses of cortisone and without any painkillers, but I know what it feels like to cough them all out of my back at the same time. Not to be able to put a foot in front of another or turn your neck for weeks. (I'm a huuuuge fan of intramuscular cortisone injections, though. Like 5 or 6 large cortisone over a week, with some B-12. Every couple years. Not in the spine... fuck that. Alternating butt cheeks. You won't feel any benefit until the third day at least. If you can convince a doctor to give you that for a week, you will be fucking superman. They won't do this in America unless you know a doctor personally, but they'll do it in Mexico or Spain. I had it the last time my discs went out and it's been 6 years and the inflammation has not come back. They thought it would).
Anyway, before you off yourself, do try a fuckton of intramuscular steroids. The fifth day I levitated off a bed in the hospital; I hadn't walked in a week; I felt so good I went to a club; I got drunk and spent the night on a beach drinking and making out with an 18 year old model from Denmark. Seriously. There were wild cats walking around; it was winter on the Spanish coast. If you do one thing before you die, go get five cortisone shots in your ass, in a week.
I also got the hiccups for 24 hours and couldn't sleep, but that's neither here nor there. And I got temporary blindness in my left eye from fluid behind the retina, caused probably by too much testosterone. But. Goddamn it, I'm ok. You can be okay.
Enough about that.
About Yahoo and Google. That entrepreneurial spirit is, in my experience, way too often just about getting the funding and fucking off. We all know why these companies go downhill, but somehow it's always such a shock when they actually deteriorate in front of our eyes, huh? Google's search results, for instance. I would have expected their cofre business to stay more or less fine, not collapse a couple years after all the competition was eliminated.
It would be fine if they didn't grow into every crevice. Get search right, that's all we ask. I don't want Google to be my chat room or my shopping site. Why do they need to? Search is huge. They own 90% of the market.
>> but organizing to turn that into adequate specs to get solid estimates or fixed price quotes for work is extremely difficult to me.
That's always the worst. The business side. I've always just built things and hoped for the best. It sounds like you've got something interesting going there, although I have no damn clue what you're building, that's an exciting feeling. ENGN is killer. If you own ENGN.com, hell, money well spent.
I don't understand what you mean about "executive function", since you obviously have the capacity to write well-crafted email and think pretty clearly; perhaps I lack the executive function to discern your lack of executive function (I'm a brutally self-punishing alcoholic, but otherwise a damn good programmer)
Anyway I don't know if you're trying to ask for pointers to workers for this concept, I'm probably not it; I'm $200/hr and I'm already covered for the next year. This, however, should be your symphony. And I think you know how to do it.
and then the original vice documentary: https://www.youtube.com/watch?v=VaMjhwFE1Zw
and then tried the breathing: https://www.youtube.com/watch?v=tybOi4hjZFQ and cold showers and they've been helping a lot with inflammation!
There are tons of benefits of cold showers, so if you can start doing that, that might already help a lot! Here's a bunch of results you could check out: https://www.youtube.com/results?search_query=cold+showers+be...
Central sensitization and hyperalgesia is really a different kind of beast when it comes to pain, people including most doctors really seem to have a hard time understanding it. That added stress to my nervous system from opening my right eye and triggering/ramping up the sensitization/pain from just low activity and careful movement was enough to lock up my thinking, and arguably my emotions, but more specifically the locking/the eye pain and increased pain from movement makes fluid thinking have much more friction added to it; one good or as relatable as possible thought experiment for a "normal"/unaffected person may be to imagine the feeling when you get something in your eye and you desperately react to get it out because it's so painful: now imagine that foreign object feeling is immense because it's permanent and broad, it's "a lot of objects" in your eye(s) because your cornea and nerves were sliced across 90%+ of the cornea (and so abnormally/constantly signaling as such), and imagine how your brain/nervous system may try to cope from such an overwhelming/overriding system constantly firing to draw your full attention to what your eye is telling your body is an active/present moment object in your eye - potentially as if you just walked into a sharp object and your eye is signalling for you to literally freeze still because it thinks you're in the moment just done something to critically wound your eye and moving another millimetre or less would threaten your survival (as the evolutionary strength of the reaction has come to dictate).
Well, the answer is some people after LASIK get this severe reaction, their nervous system gets overwhelmed - arguably the more naturally sensitivity, creative, healthy, and grounded people will have stronger/more detrimental symptoms/reaction to the eye damage; central sensitization and hyperalgesia that LASIK for years completely swept under the rug as being possible and only recently admitting to being a potential "side" effect. In fact they purposefully mislabeled what should be called corneal neuralgia syndrome as "dry eye syndrome" to mislead people away from learning that the "dry eye" part is actually on a spectrum of symptoms caused by the damaged cornea/nerves that happens in 100% of their surgeries; non-LASIK done research done has shown up to 40% of people have permanent problems after LASIK, and in 2011 one of the expert FDA advisors, who votes yes to approving LASIK, published a letter to the FDA asking them to immediately recall LASIK because there was data they were ignoring that they should have never been ignoring, and that it should have never been approved in the first place. Part of the medical industrial complex where arguably regulatory and institutional capture has occurred in the name of profits.
So again, I'...
Not necessarily: https://yacy.net
This is a good perspective. Where can Google not go? Places that don't lead to profit. They will try (cough Wave cough) but will give up.
E.g., Linus wasn't looking for profit, and Linux ate the world.
This is very true. How many times have I clicked on a site met with ads so bad that the browser slows down, and after 10 seconds the page gets covered up by more and more crap and then a paywall shows up sometimes too. Now here's the thing - a competitor to Google might detect you clicking back and then pop-up a special set of controls near the search result that lets you say: "too many ads" or "paywall".
However, if such an engine were to start beating Google, I'm sure Google would implement it in their own way: automatically detect why you clicked back in such a short timespan.
Perhaps ML will help in detecting such campaigns.
Do you seriously believe that Google doesn't use that as a datapoint already?
I'm not saying that curated search results for particular verticals is a terrible idea (though I'm sure like anything the devil is in the details), but on the whole Google search is very, very good considering the constant assault they are under from spammers (which most other search engines are not, at least directly).
There is no scenario - none - where thousands of engineers at Google working on search wake up in the morning and say "we sure have made it good enough wr2 SPAM. I think I'll have another Danish."
I think it’s more likely that because they are just building hundreds of tiny tweak experiments and it’s someone else who desides what to build and if it even worked. Search quality is such a meta-problem that it goes beyond any real hope of simply working on it in anything beyond piecemeal trial and error fashion on their dataset.
https://en.wikipedia.org/wiki/Danish_pastry
I don't claim that Google has any of these and certainly have no insight into their search group. But I've personally been at powerful companies with best-of-the-best talent that were blind to the decay in their own living room, so I would caution against immediate dismissal of PG's take.
The hard part in all of this isn't finding and stopping spam - it's defining what spam is. Are all the pie recipes where there's a 2000 word essay about their grandma at the top 'spam'? They still have the recipe, and Google Home devices pick up the recipe instructions just fine so people end up not reading it, but many people would still consider that spam since it adds such an obstacle to getting the information you want. Same for cnet articles like "Best smart home devices to buy in 2022" - it's a reputable brand with a list of smart home devices, but it's hardly a review and exists to funnel people to their Amazon affiliate link.
What's the point of writing succinct, to-the-point mini articles about problems and solutions if nobody finds them on Google?
I've done blogging for the last 10+ years, and many of those I spent as a freelancer working with startups/brands/editorials. Everyone is after "word count" and I absolutely hate it.
Whenever I work on articles for my own blog, I just don't consider word-count at all. I think if your content is great and informative, then readership will be natural.
Later on, I sold it because I needed the money. Not so much that I didn't want to keep working on it. Unfortunately, the new owners didn't have any idea how to maintain a "healthy" content blog, and it has plummeted down to around 30,000 monthly visitors. All the content they're publishing now is some thin headline-clickbait bullshit.
I even gave them free advice on how to fix it, but I think that for a lot of people, they just don't care and will mindlessly pump out as many pieces of content as possible. And such blogs can be identified from a mile away.
And therein lies the problem with Google SEO at the moment. Even myself, someone who has done SEO work for more than a decade, I can see that results are getting worse. In some niches, the same crappy articles that dominated 6-7 years ago are still dominant today.
I guess we're stuck in time, or so Google thinks.
Incidentally, prioritizing long content seems odd to me, in my experience the best pages are short and get right to the point, at least in the context of something like a recipe or other "how to" resources.
Google's algos, while advanced, still rely a ton on text to actually tell what the page is about. They need it.
If they just relied on other factors (title, links, website, etc.) they would end up with worse results for users. Im sure they've tested it.
Google's core algo in a lot of ways is much simpler than people think (in other ways of course it's very complex).
(The fact that many of these books were rushed out with very poor quality control also didn't help.)
I think this isn't entirely related, but that's perhaps the beginning of a bias you might end up having that everyone experiences technology in the same way as it marches on. I've yet to encounter a Google Home in the wild, I imagine far more people are consuming recipes on phones, tablets and PCs.
Hmm why stop there let's actually make the users do the curating and even the content creation by rewarding them with social validation. Let’s have hard working moderators who work on the community full time.
Then we could just build a search engine over it. We could call it Reddit. Or HackerNews.
Maybe the users aren't all as good as professionals at curating the information. Let's hire professionally trained curators pay them well and we could call them newspapers. Then we can come in disrupt them and replace them with an algorithmic marketplace that eventually becomes infested with click bait.
This is one area where Google could use personalised results to provide a better experience for the user. Let me decide what spam is for me. Let me mark results as good or bad, so that the algorithm knows what kind of pages should be prioritised or filtered out the next time. Google SearchWiki was a step towards this but they killed it off.
We have seen what this leads to inside the social networks as well as YouTube, and at a macro scale I think we might want to have a shared concept of what constitutes a good search result for a given query.
At micro scale, it can seem more optimized to get exactly the type of result you want, but if we take an absurd example like an Apple Pie recipe shouldn't we all have shared understanding of what types of ingredients would make for an Apple Pie?
The shared understanding, I believe, is core to communication. If all of us have our own specific ideas of Apple Pie, then who is actually right on what an Apple Pie really is? What happens when your search results insist that an Apple Pie doesn't actually have apples in it, but instead pears?
At some point, Google must have moved away from using site-level reputation in search rankings, as I almost never see recipes from reputable sources like King Arthur Baking, Serious Eats, or Food52 in the first page of results.
Not only that, Paul and Michael have seen plenty of startups, and at least in recent memory, the number of vertical search and consumer startups that Y Combinator has funded hasn't been that high
As a consumer startup, I know this issue firsthand. Paul and Michael assume that if you build a better product, they will come! That's simply not true these days.
Instead, you need to:
- Build a better product
- Option 1: Figure out a channel with enough growth on an existing platform. This likely means you're doing SEO for your new search engine
- Option 2: Get your customer lifetime value high enough so you can pay for ads. This is tough, since it's a bit of a chicken and the egg problem since most search engines are monetized with ads
As the founder of Wanderlog (YC W19; https://wanderlog.com), a consumer vacation planning app [1], I definitely remember the idealistic days when I thought the best consumer product on its own would win! But growth doesn't just come, and the same can be said of vertical-specific search engines.
[1] Try searching "[your city] itinerary" on Google vs. Bing: it's much more likely you'll find a small blog rather than Lonely Planet or the local travel bureau as the top result
nonoonononooonono. No. Don't monetize anything for the first 10 years. That's the only way it can work. Then you can go monetize it and buy an island and not give a shit if you destroy what you created.
Oh but don't worry. You'll have investors.
Bing: https://i.judge.sh/ShareX/2022/01/www.bing.com_search_q%3Dat...
Google: https://i.judge.sh/ShareX/2022/01/www.google.com_search_q%3D...
Interestingly Google didn't have a top-result ad and the google.com/travel carousel is 4th from the bottom.
For the actual results, both thefearlessforeigner.com and paigemindsthegap.com seem to be actual travel blogs (the pictures didn't appear in a reverse image search, so they are probably organic), but they're clearly geared towards being a 'faq' for visiting the city and have affiliate links where appropriate. Bing went straight for discoveratlanta.com, and frommers.com is well-thought-out but not a personal travel blog.
The best part of it was (going to a foreign country) being able to find / identify all the attractions relative to each other, so I could go to cluster A on Monday, cluster B, on Tuesday, etc.
The hardest part of it (and why I needed to create a separate google sheets anyways) was--once I figured out opening hours of different locations, hard-to-book activities with limited reservations--the ease of moving things around more fluidly e.g. cluster B on Monday, cluster A on Tuesday, etc. and having a more information-dense view so I could see larger portions of the itinerary at once.
It would be cool to have an "input everything" --> "input time restrictions / unmovable things" --> output planned activity cluster type workflow.
It's like the bad-old-days of ExpertsExchange, which somehow was never delisted by Google for its shady SEO tactics.
As a user, your best personal and ethical move is to install an ad-blocker, to make ad-based business models less viable, which will help promote business models that don't abuse the customer.
This seems a bit overly cynical. Some search engines only served ads, but they're long gone. The survivors are those who dedicated themselves to finding links which were responsive to people's search intent. They seem to have gotten into ads because it was the best business model in this market.
Some even made slideshows of SO screen captures and put that on Youtube, with a fake video or spoken intro to make believe an actual content will be discussed... A number of shameless people would go any length to grab bits of money anywhere and anyhow, and I've hit those links a couple of times.
This is really what made me suspect that Google was teetering on the edge of the MBA death spiral: these problems run for years when they'd be easy to block, which suggests to me that whatever metric gets you a bonus / promoted doesn't include things like that which are long-term threats to their core business even if it's selling a lot of ads short-term.
The only advantage a startup might have is that they could do completely new concepts, such as specifying what area you search in, allow you to modify their classification of your query and/or moderating sites you include - which is probably necessary anyway, since you'll hardly have the budget to fully index the web. I'm not saying it's impossible, but it's not going to be easy at all.
And after all of that, you still need a way to make some money.
The reality is more that some Google engineer will come up with an algorithm change that makes the result 40% better, but it will come at the expense of making that search 3ms slower so the change won't get merged. Or it will make the results worse for some niche set of queries that the business team really cares about, so again it won't get merged.
There are lots of consumers who would gladly pay $1 a month or whatever in order to use a couple extra milliseconds of compute power per per search in exchange for drastically better results, so there is lots of room for a startup to compete.
Google has a paid-for Search API, so they could do that if they chose to pursue it. And then they could let Google One users opt-in to the same thing via ordinary Search. I'm not sure whether Bing has anything equivalent.
This is hyperbolic, right? But they can solve spam in a split second, if they just admit they're watching you all the time.
[edit] /s thx for reading to the end, folks.
To put it simply, a new generation of the people who used to make the reliable niche websites that not just answered your questions but also helped you learn a particular topic have moved to youtube instead.
Google search is hollowing out as a result with the meat going and the SEO'd fluff that kinda answers the question but ONLY the direct question being asked with none of the wider expertise that more educated people in what they were searching for.
Of course google owns youtube as well.. so perhaps they just see it as an inevitable transition.
It takes time and effort to build up a spam site's ranking but it is trivial to blacklist those who get to the top.
The complaints I've read are from exactly the kind of generated content farms people are complaining about in this thread.
Really?
I can point you to Hard Problems that have been solved better at little startups than at Google - or, indeed, at any other bigco. That's why acquisitions happen.
Why does Google having 1000 engineers working on a problem automatically mean they are the smartest?
It's that Google has destroyed their own search results in order to continue to expand their revenue opportunities.
If Google:
- Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT)
- Allowed you to block certain domains from your search results, like YT videos. (If they added some kind of "coordinated network detection" and down-ranked domains coordinating with ones you've blocked, that'd be pretty cool).
- Allowed you to create your own custom search engines, like "Programmable Search Engine".
That would be incredibly valuable. They already have most of the tech. They could even create a subscription service around custom search engines if they really wanted. Plenty of people would find something like that incredibly valuable.
Anyhow, buried in there is your startup idea. Remember: your startup doesn't have to generate the same revenue or profit as the incumbent on day one to be successful.
Are there any search engines that do this? It's a great, simple idea.
Couple of positions up or down in google results for somewhat popular and valuable keywords can mean the difference in thousands of dollars per day of ad or affiliate revenue. I suspect it would get pretty wild if google launched something like this. There already are black-hat seo methods and services, but something so simple and direct would turn it up to 11.
They already have the tech to fight this on YT. They, in theory, are supposed to be doing the same thing to detect inauthentic behavior on ad placement and click abuse.
Why would they do this? Google's customers are the advertisers, not the end-users. And no one is going to pay for a search engine, it's been tried and has failed.
A company with an objectively superior search engine could make even more money with ads so now you’re back to the beginning
An “objectively superior” search engine, from an end user’s perspective, might have to make engineering choices that come at the expense of ad revenue.
But we’re all just talking hypothesis, it’d be cool to see someone launch a startup to get some answers.
It's not open source as far I know and there's only the free trial way to try it.
Always curious about things like this. I certainly would pay for this; it sounds like many other people here as well would. I'm curious if the constraint is that there aren't enough people to actually pay for the investment required for the service, or if there aren't enough people willing to pay to meet the standard VC notions of success. We seem to have a problem with building and supplying services for niche (read: "not expressable as an integer percent of the world's population") customer bases, and I'm never sure if that's a business problem or a cultural problem.
Nobody is going to pay for /just/ a search engine. But they might pay for, say, a /better/ search engine, plus additional features around gmail/gcal/gdrive.
Think of it more as subscribing "to google" and less as subscribing to "google search".
Regardless, the point isn't to "fix" google. It's to highlight a possible path for a new market entrant.
... If an existing player wanted to make a move here, I would say that both Mozilla and Apple are well positioned to add "personalized search" to a subscription service. Same with Microsoft. DDG could also make moves here if they expanded beyond search.
Google could easily use its current fingerprinting to constrain (to an extent) multiple votes. Even knowing only a portion of the population will participate in the voting they can use a Wilson confidence interval[0] or similar to properly weight votes.
Random sampling works here since you're not guaranteed one vote per user per page and the outcome in binomial, seen and downvoted or seen and not downvoted.
[0] https://www.mikulskibartosz.name/wilson-score-in-python-exam...
Internal Search owners can push for better algos, but what if the algo causes revenue to fall? Are there strong forces strong enough within the organisation to ensure that search quality prevails?
If this is the case, the problem is existential. It can only be arrested at the very top
https://en.wikipedia.org/wiki/Perverse_incentive
This gets close to the real root of the issue -- attention is monetizable independently of the quality of content. There would be much less incentive to create SEO spam if search engines negatively weighted pages with ads and affiliate links, and if manufacturers were barred (e.g. by the FTC) from owning or imitating reviewers.
This is also something that Google can control if competitors come along.
i.e. If a reasonable competitor comes along that is willing to sacrifice ad revenue for better search result quality than Google, google can just adjust their search quality upwards to knock them out (and then adjust it back once the competitive threat is gone).
Perverse incentives from Google are all over the place - Searching for the delivery business "Just Eat" in the UK for instance returns an ad for their competitor Deliveroo above the legitimate organic search result for me - and I can also see that JustEat are trying to pay for their own brand name just to compete - and IMO this sort of behaviour is anti-competitive, borderline extortion considering Google is the de-facto way of searching for a business, and shocking from a search-quality perspective (where the wrong result is intentionally shown at the top because they paid more money).
Things have changed in the past few years, now that Google has developed advanced transformer models, but for a long time Google's question answering facility has been: "let spammers make 10^8 pages where the title is the question and the answer is in the page".
The trouble is that there's a fine line between "answer is in the page" and "word salad!"
They're going the OPPOSITE DIRECTION from this!! They recently removed all downvote visibility of YouTube videos from the user, so now downvotes only feed into their algorithm. So in the last line of defense of me ending up watching a shitty video, one of the most valuable tools has been removed by my betters. It's preposterous that people think that Google is doing a good job. They're actively getting worse, and ignoring everyone saying so.
All the big channels I follow are mad at this change, and there's a coordinated effort to bring it back.
Google's search algorithm is tuned up for searching the whole web. It turns out the heuristics you need are very different depending on the size of the collection.
When Gerard Salton was doing IR experiments with punched cards he was working with collections of as little as 70 documents and in that case you are going to be very concerned about recall and not precision. Maybe there is 1 relevant document and if you miss it you failed.
If you had 70 billion documents you might have 10,000 relevant documents and if you lost 60% of them you still have 4,000 documents. The end user gets more results than they can sift through.
Thus I always groan when I see a site is using "Google Site Search" because the relevance is usually worse than you'd get with the alternatives.
Connected with that is the tuning work: Google has sufficient data to tune up a big model for everybody but true personalized search eludes them because they don't have enough data from you to tune up a model for you.
you mean the dislike counter they just disabled to force people to sit through more low quality content and pre-roll ads to claim increase in platform engagement and viewership?
The only thing matters is revenue and Google had increases in acquisition costs in prior revenue reports. Expect to see the data points for the latter metrics to be highlighted on the earnings announcement, and a record quarter for YT coming out of the change.
Not convinced this would help. The spammers would just hire people to dislike competitors
> - Allowed you to block certain domains from your search results
This I would use. Never show me results form collider, watchmojo, ranker,
> - Allowed you to create your own custom search engines, like "Programmable Search Engine".
I think this would lead to people writing highly polarized engines. The Red Pill engine for example and we'd have a new problem, the proliferation of popular highly biased results. Of course that's not to say Google's results aren't already biased but they certainly are trying to cover everyone.
I would love for Google to build this in. Until they do, there is a WebExtension that does this: https://addons.mozilla.org/en-US/firefox/addon/hohser/ ("Block or Highlight Search Engine Results"). I use it to block stuff like W3Schools so when I search for something, MDN is always #1. Saves me a lot of time having to add "MDN" to the end of every query.
Blocking pintrest would be a dream come true.
This is def a user-engagement strategy -- but it has cons as well.
Part of the complaints in the thread were spam related, other were something deeper
These sort of things are easy to spot, but only if you actually have a basic amount of familiarly with the topic. It's hard to spot with "AI" or super-cheap labor.
(Disclosure: I work at Google, but not on search)
I still think the model could work if the algorithm is sufficiently different than Google's. Ideally, people would go "I did not find anything I cared about on Google, I know, I'll use Bing!" - but nobody does this, because the results are consistently worse.
Don't get me wrong, I like G as a company, I think they do worthwhile things! But they have left things slip and need competition into this field, I mean real competition, then maybe they would actually address issues.
Maybe the issue is also on the incentive level as well. I mean more searches means more eyeballs and more money for Google. If someone searches one thing and they are done that is less interaction! I hope they don't work like this, but it's possible.
And another possible problem is the opposite. Maybe Google is optimizing search for what it thinks people want, but it uses the wrong metric. Or it gives people what they want but not what they need.
I think a really good search engine would still algorithmically search it's index, but the content library should be human-curated with a goal of ingesting content via author, not via platform. Once a given author was human-approved as a quality source of information, content they produce could be automatically ingested going forwards, and conditionally re-reviewed by a human if there were reports the quality had decreased.
I think the death of the directory search dramatically dropped the number of self-curated, informative sites from a domain expert that were common in the early internet. Now instead of making a website, many people are on content silos like Reddit/FB
Assuming a method also existed for an author to authenticate themselves with the search engine, one could also enable an author to help identify their content across multiple platforms, as well as suggest other quality authors to consider.
That may be true, but I think one of the good points made on the OP is that it might actually be cultural constraints that keep them from solving the problem:
https://twitter.com/paulg/status/1477761335412809729:
> You might need to do a lot of manual spam fighting initially. That could be both the thing-that-doesn't-scale, and the thing that differentiates you by being alien to Google's DNA. (They must hate manual interventions; so inelegant).
Google has some very smart and knowledgeable people, but the things they do have to fit into certain boxes, which means there are some problems they just can't fix, e.g.
* Everything has to be automated at scale, which leads to consistent poor user experience (unappealable account closures initiated by inscrutable algorithms, SEO spam).
* You get promoted by building new products, not maintaining existing ones, which leads to self-defeating churn outside of core areas (e.g. abandoning Google Talk and squandering their position in the messenger market).
* etc.
Of course you can always hardcode the site you want in the Google search results but this is hacky and not very expressive.
The fact that this feature does not exist shows that there is something deep within Google's core that is preventing them from addressing SEO spam, just like there is something deep within Airbnb that makes it difficult to filter out Airbnbs with problem reviews.
Google has been coasting for a good long time and now major players are realizing they are wide open for disruption.
Google is smart so I assume they crunched the numbers and figured out they make more money from people filtering through crappy results that include viewing and clicking ads than by surfacing good content.
I think Google is optimizing for ad revenue, not for good search.
On the other hand, Google needs to maintain the ballistic trajectory of its revenue growth. So how can they fix search quality when they've minmax'd themselves into this situation in the first place? If they were to make the ads background yellow again, that would have negative short term effects that I doubt any career exec can stomach.
The other moats are lock-in to advertising networks, website metrics, and effective control over Web standards through the Chrome browser.
An alternative search platform might provide better search. It would be fighting Google on at least three other fronts. It might have some success, but it would be challenging. (As history largely demonstrates.)
Even a rival tech monopolist, Microsoft, barely holds even with its own search offering (I use that indirectly via DDG), and scrapped its own web-browser development
If you have a good search engine, people will flock to it, and search ads will be valuable. That's it. That's how Google became Google.
The fact that Microsoft couldn't do it honestly doesn't mean much. Microsoft also couldn't do a phone OS, a portable music player, and many other things. They have a complex web of conflicting interests that $SEARCH_ENGINE_STARTUP does not.
It's just that "search" is really a web of interrelated services, capabilities, and revenue streams, and they tend to reinforce each other strongly. I'd like to see the monopoly disrupted.[1] But I don't think it's just a matter of "build a better mousetrap^Wsearch engine." Attack one corner, and Google will snipe at you from the others.
And with the AdWords cash cow, they've got an immense revenue stream.
________________________________
Notes:
1. Well, mostly. Google's acquired so goddamned much personal data that the premise is frankly kind of terrifying as well --- a weakened Google with neither the revenues nor talent to defend that pile.... And I'd really like to see the toppling occur without simply raising a new monopoly in its place.
Unfortunately, defaults matter and Google is spending billions of dollars yearly to make sure they are the default search engine wherever they can. Most people don’t switch from default.
>You might need to do a lot of manual spam fighting initially. That could be both the thing-that-doesn't-scale, and the thing that differentiates you by being alien to Google's DNA. (They must hate manual interventions; so inelegant).
Is he describing...Yahoo circa 1994? A manually curated directory service.
https://en.m.wikipedia.org/wiki/DMOZ
The gamification of the system then would have to come through onboarding fake users, pretending/mimicking real user behaviour to send that signal into the system; not sure if SU ever ran into that problem or was actively paying attention to trying to identify and removing fake or suspicious signals from their output?
I feel a much better system is easily within reach, it's simply getting the right structure to it, the right foundation, and then it will quickly take off due to the quality difference. I've already figured out a design pattern that Twitter and Facebook has indoctrinated us with, making us think it is normal - and keeping us blind to an actual normal way or organizing or communicating, but that isn't conducive to control or ad revenues - and so extending my future plans to include a better search-directory system would fit snugly into my efforts.
For health information and recipes in particular there are only a handful of really high quality sites that have quality content for 95% of the information most people need. I bet if you wanted to increase the coverage to 99%, that list would expand to less than a thousand sites. At those numbers manually curating the information would be easily achievable.
How to get people to use your top notch Google replacement instead of Google, however. That's the hard problem.
Siebel and PG see blood in the water no doubt, they see G's market share and want to fund companies to take some of this.
https://cse.google.com/cse?cx=dc408db269da4e769 (try searching for something you want a review of)
Make a search, whitelist the domains. Every time you run into a good review site, add it to the searchable list.
Over COVID, I did the whole fitness thing from a few different angles (overhauled diet, trained for a marathon, now lifting weights a lot). I found I could only find good info by going directly to a trusted source - literally, typing http://www. like I’m in the 90s or something. This is the exact issue a search engine should solve, but Google doesn’t.
If not, at least one of us is going to hate this new search engine.
What sorts of unique things did you do that Google failed at? Maybe you read through discussion sites or got tips from books or something like that
Importantly, this was bottom up: it was largely recommendation engines suggesting people I then filtered through for what I wanted (running, bodybuilding) vs didn’t (traditional weight loss). I couldn’t specify what I wanted, or it would be garbage SEO spam.
That helps a lot! Thank you.
I'll start: "how to rent a car" [0]
[0] worth noting that I personally get somewhat reasonable results for this, with a 3rd result from nerdwallet.com and a 4th from wikihow.com, both of which seem to answer the question in an unbiased way
They don't exist without a search engine.
I’m curious what would happen if those products were split up into 2 separate companies.
Doesn't Google already consider that if a user returns to the results page (or clicks a second link) then the first link visited was not satisfactory. Seems like a pretty elegant solution.
God please no. YouTube premium shows what Google would do, i.e., they would further ruin the free experience by ramping up the amount of ads you see to "incentivize" the premium search.
Is it viable competition?
I never considered the possibility that an incubator would support a specific product, then later on call for alternatives that would essentially freeze out the original product that they supported. I'm sure this very rarely happens, but it's interesting to see a real-world example in action.
I view it as something like the rich folks who call for additional taxation of the rich. They're not going to just pay extra money that they don't have to under the current tax rules, both because it's not particularly fair and because one person paying extra taxes, even if they're very wealthy, isn't going to make a big impact. That doesn't mean they can't lobby to change the rules and be totally fine with it if everyone is paying additional taxes.
Also, why hasn't Apple built a search engine yet? It baffles me that they chose to go head-to-head with Google on Maps, yet outsourced their search engine. I would've liked it the other way around: Google Maps and Apple Search.
I agree they should make their own search engine. But currently they’re being paid a ton of money not to.
But this only works for certain industries. It’s much less common to see this kind of tactic if your searching for say a coffee shop because they sheer number of local results let’s Google be “hyper local” with these kinds of results.
For licensed trades the solution is to go to your official trade licensing body (for the UK it's the NICEIC for electricians and the Gas Safe Register for gas/HVAC technicians), for unlicensed ones it's more difficult. There are "review" sites that claim to provide good results but their business incentives & vulnerability to spam/fake reviews are unknown.
Back in the olden days, there were lots of organizations that collated high quality content from the best writers. They nurtured expert writers and paid them well. They fact-checked the content and employed diligent editors and proofreaders so it was accurate and well-written. Over the years, they'd build a reputation for reliability and trustworthiness that kept people coming back for more. If you wanted to learn about fitness, or cars, or cooking, or science, you'd find a reputable author and publisher and buy their magazines or books.
But then, in the early 2000s, the geniuses from SV "disrupted" the publishing industry and its financial model. They brought us a much better way to find content, the search engine. Because they were so much better than the old-fashioned publishers, search engines gobbled up the advertising money and became the dominant gateway to content. Publishers had to abandon expensive high-quality writing because rankings and eyeballs now mattered more than quality and trustworthiness. Instead of investing in writers, they invested in marketers and SEO specialists.
The result: worthless content, writers banging out garbage for peanuts, and useless search engines.
Two decades later, looking at the barren wasteland they had created, the SV geniuses thought: I know what we need, more search engines, but smaller ones that collate high-quality content from the best writers. There must be money in that, right?
This implies that big budget advertisers (the CPGs, like Coke and P&G), are buying Google/FB because they have better conversion metrics. That isn't true today; only SMBs and gaming companies care about conversion metrics. There are interns in LA/NY probably collectively spending millions on FB for P&G and only reporting the number of likes back to their bosses. Google and FB has never meaningfully delivered on conversions past anything like app downloads.
Tech undermined publisher's revenue because the internet cratered distribution costs. Advertising revenues for big media crashed because the eyeballs moved away, not because it was any less efficient.
Did any of these heavily buy newspaper ads before 2000? Definitely TV, possibly magazine, but newspaper? I just don't remember seeing ads for Tide in newspapers.
I stand by my point that tech just exposed a bad business model. Newspapers were only viable because they were the ~only game in town.
Now what actually killed newspapers wasn't search, it was Craigslist killing the classifieds...at least according to a study Google paid for.
It went from "search engines and the web will usher in a new era of wisdom and democracy" to "useful content is dying at the hands of monetization schemes, and also the internet will be the death of liberal democracy, woe unto us all" in about 15 years.
So from an individual developer perspective, it's very hard to get people to change their habits. And Google/duck/Bing is the one stop shop.
It's still out there, but I haven't worked on it much lately. I always think that if I had some good advertisers, a better UI, and a salary coming in, maybe it could take over some of Google's usage!
(2) Do you have a bang on DuckDuckGo? I'm pretty aggressive with bangs, and I suspect a lot of DDG users end up being aggressive with them as well.
Link please
I like and use Duck Duck Go‘s !bangs [1] all the time, maybe try to add your site with a rememberable name.. may I suggest !garlic ?
[1] - https://duckduckgo.com/bang [2] - https://duckduckgo.com/newbang
One piece of feedback: It seems to somewhat liberally fuzz the term, doesn't tell me about it and not toggle it in the UI. I searched for "Natto" and got some great results at the top (Spicy Kimchi Natto!). However, a few results down the recipes start including "Nattu" which seems to be a Indian chicken dish and then quickly have nothing to do with the term at all and give me stuff like Tafelspitz.
Tweaking search can be a whole full time thing. I'll write it down and maybe look into if I can tweak the search so if there are exact matches to not show alternatives.
1 thing that would be helpful is to add in the ingredient amounts.
Never quite got around to it unfortunately, as I wanted more users before new features.
Of course, the massive volume of content creates a fundamental problem, but user curation & categorization on sites like Youtube would be possible, were Google to provide the software support so people could do that. Whether this and similar decisions are deliberate or accidental is likely one of those things that we will never know.
I've noticed this, and it's frustrating. I have assumed it's intentional. I am left to guess as to what a change in this behavior would accomplish.
The invariant has always been: find people who make falsifiable predictions and improve. Back then the pool was small and you had no choice. Now, fortunately we have a choice.
No, what happened is that the publishing industry lost their monopoly. They could no longer extract monopoly rents from advertisers.
One problem with search engines is affiliate marketing. If Google de-indexed the junk affiliate sites, the web would be much less polluted with affiliate spam.
Publishing was not a monopoly. Google/FB are a duopoly. If publishers capture rent from advertisers, they plow it back into content, aka the thing consumers actually want. If Google/FB capture rent, they don’t provide a living wage to content creators and plow the money into buying other startups and whatever “metaverse” is.
Popular HN post in 2022: “Search engines and SEO spam”.
Inevitable popular HN post 2025: “How to avoid getting flagged when content marketing“.
But with 3x memory needed for the indexes, the server costs probably aren't going to be bootstrap'able.
Especially for a "small" crawl of a billion web pages, event at just 10k per page.
What sort of memory and space do you have on the single server?
What's the average document size that you index?
Genuinely curious on how doable a modern search engine is on modern hardware.
The server has 128 Gb RAM and the index currently fits on a single 1 Tb SSD + an Optane drive of 480 Gb.
I find the average document to clock in at 7 Kb, in terms of raw HTML. In the index that's, dunno, probably less than 1 KB/doc.