Ask HN: Is 'search' a solved problem?

It depends what you are looking for. I have the hardest time searching for laptops that meet all of the specs I want, for instance.

Takes days.

Minimum screen brightness. 1440p touch. Needs USB-A. 8th gen quad core. 20+ watt TDP. etc. Not too thin, not too thick. decent graphics.

I think Google has pretty much monopolized search. You could say it's solved. I doubt if there are any complaints that sound like 'i couldn't find something using Google'

I remember reading about this exact topic somewhere but I can't seem to find the link.

Lucene/Solr, Elastic Search and Algolia did a great job creating search tools and services and this extinguished the thirst of the masses. I don't think it's a solved problem, it's just a problem that has commercially viable solutions. When it comes to resources, I've found valuable knowledge in Lucene/Solr forums and mailing lists, back in the day. It's worth a read.

I don't see 'search,' aka information retrieval (IR), as a solved problem. I went to the last SIG IR conference in Tokyo, and yes, the heavy hitters in the field were there to promote and present their latest research. It is no doubt a very active research field using machine learning techniques. Reading the papers published could give a view of the (academic) state of the art.

Whether it is a good business strategy to challenge Google head-on is another question. Whether there is a sure way to learn and engineer said systems on extremely large scales is another question.

To break into this space, I would recommend starting with a subset that people want to search. Like the facebook model of only being available for some colleges.

Some ideas:

- Only academic papers

- Only news sources

- Only hacker topics

- Only financial topics

- Only small bloggers

- Only literal keyword search which Google discontinued

Get traction in that domain, then build out from there.

Not really, no. Some parts of search (Internet) have been cornered/solved by big players (e.g. Google), but many other parts (e.g. search in Intranets) are still an open problem. No one has found a solution that's as simple as PageRank for Intranets. No one has found a solution for the "the author of half of these documents is the intern who wrote the template"-problem and many other things. There are good products out there, but a "Google for Non-Internet" is still far away.

p.s.: All the bad searches on the various products I use or the websites of some companies make me almost want to get back into search. Because even for things that ARE solved in the technical sense there seems to be no "out of the box" solution or people would use it.

edit: To expand a bit - all the search solutions I've seen which weren't for Internet search were more or less bespoke, so you needed a project to get something decent. Sure, you can install a plain Lucene/Solr, but Lucene/Solr cannot understand the way your data works, which parts are important or if you want to show results which are older further down (or not!). They have decent defaults for the "common case", but you have to tune them for every customer/installation for good results and that makes it non-scalable. And being scalable without effort is usually one of the requirements people have for something to be "solved".

I've learned that search is very domain-specific and it's only "solved" if you can pin down what "relevant" means for your corpus of documents.

In terms of fundamentals, I'd suggest reading about tf-idf, which is the basis of Lucene (which powers Solr and Elasticsearch).

The solution we need is search that doesn’t rely on an information monopoly like Google.

You might find this interesting https://dynamicguy.com/post/160792434677/1-search-engine-not...

If you just want a primer on how to think about adding search to a product, this piece by Max Grigorev is a great starting point: https://medium.com/startup-grind/what-every-software-enginee... It's ostensibly for the engineer, but actually feels more like it's written from the POV of a product manager.

Google's good for 1 - 3 key words / phrases but anything longer can still be difficult.

Regardless of query length it's strange that I often find the second result better than the first one.

Finding something in Gmail can be a real challenge, I wonder if would be possible to unleash Algolia on your inbox.

It's not that people have moved on. It's that the entire culture of the ecosystem is built upon a narrative that Google is an all-powerful machine that cannot be stopped or contested. So people don't try to compete, and if they do, they will be ridiculed for it.

And for what good reason? Certainly not because of past attempts. In fact, Google has bought some startups that were involved with search.

I'd be interested in knowing what you find. There are bound to be many relevant texts not labeled as search, if you know what you are looking for. Perhaps I would start with web crawlers and go from there.

I'd like a search for things that I've read or seen in the last few days. Or years.

https://twitter.com/patrickc/status/953011978217205760

(Disclaimer: I am an Apache Solr committer and popularizer)

Search is interesting! And it is important to differentiate the web search (Google) and domain-specific search (Solr, Elasticsearch, recent release of http://vespa.ai/). You cannot tune Google to your domain needs and understanding.

For domain-specific search, the basics are there. Even the fancy "basics". It is now very easy to add search to one's stack. In fact, Solr is in so many stacks, it is not even mentioned much anymore. But we still get the contributions back from Cloudera, Bloomberg, Alfresco, etc.

So, the cutting edge in Search is now on personalization, relevancy-tuning, indexing non-text content (music, images, etc), multi-word semantic search, graph traversal and, yes, Machine-Learning. See, for example, https://lucene.apache.org/solr/guide/7_3/learning-to-rank.ht...

In fact, the Solr conference that used to be called Lucene/Solr Revolution is now Activate and has focus on ML/AI because the topics are really starting to overlap (https://activate-conf.com/). You can see the interesting topics from last conference: https://www.youtube.com/playlist?list=PLU6n9Voqu_1FMt0C-tVNF...

Learning (Solr at least) is a different issue. There are so many features now that the Reference Guide is absolutely enormous. And the demo schemas are still a bit of a kitchen sync, making it look more complicated than it needs to be. And, the last comprehensive book was several versions back. Again, that's because Solr is big and is growing really fast still...

Actually that's why I chose to be a popularizer within the Solr community and focus on making it easier for beginners to start.

See, for example, my latest presentation slides at: https://www.slideshare.net/arafalov/rapid-solr-schema-develo... and the backing configuration repo: https://github.com/arafalov/solr-presentation-2018-may (includes smallest viable useful schema)

(tl;dr) Search is still exciting, lots of cutting edge cool stuff, and there are people trying to make it easy for beginners to start.

Great topic. Part of the issue is probably a transition from algorithmic approaches to data-driven approaches. What did previous users search for and click on? Existing companies have a huge advantage from years of data, and not the kind of advantage that others can learn from (compare to publishing a better algorithm). Another factor may be that parts of the problem can separated out and are studied on their own, such as natural language processing.

"Search" is too broad to ever be solved. That's like "solving entropy".

Google focused on a specific subset — you enter a few keywords or a phrase, and the machine returns the top ~10 links to pre-existing (indexed) web pages. But that's not all there is to search!

Challenges:

1. Intranets: internal documents, typically in different modalities (FAQs, support cases, wikis, public pages) and across diverse storages that evolved throughout the years via acquisitions and osmosis.

2. Clustering: you don't have any keywords, but rather want to find how a particular document (legal template, its clause section) evolved over time. You want to avoid using keywords. Search for similar documents or document sections. Find similarity between two documents that is based on semantics rather than query keywords. Applications: eDiscovery, contract management…

3. SME & Intent: "relevant result" means different things in different domains, or even different aspects of a single domain. Google is doing an amazing job with their "single search box", but there are industries (for example, HR) where search precision matters much more than recall. More elaborate, focused, domain-specific facets or even dialogue systems make sense there.

Commercial plug: we built a search solution focused around semantic search (in the "machine learning and vectors" sense, not "sematic web and RDFs" sense), https://scaletext.ai. It's still early days in that our clients are all over the place, but to say Google/Lucene solved search is patently false.

The search problem is connected to the spam filtering problem which is an ever advancing arms race - it's never solved and depends on the new schemes spammer come up with. So search itself is never a solved problem.

I've confronted my PhD supervisor (Professor of Library and Information Science) with this statement once, and she almost went berserk. Her take is that free text search is approaching the solved problem stage, but almost all other search isn't.

We are still in the stone age when it comes to search.

Ask Google who played the mens semifinals of Wimbeldon three years ago and Google will tell you it indexed 6 million pages to provide a link that may or may not have the 4 names I am looking for. Why is it doing all this pointless work? And why is it that dumb in 2018?

We have got so used to what it does that lot of people have stopped asking questions about how it does things and wether all the stuff it does is required.

Wolframalpha, Freebase/SemanticWeb/Wikidata/dbpedia approaches, NLP/NLU are still very underdeveloped and untapped.

Having open and distributed indexes like we see in nature with DNA is also totally unexplored because of Google type centralised index monopolies in various domains. It just takes a Gig or so to store a local offline index off all Wikipedia or Stackoverflow pages. And given the massive RAM and hard disks everyone has these days why aren't we seeing sophisticated local offline search apps?

The internet is getting exponentially more noisy day by day and in many ways its easier to find quality info going through a top notch library's index than wading through Google's. So there are lots of blindspots and areas to explore in search right now imho.

I tend to think the search is often the last resort and indicative of the other navigation system being broken. When people can reach the info they need in a more organized way quickly, they'd probably do so. Therefore full text search has to cover every residual task; it's bound to be messy.

If anything, it's an abandoned problem. A lot of companies bought really expensive enterprise search systems, which are sitting dormant because the results are so bad.

With advances in spamming, internet/email search is getting to be a harder problem every year.

I remember when Google was quite effective in finding what I need, but nowadays it's dismal. As an example, I googled for "storename return policy", and got page after page of results. All of them were tagged "missing storename", so just randomly picked return policies from other stores.

Search is their bread and butter, and they're probably keenly aware of the diminishing quality of results. I'd love to hear what's caused the recent trend for results that are missing a few of the most crucial keywords. Probably over-enthusiastically trying to filter out keyword farms.

My personal view is that there should be dedicated search providers for specific areas, such as academic, or technical, or news.

That way each provider can focus on building a good platform with clever machine learning tailored to that dataset.

We also need the return of proper Boolean operators and complex nested queries. Yes, Joe Public will never use them, but a lot of people who search the internet as part of their jobs, or just have deep interests and would love to have all those advanced search features back, and be able over to override the 'fuzzy logic' that generic Search engines such as Google enforce on us.

I also disagree that all sites need to be mobile first. If I have a site that provides software for the Enterprise Sector, Google will still penalise me if my site isn't responsive, even though my target market is IT professional sitting in front of powerful laptops/desktops.

As many would agree, Google have too much power in the Search space, and they have basically dictated to the world how Search should be done, whether they are actually right or not.

I don't think it is a solve problem. It just seems daunting or difficult to take on Billion dollar companies in the space.

The default search engine in people's web browsers is a solved problem. Search itself, not so much.