Ask HN: A search engine that finds things and doesn't use GPT et al.?
Looking for a recommendation. Google doesn't find things I search for any more. Bing has gone with making things up via AI instead of finding websites. I'm not very interested in a machine guessing answers to things I'm asking.
Is there anyone else left standing? I assume yahoo et al died years ago.
Usually looking for programming related information or research papers.
5 comments
[ 3.9 ms ] story [ 21.5 ms ] threadIt is so often useful to use Google to search Reddit (or Twitter, SO, etc.) to find a discussion that provides useful context.
https://www.sbert.net/
It uses large language models trained to: (1) produce a document embedding vector from a document that captures the “gist” of the document, (2) given a query and document, predict if the document is relevant to the query.
I was working on a search engine that used a dense document embedding about a decade ago which was highly successful, but at the time that kind of embedding helped with recall instead of precision since it could discover alternative meanings of words but not differentiate between different meanings of words. Transformer-based embeddings can do just that and are a revolution in IR. (Some significant develoment in that field happens about once a decade, so it is a big deal.)
It helps that that sbert.net is super-reliable and easy to use: I’ve gotten demos up and running in half an hour whereas doing something with raw huggingface transformers usually means a week tuning hyperparameters.