That article is very basic and fluffy, but the more technical article that it is based on and links to is excellent, so thanks for (indirectly) pointing to it:
The other stuff on that guy's (Han Xiao's) blog also looks interesting. I think this neural search stuff explains why we're all getting such low-precision results on Google/DDG by the standards we were used to. But, the claimed advantages are impressive. This is a big change in how search engines work, and I was unfamiliar with it before.
I still have reservations about where enough training data is supposed to come from, without already having a big system and user base. But, maybe some more info about that is out there. So this is another thing to look into. Wow.
Great to see another neural search (better known as vector search) option, especially with an end-to-end pipeline. There were 3 good discussions on vector search about a month ago:
I think 10 million documents is a large corpus. A retriever like Sklearn TfIdf will have a hard time handling it in a reasonable time. The main goal of Cherche is to prototype a neural search engine quickly and with a large choice of retrievers and rankers for corpus sizes < 1 million documents which is a common use case in the industry.
Search implements a wrapper of the Python ElasticSearch client that is scalable and dedicated to corpora composed of tens of millions of documents.
8 comments
[ 3.1 ms ] story [ 9.3 ms ] threadhttps://hanxiao.io/2018/01/10/Build-Cross-Lingual-End-to-End...
The other stuff on that guy's (Han Xiao's) blog also looks interesting. I think this neural search stuff explains why we're all getting such low-precision results on Google/DDG by the standards we were used to. But, the claimed advantages are impressive. This is a big change in how search engines work, and I was unfamiliar with it before.
I still have reservations about where enough training data is supposed to come from, without already having a big system and user base. But, maybe some more info about that is out there. So this is another thing to look into. Wow.
https://news.ycombinator.com/item?id=29551947 - About semantic search (ie, vector search applied to text)
https://news.ycombinator.com/item?id=29554986 - About vector search at Google
https://news.ycombinator.com/item?id=29555780 - Open-source vector search index from Facebook
Search implements a wrapper of the Python ElasticSearch client that is scalable and dedicated to corpora composed of tens of millions of documents.