HN Explorer

11 comments

[ 3.4 ms ] story [ 28.2 ms ] thread
leobg
> Entropy-weighted similarity: We adjust the similarity scores between query tokens and related documents based on the entropy of each token.

Sounds a lot like BM25 weighted word embeddings (e.g. fastText).

dmezzetti
> Sounds a lot like BM25 weighted word embeddings (e.g. fastText).

If you're interested in this topic, I wrote an article on this method back in 2020: https://medium.com/towards-data-science/building-a-sentence-...

nblgbg
Thanks for sharing the article. How exactly are you combining BM25 and fastText? Are you combining the TF-IDF score + embedding distance? What are the weights for each of these?
flawn
Gemischtes Brot!
deepsquirrelnet
Very cool! Glad to see continued research in this direction. I’ve really enjoyed reading the Mixedbread blog. If you’re interested in retrieval topics, they’re doing some cool stuff.
timsuchanek
Amazing! When will we have this in the major databases?
yokee
Super cool! It is definitely a good choice for the RAG system.
antman
How about computational complexity? There seems to be a small improvement in metrics but not sure if it is enough to switch to bmx
intalentive
Neat. I wonder how GPT-4’s query expansion might compare with SPLADE or similar masked BERT methods. Also if you really want to go nuts you can apply term expansion to the document corpus.