Show HN: CandleEmbed – A crate for generating text embeddings from HF models (github.com)

2 points by J_Shelby_J 2y ago ↗ HN

I wanted to try out the new snowflake embedding model a few weeks ago in a personal project, and I couldn't find an easy "cargo add" way to do so.

By the time I got done implementing Candle's text embedding example for my needs, I realized it wouldn't take much more work to make it a crate.

Running on a 3090 and it's extremely fast. It generates the ~5k embeddings in MTEB's SCIFACT benchmark in 2-3m. CPU is very slow. I know this is not particularly impressive as Rust project, but this is my first crate and I enjoyed making something "user friendly" - even if I end up being the only user!

2 comments

[ 3.1 ms ] story [ 12.4 ms ] thread

sbt567 2y ago ↗

Neat! When you say that "CPU is very slow", do you mean that using the CPU is (and always will be) slower than using the GPU, or that you could still optimize it further?

J_Shelby_J 2y ago ↗

The current implementation is very slow with a CPU. Probably, it could be optimized with parallelism for batches. I'm not sure how it could be improved for single embeddings.