Official docs for this endpoint: https://docs.github.com/en/rest/meta/meta?apiVersion=2022-11...
Creator of sqlite-vec here, happy to help debug anything if you do attempt to try out SQLite vector search again. You mentioned "VSS" in another comment, which if that was my sqlite-vss extension, I more-or-less…
I really dig more content about how vector databases/tools handle problems like this! In sqlite-vec, there's only a flat brute-force index (though DiskANN/IVF will be coming soon). But we do have a concept of partition…
vec0 virtual tables have a hard-coded max of 8192 dimensions, but I can raise that very easily (I wanted to reduce resource exhaustion attacks). But if you're comparing vectors manually, then the `vec_distance_ls()` and…
Really curious to hear about what kind of music embedding models/tools you used! I've tried finding some good models before but they were all pretty difficult to use
You can generate "text embeddings" (ie vectors) from your text with an embedding model. Once you have your text represented as vectors, then "closest vector" means "more semantically similar", as defined by the…
Author here, happy to answer any questions! Been working on this for a while, so I'm very happy to get this v0.1.0 "stable" release out. sqlite-vec works on MacOS, Linux, Windows, Raspberry Pis, in the browser with…
No, libsql added custom vector search directly into their library, while sqlite-vec is a separate SQLite extension. The libsql vector feature only works in libsql, sqlite-vec works in all SQLite versions. The libsql…
Because of SQLite. The SQLite official WASM build is a very complicated emscripten build, so if you want to write a SQLite extension for the WASM build, you need to statically compile it into the complex emscripten…
This isn't a complete extension, as mentioned in the README several times. I wanted a poc to use the new jiff library in SQL functions to test if it works. It did, so I put it in a repo for others to see. Definitely not…
I think so - I wrote my own Rust FFI bindings for SQLite extensions, and I tried to make it as fast as possible, but a "hello world" extension was still 10-15% slower in Rust than C[0]. That being said, it depends what…
Even though it's written in Rust, you can still use it in other languages like Python or Node.js. It compiles to a shared library file, which most SQLite clients support with `.loadExtension()` or another similarly…
Ya you'll be able to do things like: SELECT rowid, distance FROM vec_items WHERE rowid IN (SELECT rowid FROM ...) AND title_embeddings MATCH ? ORDER BY distance LIMIT 10 ^ This would do a KNN style query, k=10, but only…
Ya I'll put a license soon, will be MIT
Internally vectors (at least float vectors) get converted to float * arrays, where the vectors lengths are manually checked. If you provide a vector in JSON form, then it parses that JSON into a float * array. Vectors…
Ya I worded this part awkwardly - I was hinting that querying a vector index and joining with metadata with sqlite + sqlite-vec (in a single SQL join) will probably be faster than other methods, like txtai, which do the…
Yup, you'll be able to bind blobs, you'll currently have to do use vec_bit(?) which is a bit awkward, but it works!
re distance functions: current L2+cosine for float/int8 vectors, and hamming for bit vectors. There are explicit vec_distance_l2()/vec_distance_cosine()/vec_distance_hamming() SQL functions, and the vec0 table will…
Ya dynamically linking extensions in WASM would be amazing, but really hard to see how it would work. DuckDB was able to figure it out[0], but I struggle to see how it would work for SQLite. Though if anything, I think…
Went back, and forth a lot, but the short version: 1. Having directly access to SQLite's C APIs is really useful, especially for BLOB I/O and some rarely-used APIs that sqlite-vec uses and aren't available in some…
It will! I've neglected that project a bit, but in some of the alpha builds I was able to get WASM to work, I just have documented or blogged about it yet. SQLite extensions in WASM written with sqlite-loadable-rs are a…
Benchmarking for this project is a bit weird, since 1) only linear scans are supported, and 2) it's an "embeddable" vector search tool, so it doesn't make a lot of sense to benchmark against "server" vector databases…
Thanks for the kinds words! Really cool to hear that people were able to use sqlite-vss, lol.
Hey really cool to see re sqlite-vss+langchain! You could try a langchain integration now, there's an (undocumented) sqlite-vec pypi package you can install that's similar to the sqlite-vss one. Though I'd only try it…
Oooh I haven't thought of seperating the SQLite stuff and the vector search stuff. Good idea, will keep that in mind!
Official docs for this endpoint: https://docs.github.com/en/rest/meta/meta?apiVersion=2022-11...
Creator of sqlite-vec here, happy to help debug anything if you do attempt to try out SQLite vector search again. You mentioned "VSS" in another comment, which if that was my sqlite-vss extension, I more-or-less…
I really dig more content about how vector databases/tools handle problems like this! In sqlite-vec, there's only a flat brute-force index (though DiskANN/IVF will be coming soon). But we do have a concept of partition…
vec0 virtual tables have a hard-coded max of 8192 dimensions, but I can raise that very easily (I wanted to reduce resource exhaustion attacks). But if you're comparing vectors manually, then the `vec_distance_ls()` and…
Really curious to hear about what kind of music embedding models/tools you used! I've tried finding some good models before but they were all pretty difficult to use
You can generate "text embeddings" (ie vectors) from your text with an embedding model. Once you have your text represented as vectors, then "closest vector" means "more semantically similar", as defined by the…
Author here, happy to answer any questions! Been working on this for a while, so I'm very happy to get this v0.1.0 "stable" release out. sqlite-vec works on MacOS, Linux, Windows, Raspberry Pis, in the browser with…
No, libsql added custom vector search directly into their library, while sqlite-vec is a separate SQLite extension. The libsql vector feature only works in libsql, sqlite-vec works in all SQLite versions. The libsql…
Because of SQLite. The SQLite official WASM build is a very complicated emscripten build, so if you want to write a SQLite extension for the WASM build, you need to statically compile it into the complex emscripten…
This isn't a complete extension, as mentioned in the README several times. I wanted a poc to use the new jiff library in SQL functions to test if it works. It did, so I put it in a repo for others to see. Definitely not…
I think so - I wrote my own Rust FFI bindings for SQLite extensions, and I tried to make it as fast as possible, but a "hello world" extension was still 10-15% slower in Rust than C[0]. That being said, it depends what…
Even though it's written in Rust, you can still use it in other languages like Python or Node.js. It compiles to a shared library file, which most SQLite clients support with `.loadExtension()` or another similarly…
Ya you'll be able to do things like: SELECT rowid, distance FROM vec_items WHERE rowid IN (SELECT rowid FROM ...) AND title_embeddings MATCH ? ORDER BY distance LIMIT 10 ^ This would do a KNN style query, k=10, but only…
Ya I'll put a license soon, will be MIT
Internally vectors (at least float vectors) get converted to float * arrays, where the vectors lengths are manually checked. If you provide a vector in JSON form, then it parses that JSON into a float * array. Vectors…
Ya I worded this part awkwardly - I was hinting that querying a vector index and joining with metadata with sqlite + sqlite-vec (in a single SQL join) will probably be faster than other methods, like txtai, which do the…
Yup, you'll be able to bind blobs, you'll currently have to do use vec_bit(?) which is a bit awkward, but it works!
re distance functions: current L2+cosine for float/int8 vectors, and hamming for bit vectors. There are explicit vec_distance_l2()/vec_distance_cosine()/vec_distance_hamming() SQL functions, and the vec0 table will…
Ya dynamically linking extensions in WASM would be amazing, but really hard to see how it would work. DuckDB was able to figure it out[0], but I struggle to see how it would work for SQLite. Though if anything, I think…
Went back, and forth a lot, but the short version: 1. Having directly access to SQLite's C APIs is really useful, especially for BLOB I/O and some rarely-used APIs that sqlite-vec uses and aren't available in some…
It will! I've neglected that project a bit, but in some of the alpha builds I was able to get WASM to work, I just have documented or blogged about it yet. SQLite extensions in WASM written with sqlite-loadable-rs are a…
Benchmarking for this project is a bit weird, since 1) only linear scans are supported, and 2) it's an "embeddable" vector search tool, so it doesn't make a lot of sense to benchmark against "server" vector databases…
Thanks for the kinds words! Really cool to hear that people were able to use sqlite-vss, lol.
Hey really cool to see re sqlite-vss+langchain! You could try a langchain integration now, there's an (undocumented) sqlite-vec pypi package you can install that's similar to the sqlite-vss one. Though I'd only try it…
Oooh I haven't thought of seperating the SQLite stuff and the vector search stuff. Good idea, will keep that in mind!