Official docs for this endpoint: https://docs.github.com/en/rest/meta/meta?apiVersion=2022-11...
Creator of sqlite-vec here, happy to help debug anything if you do attempt to try out SQLite vector search again. You mentioned "VSS" in another comment, which if that was my sqlite-vss extension, I more-or-less…
I really dig more content about how vector databases/tools handle problems like this! In sqlite-vec, there's only a flat brute-force index (though DiskANN/IVF will be coming soon). But we do have a concept of partition…
vec0 virtual tables have a hard-coded max of 8192 dimensions, but I can raise that very easily (I wanted to reduce resource exhaustion attacks). But if you're comparing vectors manually, then the `vec_distance_ls()` and…
Really curious to hear about what kind of music embedding models/tools you used! I've tried finding some good models before but they were all pretty difficult to use
You can generate "text embeddings" (ie vectors) from your text with an embedding model. Once you have your text represented as vectors, then "closest vector" means "more semantically similar", as defined by the…
Author here, happy to answer any questions! Been working on this for a while, so I'm very happy to get this v0.1.0 "stable" release out. sqlite-vec works on MacOS, Linux, Windows, Raspberry Pis, in the browser with…
No, libsql added custom vector search directly into their library, while sqlite-vec is a separate SQLite extension. The libsql vector feature only works in libsql, sqlite-vec works in all SQLite versions. The libsql…
Because of SQLite. The SQLite official WASM build is a very complicated emscripten build, so if you want to write a SQLite extension for the WASM build, you need to statically compile it into the complex emscripten…
This isn't a complete extension, as mentioned in the README several times. I wanted a poc to use the new jiff library in SQL functions to test if it works. It did, so I put it in a repo for others to see. Definitely not…
I think so - I wrote my own Rust FFI bindings for SQLite extensions, and I tried to make it as fast as possible, but a "hello world" extension was still 10-15% slower in Rust than C[0]. That being said, it depends what…
Agreed, though the topic of timezone+DST support has come up a few times in the SQLite forums, and the SQLite team has never expressed much interest in it. Which I don't blame them, they would need to write it from…
Even though it's written in Rust, you can still use it in other languages like Python or Node.js. It compiles to a shared library file, which most SQLite clients support with `.loadExtension()` or another similarly…
Ya you'll be able to do things like: SELECT rowid, distance FROM vec_items WHERE rowid IN (SELECT rowid FROM ...) AND title_embeddings MATCH ? ORDER BY distance LIMIT 10 ^ This would do a KNN style query, k=10, but only…
Ya I'll put a license soon, will be MIT
Internally vectors (at least float vectors) get converted to float * arrays, where the vectors lengths are manually checked. If you provide a vector in JSON form, then it parses that JSON into a float * array. Vectors…
Ya I worded this part awkwardly - I was hinting that querying a vector index and joining with metadata with sqlite + sqlite-vec (in a single SQL join) will probably be faster than other methods, like txtai, which do the…
Yup, you'll be able to bind blobs, you'll currently have to do use vec_bit(?) which is a bit awkward, but it works!
re distance functions: current L2+cosine for float/int8 vectors, and hamming for bit vectors. There are explicit vec_distance_l2()/vec_distance_cosine()/vec_distance_hamming() SQL functions, and the vec0 table will…
Ya dynamically linking extensions in WASM would be amazing, but really hard to see how it would work. DuckDB was able to figure it out[0], but I struggle to see how it would work for SQLite. Though if anything, I think…
I plan to have v0.1.0 in about a month or so! Definitely will include a ton of docs and a quickstart guide. There's an undocumented pip package "sqlite-vec" that you might be able to use now, if you wanted to call it…
Went back, and forth a lot, but the short version: 1. Having directly access to SQLite's C APIs is really useful, especially for BLOB I/O and some rarely-used APIs that sqlite-vec uses and aren't available in some…
It will! I've neglected that project a bit, but in some of the alpha builds I was able to get WASM to work, I just have documented or blogged about it yet. SQLite extensions in WASM written with sqlite-loadable-rs are a…
Only public SQLite APIs! So no need to append to amalgamation. I'm a big fan of your wazero SQLite bindings! I actually plan on providing 1) CGO bindings to sqlite-vec and 2) a custom WASI build sqlite-vec that can be…
Benchmarking for this project is a bit weird, since 1) only linear scans are supported, and 2) it's an "embeddable" vector search tool, so it doesn't make a lot of sense to benchmark against "server" vector databases…
Official docs for this endpoint: https://docs.github.com/en/rest/meta/meta?apiVersion=2022-11...
Creator of sqlite-vec here, happy to help debug anything if you do attempt to try out SQLite vector search again. You mentioned "VSS" in another comment, which if that was my sqlite-vss extension, I more-or-less…
I really dig more content about how vector databases/tools handle problems like this! In sqlite-vec, there's only a flat brute-force index (though DiskANN/IVF will be coming soon). But we do have a concept of partition…
vec0 virtual tables have a hard-coded max of 8192 dimensions, but I can raise that very easily (I wanted to reduce resource exhaustion attacks). But if you're comparing vectors manually, then the `vec_distance_ls()` and…
Really curious to hear about what kind of music embedding models/tools you used! I've tried finding some good models before but they were all pretty difficult to use
You can generate "text embeddings" (ie vectors) from your text with an embedding model. Once you have your text represented as vectors, then "closest vector" means "more semantically similar", as defined by the…
Author here, happy to answer any questions! Been working on this for a while, so I'm very happy to get this v0.1.0 "stable" release out. sqlite-vec works on MacOS, Linux, Windows, Raspberry Pis, in the browser with…
No, libsql added custom vector search directly into their library, while sqlite-vec is a separate SQLite extension. The libsql vector feature only works in libsql, sqlite-vec works in all SQLite versions. The libsql…
Because of SQLite. The SQLite official WASM build is a very complicated emscripten build, so if you want to write a SQLite extension for the WASM build, you need to statically compile it into the complex emscripten…
This isn't a complete extension, as mentioned in the README several times. I wanted a poc to use the new jiff library in SQL functions to test if it works. It did, so I put it in a repo for others to see. Definitely not…
I think so - I wrote my own Rust FFI bindings for SQLite extensions, and I tried to make it as fast as possible, but a "hello world" extension was still 10-15% slower in Rust than C[0]. That being said, it depends what…
Agreed, though the topic of timezone+DST support has come up a few times in the SQLite forums, and the SQLite team has never expressed much interest in it. Which I don't blame them, they would need to write it from…
Even though it's written in Rust, you can still use it in other languages like Python or Node.js. It compiles to a shared library file, which most SQLite clients support with `.loadExtension()` or another similarly…
Ya you'll be able to do things like: SELECT rowid, distance FROM vec_items WHERE rowid IN (SELECT rowid FROM ...) AND title_embeddings MATCH ? ORDER BY distance LIMIT 10 ^ This would do a KNN style query, k=10, but only…
Ya I'll put a license soon, will be MIT
Internally vectors (at least float vectors) get converted to float * arrays, where the vectors lengths are manually checked. If you provide a vector in JSON form, then it parses that JSON into a float * array. Vectors…
Ya I worded this part awkwardly - I was hinting that querying a vector index and joining with metadata with sqlite + sqlite-vec (in a single SQL join) will probably be faster than other methods, like txtai, which do the…
Yup, you'll be able to bind blobs, you'll currently have to do use vec_bit(?) which is a bit awkward, but it works!
re distance functions: current L2+cosine for float/int8 vectors, and hamming for bit vectors. There are explicit vec_distance_l2()/vec_distance_cosine()/vec_distance_hamming() SQL functions, and the vec0 table will…
Ya dynamically linking extensions in WASM would be amazing, but really hard to see how it would work. DuckDB was able to figure it out[0], but I struggle to see how it would work for SQLite. Though if anything, I think…
I plan to have v0.1.0 in about a month or so! Definitely will include a ton of docs and a quickstart guide. There's an undocumented pip package "sqlite-vec" that you might be able to use now, if you wanted to call it…
Went back, and forth a lot, but the short version: 1. Having directly access to SQLite's C APIs is really useful, especially for BLOB I/O and some rarely-used APIs that sqlite-vec uses and aren't available in some…
It will! I've neglected that project a bit, but in some of the alpha builds I was able to get WASM to work, I just have documented or blogged about it yet. SQLite extensions in WASM written with sqlite-loadable-rs are a…
Only public SQLite APIs! So no need to append to amalgamation. I'm a big fan of your wazero SQLite bindings! I actually plan on providing 1) CGO bindings to sqlite-vec and 2) a custom WASI build sqlite-vec that can be…
Benchmarking for this project is a bit weird, since 1) only linear scans are supported, and 2) it's an "embeddable" vector search tool, so it doesn't make a lot of sense to benchmark against "server" vector databases…