This works by streaming results from the similarity search to the filter, in descending order of similarity. As soon as enough matches are found, the similarity search is terminated.
Postgres does the same thing for many queries if you look at the query plan.
Not really worthy of a blog post - especially one that says "wait till the next blog post to find out how it works!!".
How is what you describe _not_ just an efficiently implemented post filter with early out?
If it turns out that’s all pinecone have, then yeah, I’m gonna be disappointed. My mind is working overtime imagining prefixing vectors with their filter terms to root everything and other naive things…
Very nice writeup. I have an idea how this might be implemented. Since you rightfully want to keep the details private, let me just ask:
How do you handle the case where the probed clusters contain no elements satisfying the filter criteria? Is the probe extended to additional clusters, or terminated with no results?
I realize you are hinting at it as the subject of a later article, but could you share with us a little bit more about single-stage-filtering :D ? how does it work? How can metadata and vector data "coexist" in a same index?
I'm from Pinecone and can tell you we are figuring out now exactly how much to say about this.
We know you want to know, and we also know other vector databases want to know -- because this is a highly requested feature and it's far from trivial to implement in a low-latency way. That's why we have to give the admittedly non-satisfying answer of "stay tuned!"
There are different approaches for turning vectors into terms which you can then simply index with something like Elasticsearch. There are papers by the Walmart and Jet.com teams that describe different approaches, starting with both binarised vectors as well as full float vectors. Not saying that this is what Pinecone is using, so it will be interesting to read their follow-up.
Yes as gk1 said, we're figuring out what can be said! I can't fully answer your questions beyond, having both indexes together helps us speed up the search (thanks to a lack of network/serialization/parsing overhead), and because of the structure we store metadata tags in, we also don't need to check the metadata for every sample (hence the cool speedup you get when filtering) - sorry that we can't say anything more yet!
I'm not disputing that the image may not add anything to the article but if a 10MiB GIF image can crash your browser maybe you should look at other vendors. I'm on Firefox Developer Edition with approximately 100 tabs open and didn't notice that image as a resource hog.
Amazon's Opensearch (fork of Elasticsearch) natively supports vector-based approximate KNN (using https://github.com/nmslib/nmslib/) which is integrated with Opensearch's native filtering functionality. Elasticsearch also has similar functionality, but I don't know if their KNN code scales quite as well.
Opensearch only supports "pre-filtering" or "post-filtering," which leads to either high latency or incomplete results, as explained in the article.
This is why single-stage filtering was the most-requested feature for us.
From the Opensearch docs:
> You should not use approximate k-NN if you want to apply a filter on the index before the k-NN search, which greatly reduces the number of vectors to be searched.
> Because the graphs are constructed during indexing, it is not possible to apply a filter on an index and then use this search method. All filters are applied on the results produced by the approximate nearest neighbor search.
> If you use the knn query alongside filters or other clauses (e.g. bool, must, match), you might receive fewer than k results.
The approximate kNN is quite nice for many use cases, and scales to billions of documents. However, you're correct that filtering happens on the results. This is only an issue in certain use cases where filtering is very narrow, as you can often just request much higher k than the number of results you really need without much slowdown.
If the filtering is very narrow, as you commented they also provide functionality to perform pre-filtering and then exact kNN on the results. This is of course higher latency, but still quite acceptable for many use cases (this is how I use it).
I believe there are use cases that Pinecone addresses better than Opensearch, but I want to let people know that there is a free, open-source solution which _may_ also work for their use case.
Elasticsearch does currently support vector search through script score using dense vector fields, however I suspect they are still working on improving it and I prefer the Opensearch implementation for the time being https://www.elastic.co/guide/en/elasticsearch/reference/curr...
25 comments
[ 4.7 ms ] story [ 75.3 ms ] threadPostgres does the same thing for many queries if you look at the query plan.
Not really worthy of a blog post - especially one that says "wait till the next blog post to find out how it works!!".
If it turns out that’s all pinecone have, then yeah, I’m gonna be disappointed. My mind is working overtime imagining prefixing vectors with their filter terms to root everything and other naive things…
This is not a trivial problem when dealing with vector similarity search, which Postgres does not have.
How do you handle the case where the probed clusters contain no elements satisfying the filter criteria? Is the probe extended to additional clusters, or terminated with no results?
https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot....
I realize you are hinting at it as the subject of a later article, but could you share with us a little bit more about single-stage-filtering :D ? how does it work? How can metadata and vector data "coexist" in a same index?
We know you want to know, and we also know other vector databases want to know -- because this is a highly requested feature and it's far from trivial to implement in a low-latency way. That's why we have to give the admittedly non-satisfying answer of "stay tuned!"
Did animation actually add anything significant anyway.
Please open firefox to that page, leave it open for a few minutes and observe the memory. I suspect FX will bloat too.
This is why single-stage filtering was the most-requested feature for us.
From the Opensearch docs:
> You should not use approximate k-NN if you want to apply a filter on the index before the k-NN search, which greatly reduces the number of vectors to be searched.
> Because the graphs are constructed during indexing, it is not possible to apply a filter on an index and then use this search method. All filters are applied on the results produced by the approximate nearest neighbor search.
> If you use the knn query alongside filters or other clauses (e.g. bool, must, match), you might receive fewer than k results.
(https://opensearch.org/docs/search-plugins/knn/approximate-k...)
I know Elasticsearch is working on introducing vector search but it is not yet available. I don't know how they will support filtering.
If the filtering is very narrow, as you commented they also provide functionality to perform pre-filtering and then exact kNN on the results. This is of course higher latency, but still quite acceptable for many use cases (this is how I use it).
I believe there are use cases that Pinecone addresses better than Opensearch, but I want to let people know that there is a free, open-source solution which _may_ also work for their use case.
Elasticsearch does currently support vector search through script score using dense vector fields, however I suspect they are still working on improving it and I prefer the Opensearch implementation for the time being https://www.elastic.co/guide/en/elasticsearch/reference/curr...