This was a relatively quick project for us and the index is currently in-memory. However it is fast! We would love your feedback and excited to invest further.
Looks like a cool project, but it would be great if you could add a license to the project since under US copyright lack of a license implies all rights are reserved and not open source. If you don't know much about software licenses https://choosealicense.com/ is a helpful site.
Yeb - we use pgvector in production[0]. Great results. There are some secret sauce stuff, but basically - depending on your source material and your user use-case you have to do more than just call cosine similarity and call it a day. (Most people don't even know what's happening, just using langchain which has different implementations, but all essentially call cosine similarity on raw text).
In my opinion, until LLMs learn to say I don't know, this is the way to do it in accuracy-first tasks.
Yes, I use retrieval for Endless Academy [1] , and it works well.
Some tips:
- Most vector search is basically kNN under the hood, with some kind of compression. If you have too many embeddings in your DB, this starts to pull up irrelevant text very quickly. The key is to segment the DB using other data before doing the embedding search. Postgres extensions are good for this.
- The quality of the data you put into your embedding DB matters a lot.
- How you chunk text matters. Chunking by paragraph is much better than naive chunking, for example.
- This is a good benchmark for embedding models [2]
> The key is to segment the DB using other data before doing the embedding search
By segmenting data, is this as simple as adding a further condition to the SQL? E.g. if your embeddings belonged to a "client" you would start with WHERE client_id - x?
> How you chunk text matters. Chunking by paragraph is much better than naive chunking
Would love some more info on this.
SO if you had a 400 word slab of text, it would be better to create m3 or 4 embeddings from that?
Yes. I work in the oil and gas industry. When our employees do their pre-job safety meeting, we have an app that makes tailored recommendations about job hazards. The steps are:
1. User types in the planned work (e.g., "replace discharge line on water pump")
2. The app embeds that and look for similar historical work in a vector database
3. We retrieve the most similar historical work and the safety observations that were made on those jobs
4. We summarize the major hazards or mistakes from those previous jobs using gpt-3.5-turbo and deliver that to the user
- Given how neon architecture decouples compute with storage using the safekeepers & pageservers will this still just work with neon? (Wondering because you mention the index is in-memory so unless your stateless compute nodes can somehow hot-swap the indexes I wasn't sure how that'd work?)
- Do you plan to offer vector search as a plug-and-play offering? If so does neon as a product offering plan to introduce more "out-of-the-box" functionalities like vector search? (Similar to xata's offerings like search & vector search.)
- unrelated question- I believe neon has very small cold startups for free/scale-to-zero configurations but are there also inherent small latencies for infrequently accessed data? In other words is it safe to say if there were a large table with records that are "old"/"archival" but also always a sort of ~last 30-days of records that are "fresh"/"accessed with more frequency" would there more likely be slight latency introduced when accessing the older records?
Neon looks awesome and thanks for neons open source contributions!
- This does work with Neon today, please go ahead and try. Today the index is in-memory and we are working on the on-disk implementation. Neon architecture does make it trickier - no local disk, so we are actively experimenting as we speak. Anything we ship will of course work on Neon.
- We will start with vector ANN and see the reception and user feedback. Now that we have 100K databases we are super attuned to what our users tell us.
- There is a cache/storage hierarchy: local cache on compute, cache on pageserver, and S3. We are working on cold start (stay tuned for more announcements this week). The goal is to make this unnoticeable for the app.
They are different, but that doesn't preclude pgvector supporting both indexes. I get that it's slower to take a collaborative approach - but that's part of open source.
> will require us to run a number of experiments with the storage format
I'd personally consider this a feature, if you want to maintain Postgres compatibility and allow your users to migrate in/out easily
A more important question is why supabase went with one of the slowest [0] implementations instead of building a better plugin especially since you are a VC funded company that's hardly lacking in cash. Simply gluing together a few random open source extensions is not great for developer experience.
that's a fair question. it's our policy to support existing open source tools/companies as much as we can[0]
Andrew is working hard on pgvector, and his goals are aligned to ours (and the industry's). We'll continue to support him as long as that's true.
> one of the slowest
Benchmarks are simply a snapshot in time - there's nothing fundamentally flawed with pgvector, it just needs more support and some better indexes (which are already being worked on)
>> [...] why supabase went with one of the slowest implementations
> Benchmarks are simply a snapshot in time - [...] it just needs [...] some better indexes
But the choice for IVF-flat indexes was made by the authors of pgvector, and that is one of the reasons why pgvector is slow: O(√n) optimal performance is just not great with large datasets, and this is especially true for the linked list approach chosen by the authors, as it results in large amounts of random IO.
You can't only implement a slow index and then shove the blame of the extension being slow on the index that was chosen by the extension authors: it is an inherent design issue. Sure, those can be fixed, but that doesn't mean it doesn't exist.
it would definitely be a design issue if the extension could only support a single index type, but Andrew already has plans to implement HNSW in 0.5.0:
If you say "no, pgvector is not slow, it is the index that is slow", while pgvector has been designed with only a single index type, then it is a design issue of pgvector in that the wrong type of index was chosen as the included index type.
That there are plans to solve this issue doesn't mean it is not an issue; indeed it validates my point that pgvector currently has design flaws.
Telling (potential) users to wait for 6 months for performance doesn't mean much if there are more performant alternatives readily available right now; a starving person isn't helped by a meal made available half a world away.
Didn't you (Neon) just announce pg_embedding and claim 20x performance over pg_vector even though pg_embedding just runs in memory and doesn't replicate the data?
Scary that database engineers do not understand how reckless that is.
What's the plan/timeline for offering cosine similarity support, given that most OSS embedding models are fine tuned on a contrastive cosine distance objective?
So it would be interesting to see benchmarks between pg_embedding and Qdrant. I would expect them to perform similarly, but perhaps there are other factors?
Very cool! It would be nice to see a working end to end integration with an LLM. Using this to generate relevant context for example. I see multiple folks mention cos similarity which HNSW doesn’t support - how does the lack of that limit what you can do with this library?
Also, since this is in memory, I assume this significantly affects startup time in order to rebuild the index? Would be nice to see how bad that is for larger vector datasets.
This blog post makes me uneasy. The pg_embedding code on github gives impression of PoC, while blog post creates impression that pg_embedding is ready for use.
If we consider pg_embedding ready for use, why don't you compare with pgvector:
1) The need to rebuild index every instance restart,
The blog post does mention memory limitations, but we could elaborate more on the fact that the index is entirely in-memory and clarify the user-visible consequences of this fact. We'll edit the post for better clarity.
With people talking about pgvectors current scaling issues, one thing I'm not sure about is whether it's if the Postgres DB table simply contains a lot of vectors - e.g. 500k vectors - or if you are searching over 500k vectors?
E.g. If the DB table had 500k vectors, but you were pre-filtering by WHERE client_id = X (returning only 200 rows) then with an AND <embedding search> (returning only 6 rows) - would this still have the same performance issue?
Or is it literally if the embedding search is over 500k rows?
55 comments
[ 3.5 ms ] story [ 128 ms ] threadThis was a relatively quick project for us and the index is currently in-memory. However it is fast! We would love your feedback and excited to invest further.
I am working quite a bit with normal and chained LLMs but so far haven't explored the retrieval route.
In my opinion, until LLMs learn to say I don't know, this is the way to do it in accuracy-first tasks.
0. https://galenai.co
Some tips:
[1] https://www.endless.academy[2] https://huggingface.co/blog/mteb
By segmenting data, is this as simple as adding a further condition to the SQL? E.g. if your embeddings belonged to a "client" you would start with WHERE client_id - x?
> How you chunk text matters. Chunking by paragraph is much better than naive chunking
Would love some more info on this. SO if you had a 400 word slab of text, it would be better to create m3 or 4 embeddings from that?
1. User types in the planned work (e.g., "replace discharge line on water pump")
2. The app embeds that and look for similar historical work in a vector database
3. We retrieve the most similar historical work and the safety observations that were made on those jobs
4. We summarize the major hazards or mistakes from those previous jobs using gpt-3.5-turbo and deliver that to the user
It doesn't yet have a LICENSE file, so it's not strictly OS quite yet, but that might have been an oversight.
- Given how neon architecture decouples compute with storage using the safekeepers & pageservers will this still just work with neon? (Wondering because you mention the index is in-memory so unless your stateless compute nodes can somehow hot-swap the indexes I wasn't sure how that'd work?)
- Do you plan to offer vector search as a plug-and-play offering? If so does neon as a product offering plan to introduce more "out-of-the-box" functionalities like vector search? (Similar to xata's offerings like search & vector search.)
- unrelated question- I believe neon has very small cold startups for free/scale-to-zero configurations but are there also inherent small latencies for infrequently accessed data? In other words is it safe to say if there were a large table with records that are "old"/"archival" but also always a sort of ~last 30-days of records that are "fresh"/"accessed with more frequency" would there more likely be slight latency introduced when accessing the older records?
Neon looks awesome and thanks for neons open source contributions!
- This does work with Neon today, please go ahead and try. Today the index is in-memory and we are working on the on-disk implementation. Neon architecture does make it trickier - no local disk, so we are actively experimenting as we speak. Anything we ship will of course work on Neon.
- We will start with vector ANN and see the reception and user feedback. Now that we have 100K databases we are super attuned to what our users tell us.
- There is a cache/storage hierarchy: local cache on compute, cache on pageserver, and S3. We are working on cold start (stay tuned for more announcements this week). The goal is to make this unnoticeable for the app.
Virtual screens of molecules frequently use steps to filter out molecules, one of which is via chemical similarity.
It would have been nice to get the support of neon in progressing pgvector - since it’s already so widely adopted by the community
(disclosure: supabase ceo)
Our on disk path will require us to run a number of experiments with the storage format. We will move faster with a separate project.
Medium to long term we are very open to merge them.
> will require us to run a number of experiments with the storage format
I'd personally consider this a feature, if you want to maintain Postgres compatibility and allow your users to migrate in/out easily
[0] https://github.com/erikbern/ann-benchmarks
Andrew is working hard on pgvector, and his goals are aligned to ours (and the industry's). We'll continue to support him as long as that's true.
> one of the slowest
Benchmarks are simply a snapshot in time - there's nothing fundamentally flawed with pgvector, it just needs more support and some better indexes (which are already being worked on)
[0] https://supabase.com/docs/guides/getting-started/architectur...
> Benchmarks are simply a snapshot in time - [...] it just needs [...] some better indexes
But the choice for IVF-flat indexes was made by the authors of pgvector, and that is one of the reasons why pgvector is slow: O(√n) optimal performance is just not great with large datasets, and this is especially true for the linked list approach chosen by the authors, as it results in large amounts of random IO.
You can't only implement a slow index and then shove the blame of the extension being slow on the index that was chosen by the extension authors: it is an inherent design issue. Sure, those can be fixed, but that doesn't mean it doesn't exist.
it would definitely be a design issue if the extension could only support a single index type, but Andrew already has plans to implement HNSW in 0.5.0:
https://github.com/pgvector/pgvector/issues/27
0.5.0 according to the GitHub issue tracker has been more than a year in the making and it is not any closer to release.
If you say "no, pgvector is not slow, it is the index that is slow", while pgvector has been designed with only a single index type, then it is a design issue of pgvector in that the wrong type of index was chosen as the included index type.
That there are plans to solve this issue doesn't mean it is not an issue; indeed it validates my point that pgvector currently has design flaws.
Telling (potential) users to wait for 6 months for performance doesn't mean much if there are more performant alternatives readily available right now; a starving person isn't helped by a meal made available half a world away.
Scary that database engineers do not understand how reckless that is.
"Qdrant currently only uses HNSW as a vector index."
- https://qdrant.tech/documentation/concepts/indexing
So it would be interesting to see benchmarks between pg_embedding and Qdrant. I would expect them to perform similarly, but perhaps there are other factors?
This is different from the dataset we used. We used GIST-960. It could interesting to test different vector store indeed.
Also, since this is in memory, I assume this significantly affects startup time in order to rebuild the index? Would be nice to see how bad that is for larger vector datasets.
If we consider pg_embedding ready for use, why don't you compare with pgvector:
1) The need to rebuild index every instance restart,
2) Replication support?
The blog post does mention memory limitations, but we could elaborate more on the fact that the index is entirely in-memory and clarify the user-visible consequences of this fact. We'll edit the post for better clarity.
E.g. If the DB table had 500k vectors, but you were pre-filtering by WHERE client_id = X (returning only 200 rows) then with an AND <embedding search> (returning only 6 rows) - would this still have the same performance issue?
Or is it literally if the embedding search is over 500k rows?