To your point, the market for vector db solutions feels very undifferentiated. I am genuinely curious -- what are the types of ANN use-cases that truly require XXms lookup latency, XXX QPS, and capacity for billions of documents?
When I prototype RAG systems I don’t use a “vector database.” I just use a pandas dataframe and I do an apply() with a cosine distance function that is one line of code. I’ve done it with up to 1k rows and it still takes less than a second.
Also, you are probably doing it wrong by turning a matrix to matrix multiplication into a for loop (over rows). The optimal solution results in better performance
Everyone is piling on you but Id love to see what their companies are doing. Cosine similarity and loading a few thousand rows sounds trivial but most of the enterprise/b2b chat/copilot apps have a relatively small amount of data whose embeddings can fit in RAM. Combine that with natural sharding by customer ID and it turns out vector DBs are much more niche than an RDBMS. I suspect most people reaching for them haven’t done the calculus :/
1k is not much. My first RAG had over 40K docs (all short, but still...)
The one I'm working on right now has 115K docs (some quite big - I'll likely have to prune the largest 10% just to fit in my RAM).
These are all "small" - for personal use on my local machine. I'm currently RAM limited, otherwise I can think of (personal) use cases that are an order of magnitude larger.
Of course, for all I know, your method may still be as fast on those as on a vector DB.
I must be missing something -- why is the size of the documents a factor? If you embeded a document it would become a vector of ~1k floats, and 115k*1k floats is a couple hundred MB, trivial to fit in modern day RAM.
Embeddings are a type of lossy compression, so roughly speaking, using more embedding bytes for a document preserves more information about what it contains. Typically documents are broken down into chunks, then the embedding for each chunk is stored, so longer documents are represented by more embeddings.
Always felt they're more like hashes/fingerprints for the RAG use cases.
> Typically documents are broken down into chunks
That's what I would have guessed. It's still surprising that the embeddings don't fit into RAM though.
That said (the following I just realized), even if the embeddings don't fit into RAM at the same time, you really don't need to load them all into RAM if you're just performing a linear scan and doing cosine similarity on each of them. Sure it may be slow to load tens of GB of embedding info... but at this rate I'd be wondering what kind of textual data one could feasibly have that goes into the terrabyte range. (Also, generating that many embedding requires a lot of compute!)
> Always felt they're more like hashes/fingerprints for the RAG use cases.
Yes, I see where you’re coming from. Perceptual hashes[0] are pretty similar, the key is that similar documents should have similar embeddings (unlike cryptographic hashes, where a single bit flip should produce a completely different hash).
Nice embeddings encode information spatially, a classic example of embedding arithmetic is: king - man + woman = queen[1]. “Concept Sliders” is a cool application of this to image generation [2].
Personally I’ve not had _too_ much trouble with running out of RAM due to embeddings themselves, but I did spend a fair amount of time last week profiling memory usage to make sure I didn’t run out in prod, so it is on my mind!
Each vector is 1536 numbers. I don't know how many bits per number, but I'll assume 64 bits (8 bytes). So total size is 1536 * 115K * 8 / 1024^2 gives 1.3GB.
So yes, not a lot.
I still haven't set it up so I don't know how much space it really will take, but my 40K doc one took 2-3 GB of RAM. It's not pandas DF, but in an in-memory DB so perhaps there's a lot of overhead per row? I haven't debugged.
To be clear, I'm totally fine with your approach if it works. I have very limited time so I was using txtai instead of rolling my own - it's nice to get a RAG up and running in just a few lines of code. But for sure, if the overhead of txtai is really that significant, I'll need to switch to pure pandas.
That's kinda why I use LanceDB. It works on all three OSes, doesn't require large installs, and is quite easy to use. The files are also just Parquet, so no need to deall with SQL.
You may benefit from polars, it can multi-core better than pandas, and has some of the niceties from Arrow (which was the written / championed by the power duo of Wes and Hadley, authors of pandas and the R - tidyverse respectively).
1k rows isn't really at a point where you need any form of database. Vector or BOW, you can just bruteforce the search with such a miniscule amount of data (arguably this should be true into the low millions).
The problem is what happens when you have an additional 6 orders of magnitude of data, and the data itself is significantly larger than the system RAM, which is a very realistic case in a search engine.
This is exactly what I do. No one talks about how many GPUs you need to generate enough embeddings that you need to do something else.
Here's some back of the envelope math. Let's say you are using a 1B parameter LLM to generate the embedding. That's 2B FLOPs per token. Let's assume a modest chunk size, 2K tokens. That's 4 trillion FLOPs for one embedding.
What about the dot product in the cosine similarity? Let's assume an embedding dim of 384. That's 2 * 384 = 768.
So 4 trillion ops for the embedding vs 768 for the cosine similarity. That's a factor of about 1 billion.
So you could have a billion embeddings - brute forced - before the lookup became more expensive than generating the embedding.
What does that mean at the application level? It means that the time needed to generate millions of embeddings is measured in GPU weeks.
The time needed to lookup an embedding using an approximate nearest neighbors algorithm from millions of embeddings is measured in milliseconds.
The game changed when we switched from word2vec to LLMs to generate embeddings.
1 billion times is such a big difference that it breaks the assumptions earlier systems were designed under.
The embedding is generated once. Search is done whenever a user inputs a query. The cosine similarity is also not done on a single embedding, it's done on millions or billions of embeddings if you are not using an index. So what the actual conclusion is, is that once you have a billion embeddings a single search operation costs as much as generating an embedding.
But then, you are not even taking into account the massive cost of keeping all of these embeddings in memory ready to be searched.
Prototyping is one scenario I have seen this in. Prototyping is iterative - you experiment with the chunk size, chunk content, data sources, data pipeline, etc. every change means regenerating the embeddings
Another one is where the data is sliced based on a key, eg user id, particular document being worked on right now, etc
I'm expecting to deploy a 6-figure "row count" RAG in the near future... with CTranslate2, matmul-based, at most lightly (like, single digits?) batched, and probably defaulting to CPU because the encoder-decoder part of the RAG process is just way more expensive and the database memory hog along with relatively poor TopK performance isn't worth the GPU.
There is certainly some scale at which a more sophisticated approach is needed. But your method (maybe with something faster than python/pandas) should be the go-to for demonstration and kept until it's determined that the brute force search is the bottleneck.
This issue is prevalent throughout infrastructure projects. Someone decides they need a RAG system and then the team says "let's find a vector db provider!" before they've proven value or understood how much data they have or anything. So they waste a bunch of time and money before they even know if the project is likely to work.
It's just like the old model of setting up a hadoop cluster as a first step to do "big data analytics" on what turns out to be 5GB of data that you could fit in a dataframe or process with awk https://adamdrake.com/command-line-tools-can-be-235x-faster-... (edit: actually currently on the HN front page)
It's a perfedt storm of sales led tooling where leadership is sold something they don't understand, over-engineering, and trying to apply waterfall project management to "AI" projects that have lots of uncertainty and need a re-risking based project approach where you show that it's liable to work and iterate instead of building a big foundation first.
I agree pandas or whatever data frame library you like is ideal for prototyping and exploring than setting up a bunch of infrastructure in a dev environment. Especially if you have labels and are evaluating against a ground truth.
You might be interested in SearchArray which emulates the classic search index side of things in a pandas dataframe column
Thanks for the article and definitely agree you are better off to start it simple like a parquet file and faiss and then test out options with your data. I say that mainly to test chunking strategies because of how big an effect it has on everything downstream whatever vector db or bert path you take -- chunking is a much bigger impact source than most people acknowledge.
Even on the production side there is something to be said about just doing things in memory, even over larger datasets. Certainly like all things there is a possible scale issue but I would much rather spin up a dedicated machine with a lot of memory than pay some of the wildly high fees for a Vector DB.
Not sure if others have gone down this path but I have been testing out ways to store vectors to disk in files for later retrieval and then doing everything in memory. For me the tradeoff of a sligtly slower response time was worth it compared to the 4-5 figure bill I would be getting from a vector DB otherwise.
Even up to 1M or so rows you can just store everything in a numpy array or PyTorch tensor and compute similarity directly between your query embedding and the entire database. Will be much faster than the apply() and still feasible to run on a laptop.
As someone who has been using pgvector for a while and is vaguely curious about alternatives without having the bandwidth to investigate -- is there anything out there that offers truly differentiated advantages over pgvector? I'm extremely wary of non-OSS solutions in this area, it seems ripe for enshittification and attempts at vendor lock-in.
I'd also start with pgvector (it's easy to switch), but the limitations around hybrid search and filtering + ANN are real and if you're doing any kind of RAG-like thing it's worth being aware of them upfront. pgvector is also an open-source project with way less manpower behind it than a bunch of venture-backed companies, so while you can expect it to pick up important features, it takes much longer (support for HNSW indices was a good example).
If you're still in the "millions of documents" scale range, then PostgreSQL on a beefy EPYC can probably handle everything fast enough so that it doesn't make sense to spend engineering time on using a vector db which would only shave off a few ms in latency.
Context: I'm working on an e2ee alternative to Google Photos[1] where we have to cluster embeddings (for face recognition) and run similarity searches (for semantic search[2]) on device.
>Real-time recommendations, but driven by vector (and other kinds of) retrieval that looks more like a search engine - not batch computed, nightly jobs common these days.
This is already the case. Recommendations are just a fancy search where the query is a vector representing the user. Whether the learning is batched or not doesn't change the fact that it will use vector search for at least candidate generation.
There is still plenty of room for innovation in this space. Just need to focus on the right projects that are innovating and not the ones (re)working on problems solved in 2020/2021.
I agree. Honestly even the fundamentals of vector databases aren't really "solved" in the way they are for other databases. Vector indexing, embedding generation, horizontal scaling, etc. can probably still improve a lot. And don't forget, even if Postgres and MySQL are the only traditional databases in town, every tech company had their own SQL database once. Many of them are still around too. No need to get pissy about these companies.
As others have said in this thread, cosine similarity on arrays of vectors isn't novel. But there are many possibilities past that, many we haven't thought of yet too.
Most of that is recycled. They don’t break it down because it would make the obvious, obvious. Microsoft pays OpenAI but requires them to use Azure, and OpenAI pays Microsoft the same money back. This is why they continually need billions in investment, because they are far from profitable.
The same principle applies to defense. The US gives Israel and Ukraine tens of billions, but that’s a credit to buy from US defense firms. That money gets recycled right back to US weaponry.
Your logic seems highly flawed. I get what you are saying in the example, yes the government provides weapons which are paid for by the government but produced by defense companies.
But in the Microsoft example it is customers who are paying Microsoft to use OpenAI via Azure. Thats a free market of money inflows. Same with all the people using OpenAI directly. Not sure how you would even think of the money being recycled in this scenario. Yes of course there is some back scratching in the sense that Microsoft invested in OpenAI with a large portion of that investment in Azure credits which makes the investment quite nice from MSFT's side but there is still real demand for Azure services to use OpenAI apis.
For big enterprises it is either included in existing licenses (i.e. Microsoft Word has ChatGPT embedded) or pilots. No one is cutting massive checks to Microsoft specifically for OpenAI services.
This is obvious, but if you need some journalist to validate what is already logically clear:
I can see how its easy to get confused in this area but there are indeed large checks gettign written for using services like OpenAI or Anthropic.
You are really conflating too many things at once.
1) Yes, big tech is having a hard time monetizing their bespoke AI tooling within their own ecosystem.
2) Yes, big tech has made investments in the AI space where they are providing a portion of that funding as credits to use in their cloud offerings.
3) Here is where you are incorrect though. Companies are writing large checks for the raw compute/access to AI models. It is true across the spectrum of Azure OpenAI, OpenAI directly, AWS Bedrock etc, there are a lot of companies both big and small using these services heavily. To think otherwise is naive.
The investment is massive, but real tangible products that have purchasers for sustainable contracts is miniscule. We are in the experimental phase and hype cycle, and the trough of despair is next. I do think real products will come from this, but the actual productivity enhancements at the scale necessary to justify the investment have not materialized.
Wow. Back in the day, I had to do cosine similarity indexing with pg-cube. It only did euclidean distance, so I had to store a separate column with normalized vectors.
Exactly. Some other context implied here, these vectors were ML embeddings. In that case, like 100 dimensions that vaguely represented our input data in compact form. There was probably a better solution out there, this was just the most readily available for us.
Cosine similarity suffers from a curse of dimensionality, just as distance does (it’s just one dimension less). The angle between two random vectors in N dimensions approaches zero with a power of N. The main reason this metric is useful in practice is because it better relates to how certain neural networks use/train their embeddings internally.
Investors don't really care if it actually creates more value. They only care if the story can attract the public. They just want to profit by taking next investors' money.
IMO we are well past peak cosine-similarity-search as a service. Most people I talk to in the space don't bother using specialized vector DBs for that.
I think there's space for a much more interesting product that is longer-lived (since it's harder to implement than just cosine-similarity-search on vectors), which is:
1. Fine-tuning OSS embedding models on your real-world query patterns
2. Storing and recomputing embeddings for your data as you update the fine-tuned models.
MTEB averages are fine, but hardly anyone uses the average result: most use cases are specialized (i.e. classification vs clustering vs retrieval). The best models try to be decent at all of those, but I'd bet that finetuning on a specific use case would beat a general-purpose model, especially on your own dataset (your retrieval is probably meaningfully different than someone else's: code retrieval vs document Q&A, for example). And your queries are usually specialized! People using embeddings for RAG are generally not also trying to use the same embeddings for clustering or classification; and the reverse is true too (your recommendation system is likely different than your search system).
And if you're fine-tuning new models regularly, you need storage + management, since you'll need to recompute the embeddings every time you deploy a new model.
I would pay for a service that made (1) and (2) easy.
We (Marqo) are doing a lot on 1 and 2. There is a huge amount to be done on the ML side of vector search and we are investing heavily in it. I think it has not quite sunk in that vector search systems are ML systems and everything that comes with that. I would love to chat about 1 and 2 so feel free to email me (email is in my profile).
I've been working on (3) embeddings translation with the goal being to translate something like OpenAI embeddings to UAE-Large. So far, I have had success using them for cosine similarity with around a 99.99% validation rate, but only 80% using Euclidean distance.
I’m fascinated by embeddings translations and compatible embeddings with different numbers of dimensions. Can you share more about your work / findings?
I mean, the simplest answer is a matmul... Given embedding x, y, find M such that Mx ~= y. Easy to train so long as you've got access to both models to compute embedding over whatever you're interested in...
(easy to extend to two layers mlp as needed. maybe ensure that x and y are zero mean and unit length to make training the matmul a bit easier.)
I no longer work there, but Lucidworks has had embedding training as a first-class feature in Fusion since January 2020 (I know because I wrapped up adding it just as COVID became a thing). We definitely saw that even with just slightly out-of-band use of language - e.g. in e-commerce, things like "RD TSHRT XS", embedding search with open (and closed) models would fall below bog-standard* BM25 lexical search. Once you trained a model, performance would kick up above lexical search…and if you combined lexical _and_ vector search, things were great.
Also, a member on our team developed an amazing RNN-based model that still today beats the pants off most embedding models when it comes to speed, and is no slouch on CPU either…
(* I'm being harsh on BM25 - it is a baseline that people often forget in vector search, but it can be a tough one to beat at times)
Totally. And this has even happened in search. Open source search engines like Elasticsearch, etc did this... Google etc did this in the early Web days, and so on :)
> 1. Fine-tuning OSS embedding models on your real-world query patterns
This is not as easy as you make it sound :)
Typically, the embeddings are multi-modal: the query string maps to a relevant document that I want to add as context to my prompt.
If i collect lots of new query strings, i need to know the ground truth "relevant document" it maps to. Then I can use the two-tower embedding model to learn the "correct" document/context for a query.
I have thought about this problem for LLMs that do function calling. And what you can do is collect query strings and the function calling results, and ask GPT-4 - "is this a 'good' answer?". GPT-4 can be a teacher model for collecting training data for my two-tower embedding model.
Embeddings are good at capturing surface level information but can't match implicit/deeper/conclusion level information. Say you have a collection of 100,000 math problems, and you want to embed them to search problems that give result "0". Any number of problems can give this result and it is not explicit in the problem statement. But if you solve the problems you can see the data was in there, just not apparent.
In general you can see the raw text as a simulation premise that will generate inferences when "executed". The inferenced part is like the hidden part of the iceberg, you don't see it but it is there, implicit in the source text. Not just in math, but in all fields.
Embeddings are only good at superficial retrieval. The text needs to be fully analyzed with LLMs before embedding. Thus my conclusion is that we still have a long way to go, we haven't peaked.
Oh the embedding LLMs are usually lightweight BERT models with few layers and <<1B weights, while LLMs are easily 10-100x larger. The idea is to ingest the text in a LLM to extract the facets you are going to search and add those extra tokens to the original text. Then you do regular RAG.
> The text needs to be fully analyzed with LLMs before embedding.
If you happen to know what kinds of questions you will be asking about your RAG index, you should pre-process the texts to add QA pairs. Otherwise you can prompt the LLM to do chain-of-thought inferences based on the source text and add them to the material.
The thing to know about Mongo is, every database involves design choices that balance ergonomics, performance, and reliability. Every one, except Mongo which, according to their sales team, is the best at everything and has no faults, unless your technical choices are incorrect. In fact I just learned (in a lunch and learn with their team) that when you de-normalize data, inconsistency issues aren't really a problem, and joins are so unusably slow in ALL use cases anyways. Went ahead and just threw my DDIA book in the trash, as they nodded approvingly.
I think https://vespa.ai/ has the right approach in this space by focusing on being hybrid - vectors alone aren't great for production use cases, it's the combining of vectors+text that lets you use ranking to get meaningful result.
(I'm an investor so I'm biased; but it's also the reason why I invested)
This is a ridiculous rant. “ oh no! We have choices”. Then you list out every choice available for what is a new space people are exploring and the list is barely a half dozen long? It’s more like this is peak “claiming everything is peak”.
Yeah, I'm happy there's a lot of development in this area - even if it's fueled by the LLM frenzy, good nearest neighbor search solutions are useful in a lot of domains. Though I worked a little bit on this problem over 10 years ago (with an application to visual SLAM), and it is a bit amusing to see that a lot of the ideas and even the libraries are still the same!
Author here, well yeah, I agree its probably ridiculous. Sort of testing the waters to see if I'm way off base.
I think what I mean to say is that, in my experience, practitioners and vendors alike are overly focused on "just put embeddings somewhere and do cosine similarity" and that's the only problem to solve. In fact, that's a teeny tiny part of it. Hence "peak vector DB".
So I think the market needs some education that its harder than that. That part is my rant :). I've spoken / worked on enough problems now to see that disconnect between market and reality.
Though I think "vector DB" is actually a place for capital/brainpower to concentrate to solve these other problems. And I think we'll see the vector DB vendors pivot there. It's just taking a while for the market and investors to see this...
It sounds like you’ve conflated “gold rush” with “peak”. All sorts of novel technologies had mad rushes when they’re new, but that does not mean they have peaked. The dot bomb era with its ridiculous overvalued useless startups was a gold rush, but it was in no way peak Internet.
> practitioners and vendors alike are overly focused on "just put embeddings somewhere and do cosine similarity" and that's the only problem to solve
I agree, and as one who does exactly and only this on the search side, it's also something that falls flat on its face if you don't think a little more about the data and tasks involved.
I wrote about it here[0], but the gist of it for our use case is that if we don't intentionally include what may be considered "less relevant" data then we stand a good chance at failing our main generative task.
Normally having a lot of choices is a good thing, but here we are facing a dozen of vector dbs with very similar features - to the root it's just some version of ANN implemented in C++/Rust/whatever, the "peak" means there's nothing new. People are flooding into this field not because there's something worth inventing, but more of fear to lag behind and miss the quick money. That's what I feel about vector DBs in Jan, 2024.
Don't confuse a feature with a product. Postgres works great and you can layer in cosine similarity along with full-text search in a single query if you need to.
The big LLM companies are well positioned to build a lot of what a vector database is used for into their existing APIs and offerings. Both simplifying DX and devops.
Then on the other side existing databases will want to add functionality to be used as vector databases as well.
I think there’s lots of innovation ahead and it’s too soon to know what the end outcome will be.
Not yet. There is excitement for vector databases in some specialized areas but it hasn't really filtered out to the wider rank-and-file software engineering circles. You know it will be 'peak vector database' when you'll see blog posts on migrating your relational data to a vector database (with a follow-up 2 years later about moving back to PostgreSQL due to the shitshow that ensued).
147 comments
[ 5.0 ms ] story [ 213 ms ] threadAlso, you are probably doing it wrong by turning a matrix to matrix multiplication into a for loop (over rows). The optimal solution results in better performance
sim = np.vstack(df.col) @ vec
The one I'm working on right now has 115K docs (some quite big - I'll likely have to prune the largest 10% just to fit in my RAM).
These are all "small" - for personal use on my local machine. I'm currently RAM limited, otherwise I can think of (personal) use cases that are an order of magnitude larger.
Of course, for all I know, your method may still be as fast on those as on a vector DB.
Going further down the AI == compression path, there’s: http://prize.hutter1.net/
Always felt they're more like hashes/fingerprints for the RAG use cases.
> Typically documents are broken down into chunks
That's what I would have guessed. It's still surprising that the embeddings don't fit into RAM though.
That said (the following I just realized), even if the embeddings don't fit into RAM at the same time, you really don't need to load them all into RAM if you're just performing a linear scan and doing cosine similarity on each of them. Sure it may be slow to load tens of GB of embedding info... but at this rate I'd be wondering what kind of textual data one could feasibly have that goes into the terrabyte range. (Also, generating that many embedding requires a lot of compute!)
Yes, I see where you’re coming from. Perceptual hashes[0] are pretty similar, the key is that similar documents should have similar embeddings (unlike cryptographic hashes, where a single bit flip should produce a completely different hash).
Nice embeddings encode information spatially, a classic example of embedding arithmetic is: king - man + woman = queen[1]. “Concept Sliders” is a cool application of this to image generation [2].
Personally I’ve not had _too_ much trouble with running out of RAM due to embeddings themselves, but I did spend a fair amount of time last week profiling memory usage to make sure I didn’t run out in prod, so it is on my mind!
[0] https://en.m.wikipedia.org/wiki/Perceptual_hashing
[1] https://www.technologyreview.com/2015/09/17/166211/king-man-...
[2] https://github.com/rohitgandikota/sliders
Each vector is 1536 numbers. I don't know how many bits per number, but I'll assume 64 bits (8 bytes). So total size is 1536 * 115K * 8 / 1024^2 gives 1.3GB.
So yes, not a lot.
I still haven't set it up so I don't know how much space it really will take, but my 40K doc one took 2-3 GB of RAM. It's not pandas DF, but in an in-memory DB so perhaps there's a lot of overhead per row? I haven't debugged.
To be clear, I'm totally fine with your approach if it works. I have very limited time so I was using txtai instead of rolling my own - it's nice to get a RAG up and running in just a few lines of code. But for sure, if the overhead of txtai is really that significant, I'll need to switch to pure pandas.
You'll realize that it scales well beyond 1k.
The problem is what happens when you have an additional 6 orders of magnitude of data, and the data itself is significantly larger than the system RAM, which is a very realistic case in a search engine.
Here's some back of the envelope math. Let's say you are using a 1B parameter LLM to generate the embedding. That's 2B FLOPs per token. Let's assume a modest chunk size, 2K tokens. That's 4 trillion FLOPs for one embedding.
What about the dot product in the cosine similarity? Let's assume an embedding dim of 384. That's 2 * 384 = 768.
So 4 trillion ops for the embedding vs 768 for the cosine similarity. That's a factor of about 1 billion.
So you could have a billion embeddings - brute forced - before the lookup became more expensive than generating the embedding.
What does that mean at the application level? It means that the time needed to generate millions of embeddings is measured in GPU weeks.
The time needed to lookup an embedding using an approximate nearest neighbors algorithm from millions of embeddings is measured in milliseconds.
The game changed when we switched from word2vec to LLMs to generate embeddings.
1 billion times is such a big difference that it breaks the assumptions earlier systems were designed under.
The embedding is generated once. Search is done whenever a user inputs a query. The cosine similarity is also not done on a single embedding, it's done on millions or billions of embeddings if you are not using an index. So what the actual conclusion is, is that once you have a billion embeddings a single search operation costs as much as generating an embedding.
But then, you are not even taking into account the massive cost of keeping all of these embeddings in memory ready to be searched.
Another one is where the data is sliced based on a key, eg user id, particular document being worked on right now, etc
This issue is prevalent throughout infrastructure projects. Someone decides they need a RAG system and then the team says "let's find a vector db provider!" before they've proven value or understood how much data they have or anything. So they waste a bunch of time and money before they even know if the project is likely to work.
It's just like the old model of setting up a hadoop cluster as a first step to do "big data analytics" on what turns out to be 5GB of data that you could fit in a dataframe or process with awk https://adamdrake.com/command-line-tools-can-be-235x-faster-... (edit: actually currently on the HN front page)
It's a perfedt storm of sales led tooling where leadership is sold something they don't understand, over-engineering, and trying to apply waterfall project management to "AI" projects that have lots of uncertainty and need a re-risking based project approach where you show that it's liable to work and iterate instead of building a big foundation first.
These days anything less than 2TB should be done 100% in memory.
You might be interested in SearchArray which emulates the classic search index side of things in a pandas dataframe column
https://github.com/softwaredoug/searcharray
Not sure if others have gone down this path but I have been testing out ways to store vectors to disk in files for later retrieval and then doing everything in memory. For me the tradeoff of a sligtly slower response time was worth it compared to the 4-5 figure bill I would be getting from a vector DB otherwise.
- Vectors are massive data wise. In our current production database they take up 95% of the memory - should they be stored separately?
- Better support for easily re-embedding, hybrid search, certain RAG workflows
- Stronger performance once you're dealing with millions of vectors.
I would still stick with PgVector until you're dealing with non trivial scale.
There are ways to speed things up dramatically. Index build just became multithreaded (see above).
We have ideas on what to do with ingest.
Also do you interest from S3 ?
https://neon.tech/docs/extensions/pgvector
https://github.com/pgvector/pgvector/issues/409#issuecomment...
But if Postgres wins we all win!
The closest I can see is the VSS extension[1] for Sqlite.
[1]: https://github.com/asg017/sqlite-vss
Context: I'm working on an e2ee alternative to Google Photos[1] where we have to cluster embeddings (for face recognition) and run similarity searches (for semantic search[2]) on device.
[1]: https://ente.io
[2]: https://openai.com/research/clip
[1]: https://github.com/unum-cloud/usearch
This is already the case. Recommendations are just a fancy search where the query is a vector representing the user. Whether the learning is batched or not doesn't change the fact that it will use vector search for at least candidate generation.
There is still plenty of room for innovation in this space. Just need to focus on the right projects that are innovating and not the ones (re)working on problems solved in 2020/2021.
And here's a post on an alternative way to integrate vectors with traditional databases (Postgres, MySQL) - https://neuml.hashnode.dev/external-database-integration
As others have said in this thread, cosine similarity on arrays of vectors isn't novel. But there are many possibilities past that, many we haven't thought of yet too.
https://www.reuters.com/technology/openai-annualized-revenue...
The same principle applies to defense. The US gives Israel and Ukraine tens of billions, but that’s a credit to buy from US defense firms. That money gets recycled right back to US weaponry.
But in the Microsoft example it is customers who are paying Microsoft to use OpenAI via Azure. Thats a free market of money inflows. Same with all the people using OpenAI directly. Not sure how you would even think of the money being recycled in this scenario. Yes of course there is some back scratching in the sense that Microsoft invested in OpenAI with a large portion of that investment in Azure credits which makes the investment quite nice from MSFT's side but there is still real demand for Azure services to use OpenAI apis.
This is obvious, but if you need some journalist to validate what is already logically clear:
https://www.wsj.com/tech/ai/ais-costly-buildup-could-make-ea...
https://www.wsj.com/tech/ai/ai-deals-microsoft-google-amazon...
You are really conflating too many things at once.
1) Yes, big tech is having a hard time monetizing their bespoke AI tooling within their own ecosystem.
2) Yes, big tech has made investments in the AI space where they are providing a portion of that funding as credits to use in their cloud offerings.
3) Here is where you are incorrect though. Companies are writing large checks for the raw compute/access to AI models. It is true across the spectrum of Azure OpenAI, OpenAI directly, AWS Bedrock etc, there are a lot of companies both big and small using these services heavily. To think otherwise is naive.
Cosine similarity measures the angle between two vectors instead, and doesn't suffer from the curse of dimensionality.
I guess it's important to have this in your DB, so you make "nearby" queries (give me text that's similar to this other text) in an efficient way.
I know it's subjective, but databases have started to feel like running a window manager and desktop on a server.
I feel like software needs to take a step back and rethink itself after years of putting chimps at typewriters searching for Shakespeare.
Why not a Linux kernel with a module(s) to provide the same assurances, SQL operations? Write directly to the filesystem?
Why is all the mathematical concept that we derive software from packaged into endless conceptual blobs of black box state?
If there is, what api differentiates it, and why can’t this be expressed in either elasticsearch or Postgres?
I think there's space for a much more interesting product that is longer-lived (since it's harder to implement than just cosine-similarity-search on vectors), which is:
1. Fine-tuning OSS embedding models on your real-world query patterns
2. Storing and recomputing embeddings for your data as you update the fine-tuned models.
MTEB averages are fine, but hardly anyone uses the average result: most use cases are specialized (i.e. classification vs clustering vs retrieval). The best models try to be decent at all of those, but I'd bet that finetuning on a specific use case would beat a general-purpose model, especially on your own dataset (your retrieval is probably meaningfully different than someone else's: code retrieval vs document Q&A, for example). And your queries are usually specialized! People using embeddings for RAG are generally not also trying to use the same embeddings for clustering or classification; and the reverse is true too (your recommendation system is likely different than your search system).
And if you're fine-tuning new models regularly, you need storage + management, since you'll need to recompute the embeddings every time you deploy a new model.
I would pay for a service that made (1) and (2) easy.
(easy to extend to two layers mlp as needed. maybe ensure that x and y are zero mean and unit length to make training the matmul a bit easier.)
Also, a member on our team developed an amazing RNN-based model that still today beats the pants off most embedding models when it comes to speed, and is no slouch on CPU either…
(* I'm being harsh on BM25 - it is a baseline that people often forget in vector search, but it can be a tough one to beat at times)
When the time is finally right, people just "invent" what you made all over again.
I know nothing about search, but a bit about ML, so I'm curious
This is not as easy as you make it sound :) Typically, the embeddings are multi-modal: the query string maps to a relevant document that I want to add as context to my prompt. If i collect lots of new query strings, i need to know the ground truth "relevant document" it maps to. Then I can use the two-tower embedding model to learn the "correct" document/context for a query.
I have thought about this problem for LLMs that do function calling. And what you can do is collect query strings and the function calling results, and ask GPT-4 - "is this a 'good' answer?". GPT-4 can be a teacher model for collecting training data for my two-tower embedding model.
Reference: https://www.hopsworks.ai/dictionary/two-tower-embedding-mode...
In general you can see the raw text as a simulation premise that will generate inferences when "executed". The inferenced part is like the hidden part of the iceberg, you don't see it but it is there, implicit in the source text. Not just in math, but in all fields.
Embeddings are only good at superficial retrieval. The text needs to be fully analyzed with LLMs before embedding. Thus my conclusion is that we still have a long way to go, we haven't peaked.
It’s a pretty simple thing to add to a pipeline. Have you tried?
I am currently testing embeddings/RAG and could use some insight on how to make the results better.
If you happen to know what kinds of questions you will be asking about your RAG index, you should pre-process the texts to add QA pairs. Otherwise you can prompt the LLM to do chain-of-thought inferences based on the source text and add them to the material.
I guess you put put a whole doc into the I’ll and ask what questions it answers?
And then use those question plus a piece of the text and do an embedding?
Did it? After using Mongo in my current job (not my choice), I'd choose Postgres again for my next project.
(I'm an investor so I'm biased; but it's also the reason why I invested)
I think what I mean to say is that, in my experience, practitioners and vendors alike are overly focused on "just put embeddings somewhere and do cosine similarity" and that's the only problem to solve. In fact, that's a teeny tiny part of it. Hence "peak vector DB".
So I think the market needs some education that its harder than that. That part is my rant :). I've spoken / worked on enough problems now to see that disconnect between market and reality.
Though I think "vector DB" is actually a place for capital/brainpower to concentrate to solve these other problems. And I think we'll see the vector DB vendors pivot there. It's just taking a while for the market and investors to see this...
I agree, and as one who does exactly and only this on the search side, it's also something that falls flat on its face if you don't think a little more about the data and tasks involved.
I wrote about it here[0], but the gist of it for our use case is that if we don't intentionally include what may be considered "less relevant" data then we stand a good chance at failing our main generative task.
[0]: https://phillipcarter.dev/2024/01/15/three-properties-of-dat...
Same with languages.
Why so many languages.
Why can't we all get behind a few, do we need more than 6? For every case/problem? Put all our combined resources towards a smaller set.
We need a few DB's, a few languages, a few frameworks. Do we need hundreds?
Like everyone rolls their own everything.
Then on the other side existing databases will want to add functionality to be used as vector databases as well.
I think there’s lots of innovation ahead and it’s too soon to know what the end outcome will be.
Cassandra and Scylla are row based distributed key value stores.