> The Gemini embedding model, gemini-embedding-001, is trained using the Matryoshka Representation Learning (MRL) technique which teaches a model to learn high-dimensional embeddings that have initial segments (or prefixes) which are also useful, simpler versions of the same data. Use the output_dimensionality parameter to control the size of the output embedding vector. Selecting a smaller output dimensionality can save storage space and increase computational efficiency for downstream applications, while sacrificing little in terms of quality. By default, it outputs a 3072-dimensional embedding, but you can truncate it to a smaller size without losing quality to save storage space. We recommend using 768, 1536, or 3072 output dimensions. [0]
looks like even the 256-dim embeddings perform really well.
To anyone working in these types of applications, are embeddings still worth it compared to agentic search for text? If I have a directory of text files, for example, is it better to save all of their embeddings in a VDB and use that, or are LLMs now good enough that I can just let them use ripgrep or something to search for themselves?
If you have a million records of unstructured text (very common, maybe website scrapes of product descriptions, user reviews, etc) you want to be doing an embedding search on these to get the most relevant docs.
If you have a hundred .py files than you want your agent to navigate through these with a grep tool
Question to other GCP users, how are you finding Google's aggressive deprecation of older embedding models? Feels like you have to pay to rerun your data through every 12 months.
Tool calling complements RAG. You build a full scale RAG (embedding, reranker, create prompt, get output from LLM) and hook that to a tool another agent can see. That combines both their power.
> Embeddings are crucial here, as they efficiently identify and integrate vital information—like documents, conversation history, and tool definitions—directly into a model's working memory.
I feel like I'm falling behind here, but can someone explain this to me?
My high-level view of embedding is that I send some text to the provider, they tokenize the text and then run it through some NN that spits out a vector of numbers of a particular size (looks to be variable in this case including 768, 1536 and 3072). I can then use those embeddings in places like a vector DB where I might want to do some kind of similarity search (e.g. cosine difference). I can also use them to do clustering on that similarity which can give me some classification capabilities.
But how does this translate to these things being "directly into a model's working memory'? My understanding is that with RAG I just throw a bunch of the embeddings into a vector DB as keys but the ultimate text I send in the context to the LLM is the source text that the keys represent. I don't actually send the embeddings themselves to the LLM.
So what is is marketing stuff about "directly into a model's working memory."? Is my mental view wrong?
Perhaps the person that wrote it is also confused. I guess Geminis embedding model offers multilingual support, but we can use anything. The assumption is the developer uses these embeddings on their end with their implementation of storage/querying (their own vector db). The confusing thing is the article is suggesting that whole process is now done automatically soon as you send the embeddings to Gemini (which doesn’t even make sense, shouldn’t it only take text?).
Your comment really helps me improve my mental model about LLM. Can someone smarter help me verify my understanding:
1) at the end of the day, we are still sending raw text over LLM as input to get output back as response.
2) RAG/Embedding is just a way to identify a "certain chunk" to be included in the LLM input so that you don't have to dump the entire ground truth document into LLM
Let's take Everlaw for example: all of their legal docs are in embeddings format and RAG/tool call will retrieve relevant document to feed into LLM input.
So in that sense, what do these non-foundational models startups mean when they say they are training or fine tuning models? Where does the line end between inputting into LLM vs having them baked in model weights
Qwen3 is the open weight state of the art at the moment. Qwen3-embedding-8B and Qwen3-reranker-8b are surprisingly good (according to some benchmarks, better than Gemini 2.5 embedding). 4B is also nearly as good so you might as well use that too unless 8B benefits your usecase. If you don't need a SOTA-precise embedding model because you'll run a more powerful reranker, you could run qwen3-embedding-4B at Q4 which is only 2GB, and will process extremely fast in most hardware. A weaker but close choice is `Qwen3-Embedding-0.6B` at Q8 which is about 600MB and will run just fine on most powerful CPUs. So if that does the job for you, you may not even need GPU, just grab an instance with 16 vCPUs, that'll give you plenty of throughput, probably more than you need until your RAG has thousands of active users.
Another benefit of having your own test dataset, is that it can grow as your data grows. And you can quickly test new models to see how it performs with YOUR data.
> Everlaw, a platform providing verifiable RAG to help legal professionals analyze large volumes of discovery documents, requires precise semantic matching across millions of specialized texts. Through internal benchmarks, Everlaw found gemini-embedding-001 to be the best, achieving 87% accuracy in surfacing relevant answers from 1.4 million documents filled with industry-specific and complex legal terms, surpassing Voyage (84%) and OpenAI (73%) models. Furthermore, Gemini Embedding's Matryoshka property enables Everlaw to use compact representations, focusing essential information in fewer dimensions. This leads to minimal performance loss, reduced storage costs, and more efficient retrieval and search.
This will make a lot of junior lawyers or their work obsolete.
Here is a good podcast on the topic how will AI affect legal industry
I'm short on vocabulary here but it seems that using content embedding similarity to find relevant (chunks of) content to feed an LLM is orthogonal to the use of LLMs to take automatically curated content chunks and use them to enrich a context.
Is that correct?
I'm just curious why this type of content selection seems to have been popularized and in many ways become the defacto standard for RAG, and (as far as I know but I haven't looked at 'search' in a long time) not generally used for general purpose search?
What do you mean by automatically curated content chunks? RAG with Embedding search is the process of deciding which chunks go into the context of the bot so that it can reference them to answer a user question
Possibly because up until now the performance of semantic based search wasn't worth the complexity tradeoff. I mean NLP was a hard problem, and we'd spent decades fine-tuning traditional keyword based search.
has anyone done some simple latency profiling of gemini embedding vs open ai embedding api? seem like that api call is one of the biggest chunks of time in a simple rag setup.
Good luck to anyone using it. We used it for embedding about 6k documents.
The API constantly gives you quota errors when you reach about 150 requests/min eventhough the quota should allow about 50_000 requests/min.
We’d like to use the Batch API, but the model isn’t available yet.
Quite a nice model though. Being able to get embeddings for a specific task type [1] is very interesting. We used classification specific embeddings and noticed a meaningful improvment when we used the embeddings as input for a classifier.
Anyone who has recently worked on embedding model finetuning, any useful tools you'd recommend (both for dataset curation and actual finetuning)? Any models you'd recommend as especially good for finetuning?
I'm interested in both full model finetunes, and downstream matrix optimization as done in [1].
I have been thinking around solving this problem. I think one of the reasons some AI assistants shine vs others is how they can reduce the amount of context the LLM needs to work with using in-built tools. I think there's room to democratize these capabilities. One such capability is allowing the LLMs to directly work with the embeddings.
I wrote an MCP server directory-indexer[1] for this (self-hosted indexing mcp server). The goal being indexing any directories you want your AI to know about and gives the it MCP tools to search through the embeddings etc. While an agentic grep may be valuable, when working with tons of files with similar topics (like customer cases, technical docs), pre-processed embeddings have proven valuable for me. One reason I really like it is that it democratizes my data and documents: giving consistent results when working with different AI assistants - the alternative being vastly different results based on the in-built capabilities of the coding assistants. Another being having access to you "knowledge" from any project you're on. Though since this is selfhosted, I use nomic-embed-text for the embedding which has been sufficient for most use cases.
28 comments
[ 3.2 ms ] story [ 46.3 ms ] thread> The Gemini embedding model, gemini-embedding-001, is trained using the Matryoshka Representation Learning (MRL) technique which teaches a model to learn high-dimensional embeddings that have initial segments (or prefixes) which are also useful, simpler versions of the same data. Use the output_dimensionality parameter to control the size of the output embedding vector. Selecting a smaller output dimensionality can save storage space and increase computational efficiency for downstream applications, while sacrificing little in terms of quality. By default, it outputs a 3072-dimensional embedding, but you can truncate it to a smaller size without losing quality to save storage space. We recommend using 768, 1536, or 3072 output dimensions. [0]
looks like even the 256-dim embeddings perform really well.
[0]: https://ai.google.dev/gemini-api/docs/embeddings#quality-for...
If you have a million records of unstructured text (very common, maybe website scrapes of product descriptions, user reviews, etc) you want to be doing an embedding search on these to get the most relevant docs.
If you have a hundred .py files than you want your agent to navigate through these with a grep tool
But am I crazy or did the pre-production version of gemini-embedding-001 have a much larger max context length?
Edit: It seems like it did? 8k -> 2k? Huge downgrade if true, I was really excited about the experimental model reaching GA before that
I feel like I'm falling behind here, but can someone explain this to me?
My high-level view of embedding is that I send some text to the provider, they tokenize the text and then run it through some NN that spits out a vector of numbers of a particular size (looks to be variable in this case including 768, 1536 and 3072). I can then use those embeddings in places like a vector DB where I might want to do some kind of similarity search (e.g. cosine difference). I can also use them to do clustering on that similarity which can give me some classification capabilities.
But how does this translate to these things being "directly into a model's working memory'? My understanding is that with RAG I just throw a bunch of the embeddings into a vector DB as keys but the ultimate text I send in the context to the LLM is the source text that the keys represent. I don't actually send the embeddings themselves to the LLM.
So what is is marketing stuff about "directly into a model's working memory."? Is my mental view wrong?
1) at the end of the day, we are still sending raw text over LLM as input to get output back as response.
2) RAG/Embedding is just a way to identify a "certain chunk" to be included in the LLM input so that you don't have to dump the entire ground truth document into LLM Let's take Everlaw for example: all of their legal docs are in embeddings format and RAG/tool call will retrieve relevant document to feed into LLM input.
So in that sense, what do these non-foundational models startups mean when they say they are training or fine tuning models? Where does the line end between inputting into LLM vs having them baked in model weights
There are some good open models there that have longer context limits and fewer dimensions.
The benchmarks are just a guide. It's best to build a test dataset with your own data. This is a good example of that: https://github.com/beir-cellar/beir/wiki/Load-your-custom-da...
Another benefit of having your own test dataset, is that it can grow as your data grows. And you can quickly test new models to see how it performs with YOUR data.
This will make a lot of junior lawyers or their work obsolete.
Here is a good podcast on the topic how will AI affect legal industry
https://open.spotify.com/episode/4IAHG68BeGZzr9uHXYvu5z?si=q...
Is that correct?
I'm just curious why this type of content selection seems to have been popularized and in many ways become the defacto standard for RAG, and (as far as I know but I haven't looked at 'search' in a long time) not generally used for general purpose search?
Possibly because up until now the performance of semantic based search wasn't worth the complexity tradeoff. I mean NLP was a hard problem, and we'd spent decades fine-tuning traditional keyword based search.
https://huggingface.co/spaces/mteb/leaderboard
Particularly Qwen3-Embedding-8B and Qwen3-Embedding-4B:
https://huggingface.co/Qwen/Qwen3-Embedding-8B
For now there is only https://exa.ai/ that is currently doing something similar it seems.
The API constantly gives you quota errors when you reach about 150 requests/min eventhough the quota should allow about 50_000 requests/min.
We’d like to use the Batch API, but the model isn’t available yet.
Quite a nice model though. Being able to get embeddings for a specific task type [1] is very interesting. We used classification specific embeddings and noticed a meaningful improvment when we used the embeddings as input for a classifier.
1: https://ai.google.dev/gemini-api/docs/embeddings#supported-t...
I'm interested in both full model finetunes, and downstream matrix optimization as done in [1].
[1] https://github.com/openai/openai-cookbook/blob/main/examples...
I wrote an MCP server directory-indexer[1] for this (self-hosted indexing mcp server). The goal being indexing any directories you want your AI to know about and gives the it MCP tools to search through the embeddings etc. While an agentic grep may be valuable, when working with tons of files with similar topics (like customer cases, technical docs), pre-processed embeddings have proven valuable for me. One reason I really like it is that it democratizes my data and documents: giving consistent results when working with different AI assistants - the alternative being vastly different results based on the in-built capabilities of the coding assistants. Another being having access to you "knowledge" from any project you're on. Though since this is selfhosted, I use nomic-embed-text for the embedding which has been sufficient for most use cases.
[1] https://github.com/peteretelej/directory-indexer