HN Explorer

Ask HN: How are you doing RAG locally?

413 points by tmaly ↗ HN
I am curious how people are doing RAG locally with minimal dependencies for internal code or complex documents?

Are you using a vector database, some type of semantic search, a knowledge graph, a hypergraph?

99 comments

[ 1.6 ms ] story [ 106 ms ] thread
eajr
Local LibreChat which bundles a vector db for docs.
rahimnathwani
If your data aren't too large, you can use faiss-cpu and pickle

https://pypi.org/project/faiss-cpu/

notyourwork
For the uneducated, how large is too large? Curious.
hahahahhaah
Shoud it be:

If the total size of your data isn't loo large...?

Data being a plural gets me.

You might have small datums but a lot of kilobytes!

pousada
Data is technically a plural but nobody uses the singular and it’s being used as a singular term often - which is completely fine I think, nobody speaks Latin anyway
motakuk
LightRAG, Archestra as a UI with LightRAG mcp
nineteen999
A little BM25 can get you quite a way with an LLM.
jeffchuber
try out chroma or better yet as opus to!
CuriouslyC
Don't use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram, that gets better results while keeping search responses snappy.
Der_Einzige
This is true in general with LLMs, not just for code. LLMs can be told that their RAG tool is using BM25+N-grams, and will search accordingly. keyword search is superior to embeddings based search. The moment google switched to bert based embeddings for search everyone agreed it was going down hill. Most forms of early enshittification were simply switching off BM25 to embeddings based search.

BM25/tf-idf and N grams have always been extremely difficult to beat baselines in information retrieval. This is why embeddings still have not led to a "ChatGPT" moment in information retrieval.

lee1012
lee101/gobed https://github.com/lee101/gobed static embedding models so they are embedded in milliseconds and on gpu search with a cagra style on gpu index with a few things for speed like int8 quantization on the embeddings and fused embedding and search in the same kernel as the embedding really is just a trained map of embeddings per token/averaging
pdyc
sqlite's bm25
[deleted]
(comment deleted)
dvorka
Any suggestion what to use as embeddings model runtime and semantic search in C++?
cbcoutinho
The Nextcloud MCP Server [0] supports Qdrant as a vectordb to store embeddings and provide semantic search across your personal documents. This enables any LLM & MCP client (e.g. claude code) into a RAG system that you can use to chat with your files.

For local deployments, Qdrant supports storing embeddings in memory as well as in a local directory (similar to sqlite) - for larger deployments Qdrant supports running as a standalone service/sidecar and can be made available over the network.

[0] https://github.com/cbcoutinho/nextcloud-mcp-server

lormayna
I have done some experiments with nomic embedding through Ollama and ChromaDB.

Works well, but I didn't tested on larger scale

baalimago
I thought that context building via tooling was shown to be more effective than rag in practically every way?

Question being: WHY would I be doing RAG locally?

beret4breakfast
For the purposes of learning, I’ve built a chatbot using ollama, streamlit, chromadb and docling. Mostly playing around with embedding and chunking on a document library.