I had an issue. A documents folder with over 12k objects in it. A hodgepodge of folders and sub-folders. That over time had created a mess that no amount of file movement was ever going to make it usable. I wanted:
1) To keep my data local
2) be able to filter out PII and other data
3) Be able to find and delete duplicates
4) Get short synopsis of what a document is
5) Semantic and keyword search
6) All of this kept local to me requiring no internet access and no tokens spent to train someone elses AI.
The result I call DocuBrowser and in it's current form is FOSS (GPL-3) licensed for your personal use. The UI is in your browser. The AI models used are held local and are tiny, Available for Linux(RPM,Deb, and tgz) Windows and Mac. Let me know what you think and thanks for taking the time to try it out.
Could it be extended so it also extracts pictures from pptx and xlsx and run vision to get a description to be added to the text content before indexing?
I went with the python libraries (pydoc and pyxls for example), because it's portable and doesn't require a big download to a users system if they don't already have it installed.
I've run it on a VM with 4G ram and no GPU. It runs, But I really recommend 8G ram at least. If you have a GPU (like I do) with 4G vRAM that is ideal. Will get this in the readme. Thanks for the suggestion. I really tried to build this to minimal spec.
By restricted for personal use I mean it's not networked. It's running on your system only. It's not a networked commercial product able to do SSO etc. It's not an enterprise level product.
I have not set up Hister yet but it's on my list to try out. How would I do something like host it on my Unraid box but have it index/persist my local MacBook browsing history?
I just had a wild thought. Combine Hister with my RepoSearch app. Point it at a companies Internal github/gitlab and have a searchable knowledge base of your git repos.
I'm actually thinking of this for a commercial product feature. However, if you use a tool like Rclone on Windows, Linux or Mac. Mount the s3 bucket and you can then run DocuBrowse as if the s3 bucket were local.
Consider it yes, However having experience in this ... not really. For now there is a file called Decisions.md in the repo that is my "notes to self" if you will about where and what I need to do.
- Filling a need I personally have.
- Learning how to leverage AI for real world use not just to fill up a data center.
- Personal knowledge
-developing skills
I learned a solution is to turn the documents into vectors in say PostgreSQL (with pgvector) and do a cosine similarity search with a search vector. Doing a search for embed models on HuggingFace shows nomic-ai/nomic-embed-text-v1.5 and Qwen/Qwen3-Embedding-0.6B. I might have used a larger one like Qwen/Qwen3-Embedding-4B.
There's some info for AnythingLLM[0] which supports RAG. AnythingLLM has LanceDB out of the box but also supports others including pgvector.
Many such open source projects already (which is fantastic). I lose track of them.
Today it happened I needed a simple way to embed & query 1TB+ of documents, and I was looking at open source options. Can anyone tell me what their go-to solution is now? Could this be the one?
And what are the key differences vs. other open source RAG tools like kotaemon?
53 comments
[ 2.5 ms ] story [ 57.4 ms ] threadThe result I call DocuBrowser and in it's current form is FOSS (GPL-3) licensed for your personal use. The UI is in your browser. The AI models used are held local and are tiny, Available for Linux(RPM,Deb, and tgz) Windows and Mac. Let me know what you think and thanks for taking the time to try it out.
Key difference I see is that you point it to a folder instead of uploading to a system.
It's pretty cool, I've set up a share where the scanner scans, and it automatically picks it up from there and ingests it into the system.
Also need to search git repos including all branches and history (TIL/xkcd#153'd GitLab's web search can basically only do one branch at a time).
I'm working on a similar application called Hister (https://github.com/asciimoo/hister). I should borrow some of your ideas. =]
I will be using this, and I will be making contributions to it as well.
> I'm actually thinking of this for a commercial product feature
Would you consider writing down which features you would like to make commercial product features and how you would like to price them?
Pretty much in that order
There's some info for AnythingLLM[0] which supports RAG. AnythingLLM has LanceDB out of the box but also supports others including pgvector.
[0] https://docs.anythingllm.com/features/embedding-models
https://antfly.io/ https://github.com/antflydb/antfly
They’ve put a lot of effort into optimizing the local llm pipelines and I have a lot of faith in the devs working on it.
Today it happened I needed a simple way to embed & query 1TB+ of documents, and I was looking at open source options. Can anyone tell me what their go-to solution is now? Could this be the one?
And what are the key differences vs. other open source RAG tools like kotaemon?