TinySearch wraps this and works well for agents. It's better than the native SearXNG MCP because it optimizes the context before it even gets to the agent so as to not waste tokens.
I’ll have to try, I’ve only recently learned Exa pricing is a bit crazy (especially on searches where you source 30-40 sources)I just used it be default and then was like oh damn when I got hit
I've been using this for some projects. It's exceptional and I recommend it highly.
I actually included a recipe to deploy it to kubernetes in typekro, my TypeScript infrastructure-as-code project for kubernetes: https://typekro.run/api/searxng/
I've been using SearXNG for a few years now, however I've been trying out Degoog as a SearXNG alternative since I've had issues with engines constantly failing or being slow since day 1 of using SearXNG, but Degoog has worse results with the same engines. It's a shame since I'm having to pick between slower but better results, or very fast but worse results.
Yeah, I find that searx results are way more relevant to what I’m actually looking for than a single engine. There’s so much manipulation going on that if you don’t aggregate multiple engines, it’s near impossible to get what you want.
I am also interested in what a full local AI stack with web search and other tools looks like. As far as I can tell, SearX does not embed an MCP server, so it can't be directly called from llama-server for example. Open WebUI does have an integration for SearX and other providers, but the results I obtained weren't particularly impressive.
I use Searxng through Onyx, both as regular search and Onyx's Deep Research mode. I also have https://github.com/ihor-sokoliuk/mcp-searxng to add search to coding agents. Haven't really had many issues with it.
It has a JSON mode that you need to enable in settings and then you can create a simple python script to interact with it or have the agent use `curl` and `jq` to interact with it.
i have a friend with a 4080 that is wanting to experiment with local models and those cards should be similar enough. can you give any more detail about your setup? ty!
I know I'm replying to comment from yesterday that nobody will see, but here's my setup:
I run a local Qwen 3.6 27B at a 5-bit quantization on a 3090ti - it's a dedicated server that used to be my old gaming PC/workstation. I mostly use it with a chat interface for deep web searching and simple script generation. Here's what my setup looks like:
LLM host - the aforementioned physical box that contains the GPU and hosts Qwen 3.6 27B and also a CPU-only Qwen 2.5 1.5B that I use for generating chat titles in Librechat.
Librechat host - a VM hosting librechat; this provides the chat interface and interfaces with the search provider.
Search host - a VM with SearXing and Firecrawl.
The flow is:
- Qwen decides it needs to make a search tool call and does so
- Librechat receives the call, and feeds it to the SearXing API. It receives search results in JSON format.
- Librechat feeds the result URLS to firecrawl, which scrapes the pages and returns the contents to Librechat.
- Librechat prompts Qwen with the results of the search, and it either formulates an answer or performs additional searches based on the results.
What I don't have yet is a reranker; this is a specialized model that analyzes the scraped documents and returns only the ones (or even just snippets) most relevant to the query. This would be useful, because sometimes the LLM ends up semi-randomly with very irrelevant results and spends multiple turns trying to get better ones and/or gives a bad answer. No specific reason not, I just haven't implemented it yet.
Ohi, I'm the original creator of Searx, but due to the limitations of the metasearch concept I'm not involved in the development anymore. My new search project is https://github.com/asciimoo/hister (https://hister.org/).
Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews and the full page content via MCP.
Sure, it cannot fully replace web search engines (yet), but it can reduce the dependence on these services more and more as your index grows. Hister is designed to support quickly falling back to traditional search engines with a single hotkey if no results found.
I agree, we should add more extractors [1]. Can you recommend extractors you missed?
Both are search engines, but that's all the similarity. Hister has a traditional crawler, but its biggest strength is automatically indexing browser tabs as those are rendered. This way it bypasses authentication, CloudFlare, captchas and most of the annoying limitations of traditional crawlers.
Hister also provides full offline result previews. Check out the small read-only demo: https://demo.hister.org/
I love your idea and wondered why saving and indexing browser visited pages we done being done at scale. Does this handle large amounts of local files, for example 10-20TB across file types like Powerpoint, Excel, Word, and PDF?
In its current form it cannot handle this amount of data (and doesn't support powerpoint/excel/word yet), but this is a valid use-case, I've added a TODO item to experiment with it.
I'm sorry for not taking the time to read the docs, but I have a question.
Some 20 years ago a friend of mine has set up a local proxy (python if I'm not mistaken) that was gathering all his web traffic and served him as a long term memory. The proxy had a web interface and allowed him to quickly find something he saw ca. 10 days ago, or that specific algorithm he recalls but can't remember it's name.
For years I've been collecting links to different work related trivia which I use on a daily basis as a rabbit-from-a-hat solution to answer random question from friends and coworkers. For example someone randomly asked me for an idea for color palette for data charts and I can immediately give them a scientific research into the color palette. Or an obscure algorithm.
But with time the collection has grown substantially and it's really cumbersome to find the proper things.
Also very interested in this. I was playing around with doing the same thing with YaCY. I want the proxy aspect so that I can proxy my phone traffic through it as well.
Absolutely, this is a great example where Hister can shine.
I started Hister as a proxy as well, but quickly switched to the current extension based approach, because intercepting HTTPS traffic requires a MiTM proxy which is much more painful to setup than installing a browser extension.
would it be possible to gdrive/rsync/git the data between machines and then use the data on an online server for retrieval (given that I would handle data sync myself)?
also what exactly are you using for search? does it support trigrams? how do you sort results?
I found Hister a few month ago and was amazed by it.
Now for many of us the browser extension approach is not possible (mobile usage, security, etc.)
My feeling is for a lot of users there is really a third way apart from the MiTM proxy or Browser extension approach. I actually do not want my "personal" / "logged in" pages to be indexed. This is a bit like the MS recall nightmare (self hosted version).
Any way to get the list of URL visited (with something like Privoxy, or maybe one of those popular ad blockers like Pi Hole but I guess they just get DNS queries?) and then importing it with some filtering rules with a nightly batch job is good enough for a lot of people.
The browser import [1] is great but I guess hard to use with mobile...
Always excited to see new things like Hister in the search space. What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful? Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?
> What are the scaling limits, as far as you can tell in terms of how much can it hold before queries start breaking down or become too slow to be useful?
There has been no stress tests in this regard. The indexer lib Bleve [1] can handle millions of documents according to their documentation.
> Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?
My long term goal is exactly this. I'd like to add federation/P2P feature [2] to evolve from being a private search companion. I'd appreciate any help designing the system.
Wow! that looks like a bit of software I have been dreaming about for awhile - will definately check out! You're at least doing something right in communicating the reasons why and appeal for starters! All the best!
Hister sounds like something I wanted for a while, but never got around to building. Searching stuff I’ve seen before is most of what I do with a search engine, so having it local and fast would be amazing. Eager to give it a try.
I was considering paying someone to build something like this at some point. With two jobs, I eventually had no time to even organize what I find. It's just piles of links in text files.
Can I give your software a huge list of URL's to index? Or do I need to use browser automation to open them a few at a time with it caching and indexing them?
Hister has a built in crawler with standard HTTP lib and browser based backends, you can feed your link collection to it. Also, Hister supports importing your existing browser history automatically using either of the mentioned backends.
This is great, like many others I've been thinking of something like hister but only for bookmarked web pages. I presume it should be straightforward with hister to do that?
It is possible. The automatic website indexing can be turned off in the extension and manual indexing can be triggered via the command line tool, the extension popoup or by hotkeys.
YaCY has a proxy mode that automatically index your web-serving. In my experience, the index grow in size very fast and reaches ~100GB or more. How does the index size of Hister compare to that?
Hister stores only the text content of HTML/pdf pages. 1000 documents require around 80-100MB of storage and there is still plenty of room to optimize for storage space.
I'm using it for 6-7 months and my index size is below 1GB with almost 10k pages.
Also, a downside of the proxy approach: it does not handle properly JS based websites and cannot identify dynamic content changes. Our extension periodically checks if the browser tabs' content has been changed and automatically updates the index when change detected.
Hister is a great idea and the creator is a really nice person, please give it an honest look and consider supporting them (I'm Uruky's co-founder and we sponsored them)!
Interesting, a local search option. I made the recoll engine for SearX and now SearXNG and still use this daily over a rather large archive of journal articles and other non-fiction texts. Recoll's indexer can extract text from just about anything I throw at it, it also extracts and indexes metadata. Would Hister serve the same purpose and if so is there a SearXNG engine to integrate it into the result stream?
this is really cool, first time hearing about this, is there any org level model for this so you can promote individual's indexed websites into an organization/team owned model?
I've been trying to find something to use for enriching my own self-hosted LLMs and agentic tools with information I find useful. Metasearch tools like SearXNG make it less likely you'll get blocked by bot detection tools when finding information, but usually it's something I've already found, read, or seen that I want to incorporate into my tooling.
I came to the conclusion that a self-hosted content storage system with a search engine and a browser extension that can extract and save web page content and metadata was the ideal setup for me. Preferably with some sort of federated content sharing ability and the ability to import creative commons content like Wikipedia and Gutenberg.
This looks almost exactly like what I wanted.
It'll be a few weeks before I have time to audit the code and deploy it, but I'm really looking forward to trying it out.
SearXNG is my daily internet search now +5 years; with YaCY Backends and else as fallback. I also build internal document search or RAG applications with this setup (SearXNG also support json results).
However, there are some downer I accept because of privacy:
1. Its slower and the results are not that good then a others. But fast and good enough for most of my queries
2. From time to time you get blocked on the duckduckgo, brave or whatever search and you must solve some captures
> SearXNG is my daily internet search now +5 years
Same here
> with YaCY Backends and else as fallback.
Do you run your own "super fast" YaCy instance? or with specific settings?
My experience with YaCy is it doesn't fit in the backend of SearX since YaCy kind of slowly stream results for about 30 seconds...
I also have a local `kiwix-serve` serving ZIM files of wikipedia, wiktionary, gutemberg, archwiki, etc. but same problem the kiwix search engine [0] doesn't really fit as a backend for SearX as it returns too many results and pollute the SearX result page.
What I haven't done yet is trying to plug SearX to a local Recoll instance [1]. But Recoll doesn't support indexing ZIM files... but could be useful for other archived html documents.
I would be curious to know more about a working setup since search is hard to get right.
I ran my own YaCY instances. Three of them to be specific, because they are "super fast" and "reboot" often. I crawl with them the smallweb, smallcomic and smallyt sites and also all feeds from my miniflux instance; getting them via the miniflux api. Beside that i have other static entries that i crawl.
For wikibooks and wikipedia i tried and use also YaCY, but it use a lot of resources. So its only in one instance. I suggest >16GB RAM and 300GB+ HDD if you want to do this.
To access wikimedia, gutemberg, archwiki or media.ccc.de directly, I use also SearXNG.
Usually it takes 1-3 Seconds to get search results from YaCY in my setup.
I run them in docker on aarch64 with ~6GB of RAM and 200GB HDD. The VPS it-self has 8GB RAM, 6 arm cores and 250GB HDD. If YaCY hang, i just restart it.
This are my pretty good working docker deploy and java settings I use currently:
That's the smallest I got it running mostly stable and self-healing with a index size of +100GB. I also avoid to use crawling by the build in tasks and use the API and cron jobs for weekly feed importing, because I found out, that kind of crawling eats up less resources then the usual. All-Over, to much running crawlers, make retrieving search results slow.
For production use, I suggest to min. double the resources. If you do this, it becomes very stable.
Thanks you so much this is highly very valuable information.
> Thanks to pointing out kiwix. I'll give it a try.
I see YaCy works with ZIM files [0] packaged by Kiwix so this is great.
In theory if you run YaCy kiwix is not necessary but they do package already valuable sites likes Wikipedia, iFixit, archwiki, etc. [0] so you do not have the worry of your crawler to be blocked and have local copy anyway [1]. So a lot of bandwidth and headache saved.
I’ve been a big fan of SearXNG for a while now. My distain for google has only grown, so having the ability to search and avoid things like yk, small AI models being installed on my PC without my consent, is awesome.
Highly recommend this. I set up a self-hosted instance and been using it exclusively for months. It's better than DDG and i don't miss Google whatsoever.
Image search is worse i guess mostly for lack of CDN so it's slow but whatever.
I built something similar ([1]) that you might find interesting. Similar to your project, but with the fun tweak that it bundles searxng inside itself, so you don't need to run or find a searxng instance to use it.
85 comments
[ 0.28 ms ] story [ 50.9 ms ] threadhttps://github.com/MarcellM01/TinySearch
I actually included a recipe to deploy it to kubernetes in typekro, my TypeScript infrastructure-as-code project for kubernetes: https://typekro.run/api/searxng/
TOO BAD YOU IDIOTS CAN'T FIGURE OUT HOW TO BAN PEOPLE. RETARDS. LEARN TO CODE.
I'm curious what setups folks use to provide this functionality.
Since the quantized 24B parameter Gemma model came out, I've had good luck with tool calling on a 4070 Ti Super.
Successful tool calling is what finally made the local experience useful.
I should note this is for the general and not coding specific context.
It's at the bottom of this page: https://docs.searxng.org/admin/settings/settings_search.html
i have a friend with a 4080 that is wanting to experiment with local models and those cards should be similar enough. can you give any more detail about your setup? ty!
I run a local Qwen 3.6 27B at a 5-bit quantization on a 3090ti - it's a dedicated server that used to be my old gaming PC/workstation. I mostly use it with a chat interface for deep web searching and simple script generation. Here's what my setup looks like:
LLM host - the aforementioned physical box that contains the GPU and hosts Qwen 3.6 27B and also a CPU-only Qwen 2.5 1.5B that I use for generating chat titles in Librechat.
Librechat host - a VM hosting librechat; this provides the chat interface and interfaces with the search provider.
Search host - a VM with SearXing and Firecrawl.
The flow is:
- Qwen decides it needs to make a search tool call and does so - Librechat receives the call, and feeds it to the SearXing API. It receives search results in JSON format.
- Librechat feeds the result URLS to firecrawl, which scrapes the pages and returns the contents to Librechat.
- Librechat prompts Qwen with the results of the search, and it either formulates an answer or performs additional searches based on the results.
What I don't have yet is a reranker; this is a specialized model that analyzes the scraped documents and returns only the ones (or even just snippets) most relevant to the query. This would be useful, because sometimes the LLM ends up semi-randomly with very irrelevant results and spends multiple turns trying to get better ones and/or gives a bad answer. No specific reason not, I just haven't implemented it yet.
Hister is a full text indexer for websites and local files which automatically saves all the visited pages rendered by your browser. Storing full page content allows serving offline result previews and the full page content via MCP.
Take a look at how the MCP can be utilized: https://hister.org/posts/give-your-ai-assistant-a-private-me...
I agree, we should add more extractors [1]. Can you recommend extractors you missed?
[1] https://github.com/asciimoo/hister/issues/305
Some 20 years ago a friend of mine has set up a local proxy (python if I'm not mistaken) that was gathering all his web traffic and served him as a long term memory. The proxy had a web interface and allowed him to quickly find something he saw ca. 10 days ago, or that specific algorithm he recalls but can't remember it's name.
For years I've been collecting links to different work related trivia which I use on a daily basis as a rabbit-from-a-hat solution to answer random question from friends and coworkers. For example someone randomly asked me for an idea for color palette for data charts and I can immediately give them a scientific research into the color palette. Or an obscure algorithm.
But with time the collection has grown substantially and it's really cumbersome to find the proper things.
Would your project be a good fit for my problem?
I started Hister as a proxy as well, but quickly switched to the current extension based approach, because intercepting HTTPS traffic requires a MiTM proxy which is much more painful to setup than installing a browser extension.
also what exactly are you using for search? does it support trigrams? how do you sort results?
Now for many of us the browser extension approach is not possible (mobile usage, security, etc.)
My feeling is for a lot of users there is really a third way apart from the MiTM proxy or Browser extension approach. I actually do not want my "personal" / "logged in" pages to be indexed. This is a bit like the MS recall nightmare (self hosted version).
Any way to get the list of URL visited (with something like Privoxy, or maybe one of those popular ad blockers like Pi Hole but I guess they just get DNS queries?) and then importing it with some filtering rules with a nightly batch job is good enough for a lot of people.
The browser import [1] is great but I guess hard to use with mobile...
- [0] https://www.privoxy.org/
- [1] https://hister.org/docs/importing-browser-history
There is already an ongoing discussion about the topic: https://github.com/asciimoo/hister/issues/387
The currently discussed solution relies on the browser extension, but mobile Firefox has extension support.
There has been no stress tests in this regard. The indexer lib Bleve [1] can handle millions of documents according to their documentation.
> Could it evolve into a general internet search engine if, say, enough trusted members of a geo-distributed YugabyteDB cluster and an army of crawlers built a sufficient index?
My long term goal is exactly this. I'd like to add federation/P2P feature [2] to evolve from being a private search companion. I'd appreciate any help designing the system.
[1] https://blevesearch.com/docs/Home/ [2] https://github.com/asciimoo/hister/discussions/432
Can I give your software a huge list of URL's to index? Or do I need to use browser automation to open them a few at a time with it caching and indexing them?
Hister has a built in crawler with standard HTTP lib and browser based backends, you can feed your link collection to it. Also, Hister supports importing your existing browser history automatically using either of the mentioned backends.
All the best!
I'm using it for 6-7 months and my index size is below 1GB with almost 10k pages.
Also, a downside of the proxy approach: it does not handle properly JS based websites and cannot identify dynamic content changes. Our extension periodically checks if the browser tabs' content has been changed and automatically updates the index when change detected.
I've been trying to find something to use for enriching my own self-hosted LLMs and agentic tools with information I find useful. Metasearch tools like SearXNG make it less likely you'll get blocked by bot detection tools when finding information, but usually it's something I've already found, read, or seen that I want to incorporate into my tooling.
I came to the conclusion that a self-hosted content storage system with a search engine and a browser extension that can extract and save web page content and metadata was the ideal setup for me. Preferably with some sort of federated content sharing ability and the ability to import creative commons content like Wikipedia and Gutenberg.
This looks almost exactly like what I wanted.
It'll be a few weeks before I have time to audit the code and deploy it, but I'm really looking forward to trying it out.
I download only title, description, thumbnail, common og fields.
My index is very lean, I think. I have 2m of pages crawled.
https://github.com/rumca-js/Internet-Places-Database
It has tags, and votes support.
Recently I also launched my first fdroid app
https://github.com/rumca-js/OfflineWebSearch
https://f-droid.org/en/packages/io.github.rumcajs.offlineweb...
Same here
> with YaCY Backends and else as fallback.
Do you run your own "super fast" YaCy instance? or with specific settings?
My experience with YaCy is it doesn't fit in the backend of SearX since YaCy kind of slowly stream results for about 30 seconds...
I also have a local `kiwix-serve` serving ZIM files of wikipedia, wiktionary, gutemberg, archwiki, etc. but same problem the kiwix search engine [0] doesn't really fit as a backend for SearX as it returns too many results and pollute the SearX result page.
What I haven't done yet is trying to plug SearX to a local Recoll instance [1]. But Recoll doesn't support indexing ZIM files... but could be useful for other archived html documents.
I would be curious to know more about a working setup since search is hard to get right.
- [0] https://kiwix-tools.readthedocs.io/en/latest/kiwix-serve.htm...
- [1] https://docs.searxng.org/dev/engines/online/recoll.html
Thanks to pointing out kiwix. I'll give it a try.
> Thanks to pointing out kiwix. I'll give it a try.
I see YaCy works with ZIM files [0] packaged by Kiwix so this is great.
In theory if you run YaCy kiwix is not necessary but they do package already valuable sites likes Wikipedia, iFixit, archwiki, etc. [0] so you do not have the worry of your crawler to be blocked and have local copy anyway [1]. So a lot of bandwidth and headache saved.
- [0] https://github.com/yacy/yacy_search_server/tree/master/sourc...
- [1] https://browse.library.kiwix.org/#lang=eng
Image search is worse i guess mostly for lack of CDN so it's slow but whatever.
It also needs a SearXNG server to run, so I recently pivoted towards a self-contained solution: https://github.com/denysvitali/search-mcp
[1]: https://github.com/nikvdp/searxng-ai-kit