Not in the near term. Right now I am focused on developing Hister rather than operating a hosted service. There are already plenty of centralized hosted search engines, so my longer term interest is in federation and distributed search. I want to make the core system mature first.
> would've been great with a more liberal license
It depends on how do you define liberal. =] I chose AGPLv3+ because I want Hister to remain free software and available to their users.
I agree with AGPLv3, I don't really understand why people have an objection. All it prevents is someone taking your work and making a business out of it.
Safari support is currently stalled in this PR: https://github.com/asciimoo/hister/pull/46. I do not have access to macOS, so help with development and testing would be very welcome.
>Hister builds a personal search index from pages you visit, bookmarks, browser history, local files, and crawled websites.
Immediately interested and will check it out, thank you! I've wanted a "search stuff you've seen online" tool for a long time, but everything seems to be research-oriented or "archive but don't search" or some weird combination that means it's nigh useless to me. I've got decades of bookmarks and archives and I've kinda been stuck grepping them at best (it's rare but I do sometimes want a page I saw once three years ago and I love having that option), while hoping someone would build something better.
One question if ya don't mind, while I explore: any chance of singlefile support? Content-extraction is useful in lots of situations (e.g. wallabag), but sometimes it fails and sometimes you really do want the page, relatively close to how it actually was. Singlefile does that much better than most, and it does so well enough (and manually-capable enough) that I don't feel any desire to switch to WARCs or similar.
Exactly! I had the very same issues before Hister.
> One question if ya don't mind, while I explore: any chance of singlefile support?
Yes, partially. Hister can already import HTML files created by SingleFile, but there is no direct integration yet. In the longer term, I would like the SingleFile extension to be able to send snapshots directly to Hister.
I assume it content-extracts that on upload? I'd really like to move storage into hister too, if possible. That way you could also switch from an extracted view to a "full" view in the UI. Though I assume that'd be fairly simple to build later.
Overall I really like what I'm seeing, it ticks a lot of important boxes for me and it's pleasantly straightforward. Hopefully I'll find time to contribute!
> That way you could also switch from an extracted view to a "full" view in the UI.
It isn't even needed, we just need a SingleFile specific extractor (an interface in Hister to parse specific page content and provide custom previews) that provides the full original HTML for the preview panel.
I came here to mention https://github.com/gildas-lormeau/singlefile, I couldn't get it to do what I want so I built my own, for watched domains it pushes a copy of the serialized DOM to a local search database. For structured data, either extract and enrich in the browser or enrich on the server side.
Would Hister support this basic workflow? I'd love to retire my own software.
The next phase was going to move to a recording proxy.
Singlefile based web pack+download is exactly the next feature I need besides indexing URLs / browser tabs with[0], have it on my to-research for a looong time. I do have a server-side script that can be triggered on doc ingestion as a hook "if data/schema/tab is indexed and path is /ml/* then run script "fetch-url.sh" and file the result into {{doc.path}}" but being able to do that in-browser is more user-friendly.
I use Linkding for archiving links I've seen online and I love it's archive feature. But often enough it didn't work, until they added support for Singlefile from the browser.
You should integrate with Karakeep.app, it's only natural that you'd want both a search engine, and a nice "archive" and "article pretty view" features :)
I was thinking about making a "personal data search engine". I have almost 30GB of email archives and a bunch of Google Drive and offline files. I am thinking about downloading it all from the cloud and storing it on a large RAID array, with an indexing and search interface (and possibly a local MCP server).
Would Hister be suitable for this? Can it index mbox files? Would it handle this amount of data? Does it have a search API so I can build an MCP server?
The indexer it uses (Bleve) can handle millions of records according to their docs, but sure that Hister would be the best choice for this task. I'd probably use Meilisearch (https://www.meilisearch.com/).
> Can it index mbox files?
Not yet.
> Does it have a search API so I can build an MCP server?
This sounds very interesting. Wanted something similar, so going to check it out:)
One thing at first look. Personally I think it would be very nice if it was possible to index different sites to different search index. So you could separate different stuff, like job, different projects you have, and other stuff and then last everything else etc.
So it would be possible to have like "profiles" you could easy choose from. So in the addon, when you click on the hister icon on the toolbar there would be a list of profiles that you could choose where to index to, but also a setting to choose which domains and sites that always should index to one seach index profil. And all sites that dont is added to a profile is added to the default index instead.
For me this would make it much more easy to find stuff and sort out what Im looking for (for me things get very "thing"/project based what I want to find), so having this would...
1 - become like an important filter. When you search you could easy mark one or more "profiles" which you would search from, and you remove alot of unwanted data automatically.
2 - It would also make it easy if the db gets big over time to remove data that is less important later on and that you dont need to be index anymore.
3 - It would also make it easier to backup only the most important search index data DB:s if the DB for the differnt seach index:s where in different folders/files. I guess the "default index" that index all sites could easy get big, but there it would be alot of not important data. So would be nice to be able to easy to use a backup program and only backup the only important search databases.
You can create multiple users for different use-cases. Also, it is possible to automatically assign labels to documents coming from different sources and use these labels to filter results. E.g. using multiple browser profiles where different profiles apply different labels to submitted documents.
I used to be interested in the prophecies of Nostradamus a lifetime ago. You may be surprised to learn that “Hister” is a name mentioned in those prophecies that is commonly claimed by believers to refer to the future Adolf Hitler.
Academics usually take it to mean the Danube river instead, but in conspiracy/New Age contexts, the association with Hitler is prevalent, and mentioned in several pop culture works, including at least one feature film.
I’m pretty sure it’s just an unintentional name collision. The vast majority of people are unaware of this name’s connotation in Nostradamus, unless they happen to have read or watched some pop culture work that references it.
Thanks you for making Hister. I have been using it since May and find it immensely useful. It's a piece of mind to know that I always will be able to easily find what I looked at. So I don't feel the need to bookmark things, which I used to do, but then never looked at anyways ;).
Thanks for your contribution to the Universe! :)
I'm hacking on a in-process LMDB+Lance(for now, hilbert space kung-fu with a custom matryoshka embedding setup with separate spatial + temporal + internal and content derived anchors will replace that ~last-century~ last-year tech) + roaring bitmaps as the primary indexing engine with the same or similar goal[0] and will definitely deep-dive into yours.
U'm also trying to index documents(tabs, emails, files, notes, identities etc)
Organize them into semantically meaningful context or directory-like virtual trees (a photo of a nice kitchen may be indexed under /travel/barcelona and /arch/interieour/kitches) - trees are bitmaps - /travel/barcelona does a fast and cheap "travel" AND "barcelona", want to "zoom-out" you just go one directory up to /travel and see all documents tagged with travel)
You can use multiple timelines - extract that fancy md-converted en-wiki hf dataset into a wikipedia db dataset + timeline, tag your personal timeline as "personal" - wanna know the zeitgeist of your grandmothers birth date - search for it with timelines personal + wikipedia in layered mode and you'll get everything that happened or was happening during that time.
You can have long-running stateful query sessions and refine your searches dynamically - search for "winter" and get all documents with a winter scenery or mentioning winter - refine with "nice view" then "laptop" - citing a recent example[1]
Documents have relations that would be cumbersome to map in a virtual tree structure hence on top of the initial structure you can use graph edges(also powered by bitmaps - as most indexes are)
All vector queries always run on top of the candidate set you get by the bitmap/bitmap-based filter algebra hence searching through 100k+ docs is fast
Is the extracted content stored in a way that could be easily converted to markdown? Seems like this could be integrated incredibly well with obsidian as a way to automatically expand your knowledge base, as well as a better search for it.
The extracted content is stored and displayed as HTML, so converting it to markdown doesn't guarantee lossless transformation. But, the other direction works well: Hister can live track and import markdown files providing full text search and rendered previews for all your files.
I'm running hister locally, I see in the MCP docs that a third party can query the server, but is there a way to import data as well? I find that I often do research using pi, and would love to save the context to hister to query later on. The closest thing I see is the "hister import file" command, which is great, but inconvenient. If MCP allowed third parties to dump text data in, that would be ideal.
I tried this out this week and liked it but really wish this project had some form of auth. Opening the contents of every page you’ve ever visited, even to the local network, is not the best idea.
Hister supports token based, password based, and OIDC/OAuth authentications with optional multi-user handling. Details about user handling can be found here: https://hister.org/docs/user-handling
It also has a "public mode" where anyone can search the indexed content, but only authenticated users can add or modify it.
It seemed like the public mode was the default when I set it up. If so, that’s a fairly dangerous default as keeping a “clean” history with no secrets leaked seems neigh impossible.
I just finished setting this up yesterday and I'm kind of obsessed with it. I was previously a heavy user of Karakeep, but I hated forgetting to save something and losing it. I also think Hister's semantic search is a better solution than AI generated summaries and tags. Support for local docs is super cool too, I have it set up to index my org notes directory.
Really? The website, you mean? :( I thought to myself, "what a cute, well designed webpage!" when I saw it. Maybe with time this site will stand out like those Claudeslop ones.
I think that the tells, if they are in fact legitimate, are negligible at best or subtle at worst. It looks like the site uses Tailwind, so if AI was involved it wasn't generating the design from scratch. If it did then I'm impressed because usually it sucks. This is alright.
I love the idea overall, but something about browser extensions give me a bad taste. Am I overthinking it? I realize the extension is open source, of course.
This is so cool! For the time being I’m still locked into notion for my handwritten knowledge base, but I love this for incorporating external information.
For the semantic search is there any chunking/processing that happens with the content or do you need to be diligent about having a large embedding context (and/or small content)?
90 comments
[ 0.22 ms ] story [ 14.0 ms ] threadwould've been great with a more liberal license
Not in the near term. Right now I am focused on developing Hister rather than operating a hosted service. There are already plenty of centralized hosted search engines, so my longer term interest is in federation and distributed search. I want to make the core system mature first.
> would've been great with a more liberal license
It depends on how do you define liberal. =] I chose AGPLv3+ because I want Hister to remain free software and available to their users.
In fact many companies avoid AGPLv3 licensing for this reason.
> All it prevents is someone taking your work and making a business out of it.
Not really. I say licensing is moot now because of LLMs.
Immediately interested and will check it out, thank you! I've wanted a "search stuff you've seen online" tool for a long time, but everything seems to be research-oriented or "archive but don't search" or some weird combination that means it's nigh useless to me. I've got decades of bookmarks and archives and I've kinda been stuck grepping them at best (it's rare but I do sometimes want a page I saw once three years ago and I love having that option), while hoping someone would build something better.
One question if ya don't mind, while I explore: any chance of singlefile support? Content-extraction is useful in lots of situations (e.g. wallabag), but sometimes it fails and sometimes you really do want the page, relatively close to how it actually was. Singlefile does that much better than most, and it does so well enough (and manually-capable enough) that I don't feel any desire to switch to WARCs or similar.
> One question if ya don't mind, while I explore: any chance of singlefile support?
Yes, partially. Hister can already import HTML files created by SingleFile, but there is no direct integration yet. In the longer term, I would like the SingleFile extension to be able to send snapshots directly to Hister.
Overall I really like what I'm seeing, it ticks a lot of important boxes for me and it's pleasantly straightforward. Hopefully I'll find time to contribute!
Hister always stores the original material.
> That way you could also switch from an extracted view to a "full" view in the UI.
It isn't even needed, we just need a SingleFile specific extractor (an interface in Hister to parse specific page content and provide custom previews) that provides the full original HTML for the preview panel.
> Hopefully I'll find time to contribute!
I'd appreciate it. <3
Thank you again!
Would Hister support this basic workflow? I'd love to retire my own software.
The next phase was going to move to a recording proxy.
[0] https://github.com/canvas-ui/canvas/tree/main/apps/browser-e...
https://beepb00p.xyz/promnesia.html
I use Linkding for archiving links I've seen online and I love it's archive feature. But often enough it didn't work, until they added support for Singlefile from the browser.
I often find myself irritated because I read an article on my phone 6 months ago and the history is gone.
Would Hister be suitable for this? Can it index mbox files? Would it handle this amount of data? Does it have a search API so I can build an MCP server?
The indexer it uses (Bleve) can handle millions of records according to their docs, but sure that Hister would be the best choice for this task. I'd probably use Meilisearch (https://www.meilisearch.com/).
> Can it index mbox files?
Not yet.
> Does it have a search API so I can build an MCP server?
It has both search API and MCP server endpoints.
One thing at first look. Personally I think it would be very nice if it was possible to index different sites to different search index. So you could separate different stuff, like job, different projects you have, and other stuff and then last everything else etc.
So it would be possible to have like "profiles" you could easy choose from. So in the addon, when you click on the hister icon on the toolbar there would be a list of profiles that you could choose where to index to, but also a setting to choose which domains and sites that always should index to one seach index profil. And all sites that dont is added to a profile is added to the default index instead.
For me this would make it much more easy to find stuff and sort out what Im looking for (for me things get very "thing"/project based what I want to find), so having this would... 1 - become like an important filter. When you search you could easy mark one or more "profiles" which you would search from, and you remove alot of unwanted data automatically. 2 - It would also make it easy if the db gets big over time to remove data that is less important later on and that you dont need to be index anymore. 3 - It would also make it easier to backup only the most important search index data DB:s if the DB for the differnt seach index:s where in different folders/files. I guess the "default index" that index all sites could easy get big, but there it would be alot of not important data. So would be nice to be able to easy to use a backup program and only backup the only important search databases.
Academics usually take it to mean the Danube river instead, but in conspiracy/New Age contexts, the association with Hitler is prevalent, and mentioned in several pop culture works, including at least one feature film.
U'm also trying to index documents(tabs, emails, files, notes, identities etc) Organize them into semantically meaningful context or directory-like virtual trees (a photo of a nice kitchen may be indexed under /travel/barcelona and /arch/interieour/kitches) - trees are bitmaps - /travel/barcelona does a fast and cheap "travel" AND "barcelona", want to "zoom-out" you just go one directory up to /travel and see all documents tagged with travel) You can use multiple timelines - extract that fancy md-converted en-wiki hf dataset into a wikipedia db dataset + timeline, tag your personal timeline as "personal" - wanna know the zeitgeist of your grandmothers birth date - search for it with timelines personal + wikipedia in layered mode and you'll get everything that happened or was happening during that time. You can have long-running stateful query sessions and refine your searches dynamically - search for "winter" and get all documents with a winter scenery or mentioning winter - refine with "nice view" then "laptop" - citing a recent example[1] Documents have relations that would be cumbersome to map in a virtual tree structure hence on top of the initial structure you can use graph edges(also powered by bitmaps - as most indexes are) All vector queries always run on top of the candidate set you get by the bitmap/bitmap-based filter algebra hence searching through 100k+ docs is fast
Anyhow, let me stop here, thank you once again!
[0] https://github.com/canvas-ui/canvas-synapsd (sorry for the sloppy ai readme, no time to resurrect my old one with the updated APIs) [1] https://demo.cnvs.ai/pub/c/aks6zaf8
It also has a "public mode" where anyone can search the indexed content, but only authenticated users can add or modify it.
Also indexed data is persisted on a per-user basis, so you got this isolation and certainty that your searches will not be polluted by your family's
https://github.com/rumca-js/Internet-Places-Database
Also I maintain android app that can be used to search places.
https://f-droid.org/en/packages/io.github.rumcajs.offlineweb...
I believe hister you have to fill in with your data, right?
What tools or features would Hister need to support your complete search workflow?
Got enough Hitler in my life with Google...
It's a terrible choice of name
Unlike say Hipster with the descender on the p.
Webtm.io
All open source and small enough to deploy. I deploy to cf webworkers so it’s the only place it’s tested.
One cool thing is we work on iOS, chrome and friends, Firefox and pretty much everywhere. We do require you bring your own LLM though.
Planning to contribute significant improvements to the vector search side here.
For the semantic search is there any chunking/processing that happens with the content or do you need to be diligent about having a large embedding context (and/or small content)?
- let us say I want to index every blog ever listed on HN
- should be a small subset of the 400 billion pages out there on the internet no?
- First I need to gather data, what do you use to load so many webpages rapidly? asyncio with aiohttp in python? are there better options?
- how do you handle proxies? rotation? are there libraries you recommend for this?
- what about pages that use cloudflare? or block your request
- what are the filetypes you collect? only html or media as well?
- where and in what format do you store all these collected files? flat file storage? duckdb? postgres? hstore? something else?
- what is the frequency at which you refresh each page? once a day? once a week? something else?
- what kind of pre-processing do you use on the collected data? remove extra spaces? special characters? some kind of complex regex pipeline? LLM?
- how do you match the incoming query with processed data? simple text matching? regex? vector embedding match? something else?
Others could be answered by examining the source code, as it is open source.
It's good etiquette to check existing documentation and resources before badgering OSS project teams with a lengthy list of questions.