Hey HN, we're Ian and Nik and we're a bit obsessed with data. Specifically, being able to own your own data and make it useful.
We're rethinking how we interact with and manage all the data in our lives. Data is all over the place now (computer, mobile, multiple cloud providers...) and it's more than a bit unwieldy. Not only is it unwieldy, you can also lose access to your data at the drop of a hat due to an outage, a policy change, or a bug. This doesn't sit well with us.
Of course, to really tackle this problem will be the work of years, so we're starting with something more manageable. We've created a tool to aggregate all your browser history and make it full-text searchable. If you've read something online, you should be able to find it again—quickly.
There's plenty of prior art in the search-your-own-history space, so why might you care? When we tried to find an existing product to fit our needs we were unable to find something that checks all of these boxes:
- All history, from all browsers.
- Fast. It should feel more like autocomplete than like web search.
- Completely local. No need to trust someone's server.
- No user input required. I.e. no need to bookmark. Store everything.
- No limits. Don't delete my history unless I say so (browsers will delete your history after a few months, 4 months for Chrome).
I'm sure we've missed a few though. How do you keep track of everything you've read online? Any workflows or killer hacks you've discovered?
This is great! Can't wait to give it a try. Waiting for the Windows version though.
Does the electron GUI need to be open for it to fetch the data, or can it do that as a background service as well, for example if one just wanted an always up to date persistory.db to use directly? For example an api server and a custom browser extension consuming it, or are these also being considered for browserparrot
Heh, ah yes windows. Definitely want to support this. Will be a bigger lift than Linux, but definitely something we'd like to support (we both have windows boxes too so there's a bit of skin in the game).
Regarding the GUI, good question. The GUI does _not_ need to be open. There's a background process that periodically copies new history from the various sqlite databases into the BrowserParrot db.
In Activity Monitor this process appears as "core." I know, terribly ambiguous name. It was originally named for the Clojure namespace where that bit of code lives, but renaming the process to make it more clear what's running is on our backlog.
Seems like a cool project! I use my browsing history a lot to find certain things again and it’s always frustrated me that it’s slow and horrible at finding what I need.
But please release a linux version ASAP :)
I’m also not sure if I like that is’s always running in the background, not because of privacy but because it just seems a bit unnecessary - something that I use maybe a few times in a week and that’s nothing more than a search engine shouldn’t be using system resources all the time
Thanks for the feedback. Shouldn't be a big lift to add linux support (it's electron after all). We just wanted to get this out the door asap and hopefully receive some community pressure to add a linux build step
Do you guys plan on providing similar client-side search tools?
I’d love to be able to search my emails, notes, or third party services locally. Neeva is a search engine that provides this capability, but you need to grant them access to all your third party services so they can do the processing on their servers. YUCK!
Thanks! Yes, absolutely that's the plan. We too _really_ want to search over everything and have it all stored locally (or remotely on user-controlled VPS). This was our original idea, but we decided we should launch something and hopefully get feedback or just meet other people who care about this problem space.
So yes, we want to search everything locally. For me personally I find Notion, Roam and _all_ chat apps to be particular pain points.
Email too, although I've found that email search often delivers what I'm looking for so the pain point is much lower.
Thanks. We first grab all browser history from your existing browsers and slurp it into a single sqlite database. We do this sequentially but since it's sqlite-to-sqlite it's usually very qucik.
For the full text portion we send out a request to the URL for the full HTML, distill it (similar to "reader" mode in some browsers) and add that distilled content to the full text index.
The full-text generation happens with some concurrency but we intentionally didn't want to spam anyone's server so it's limited and takes a while to populate.
Yup! Specifically we extract the full text for each website 8 URLs at a time right now, but you can still search over the titles and URLs while the full text is being collected
Hey wuyishan, right now it isn’t able to retrieve the full text of content that’s behind login walls, but we are looking at ways to allow it to handle those sites as well
Very interesting tool, just downloaded and started it. I'm curious: You said that it simply accesses the browser history from the browser's SQLite databases. How come that this app can even do it without asking any permissions on macOS? I thought that apps on macOS are sandboxed in a way.
Yes, for sandboxed apps distributed via the Mac app store that is true. For apps distributed externally, such as ours, it is optional. No additional permissions are required to access ~/Library/ApplicationSupport/* which is where most history databases live.
The exception is Safari, which lives at ~/Library/Safari and requires Full Disk Access, granted via System Preferences, in order to be read.
Thanks for answering. Some suggestions after using it for 5 minutes: I apparently have close to 100,000 URLs from Firefox waiting to be indexed.
- Show the date range from the earliest bookmark to the latest
- Let me open the downloaded content for a URL instead of redirecting me to the current online version.
I'm also wondering if the downloaded "extract" is somewhat complete. For example, reader mode doesn't really work on HN or Reddit, but the individual comments could be what I'm looking for. Anyways, I'm keeping an eye on this tool. Not sure if it really fits into my workflow, but maybe it will :)
It should indeed work for HN posts (this was something we specifically wanted). For example, here's this thread at the time of indexing: https://pastebin.com/shx8miZk
You can also access all this yourself within the datasource_browsing_history_fts table of ~/.config/persistory/persistory.db (so named for legacy reasons...)
Regarding the point about accessing the content directly, this is something we're considering. In our testing there's a bit of tension between ideal text for searching (limited formatting) and ideal text for rendering (probably markdown). However, having the local db serve as your own personal archive.org is one of our secondary goals.
Hey WA, Nik here, thanks for the suggestions. Showing the date ranges is a good idea, and we’d also like to make what’s shown in the search results more configurable since we have some potentially interesting metadata there (date last visited, date we downloaded the full text etc.)
We do keep a fairly complete copy of the URL’s contents in the db, though we don’t currently preserve all the html or store embedded content such as images, so some formatting may be lost. Adding a method to view the stored version from the GUI is definitely on the todo list though, we want that feature too!
Being flexible and able to fit into a lot of workflows is high priority for us, if you have more thoughts on how it could more easily fit into your workflow please feel free to let us know here or on the discord
Just tried this out, looks great, I've wanted something like this for a long time!
I often use Cmd-tab to switch between apps on the mac, and I noticed that this search pane shows up in the tab switcher, which can get in the way a bit.
* In vscode, cmd-tab to chrome
* cmd-E to search for something
* cmd-E to close the search (goes back to chrome)
* cmd-tab to go back to vscode (but this goes to browserparrot)
Strongly agree. Something like this is on our todo list. Our thinking is an optionally-open, in-the-dock UI to manage settings and a floating non-dock search bar for quick access. Perhaps with a menu bar item to make it clear something is still running.
So the dock app could be closed when not in use, which is would probably be most of the time.
Hey shmapf, Nik here. I know what you mean, Alfred is a great tool.
We’ll be adding an option to run in the taskbar rather than as a full-fledged app-with-dock-icon soon to help with this —- being able to do everything fast and from the keyboard (including switching apps) is high priority for us
Amazing work. Shocked that in 2021 we do not have more such tools. One thing -- please give a way to change the search hotkey from Cmd-E to something else.
I need that for PyCharm :)
We haven’t added a UI for it yet, but you can actually change the hotkey by editing ~/.config/persistory/config.edn and changing the :quick_search entry (CommandOrControl+E by default) to your desired keyboard binding.
This functionality comes from electron, more info on the hotkey syntax in their docs [0]. Definitely don’t want to be getting in the way of your PyCharming!
26 comments
[ 4.2 ms ] story [ 77.0 ms ] threadWe're rethinking how we interact with and manage all the data in our lives. Data is all over the place now (computer, mobile, multiple cloud providers...) and it's more than a bit unwieldy. Not only is it unwieldy, you can also lose access to your data at the drop of a hat due to an outage, a policy change, or a bug. This doesn't sit well with us.
Of course, to really tackle this problem will be the work of years, so we're starting with something more manageable. We've created a tool to aggregate all your browser history and make it full-text searchable. If you've read something online, you should be able to find it again—quickly.
There's plenty of prior art in the search-your-own-history space, so why might you care? When we tried to find an existing product to fit our needs we were unable to find something that checks all of these boxes:
- All history, from all browsers.
- Fast. It should feel more like autocomplete than like web search.
- Completely local. No need to trust someone's server.
- No user input required. I.e. no need to bookmark. Store everything.
- No limits. Don't delete my history unless I say so (browsers will delete your history after a few months, 4 months for Chrome).
I'm sure we've missed a few though. How do you keep track of everything you've read online? Any workflows or killer hacks you've discovered?
edit: formatting
Does the electron GUI need to be open for it to fetch the data, or can it do that as a background service as well, for example if one just wanted an always up to date persistory.db to use directly? For example an api server and a custom browser extension consuming it, or are these also being considered for browserparrot
Regarding the GUI, good question. The GUI does _not_ need to be open. There's a background process that periodically copies new history from the various sqlite databases into the BrowserParrot db.
In Activity Monitor this process appears as "core." I know, terribly ambiguous name. It was originally named for the Clojure namespace where that bit of code lives, but renaming the process to make it more clear what's running is on our backlog.
Can also find the process via `lsof -i :44444`.
But please release a linux version ASAP :)
I’m also not sure if I like that is’s always running in the background, not because of privacy but because it just seems a bit unnecessary - something that I use maybe a few times in a week and that’s nothing more than a search engine shouldn’t be using system resources all the time
Do you guys plan on providing similar client-side search tools?
I’d love to be able to search my emails, notes, or third party services locally. Neeva is a search engine that provides this capability, but you need to grant them access to all your third party services so they can do the processing on their servers. YUCK!
So yes, we want to search everything locally. For me personally I find Notion, Roam and _all_ chat apps to be particular pain points.
Email too, although I've found that email search often delivers what I'm looking for so the pain point is much lower.
Is the idea to crawl locally? How do you crawl - sequentially or in parallel?
For the full text portion we send out a request to the URL for the full HTML, distill it (similar to "reader" mode in some browsers) and add that distilled content to the full text index.
The full-text generation happens with some concurrency but we intentionally didn't want to spam anyone's server so it's limited and takes a while to populate.
The exception is Safari, which lives at ~/Library/Safari and requires Full Disk Access, granted via System Preferences, in order to be read.
- Show the date range from the earliest bookmark to the latest
- Let me open the downloaded content for a URL instead of redirecting me to the current online version.
I'm also wondering if the downloaded "extract" is somewhat complete. For example, reader mode doesn't really work on HN or Reddit, but the individual comments could be what I'm looking for. Anyways, I'm keeping an eye on this tool. Not sure if it really fits into my workflow, but maybe it will :)
It should indeed work for HN posts (this was something we specifically wanted). For example, here's this thread at the time of indexing: https://pastebin.com/shx8miZk
You can also access all this yourself within the datasource_browsing_history_fts table of ~/.config/persistory/persistory.db (so named for legacy reasons...)
Regarding the point about accessing the content directly, this is something we're considering. In our testing there's a bit of tension between ideal text for searching (limited formatting) and ideal text for rendering (probably markdown). However, having the local db serve as your own personal archive.org is one of our secondary goals.
Edit: added a bit more context
We do keep a fairly complete copy of the URL’s contents in the db, though we don’t currently preserve all the html or store embedded content such as images, so some formatting may be lost. Adding a method to view the stored version from the GUI is definitely on the todo list though, we want that feature too!
Being flexible and able to fit into a lot of workflows is high priority for us, if you have more thoughts on how it could more easily fit into your workflow please feel free to let us know here or on the discord
I often use Cmd-tab to switch between apps on the mac, and I noticed that this search pane shows up in the tab switcher, which can get in the way a bit.
* In vscode, cmd-tab to chrome
* cmd-E to search for something
* cmd-E to close the search (goes back to chrome)
* cmd-tab to go back to vscode (but this goes to browserparrot)
Alfred, a similar search app doesn't do this, and I quite like that behaviour. I found out how to change this, as a user, here, but it would be nice if it was done by default: https://apple.stackexchange.com/questions/92004/is-there-a-w...
Strongly agree. Something like this is on our todo list. Our thinking is an optionally-open, in-the-dock UI to manage settings and a floating non-dock search bar for quick access. Perhaps with a menu bar item to make it clear something is still running.
So the dock app could be closed when not in use, which is would probably be most of the time.
We’ll be adding an option to run in the taskbar rather than as a full-fledged app-with-dock-icon soon to help with this —- being able to do everything fast and from the keyboard (including switching apps) is high priority for us
We haven’t added a UI for it yet, but you can actually change the hotkey by editing ~/.config/persistory/config.edn and changing the :quick_search entry (CommandOrControl+E by default) to your desired keyboard binding.
This functionality comes from electron, more info on the hotkey syntax in their docs [0]. Definitely don’t want to be getting in the way of your PyCharming!
[0]: https://www.electronjs.org/docs/api/accelerator#available-mo...
Not sure, but I think they only search pages that you explicitly bookmark, unlike BrowserParrot