Looks fun reminds me of Google Desktop search from way back in the day - you know back in the days of google also selling a search appliance for enterprise data centers.
No, they didn't. Back in the day Google sold 'Google-In-A-Box' which was a rack mounted hardware device that you installed into your company's server rack (either on-prem or in a hosted data centre) and once configured it crawled all your internal resources and presented an internal search engine with all the power of Google search.
Although a good idea in principle, I found it cumbersome to configure and it only (at the time) supported ftp and http based resources - if your company made heavy use of Windows file shares you couldn't index them without also serving them over ftp or http.
I believe later versions improved on that though, but by then it was a dead product, and Google discontinued it.
Interesting idea however I can't see many practical uses for this ... how often do you really need to search through the text of something you browsed some time ago and would it really offset the disk-space cost? I personally would have benefited from this on a few occasions however SQL+title-keywords+timestamps were effective enough and it's a rare occurrence. And if you need to archive something, why do it locally when there are projects like the Wayback machine (centralized is an argument against it I suppose), or why do it for everything?
Everyone is different. Sometimes even just saving someone that inconvenience and time and stress in that edge case where like you have this bookmark that you know that you had or this page that you visited but you can't find it again through a search engine, sometimes just saving people the headache of going through that is a really valuable thing like even if it only occurs a couple of times per year.
Personally I couldn't understand why people were so into this project when I created it. I didn't understand the potential uses when I created it. I've liked that people have shown me, and also that I've had to market this way less than my other products.
I created it as like you know a technical proof of concept kind of hack because I wanted to see if it was possible and then I kind of discovered that there were all these like archive nuts who were really into archiving. And then over time I started to think well sometimes I've got a bookmark but I can't remember how to find it and I couldn't find it in in the main search engine but I knew that I bookmarked it somewhere but I can't identify it from the title search. So having a full text search over the page content like a personal local search engine I thought it was going to be useful. But the main cool thing I think is the fact that you know you can have pages running as if they're on the internet when you're offline. So on the plane say you can have all of this you know mdn stuff archived all of these useful resources useful documents and then you can be running on the plane you don't have to use the Wi-Fi but you can have this like offline internet so you can keep working.
That's me and my perspective maybe let me finish with the words of someone else who commented here just a little while ago but then deleted the comment, but I thought it was a really good perspective so I'll share it below
> I thought the same thing 10 years ago. I have(had?) many thousands of bookmarks collected over 10 years. All of the most interesting things I ever read. Imagine my disappointment when I eventually realized most of them are now gone. Site is gone, URL is changed, videos taken down (so many of them it's amazing), websites weren't properly archived etc.
> I tried to curl all my bookmarks for a half-assed solution but dynamic loading and unusable content presentation without running all js and saving dependencies, timestamps, replies etc makes it impossible. Even worse, it's impossible to discover something relevant when there is no search function. What you get is the page title and a URL. Usually both of them useless. I tried categorizing them and adding comments but that was a lot of work and led to me deleting stuff that are hard to categorize. I'm not a professional programmer and I'm definitely not fully aware of how modern websites work (particularly on the front end).
> But if someone provided a locally run telemetry-free open-source (so it's auditable) searchable archive-as-you-browse tool that legitimately works reliably and with some kind of hard (technical) guarantee that it won't break with the next browser update, I'd pay good money for it.
Desktop file search sucks. But I think you mean you don't have trouble finding pages you've visited before just using a search engine.
That's good but that's not everyone's experience. Sometimes you're like oh what's that page but you just can't find a way back to it because you didn't discover it through a search engine originally.
You discovered it through some recommended article on some other website or you discovered it through social media or someone sent you a link. And you can't remember where you saw it but you know you saw it and there was something in there that you wanted, but you can't find it through search because you've never actually searched for it before, you tried to search it now but it didn't work, and sometimes that happens. That's a real thing that happens.
If you think about it around the world like if this happens to you say four times a year and there's you know 3 billion people online that's happening a lot of times (hundreds of millions of times) every day and you could save a lot of time for people if there was some solution to this.
Your experience is valid, and I get if you've never experienced that and that's cool, that's a good thing--that's great!--but that's not everyone's experience.
And there's a lot of cases for me where this has happened and I found it really annoying you know like I couldn't find something that I know that I visited and it's really annoying because you try and find it and it takes longer than you think it should, and sometimes you can't find it at all it's like the word that's on the tip of your tongue, but you just don't get it. I definitely think it is a pain point, that is a market opportunity.
Why build it with an intimate dependency on Chrome, and not as a general HTTP/HTTPS proxy?
Not that it doesn't look neat, sure. But I do a lot of browsing from my phone, and I've just been spending some money and considering spending more on even a partial archiving solution for that.
If this could support that use case, you'd have a chance at a paying customer for life. As it is, you've already told me you don't want me, or anyone with a similar use case, as a customer at any price. I assume there has to be some engineering justification for a tradeoff like that, but I admit I can't fathom what it might be.
> Not that it doesn't look neat, sure. But I do a lot of browsing from my phone, and I've just been spending some money and considering spending more on even a partial archiving solution for that.
DK has this thing called "bookmark mode" where basically like you know it will only archive the stuff you bookmark. And because your bookmarks are synced from your Chrome mobile browsing to your Chrome desktop the next time you're on your desktop and running DK there it will detect those new bookmarks from your mobile browsing and it will archive and index them for you.
> If this could support that use case, you'd have a chance at a paying customer for life. As it is, you've already told me you don't want me, or anyone with a similar use case, as a customer at any price.
So there's plenty of space for you in the current system! :)
> Why build it with an intimate dependency on Chrome, and not as a general HTTP/HTTPS proxy?
I thought about doing that (not the general HTTP proxy idea but maybe I should look into that) I thought about doing that by somehow hacking a way to get the remote debugging port to work on the device (Android or iPhone) versions of Chrome. Basically I think the only way that it opens is if you have your phone plugged in to the Android debugger and hooked up the USB debugging or something like that so I can set it and it may be possible with like a little kind of USB type of dongle stuck in the bottom of your phone or something they may be a way to sort of, while the dongle is stuck in, you can archive your browser history... but I thought that's probably quite a number of steps from here to there.
Well yeah, that's the thing about a proxy - ~every HTTP/HTTPS user agent supports them, and neither you nor your users need to mess around with Chrome-specific debugging stuff. Just set a proxy up in whatever browser, and all the traffic you want to archive goes through it with no extra effort.
Fair warning, HTTPS proxying with packet inspection can be complicated since the proxy needs to terminate TLS in order to see the content; it's exactly like a MITM attack, but in this case performed among friends. This can be problematic in an environment where certificate pinning also exists, but that can be overcome for browsers at least if the root of the proxy's MITM cert is trusted in the system's or browser's certificate store. This is trivial as far as I know on all desktop OSes and on Android; I think you can only do it at the system level in iOS via an MDM profile, and I'm not sure how complicated those are to generate but iirc Boxen and similar OSS Mac config management tools do it, so may serve as a useful reference on the mechanics.
I think that's the reason I don't want to do a proxy because I don't like the idea of terminating TLS.
I think it introduces some sort of vulnerability plus as you say technically it can be weird. The advantage of the current system is the users don't have to mess around with browser stuff at all.
I just don't like that technical solutions space and solving problems in that right now anyway as much as I do the browser side. So I don't think I'll do that for now. But it's good to have something in the future I think if this product really scaled I could definitely look at the proxy solution as a way to sort of take it to maximum scale for you know the maximum number of users.
But I think the way that it currently is would let it achieve some sort of product market fit and scale within a niche so I could see if it was really worth growing further. If it did grow then I could maybe use other solutions like your proxy one to scale it to a high number of people.
There's actually another solution though which involves remote browsing. So I could bundle it incorporate it it's cloud surface with another product that I have browser box and so people do all their browsing on the cloud and we can do all the archiving on the cloud and you can access that where my browser from your mobile device so anything you browse through that will be archived for you and then we could sync it so you could like download those archives that you know various points or search through them online or something like that.
Hopefully for now the bookmark mode can satisfy your use case but if not I understand and I feel a bit sorry for you if that's the case because I would like to think that you know you could benefit from this but you know we're not able to please everybody and while I feel sad for you about that that's just the way that it is, we can only deliver to a niche right now. So thank you for your technical information and sharing about your use case anyway I really appreciate that so I hope you give the product a try and you know let me know what you think.
Replying to an (inexplicably, as it was high quality) deleted comment:
> I thought the same thing 10 years ago. I have(had?) many thousands of bookmarks collected over 10 years. All of the most interesting things I ever read. Imagine my disappointment when I eventually realized most of them are now gone. Site is gone, URL is changed, videos taken down (so many of them it's amazing), websites weren't properly archived etc.
> I tried to curl all my bookmarks for a half-assed solution but dynamic loading and unusable content presentation without running all js and saving dependencies, timestamps, replies etc makes it impossible. Even worse, it's impossible to discover something relevant when there is no search function. What you get is the page title and a URL. Usually both of them useless. I tried categorizing them and adding comments but that was a lot of work and led to me deleting stuff that are hard to categorize. I'm not a professional programmer and I'm definitely not fully aware of how modern websites work (particularly on the front end).
> But if someone provided a locally run telemetry-free open-source (so it's auditable) searchable archive-as-you-browse tool that legitimately works reliably and with some kind of hard (technical) guarantee that it won't break with the next browser update, I'd pay good money for it.
This is basically what DK is except, I have no guarantee as it depends on Chrome Remote Debugging Protocol (aka DevTools API.).
I could hack you a guarantee by packaging it with a fixed version of Chrome (thank you for this idea I have never thought that that would be something people would give it their purchase on but thank you very much for informing me of this it's very valuable to know thank you) which would guarantee that all your sites archived up to the version of Chrome used to archive them would continue to work into the future because they worked with that version of Chrome in the first place...and then you would have access to that version of Chrome to run the archive.
Aside from that we can have the hand waving heuristic argument (which is not a hard technical guarantee) that Chrome has invested a lot in puppeteer and the whole browser automation API thing is blowing up and it's very big among developers and supported by a variety of browser vendors (and on track to become a bona fide w3c standard if it's not already the web driver or web devital standard I mix up the name because there was a historical one as well) and there's lots of forks of browsers so the idea that some part of the puppeteer API (CRDP API), as in the ability to intercept requests, will go away at some point in the future seems kind of unlikely but it's not something you can guarantee.
Although in the limit I think the idea of packaging it with a historical version of Chrome works because in the limit your archive size is going to be larger than the 60 to 100 MB size of the Chrome binary.
20 comments
[ 3.3 ms ] story [ 29.9 ms ] threadAlthough a good idea in principle, I found it cumbersome to configure and it only (at the time) supported ftp and http based resources - if your company made heavy use of Windows file shares you couldn't index them without also serving them over ftp or http.
I believe later versions improved on that though, but by then it was a dead product, and Google discontinued it.
Personally I couldn't understand why people were so into this project when I created it. I didn't understand the potential uses when I created it. I've liked that people have shown me, and also that I've had to market this way less than my other products.
I created it as like you know a technical proof of concept kind of hack because I wanted to see if it was possible and then I kind of discovered that there were all these like archive nuts who were really into archiving. And then over time I started to think well sometimes I've got a bookmark but I can't remember how to find it and I couldn't find it in in the main search engine but I knew that I bookmarked it somewhere but I can't identify it from the title search. So having a full text search over the page content like a personal local search engine I thought it was going to be useful. But the main cool thing I think is the fact that you know you can have pages running as if they're on the internet when you're offline. So on the plane say you can have all of this you know mdn stuff archived all of these useful resources useful documents and then you can be running on the plane you don't have to use the Wi-Fi but you can have this like offline internet so you can keep working.
That's me and my perspective maybe let me finish with the words of someone else who commented here just a little while ago but then deleted the comment, but I thought it was a really good perspective so I'll share it below
> I thought the same thing 10 years ago. I have(had?) many thousands of bookmarks collected over 10 years. All of the most interesting things I ever read. Imagine my disappointment when I eventually realized most of them are now gone. Site is gone, URL is changed, videos taken down (so many of them it's amazing), websites weren't properly archived etc.
> I tried to curl all my bookmarks for a half-assed solution but dynamic loading and unusable content presentation without running all js and saving dependencies, timestamps, replies etc makes it impossible. Even worse, it's impossible to discover something relevant when there is no search function. What you get is the page title and a URL. Usually both of them useless. I tried categorizing them and adding comments but that was a lot of work and led to me deleting stuff that are hard to categorize. I'm not a professional programmer and I'm definitely not fully aware of how modern websites work (particularly on the front end).
> But if someone provided a locally run telemetry-free open-source (so it's auditable) searchable archive-as-you-browse tool that legitimately works reliably and with some kind of hard (technical) guarantee that it won't break with the next browser update, I'd pay good money for it.
That's good but that's not everyone's experience. Sometimes you're like oh what's that page but you just can't find a way back to it because you didn't discover it through a search engine originally.
You discovered it through some recommended article on some other website or you discovered it through social media or someone sent you a link. And you can't remember where you saw it but you know you saw it and there was something in there that you wanted, but you can't find it through search because you've never actually searched for it before, you tried to search it now but it didn't work, and sometimes that happens. That's a real thing that happens.
If you think about it around the world like if this happens to you say four times a year and there's you know 3 billion people online that's happening a lot of times (hundreds of millions of times) every day and you could save a lot of time for people if there was some solution to this.
Your experience is valid, and I get if you've never experienced that and that's cool, that's a good thing--that's great!--but that's not everyone's experience.
And there's a lot of cases for me where this has happened and I found it really annoying you know like I couldn't find something that I know that I visited and it's really annoying because you try and find it and it takes longer than you think it should, and sometimes you can't find it at all it's like the word that's on the tip of your tongue, but you just don't get it. I definitely think it is a pain point, that is a market opportunity.
https://github.com/beenotung/personal-search-engine
https://github.com/hamsterbase/hamsterbase
Render and upload mhtml(chrome)、webarchive(safari) website.
full text searching is WIP
Not that it doesn't look neat, sure. But I do a lot of browsing from my phone, and I've just been spending some money and considering spending more on even a partial archiving solution for that.
If this could support that use case, you'd have a chance at a paying customer for life. As it is, you've already told me you don't want me, or anyone with a similar use case, as a customer at any price. I assume there has to be some engineering justification for a tradeoff like that, but I admit I can't fathom what it might be.
DK has this thing called "bookmark mode" where basically like you know it will only archive the stuff you bookmark. And because your bookmarks are synced from your Chrome mobile browsing to your Chrome desktop the next time you're on your desktop and running DK there it will detect those new bookmarks from your mobile browsing and it will archive and index them for you.
> If this could support that use case, you'd have a chance at a paying customer for life. As it is, you've already told me you don't want me, or anyone with a similar use case, as a customer at any price.
So there's plenty of space for you in the current system! :)
> Why build it with an intimate dependency on Chrome, and not as a general HTTP/HTTPS proxy?
I thought about doing that (not the general HTTP proxy idea but maybe I should look into that) I thought about doing that by somehow hacking a way to get the remote debugging port to work on the device (Android or iPhone) versions of Chrome. Basically I think the only way that it opens is if you have your phone plugged in to the Android debugger and hooked up the USB debugging or something like that so I can set it and it may be possible with like a little kind of USB type of dongle stuck in the bottom of your phone or something they may be a way to sort of, while the dongle is stuck in, you can archive your browser history... but I thought that's probably quite a number of steps from here to there.
Fair warning, HTTPS proxying with packet inspection can be complicated since the proxy needs to terminate TLS in order to see the content; it's exactly like a MITM attack, but in this case performed among friends. This can be problematic in an environment where certificate pinning also exists, but that can be overcome for browsers at least if the root of the proxy's MITM cert is trusted in the system's or browser's certificate store. This is trivial as far as I know on all desktop OSes and on Android; I think you can only do it at the system level in iOS via an MDM profile, and I'm not sure how complicated those are to generate but iirc Boxen and similar OSS Mac config management tools do it, so may serve as a useful reference on the mechanics.
I think it introduces some sort of vulnerability plus as you say technically it can be weird. The advantage of the current system is the users don't have to mess around with browser stuff at all.
I just don't like that technical solutions space and solving problems in that right now anyway as much as I do the browser side. So I don't think I'll do that for now. But it's good to have something in the future I think if this product really scaled I could definitely look at the proxy solution as a way to sort of take it to maximum scale for you know the maximum number of users.
But I think the way that it currently is would let it achieve some sort of product market fit and scale within a niche so I could see if it was really worth growing further. If it did grow then I could maybe use other solutions like your proxy one to scale it to a high number of people.
There's actually another solution though which involves remote browsing. So I could bundle it incorporate it it's cloud surface with another product that I have browser box and so people do all their browsing on the cloud and we can do all the archiving on the cloud and you can access that where my browser from your mobile device so anything you browse through that will be archived for you and then we could sync it so you could like download those archives that you know various points or search through them online or something like that.
Hopefully for now the bookmark mode can satisfy your use case but if not I understand and I feel a bit sorry for you if that's the case because I would like to think that you know you could benefit from this but you know we're not able to please everybody and while I feel sad for you about that that's just the way that it is, we can only deliver to a niche right now. So thank you for your technical information and sharing about your use case anyway I really appreciate that so I hope you give the product a try and you know let me know what you think.
> I thought the same thing 10 years ago. I have(had?) many thousands of bookmarks collected over 10 years. All of the most interesting things I ever read. Imagine my disappointment when I eventually realized most of them are now gone. Site is gone, URL is changed, videos taken down (so many of them it's amazing), websites weren't properly archived etc.
> I tried to curl all my bookmarks for a half-assed solution but dynamic loading and unusable content presentation without running all js and saving dependencies, timestamps, replies etc makes it impossible. Even worse, it's impossible to discover something relevant when there is no search function. What you get is the page title and a URL. Usually both of them useless. I tried categorizing them and adding comments but that was a lot of work and led to me deleting stuff that are hard to categorize. I'm not a professional programmer and I'm definitely not fully aware of how modern websites work (particularly on the front end).
> But if someone provided a locally run telemetry-free open-source (so it's auditable) searchable archive-as-you-browse tool that legitimately works reliably and with some kind of hard (technical) guarantee that it won't break with the next browser update, I'd pay good money for it.
This is basically what DK is except, I have no guarantee as it depends on Chrome Remote Debugging Protocol (aka DevTools API.).
I could hack you a guarantee by packaging it with a fixed version of Chrome (thank you for this idea I have never thought that that would be something people would give it their purchase on but thank you very much for informing me of this it's very valuable to know thank you) which would guarantee that all your sites archived up to the version of Chrome used to archive them would continue to work into the future because they worked with that version of Chrome in the first place...and then you would have access to that version of Chrome to run the archive.
Aside from that we can have the hand waving heuristic argument (which is not a hard technical guarantee) that Chrome has invested a lot in puppeteer and the whole browser automation API thing is blowing up and it's very big among developers and supported by a variety of browser vendors (and on track to become a bona fide w3c standard if it's not already the web driver or web devital standard I mix up the name because there was a historical one as well) and there's lots of forks of browsers so the idea that some part of the puppeteer API (CRDP API), as in the ability to intercept requests, will go away at some point in the future seems kind of unlikely but it's not something you can guarantee.
Although in the limit I think the idea of packaging it with a historical version of Chrome works because in the limit your archive size is going to be larger than the 60 to 100 MB size of the Chrome binary.
https://www.browserparrot.com/
https://github.com/ArchiveBox/ArchiveBox
HN thread: https://news.ycombinator.com/item?id=27842740