Ask HN: How do you save and browse external interesting URLs?

43 points by bluewalt ↗ HN
As a curious developer, my knowledge is scattered between many external resources I consumed and want to keep at my fingertips: blog posts I read, Youtube videos I watched, Stack Overflow answers I read, Github repos I follow, etc. My knowledge is NOT the notes I took, but these external resources I consumed and loved.

But over time, I forget. I don't know what I know, and as soon as I need something like, I google it. For example, it could be the 10th time I google "efficient logging with Python". I may come across a link I already clicked, or not.

To me, it would be much more efficient to be able to search among all my external resources I already read and decided to keep, because it is limited to quality contents that I have already filtered, and that I already read, so that memory will activate when I read it another time.

At that point, you could tell me to use bookmarks. And it's what I do. Then 6 months later, I end up with 200 bookmarks I will not sort. And even if they were sorted, I will be too slow to find something in them with no tagging, I and I would use Google anyway.

In a ideal world, It would be easy to save and tag external resources (one click from the browser), and then, browse and find them back easily.

Do you have this feeling too, or it's just me? If so, what do you use for this?

68 comments

[ 3.2 ms ] story [ 144 ms ] thread
This is actually the reason why I stated a newsletter (weeklyrobotics.com). I was finding things on the Internet that were interesting, but then I could never find them again. I discovered that limiting search to my own domain made it much easier.

I’m using raindrop.io to save all the bookmarks in one place, and then while working on them I go through the top of the unsorted bookmarks, and select the ones I will feature in the newsletter.

This still doesn’t solve the problem of heaps of unsorted links (over 1.4k after around 3 years).

If I was to do something that you describe, I would probably look into dedicating some time into organizing your bookmarks, and if taking it to the next level then I would consider using TriliumNotes for making notes and categorizing the knowledge.

Thanks! I took a quick look to Raindrop.io, it seems powerful and full of features. At this point I'm wondering why they did not solve the problem of unsorted links. Since there is a browser extension, a simple solution would be to allow user to add optional tags when the user bookmarks the link.
When you add a link to raindrop.io you can add it to any of your folders. Tags are yet another thing they support but I don’t use them extensively.
I use the Google Keep browser extension.

When I click on the extension's icon (or right-click > Save to Keep), a draft note pops up with the page link in the body of the note, as well as any text selected on the page prior to clicking the icon.

From there I can add more text to the body, insert a title, and add or select a tag. Once the note is in Keep more features are available, such as attaching an image or archiving the note to move it out of the way. And the note is accessible and searchable from any device.

One solution is to continuously record web history and then just search it. The advantage is:

1. you often don't know what resources you will really "value" in the future, so no more "to save or not to save, this is the question"

2. tagging, to be effective, requires discipline (thinking about then sticking to an agile system). So, we just replace it with search, preferably NLP/AI (so you don't have to remember the exact keywords)

Apps do exist, from the expensive [1] to the experimental [2].

[1] https://heyday.xyz/

[2] https://github.com/karlicoss/promnesia

I use a system of personal knowledge management (PKM), which I've posted about before:

1. Capture: every interesting idea that I think up or read is immediately stored in Google Keep (on mobile or laptop). It can be very rough at this point, the goal is simply to not forget.

2. Transcribe & Organize: every weekend, I go through the notes I accumulated during the week. It tends to be between 10 and 30 notes. Sometimes the note is "read this article" or "catch up on all newsletters", so understanding a single note can take over an hour. On some tough weekends the process takes an entire day, but that is invariably a day where I feel like I learned a ton. Once the note is cleaned up (transcribed), I feel like I understand it. At this point I rarely forget it - it has been absorbed into my brain. The final step here is "categorizing" the note. I classify it using OneNote with tabs like "Clinical psychology" (nested under "Psychology") or "Investment management" (nested under "Finance") or "Math" or "Physics". This way, in the future, I don't have a million notes scattered around, but one clear place I know where to look. On average, this process takes 2-4 hours per weekend. I never accumulate bookmarks, Google Keep notes or unread emails more than a week to prevent existential dread.

3. Revisit: generally, people recommend you revisit your notes from time to time. I almost never do this. But if I ever am thinking about "Marketing" or "Sociology", I have an immense, high SNR repository of everything I've ever found valuable on the topic. I've done this for software interviews and it's been incredibly helpful.

Overall, I attribute this system to making me much smarter. It has been an invaluable investment.

https://news.ycombinator.com/item?id=25302365

For those interested in PKM, that's also how I manage mine but I follow something closer to Tiago Forte's "Building a second brain" method.

I use Logseq, and I have a sandbox for the particular day. Using Tagging, I can have a #inbox #interest - or any sort of sorting, and then create queries for pages where I process that reading when it's tied to a project. A lot of them never end up being read, and as they go further down the list, they become less important to me.

The thing is, your system requires a lot of discipline. If I had discipline, I would almost not need a tool :/
A very low effort way to do something like this is the following:

1) for any article, I save to read later. I used to use Pocket for many years, but recently switched to the new readwise reader. This makes it easy to search later.

2) if it’s a website or service/repo I want to be able to find later, I often screenshot it on the iPhone, and that’s it. The only downside is that you end up with a lot of screenshots in your camera roll. But now it’s quite easy to search for any text and the new iPhone automated OCR works incredibly well. I also sync all these images to google photos, which also lets you search for text in the images.

My system is very much like this. I capture most things to Drafts app with a “to process” or “to read” tag. And then review and handle at least once a week.

If it’s a resource that I want to refer back to (like how to do something or a tool I found interesting) I keep it in Drafts in a resource workspace appropriately tagged. If it’s a resource I keep and share a lot, I move it into Obsidian and write text around it to make sharing easy.

If it’s a longer item — say a paper that’s building my knowledge in a subject area — I move it into DEVONThink and annotate.

I do need to recall and reuse things and I find having different tools and workflows for different kinds of information helps me.

Our solution is so close that I put it into the github repository. I don't like to depend on any software, vscode is good enough for me. The key problem is this: there isn't that much valuable information worth recording each week. Just saving doesn't solve the problem, it requires understanding and thinking, as well as regular review.
I save everything I consider useful as an MHTML file (HTML/CSS/images in one file), which is native to Chromium browsers. This has the benefit of the page not being split into separate files as the normal save page does, doesn't break when the file is renamed like the normal save page does (on Windows at least), removes the need for bookmarks which go dead over time, preserves the original source URL in the file for later reference.

There are over 10k of such files I've saved in this manner. With practice it becomes second nature to categorize and tag them just using the filename, which makes them findable within seconds.

These have become like my own private search engine, without the issue of not being able to find answers to queries you know exist (which increasingly has become an issue with online search engines).

In Chromium the saving to MHTML feature is enabled by launching the browser with the CLI argument `--save-page-as-mhtml` (Vivaldi browser enables this by default without any arguments needed). Firefox used to support it via excellent addons up until their Quantum update but they haven't supported it since and is a dealbreaker to my daily use of it.

I want this but done automatically (save every webpage I spend more than 10 seconds in) and also work on my phone. Which is impossible I suppose.
There are some 'save everything' programs/extensions I seen mentioned in the past (and in this topic) but have wondered how users deal with the signal-to-noise ratio, later filtering for desired content, or whether the users find the storage requirements are worth it.

Often when searching for a specific thing, be it technical queries, useful reviews, troubleshooting, etc, it can take many, many searches and looking through pages before finding something worthwhile, at which point I'll save it (and name it with keywords for future me) since the effort required to even find it is not worth repeating. I'm not sure how such save everything utilities would make that part easier when I wish to return to the relevant page in the future (ie: finding the relevant page, among all the pages that aren't relevant when they share so many similar internal keywords).

MHTML looks interesting. Avoiding "Save as" into multiple files, I've been capturing the websites as screenshots to preserve the layout and printing as PDFs to preserve the content, and honestly it works so-so...
MHTML is almost as old as Google and was first implemented by Internet Explorer in 2001. They're identical to .eml files too (it's plain old MIME), so you can load them in your email client if you're so inclined.
Not MHTML I guess and not native to the browser, but in Firefox you can use the popular SingleFile addon to save any web page into a single HTML file.

https://addons.mozilla.org/en-GB/firefox/addon/single-file/

Seems like a decent project, which has come up before. The sibling commenter ramraj07 would likely be interested in what seems like a new feature listed to automatically save pages once loaded, which according to that link works on Firefox for Android (mobile).

Back when Firefox supported the UnMHT addon it would also work on Firefox for Android and had the nice feature of being able to customize which replacement Unicode was used for illegal filename characters (instead of just using underscores like Chromium does and SingleFile says it does).

Btw the comparison table of SingleFile mentions MHTML can't be 'unzipped' to extract the resources but there is an open source program on Windows which can extract the resources called ExtractMHT[1], which later became bundled with Universal Extractor.

[1] https://www.legroom.net/software/extractmht

I used to think like that, stored everything in Pinboard, tagged properly. I also used to use nvalt (https://brettterpstra.com/projects/nvalt/) for that as it had good search and I didn't have to switch to other tabs to search Pinboard.

It felt good to "catalog" all this knowledge but in reality I never went back to it, just like bookmarks and I realized that if something is important enough I'll always be able to re-find or re-download almost everything I ever found.

Not feeling like you have to catalog and store everything in personal knowledge management apps is very liberating.

That's a very good point. I love the "the best solution to your problem is to realise you don't have a problem" mindset.
That was the goal of the Memex proposed back in 1945, still unrealized. 8(
I take structured notes with Obsidian, then add a backlink to the resource. No bookmarking anymore because there's a probability of exactly 0% that I ever click on bookmarked resource again. Google indexes content, so you should index your own with obsidian and add semantic research on top.
(comment deleted)
I take notes in Joplin in general and also use the Joplin web clipper [1] to make sure that I have all the interesting articles I read or want to read in markdown

[1] https://joplinapp.org/clipper

I wrote a small app to scan my comments and stories on hacker news. It would dump them into a postgres database and use the full text search to normalize the text of the articles to make for easy searching at a later date when I inevitably would think “where did I read that thing about this?”
Sometimes I'll spend 15 minutes going through my (unordered) bookmarks and put them into folders, in the context of the things I'm working on. Some also get deleted.

There's a higher likelihood of them being deleted if they can't fit into a folder shared by other bookmarks.

I run into the same issue you do, and I just paste interesting links into Apple’s notes app since it’s convenient to use across my devices.

Personally I’ve found it useful to just scroll through it time to time to refresh my memory on what I’ve read/watched/listened to.

Maybe not the most ideal or fancy solution, but it works well for me, even as my note has gotten longer and longer.

Yep I do the same. From time to time, I move all my bookmarks to my notes, adding a title as a description, and removing useless bookmarks. I use a classification. It helps keep the bookmarks somewhere and kind of organised, but is very hard to search for.
I see others have already mentioned software I use for categorisation, i.e. Google Keep, Obsidian. If you like bookmarks and/or you want to organize and tag yours, I've recently found raindrop.io
In Linux you can use xsel to grab the url and put it in a tagger using one shortcut, then add tags or comments.

As you can see, the harder part is using what you have tagged. If you have a custom menu that you do searches with, it can show matches, and if you keep a history of searches then more frequently used links can be seen first.

You can do all sorts of things with tag hierarchies and other stuff that a browser won't do, so tagging is often saner than thinking of how to make hierarchies using folders in a browser.

My personal solution to this is YAGNI. In very rare situations I visit chrome://history and search, if it is only a couple of days old.

But I do sorta like the idea you hint at, of a 1 click "add to my web" with its own personal webcrawler. I'd give it a try.

(comment deleted)
I run an instance of yacy (in non-distributed Robinson mode). Any interesting links I add to my index (with a crawl depth of 0 since I only want that page, not the whole site).

This gives me a good search history, plus it automatically creates a cache of the page!

I just set up a YaCy instance myself last week after being inspired by your post on hackernews from ~7 months ago. Thanks for sharing that. I found some other helpful resources online for this too. I wrote about my configuration in case it might be helpful for anyone else who'd like yo play with YaCy for bookmarking. I haven't had it up long enough to know how useful it will be for me yet, but its been fun to play with. I also wrote an Android app to help me index specific pages from my mobile device with just a few taps.

https://www.richardosgood.com/posts/yacy-personal-search-eng...

I've never done something like that, but after reading your post and the comments, I'm tempted to start doing so!

I'm thinking about a script that receives the URL and saves its main content (e.g., using Mozilla's Readability) in a text file. It also stores the URL on the first line and maybe sends a request to Archive.org to take a snapshot of the page and adds its URL to the second line in the file. Then, whenever I need something, I can search the content of those files (I use the Silver Searcher) and find what I'm looking for. If the main content stored in the file is not enough, I can open the original URL or the Archive's snapshot.

I think I won't need to categorize or tag them; searching seems enough to me.

The only difficulty I can think of is that extracting the main content of pages is not easy, and, for example, Mozilla's Readability doesn't work well all the time. It may be required to have a manual process for copying and pasting the data.

(comment deleted)
I created a section on my website for shower thoughts and link reminders. I'm building it as I go and as I require new features. So far it's a title/desc+tags that I can search and filter. I'm planning to add a random reminder every day and some sort of "check back in X months/years if what I predict happen to be true". I can't remember the name of the addon I had to create bookmarks in the 2000's with tags rather than folder and but it was the best organization tool I ever had (as a very disorganized person), trying to recreate what I remember from it.
Oh and I'm also working on an extension to quick add links, with an AI summary on what's on the page + tagging. But that far off in the future!
Browser bookmarks with Netscape Bookmarks format (yuck!) for transferring between browsers when I access stuff regularly and don't want to discover it through the search engine every time. Self-hosted shortener service when I need to access URLs from any device in any location. So called "url files" which browser opens for me with every URL in a separate tab when there is temporary high volume inflow of URLs, e.g. researching a purchase, rental, or travel.
I keep everything in Evernote. The clipper is so good that I only have to clip the part of the page I care about, whether it's comments or pictures or whole article. It has fantastic search function with OCR.

I have about 5,000 notes in Evernote. I haven't found a good alternative so far.

I use Pocket[1] for this. There’s a browser extension and an iOS app; both are optimized for saving quickly. Later, I tag each item for easier search.

[1] https://getpocket.com/en/

Do you use paid version? I considered buying it but then read some reviews that their search in paid version isn't that good.