> my computer should search across everything I’ve read and some small high-quality subset of the Web
Even without the contextual search—which is initeresting mind you!—I want this part, and I’ve wanted it for a long long time.
I have it for journal articles simply by the virtue of obsessively saving and filing every PDF I’ve read for the last decade, but for the Web I don’t know how I could just plain index everything I’ve read (and not just everything in Pocket/Wallabag/etc). For example, is there anything that can pull links from Firefox Sync and save/index extracted text from them in a halfway reliable fashion? (I don’t expect it to be able to bypass yucky stuff like CAPTCHAs on archive.is.)
I know there are a handful of WARC-ecosystem tools, but AFAIU these are rarely that automatic, are tied to a browser plugin as opposed to just pulling the URLs from history sync (yes, I’m unreasonably fond of Epiphany/GNOME Web), and kind of heavy. Am I wrong? Any alternatives?
Pushes your history to ArchiveBox, which does the heavy lifting storing/processing the content.
Alas, might not work with Epiphany because there's no complete extension support.
But IIRC, it stores its urls in $XDG_DATA_HOME/epiphany/ephy-history.db - so a bit of sqlite and ArchiveBox might do the trick for you.
Note: I'm running something similar, but find that I'd rather not rely on my history, I tend to click on a lot of garbage ;) You might want to curate a bit.
There's a different strategy, it's a VC thing that's mac only but pretty fun to play with:
http://rewind.ai takes screenshots continuously and uses mac's system OCR to index everything you've ever looked at, whether that's web, code, or messages. When you search it pulls up screenshots that match.
I've never heard of this service but am intrigued. Also, props to the Rewind team for putting privacy front and center. (Now I just have to decide if I trust what they're saying.)
I built something like this over a weekend. Saves every page title, html body, and time and lets you search them in the chrome new tab page. It also indexes on the fly.
I have it installed on my browser and I’ll probably release it when I can figure out how to make it more useful.
The first part is more the hook than the thesis, as the rest of TFA explains. (The first paragraphs outright admit that it’s not actually that defensible if taken literally.)
The actual point is that every potential term and phrase in a text you’re reading should be preemptively searched for (across e.g. your web browsing history, reading list, etc.), marked up inline according to the rough potential usefulness of the results, and in any case those results should be made quickly accessible through user action, whatever the machine’s opinion on their usefulness.
So less manual hyperlinking and more web searches. (Manual hyperlinks have their place as well, but that place is things the author thinks especially relevant, not literally every point they want to make or refer to.)
> The computer can look at every word on the page, every phrase, name, quote, and section of text, and show me a “map” of the words and ideas behind which lay the most interesting ideas I might want to know about
This would be really useful when reading research papers.
In biology, it would already be a huge upgrade if you could click on the name of any gene or protein and see a summary of the protein function, links to sequence info, a list of other names that the gene has historically been labelled with, and links to other recent paper where the gene was studied.
In the non-biologies, I'm sure something similar would also be useful.
> Yeah, if one could solve the problem of identifying "most interesting ideas,"
I think ChatGPT can already categorize ideas, so you could at the very list see information about related concepts?
Perhaps this conjures different sentiments from other HN readers, but what instantly pops to mind for me is my childhood web browsing experience where I had the wrong toolbar or plugin or whatever, and thus every page was filled with automatically generated double-underlined green links that invariably went to a page I regretted visiting.
If there is one thing the SEO and LLM crazes have taught us, it is that small curated beats big generated, every single time.
That or some Yahoo! sites or random blogs that did this on their own. The links didn't go to malicious sites, just useless ones.
I can see this actually being useful if done well, but I didn't like the demo just because of the annoyingly highlighted words. It should only show up if I mouse over.
The best (worst) were all the ads that were a pop up underneath the cursor. There was absolutely no safe place to park your cursor. The ad of course occulted the page and you couldn't read anything. The bastard sibling of that type of ad was the popup under the cursor when you were just hovering over a link to see where it went.
That was my first reaction too, but on second thought it could be quite useful for spotting one's topics of interest on a page. I often do a cursory scan of the HN front page for discussions which might interest me, for example, and I bet I often miss stuff. It could make it less likely for me to miss comments of interest, too. If this is would be a plugin which could be turned on or off per site, I'd use it.
I didn't see it like that. It makes perfect sense in the context it's being shown in: a personal wiki system. Even if such a system would be extended to the browser, I imagine the principle of auto-linking based on the user's already-accumulated knowledge would stay. And would be nothing like SEO spam, since it would link back to the user, not to some third party. I would still want to be able to distinguish regular links from generated ones (and the heatmap UI would allow that), and it would make it easy to spot and connect to my topics of interest in a larger body of text.
For the many Obsidian users here, wanted to share an Obsidian demo/plugin that I saw recently by Justin Smith[0] that I think faithfully carries over a lot of what I liked about this idea into the Obsidian land, complete with a semantic index w/ language models.
If you're an Obsidian user, do check out the demo. I can't take credit for any part of building it, but it's really cool to see the idea in action :)
Hey. Any source for notation available? I fiddled around with codemirror to have my own notes app and iam curious about your implementation and would love to dig into it.
True. Like on the Hacker News website, where almost all of the posts are hyperlinks. People clearly get overwhelmed and jusy comment instead of opening the articles ;)
An ordered list of headlines is very different from a body of text littered with random hyperlinked phrases. Even in the article, the example screenshots are super messy.
There is a huge advantage to surfacing search results directly in the document. You drop the friction to effectively zero, which means you're far more likely to search random phrases in a document when you never would have before.
I've read some research papers where I need to look up a word in like every sentence. It's hard to focus when I have to keep going to a separate tab/window.
Should this be a system-level service in many circumstances? I love that with a three-finger tap, I get an instant dictionary definition of whatever word I'm hovering over (not even selected). I want that reliable availability, but more expansive.
The concept reminds me of Vannevar Bush's Memex device/system [https://en.wikipedia.org/wiki/Memex] - very cool! Though, i can imagine the UX, UI would have to be very tight and deliberate, lest the user can be easily overwhelemed, overloaded with too much noise (and not enough signal).
A well written document should focus the reader. When everything is a hyperlink and a "thought map" then you'll just end up being distracted and either not reading the original doc, or clicking on nothing and ignoring the deluge of hyperlinks. The demos in this article are heavily obstructed by all the visual noise.
If something is important enough, the author will discuss it directly in the document. Hyperlinks can provide context when the reader may not have the context assumed of the intended audience, or they're a poor man's bibliographic citation (because the links always break eventually), but otherwise it's a signal that it's _not_ important.
When everything is a hyperlink, nothing is a hyperlink. You can trivially make every word a hyperlink by running a script which attaches the dictionary.com entry for each word. I would never click on those links, and never know if a link was legitimate. Instead of link quantity, you need link quality.
That seems like common sense, but most major wikipedia articles agree with the this piece's author more than me. When I open a big article, I would personally prefer it if 1/4 as much of the text is blue-underlined as what I find. Links should be used to bring the reader's attention to key information that is useful context but only expanded on elsewhere. Not any text that could be conceivably linked to some other random article. If I needed that, I would just manually search for the topic.
In a world of scarce and distinct links, my curiosity would draw me to click on many, and I would go down wormholes in the way tvtropes pulls of. But spammed fluff links are useless.
In Wikipedia, being an encyclopedic work, heavy linking is a good thing, and one I rely heavily on. In regular articles and blog posts, I definitely don't want that many.
One of the HN regular contributor's personal site is similar. Extensively cross-linked and hovering over any links opens in its own popup window. A unique site.
I don't understand why, with all the UI/UX experience we have with hypertext, that have not moved the semantic of a hyperlink beyond a dumb one-way pointer.
Assuming that a maximal hypertextual interface needs to dynamically display relationships (rather than just a huge tag cloud or making all the body text hyperlink blue), a machine-readable characterization of the nature of those relationships would be invaluable. Does my text refute the linked resource? Or support it, or provide related factual statements, or is it chronologically or conceptually derivative? Some sort of ontological palette to write with could produce something that could be traversed in all sorts of novel ways. It would give writers more agency over external systems that will otherwise apply their own semantic meaning to the text, and its links.
Also, Hypertext (HTML, anyway) is oddly asymmetrical in its normal usage. We link a small string of text within a document, one-way, to the entirety an entire separate document. I think convention is the primary obstacle, but why can't we describe a relationship from our entire document to some other resource?
If you haven't run across it, you should read up on Ted Nelson's Xanadu project.
I've heard different arguments for why that failed. I think the likeliest is just that simple and dumb is often more successful than complex and nuanced.
> I think convention is the primary obstacle
At this point, I think the existence of the web as it is is the primary obstacle. Business models built on laws and case rulings depend on links as they currently exist. So changes here would need to get buy in from those folks, browser makers, standards body folks (but I repeat myself), and then would have to somehow be reconciled with laws that make assumptions about how they work.
As _jal notes, Xanadu affords this. (I've read about and watched demonstrations of the project, I've not used it.)
This is also a feature of many wiki systems, and in particular Mediawiki (the engine behind Wikipedia, Wiktionary, Wikisource, etc.). Here, both the linking and linked pages are aware of the link, at least so long as they occur within the same underlying Wiki. External links, including links to other Wikis (both within and outside the Wikipedia Foundation's remit) aren't treated like this, however.
Where it exists, though, it's a powerful model.
This is still only a subset of the features you're referring to, though again Mediawiki has some more complex relationships as well, through tools such as infoboxes, categories (which collect numerous other related pages), and more. Within Wikipedia inclusion in a category page is achieved not by editing the category page, but by editing the included page to indicate that it's a part of that category, which is quite powerful.
You might also want to look at various notions of the semantic web which have been floated. Limited adoption seems to be largely a combination of coordination problem and general complexity. Basic HTML is reasonably simple, especially where relationships between pages (and pages on multiple sites) are concerned.
1. Most people who publish their writings online are too lazy to hyperlink, even where it is extremely appropriate. Even intra-site links.
2. Link rot is real. Maintainers of web sites are too arrogant to make redirects, and happily destroy all their incoming links when playing with a new framework. So now links from your site lead to 404s, because of somebody on the other end, and you have to scan your outgoing links every few months and change them.
In a way Mac OS and iOS have already implemented the "everything is a link"-approach, with their look up feature. It's much more powerful than it used to be, and you can select any word or words anywhere and find all kinds of information.
- Text linking to the author's or publisher's choice of specific reference.
- Text linking to the reader's choice of specified reference.
There are pros and cons to each of these options, mind, but I'd specifically draw a distinction between a search engine's algorithmic approach and an author's or reader's volitional choices.
Love the fact that people still think and work on human centric computing.
There should clearly be some low hanging fruit in revisiting the hypertext concept with the benefit of a few decades of further tech development.
Its not clear though which aspect(s) would be the most beneficial. Eg when scanning text for information and connections, our context is frequently an important factor, but that context is not available to the computer.
How to add hyperlinks is something I've thought a bit about for Gwern.net: there's no point having all these fancy popups if there are no hyperlinks exploiting them, right?
The way I currently do it is that first, I make hyperlinks stable by automatically snapshotting & making local archives of pages (https://gwern.net/archiving#preemptive-local-archiving). There is no point in adding links if linkrot discourages anyone from using them, of course, and I found that manual linkrot fixing did not scale to the amount of writing & hyperlinking I want to do.
The next step is adding links automatically. Particularly in the STEM topics I write most about these days, AI, there are many acronyms & named systems which mean specific things but it's easy to get lost in. Fortunately, that makes them easy to write automatic link rules for: https://github.com/gwern/gwern.net/blob/master/build/Config/... These run automatically on essay bodies when compiling the site, and on annotations when created. If a URL is already present, its rule doesn't run; and if it's not, only the first instance gets linked and the rest are skipped. (This is important: there are some approaches which take the lazy approach of hyperlinking every instance. This is bad and discredits linking.) This code is very slow but fast enough for static site building, anyway.
Sometimes terms are too ambiguous or too rare or too much work to write an explicit rewrite rule for. But it will still exist on-site. In fact, you can say that the site corpus defines a set of rewrite rules: everytime I write by hand `[foo](http://bar)`, am I not implicitly saying that there ought to be a rewrite rule for the string `foo` which ought to hyperlink `http://bar`? So there is a script (https://github.com/gwern/gwern.net/blob/master/build/link-su...) which will parse the site corpus, compile all the text/link pairs, create/remove a bunch of them per whitelist/blacklists and a frequency/length threshold, and then generate a bunch of Emacs Lisp pairs. This master list of rewrites then gets read by an Elisp snippet in my Emacs and turned into several thousand interactive search-and-replace commands when I run my generic formatting command on a buffer.
The effect of this second script is that after I have linked `Foo et al 2023` to `/doc/2023-foo.pdf` a few times (perhaps I went back and hyperlinked all instances of it after realizing it's an important paper), any future instances of 'Foo et al 2023' will pop up a search-and-replace asking to hyperlink it to `/doc/2023-foo.pdf`, and so on.
Third, I exploit my link-recommendations for manually-curated 'see also' sections appended to annotations. I have a fairly standard link-recommender approach where each annotation is embedded by a neural network (OA API for now), and one does nearest-neighbor lookups to find _n_ 'similar' annotations, and shows the reader them in case any are relevant. So far so good. But I also do that after editing each annotation: embed-recommend-list, and spits out a HTML list of the top 20 or so similar-links appended to the annotation. I can look at that and delete the irrelevant entries, or the entire list. This means that they'll be included in the final embedded version of the annotation, will show up in any fulltext searche I run, are more visible to the reader, can be ...
As far as OP goes, matching on n-grams is a reasonable & fast idea, but the presentation is not good. The problem with lots of links is that they are a tax on the reader's attention and limited decision-making reserves; unfortunately, not having links is an even greater tax, because now one has to exert much more effort and use one's fingers as well as one's eyeballs to see if there is anything at all worth looking at.
My suggestion would be to relegate this to the authoring interface, and just show regular links to readers. This is still bad for authors, however, and so my suggestion would be to explicitly annotate 'possible links'. For example, you could draw an underline under all words, count how many hits that word would have across all possible n-grams, normalize document-wide, and set the opacity 0-100% based on that. So you would see most words have no underline (0% opacity), dubious links get hard to see lines, and then various proper nouns and titles and technical terms would have heavy dark underlines; the author can then easily see which ones might be viable links, and right-click or hover to check & instantiate. And this could be set to automatically hyperlink at certain % or run in an interactive modal to approve/disapprove hyperlinks as a final editing pass.
I'm increasingly inclined to see classical footnotes or endnotes as a preferable compromise.
Simply hyperlinking text often provides little context as to where that link is going, though alternatives (as with your own pop-up context cards, see also Mediawiki's use of same) are useful. I've played with sidenotes as an option, implemented in CSS (I've a few CodePens on this concept). That also works reasonably well (and can even be responsive to screen size).
There's also a fantastic quote of Walter Benjamin's I've just run across:
“Already today, as the current scientific mode of production teaches, the book is already an obsolete mediation between two different card file systems. For everything essential is found in the index box of the researcher who wrote it, and the scholar who studies it assimilates it in his own card file.”
-- Walter Benjamin - Attested Auditor of Books, in One Way Street (1928)
The thought of books (or Web articles) as inter-process communication between author's personal datastors (Zettelkasten, databases, wikis, org-mode notes, ...) is a fresh and illuminating one.
I don't like footnotes/endnotes at all, because they still incur the opacity of a link and require reader effort. Even a simple link-icon (to denote within-site links, before/after self-links etc) gives you more information than that!
> I've played with sidenotes as an option, implemented in CSS (I've a few CodePens on this concept).
One idea I've played with is having links be partially expanded and sort of 'docked' into the side, so you can glance over at any link and see the title & snippets. Crude mockup: https://gwern.net/doc/design/typography/sidenote/2022-11-16-... It doesn't work with 1-column layouts on most monitors because not enough space, but could work with more newspaper-like layouts or a 2-column layout with the right column reserved for the bibliography/sidenotes-esque cards.
Any thoughts of representing topics across different notes or pages visually? I have always wanted something similar to the https://twitchatlas.com/ but for topics of interest to me. For example Quantum Physics would be a very large bubble. Within that bubble could be links to several sub-topics and branching out from there would be related topics.
I know I am not articulating this the best. I am a visual learner and going through pages and pages in OneNote and cherrytree has been ineffective.
>In this vision, links are no longer lonesome strands precariously holding together a sparsely connected Web, but a booming choir of connections tightly binding together everything I have read and I will read
That isn't a good idea. For links to be purposeful they need to be high in signal and low in noise. If machines auto-link the word 'vision' in the quoted text above, I'll probably get a long list of optometrists. There's a reason the text itself has maybe half a dozen links in it, like " Andy and Michael’s excellent essay[...]". That's a specific piece if information that's relevant to the context.
Sparseness is a feature and not a bug when linking as you otherwise very quickly end up with a network where everything points to everything else, and that's meaningless. You don't want to 'tightly bind' everything together, because many of the things you read have nothing in common. In Schizophrenics, one important symptom is Apophenia, the tendency to see patterns in unrelated things. Hyperlink Maximalism is basically doing that to your text processing.
It would be nice to have hyperlinks to discussion, explanations and clarifications about every sentence.
Note that this is very different from a hyperlink to some "related" content that leads you on a tangent.
Imagine reading a book (about something complicated, perhaps math) and immediately when you don't understand something you can tap into discussion/QA about the exact sentence you just read.
Of course it would only work for books that are read frequently enough to build that bank of context, but I imagine for many common topics it would work.
I wanted this for a long time. Like you right click and get a choice of discussion, explanations, or clarifications, etc. I haven't studied Xanadu, but there just has to be something that does this out there already
There's a value and a cost to hyperlinks. Amongst other issues, they are distracting, and given linkrot often perform far more poorly than more traditional citations. Note that, say, Wikipedia, strongly prefers citations to naked URLs. (Though of course citations may link to sources.)
That said, I've occasionally written (or adapted) materials to include heavy hyperlinking. One example would be a bibliographic note (itself already a reference document), which I extended by linking names to Wikipedia entries and texts to their Worldcat entries:
One thing that made this easier is Wikipedia's convention for article links. I could auto-generate these when composing the document, then (reasonably quickly) verify the actual links were as expected. I think I'd used some shell functions to confirm Worldcat entries, which was possible at the time. A subsequent site degredation means that Worldcat is no longer usable at all without a JS-enabled browser, which w3m (my go-to terminal-mode browser) is unfortunately not.
I built a variant of this for learning Japanese (and soon more language pairs) by reading. It tracks (on your local device/personal iCloud) all the words and kanji you read, the sentences that contained them, and the original source materials. So when you encounter a word, you can view your personal corpus of example sentences and articles to reference from your reading.
I plan to expand this to a general purpose web browser (or a kind of alternative companion reading-oriented browser).
60 comments
[ 4.6 ms ] story [ 125 ms ] threadEven without the contextual search—which is initeresting mind you!—I want this part, and I’ve wanted it for a long long time.
I have it for journal articles simply by the virtue of obsessively saving and filing every PDF I’ve read for the last decade, but for the Web I don’t know how I could just plain index everything I’ve read (and not just everything in Pocket/Wallabag/etc). For example, is there anything that can pull links from Firefox Sync and save/index extracted text from them in a halfway reliable fashion? (I don’t expect it to be able to bypass yucky stuff like CAPTCHAs on archive.is.)
I know there are a handful of WARC-ecosystem tools, but AFAIU these are rarely that automatic, are tied to a browser plugin as opposed to just pulling the URLs from history sync (yes, I’m unreasonably fond of Epiphany/GNOME Web), and kind of heavy. Am I wrong? Any alternatives?
Pushes your history to ArchiveBox, which does the heavy lifting storing/processing the content.
Alas, might not work with Epiphany because there's no complete extension support.
But IIRC, it stores its urls in $XDG_DATA_HOME/epiphany/ephy-history.db - so a bit of sqlite and ArchiveBox might do the trick for you.
Note: I'm running something similar, but find that I'd rather not rely on my history, I tend to click on a lot of garbage ;) You might want to curate a bit.
Recoll together with the extension can be configured to index every single web page you browse.
https://addons.mozilla.org/en/firefox/addon/recoll-we/
http://rewind.ai takes screenshots continuously and uses mac's system OCR to index everything you've ever looked at, whether that's web, code, or messages. When you search it pulls up screenshots that match.
I have it installed on my browser and I’ll probably release it when I can figure out how to make it more useful.
>almost nothing in the article is hyperlinked
What did he mean by this?
The actual point is that every potential term and phrase in a text you’re reading should be preemptively searched for (across e.g. your web browsing history, reading list, etc.), marked up inline according to the rough potential usefulness of the results, and in any case those results should be made quickly accessible through user action, whatever the machine’s opinion on their usefulness.
So less manual hyperlinking and more web searches. (Manual hyperlinks have their place as well, but that place is things the author thinks especially relevant, not literally every point they want to make or refer to.)
This would be really useful when reading research papers.
In the non-biologies, I'm sure something similar would also be useful.
> Yeah, if one could solve the problem of identifying "most interesting ideas,"
I think ChatGPT can already categorize ideas, so you could at the very list see information about related concepts?
If there is one thing the SEO and LLM crazes have taught us, it is that small curated beats big generated, every single time.
I can see this actually being useful if done well, but I didn't like the demo just because of the annoyingly highlighted words. It should only show up if I mouse over.
https://en.wikipedia.org/wiki/Smart_tag_(Microsoft)
For the many Obsidian users here, wanted to share an Obsidian demo/plugin that I saw recently by Justin Smith[0] that I think faithfully carries over a lot of what I liked about this idea into the Obsidian land, complete with a semantic index w/ language models.
If you're an Obsidian user, do check out the demo. I can't take credit for any part of building it, but it's really cool to see the idea in action :)
[0] https://twitter.com/justindsmith/status/1679978286955532296
If something is important enough, the author will discuss it directly in the document. Hyperlinks can provide context when the reader may not have the context assumed of the intended audience, or they're a poor man's bibliographic citation (because the links always break eventually), but otherwise it's a signal that it's _not_ important.
That seems like common sense, but most major wikipedia articles agree with the this piece's author more than me. When I open a big article, I would personally prefer it if 1/4 as much of the text is blue-underlined as what I find. Links should be used to bring the reader's attention to key information that is useful context but only expanded on elsewhere. Not any text that could be conceivably linked to some other random article. If I needed that, I would just manually search for the topic.
In a world of scarce and distinct links, my curiosity would draw me to click on many, and I would go down wormholes in the way tvtropes pulls of. But spammed fluff links are useless.
<https://gwern.net/>
They discuss the site in this thread here: <https://news.ycombinator.com/item?id=36870640>
Assuming that a maximal hypertextual interface needs to dynamically display relationships (rather than just a huge tag cloud or making all the body text hyperlink blue), a machine-readable characterization of the nature of those relationships would be invaluable. Does my text refute the linked resource? Or support it, or provide related factual statements, or is it chronologically or conceptually derivative? Some sort of ontological palette to write with could produce something that could be traversed in all sorts of novel ways. It would give writers more agency over external systems that will otherwise apply their own semantic meaning to the text, and its links.
Also, Hypertext (HTML, anyway) is oddly asymmetrical in its normal usage. We link a small string of text within a document, one-way, to the entirety an entire separate document. I think convention is the primary obstacle, but why can't we describe a relationship from our entire document to some other resource?
I've heard different arguments for why that failed. I think the likeliest is just that simple and dumb is often more successful than complex and nuanced.
> I think convention is the primary obstacle
At this point, I think the existence of the web as it is is the primary obstacle. Business models built on laws and case rulings depend on links as they currently exist. So changes here would need to get buy in from those folks, browser makers, standards body folks (but I repeat myself), and then would have to somehow be reconciled with laws that make assumptions about how they work.
Easy peasy.
This is also a feature of many wiki systems, and in particular Mediawiki (the engine behind Wikipedia, Wiktionary, Wikisource, etc.). Here, both the linking and linked pages are aware of the link, at least so long as they occur within the same underlying Wiki. External links, including links to other Wikis (both within and outside the Wikipedia Foundation's remit) aren't treated like this, however.
Where it exists, though, it's a powerful model.
This is still only a subset of the features you're referring to, though again Mediawiki has some more complex relationships as well, through tools such as infoboxes, categories (which collect numerous other related pages), and more. Within Wikipedia inclusion in a category page is achieved not by editing the category page, but by editing the included page to indicate that it's a part of that category, which is quite powerful.
You might also want to look at various notions of the semantic web which have been floated. Limited adoption seems to be largely a combination of coordination problem and general complexity. Basic HTML is reasonably simple, especially where relationships between pages (and pages on multiple sites) are concerned.
1. Most people who publish their writings online are too lazy to hyperlink, even where it is extremely appropriate. Even intra-site links.
2. Link rot is real. Maintainers of web sites are too arrogant to make redirects, and happily destroy all their incoming links when playing with a new framework. So now links from your site lead to 404s, because of somebody on the other end, and you have to scan your outgoing links every few months and change them.
In a way Mac OS and iOS have already implemented the "everything is a link"-approach, with their look up feature. It's much more powerful than it used to be, and you can select any word or words anywhere and find all kinds of information.
So I think we already have this feature. Now we just need better search-engines. Maybe AI supported ones? Wait we already have those too.
Making every word look like a hyperlink is a bad idea because often the user will want to search for a whole phrase, not just an individual word.
- Text linking to a general Web search.
- Text linking to the author's or publisher's choice of specific reference.
- Text linking to the reader's choice of specified reference.
There are pros and cons to each of these options, mind, but I'd specifically draw a distinction between a search engine's algorithmic approach and an author's or reader's volitional choices.
There should clearly be some low hanging fruit in revisiting the hypertext concept with the benefit of a few decades of further tech development.
Its not clear though which aspect(s) would be the most beneficial. Eg when scanning text for information and connections, our context is frequently an important factor, but that context is not available to the computer.
The way I currently do it is that first, I make hyperlinks stable by automatically snapshotting & making local archives of pages (https://gwern.net/archiving#preemptive-local-archiving). There is no point in adding links if linkrot discourages anyone from using them, of course, and I found that manual linkrot fixing did not scale to the amount of writing & hyperlinking I want to do.
The next step is adding links automatically. Particularly in the STEM topics I write most about these days, AI, there are many acronyms & named systems which mean specific things but it's easy to get lost in. Fortunately, that makes them easy to write automatic link rules for: https://github.com/gwern/gwern.net/blob/master/build/Config/... These run automatically on essay bodies when compiling the site, and on annotations when created. If a URL is already present, its rule doesn't run; and if it's not, only the first instance gets linked and the rest are skipped. (This is important: there are some approaches which take the lazy approach of hyperlinking every instance. This is bad and discredits linking.) This code is very slow but fast enough for static site building, anyway.
Sometimes terms are too ambiguous or too rare or too much work to write an explicit rewrite rule for. But it will still exist on-site. In fact, you can say that the site corpus defines a set of rewrite rules: everytime I write by hand `[foo](http://bar)`, am I not implicitly saying that there ought to be a rewrite rule for the string `foo` which ought to hyperlink `http://bar`? So there is a script (https://github.com/gwern/gwern.net/blob/master/build/link-su...) which will parse the site corpus, compile all the text/link pairs, create/remove a bunch of them per whitelist/blacklists and a frequency/length threshold, and then generate a bunch of Emacs Lisp pairs. This master list of rewrites then gets read by an Elisp snippet in my Emacs and turned into several thousand interactive search-and-replace commands when I run my generic formatting command on a buffer.
The effect of this second script is that after I have linked `Foo et al 2023` to `/doc/2023-foo.pdf` a few times (perhaps I went back and hyperlinked all instances of it after realizing it's an important paper), any future instances of 'Foo et al 2023' will pop up a search-and-replace asking to hyperlink it to `/doc/2023-foo.pdf`, and so on.
Third, I exploit my link-recommendations for manually-curated 'see also' sections appended to annotations. I have a fairly standard link-recommender approach where each annotation is embedded by a neural network (OA API for now), and one does nearest-neighbor lookups to find _n_ 'similar' annotations, and shows the reader them in case any are relevant. So far so good. But I also do that after editing each annotation: embed-recommend-list, and spits out a HTML list of the top 20 or so similar-links appended to the annotation. I can look at that and delete the irrelevant entries, or the entire list. This means that they'll be included in the final embedded version of the annotation, will show up in any fulltext searche I run, are more visible to the reader, can be ...
My suggestion would be to relegate this to the authoring interface, and just show regular links to readers. This is still bad for authors, however, and so my suggestion would be to explicitly annotate 'possible links'. For example, you could draw an underline under all words, count how many hits that word would have across all possible n-grams, normalize document-wide, and set the opacity 0-100% based on that. So you would see most words have no underline (0% opacity), dubious links get hard to see lines, and then various proper nouns and titles and technical terms would have heavy dark underlines; the author can then easily see which ones might be viable links, and right-click or hover to check & instantiate. And this could be set to automatically hyperlink at certain % or run in an interactive modal to approve/disapprove hyperlinks as a final editing pass.
Simply hyperlinking text often provides little context as to where that link is going, though alternatives (as with your own pop-up context cards, see also Mediawiki's use of same) are useful. I've played with sidenotes as an option, implemented in CSS (I've a few CodePens on this concept). That also works reasonably well (and can even be responsive to screen size).
There's also a fantastic quote of Walter Benjamin's I've just run across:
“Already today, as the current scientific mode of production teaches, the book is already an obsolete mediation between two different card file systems. For everything essential is found in the index box of the researcher who wrote it, and the scholar who studies it assimilates it in his own card file.”
-- Walter Benjamin - Attested Auditor of Books, in One Way Street (1928)
<https://writingslowly.com/2023/07/24/walter-benjamin-on.html>
The thought of books (or Web articles) as inter-process communication between author's personal datastors (Zettelkasten, databases, wikis, org-mode notes, ...) is a fresh and illuminating one.
> I've played with sidenotes as an option, implemented in CSS (I've a few CodePens on this concept).
One idea I've played with is having links be partially expanded and sort of 'docked' into the side, so you can glance over at any link and see the title & snippets. Crude mockup: https://gwern.net/doc/design/typography/sidenote/2022-11-16-... It doesn't work with 1-column layouts on most monitors because not enough space, but could work with more newspaper-like layouts or a 2-column layout with the right column reserved for the bibliography/sidenotes-esque cards.
I know I am not articulating this the best. I am a visual learner and going through pages and pages in OneNote and cherrytree has been ineffective.
That isn't a good idea. For links to be purposeful they need to be high in signal and low in noise. If machines auto-link the word 'vision' in the quoted text above, I'll probably get a long list of optometrists. There's a reason the text itself has maybe half a dozen links in it, like " Andy and Michael’s excellent essay[...]". That's a specific piece if information that's relevant to the context.
Sparseness is a feature and not a bug when linking as you otherwise very quickly end up with a network where everything points to everything else, and that's meaningless. You don't want to 'tightly bind' everything together, because many of the things you read have nothing in common. In Schizophrenics, one important symptom is Apophenia, the tendency to see patterns in unrelated things. Hyperlink Maximalism is basically doing that to your text processing.
Note that this is very different from a hyperlink to some "related" content that leads you on a tangent.
Imagine reading a book (about something complicated, perhaps math) and immediately when you don't understand something you can tap into discussion/QA about the exact sentence you just read.
Of course it would only work for books that are read frequently enough to build that bank of context, but I imagine for many common topics it would work.
[1]: https://book.realworldhaskell.org/read/
That said, I've occasionally written (or adapted) materials to include heavy hyperlinking. One example would be a bibliographic note (itself already a reference document), which I extended by linking names to Wikipedia entries and texts to their Worldcat entries:
<https://web.archive.org/web/20230607050023/https://old.reddi...>
One thing that made this easier is Wikipedia's convention for article links. I could auto-generate these when composing the document, then (reasonably quickly) verify the actual links were as expected. I think I'd used some shell functions to confirm Worldcat entries, which was possible at the time. A subsequent site degredation means that Worldcat is no longer usable at all without a JS-enabled browser, which w3m (my go-to terminal-mode browser) is unfortunately not.
I plan to expand this to a general purpose web browser (or a kind of alternative companion reading-oriented browser).
https://reader.manabi.io