Link rot and content drift are endemic to the web (theatlantic.com)

All the cool stuff has gone back to IRL. Once people stopped making potato gun websites, the internet really stopped blossoming into an amazingly vibrant space. I would recommend hackaday.com because it hasn't changed in quite some time.

I wish there was something like Reddit that could be organized by topic, but had the simplicity of HN's design instead of the monstrocity Reddit has become. My guess is it would still succumb to the Reddit Hive Mind effect without a reasonably benevolent moderation team though. For all the times I've said that HN basically does the same thing, I have to admit that it is much better about keeping it in check.

The article is about link rot, not cultural rot.

Google still works, you just need to be very specific on your search criteria (e.g. site:). I do agree on the premise that good quality data, art and content has been lost to the winds of time.

You either have to visit aggregating sites that have likeminded people (HN/Reddit/Obscure FB groups/Group chats/Forums) or know exactly what you're looking for.

It's depressing.

I know somebody who started a business that was successful for a while and then failed. Spammers got control of the domain and now it is full of ads for a dangerous diet drug.

What makes my blood boil is that it impugns the integrity of the founder who is a decent person who has nothing to do with that scam.

This is why when I see something I like on a website, I might bookmark it, but I'm also saving it locally.

Exactly. The Atlantic author seems to be laboring under the misguided assumption that the web is somehow the same sort of thing as a library of books. Even libraries often have some degree of garbage information in them, and represent a survival story: the vast majority of books ever written are no longer in print, or even discoverable anywhere.

Good stuff should be preserved, but it's not the Internet's job to somehow magically do it. It's OUR job, and the nature of digital information (DRM not withstanding) makes this easier than ever.

Links are the backbone of the internet. Archive.org is a huge asset, but it relies on individuals being prescient enough to archive pages that might be lost to time. That's not scalable. Plenty of people will visit Wayback Machine to pull up an old page that's gone to the big 404 in the sky, but they won't actively submit links to archive themselves.

The bad design and low quality content is a symptom of the Internet's broken underlying economics. That's a human problem, not a tech problem.

> increasingly throttled and blocked and excluded from walled garden

I keep thinking back to Jacob Applebaum's stance of "facebook and the other walled gardens are the real dark web."

Honestly, it would be a better use of surplus resources than crypto mining.

If only there are a way to algorithmically tie a proof of work for a new cryptocurrency to archival of the internet in a way that wouldn't be easily gamed (by people archiving easy to access content or highly redundant archival of trivia).

WARC can record and replay single-page apps, but it struggles with knowing where a "page" begins and ends.

There was a time when I was furious with the web going to hell and I investigated the possibility of "web without browsers" that started with making a WARC capture of page and putting pages through extensive filtering and classification before the user sees anything.

With interactive capturing you can push a button to indicate that a page is done "loading" but with automated capturing you can't really know that the page is done or that you got a good capture. That ended the project right there.

I think “right to be forgotten” is important and I’m generally against everlasting social media posts, but for copyrighted works, we really need a centralized Library of Congress that acts to archive these. In order for that to happen there needs to be an equivalent “publishing” mechanism for the web - where the user says - I created something and I want it to be archived. This would cover things that exist behind a paywall or are only delivered as newsletters.

WARC is genuine genius, and a very real and valuable contribution in large part of the Internet Archive.

Ironically, deleting stuff on the web is technically REDUCING its entropy.

Such as / directions / goals to seek / pitfalls to avoid?

I know your comment is probably tongue-in-cheek but I must say (without disagreeing, regardless) that perhaps modern day digital infrastructure can transcend the considerations early buddhists may have had for the natural world and then-contemporary human-built structures.

Indeed yes, however much we may dislike it, change is the only constant. Of course that doesn't mean we shouldn't bother archiving, but there is no need to fret over saving every byte out there on the web.

Entropy is king. Eventually all information loses its coherency.

Accepting the impermanence of all things is an important lesson, but it is not an excuse for nihilism, because even impermanent things can have value, however finite, and the impossibility of true permanence doesn’t have to distract from realising finite value while it lasts.

Buddhists have preserved most of the Tripitaka for 2500 years, and for the first 500 years it was memorized and transmitted orally from generation to generation. Buddhist monks today spend significant amounts of their time memorizing parts of it. Printed, it's about 12000 pages; it's been translated into many languages, but not all of it has been translated into English yet. Thanissaro Bhikkhu has been working on it for 20 years, publishing his translations under CC-BY, and may finish the job before he dies. Aside from its value to devotees, the Tripitaka is one of our best historical sources about everyday life in South Asia 2500 years ago.

The invention of wood block printing 1300 years ago in the Tang was apparently specifically motivated by the desire to preserve and reproduce Buddhist sutras; the oldest surviving documents printed with movable type, from 900 years ago, are also Buddhist texts.

Of course the Tripitaka is not permanent; it will be lost some day. But you seem to be implicitly claiming that Buddhists do not apply effort to preserving information and in particular textual records, because they know that ultimately they will be lost. In fact, the truth is quite the opposite, and believing your implicit claim would require almost complete ignorance of Buddhism, printing technology, and South Asian classical studies.

Maybe the reason it feels impossible to stem the tide of link rot is that it's takes tremendous energy to constantly increase the entropy of a system. And that energy has opportunity cost that no one really wants to talk about.

The article has the unstated assumption that eternal preservation of all writing ever is a net benefit. I think it's worth having a discussion on that point.

Having read the whole thing, it seems partially good, martially misguided, and partially terrible.

The overall bent is a hand-wringing about link rot, which I thought we mostly got over a decade ago. The Internet is fundamentally ephemeral. If you see something you like, save it so you can repost it later. If you rely on someone else to keep in up indefinitely, you're being foolish.

Around the edges of that main discussion, the Atlantic also touches on censorship in all the wrong ways, re-iterating the too-common view that censorship is good as long as the good guys do it. They at least argue that this censorship should be transparent and censored works still accessible in some way, but they seem to not understand the nature of what they're talking about.

Censored works aren't censored to protect the public. They're censored to protect the rich and powerful. That's why "right to be forgotten" really exists. That's why Google and Youtube and Twitter and Facebook quash anything that goes against the accepted narrative in any given field. They aren't protecting the public from dangerous misinformation. No one gives a shit about the public. They're protecting the financial and political interests of some very powerful people.

Given this, talk of a "poison cabinet" only illustrates ignorance of the issue. The Memory Hole cannot be divorced from the censorship process, it's a core part of it. If people can still find the information in some form, it's not censored enough to make the people it threatened happy.

And this leads to the final point, which is that the real reason the web is "rotting" isn't link rot, it's censorship on the part of tech monopolies, due to their joined-at-the-hip relationship with every large corporation and industry imaginable due to advertising and other deals. The fact that links die doesn't matter much: you can just repost the material. The fact that links ARE ACTIVELY KILLED to suppress their information is a much more serious problem, and one that doesn't have an easy solution besides full breakup of the tech monopolies.

> audit when content is changed via takedown requests

Not just that, but even due to politics, someone wanting their history "changed", or even worse reasons.

There were multiple reddit arguments recently, when someone posts a newsstory about something, discussion starts, two hours later the story is changed (without any footnote or old version available), and arguments start, because "there is noting in the article that says <what you said>", and no way for the original poster to prove, that it was, because noone even thought they'd need a screenshot.

The web forum that I frequent most, https://forum.nasaspaceflight.com , has a policy of not allowing embedded images but requiring them to be attached on each post. The forum has been active for well over a decade now (site was founded in 2004) and has a thriving community that continues to grow. It is the community that keeps the forum alive instead of just one random company (although it technically is a company). This helps prevent link rot. Forums seem anachronistic in 2021, but they have massive benefits versus the gigantic platforms like Facebook or Reddit or Twitter, and the quality of the discourse and analysis is far higher. There are a few ads, but they’re very unobtrusive.

(Also, Twitter posts are often linked, but usually the text is copied for archival purposes.)

My experience with Wikipedia, forums such as those, and Wayback Machine and arxiv.org make me think that people will do a lot of stuff basically for free and that by building communities, you don’t need extremely clever trustless incentive systems like Blockchain or major paywalls (although granted, the forum does have something like a paywall for unverified pre-public info) or massive platforms with multi billion dollar companies in order to disseminate information, analysis, news, etc. Best practices of web forums from the 2000s (active moderation, a sense of common purpose, expectations of non-toxicness, etc), are a really good solution.

I set up a server specifically for downloading Youtube videos from all my playlists on a daily basis just for this reason. At some point I got fed up at seeing all the missing videos on my playlists (and not even knowing what was removed)

How many how to videos, memes, concerts, etc, are really that important?

In fifty years how many people will care? How many people should care because it would mean ignoring the huge volume of newer stuff? A hundred years? Two hundred?

I haven't even read or seen many of the existing cultural artifacts we have from past decades and centuries, what would I do if orders of magnitudes more of them had been preserved?

In fact, I'm incredibly greatful that I grew up before all the random shit that I threw out there as part of my youth was subjected to obsessive cataloging and archival efforts.

I download everything I like. Storage is cheap now.

Plus smaller sites will start disappearing because of regulatory capture. It will not be possible to run a forum or similar site in few years.

It’s really up to you who cares about something to archive it. I managed to find a torrent of early days video games from my region that has almost been lost to time. Luckily I found a discord and could coax someone to hop on to seed it. If I had waited 10 more years they might have been gone for good.

Everyone’s assuming that data now stays on the internet forever because it’s so massive. It’s usually one or two people who keep the flame alive

> "and please skip the tired argument that I should just pay a subscription fee to avoid their ad crap - we are already paying them with our data. My data is worth far more to me than the value they provide for it. Plus - I won't give money to a company who forced this Faustian bargain on me."

Not only this, but many of the largest companies these days would never remove advertising even from a paid service. It's like cable TV. They wanna charge you and advertise at you for more money.

The 2014 Vulture interview with David Milch, where he reads form an unreleased Boss Tweed script, may be gone for good. https://twitter.com/mattzollerseitz/status/14096229692828753...

you're worried about....the ads? Having ads around doesn't make the content any less valuable. We're talking about the content itself still being available, who cares if there's some ads keeping the system up if all those live concerts and how-to vids are preserved forever

They won’t stop collecting your data even if you pay them

> I am increasingly worried about the valuable content on YouTube.

Download it? This continues to be not difficult for YouTube.

> I suspect that one day, they will make their ads unblockable by embedding them in the video files.

That's fine. I honestly wish they would because most of the hangs in YouTube I experience when the stream changes to an ad, and then changes back. If it was embedded in the video then the stream wouldn't be interrupted.

If I hate the ads that much one can edit them out after it's downloaded.

> that I should just pay a subscription fee to avoid their ad crap

Fuck no. Do not do this. Use youtube-dl[0], and maintain local copies of anything useful you can find.

0: http://youtube-dl.org/

Good news, you can block ads embedded in youtube videos: https://addons.mozilla.org/fr/firefox/addon/sponsorblock/

It skips sponsors (even from the youtuber itself), jingles, intro, etc.

It's awesome.

What I hate about youtube the most is their deleting of videos AND metadata... I make a "watch later" playlist, with stuff that i must watch, two weeks go by, and there is a "[Deleted video]" in my playlist... not even a title left behind (so i'll be able to find it elsewhere).

Or on Firefox.

On the bright side, using a tool like Internet archive it should be easy to filter out which articles were removed and/or edited by the BBC, in a way highlighting the most historically important articles.

    usage 1: (simple html page of all results)
    echo https://www.theatlantic.com/article/619320/|1.sh >1.htm
    firefox ./1.htm

    usage 2: (retrieve last result)
    echo https://www.theatlantic.com/article/619320/|1.sh 1 >2.htm
    firefox ./2.htm

    #!/bin/sh 
    read x0;
    x1=web.archive.org;
    curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original" \
    |case $# in :)
    ;;0)( printf "<h2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$x0</h2><ol><pre>\n";
        sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
        printf "</ol></pre><br>\n" )
    ;;1)curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p);
    esac 

    #!/bin/sh
    read x0;
    x1=web.archive.org;
    printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms 
    \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b 
    \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \
    |exec haproxy -D -f /dev/stdin;
    printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \
    \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80 \
    |case $# in :)
    ;;0)( printf "<h2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$x0</h2><ol><pre>\n";
        sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
        printf "</ol></pre><br>\n" )
    ;;1) printf "GET %s HTTP/1.1\r\nHost: $x1\r\nConnection: close\r\n\r\n" \
         $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \
         |exec nc -vvn 127.21 80;
    esac;
    if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi

     #!/bin/sh
     read x0;
     x1=web.archive.org;
     curl -s "https://$x1/cdx/search/cdx?url=$x0&fl=timestamp,original"|case $# in :)
     ;;0)( printf "<h2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$x0</h2><ol><pre>\n";
         sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
         printf "</ol></pre><br>\n" )
     ;;1)x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036');
         curl -s $(sed -n -e "s>.*>https:/$x1/web/&>;s> >/>" -e \$p)|exec tr -d '[\02\03\36]' \
         |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/;
         s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \
         |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;";
     esac

     #!/bin/sh
     read x0;
     x1=web.archive.org;
     printf "defaults\ntimeout client 50000ms\ntimeout server 50000ms\ntimeout connect 50000ms 
     \nglobal\npidfile $HOME/1.pid\nfrontend f\nbind 127.0.0.21:80\ndefault_backend b 
     \nbackend b\nserver s ipv4@207.241.237.3:443 ssl ca-file /etc/ssl/certs/ca-certificates.crt\n" \
     |exec haproxy -D -f /dev/stdin;
     printf "GET /cdx/search/cdx?url=$x0&fl=timestamp,original HTTP/1.1\r\nHost:\40$x1 \
     \r\nConnection: close\r\n\r\n"|exec nc -n 127.21 80|case $# in :)
     ;;0)( printf "<h2>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$x0</h2><ol><pre>\n";
         sed -n "/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/<li><a href=https:\/\/$x1\/web\/\1\/\2>\1<\/a>/;s/ </</;s/ //2p;}";
         printf "</ol></pre><br>\n" )
     ;;1) x=$(echo x|exec tr x '\002');y=$(echo y|exec tr y '\003');z=$(echo z|exec tr z '\036');
          printf "GET %s HTTP/1.1\r\nHost: ${x1}\r\nConnection: close\r\n\r\n" \
          $(exec sed -n -e '/[0-9]\{14\} [hf]/{s/\(.* \)\(.*\)/\/web\/\1\/\2/;s/ //p;}'|exec sed -n \$p) \
          |exec nc -vvn 127.21 80|exec tr -d '[\02\03\36]' \
          |exec sed "s|<script src=\"//archive.org/includes/analytics.js?v=|$z$x 1|;s/<.-- End Wayback Rewrite JS Include -->/$y 1/;
          s/<.-- BEGIN WAYBACK TOOLBAR INSERT -->/$z$x 2/;s/<.-- END WAYBACK TOOLBAR INSERT -->/$y 2/;" \
          |exec tr '\036' '\012'|exec sed "/$x 1/,/$y 1/d;/$x 2/,/$y 2/d;"
     esac;
     if [ -f 1.pid ];then kill -9 $(sed b 1.pid);exec rm 1.pid;fi

> so unless historians dig deep in third-party archives, they'd never understand where that belief came from

I expect future historical tooling will exist to solve exactly this problem. Assuming Archive.org and the like nabbed it, the evidence is all there for future generations to see.

The BBC publishes nothing but garbage and there are other extant sources that are more durable. It's fine to forget things. We're missing entire libraries of classical literature from great authors which would be nice to have. Missing documentary sewage isn't a tragedy.

This should show us that most of the web isn't worth preserving anyway, much like McDonald's burger wrappers aren't worth preserving like sacred artifacts. Most web and social media content is worth less than said greasy burger wrappers.

How much of these overwrites are cover ups for failures? Certainly some percentage, just not sure what.

> a rule for my website

Note that you should also have a rule to save the link content locally, to avoid single-point-of-failure problems in the unlikely-but-catastrophic case that archive.org itself goes down. (Cf the attempts to attack them over their National Emergency Library programme last year.)

We should have kept developing Usenet. Handing control over to web browser providers was a mistake.

What specifically do you hash?

How do you hash content which is programmatically determined and changes on every page load?

How do you account for the same work in multiple versions, translations, or updates, strictly using hashes?

(Note that a chained hash, e.g., a git history, is not a strict use of hashes, though it most definitely does use hashes.)