Interesting. I was looking into creating an extension that manually manipulates and intercepts the vnd.yt-ump [1] requests, then use webcodecs to process everything in the browser.
Yeah I was onboard until the re-encoding part: yt-dlp maintains the exact bits, why on earth would someone want to waste encoding time just to trash the quality?
On top of that... seriously, of all the formats one could choose, MOV?! Might as well choose DivX or RealVideo.
Cool but ... this also sounds like hording behavior. The number of things I've saved over the years only to throw them away years later and realize that saving them in the first place was a waste of time.
In the 90s my friend's mom would video tape AMC movies. She had 300+ tapes. Maybe she had a few rare ones but now all those movies are available on demand either legally or illegally and in much better quality. Another friend kept all of his 1980s computer magazines (Byte, etc...) and moved these extremely heavy boxes through 30+ years of moves. I doubt he ever opened a single magazine since the moment he saved them. Then they all appeared on The Archive and he finally got rid of them.
To be clear, I have a few youtube videos saved on my local storage. I'm just thinking that saving every video I watch reminds me of the things I've personally over-saved.
Actually that reminds me. I met up with the magazine saving friend recently which is when I verified that he finally got rid of his stash. It made me think about things I'm still saving that if I reflect on I know I will never actually look at. For example I have box of about eight 3.5 inch floppy disks from my Amiga days. The odds that I'm going to get an Amiga or download an Amiga emu and get a drive to read those are close enough to zero that I should throw them away. Similarly I have a book of CD-ROMs of backed up data from the 90s. There's a close to 0% chance that I'm never going to bother look at their contents.
>There's a close to 0% chance that I'm never going to bother look at their contents.
More likely scenario, your children, grandchildren or other family members go through your shit after you pass away and discover stuff about you that perhaps you never wanted to share.
This is something I think about a lot because I don't have a "digital legacy plan."
I would like to be able to search old videos I’ve seen sometimes. Like to find that one recipe I saw or to pull out that one fact I thought I heard. Or sometimes just to listen to a song that later got made private or deleted outright. When YouTube deletes a video it doesn’t even leave the title in your playlist so it can be frustrating to try and find the same thing again.
I am the exact opposite and sell or throw away pretty much everything that I don't use. I find that doing so not only clutters the house less, but also gives you less to worry about.
My general rule is - if I didn't use it for a year, I don't need it. There are obviously some exceptions like a fire extinguisher (which I hope to never use) and digitized photos, which only go through a careful selection.
I think the thing I kept the longest was a Libranet Linux 3.0 CD set because I worked for Libra Computer Systems for a while and this was the release that I helped building. A few years ago I threw it away, I think after I saw someone uploaded it to archive.org. When I'm 60 and want to install it again for good old time's sake I can.
tl;de: if you don't use something for a year, you probably don't need it.
PSA: if you have a collection or other artifacts for ingest by IA, I’ll cover reasonable shipping costs to get them there. Above a certain size, they’ll handle logistics of packing and shipping for ingest.
> Then they all appeared on The Archive and he finally got rid of them.
Sometimes you're the person who is uploading them to public archives. Because everybody else threw them all away, and you saved them until the technology made archiving practical enough.
I've been replacing all of my physical media for years, but the reason I can do that now is because other people scan/rip and archive/share the stuff. You also have unique stuff that you may not even know is unique. When you find something in your house that you can't find online, scan it and you're paying everybody back for all of the scanning they did for you.
With the CD-ROMs, you should just glide through them one by one and check if you can find the stuff online. If you can, throw them in the trash. If you can't, copy their contents to a folder, and throw them in the trash. Go through the folder over the next hour or next 20 years (however long it takes to get around to it) and take the things you can't find online that you think somebody might want, and get those things to that somebody (uploading to archive.org is always a good place to start.)
edit: I know for a fact that for a lot of people, uploading somewhere on the internet is their standard pre-deletion ritual.
I agree with you on a personal level. I also collected a bunch of different things but eventually threw them all away and am now extremely selective about what I keep. I even avoid hoarding digital stuff.
However, in your examples, the fact those things eventually became available in other forms is not necessarily a counterpoint to your acquaintances having kept them. The specific counterpoint being Marion Stokes.
It's only hoarding if you don't fill the collection. It also depends on what percent of your storage you devote to it. If I could store every video I ever watched locally and only pay 1% of my storage cost, why not?
> saving them in the first place was a waste of time
I think of it like this:
Automatically save everything and spend time deleting the things I don't want to keep. vs Manually saving everything.
"Don't want to keep" depends on disk space and cluttering up the list as it grows. Disk space is not really a thing. I use to have a friend who spend many hours every week cleaning up his 512 GB drive. He was quite obviously deleting things he wanted to keep but "had to" make choices. I just have enough drives to keep 1 to 3 copies of everything. (The single drive will also fail inevitably)
The clutter still seems to happen even if I make the effort to get rid of things. Organizing it a bit, say at least by date is inevitable.
Therefore there is nothing to be gained by wasting time saving things. It is more time efficient to waste time by deleting only enormous folders that you clearly don't need to keep around.
Hoarding is only an appropriate term if you don't have the space for it. If you have an empty airplane hangar a few boxes of foo isn't hoarding.
It is hoarding behavior. I am sometimes guilty of it but I have no regrets. I have DVD's and CD's that either do exist online in a modified and sometimes censored format or got cancelled for reasons that exist now but did not exist when the movie or song was created social constructs, cancel culture, etc... The original versions probably exist in some obscure place but I am happy with them existing in my collection as well for data parity purposes. The same thing occurs with online videos, podcasts, music and movies.
I prefer to preserve the original artistic works of the producers, directors and screen writers. It could be I am alone in this endeavor and perhaps I am just a hoarder in that regard. I like to watch movies or listen to songs in the way the artists originally intended rather than the current ways that society has determined I should. The exception of course is when a movie is re-mastered and they improve lighting, sound, resolution, etc...
I hesitate to even imagine how AI will sloppify these great works in the future or how they will re-imagine podcasts. The original art will be lost to time and many people will never experience the original results of the blood sweat and tears that went into making them.
Hording behaviour doesn't really apply digitally. It's not like it's physical magazines or VHS. It's just bytes and bytes are incredibly cheap. It's also easier to move, delete or search. And even if you don't want it, you can upload it to the IA and maybe someone else will use it. Try doing that with your boxes of VHS.
I creates something similar in concept but with different goal. I wanted to be able to watch videos with sponsor block on iPad ideally using Plex.
I found self hosted solution like this but I was very dissatisfied with how that worked
on other hand I wanted to check out loco.rs framework, so I decided to implement my own solution.
basically you are able to add channels/playlists on many many platforms that yt-dlp supports, you can select what should be cut out using sponsor block and you choice how many days you want it (videos older that that are automatically deleted)
It would be nice if the extension wrote them to some shared repository. That way, the videos could be preserved for humanity without Google having a say in it.
Added benefit: every video would have to be archived only once.
My archiving app called HEAP can be configured using a simple apple script and yt-dlp to do this too. And since it's a native macOS app instead of a browser extension, it works via all browsers:
I've had this idea myself so cool to see it implemented.
What I'd really like is a kind of universal web caching backend. So everything I access goes through a cache and I have the option of viewing from cache if something goes offline or changes. I could also mark things as "favourite" so they don't ever expire from the cache. Does such a thing exist?
I don’t really get the purpose of this broadly, because doesn’t YouTube keep videos online unless the creator took them down which is probably not the case 95% of the time? That said for a niche or a high likelihood of a video being removed, or if you really want to be 100% certain it makes sense, but would I be accurate in that statement or am I missing something?
I gave Claude access to supadata YT transcription and obsidian MCP to convert them to "permanent note" format and it's helped tone down my YT addiction a lot
For YouTube videos I feel are worth archiving, I just add them to playlists on my channel, then periodically download my entire channel using a single yt-dlp command (it can keep track of what's already been downloaded).
It started a line of thoughts in me. What if the backend keeps the videos around for a longer period of time, and:
* regularly checks youtube, and whenever one archived video gets deleted from youtube, it advertises the video ID on a specific set of Nostr relays
* have a different browser extension for yt viewers that activates when the user hits a deleted video.
* the backend can stream the video for continuous Bitcoin lightning payment until the stream is kept alive.
Related, but this is something that I would like if browsers gave more importance to.
Literally a few moments ago I was playing a video from a Nitter instance (Twitter proxy), and wanted to save it with Firefox but nope. Firefox already has the video there, I can see in the developer tools that it's playing from a blob. Even if the blob is then modified on-the-fly, why can't I right-click it and save that blob that you already have? If it's a stream, why can't I dump it? You are already playing it, you already have the bytes, you can "cache" the whole playback into a local file and then let me Right Click -> Save that "cache" file. Had to use `yt-dlp`.
Similar with bookmarks. When I bookmark something, it would be nice if the current state of the page were saved as a fallback (or at least have a checkbox for this behavior). You already have the rendered page right there. I can literally do `Ctrl+S` and save this page as it is right now, even with this half-typed comment. The code for saving the current state of a page is already there. And you can already link binary data to a bookmark (favicon), and text data (description, tags, etc). I wouldn't even mind if I have to give up the description field in exchange for a base64-encoded version of the bookmarked page, I'd be willing to accept that additional storage requirement.[1]
Sites like YouTube that actively work to prevent easy downloads and archiving might still need extensions like the one linked in the submission, but browsers don't make it easy to save some data that they already have, and it's difficult enough to contribute to (or patch for yourself) those massive codebases unless you are already familiar with them.
I'm willing to patch things for myself and repackage them for myself, but unfortunately for these things it's easier and more reliable long-term to just do workarounds.
(Disclaimer: I'm using Firefox.)
[1] EDIT: Having history enabled doesn't seem to do save an offline version, either.
Speaking of archiving data, is there a search engine that also returns search results from older (no-longer-existent but archived) web pages? The issue I've had with things like the Wayback Machine is that I need to recall the URL or find some other web page still linking to the (now broken) URL.
42 comments
[ 4.2 ms ] story [ 40.2 ms ] threadtotal 3207312
-rw-r--r-- 1 aa staff 525M Aug 2 09:11 2PMzaym-StM.mov
-rw-r--r-- 1 aa staff 362M Aug 2 09:10 CHbawkGc_os.mov
-rw-r--r-- 1 aa staff 658M Aug 2 09:11 lqR7VV8ftys.mov
~/os/starchive (main)[56daf7] $ ./starachive
Server starting on port 3009...
JSON received: map[videoId:CHbawkGc_os]
Added video CHbawkGc_os to queue. Queue length: 1
Processing video CHbawkGc_os. Remaining in queue: 0
[1]: https://github.com/gsuberland/UMP_Format/blob/main/UMP_Forma...
Transcoded (ouch) or just remuxed to a mov container? Have to investigate.
On top of that... seriously, of all the formats one could choose, MOV?! Might as well choose DivX or RealVideo.
https://github.com/tubearchivist/browser-extension
I really like the WebUI of Tubearchivist itself.
In the 90s my friend's mom would video tape AMC movies. She had 300+ tapes. Maybe she had a few rare ones but now all those movies are available on demand either legally or illegally and in much better quality. Another friend kept all of his 1980s computer magazines (Byte, etc...) and moved these extremely heavy boxes through 30+ years of moves. I doubt he ever opened a single magazine since the moment he saved them. Then they all appeared on The Archive and he finally got rid of them.
To be clear, I have a few youtube videos saved on my local storage. I'm just thinking that saving every video I watch reminds me of the things I've personally over-saved.
Actually that reminds me. I met up with the magazine saving friend recently which is when I verified that he finally got rid of his stash. It made me think about things I'm still saving that if I reflect on I know I will never actually look at. For example I have box of about eight 3.5 inch floppy disks from my Amiga days. The odds that I'm going to get an Amiga or download an Amiga emu and get a drive to read those are close enough to zero that I should throw them away. Similarly I have a book of CD-ROMs of backed up data from the 90s. There's a close to 0% chance that I'm never going to bother look at their contents.
>There's a close to 0% chance that I'm never going to bother look at their contents.
More likely scenario, your children, grandchildren or other family members go through your shit after you pass away and discover stuff about you that perhaps you never wanted to share.
This is something I think about a lot because I don't have a "digital legacy plan."
My general rule is - if I didn't use it for a year, I don't need it. There are obviously some exceptions like a fire extinguisher (which I hope to never use) and digitized photos, which only go through a careful selection.
I think the thing I kept the longest was a Libranet Linux 3.0 CD set because I worked for Libra Computer Systems for a while and this was the release that I helped building. A few years ago I threw it away, I think after I saw someone uploaded it to archive.org. When I'm 60 and want to install it again for good old time's sake I can.
tl;de: if you don't use something for a year, you probably don't need it.
Maybe something a bit more selective than this though!
https://help.archive.org/help/how-do-i-make-a-physical-donat...
Tools to make this easy exist if you already have digital versions.
https://github.com/jjjake/internetarchive
And don’t forget to send a few dollars if and when you can.
https://archive.org/donate
(no affiliation, I just like the public good)
Sometimes you're the person who is uploading them to public archives. Because everybody else threw them all away, and you saved them until the technology made archiving practical enough.
I've been replacing all of my physical media for years, but the reason I can do that now is because other people scan/rip and archive/share the stuff. You also have unique stuff that you may not even know is unique. When you find something in your house that you can't find online, scan it and you're paying everybody back for all of the scanning they did for you.
With the CD-ROMs, you should just glide through them one by one and check if you can find the stuff online. If you can, throw them in the trash. If you can't, copy their contents to a folder, and throw them in the trash. Go through the folder over the next hour or next 20 years (however long it takes to get around to it) and take the things you can't find online that you think somebody might want, and get those things to that somebody (uploading to archive.org is always a good place to start.)
edit: I know for a fact that for a lot of people, uploading somewhere on the internet is their standard pre-deletion ritual.
However, in your examples, the fact those things eventually became available in other forms is not necessarily a counterpoint to your acquaintances having kept them. The specific counterpoint being Marion Stokes.
https://en.wikipedia.org/wiki/Marion_Stokes
I think of it like this:
Automatically save everything and spend time deleting the things I don't want to keep. vs Manually saving everything.
"Don't want to keep" depends on disk space and cluttering up the list as it grows. Disk space is not really a thing. I use to have a friend who spend many hours every week cleaning up his 512 GB drive. He was quite obviously deleting things he wanted to keep but "had to" make choices. I just have enough drives to keep 1 to 3 copies of everything. (The single drive will also fail inevitably)
The clutter still seems to happen even if I make the effort to get rid of things. Organizing it a bit, say at least by date is inevitable.
Therefore there is nothing to be gained by wasting time saving things. It is more time efficient to waste time by deleting only enormous folders that you clearly don't need to keep around.
Hoarding is only an appropriate term if you don't have the space for it. If you have an empty airplane hangar a few boxes of foo isn't hoarding.
I prefer to preserve the original artistic works of the producers, directors and screen writers. It could be I am alone in this endeavor and perhaps I am just a hoarder in that regard. I like to watch movies or listen to songs in the way the artists originally intended rather than the current ways that society has determined I should. The exception of course is when a movie is re-mastered and they improve lighting, sound, resolution, etc...
I hesitate to even imagine how AI will sloppify these great works in the future or how they will re-imagine podcasts. The original art will be lost to time and many people will never experience the original results of the blood sweat and tears that went into making them.
I found self hosted solution like this but I was very dissatisfied with how that worked
on other hand I wanted to check out loco.rs framework, so I decided to implement my own solution.
basically you are able to add channels/playlists on many many platforms that yt-dlp supports, you can select what should be cut out using sponsor block and you choice how many days you want it (videos older that that are automatically deleted)
if you are interested, you can check it out: https://github.com/Szpadel/LocalTube
Added benefit: every video would have to be archived only once.
[1] https://news.ycombinator.com/item?id=37885584
Youtube is an archive like a grocery store is a food archive. [1]
If it was worth watching in the first place, it's worth saving. Reducing the friction of doing so is going to help a lot of people.
(1: I'm getting this quote wrong, what's the actual and attribution??)
https://github.com/ArchiveBox/ArchiveBox
https://apps.apple.com/ca/app/heap-website-full-page-image/i...
What I'd really like is a kind of universal web caching backend. So everything I access goes through a cache and I have the option of viewing from cache if something goes offline or changes. I could also mark things as "favourite" so they don't ever expire from the cache. Does such a thing exist?
So you can make some money on free disk space.
Literally a few moments ago I was playing a video from a Nitter instance (Twitter proxy), and wanted to save it with Firefox but nope. Firefox already has the video there, I can see in the developer tools that it's playing from a blob. Even if the blob is then modified on-the-fly, why can't I right-click it and save that blob that you already have? If it's a stream, why can't I dump it? You are already playing it, you already have the bytes, you can "cache" the whole playback into a local file and then let me Right Click -> Save that "cache" file. Had to use `yt-dlp`.
Similar with bookmarks. When I bookmark something, it would be nice if the current state of the page were saved as a fallback (or at least have a checkbox for this behavior). You already have the rendered page right there. I can literally do `Ctrl+S` and save this page as it is right now, even with this half-typed comment. The code for saving the current state of a page is already there. And you can already link binary data to a bookmark (favicon), and text data (description, tags, etc). I wouldn't even mind if I have to give up the description field in exchange for a base64-encoded version of the bookmarked page, I'd be willing to accept that additional storage requirement.[1]
Sites like YouTube that actively work to prevent easy downloads and archiving might still need extensions like the one linked in the submission, but browsers don't make it easy to save some data that they already have, and it's difficult enough to contribute to (or patch for yourself) those massive codebases unless you are already familiar with them.
I'm willing to patch things for myself and repackage them for myself, but unfortunately for these things it's easier and more reliable long-term to just do workarounds.
(Disclaimer: I'm using Firefox.)
[1] EDIT: Having history enabled doesn't seem to do save an offline version, either.