I remember the first time I was in awe of storage capabilities was roughly 2003 when someone sent me a screenshot of his pirated porn collection, it was in the hundreds of TBs, at that point already, mostly stored on Disc spindles. I often wonder if he's stopped accumulating at some point.
In /r/datahoarders people are abusing Google Drive Quotas for the same purpose, but the capital required doesn't remotely compare to what it seems some people are willing to invest.
I think / hope those are archived as well; e.g. what.cd and waffles had a huge archive of all kinds of rare music, bootlegs, obscure bands, limited releases, etcetera. I hope there's an organization that made sure those were archived.
Are museums exempt from copyright laws for archival purposes?
When I was living in Australia (2007-2012) I had a tv card in my computer so I could watch tv in my room, and I could record another channel while watching. So I used to record shows to mpg to watch later.
I happen to have been doing this since Feb 10 this year, as I've been using a USB DVB card to capture/watch the news. Up to 370 GiB so far, just storing the MPEG-TS streams on hard drives.
Why do I watch the news? I don't like the idea of relying on the news sites here (New Zealand) in getting a brief overview of the current stories.
If you meant why I'm keeping them .. I just don't really like deleting things. If the space usage ends up being impractical, I'll probably just end up reencoding them.
Edit: to clarify, by "the news", I mean the hour-long show (though I skip the sports segments) that's on at 18:00 every day; we don't have the "24-hour" news cycle that seems to be common in the US.
Reddit's /r/datahoarder is full of people archiving all sorts of things that I wouldn't expect to be archived. Recently there was a community effort to archive YouTube's annotations before they were removed.
Link here:
I only watch TV through downloads (get_iplayer & youtube-dl), and I have kept it all [edit: only things I downloaded intending to watch, not all TV]. This is since about 2008:
fs:/volume1/media 14T 8.0T 5.6T 59% /mnt/media
The reason? Why not - disks are cheap. The hard thing is deleting something since I don't know if I might want to watch it again. And at almost no real cost I don't have to make that decision.
Edit: Some stats: 20,269 TV programmes; 11,830 radio programmes; 925 movies
I wish I started doing similar thing way earlier, because now I can't refer to a lot of content that vanished from the net for a number of reasons (creators getting bored, flagged out from youtube by bots that random haters set up, bogus copyright claims from impostors that upload all sorts of stolen things into contentid in order to create a quick buck, UMG taking down videos from music theory channels despite it obviously falling under the Fair Use and so on)
YouTube audio is usually aac. Sometimes I have saved audio as an .aac file without re-encoding. (Not using the same scripts, what I have done typically is get the mp4 and do: ffmpeg -i foo.mp4 -acodec copy foo.aac)
Any audio player worth its salt should play .aac files. I think for the most common types [low-complexity] the patents are expired. (Curious if somebody has one that doesn't.)
Do you maintain any kind of database or additional metadata alongside all of those video files? I only ask because the default filenames generated by youtube-dl don't give you anything more than the title and ID for a particular video and I imagine it's difficult to search among tens of thousands of media files without contextual data about the files.
No, just filenames and whatever metadata is stored inside the file. For get_iplayer this isn't very much unless you have "AtomicParsely" installed too, but I uninstalled AP after it segfaulted once too often. Note for BBC TV programmes you can search on the "PID" (which is part of the filename), eg: try searching for "m000427z".
However I also download subtitles where available so I guess I could search those as they are just text.
Also, it has the --write-* switches, in particular --write-info-json, which allow to store metadata data. in files, and --add-metadata to embed (some) of the metadata in the file itself.
Of course, the metadata is only as good as the site in question provides + the yt-dl plugin handing a particular site supports.
I was familiar with the -o template system for filenames, but wasn't aware of the --write-info-json flag which is very useful to me. It makes a lot more sense than overloading the filenames with usernames and everything else only to parse it out after download.
Oh. If I didn't know any better I would suspect Stephen Wolfram made a blogpost for shamelessly advertising the data science capabilities of Mathematica for the purpose of selling it to data scientists.
I think the blogposts are being generated by a collection of cellular automata that are being created by a learning algorithm with access to that dataset. In other words, Stephen Wolfram.
Transdiffusion, a small UK based group of TV presentation enthusiasts, have been recording and archiving material related to TV logos, idents and presentation since the 1950s.
Anyone know if this has been digitized yet or when it will be available? I am so impressed by this woman’s work. So much of our history is being lost, forgotten, or is inaccessible by the general public.
In the US there is the Vanderbilt Television News Archive, dating to 1968:
The core collection of the Archive consists of regularly scheduled newscasts from ABC, CBS, NBC, CNN and Fox News. Special news reports are recorded in addition to these newscasts, including material from other networks. We record broadcasts as they are televised, provide the widest access allowable within copyright for scholarship and research, and preserve the content for future generations. The database currently includes 1,116,752 records, including abstracts at the story level of regular evening newscasts and catalog records for each special news report.
The archive itself is not publicly viewable online, but the rundowns are (including national advertising spots running with the programmes), and are text searchable. Trends in programme length, topics, and to some extent quality and bias, are evident. It's a hugely useful, little-known, treasure.
Example; every nightly news story on "privacy" since 1968:
Archives, including (again, text-searchable) close captioning, here at the Internet Archive (https://archive.org/details/tv), are hugely useful to researchers and even investigative reporters. Here at WashPo:
It's gonna be awesome to learn history in the next 100, 200 or 500 years. Video records for all the major and minor events, archived since the second half of 20th century.
Although they may complain that it's "just a video", not a virtual reality or something. Also, we need to last that long.
Now that video forgery is easy [1], I think the mountain of video footage will make learning history in a few hundred years painful and difficult.
How would you have any idea what happened and what didn't, from the 21st century on?
Yes, historians worked without video or photography for centuries, but most people aren't as rigorous as a historian, and it's going to take a long time for our cultural norms to adjust to video being "default untrusted".
Furthermore, the written word and eyewitness testimony aren't nearly as important in our culture as they used to be, precisely because of the "pics/video or it didn't happen" mindset. Again, that's what historians have relied on for centuries, but it's considered unacceptable now in favor of technology that's just become completely untrustable. There won't be trustable records for people studying our near-future.
Finally, since we've all given up on "primitive" technology like dead trees, most information is stored digitally now. That's great for convenience but terrible for long-term archives [2]. Digital information will vanish if not tended actively and carefully, and historians have long relied on physical artifacts like letters that largely don't exist any more.
Our culture is a nightmare for future historians.
Edit: expanded on why it will be hard to study history in a few centuries
I am not a blockchain enthusiast but proof of authenticity is a use case that I've heard. It could be used to prove (for example), that a specific video existed in it's current form at a certain point in time.
I would. The market has not demanded this yet, so why on earth would they waste the resources on building it?
I might wish they did, but my experience of life says that product won't exist it until it looks profitable, and maybe not even then.
That aside, watermarking video from cameras buys you nothing in terms of trust.
Not all cameras will have the feature, so videos without it will not be known to be untrustworthy. For cameras that have it, the owners will be able to forge videos using them one way or another, so videos with watermarks will not be known trustworthy.
Chains of trust are fundamentally about humans, and now that forging videos is trivial, we will never be able to safely assume in a vacuum that one is not forged.
It's back to credible eyewitnesses, like we had to rely on for most of history.
With a low definition video that's probably challenging. Maybe it's possible to assert correct physics happening. To my knowledge at least 5-10 years ago Quantum Chromodynamics simulations were so prohibitively expensive that they were simulated on 20x20x20 ("pixel") grids. There aren't that many studies about (boring) Newtonian mechanics but even that provides its complexity challenges. Same holds probably true for most liquids.
The US never used Teletext, but we do have closed captioning, which should be recoverable and would provide some machine-processable date. Much less than Teletext would, though, for sure.
It's funny you should mention this - it's exactly what is on my mind at the moment too. I'm trying to resist the temptation to trawl ebay for peoples old VHS collections (search "used VHS" gives good results) and start building an archive.
Pity there doesn't yet seem to be any centralized effort to this though. I can imagine a wayback machine-like archive being an amazing resource.
64 comments
[ 5.0 ms ] story [ 128 ms ] threadhttps://archive.org/details/tvnews
[0] https://www.youtube.com/watch?v=IMVTB2aVUg0
In /r/datahoarders people are abusing Google Drive Quotas for the same purpose, but the capital required doesn't remotely compare to what it seems some people are willing to invest.
Are museums exempt from copyright laws for archival purposes?
If you meant why I'm keeping them .. I just don't really like deleting things. If the space usage ends up being impractical, I'll probably just end up reencoding them.
Edit: to clarify, by "the news", I mean the hour-long show (though I skip the sports segments) that's on at 18:00 every day; we don't have the "24-hour" news cycle that seems to be common in the US.
https://old.reddit.com/r/DataHoarder/comments/b7imx9/youtube...
Edit: Some stats: 20,269 TV programmes; 11,830 radio programmes; 925 movies
Any audio player worth its salt should play .aac files. I think for the most common types [low-complexity] the patents are expired. (Curious if somebody has one that doesn't.)
However I also download subtitles where available so I guess I could search those as they are just text.
https://github.com/ytdl-org/youtube-dl/blob/master/README.md...
Also, it has the --write-* switches, in particular --write-info-json, which allow to store metadata data. in files, and --add-metadata to embed (some) of the metadata in the file itself.
Of course, the metadata is only as good as the site in question provides + the yt-dl plugin handing a particular site supports.
https://blog.stephenwolfram.com/2012/03/the-personal-analyti...
He captured every keystroke (over 100 million) on his computer.
Which is a great display of Mathematica's power over people's wallets
Did you ever digitize them?
I haven't looked too closely, but what's this then:
https://120minutes.tylerc.com/
I always thought the original draw of Beavis and Butthead was the MST3K approach to music videos, but those are often stripped from current versions.
The State, WKRP, and The Wonder Years all lost great original soundtracks too:
https://www.techdirt.com/articles/20110801/11062015346/new-b...
https://news.ycombinator.com/item?id=19471312
https://www.transdiffusion.org/
The core collection of the Archive consists of regularly scheduled newscasts from ABC, CBS, NBC, CNN and Fox News. Special news reports are recorded in addition to these newscasts, including material from other networks. We record broadcasts as they are televised, provide the widest access allowable within copyright for scholarship and research, and preserve the content for future generations. The database currently includes 1,116,752 records, including abstracts at the story level of regular evening newscasts and catalog records for each special news report.
https://tvnews.vanderbilt.edu
https://tvnews.vanderbilt.edu/about
The archive itself is not publicly viewable online, but the rundowns are (including national advertising spots running with the programmes), and are text searchable. Trends in programme length, topics, and to some extent quality and bias, are evident. It's a hugely useful, little-known, treasure.
Example; every nightly news story on "privacy" since 1968:
https://tvnews.vanderbilt.edu/search?utf8=&query=privacy&but...
Archives, including (again, text-searchable) close captioning, here at the Internet Archive (https://archive.org/details/tv), are hugely useful to researchers and even investigative reporters. Here at WashPo:
https://www.washingtonpost.com/politics/2019/04/08/televisio...
Andrew Tyndall, independent news analyst, has monitored TV news since 1988, publishing review and data at The Tyndall Report:
http://tyndallreport.com
Although they may complain that it's "just a video", not a virtual reality or something. Also, we need to last that long.
How would you have any idea what happened and what didn't, from the 21st century on?
Yes, historians worked without video or photography for centuries, but most people aren't as rigorous as a historian, and it's going to take a long time for our cultural norms to adjust to video being "default untrusted".
Furthermore, the written word and eyewitness testimony aren't nearly as important in our culture as they used to be, precisely because of the "pics/video or it didn't happen" mindset. Again, that's what historians have relied on for centuries, but it's considered unacceptable now in favor of technology that's just become completely untrustable. There won't be trustable records for people studying our near-future.
Finally, since we've all given up on "primitive" technology like dead trees, most information is stored digitally now. That's great for convenience but terrible for long-term archives [2]. Digital information will vanish if not tended actively and carefully, and historians have long relied on physical artifacts like letters that largely don't exist any more.
Our culture is a nightmare for future historians.
Edit: expanded on why it will be hard to study history in a few centuries
1: https://en.wikipedia.org/wiki/Deepfake 2: http://howicode.nateeag.com/data-preservation.html
I might wish they did, but my experience of life says that product won't exist it until it looks profitable, and maybe not even then.
That aside, watermarking video from cameras buys you nothing in terms of trust.
Not all cameras will have the feature, so videos without it will not be known to be untrustworthy. For cameras that have it, the owners will be able to forge videos using them one way or another, so videos with watermarks will not be known trustworthy.
Chains of trust are fundamentally about humans, and now that forging videos is trivial, we will never be able to safely assume in a vacuum that one is not forged.
It's back to credible eyewitnesses, like we had to rely on for most of history.
Pity there doesn't yet seem to be any centralized effort to this though. I can imagine a wayback machine-like archive being an amazing resource.