Well there is sowmthing to be said for simpler formats, that can be read by basic engineering skills, compared to for example storing something on a flash drive the requires millions of lines of code to read data of.
For the same reason, I print all important photos for posterity. Who knows if apple / google cloud photo library will be around in 50 years when I grow old? I can make sure my printed material remains much more easily.
I don’t disagree with your general point, but for this specific assertion:
> something on a flash drive the requires millions of lines of code to read data of
Does it really require millions of lines? Even if you’re just targeting that one model of drive, rather than writing a general purpose SATA/USB kernel stack? If what’s on there is, say, JPEG images, libjpeg (plus its dependencies in glibc, I guess) isn’t “millions of lines” either. Where are these millions of lines?
(I’m speaking as someone who has written a unikernel with disk IO and graphics support. It was far less than a million lines!)
Compare the BBC Domesday Project to the original Domesday Book from 1085. One can be read today, the other not so much. You ask what the folks at BBC were thinking, did they never consult an archival librarian?
I played with that when it first came out. We had the discs and the player, it was very impressive but even then I knew that there was no way that it would last for even a decade.
It was a puny bit of data for todays standards (about 500M in total), but of the day it was incredible. Back then a mainframe winchester pack with the same capacity would be roughly the size of a washer-dryer combination.
The data not being publicly accessible at the moment is more of a copyright rather than a technical issue.
>For the same reason, I print all important photos for posterity. Who knows if apple / google cloud photo library will be around in 50 years when I grow old? I can make sure my printed material remains much more easily.
You can be sure the JPEG and BMP and PNG format will be, and plain text and basic html as well.
So you don't need to print anything, just to have plenty of backup and repeat the backup process every few years (which takes all of 2-3 minutes even for 1000s of images).
> It's not like printed photos can't be burned, scratched, lost, discolor, soak, stained and many other things besides...
Not that I disagree with you, but this is an interesting point. I've heard archivists argue that with simpler analog storage, the curve for corrupt bits vs interpretability is smoothish, but with digital storage it's often a step function where it's like "whoops, a few bits flipped, can't read at all anymore"
> with digital storage it's often a step function where it's like "whoops, a few bits flipped, can't read at all anymore"
Digital anything, really. Analog signals degrade somewhat gracefully—you have to explicitly build error correction into a digital signal.
A relatively recent example of a bungled transition from analog to digital: when the US switched to digital television in 2009, my family suddenly and completely lost many local/lower-powered/more distant stations that we used to be able to get with just a little static or distortion. And the channels we continued to get wou— —ut out occasio— —in a really a—oying wa— and became unwatchable (well, unlistenable).
>but with digital storage it's often a step function where it's like "whoops, a few bits flipped, can't read at all anymore"
That's not inherent in digital copies though. It depends on how much redundancy one adds. We've had self-correction codes that sacrifice size for redundancy and reliability since forever, even in simple formats like ZIP and RAR archives.
And with digital one can always make like 10 copies in the same disk too. If one is unreadable due to single bit flipped or a whole sector, some of the others will still be fine.
I interpreted your statement of "just to have plenty of backup" as having plenty of storage space rather than multiple copies / redundant backup.
The problem I have had is that when one is trying to duplicate/copy > 1 Tb of data across tens, if not thousands of files, that the bitrot is not always easy to find. Windows and OS X / MacOS treat bitrot differently and, it seems, across different file types.
And yeah, lots of issues with scratching, discolouration, lost, etc., of any printed or optical media. That's why a hybrid physical-digital solution is a good half-way house. Same goes for encryption keys or one's crypto wallets, best to hedge one's bets over physical and digital media.
>I interpreted your statement of "just to have plenty of backup" as having plenty of storage space rather than multiple copies / redundant backup.
Yeah, plenty of backup copies. One can always die on you, and even just 2 is stretching it. And the working ones should be verified and copied to new copies every 3-4 years (but in overall, it's not a big deal).
>The problem I have had is that when one is trying to duplicate/copy > 1 Tb of data across tens, if not thousands of files, that the bitrot is not always easy to find
Yes, but for the order of 100s or 1000s of images (equivalent to those one would be able to print) this shouldn't be a problem though. Tiny HDs, SSDs, and USB drives + MD5 (or better) hashes/CRC codes should do.
As a kid I got to play with microfilm and microfiche at the library while my mom did research. I don't have a lot of pleasant memories from childhood, but this was an exception.
My mind was intrigued and blown by this.
Thinking back on the experience, it was incredible. I could go grab a copy of a newspaper from any particular day in any year. I remember reading the actual newspapers from the early 20th century. Big historical events that are in history books, really take on a different meaning to you when you read them as they were reported in near-real-time. The ability of microfilm to snapshot history in a way different than say viewing a jpg or png, is fascinating.
If you are ever bored, and can find a library that will give you access, you should definitely check it out. It's an experience I think all hackers should have. Such simple yet powerful machines, built on a highly non-technical system of convention. Very cool. Sorry for the long walk down nostalgia lane.
You will find good holdings of national and international periodicals in university and higher ed libraries, in Frankfurt/Main, Marburg, Darmstadt, Wiesbaden, Gießen, Kassel and so on. Also, the German National Library (Deutsche Nationalbibliothek) is in Frankfurt/Main.
I wonder if there are any non-profit archival bodies that would partner with a tech org on such a venture. Finding a non-profit tech org that could do it in a sound way might be challenging*
Perhaps a Google project like the books scanning they did? That would be really cool.
* it doesn't have to be non-profit tech org of course (I did mention google's book project after all), but finding a for-profit that would make things freely available might be a challenge in itself since the costs would not likely be low, either for development or infrastructure/maintenance.
My local newspaper (in Sweden) does this. All subscriber has access on the web to the archive back to 1910. Very useful and fun to browse ex. birthdays or things from the world wars.
If you have a library card, you can get past many of those paywalls, at least in the U.S.
The library systems in the last two places I lived both subscribed to services where when you put your library card number in, you could browse the archives of hundreds of newspapers, magazines and journals. Some of them going back into the 1800's.
There's also "Chronicling America" - it's a federal government (Library of Congress) web site that has full page scans of 500 different newspapers from 1836-1922. Free and open to everyone in the world: https://chroniclingamerica.loc.gov
A few years ago I explored this concept and made a CMS for exactly these kinds of situations, also textbook reading, and in general any kind of endless/near-endless content that benefits from seamless space/time organization and navigation. See https://eagrereader.appspot.com/ or my website itself https://dougkoellmer.com for two uses of it.
You could make something similar, but it won’t be nearly the same experience. The speed at which you can scroll through enormous amounts of visual data is staggering. Far more responsive than your average Google Maps-style tiled image app.
If it ever feels like the news is getting you down, I strongly recommend substituting your normal news habit for watching old news bulletins on YouTube. Try it for a week or two - if you feel the urge to check the news, go and watch an NBC news bulletin from 1988 or a BBC news bulletin from 1991 instead. Lots of people have uploaded complete bulletins, which I'd recommend over isolated clips.
I found it to be a revelatory experience. The news, ultimately, is just stories. It's not a definitive or objective account of what's happening in the world, it's just industrialized gossip with a veneer of legitimacy. Without the sense of urgency that comes with newness, it's often remarkably dull. The news shows you the variance, but not the trend; someone who caught up on current events only once a year would probably have a better sense of where the world is headed than someone who checks the news every day. Some stories that were huge dramas at the time are now long forgotten; some stories that barely registered at the time eventually turned out to be part of a crucial change in society.
I get a lot of flak for it, but this is why I stopped following the daily news. So much of it is noise. People are flabbergasted that I am not following the latest turn in some months long drama, but it is precisely because it is months long that I don’t need to follow the daily changes. My hope is that it leaves me better able to filter the noise. I couldn’t say if it is working, but it feels like it is.
> The news, ultimately, is just stories. It's not a definitive or objective account of what's happening in the world, it's just industrialized gossip with a veneer of legitimacy
Or it may be that people have a hard time to figure out what matters at any given moment in time - you never quite know how a news item will play out - will it become part of History or will it turn out to be just another detail? go figure when it is happening...
Maybe grandparent is just right: when it is all happening people may be more honest than after the fact, when positions are clear and everyone has to justify his own stake in the grand puzzle.
>Or it may be that people have a hard time to figure out what matters at any given moment in time
It's called "the news", not "the importants". Unusual, shocking and lurid stories are much more likely to be newsworthy, but by definition they aren't representative of what is actually happening in the world. Our perception of the world is much more strongly influenced by emotive anecdotes than cold hard facts. Violent crime rates in the western world have been significantly declining for decades, but most people believe that the opposite is true; multiple studies have found a correlation between news consumption and the perception of high crime. It's very difficult for the news media to usefully communicate important but gradual trends - rising incomes across the developing world, wage stagnation in the developed world, the impact of climate change, demographic shifts, the erosion of civil society etc.
I think that most journalists are well-intentioned and genuinely believe that they are seekers of truth, but they operate in a system that reflects our innate perceptual biases, our acquired cultural biases and the commercial exigencies of media organisations.
Another useful experiment is to exclusively consume another country's news media for a week or two, preferably a country you know very little about. A week of reading a Ugandan or Nigerian newspaper provides a very different perspective on world events.
I was blown away by two very similar things. The first was microfilm in my local library. The second was putting the needle on a record and hearing the music without the amplifier being on. In both cases it's just so obvious how it works yet absolutely amazing that it works so well.
I once read the New York Times from 1917 to 1922 on microfilm, guided somewhat by an index. Every article was about how the government of "Mr. Lenine and Mr. Trotzky" was on the verge of collapse. It gave a very false picture - well, the Times still gives a false picture of Russia today.
If you have access to microfilm, instructive is an article from June 23, 1918 titled "Lenine ready to resign". Of course this was another false story - although he was shot two months later which caused an illness in him that caused him to slowly withdraw until his death five years after.
But there are many stories of this type. Ultimately, however, dominant communist power in Russia ended because Russian communists themselves decided to hold Multi-Party elections (to be even more tangential - some think Lenin would have had a coalition government of Bolsheviks and left socialist revolutionaries if he had not been shot and retired to Gorki).
You don't need microfilm to read the NY Times; the entire archive is available on their website.
> I once read the New York Times from 1917 to 1922 on microfilm
You read ~1800 newspapers from 100 years ago?
Anyway, can you provide a basis that substantiates any claim in the parent? I don't mean one link to one article, which tells us nothing about an overall trend. Ironically, there is nothing like HN for unsubstantiated claims about Russia like the parent, which are ceaseless.
> You don't need microfilm to read the NY Times; the entire archive is available on their website.
This was not the case when I did this thirty years ago.
> You read ~1800 newspapers from 100 years ago?
I concur with your observation that if you stop reading that sentence after the first few words, it makes less sense than if you read the whole sentence.
> Anyway, can you provide a basis that substantiates any claim in the parent?
I already did.
> I don't mean one link to one article
August 13, 1918 NY Times front page - article headline "Red leaders flee, reach Kronstadt, entire Bolshevist government escaping from Moscow". Of course history shows us that the Bolshevist government did not collapse in August 1918.
> there is nothing like HN for unsubstantiated claims about Russia like the parent, which are ceaseless
I have received ceaseless upvotes for my comments on the lies the Times has told about Russia over the past century, and I wish to thank the товарищи who gave them to me - from wherever they are in the world.
> August 13, 1918 NY Times front page - article headline "Red leaders flee, reach Kronstadt, entire Bolshevist government escaping from Moscow". Of course history shows us that the Bolshevist government did not collapse in August 1918.
That doesn't establish anything in the original claim. First, the quote says the government left Moscow; it didn't say the government collapsed permanently. Similarly, saying De Gaulle fled France doesn't say that France's free government had collapsed permanently. It would have been false to say Mao and the Chinese Communists didn't flee to northwest China (the Long March); should the newspaper have reported that they were in Beijing? Are you saying the Bolshevik government didn't leave Moscow? Can you provide a basis for that? Second, it's just one headline and you had claimed a trend that lasted for years; one headline, even if mistaken, is not a sign of bias; nobody would say that newspapers are perfect and errors happen for many reasons. Finally, it's from a century ago; I don't see what it says about anything current; does the Russian government of 100 years ago tell us about the current organization? Does IBM of 100 years ago tell us how the current organization functions?
The Times's highly inaccurate coverage of the Russian Revolution was an inspiration for Walter Lippmann's commentary on improving media accuracy, in the first decades of the 20th century:
The analysis shows how seriously misleading was the Times by its reliance on the offical purveyors of information. It indicates that statements of fact emanating from governments and the circles around governments cannot be taken as judgements of fact by an independent press. They indicate opinion, they are controlled by special purpose, and they are not trustworthy news.
Even more problematic than official sources, the authors continue, are the semi-official anyomous reports often relied upon.
All the more reason to shun official Information Ministers and Spokesmen, press briefings, and the like.
The quote comes from "A Test of the News", the researchers were Walter Lippman and Charles Merz, the publication The New Republic, and the date, August 4, 1920.
When I was a kid, the book catalogue for larger libraries was itself on microfiche (smaller libraries used card catalogues, as seen in the opening scene of the original Ghostbusters).
Does any library not have this stuff? It seems absolutely standard for universities, the main/research branches of public library systems, etc. Even my small suburban branch library had a microfilm collection and a few readers.
I grew up in Poland and I have the same experience - I learned about microfilm from spy movies, and am aware of its use in libraries from the Internet.
I haven't been to a university library in many years, so things may be different there (especially with the research requirements of the institution) but at most public libraries the machines are either restricted or absent entirely. It's sad but logical. Demand was quite low.
I think library /systems/ will probably always have such an archive stack in their central repository.
However newer libraries, particularly in affluent areas, are more likely to go to a digital stack since it will be more readily searched and comprehensively indexed.
“Microfilm machines trained people’s eyes to read differently: A blur of rapidly advancing images replaced flipping through pages, a precursor to the transition from reading books to surfing the web. Once we adjusted to the nonlinear reading devices, we wanted to jump around instead of advance through page after page.“
I oft find it interesting how a medium (and/or technology) can fundamentally alter the basic experience of reading.
It was definitely a different way to interact with the medium. Back in the early 80s I helped my mother do some genealogy research and spent many weekends in various libraries going through obituary sections of a lot of major metro papers from 1920-1940. You quickly learned to recognize the 'shape' of the sort of page you were looking for and then would hit the advance button on the machine to spin forward until you hit something that looked like the right section. It was the sort of pattern recognition you probably do today when scrubbing through a video looking for a particular type of scene, but with pages of text. Every paper had its own style of page layout so it took an hour or two to learn what you were looking for when you switched to a different paper, but the process became fairly automatic with a bit of practice.
Human binary search algo for page numbers, you meld with the machine and after a couple dozen hours its creepy how I could advance exactly 143 pages if I so desired by turning the dial just so; not any more surprising than video game accomplishments after a couple dozen hours.
I'm a gen-x so I'm so old my youth predated even cdroms; in fact I got an off brand early cdrom reader in high school as a very expensive birthday gift. So in early high school I spent a couple weeks reading microfilm of Scientific American magazine searching out all the "amateur scientist" columns from the 50s and 60s, some real gold there. I printed out some interesting columns. Still have some of those printouts. The technology to "photocopy from microfilm" is pretty obvious optically, the image is projected onto what amounts to a xerox photocopier. Of course kids these days may not believe fully analogy photocopiers ever existed, but certainly they did, until relatively recently as digital multifunction devices took over. Around the turn of the century all the Amateur Scientist columns were combined into a multimedia cdrom using turn of the century ancient java to navigate tens of thousands of one-page-per-pdf files. I no longer have infrastructure that can run that old of a java version although hand navigation of an image of the cdrom filesystem is possible (the file names aren't too insane) and I no longer own a desktop or laptop with optical drive to read the legacy disk, although I can, for now, scare up a USB cdrom reader if needed.
I lived near a depository library; did you? If you can get access to a federal depository library then you'll have access to microfilm version of complete ancient census records; I spent a long time looking at those and printing out 1800s era census records. My future wife bought "family tree maker" cdrom custom format disks of census data (this is all pre-internet pre-ancestry.com era by at least a decade). Obviously those custom cdroms of census records are useless today unless you have a complete early 90s PC emulator, and searching online works better and there's been several decades of census data released since then, LOL.
I find it fascinating that I grew up before cdroms, then in my youth cdroms would replace "everything", and now cdroms are a dead media. I'm and not really all that decrepit and old outside of SV hiring standards and ageism, I scarcely have a gray hair, maybe one or two, don't even have any grand kids yet. CDroms may have been born and died in my young adult years, but there's still microfilm out there.
I appreciate how the article highlights the longevity of polyester-based microfilm and the simple technology (lamp and magnifier) needed to read it.
But as a digital native myself, I'm disappointed that they failed to emphasize the biggest advantage of digital information storage - the fact that data can be copied perfectly, as many times as you want. Analog microfilm cannot be copied perfectly - the original is always better than the copy. This means the microfilm is a unique artifact that needs to be preserved and taken care of. One disaster, one fire, or one careless handler, and the object is damaged.
With digital technology, we pay a high price in the lines of code to build an application and interface, to specify seemingly opaque binary file formats, to have physical media (floppy, CD, etc.) go obsolete in a decade, and to build complex CPUs - but never forget that what we gain is the tremendous ability to copy perfectly. This is what allows us to share a file with someone and know that their copy is a perfect backup for mine in case mine is lost. This is what allows us to transfer files from old, slow, unreliable media (e.g. floppy disks) to new, fast media (e.g. SSDs) without missing a beat. This is why we can take a sizable collection of a thousand audio CDs and cram them into one convenient hard disk drive (as WAV, not MP3), with the exact same audio quality. This is why we can transmit files over the Internet - we surely can't transmit analog data over a distance without a high risk of degradation.
If we collectively accept that storing information digitally with the ability to make perfect copies is worth the effort, then there are things we can do to improve the status quo. We need to make it easier to find and manage file duplicates - otherwise it's hard to know whether a copied file has been modified or not. We need emphasize building applications on open standards. We need to make file format specifications accessible, well-defined, and easier to understand. We need to make systems that have low implementation complexity (e.g. CPU with few transistors, OS with few lines of code) to show what is the minimum knowledge needed to reboot the technology.
While microfilm may last 500 years, digital data could truly be forever - as long as you copy it periodically.
> the fact that data can be copied perfectly, as many times as you want
Tangentially, this makes me wonder: While microfilm can't be copied perfectly, isn't it true that the source quality (e.g. the crispness of the print) is vastly lower than the resolution of the medium itself? Which seems like it would imply that the "lossy" transformation might in some sense still lossless, at least for a while - like converting a high bitrate mp3 recording of a radio broadcast into a lower bitrate one. Or perhaps the kinds of loss are different. I'm sure I'm confusing a lot of ideas and butchering some terms here.
I think you're trying to say something, but I'm getting lost in the details. Can you elaborate more on what you mean by source quality / crispness of the print / resolution of the medium itself? What items are you comparing? Please be specific.
With arbitrary numbers: if the original paper was printed at 100dpi, and the film captured 1000dpi, then the losses from copying that film might not make a real difference in fidelity.
Sure, so long as you assume the transfer is 100% perfect with no errors or artifacts and can make subsequent copies similarly. In practice, few physical processes can offer that level of assurance, and digital copies can.
In the analog world, you have to think about signal-to-noise ratios instead of DPI. Any given copying technique will add some small but nonzero amount of randomness to the output, which destroys some information.
The way digital copying gets around this problem is by having predefined signal levels that are farther apart than the vast majority of copying error, and writing a new master every time instead of the analog copy.
The first copy is unlikely to, since a random error (i.e. an unintentionally developed silver grain) will only introduce a distortion 1/10th the size of the original smallest detail. But what about after 10,000 copies? Scattering random 1/10th sized blobs all over the original details is still going to do something to the experience.
You're right! Copying to a higher-definition medium should in theory allow for effectively lossless copies.
There is a potential catch, in that analog copies will tend to introduce some amount of physical errors no matter how much the hypothetical increase in definition should allow for perfection. Subsequent copies, no matter how high their quality, will both tend to amplify these errors and introduce new ones. You get a similar result if you repeatedly re-endcode data even in high-fidelity lossy ways - each encode introduces artifacts and amplifies previous artifacts.
It's an intriguing idea! Thank you for bringing it forth for discussion.
You are correct, that digital documents may be copied perfectly. However, I don't think the author "failed to emphasize" that topic because that wasn't the goal of the essay.
It is part of the "Object Lesson" series - "Each Object Lessons project will start from a specific inspiration: an anthropological query, ecological matter, archeological discovery, historical event, literary passage, personal narrative, philosophical speculation, technological innovation—and from there develop original insights around and novel lessons about the object in question." - http://objectsobjectsobjects.com/about/
The object in question is microfilm readers, and the things we might learn from it. Not digital technology, nor the reasons why microfilm went by the wayside.
I think you can copy microfilm perfectly, as long as you know the smallest distinguishable feature which the original microfilm archiving process could create. You just have to sample the microfilm at twice that spatial frequency, by the Nyquist theorem, right?
That only covers the errors due to aliasing. There's still going to be additive noise introduced during the transfer that will build up each time the content is transferred.
Even sampling above the Nyquist rate, you can only reconstruct a signal perfectly if the samples themselves are perfectly precise - i.e., their binary or decimal representations have infinite digits. In practice the samples are truncated to a given sampling depth, and the difference is regarded as noise.
In practice though, I think you could take your idea even further into the realm of lossy compression. If you're scanning newsprint for instance, the vast majority of the page is either 1) text in a known font or 2) images rendered through halftoning, that is a regular pattern of circles wih some finite range of radii. Conceivably a system intelligent enough to recognise and encode this information might even be considered a higher fidelity scan than a simple photo, as it will repair incidental defects from the printing process.
But digital copies ain't always perfect - you forget about bit rot [0] unfortunatelly.
One bit flip on stream of compressed picture or archive distorts remaining result.
Entire digital world is founded on analog physical world with all its imperfections.
Of course there are solutions to mitigate them: checksuming, redundancy, error correction. But how often one checks checksum hash on copy? Edit: CRC doesn't count as it is too weak.
I agree on open standards and deduplication, I hope content addresable archiving will get more mainstream, IPFS brings big hopes.
Yes, if one was going to reliably archive information they wouldn't use FAT32 on a single spinning disk. Does this really need to be pointed out? The point is digital gives you ways to have perfect copies, correct loss, and the ability to transfer to the next storage medium with no decay. Things analog could never achieve.
Hybrid analog could achieve this. Film could have a picture printed on it, but instead of going for maximum resolution (down to grain level) it could have tiny tiny pixels, not visible to the eye, but readable by a computer. A kind of steganography, if you will.
The data encoded in the steganography, could be any data of course, but for our purposes, it would be the image itself.
Such a picture could be copied, with loss, in an analog fashion, but could also be copied perfectly in a digital fashion, by a digital reader capable of reading out the steganography information. Even the analog copy could be digitally read if it was of high enough quality.
Another, more straightforward, way would be to print on the film, first an analog rendition of the picture, then a digitally encoded pattern of the same image, rinse repeat.
But from a future safe perspective, it could be smart to encode the data itself inside the image, as per the first paragraph. Someone copying the images could unwittingly copy the digital data too!
> The data encoded in the steganography, could be any data of course, but for our purposes, it would be the image itself.
This was an intriguing idea, but after I did some back of the envelope rough numbers, I tentatively concluded microform might not encode densely enough to accomplish that. It already has poor fidelity characteristics for photographic data. Right off the bat you are losing lots of information during the reducing process, and the film itself does not have the kind of qualities photographers look for. Color microform is also much more expensive than black and white. So the color film works well for newsprint photos, but for archival grade purposes, we're likely better off with the original film negatives. For deep storage archives, I could see a use case for microform, and with the encoding you suggested, but not quite at one picture per frame (page).
Steganography relies upon imperceptibly embedding information into the picture, so there is a very low encoding density limit because we're working with analog pictures. Optar at 600 ppi uses an entire page to encode 200 Kb, but microfilm is closer to 300 ppi, so at 100 percent density with no picture, we can expect around 100 Kb per page of Optar-encoded binary data. An average smartphone picture size these days is around 5 MB, closer to 15-50 MB for DSLR pictures.
Optar-encoding the smartphone picture, we are around 50 pages before we scatter it into the picture to steganographically encode it. Already we can see the picture has to be blown up across more than 50 pages and stitched back together. At imperceptible levels of encoding density, I'm guessing we're looking at 500+ pages, maybe. A standard microfiche holds about 98 pages, so we're talking about 6'ish fiches per picture at cellphone grade, much more for DSLR grade picture encoding.
For high quality archival photograph storage, I think current librarians have it right: lots of duplicates of negatives, geographically separated, then service copy negatives off those masters, digitally scan in the most-used pictures off the service copy, and let patrons use the digital copies. Migrate the digital copies until we establish a universally-accepted digital bootstrap mechanism and process for Long Now-span time scales.
Isn't it specific to magnetic media only? (I don't think optical discs susceptible to bit rot. Especially M-discs which are said to last for a thousand years.)
The general concept isn't specific to magnetic media at all, although the underlying mechanisms vary.
Any EPROM/EEPROM/Flash will leak charge from its cells and eventually lose data if not periodically refreshed. Even some mask ROMs develop bad bits over time due to latent manufacturing defects.
Many CD-Rs and DVD-Rs recorded ~15-20 years ago already have uncorrectable errors (due to the dye degrading?). Pressed discs are nominally immune, but may have manufacturing defects that lead to progressive failure (e.g. "laser rot" on LaserDiscs manufactured by certain plants).
Would a good solution for /really/ long term storage be a kind of WORM ROM based on fuses that could be burned to 'program' a given area? (I think some EPROM might be this, but I'm talking about more storage density than even current top end flash drives.) Obviously you'd only want to use this for the LONG term storage, but if it's cheep enough I could see it being useful in the same use cases that existing optical media were and even in roles like backup / audit logs.
One major benefit of microfilm in my book is precisely the simple tech - even after the zombie apocalypse, just about anyone will be able to quickly figure out how to read what's on them. (Yes, 'zombie apocalypse' is just a placeholder for any massively disruptive event...)
(That being said, digital copies would be way more useful here and now; let's do both!)
There was an effort in the 1950s and 1960s to stock major fallout shelters with a set of microfiche on how to make just about everything, along with readers that just needed sunlight. Wonder if any of those are still around.
(This wasn't a joke back then. Everybody remembered the wrecked cities of WWII. They had to be built back up. You might need those instructions.)
I think it's fair to say that American soldiers brought back those memories. My grandfather served in the Army towards the end of the conflict in Europe and, when he would speak about it, he spoke about seeing destruction. (It was not something he spoke about easily or lightly.)
>> Everybody remembered the wrecked cities of WWII
> Is this actually a thing in American memory, given that no American city was bombed in WWII?
To veer off-topic a bit, I think this is an interesting question brings up two important points:
First, as the WWII generation has died out, people seem to have become less averse to war and to care less about the institutions built to prevent it (the UN, EU, international law, etc.). They were built because the survivors of WWII thought war was the worst scourge of humanity, and that another war - with ~1945 technology - could destroy civilization.
Second, the U.S. experienced very little direct harm in WWII, and even the overseas casualties were orders of magnitude less than those of the European and Asian countries (e.g., the U.S. lost 200,000; the USSR lost 20 million). If the worst happens and the U.S. fights another major war, the enemy likely will be able to strike the U.S. with missiles. Even North Korea can do it to a limited degree.
With the loss of the WWII generation, I think Americans have become cavalier about war, and because the WWII didn't affect the U.S. nearly as badly as the Europe and East Asia, they don't conceive of a different kind of war, such as one where missiles rain down on every American city. We can assume the Bay Area would be a priority target, for example; it's hard to imagine what that would be like; it's hard to imagine the consequences of it.
Ideally let's also have various colleges create new versions of the source instructions and collectively create a better modern set to use as bootstrapping (and feed this back as the foundation for future education as well).
> ...but never forget that what we gain is the tremendous ability to copy perfectly.
Yeah, all except the first one. I'm not gonna argue with the gushing passion for all things digital in the rest of your comment, but I will first of all, point out that putting your photos, music, and videos in the cloud is absolutely not an exact copy--cloud providers don't have ToS that cover the compression that they apply to your media. There's no telling how badly they'll compress it to make room for the newest stuff years from now.
Since you offered such an untempered view of digital media as the ultimate achievement of human civilization, I'd like to offer an alternative viewpoint that recognizes that digital copies reduce very deep, meaningful experiences down to very meaningless bits. That's great for newspaper articles, but it's no way to live life. Just yesterday I was sitting in the park under a tree. The winds were blowing, thousands of leaves dancing above, and distantly a church bell began tolling, it's bang-clang rolling and washing through and around the boughs, being buffeted by the wind itself. The boughs are laden with leaves and seeds, and the sound waves are muffled at a turn, coming through at a turn, as this branch shakes and wags between the bells and me. The sun is filtered and flickered, I see the veins of leaves through. The insects buzz. One shield-like bug, as thin as a blade and about the size of a fingernail, lands and waddles around my blanket, its six legs hyper-optimized by evolution. There are perhaps 10,000 such creatures in this field. A trillion generations to produce it. It starts, its wings buzzing a thousand times a second. The hundreds of thousands of blades of grass wag in waves like grain, like carpet, like kelp in a fishtank, and they too, rustle. Could digital media capture this? I move my head to the side, angle it up, cup my ears, and I sense this field...I am absolutely surrounded by information. The amount information is incredible. I could hear the scrapes of the leaves against each other and the audio shadows of them as they blow around in front of this bell! I can hear the bells harmonics, and in my mind I see the sound waves racing round its circumference, overlaid with other harmonics making it jiggle like a waterballoon. The information there is 10^30...10^40. We can't even imagine.
And yet, we are told that VR and digital media will usher us into a wonderful world of entertainment. Pah! Lossless copies. Pah! We cannot even capture a millionth of something like that.
I envision a transparent semi-crystalline substance that can be written to by a laser or two. With the volume of a sugar cube, perhaps, with the longevity of amber. Written down on the molecular level. Formatted to let 3D readers orient themselves to the playback, regardless of considerable distortion.
Those who've seen the 60s George Pal version of "Time Machine" will remember The Traveller spinning a disk on a flat surface to activate a voiced playback. Still visionary.
One big advantage of film (especially transparencies rather than negatives) though is that it can trivially be read, regardless of technology or hardware.
I can view on a light table or project a photographic slide from 1935 just as easily as I can view a slide developed yesterday. On the other hand, it's not a trivial task to read a magnetic tape from the 70's. These days it's not even trivial for me to read a CD, I don't own a CD reader, and if I'm away from my desk I can't even read a USB flash drive on my Macbook Pro.
That's important for casual readers, but less important for historical reconstruction purpose. Consider that reseachers used a micro-CT scan followed by volume cartography analysis to locate the ink within the thoroughly burnt-to-a-crisp En-Gedi scroll [1]. Compared to that, a CD is nothing.
A few tidbits I know about the topic. One of the organizations which helped push microfilm was the Documentation Institute, founded by Watson Davis in 1935, which became the American Documentation Institute in 1937.
(As an aside, "documentation" was the hot term in library science in the 1930s. "Documentation" meaning "collection of informational papers" is only from 1927, says https://www.etymonline.com/word/documentation . As a further aside for those involved in science fairs, Watson Davis also founded "the Science Clubs of America, reaching at one point roughly a million school-age children across the United States; he also was one of the originators of the Westinghouse Science Talent Search".
> The World Congress of Universal Documentation was held from 16 to 21 August 1937 in Paris, France. Delegates from 45 countries met to discuss means by which all of the world's information, in print, in manuscript, and in other forms, could be efficiently organized and made accessible. ...
> The main resolution adopted by the congress proposed that microfilm be used to make information universally available.[8] Watson Davis, chairman of the American delegation and president of the ADI, stated that the volume of information being produced created difficult problems of access and preservation, but that these could be solved by the use of microfilm. ...
> In his address to the Congress, H. G. Wells said that he thought that his idea of the "world brain" was a precursor to the ideas other delegates were proposing, and explicitly linked the projects being discussed to the work of the encyclopédistes:
These are all ideas which percolated through Vannevar Bush's Memex and many others to become the modern internet.
Also, one of the co-founders of the ADI was Atherton Seidell. He published a series of articles in Science promoting the use of microfilm. Eg, from 24 August 1934 - http://science.sciencemag.org/content/80/2069/184 .
I've often hesitated about getting a microfilm or microfiche reader, but what would really get me over the edge is some kind of similar equipment to make microfilm or microfiche to use with the reader. Did such "consumer" equipment ever exist, or did all the libraries and institutions buy premade microfiche or microfilm, with the exception of a few dedicated conversion companies?
Are there any combined devices/stations that can be used to both read AND create either microfilm or microfiche?
Everything one needed was available; it just wasn't practical in the "consumer goods" sense. You needed an apochromatic flat-field lens, generally optimized specifically for the magnification levels involved (since sufficient optimization over a very wide range of focal distances is an extremely difficult thing). The film has to be extremely low sensitivity to have good enough acutance (think ISO 6-12 as "high speed", since the emulsion needs to be extremely thin). You need consistent lighting so you can have a decent idea of what your exposure will be like, and before Polaroid polarization film cropped up, keeping glare under control was a bit of a bugger. You needed a rock-solid setup, since your exposure was going to take a while. The film needed to be kept very flat - there's far less room for any deviation than there would be for taking general photographs of the big world. In a world of silicon wafers, it's hard to remember that keeping pieces of floppy (and often tightly-rolled) plastic flat enough was once one of the hard parts. Processing was pretty critical - a little too long in the soup, and the black bleeds (that's how halftones are created using a line screen). So, you could get it all going with a decent copy stand and lights, the right (expensive) camera and lens and a goodly amount of faff... or you could get a service that had a dedicated process camera that paid for itself through constant usage and craftspeople who do that sort of thing for a living do it for you for a smallish fee. If you had regular archiving needs, the cost of the set-up wasn't horrible - it was certainly cheaper than a process camera for printing plates. It just wasn't a home-and-small-office thing by any normal standards.
Most of it, though, you could do now with an "eBay special" camera and lens and a bit of ortho-litho film. It's still available as sheet film. That would mean red reads as black and non-repro blue is a thing. Panchromatic repro work would be a lot harder since Kodak killed off Technical Pan. Tech Pan was kind of super-fast film, though - you'd use it at "ISO" 20-50, depending on the application - so while it was spectacular for general photography, it was just at the edge of what would be useful for microfiche.
Used Kodak archive writers cost about $30K (not a typo) are quite large, the modular cassettes for film cost about $1K each, film reloads for each cassette cost about $10 each, a roll of film holds like "ten thousand" images, and the machine can burn an image in about one second. All varying by factors of perhaps four depending on individual model. They eat PDF and TIFF (fax) files, also others. Fundamentally its "just a printer". Theoretically a completely hands-off lights-out machine, much like theoretically a laser printer never needs hand-holding... of course laser printers are mfgr'd "fire and forget" whereas the kodak archive writers service contract was something like $5K/yr.
Part of the problem is the target market; this was at a financial services company where the supplier knew they could charge pretty much anything they wanted and it would get passed along as a tiny fraction of the end user customer contract cost. If a $100M financial services contract involves a $5K service contract for some obscure accounting thingy no one really cares. Given that situation is similar to pre-consumer era 3-d printing, I figure a really smart hacker who doesn't care about reproducible results or speed or reliability could squirt something out for a tenth the costs; but even at a tenth, thats still going to be "laser cutter" like costs. Film is never cheap; high res film is gonna be worse, just how it is.
I would settle for an ultraportable USB-attached reader device that scanned microfilm/microfiche, with a draft mode that lets me preview a low resolution real-time display via the USB on an app on the laptop to position the film/fiche, and a high resolution mode for scanning into the app.
I can understand the film/fiche writing machines being hideously expensive, but I have yet to find a reader that is built for the XXI century digital-oriented demographic. Of the readers built recently, they're all expensive, and are designed with firmware, instead of a design orientation of dumb manipulable hardware (preferably with open documentation) driven by modifiable software.
The earliest example of that general technology would probably be the Leica camera, as this was the first time a negative had been designed for enlargement rather than contact printing (or at least, for high degrees of enlargement). This is only really physically reducing the size of the storage device though, not really compression.
In terms of compression as a whole, the first example would probably be military/commercial codebooks. A relatively modern example would be the q-code, where a predetermined sequence of words are encoded as a 3-letter sequence ("QRA" becomes "What is the name of your vessel/station") but I'm sure there are prior examples.
This was a machine capable of storing a terabit of data on write-once photographic film. It wrote to the film using a scanning electron beam, and then developed the film using a fully automated film processing lab, and finally stored the film cartridges in a robotic storage library. Amazingly cool, but required a lot of expert maintenance and diligent cleaning.
And the early versions of the PLATO educational computing system used a computer-controlled microfiche reader to display graphics beyond what was possible at the time.
It's as though we got so busy using all of this stuff to do ordinary things that we kept making the same thing (GUI, net) better and better, but forgot to keep trying new , richer ideas. As Plato attempted to do. That came out of the U. of Illinois. Where'd the visionary academics go?
I believe that the idea that the end of some avenue of human ingenuity has been reached, and your implicit statement of differing values between the past and present, are both perennially recurrent throughout history. It's not necessarily a bad thing if today's academics are pursuing things that are not among your interests, and it's more likely than not that the speed of the dissemination of ideas, the recognition of their value, and their broad application has not appreciably changed in recent decades (assuming that it's more meaningful to talk about that in terms of a "March of Progress" than a "Random Walk of Progress"). I would think it most reasonable to assume that the academics are where they have always been, doing the same arcane and incomprehensible things that academics have always done, and that any appearance to the contrary was a fault of my own perspective.
Newer microfilm may last 500 years, but the older stuff, especially Kalvar film, has serious life problems. Kalvar film gives off volatiles which corrode everything around it except Kalvar film.
107 comments
[ 3.3 ms ] story [ 138 ms ] threadEdit: Even after disabling ad block, still says that there is no connection. Even worse, just after I actually buy a copy of their magazine.
For the same reason, I print all important photos for posterity. Who knows if apple / google cloud photo library will be around in 50 years when I grow old? I can make sure my printed material remains much more easily.
> something on a flash drive the requires millions of lines of code to read data of
Does it really require millions of lines? Even if you’re just targeting that one model of drive, rather than writing a general purpose SATA/USB kernel stack? If what’s on there is, say, JPEG images, libjpeg (plus its dependencies in glibc, I guess) isn’t “millions of lines” either. Where are these millions of lines?
(I’m speaking as someone who has written a unikernel with disk IO and graphics support. It was far less than a million lines!)
It was a puny bit of data for todays standards (about 500M in total), but of the day it was incredible. Back then a mainframe winchester pack with the same capacity would be roughly the size of a washer-dryer combination.
The data not being publicly accessible at the moment is more of a copyright rather than a technical issue.
You can be sure the JPEG and BMP and PNG format will be, and plain text and basic html as well.
So you don't need to print anything, just to have plenty of backup and repeat the backup process every few years (which takes all of 2-3 minutes even for 1000s of images).
There is something to be said for tangible, archival-paper printed photographs.
Maybe printing the top 100 photos each year is a good middle ground, but to each their own :)
That's why I advocated for multiple copies -- and in a redundant format with corrections codes even better.
It's not like printed photos can't be burned, scratched, lost, discolor, soak, stained and many other things besides...
Not that I disagree with you, but this is an interesting point. I've heard archivists argue that with simpler analog storage, the curve for corrupt bits vs interpretability is smoothish, but with digital storage it's often a step function where it's like "whoops, a few bits flipped, can't read at all anymore"
Digital anything, really. Analog signals degrade somewhat gracefully—you have to explicitly build error correction into a digital signal.
A relatively recent example of a bungled transition from analog to digital: when the US switched to digital television in 2009, my family suddenly and completely lost many local/lower-powered/more distant stations that we used to be able to get with just a little static or distortion. And the channels we continued to get wou— —ut out occasio— —in a really a—oying wa— and became unwatchable (well, unlistenable).
That's not inherent in digital copies though. It depends on how much redundancy one adds. We've had self-correction codes that sacrifice size for redundancy and reliability since forever, even in simple formats like ZIP and RAR archives.
And with digital one can always make like 10 copies in the same disk too. If one is unreadable due to single bit flipped or a whole sector, some of the others will still be fine.
The problem I have had is that when one is trying to duplicate/copy > 1 Tb of data across tens, if not thousands of files, that the bitrot is not always easy to find. Windows and OS X / MacOS treat bitrot differently and, it seems, across different file types.
And yeah, lots of issues with scratching, discolouration, lost, etc., of any printed or optical media. That's why a hybrid physical-digital solution is a good half-way house. Same goes for encryption keys or one's crypto wallets, best to hedge one's bets over physical and digital media.
Yeah, plenty of backup copies. One can always die on you, and even just 2 is stretching it. And the working ones should be verified and copied to new copies every 3-4 years (but in overall, it's not a big deal).
>The problem I have had is that when one is trying to duplicate/copy > 1 Tb of data across tens, if not thousands of files, that the bitrot is not always easy to find
Yes, but for the order of 100s or 1000s of images (equivalent to those one would be able to print) this shouldn't be a problem though. Tiny HDs, SSDs, and USB drives + MD5 (or better) hashes/CRC codes should do.
Thinking back on the experience, it was incredible. I could go grab a copy of a newspaper from any particular day in any year. I remember reading the actual newspapers from the early 20th century. Big historical events that are in history books, really take on a different meaning to you when you read them as they were reported in near-real-time. The ability of microfilm to snapshot history in a way different than say viewing a jpg or png, is fascinating.
If you are ever bored, and can find a library that will give you access, you should definitely check it out. It's an experience I think all hackers should have. Such simple yet powerful machines, built on a highly non-technical system of convention. Very cool. Sorry for the long walk down nostalgia lane.
Perhaps a Google project like the books scanning they did? That would be really cool.
* it doesn't have to be non-profit tech org of course (I did mention google's book project after all), but finding a for-profit that would make things freely available might be a challenge in itself since the costs would not likely be low, either for development or infrastructure/maintenance.
The library systems in the last two places I lived both subscribed to services where when you put your library card number in, you could browse the archives of hundreds of newspapers, magazines and journals. Some of them going back into the 1800's.
There's also "Chronicling America" - it's a federal government (Library of Congress) web site that has full page scans of 500 different newspapers from 1836-1922. Free and open to everyone in the world: https://chroniclingamerica.loc.gov
It's open source at https://github.com/dougkoellmer/swarm if anyone wants to throw money at me to continue developing it. :)
I found it to be a revelatory experience. The news, ultimately, is just stories. It's not a definitive or objective account of what's happening in the world, it's just industrialized gossip with a veneer of legitimacy. Without the sense of urgency that comes with newness, it's often remarkably dull. The news shows you the variance, but not the trend; someone who caught up on current events only once a year would probably have a better sense of where the world is headed than someone who checks the news every day. Some stories that were huge dramas at the time are now long forgotten; some stories that barely registered at the time eventually turned out to be part of a crucial change in society.
https://www.youtube.com/watch?v=P4-_85x5k3s
Or it may be that people have a hard time to figure out what matters at any given moment in time - you never quite know how a news item will play out - will it become part of History or will it turn out to be just another detail? go figure when it is happening...
Maybe grandparent is just right: when it is all happening people may be more honest than after the fact, when positions are clear and everyone has to justify his own stake in the grand puzzle.
It's called "the news", not "the importants". Unusual, shocking and lurid stories are much more likely to be newsworthy, but by definition they aren't representative of what is actually happening in the world. Our perception of the world is much more strongly influenced by emotive anecdotes than cold hard facts. Violent crime rates in the western world have been significantly declining for decades, but most people believe that the opposite is true; multiple studies have found a correlation between news consumption and the perception of high crime. It's very difficult for the news media to usefully communicate important but gradual trends - rising incomes across the developing world, wage stagnation in the developed world, the impact of climate change, demographic shifts, the erosion of civil society etc.
I think that most journalists are well-intentioned and genuinely believe that they are seekers of truth, but they operate in a system that reflects our innate perceptual biases, our acquired cultural biases and the commercial exigencies of media organisations.
Another useful experiment is to exclusively consume another country's news media for a week or two, preferably a country you know very little about. A week of reading a Ugandan or Nigerian newspaper provides a very different perspective on world events.
If you have access to microfilm, instructive is an article from June 23, 1918 titled "Lenine ready to resign". Of course this was another false story - although he was shot two months later which caused an illness in him that caused him to slowly withdraw until his death five years after.
But there are many stories of this type. Ultimately, however, dominant communist power in Russia ended because Russian communists themselves decided to hold Multi-Party elections (to be even more tangential - some think Lenin would have had a coalition government of Bolsheviks and left socialist revolutionaries if he had not been shot and retired to Gorki).
> I once read the New York Times from 1917 to 1922 on microfilm
You read ~1800 newspapers from 100 years ago?
Anyway, can you provide a basis that substantiates any claim in the parent? I don't mean one link to one article, which tells us nothing about an overall trend. Ironically, there is nothing like HN for unsubstantiated claims about Russia like the parent, which are ceaseless.
This was not the case when I did this thirty years ago.
> You read ~1800 newspapers from 100 years ago?
I concur with your observation that if you stop reading that sentence after the first few words, it makes less sense than if you read the whole sentence.
> Anyway, can you provide a basis that substantiates any claim in the parent?
I already did.
> I don't mean one link to one article
August 13, 1918 NY Times front page - article headline "Red leaders flee, reach Kronstadt, entire Bolshevist government escaping from Moscow". Of course history shows us that the Bolshevist government did not collapse in August 1918.
> there is nothing like HN for unsubstantiated claims about Russia like the parent, which are ceaseless
I have received ceaseless upvotes for my comments on the lies the Times has told about Russia over the past century, and I wish to thank the товарищи who gave them to me - from wherever they are in the world.
That doesn't establish anything in the original claim. First, the quote says the government left Moscow; it didn't say the government collapsed permanently. Similarly, saying De Gaulle fled France doesn't say that France's free government had collapsed permanently. It would have been false to say Mao and the Chinese Communists didn't flee to northwest China (the Long March); should the newspaper have reported that they were in Beijing? Are you saying the Bolshevik government didn't leave Moscow? Can you provide a basis for that? Second, it's just one headline and you had claimed a trend that lasted for years; one headline, even if mistaken, is not a sign of bias; nobody would say that newspapers are perfect and errors happen for many reasons. Finally, it's from a century ago; I don't see what it says about anything current; does the Russian government of 100 years ago tell us about the current organization? Does IBM of 100 years ago tell us how the current organization functions?
The analysis shows how seriously misleading was the Times by its reliance on the offical purveyors of information. It indicates that statements of fact emanating from governments and the circles around governments cannot be taken as judgements of fact by an independent press. They indicate opinion, they are controlled by special purpose, and they are not trustworthy news.
Even more problematic than official sources, the authors continue, are the semi-official anyomous reports often relied upon.
All the more reason to shun official Information Ministers and Spokesmen, press briefings, and the like.
The quote comes from "A Test of the News", the researchers were Walter Lippman and Charles Merz, the publication The New Republic, and the date, August 4, 1920.
Plus ça change, plus c'est la même chose.
https://archive.org/stream/LippmannMerzATestoftheNews/Lippma...
Does any library not have this stuff? It seems absolutely standard for universities, the main/research branches of public library systems, etc. Even my small suburban branch library had a microfilm collection and a few readers.
However newer libraries, particularly in affluent areas, are more likely to go to a digital stack since it will be more readily searched and comprehensively indexed.
I oft find it interesting how a medium (and/or technology) can fundamentally alter the basic experience of reading.
I'm a gen-x so I'm so old my youth predated even cdroms; in fact I got an off brand early cdrom reader in high school as a very expensive birthday gift. So in early high school I spent a couple weeks reading microfilm of Scientific American magazine searching out all the "amateur scientist" columns from the 50s and 60s, some real gold there. I printed out some interesting columns. Still have some of those printouts. The technology to "photocopy from microfilm" is pretty obvious optically, the image is projected onto what amounts to a xerox photocopier. Of course kids these days may not believe fully analogy photocopiers ever existed, but certainly they did, until relatively recently as digital multifunction devices took over. Around the turn of the century all the Amateur Scientist columns were combined into a multimedia cdrom using turn of the century ancient java to navigate tens of thousands of one-page-per-pdf files. I no longer have infrastructure that can run that old of a java version although hand navigation of an image of the cdrom filesystem is possible (the file names aren't too insane) and I no longer own a desktop or laptop with optical drive to read the legacy disk, although I can, for now, scare up a USB cdrom reader if needed.
I lived near a depository library; did you? If you can get access to a federal depository library then you'll have access to microfilm version of complete ancient census records; I spent a long time looking at those and printing out 1800s era census records. My future wife bought "family tree maker" cdrom custom format disks of census data (this is all pre-internet pre-ancestry.com era by at least a decade). Obviously those custom cdroms of census records are useless today unless you have a complete early 90s PC emulator, and searching online works better and there's been several decades of census data released since then, LOL.
I find it fascinating that I grew up before cdroms, then in my youth cdroms would replace "everything", and now cdroms are a dead media. I'm and not really all that decrepit and old outside of SV hiring standards and ageism, I scarcely have a gray hair, maybe one or two, don't even have any grand kids yet. CDroms may have been born and died in my young adult years, but there's still microfilm out there.
But as a digital native myself, I'm disappointed that they failed to emphasize the biggest advantage of digital information storage - the fact that data can be copied perfectly, as many times as you want. Analog microfilm cannot be copied perfectly - the original is always better than the copy. This means the microfilm is a unique artifact that needs to be preserved and taken care of. One disaster, one fire, or one careless handler, and the object is damaged.
With digital technology, we pay a high price in the lines of code to build an application and interface, to specify seemingly opaque binary file formats, to have physical media (floppy, CD, etc.) go obsolete in a decade, and to build complex CPUs - but never forget that what we gain is the tremendous ability to copy perfectly. This is what allows us to share a file with someone and know that their copy is a perfect backup for mine in case mine is lost. This is what allows us to transfer files from old, slow, unreliable media (e.g. floppy disks) to new, fast media (e.g. SSDs) without missing a beat. This is why we can take a sizable collection of a thousand audio CDs and cram them into one convenient hard disk drive (as WAV, not MP3), with the exact same audio quality. This is why we can transmit files over the Internet - we surely can't transmit analog data over a distance without a high risk of degradation.
If we collectively accept that storing information digitally with the ability to make perfect copies is worth the effort, then there are things we can do to improve the status quo. We need to make it easier to find and manage file duplicates - otherwise it's hard to know whether a copied file has been modified or not. We need emphasize building applications on open standards. We need to make file format specifications accessible, well-defined, and easier to understand. We need to make systems that have low implementation complexity (e.g. CPU with few transistors, OS with few lines of code) to show what is the minimum knowledge needed to reboot the technology.
While microfilm may last 500 years, digital data could truly be forever - as long as you copy it periodically.
Tangentially, this makes me wonder: While microfilm can't be copied perfectly, isn't it true that the source quality (e.g. the crispness of the print) is vastly lower than the resolution of the medium itself? Which seems like it would imply that the "lossy" transformation might in some sense still lossless, at least for a while - like converting a high bitrate mp3 recording of a radio broadcast into a lower bitrate one. Or perhaps the kinds of loss are different. I'm sure I'm confusing a lot of ideas and butchering some terms here.
The way digital copying gets around this problem is by having predefined signal levels that are farther apart than the vast majority of copying error, and writing a new master every time instead of the analog copy.
There is a potential catch, in that analog copies will tend to introduce some amount of physical errors no matter how much the hypothetical increase in definition should allow for perfection. Subsequent copies, no matter how high their quality, will both tend to amplify these errors and introduce new ones. You get a similar result if you repeatedly re-endcode data even in high-fidelity lossy ways - each encode introduces artifacts and amplifies previous artifacts.
It's an intriguing idea! Thank you for bringing it forth for discussion.
It is part of the "Object Lesson" series - "Each Object Lessons project will start from a specific inspiration: an anthropological query, ecological matter, archeological discovery, historical event, literary passage, personal narrative, philosophical speculation, technological innovation—and from there develop original insights around and novel lessons about the object in question." - http://objectsobjectsobjects.com/about/
The object in question is microfilm readers, and the things we might learn from it. Not digital technology, nor the reasons why microfilm went by the wayside.
In practice though, I think you could take your idea even further into the realm of lossy compression. If you're scanning newsprint for instance, the vast majority of the page is either 1) text in a known font or 2) images rendered through halftoning, that is a regular pattern of circles wih some finite range of radii. Conceivably a system intelligent enough to recognise and encode this information might even be considered a higher fidelity scan than a simple photo, as it will repair incidental defects from the printing process.
I agree on open standards and deduplication, I hope content addresable archiving will get more mainstream, IPFS brings big hopes.
[0] https://en.m.wikipedia.org/wiki/Data_degradation
Your operating system does this.
The data encoded in the steganography, could be any data of course, but for our purposes, it would be the image itself.
Such a picture could be copied, with loss, in an analog fashion, but could also be copied perfectly in a digital fashion, by a digital reader capable of reading out the steganography information. Even the analog copy could be digitally read if it was of high enough quality.
Another, more straightforward, way would be to print on the film, first an analog rendition of the picture, then a digitally encoded pattern of the same image, rinse repeat.
But from a future safe perspective, it could be smart to encode the data itself inside the image, as per the first paragraph. Someone copying the images could unwittingly copy the digital data too!
This was an intriguing idea, but after I did some back of the envelope rough numbers, I tentatively concluded microform might not encode densely enough to accomplish that. It already has poor fidelity characteristics for photographic data. Right off the bat you are losing lots of information during the reducing process, and the film itself does not have the kind of qualities photographers look for. Color microform is also much more expensive than black and white. So the color film works well for newsprint photos, but for archival grade purposes, we're likely better off with the original film negatives. For deep storage archives, I could see a use case for microform, and with the encoding you suggested, but not quite at one picture per frame (page).
Steganography relies upon imperceptibly embedding information into the picture, so there is a very low encoding density limit because we're working with analog pictures. Optar at 600 ppi uses an entire page to encode 200 Kb, but microfilm is closer to 300 ppi, so at 100 percent density with no picture, we can expect around 100 Kb per page of Optar-encoded binary data. An average smartphone picture size these days is around 5 MB, closer to 15-50 MB for DSLR pictures.
Optar-encoding the smartphone picture, we are around 50 pages before we scatter it into the picture to steganographically encode it. Already we can see the picture has to be blown up across more than 50 pages and stitched back together. At imperceptible levels of encoding density, I'm guessing we're looking at 500+ pages, maybe. A standard microfiche holds about 98 pages, so we're talking about 6'ish fiches per picture at cellphone grade, much more for DSLR grade picture encoding.
For high quality archival photograph storage, I think current librarians have it right: lots of duplicates of negatives, geographically separated, then service copy negatives off those masters, digitally scan in the most-used pictures off the service copy, and let patrons use the digital copies. Migrate the digital copies until we establish a universally-accepted digital bootstrap mechanism and process for Long Now-span time scales.
Isn't it specific to magnetic media only? (I don't think optical discs susceptible to bit rot. Especially M-discs which are said to last for a thousand years.)
Any EPROM/EEPROM/Flash will leak charge from its cells and eventually lose data if not periodically refreshed. Even some mask ROMs develop bad bits over time due to latent manufacturing defects.
Many CD-Rs and DVD-Rs recorded ~15-20 years ago already have uncorrectable errors (due to the dye degrading?). Pressed discs are nominally immune, but may have manufacturing defects that lead to progressive failure (e.g. "laser rot" on LaserDiscs manufactured by certain plants).
Would a good solution for /really/ long term storage be a kind of WORM ROM based on fuses that could be burned to 'program' a given area? (I think some EPROM might be this, but I'm talking about more storage density than even current top end flash drives.) Obviously you'd only want to use this for the LONG term storage, but if it's cheep enough I could see it being useful in the same use cases that existing optical media were and even in roles like backup / audit logs.
https://www.archives.gov/preservation/technical/storage-stra...
https://en.wikipedia.org/wiki/Digital_preservation
The Internet Archives preferred bmedium for book storage is ... books.
https://www.theguardian.com/books/2011/aug/01/internet-archi...
Quartz glass is a proposed medium, with a potential life of 300 million years.
https://www.scientificamerican.com/article/data-saved-quartz...
(That being said, digital copies would be way more useful here and now; let's do both!)
(This wasn't a joke back then. Everybody remembered the wrecked cities of WWII. They had to be built back up. You might need those instructions.)
Is this actually a thing in American memory, given that no American city was bombed in WWII?
> Is this actually a thing in American memory, given that no American city was bombed in WWII?
To veer off-topic a bit, I think this is an interesting question brings up two important points:
First, as the WWII generation has died out, people seem to have become less averse to war and to care less about the institutions built to prevent it (the UN, EU, international law, etc.). They were built because the survivors of WWII thought war was the worst scourge of humanity, and that another war - with ~1945 technology - could destroy civilization.
Second, the U.S. experienced very little direct harm in WWII, and even the overseas casualties were orders of magnitude less than those of the European and Asian countries (e.g., the U.S. lost 200,000; the USSR lost 20 million). If the worst happens and the U.S. fights another major war, the enemy likely will be able to strike the U.S. with missiles. Even North Korea can do it to a limited degree.
With the loss of the WWII generation, I think Americans have become cavalier about war, and because the WWII didn't affect the U.S. nearly as badly as the Europe and East Asia, they don't conceive of a different kind of war, such as one where missiles rain down on every American city. We can assume the Bay Area would be a priority target, for example; it's hard to imagine what that would be like; it's hard to imagine the consequences of it.
Yeah, all except the first one. I'm not gonna argue with the gushing passion for all things digital in the rest of your comment, but I will first of all, point out that putting your photos, music, and videos in the cloud is absolutely not an exact copy--cloud providers don't have ToS that cover the compression that they apply to your media. There's no telling how badly they'll compress it to make room for the newest stuff years from now.
Since you offered such an untempered view of digital media as the ultimate achievement of human civilization, I'd like to offer an alternative viewpoint that recognizes that digital copies reduce very deep, meaningful experiences down to very meaningless bits. That's great for newspaper articles, but it's no way to live life. Just yesterday I was sitting in the park under a tree. The winds were blowing, thousands of leaves dancing above, and distantly a church bell began tolling, it's bang-clang rolling and washing through and around the boughs, being buffeted by the wind itself. The boughs are laden with leaves and seeds, and the sound waves are muffled at a turn, coming through at a turn, as this branch shakes and wags between the bells and me. The sun is filtered and flickered, I see the veins of leaves through. The insects buzz. One shield-like bug, as thin as a blade and about the size of a fingernail, lands and waddles around my blanket, its six legs hyper-optimized by evolution. There are perhaps 10,000 such creatures in this field. A trillion generations to produce it. It starts, its wings buzzing a thousand times a second. The hundreds of thousands of blades of grass wag in waves like grain, like carpet, like kelp in a fishtank, and they too, rustle. Could digital media capture this? I move my head to the side, angle it up, cup my ears, and I sense this field...I am absolutely surrounded by information. The amount information is incredible. I could hear the scrapes of the leaves against each other and the audio shadows of them as they blow around in front of this bell! I can hear the bells harmonics, and in my mind I see the sound waves racing round its circumference, overlaid with other harmonics making it jiggle like a waterballoon. The information there is 10^30...10^40. We can't even imagine.
And yet, we are told that VR and digital media will usher us into a wonderful world of entertainment. Pah! Lossless copies. Pah! We cannot even capture a millionth of something like that.
Those who've seen the 60s George Pal version of "Time Machine" will remember The Traveller spinning a disk on a flat surface to activate a voiced playback. Still visionary.
I can view on a light table or project a photographic slide from 1935 just as easily as I can view a slide developed yesterday. On the other hand, it's not a trivial task to read a magnetic tape from the 70's. These days it's not even trivial for me to read a CD, I don't own a CD reader, and if I'm away from my desk I can't even read a USB flash drive on my Macbook Pro.
https://en.wikipedia.org/wiki/En-Gedi_Scroll
(As an aside, "documentation" was the hot term in library science in the 1930s. "Documentation" meaning "collection of informational papers" is only from 1927, says https://www.etymonline.com/word/documentation . As a further aside for those involved in science fairs, Watson Davis also founded "the Science Clubs of America, reaching at one point roughly a million school-age children across the United States; he also was one of the originators of the Westinghouse Science Talent Search".
He participated in the World Congress of Universal Documentation. Quoting from https://en.wikipedia.org/wiki/World_Congress_of_Universal_Do...
> The World Congress of Universal Documentation was held from 16 to 21 August 1937 in Paris, France. Delegates from 45 countries met to discuss means by which all of the world's information, in print, in manuscript, and in other forms, could be efficiently organized and made accessible. ...
> The main resolution adopted by the congress proposed that microfilm be used to make information universally available.[8] Watson Davis, chairman of the American delegation and president of the ADI, stated that the volume of information being produced created difficult problems of access and preservation, but that these could be solved by the use of microfilm. ...
> In his address to the Congress, H. G. Wells said that he thought that his idea of the "world brain" was a precursor to the ideas other delegates were proposing, and explicitly linked the projects being discussed to the work of the encyclopédistes:
These are all ideas which percolated through Vannevar Bush's Memex and many others to become the modern internet.
Also, one of the co-founders of the ADI was Atherton Seidell. He published a series of articles in Science promoting the use of microfilm. Eg, from 24 August 1934 - http://science.sciencemag.org/content/80/2069/184 .
Are there any combined devices/stations that can be used to both read AND create either microfilm or microfiche?
Most of it, though, you could do now with an "eBay special" camera and lens and a bit of ortho-litho film. It's still available as sheet film. That would mean red reads as black and non-repro blue is a thing. Panchromatic repro work would be a lot harder since Kodak killed off Technical Pan. Tech Pan was kind of super-fast film, though - you'd use it at "ISO" 20-50, depending on the application - so while it was spectacular for general photography, it was just at the edge of what would be useful for microfiche.
Part of the problem is the target market; this was at a financial services company where the supplier knew they could charge pretty much anything they wanted and it would get passed along as a tiny fraction of the end user customer contract cost. If a $100M financial services contract involves a $5K service contract for some obscure accounting thingy no one really cares. Given that situation is similar to pre-consumer era 3-d printing, I figure a really smart hacker who doesn't care about reproducible results or speed or reliability could squirt something out for a tenth the costs; but even at a tenth, thats still going to be "laser cutter" like costs. Film is never cheap; high res film is gonna be worse, just how it is.
I can understand the film/fiche writing machines being hideously expensive, but I have yet to find a reader that is built for the XXI century digital-oriented demographic. Of the readers built recently, they're all expensive, and are designed with firmware, instead of a design orientation of dumb manipulable hardware (preferably with open documentation) driven by modifiable software.
https://gmpphoto.blogspot.com/2018/04/ur-leica-most-influent...
In terms of compression as a whole, the first example would probably be military/commercial codebooks. A relatively modern example would be the q-code, where a predetermined sequence of words are encoded as a 3-letter sequence ("QRA" becomes "What is the name of your vessel/station") but I'm sure there are prior examples.
https://en.wikipedia.org/wiki/Q_code
Here's a link I found on some early codebooks: http://www.cryptomuseum.com/crypto/codebook/index.htm
This was a machine capable of storing a terabit of data on write-once photographic film. It wrote to the film using a scanning electron beam, and then developed the film using a fully automated film processing lab, and finally stored the film cartridges in a robotic storage library. Amazingly cool, but required a lot of expert maintenance and diligent cleaning.
https://en.wikipedia.org/wiki/PLATO_(computer_system)