Even storing everything in 100 Gigabyte “8k Blu-ray quality chunks”, you would have to be storing 10 Million Movies and TV shows to reach 1 exabyte worth of storage. At an average of 30 minutes each, that would take something like 500 years to watch all of 24/7.
And that’s 1/18th the total space a 64 bit number could address. So yeah, 64 bits should be enough.
4K Blu-ray quality video requires a 144 Mbps bit rate. On a triple layer Blu-ray Disc (100GB in size) that nets you ~90 minutes of run time. That doesn't include audio, which can require extraordinary amounts of space as well.
For comparable quality 8K recordings you need 4 time as much space as needed for 4K.
By your calculations the 4k blu-ray box set for Lord of the Rings, Return of the King would take 5 triple layered disks (for both the extended and theatrical versions it comes with) for just the video and no audio or anything else like the behind the scenes content when the box set only comes with 9 disks for all 3 movies. Likely not using the full bitrate you provided but still. Hard to say they aren’t getting really high quality content on the screen.
Actually, putting it this way makes me think that 64 bits may not be enough for anybody. 1/18th of a total space is a lot. And as soon as you try to use any of those bits for routing or subdivision of your capacity, you start underutilizing the whole space.
And if you argue that you should keep your routing info out of your addresses, then I feel like that's already an admission that the address isn't large enough.
“Even after the drastic data reduction performed by the experiments, the CERN Data Centre used to process on average one petabyte (one million gigabytes) of data per day during LHC Run 2.
The LHC experiments plan to collect more data during LHC Run 3 than they did in the first two runs combined.”
"the one terabyte per second (1 TB/s) threshold". This is definitely something extremely good for the research pipelines. Also, Moore's Law and Kryder's Law seems not applicable that much on their current state.
Ha, I remember doing a[0] 3-month stint as a student there back when a single PiB got oohs-and-aaahs! So crazy to see how much happened in terms of storage capacity in the intervening 20+ years.
[0] Completely insignificant to them, but a good learning experience to me!
> CERN’s data store has now crossed the remarkable capacity threshold of one exabyte, meaning that CERN has one million terabytes of disk space ready for data!
1MTB -> 1EB ready for data. It is _effective space_, not raw space. And it keeps growing.
Speaking as someone who designs hardware to process data streaming off just one of the detectors on one of the experiments at the LHC, its hard to imagine that is enough!
As a computational physicist, I often hear people in industry talk about "big data" and then laugh when I understand the amount of data they're talking about. I once accidentally wrote a hundred TiB to tape as part of a side project.
The raw LHC data is already filtered down to a tiny percentage (<0.1% IIRC) using several levels of hardware and then software triggers to make sure only the interesting stuff is recorded. So LHC could very easily fill several orders of magnitude more than this.
24 comments
[ 3.2 ms ] story [ 289 ms ] threadAnd that’s 1/18th the total space a 64 bit number could address. So yeah, 64 bits should be enough.
For comparable quality 8K recordings you need 4 time as much space as needed for 4K.
8K won't be the stopping point either.
By your calculations the 4k blu-ray box set for Lord of the Rings, Return of the King would take 5 triple layered disks (for both the extended and theatrical versions it comes with) for just the video and no audio or anything else like the behind the scenes content when the box set only comes with 9 disks for all 3 movies. Likely not using the full bitrate you provided but still. Hard to say they aren’t getting really high quality content on the screen.
- The Fellowship of the Ring
- The Two Towers - The Return of the KingAnd if you argue that you should keep your routing info out of your addresses, then I feel like that's already an admission that the address isn't large enough.
https://www.computerworld.com/article/2960642/cerns-data-sto...
“In the next 10 to 20 years, data will grow immensely because the intensity of accelerator will be ramped up”
https://home.cern/science/computing/storage
“Even after the drastic data reduction performed by the experiments, the CERN Data Centre used to process on average one petabyte (one million gigabytes) of data per day during LHC Run 2.
The LHC experiments plan to collect more data during LHC Run 3 than they did in the first two runs combined.”
[0] Completely insignificant to them, but a good learning experience to me!
1MTB -> 1EB ready for data. It is _effective space_, not raw space. And it keeps growing.
CERN is on a whole other level.
https://www.reddit.com/r/dataengineering/comments/y6lb2v/est...
https://eos-web.web.cern.ch/eos-web/