I wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/
So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?
Not PBS but an affiliate broadcaster and the data is not lost as in destroyed, but legally encumbered.
Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation
The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
> This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.
Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.
Ok, it sucks, losing data always sucks, but what ever happened to the 3-2-1 backup rule? even if your off site backup provider goes tits up, your on site backup is still there.
If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
The backups (and servers) seem to still exist, but KETC's vendor is estranged with its own hosting / siting provider, Iron Mountain, which is a going concern.
IM seem to be holding KETC's data hostage, which is Not Cool.
Sounds like they paid a company to digitise all their old tapes, and store the results. They possibly still have all the tapes (on site) and are just trying to avoid paying the costs to redigitise them.
But, it’s also quite likely they destroyed all original the tapes after digitisation, and this was the only copy.
> This unidentified vendor provided “hardware, software and cloud-storage services” for storing the public broadcaster’s archival materials and other data.
Isn't 50TB just 6 years of videos, in even the best case (1GB/h)?
They weren't recording in 4K last century.
One hour of 1970's-grade NTSC video is around 300MB.
More to the point, it's not like they archived 70 continuous years of video. Many local PBS stations produce an hour or less of video each week. This being a St. Louis station, however, it's probably somewhat more.
Which, in practical terms, means "while I or my investors have money". These guys have 70 years worth of data, can you guarantee to keep it for another 70? The real answer to these kinds of problems is managing the data yourself
Calling out Nine PBS for not accounting for the fact that their vendor could effectively just decide to commit crimes to get out of their contract and their vendor’s vendor could just decide to compound the problem in the face of a court order is wild. Yes, we’ve all learned to anticipate everyone else on the planet being an absolute shitheel; no, Nine PBS did not do anything wrong by assuming the professionals they were paying to render a professional service to them would do so. We’re all paranoid engineers, but this is absolutely no goddamn way to run a society.
> "their vendor could effectively just decide to commit crimes"
Going bankrupt isn't inherently criminal. And IM are likely to be within their contractual rights to lock the OSS account if IM isn't being paid.
> "their vendor’s vendor could just decide to compound the problem in the face of a court order is wild"
The initial court order was between "Nine PBS" and "OSS". Iron Mountain were not a party to the initial court order, and granting access without a court order naming Iron Mountain could open them up to liability from OSS.
This looks like a similar formality of following the process so that a court legitimises the transfer of data back to Nine PBS.
I'd suspect somewhere in the contract between Nine PBS and OSS there's some wording about what happens in the event of OSS discontinuing operations that doesn't include the words "silently fuck off into the woods without informing customers," which appears to be what happened here.
I agree that IM is probably inside of their contractual rights, but they're still handling this in a fashion that risks causing severe issues for Nine PBS. The presence of the court order seems like sufficient inducement to at least hold the deletion until the court decides, and I'd be very curious what kinds of legal remedies Nine PBS might be entitled to should IM not do so and the court decide in their favor.
"Judge sets framework for Nine PBS to retrieve archival data"
Looks like a sensible recognition of the responsibility to restore the data and make Nine PBS, whilst recognising the cost and impact to Iron Mountain and the risks and challenges of potentially co-mingled client-data from OSS.
>This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Look, I know that a certain percentage of Tom's Hardware articles are just sales pitches for a NAS, but this is asinine. Iron Mountain deals largely in tape storage. 50TB of tapes fits in one of their small archive boxes. If they're using the latest tech (LTO-9), it fits on 3 tapes.
This is exactly how you are supposed to store archival media for decades, not on a NAS. A NAS is for storing the files you need to actively access.
I thought this would be like the backups I've heard broadcasters have of every broadcast they've ever had (I've heard of people being able to request 20 year old news clips before).
But just 50TB of data? Duplicating that would have been cheap and trivial.
I think for most of broadcasting history video was fairly low quality (according to today’s standards). So I am not totally surprised that this number is low.
> But just 50TB of data? Duplicating that would have been cheap and trivial.
Seriously. I'm just a random nerd and I have a home server with 96TB of raw disk space divided in to two RAIDZ1 pools which cost me around $2500 to build five years ago and has required basically zero maintenance since then.
Hard drive prices are absolutely bonkers right now but even with that in mind if I were to build a modern version of this same machine I could do it for under $10k using brand new hardware and it'd have double the storage
Most broadcasters would consider their ultimate back up as the tape library/vault of their broadcast video tapes. Older stations will have a large collection of analog tapes. Most will only be digitized if someone requests it as capturing their entire library just to have digital files few people will ever want is an expense a station just isn't going to do for the lulz.
It's funny to me that entire companies exist that have a fraction of storage that I personally have in a datacenter on flash let alone in total. 50 TiB is so little. It's like $7k annually on R2 and under a thousand bucks on S3 Glacier Deep Archive. I suppose it's just whoever knew the cousin of whoever at whatever position of authority but it's funny how just using deep glacier means he would have such insane margins over paying Iron Mountain.
50TiB is like $4k consumer SSDs in a shoebox at this point (not actually the recommended solution, but...). And that's massively inflated thanks to the AI boom.
I work in several labs where I have to give this advice, which some labs are smart enough to make an ersatz backup that we store somewhere (PI's house) as a cheap offsite. Even a partial recovery is great in the face of fire/flood/explosion, or the most common error: IT fucks it up and then says "oh well".
Hope someone has some copies and they can rebuild, what a nightmare.
When I worked for Syncplicity, we offered an on-site version for "cloud" storage specifically for this reason. (Ok, not specifically for this reason, it was because customers demanded it and an on-site version of our product was a great differentiator in our niche.)
(I've submitted this to HN's mods via email. The original submission was to a very poor TomsHardware link.)
OT: The tendency of local broadcasters to fail to identify either on websites OR QUITE OFTEN ON BROADCASTS THEMSELVES even vaguely where they operate is ... a long-standing annoyance of mine. Sure, local audiences may know, but those passing through, or catching the signal at a distance, likely won't.
I've spent more than a few nights at higher elevations (campgrounds, highways) listening to clear channel stations reporting on weather and traffic "in the local area", sometimes for hours, without ever hearing what "the local area" is. That for signals travelling hundreds of km, and not infrequently multiple thousands of km.
(Don't get me started on entire countries where stations are identified simply by number, e.g., "Channel 4" in the UK.)
It’s entirely possible that Iron Mountain will be happy to turnover the data but needs a court judgement to do so without incurring more legal exposure. The article says the system belongs to OSS. If this is something like a colo arrangement then probably that is the only reasonable position to take. They can’t just go crack open a customer’s server and pull data out of it for whoever asserts a claim on it.
With the stuff about "corruption" and losing the data if the server is shut down, I wonder if part of the concern is about losing in-memory decryption keys.
54 comments
[ 0.21 ms ] story [ 6.3 ms ] threadI wonder how they got the contract? Seems to be 4 people 'employed' at this company. Of which, one is retired, one is "Linkedin Member", one is a controller and the last one is a social media person: https://www.linkedin.com/in/isabelle-anderson-7aa250b5/
So, just the person to deposit checks and a social media person to get paid. Who exactly is left to work on storage infrastructure?
https://www.linkedin.com/company/osstorage/people/
Full headline: PBS broadcaster loses access to 50TB of data comprising 70 years of TV history after contracted cloud storage vendor goes defunct — public TV channel sues Iron Mountain data center, which hosts archival materials, to ensure preservation
The channel’s attorney also reached out to Iron Mountain to inform the company of the pending case in Colorado; Iron Mountain finally admitted to holding the data. Iron Mountain initially agreed to turn over Nine PBS’ data, but it eventually backtracked and claimed that OSS owned the data it stored. Because of this, Nine PBS now has no choice but to sue Iron Mountain, too, to ensure the data is not deleted.
Okay hang on now.. that strategy comes with its own risks. Risks that I imagine PBS was trying to avoid by using a third party that specializes in archival data storage. This is a silly takeaway.
Edit: I was too quick to comment. I think they are suggesting that the NAS should have been used in addition to the third party storage, which is actually a great takeaway.
If you don't want on site, a second off site provider, 50TB on backblaze would be an extra ~$350 per month (prob less if they were to commit to capacity instead of using their PAYG service)
IM seem to be holding KETC's data hostage, which is Not Cool.
Sounds like they paid a company to digitise all their old tapes, and store the results. They possibly still have all the tapes (on site) and are just trying to avoid paying the costs to redigitise them.
But, it’s also quite likely they destroyed all original the tapes after digitisation, and this was the only copy.
At that point, why wouldn't you pick a major cloud vendor? This feels like more of a governance failure.
> This unidentified vendor provided “hardware, software and cloud-storage services” for storing the public broadcaster’s archival materials and other data.
They weren't recording in 4K last century.
One hour of 1970's-grade NTSC video is around 300MB.
More to the point, it's not like they archived 70 continuous years of video. Many local PBS stations produce an hour or less of video each week. This being a St. Louis station, however, it's probably somewhat more.
Just email…
Which, in practical terms, means "while I or my investors have money". These guys have 70 years worth of data, can you guarantee to keep it for another 70? The real answer to these kinds of problems is managing the data yourself
This looks like a similar formality of following the process so that a court legitimises the transfer of data back to Nine PBS.
I agree that IM is probably inside of their contractual rights, but they're still handling this in a fashion that risks causing severe issues for Nine PBS. The presence of the court order seems like sufficient inducement to at least hold the deletion until the court decides, and I'd be very curious what kinds of legal remedies Nine PBS might be entitled to should IM not do so and the court decide in their favor.
"Judge sets framework for Nine PBS to retrieve archival data"
Looks like a sensible recognition of the responsibility to restore the data and make Nine PBS, whilst recognising the cost and impact to Iron Mountain and the risks and challenges of potentially co-mingled client-data from OSS.
>This wouldn't have been an issue if the organization bought a $1,000 NAS and populated it with a few hard drives.
Look, I know that a certain percentage of Tom's Hardware articles are just sales pitches for a NAS, but this is asinine. Iron Mountain deals largely in tape storage. 50TB of tapes fits in one of their small archive boxes. If they're using the latest tech (LTO-9), it fits on 3 tapes.
This is exactly how you are supposed to store archival media for decades, not on a NAS. A NAS is for storing the files you need to actively access.
I thought this would be like the backups I've heard broadcasters have of every broadcast they've ever had (I've heard of people being able to request 20 year old news clips before).
But just 50TB of data? Duplicating that would have been cheap and trivial.
It depends - especially if you go back further than 20 years. The Wikipedia article for "Lost television broadcast" goes into detail:
https://en.wikipedia.org/wiki/Lost_television_broadcast
Seriously. I'm just a random nerd and I have a home server with 96TB of raw disk space divided in to two RAIDZ1 pools which cost me around $2500 to build five years ago and has required basically zero maintenance since then.
Hard drive prices are absolutely bonkers right now but even with that in mind if I were to build a modern version of this same machine I could do it for under $10k using brand new hardware and it'd have double the storage
Still, it is 70 years of video from that PBS station.
I work in several labs where I have to give this advice, which some labs are smart enough to make an ersatz backup that we store somewhere (PI's house) as a cheap offsite. Even a partial recovery is great in the face of fire/flood/explosion, or the most common error: IT fucks it up and then says "oh well".
Hope someone has some copies and they can rebuild, what a nightmare.
"Nine PBS" seems to be <https://www.ninepbs.org/>, A/K/A KETC, St. Louis, MO.
<https://en.wikipedia.org/wiki/KETC>
The upstream cited article which should be subbed in for this submission does clearly identify the station in the first line of its report: <https://current.org/2026/08/nine-pbs-sues-iron-mountain-over...>.
(I've submitted this to HN's mods via email. The original submission was to a very poor TomsHardware link.)
OT: The tendency of local broadcasters to fail to identify either on websites OR QUITE OFTEN ON BROADCASTS THEMSELVES even vaguely where they operate is ... a long-standing annoyance of mine. Sure, local audiences may know, but those passing through, or catching the signal at a distance, likely won't.
I've spent more than a few nights at higher elevations (campgrounds, highways) listening to clear channel stations reporting on weather and traffic "in the local area", sometimes for hours, without ever hearing what "the local area" is. That for signals travelling hundreds of km, and not infrequently multiple thousands of km.
(Don't get me started on entire countries where stations are identified simply by number, e.g., "Channel 4" in the UK.)
https://www.newson6.com/story/5e3687772f69d76f6209d656/tape-...
With the stuff about "corruption" and losing the data if the server is shut down, I wonder if part of the concern is about losing in-memory decryption keys.