I'm confused. Why are decryption keys in NVRAM? That seems to negate the purpose of at-rest encryption if you can retrieve keys from the device even after shutdown.
Wow. SubC’s software engineering needs some work. They thought the camera’s file system was unencrypted, when it was encrypted. They didn’t know where the keys were to decrypt it. It turned out the key was written unencrypted to a UFS storage device. There was a file written to /mnt/nas/Stills, which indicates that the camera was to writing to a remote file system that wasn’t mounted.
> They thought the camera’s file system was unencrypted, when it was encrypted.
Willing to bet plenty of hn readers are unaware of encryption going on at lower layers of the tech stack than they're aware of.
For example most hard drives encrypt all data, even when not commanded to, as a way to do 'data whitening' (ie making sure there are even numbers of 0's and 1's in the data stream and not some pattern which might throw off tracking.)
The encryption key is simply stored elsewhere in the drive - or nvram or in the firmware.
But it means if you extract the physical magnetic surface and read it with the right microscope, you might well find the data encrypted with no available key.
The SD card on the camera was intact but encrypted. Decrypting the data required a key stored on a separate SOM board, but the SOM was damaged. The investigation team delivered the SOM and SD card to the camera manufacturer in Newfoundland, and they were able to decrypt the card.
They found a couple of images, but
No data with a timestamp after May 16th was found on the camera, so it is likely that none of the data recorded on the SD Card were of the accident voyage or dive.
After all that work...
If you're interested in data recovery, you will enjoy reading this report, about 10 pages, clearly written. The technical language mentioned they didn't see a LUKS header on the card so they figured it was a custom dm_crypt setup.
I have seen engineers slap Teensies on a PCB and call it a day, so it’s definitely normal. It’s faster than having to route your MCU, USB, debugger, etc. manually, so there isn’t really a drawback as long as it physically fits there.
>What's with the entire dev board crammed in there? Is that... normal?
Yep. I designed boards for cameras like this (and the vehicles they are mounted on) for 20 years. When you're only going to sell ~30 a year, and it's going into a $7k enclosure, the extra $7 for the dev board you used during prototyping isn't even a consideration. Go ahead and design around the breadboard, at this low volume it's WAY cheaper than the time to re-design the support circuitry from scratch and it gives you time to start working on the NEXT project that has already been sold to customers with a delivery date quickly looming.
Many times I have heard the tech stack for the subsea industry called "Shop & Glue."
There's a fascinating and redacted interview with an "anonymous" subject about the disaster. To say the least it's an unsuccessful attempt to hide the identity of the individual:
"Q. So how did you get yourself started into submersible operations?
A. Well, I'm sure you're familiar with my film Titanic. When I set down the path to make that film, the first thing that I did was arrange to be introduced to the head of the submersible program at the P.P. Shirshov Institute in Moscow, a guy named..."
It appears that all the engineers -- system designer, material
engineer and structural analyst -- thought that OceanGate CEO was going to
kill himself:
If you ever find <name-of-the-engineer>, he’s not going
to have a whole lot of nice to say. He was very frustrated
with the company. (...) And I understand why. He thought
Stockton was going to kill himself.
And the director himself declined to dive on Titan when asked:
Now, the question is, why wouldn’t the engineer get inside
his own vehicle? It was because of what I felt -- and I have a
background in Navy diving in EOD operations. I knew firsthand
that the operations group was not the right group for that role,
and I told him as much, that I don’t trust operations and who he
has there.
An anecdotal personal story as it aligns with this exact statement although no one got killed but data breaches certainly occurred.
Many years ago now I was propositioned to be on the board of a financial technology company and they spared no expense in literally rolling out the red carpet for my arrival. I found it all very laughable being solely focused on business and the technical details as I was not being fooled by all the schmoozing. After hearing all the unrealistic business objectives and the promise of having the Philadelphia Flyers involved I then asked to meet the technology team that built the product to see a demo. They bring in one young guy who built it all, the executives are still present mind you, and they allow me to ask any and all questions about the platform that nearly no one in management comprehended. After seeing the demo which involved several blatant security issues I asked only one more question of the sole developer: "Would you put your financial information into this system?"
He provided his answer in front of the companies executive board and I can still see their reactions to this very day. I then stood up and thanked everyone for opportunity and left.
> Removed SD card. The manufacturer of the camera had requested certain components of
the device be redacted. Portions of this image have been redacted.
And so it is, but anyone who has ever seen a Sandisk SD card knows what they're looking at. I can even tell it's not the fastest V90 speed.
The things companies try ineffectually to keep out of public view are weird.
yeah, I'd guess the person who made this decision just wanted to cover their butt. it got a good laugh out of me reading the caption, though. "hmmm. who could use this distinct branding so many people are familiar with? it looks a bit orange; maybe it's Home Depot."
Amusing that the bits the “manufacturer asked to be redacted” in the images appear to be the identifiers for common off-the-shelf electronic components, including a standard memory card. Is that really super secret IP?
Crazy that it's pretty much a 3D printed assembly internally, and the manufacturer didn't know how it worked. No way that would pass any kind of vibration test.
22 comments
[ 3.2 ms ] story [ 55.2 ms ] threadhttps://data.ntsb.gov/Docket/Document/docBLOB?ID=19169363&Fi...
Full docket:
https://data.ntsb.gov/Docket/?NTSBNumber=DCA23FM036
For a machine that needs to boot unattended, what would you do with disk decryption keys?
Willing to bet plenty of hn readers are unaware of encryption going on at lower layers of the tech stack than they're aware of.
For example most hard drives encrypt all data, even when not commanded to, as a way to do 'data whitening' (ie making sure there are even numbers of 0's and 1's in the data stream and not some pattern which might throw off tracking.)
The encryption key is simply stored elsewhere in the drive - or nvram or in the firmware.
But it means if you extract the physical magnetic surface and read it with the right microscope, you might well find the data encrypted with no available key.
They found a couple of images, but
After all that work...If you're interested in data recovery, you will enjoy reading this report, about 10 pages, clearly written. The technical language mentioned they didn't see a LUKS header on the card so they figured it was a custom dm_crypt setup.
Yep. I designed boards for cameras like this (and the vehicles they are mounted on) for 20 years. When you're only going to sell ~30 a year, and it's going into a $7k enclosure, the extra $7 for the dev board you used during prototyping isn't even a consideration. Go ahead and design around the breadboard, at this low volume it's WAY cheaper than the time to re-design the support circuitry from scratch and it gives you time to start working on the NEXT project that has already been sold to customers with a delivery date quickly looming.
Many times I have heard the tech stack for the subsea industry called "Shop & Glue."
Also a good video from Scott Manley: https://youtu.be/qMUjCZ7MMWQ
"Q. So how did you get yourself started into submersible operations?
A. Well, I'm sure you're familiar with my film Titanic. When I set down the path to make that film, the first thing that I did was arrange to be introduced to the head of the submersible program at the P.P. Shirshov Institute in Moscow, a guy named..."
https://media.defense.gov/2025/Sep/17/2003800984/-1/-1/0/CG-...
https://data.ntsb.gov/Docket/Document/docBLOB?ID=17236880&Fi...
It appears that all the engineers -- system designer, material engineer and structural analyst -- thought that OceanGate CEO was going to kill himself:
And the director himself declined to dive on Titan when asked:An anecdotal personal story as it aligns with this exact statement although no one got killed but data breaches certainly occurred.
Many years ago now I was propositioned to be on the board of a financial technology company and they spared no expense in literally rolling out the red carpet for my arrival. I found it all very laughable being solely focused on business and the technical details as I was not being fooled by all the schmoozing. After hearing all the unrealistic business objectives and the promise of having the Philadelphia Flyers involved I then asked to meet the technology team that built the product to see a demo. They bring in one young guy who built it all, the executives are still present mind you, and they allow me to ask any and all questions about the platform that nearly no one in management comprehended. After seeing the demo which involved several blatant security issues I asked only one more question of the sole developer: "Would you put your financial information into this system?"
He provided his answer in front of the companies executive board and I can still see their reactions to this very day. I then stood up and thanked everyone for opportunity and left.
He didn't mince his words either; he was extremely critical of the whole thing before and after the disaster.
And so it is, but anyone who has ever seen a Sandisk SD card knows what they're looking at. I can even tell it's not the fastest V90 speed.
The things companies try ineffectually to keep out of public view are weird.