I recall this fondly when it came out. I had a few racks of Sun servers at that time...and anyone I showed the video too were convinced it was bullshit. April Fools anyone? However, Brendan and Bryan know their onions...and all the naysayers had to pause and think a bit. It too crazy to believe...and too crazy to not believe. I miss this stuff.
Edit: This was done when everyone ran spinning media. I wonder how modern SSDs hold up?
Oh man. Once, when I was a young padawan doing maintenance by myself on one of our three racks in a colocation DC, I dropped a 1U server from about 2m/6ft high. It was scary as hell. While trying to grab it I hit the off button on one of the other servers, causing an outage for about 50 customers. What's more, it fell on the floor and just missed the SAN with the spinning disk array at the bottom of the rack. Everything was fine, including the fallen server, but shit, that was stupid. The vibration alone could have caused a lot of data corruption.
Old grease, you probably broke the sticking just enough for the motor to overcome it and then after a bit of turning the grease warms up enough that it will liquify.
In case anyone needs to know, the shouter in question is none other than Brendan Gregg (of flame graph, eBPF, and Systems Performance fame). And the guy who took the video is Bryan Cantrill of Oxide Computer, DTrace, and Joyent.
He shouldn't be too upset, views and video quality only have a very rough correlation if at all. It shouldn't be surprising that his most accessible video is his most watched one.
I have a colleague who was on the on-call rotation for some of our systems hosted in a datacenter our company owned. During this rotation he got unlucky in that the datacenter detected fire and decided to disperse gas that displaces oxygen (I think it was Halon gas?). Those systems are no joke - you have a short warning before you are supposed to put on a mask or get out of the server room otherwise you will just suffocate.
Anyways, no fire damage was detected but oh boy... he had to run around various tech stores left and right to buy about 50-100 enterprise grade spinning disks before returning to the site. The fire suppression system somehow broke most of the hard drives in the server room.
This was more than a decade ago, however. Most of the disks are replaced with SSDs and so far they seem to be much more resilient than the spinning ones.
I was once told by a datacenter operator that this is due to the pressure change when they pump the gas in and that we should plan on replacing all HDDs should the fire alarm go off.
My partner worked in a Research School of Chemistry.
On the top floor of the building a tip-over failure replacing an argon bottle broke the valve off the end and emptied the compressed lot of Argon in short order.
Argon being colourless, ordourless, heavier than air = whole building "anti-suffocation" evacuation. But no fire problem!
I wouldn't be surprised if the fact that argon is heavier caused the failure. Argon gets into the drives, assisted by the larger pressure outside of the drive at the time of the release, and it wouldn't come out within a reasonable time even when the pressure came back to normal, since there's not much capability for venting gasses out of the hard drive. And since argon is significantly heavier, it causes larger air drag on the mechanical parts of the drive, eventually making the head driving coils too weak to operate at the needed speeds.
He’d be even more on an inclined plane, wrapped helically around an axis nowadays as there probably isn’t even one never mind multiple local places to buy that amount of enterprise drives.
The shockwave from a sudden release like that can take out the better part of a raid array in a matter of moments. I never understood why they have to release it all at once instead of gradually because the number of times that I've seen accidental releases dwarfs the number of real firefighting incidents.
> Please don't complain that a submission is inappropriate. If a story is spam or off-topic, flag it. Don't feed egregious comments by replying; flag them instead. If you flag, please don't also comment that you did.
There's an interesting little paper 'Acoustic Denial of Service Attacks on HDDs' where they use a function generator to find resonant frequencies which cause issues with spinning disks
There's also a paper Hard Drive of Hearing: Disks that Eavesdrop with a Synthesized Microphone. An APT attacker can potentially reflash the firmware on an HDD and using it as a microphone for eavesdropping.
Yeah, the fan noise is that loud. You should use hearing protection in there, even when staying for a short amount of time.
Though I have heard about some special DCs that use water cooling and as such can defer the noise away from the racks.
I've been in my share of datacenters and most of 'em are at least a little bit quieter now. Still loud, but not to the extent that you have to shout constantly.
It varies. Once I've been at a customer's data center with raised floors some ten years ago. It felt like an hurricane when you removed one of the tiles. Loud and cold (ear protection and wind breaker mandatory). Hard to communicate there, harder still to think. Not a nice place to be. Keep it in mind when choosing your career.
I got a "3M Peltor X5A" a couple years back as a "thinking cap": put on to think in thought-hostile environments.
Works like a charm. Probably best 30 bucks I ever spent.
For comms use a wind-proof mic on a headset boom and earphones under the X5A.
Pretty funny to see this video at the top of HN! My kids still think I'm an idiot for not having somehow parlayed the enduring popularity of this video into being a YouTuber -- despite my protestations that I was merely the videographer here. If anyone is curious about the video's origins, I had a fun conversation about it with Ben Sigelman a few years ago.[0]
We do Vibration Testing where I work (think satellite parts, simulating launch, etc). We moved to SSDs a long time ago because some of our test gear would shutdown at certain frequencies/levels during vib tests.
FWIW I did a "making of" video soon after, in 2009: https://www.youtube.com/watch?v=lMPozJFC8g0 . Only bit I missed discussing there was that I did try to download freeware frequency generators wondering if there was some "rust note" of HDD harmonics, but those I downloaded didn't work and out of frustration I decided to start yelling instead.
Years ago someone had an idea to put electron microscope in a basement of a uni building next to a busy, city-center street in Cracow, Poland, only to be able to use it during nights due to unstable images. Fun times.
81 comments
[ 2.9 ms ] story [ 163 ms ] threadEdit: This was done when everyone ran spinning media. I wonder how modern SSDs hold up?
We all laughed to the video, too.
weeks later
"Yeah, we had to let him go. Too burned out. Caught him screaming at the SAN."
I firmly believe that this [1] still should be hanging in every system administrators bedroom around.
https://www.brendangregg.com/Perf/linux_observability_tools....
Absolutely incredible!
It'd be el caro (the expensive one), in the same line as the yatch thing.
Anyways, no fire damage was detected but oh boy... he had to run around various tech stores left and right to buy about 50-100 enterprise grade spinning disks before returning to the site. The fire suppression system somehow broke most of the hard drives in the server room.
This was more than a decade ago, however. Most of the disks are replaced with SSDs and so far they seem to be much more resilient than the spinning ones.
...and I think it was Argon...
It was indeed Argon, not Halon, and the pressure issues that broke HDDs.
Looks like Siemens did an investigation and tested various parameters (pressure, noise, vibration) independently.
They point towards noise, not pressure. Really interesting as I am _sure_ I was told it is pressure. But that might be 100% wrong!
https://virz.de/wp-content/uploads/2020/05/2015-White-Paper-...
The feeling of anti-sound is quite distinctive.
On the top floor of the building a tip-over failure replacing an argon bottle broke the valve off the end and emptied the compressed lot of Argon in short order.
Argon being colourless, ordourless, heavier than air = whole building "anti-suffocation" evacuation. But no fire problem!
Possibly another case of loud noise when the gas is released - see e.g.
https://www.theregister.com/2018/04/26/decibels_destroy_disk...
https://www.bbc.co.uk/news/technology-37337868
if you disassemble one you will find the tiny air filter module which covers a hole in the casing for equalizing pressure with the outside air.
He’d be even more on an inclined plane, wrapped helically around an axis nowadays as there probably isn’t even one never mind multiple local places to buy that amount of enterprise drives.
Might be a micro center if you’re lucky.
https://hn.algolia.com/?dateRange=all&query=tDacjrSCeq4&sort...
https://news.ycombinator.com/newsguidelines.html
https://andrewkwong.org/docs/Kwong-HDDphone-IEEE-SP-2019.pdf
https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=10...
and here: https://devblogs.microsoft.com/oldnewthing/20220816-00/?p=10...
For comms use a wind-proof mic on a headset boom and earphones under the X5A.
[0] https://www.youtube.com/watch?v=_IYzD_NR0W4
> Even soullsess machine cant work properly when it`s being shouted at
Any other Data Centre classics I should watch?