I wonder how the institutional knowledge on correlated disk failure got lost. This used to be common with HDDs, and one large enterprise I am aware of makes sure that boot drives in a datacenter (the only SSDs of this type) are always from a mix batches at a mix of vendors.
Possibly because so much of what happens in the tech industry has been abstracted away.
The vast majority of the people responsible for the servers on the billion-dollar company I work for have never physically touched a server, a rack, or an Ethernet cable.
There's a whole generation of "point and click" admins quickly gaining prominence. I suspect cloud services have contributed to this lack of knowledge, because if all you know is cloud, you don't need to know things like this.
That, and the curse of working the same 18 months 4 times on the way to 'senior'.
All of these little companies starting with brand new hardware as well. Far from all but a few of those 'garage startups' I knew of did limp along on used hardware for some time. It happened again a bit after the dotcom bubble burst (especially for office equipment) and again around 2008.
But for drives in particular, people are pretty superstitious about giving them up.
The best I can say is try not to keep coming into the same point in the lifecycle of each new company, and try to pay attention to things you didn't last time instead of just jumping into your comfort zone.
It can help seeing people struggle with tech debt or bad decisions you had nothing to do with, if you can put yourself into the shoes of the person who did it. The wise man learns from the mistakes of others. But it does mean you have to do a little more sleuthing than the typical postmortem that just blames everything on an engineer who quit six months ago. That's facile, and neglects the contributions of people who still work here and/or think the same way.
"Point and click" is unnecessarily derogatory. During my career as a developer I spent my share of time racking my own systems, diagnosing (both network and disk) cable faults, etc. I also spent time working with the kinds of software tools needed to manage dozens of clusters up to 10K machines each at 36 disks per machine. At that scale you really need to minimize the number of people who get to muck with the physical bits, and the amount of time they spend doing so. The complexity has merely shifted, not disappeared, and most developers or sysadmins from the previous generation would themselves be hard pressed to master the management of modern systems. It's no more "point and click" now than it was "lift heavy things" before.
> if all you know is cloud, you don't need to know things like this.
That's part of the cloud value proposition, isn't it? Leave the physical (and other low-level) stuff to people who do know that stuff, and can also reap economies of scale, while you focus on your own parts of the problem. Do you not approve of such specialization?
I think the unfortunate reality of this is that even if you don't take care of the physical and OS stuff yourself, it will still hurt you when it goes wrong. Abstracting it away doesn't really take care of it for you, at least not in the same way that someone who really knows how to deal with it AND knows your system well does. And you pay through the nose for the privilege of that abstraction - even DigitalOcean is 2-3x more expensive than renting your hardware from a hosting provider, and things like managed databases and AWS lambda will run you 20x more than renting servers.
When AWS fails, it fails spectacularly. It also experiences outages fairly often, and you as the developer often have nothing you can do about it, but twiddle your thumbs and hope that your application was amenable to "high-availability architecture" (which is more expensive...). Finally, this particular failure mode brought down HN's servers, rented from a reputable hosting provider, and arguably this is something that a hosting provider should know.
> I think the unfortunate reality of this is that even if you don't take care of the physical and OS stuff yourself, it will still hurt you when it goes wrong. Abstracting it away doesn't really take care of it for you, at least not in the same way that someone who really knows how to deal with it AND knows your system well does
It's also really hard to fix things when they're surrounded by a black box operated by someone who doesn't care. I kind of enjoy doing it, but it would be a lot more productive if I was able to diagnose network link problems by checking the error counters on all the routers/switches rather than by sending pings that hash over the numerous lacp alternate routes between two servers to figure out how to get a traceroute that shows the issue so the network owner can look at counters and fix the underlying issue.
Once you cross a certain size, how else is it supposed to work, realistically speaking? My team administers servers in Brazil, Australia, Japan, Netherlands, the US, and several other countries. We can't do racking, cabling and hardware maintenance by ourselves. If we did, we'd spend 50% of our work hours just commuting on international flights. The only realistic option is to rely on local DC Ops teams to perform the manual steps involved in hardware buildup, maintenance and decom.
It's just so much more convenient to single source things and have a consistent experience... No worrying about why these drives failed and these other ones didn't; not so great when they all fail though.
I'm just some random guy who doesn't work in tech and even I know about this. I was hesitant to buy any SSDs when they were new because I was worried about drive failure.
They also know about mixing brands/batches/etc. to avoid exactly this kind of correlated failure. During one of my oncalls, we had an uncomfortably large number of failures fitting this kind of pattern, but they were well enough distributed and we had enough data redundancy that there was no data loss. Sometimes there's safety in numbers, if you know how to make those numbers work for you rather than against you.
They've got a lot of flash too. I think most of the small nodes are running from flash, and most of the databases too? There's a lot of longer term storage that's going to be rotational though, sure.
at a large enough scale, almost everything backblaze uses is sufficiently reliable as measured by percentage of any pool of 100 drives that fails in a year, as they have a large amount of parity / data duplication across multiple pods, hot spares, etc.
I haven't seen any drives that really stood out as terrible in their reports in the last couple of years for 12TB+ spinning disks, although the HGST product do seem to be the most reliable.
The manufacturers whose drives they use provide a good basis for my consideration. I don't see certain names on their lists sometimes, so my search usually is narrowed down by that.
there's literally only 3 spinning HDD manufacturers on earth now so if a name doesn't appear on their list now it's because it doesn't exist. some of the things on their list are now brands owned by a single company through consolidation.
Not so. Intel is the highest rated for failure from a QC point of view. We've been having a rash of such problems with our last batch that it's making everyone in our department nervous before shipment.
For an SSD, you need to buy "Enterprise" or "Data Center" class drives for this kind of application. That class of drive have higher write endurance which is important since enterprise applications can have orders of magnitude more writes to their drives than a typical desktop.
That said, there's no real standard blanket vendor recommendation so long as the "Enterprise" criteria is met.
> The update corrects a logging check: “Assert had a bad check to validate the value of circular buffer’s index value. Instead of checking the max value as N, it checked for N-1,” Dell’s advisory said.
A lot of preventable accidents come from the human brain discounting the threat after a large number of positive interactions. For all we know there once was a test three refactorings ago and someone fenced it away.
Can’t find it. But there was an old article where some young guy at a conference was explaining to Netflix why SSDs weren’t going to work due to drive failure cost.
And what does Netflix have a need for SSDs for? Their load seems entirely predictable / serveable / stable with spinning disks and reasonably (but not lightning) fast access patterns? I would have guessed the network latency is always dominant.
There's still a huge long tail, so iops matter. Netflix definitely wants to minimize any time you ever see a waiting on network spinner.
I dunno what talk the above comment is referring to but most likely the young hero was saying something somewhat clueless, not being kept down by the tyranny of older idiots or whatever is being insinuated. Netflix's content is pre baked. They have to do like a dozen versions of every episode to get the best experience across the variety of hardware, but it's not livestreaming or such. Where they use SSDs it's not an erase/update heavy workload, at least for the CDN part (Netflix does a ton of other stuff including basically running entire movie studios out of the cloud, not referring to that). Density and power also matter a ton as they want their POPs to be as easy as possible to convince people to run at peering points.
I have no idea what that guy is talking about or why Netflix would care about SSD failures more than a normal company that just uses mirrors/raidz for everything.
There's no need for a recall because it can be fixed with a firmware update. And as the article mentions, vendors did issue urgent/critical firmware updates for it back in 2020.
I think the problem is like.. if you buy a car, then they will call you if a recall happens. If they'd call you every month to sell you something you'd probably tell them off in such a way that they would stop. If food is recalled you may see it at the checkout in the grocery store.
No one registers their hardware at the vendor because all they do is spam you with offers then. Or they change their system and despite you registering you'll still get no notice.
57 comments
[ 2.5 ms ] story [ 128 ms ] threadThe vast majority of the people responsible for the servers on the billion-dollar company I work for have never physically touched a server, a rack, or an Ethernet cable.
There's a whole generation of "point and click" admins quickly gaining prominence. I suspect cloud services have contributed to this lack of knowledge, because if all you know is cloud, you don't need to know things like this.
All of these little companies starting with brand new hardware as well. Far from all but a few of those 'garage startups' I knew of did limp along on used hardware for some time. It happened again a bit after the dotcom bubble burst (especially for office equipment) and again around 2008.
But for drives in particular, people are pretty superstitious about giving them up.
This feels honestly like my whole career. Just get to a point of competence and then get an offer you can't refuse.
It can help seeing people struggle with tech debt or bad decisions you had nothing to do with, if you can put yourself into the shoes of the person who did it. The wise man learns from the mistakes of others. But it does mean you have to do a little more sleuthing than the typical postmortem that just blames everything on an engineer who quit six months ago. That's facile, and neglects the contributions of people who still work here and/or think the same way.
> if all you know is cloud, you don't need to know things like this.
That's part of the cloud value proposition, isn't it? Leave the physical (and other low-level) stuff to people who do know that stuff, and can also reap economies of scale, while you focus on your own parts of the problem. Do you not approve of such specialization?
When AWS fails, it fails spectacularly. It also experiences outages fairly often, and you as the developer often have nothing you can do about it, but twiddle your thumbs and hope that your application was amenable to "high-availability architecture" (which is more expensive...). Finally, this particular failure mode brought down HN's servers, rented from a reputable hosting provider, and arguably this is something that a hosting provider should know.
It's also really hard to fix things when they're surrounded by a black box operated by someone who doesn't care. I kind of enjoy doing it, but it would be a lot more productive if I was able to diagnose network link problems by checking the error counters on all the routers/switches rather than by sending pings that hash over the numerous lacp alternate routes between two servers to figure out how to get a traceroute that shows the issue so the network owner can look at counters and fix the underlying issue.
This is a specific bug, like an overflow that is consistently triggered after about 4 years of use.
I haven't seen any drives that really stood out as terrible in their reports in the last couple of years for 12TB+ spinning disks, although the HGST product do seem to be the most reliable.
That said, there's no real standard blanket vendor recommendation so long as the "Enterprise" criteria is met.
Well yeah that’s probably why you see “SanDisk” and think of consumer SD cards.
So the code was never tested even once?
Engineering processes are systems with their own bugs, and the people/systems that implement them can’t deliver perfectly either.
There are always going to be things that slip through, and some of them are bound to look embarrassingly sloppy.
I'm not talking automated testes or anything. Just once, on your own machine.
I agree that it is a bit unfair to pick an example after the fact. But it does not inspire confidence.
I'd love to see an entire thread on this topic alone.
They weren't end-of-life when they were sold this way. That's just the PR department trying to make their failure seem less severe than it was.
Response was epic.
Everything you say is true.
But we don’t care, we rent the servers.
As in, no one believed him at the time?
And what does Netflix have a need for SSDs for? Their load seems entirely predictable / serveable / stable with spinning disks and reasonably (but not lightning) fast access patterns? I would have guessed the network latency is always dominant.
I dunno what talk the above comment is referring to but most likely the young hero was saying something somewhat clueless, not being kept down by the tyranny of older idiots or whatever is being insinuated. Netflix's content is pre baked. They have to do like a dozen versions of every episode to get the best experience across the variety of hardware, but it's not livestreaming or such. Where they use SSDs it's not an erase/update heavy workload, at least for the CDN part (Netflix does a ton of other stuff including basically running entire movie studios out of the cloud, not referring to that). Density and power also matter a ton as they want their POPs to be as easy as possible to convince people to run at peering points.
I have no idea what that guy is talking about or why Netflix would care about SSD failures more than a normal company that just uses mirrors/raidz for everything.
https://www.dell.com/support/home/en-us/drivers/driversdetai...
https://support.hpe.com/hpesc/public/docDisplay?docLocale=en...
No one registers their hardware at the vendor because all they do is spam you with offers then. Or they change their system and despite you registering you'll still get no notice.