So they changed pins, and kept keying? That is just pure absolute evil.
In these cases 80% of retail box and the power supply itself should be covered with red warning label. "PINOUT CHANGED! WILL CAUSE PERMANENT SYSTEM DAMAGE"
Probably EVGA should cease sales of the updated supply and proactively remove them from any channel suppliers/shops/etc.
The redesigned power supplies are probably otherwise fine, but at the very least EVGA need to ensure the pin out doesn't let previous gen cables be plugged in.
Otherwise they're just asking for liability issues like happened here. :(
the cable sets are the real problem, though, and the cable sets are in the wild already with no distinctive markings.
Those cable sets are ostensibly compatible with almost every decent/good psu that evga made (the entire g2/t2/p2/g3/g5/g6/g7/gq family theoretically share a set) and are marketed as compatible, traded on eBay, etc. They’ve polluted the whole shebang, 20 years of cables need to be tested or replaced.
The only guaranteed safe way out is to either recall all cable sets from all units in those series and then ship out new ones marked as safe, or ship a free tester board to anyone with any compatible cable string (forever into the future) to validate that a given string is actually safe.
Evga isn’t going to do either of those because they’re basically on the way out of business already. They’re down to a skeleton crew etc and certainly are in no financial shape to do a giant recall, even if the big boss wanted to (and I’m guessing he’s the problem).
(The really fortunate thing is that it’s just the SATA string so far… but the other PERIF strings do plug into the same ports and are supposed to be compatible too! (I am guessing they fit and would blow up your drives too.) But modern users don’t use the SATA strings that often, because modern hdds use the 3.3v as a “power off” toggle to spin down the drives… so ideally you use molex adapters anyway. But still - I have at least 5x G2 units in my various machines, and I’ve mixed and matched cable sets without thinking - they’re sold as compatible, why wouldn’t I? I’ve used the SATA strings recently on some ssds etc. Not happy about this at all.)
We were fixing old QNAP server, and it used normal PSU keying. The thing is - it blew PSU capacitors(with a nice small explosion) after being turned on.
we suspected that the old QNAP model had proprietary pin layout for the PSU. No labels of course.
And to clarify for those who came straight to the comments - this wasn't a case of the customer buying a new PSU and just assuming the cables matched. This was an RMA unit, and was sent to him without cables. I don't see how this can be interpreted in any other way than him being intended to use existing cables (and even if that was an oversight, it's still theirs to bear the consequences)
EVGA are quoting warranty conditions, but surely this just comes under common tort liability, at least for the material cost of the damaged drives?
Yes. And they instructed him as per their RMA process not to send in accessories such as cables when he was RMAing the original defective power supply.
I believe a wise man once said The best thing about standards is there are so many to choose from.And if you don't like any of them you can make your own!
> he'll have to replace all of his storage devices out of pocket, which won't be cheap
22 TB of storage - ~500 $
the data stored on the fried disks - may be priceless
Yeah, the old adage to always make backups applies here, but I guess many people (me included) back up stuff by keeping duplicate copies on different drives, because it's very rare for two drives to fail at the same time. Except if your PSU decides to fry them, of course...
About twenty years ago, the admin of the local file sharing FTP bragged about getting a new 5TB server and invited me to take a look. Now my SFF PC has the same amount of storage like it's nothing. I wonder if in another 20 years we are going to see storage grow as much.
Ages ago had a raid5 with 4 drives. Two WD, a Seagate and a Toshiba. PSU blew up, took the two WDs with it. Game over. Still not sure if the WDs were more sensitive or just bad luck. This is obviously not statistically relevant in any way, I still wonder to this day...
You wrote that the bad reputation isn't from the drives themselves, but I consider any drive that turns voltage dips into corruption to be a problem. Especially on the time scale of milliseconds or larger, I expect it to self-monitor and either regulate the voltage or cancel the write attempt.
It only costs $0.10 and $4/month to back up 1TiB, but $100-$350 to restore it, depending on how fast you want to restore. Still, $4/mo, or $88/month for 22TB is some peace of mind.
Having paid restore costs due to user error in my home, it's worth it. I'm toying with offsite HDDs in the future, but still feel that reliability is inferior.
The restore costs are absolutely worth it to most people, but it's still important to point out.
As a teenager I would have been perfectly capable and willing to pay 5/month to backup all my stuff. But I wouldn't have been able to afford 100-150 to restore it. So not knowing about the restoration fees would have given me a false sense of security and it might have been disastrous.
That's the first thing I thought. If he's concerned about the cost of replacement drives then it could have been much worse. Seeing that the post was in /r/DataHoarder I doubt it was priceless--though it could have taken a while to accumulate.
Seems pretty open and shut to me. That power supply either has a connected equipment warranty or it does not.
Dollars to doughnuts that it doesn't (https://www.evga.com/warranty/power-supplies/), and likely has language confirming the opposite for liability reasons (it does). Commonplace for consumer gear.
Sure the good, and human thing to do would be to cut the guy a check, that's inarguably true. It's probably even a good business decision to, given that bad press is likely more expensive than a couple grand worth of drives. Legally though, they're probably in the clear.
On the other hand, this design decision was so stupid that they're likely to kill the hardware of a lot of other people too, so obviously the lawyers would want to... uh... help destroy the company's reputation.
Here's a question: If you're going to change the pinout of the connector, which is proprietary anyway, why is the connector not keyed for the pinout it's supposed to use so you can't plug in the wrong cable?
Obviously I agree with your sentiment that this was a foolish and in the end a not-actually-cheaper solution, just saying that I think that was the argument used by the people responsible for the change.
It's insane that PSU manufacturers still haven't standardised the PSU-end of modular PSUs to this day. Or at the very least, standardised within one manufacturers offerings.
But I suppose I know the reason, it's more profitable if people can't reuse cables but have to get new ones every time they get a new PSU...
And motherboards don't have a standard front panel connector after like 35 years plus even though it'd be easy and save a lot of frustration everywhere
I would argue that this is much less of an issue, especially given there's like 4 things max you wanna plug in anyway, power button, power LED, maybe activity LED, and speaker ...maybe, and you only do this once per PC, and if you fuck up you're not breaking anything.
I really wonder if this was worth the few thousand they likely saved. You have two connectors(PSU and modular cable) with cost of what 1 cent a piece...
And it is not like they could not be used in some other model later...
I'd argue that if they first tell him to keep his existing cables and then send him a PSU that fries attached peripherals when operated with his existing cables, they're very much not in the clear...
Under German Law you would likely have a claim for damages under § 823 BGB as it is a "Weiterfresserschaden" literally "damage that eats further". In the float switch case a faulty float switch caused a building to burn down and the BGH awarded damages to the building owner.
https://www-lecturio-de.translate.goog/mkt/jura-magazin/der-...
Depends on your legislation. In EU EVGA would be responsible for direct damage, i.e. the disks themselves. Indirect damage, such as loss of data, is not covered.
No, because warranty law isn't the only law at play. If you take an action that damages other people's stuff, you are usually liable in civil law. That includes shipping something that damages their stuff.
"Look, when you send your cell phone in for warranty repairs we may _occassionally_ send one back where the mainboard is replaced with thermite that will ignite the first time you connect it to a charger. If we do that without telling you, we're not liable for your house burning down because our warranty document says we don't cover that. Thanks for understanding. HTH HAND."
I'm sure that will totally fly in a court. Not one with real lawyers or jurists, but _a_ court.
This wasn't a mistake or freak accident. This was a reasonably foreseeable outcome of EVGA's decisions. They could have prevented it at multiple steps along the way. The only things the customer could have done to avoid this (i.e., replace the cables) they were directly advised not to do by EVGA.
Who could have prevent this happening, and how?
The end user could have prevented it by... not making use of the warranty he was entitled to? Ignoring EVGA's advice to keep his cables and instead replace them at his own expense without even knowing that the cables were in fact different?
EVGA could have prevented it by not sending him a PSU with a different pinout, by sending him the appropriate cables for the PSU he was sent, by providing a warning that the PSU was incompatible with previous cables and telling him to replace them, by...
It's difficult, but not impossible to do the recovery yourself. Swap the platters between two drives and clone the data to a fresh drive. I've done it on some 2 tb drives way back when.
I've done this, but on IDE drives, over a decade ago. I wonder if nodern SATA drives store their encryption keys on the platter or on the PCB. Afaik even drives where the data is accessible on boot has encryption, just that the key isn't password protected, it's a convenient way to turn on encryption, just encrypt the key instead of overwriting the whole disk...
If you have self encrypting disks you are screwed anyway and they are more common today than ever. There are some with the "nice" feature of resetting the initial key to the same value every time though.
If you're the computer nerd for your extended family, both biological and chosen, that's a lot of photos and videos of birthdays and weddings and funerals, but most likely it's for a private Netflix-like service that never (unless the drives get killed) removes videos.
Well, it might be a couple of weeks worth of some logs/stats. I use sample datasets of similar size in my routine development workflows. That data is not important though.
And I guess the lesson learned is you RMA all parts when you RMA a product.
But EVGA said that they won't return the cables if you send the power supply with the cables. From the article:
After contacting EVGA, he sent the unit (at his own expense) to EVGA to get the problem resolved. But in typical fashion, EVGA told the Reddit post to keep all his accessories and power supply cables as they wouldn't be returned if shipped. This is not an unusual practice for most power supply manufacturers.
I thought the pinout on power-supplies was standardized. Except if you're Dell of course; Dell used to make motherboards that only worked with Dell power supplies, and vice-versa. I don't know if that's changed.
Is there a way to publish open standards for this stuff and pressure oems to use it? Seems like PSU and front panel wiring is ignored and I'm sure there is more but I just don't deal with it enough.
The sata power spec gives 3.3, 5, and 12 volt rails, but I believe most devices now don't use 3.3 or 5 volt rails at all because they want to use their own internal voltage regulators where power up scheduling is more predictable (ie. Ramp rates etc).
Given that, the drives probably only used 12v and gnd, any miswiring could only have been reverse polarity??
Intel's SATA SSDs helpfully print on the label what voltages they use and the respective max current draw. Most only use 5V but I have a couple that use 5V and 12V.
I don't think this is an unlucky mistake, it's bad processes inside the company. Any other person in the same situation would have gotten the same result. It's not like somebody accidentally gave him the wrong thing, they have two variations of the same product silently being treated as the same, with pin incompatibilities.
EVGA is notorious for missing diagrams in their PSUs. This is no surprise to me. I am using one but planning to replace it with a more reputable brand.
My experience with EVGA PSUs: unstable systems regardless of the PSU being mid-range 800W or top tier 2kW one. Incompatibility with 6xxx RedDevils (the cards just don't power up, voltages out of the spec). Coil wine.
Never had an issue with seasonics and superflowers.
So after plugging the keyed sata cable into the PSU dig out a copy of the sata connection pin layout and a multimeter? I suspect that level of paranoia is beyond almost everyone.
However, EVGA swapped voltages around on sata cables without changing the keying on the proprietary connector. That is fire-the-engineers grade stupid and should probably mark their exit from the power supply market. It drastically lowers confidence that the rest of the product is sensibly put together.
Ok, I'll be that guy. Let's stay with power supplies. How do you "never assume"? Send to a lab? dump all MCU code?
Say you buy USB-C brick. How do you test that is outputs correct 9V/12V when PD or PPS requests it? Do you inspect it with thermal camera under max rated load?
World cannot function if every fault is blamed to lack of due diligence by end user
When getting a new part, just have that conversation with the supplier that it's good to plug in with the cables you have. Of course this is on EVGA and I'm not saying this isn't lack of due diligence but being paranoid might have helped this guy keep his data and he'd have an extra paper trail.
But in any case, I was never blaming the guy. I'm sure I would have blown up my SATA drives in exactly the same way. I'm only saying that from now on, the paranoid, like me will always check the voltages on a new power supply output pins before I use them. He did actually check them after he realized the problem. It doesn't take long to check. You don't need to send it out to a lab as was suggested.
Sorry to hear this and the sibling comments about the company struggling. I like my EVGA power supplies. Availability is limited for ATX capable of running multiple GPUs.
All that Linux ISO's are now gone - EVGA needs to pony up and replace his drives at least due to their idiotic design changes - they probably don't make the power supplies themselves but use an OEM.
97 comments
[ 3.4 ms ] story [ 170 ms ] threadNo clearer case of criminal incompetence. What a bunch of fuckwits. Noone at that company should ever work in engineering again.
(but still not the worst fuckwittery I've seen)
In these cases 80% of retail box and the power supply itself should be covered with red warning label. "PINOUT CHANGED! WILL CAUSE PERMANENT SYSTEM DAMAGE"
The redesigned power supplies are probably otherwise fine, but at the very least EVGA need to ensure the pin out doesn't let previous gen cables be plugged in.
Otherwise they're just asking for liability issues like happened here. :(
Those cable sets are ostensibly compatible with almost every decent/good psu that evga made (the entire g2/t2/p2/g3/g5/g6/g7/gq family theoretically share a set) and are marketed as compatible, traded on eBay, etc. They’ve polluted the whole shebang, 20 years of cables need to be tested or replaced.
https://www.evga.com/products/product.aspx?pn=101-CK-0850-B9
The only guaranteed safe way out is to either recall all cable sets from all units in those series and then ship out new ones marked as safe, or ship a free tester board to anyone with any compatible cable string (forever into the future) to validate that a given string is actually safe.
Evga isn’t going to do either of those because they’re basically on the way out of business already. They’re down to a skeleton crew etc and certainly are in no financial shape to do a giant recall, even if the big boss wanted to (and I’m guessing he’s the problem).
(The really fortunate thing is that it’s just the SATA string so far… but the other PERIF strings do plug into the same ports and are supposed to be compatible too! (I am guessing they fit and would blow up your drives too.) But modern users don’t use the SATA strings that often, because modern hdds use the 3.3v as a “power off” toggle to spin down the drives… so ideally you use molex adapters anyway. But still - I have at least 5x G2 units in my various machines, and I’ve mixed and matched cable sets without thinking - they’re sold as compatible, why wouldn’t I? I’ve used the SATA strings recently on some ssds etc. Not happy about this at all.)
We were fixing old QNAP server, and it used normal PSU keying. The thing is - it blew PSU capacitors(with a nice small explosion) after being turned on.
we suspected that the old QNAP model had proprietary pin layout for the PSU. No labels of course.
EVGA are quoting warranty conditions, but surely this just comes under common tort liability, at least for the material cost of the damaged drives?
22 TB of storage - ~500 $
the data stored on the fried disks - may be priceless
Yeah, the old adage to always make backups applies here, but I guess many people (me included) back up stuff by keeping duplicate copies on different drives, because it's very rare for two drives to fail at the same time. Except if your PSU decides to fry them, of course...
Side note, but damn, imagine telling someone in the 90s that storage would get this cheap. We've come a long way.
83-93, disk prices dropped about 300x
93-03, about 1000x
03-13, about 20x
13-23, about 3x
SSDs are catching up but they're not going much faster.
But WD always were quite demanding for the power quality and the amount, ie some drives could work with 11.75 while WD probably wouldn't.
https://news.ycombinator.com/item?id=38834108
Chinesium PSU went way lower than 11v and... would you pay 5% more for the every HDD not to have a problem if your sub $20 PSU is sub $20 one?
You quite literally wouldn't jave a problem with any HDD if you invested $20 more in your PSU.
Unless you are upstream-constrained or a data hoarder, just make offsite incremental backups. Preferably some redundant block storage.
S3 Glacier Deep Archive is 99c/TB. Just ship it.
It only costs $0.10 and $4/month to back up 1TiB, but $100-$350 to restore it, depending on how fast you want to restore. Still, $4/mo, or $88/month for 22TB is some peace of mind.
https://www.arqbackup.com/aws-glacier-pricing.html
As a teenager I would have been perfectly capable and willing to pay 5/month to backup all my stuff. But I wouldn't have been able to afford 100-150 to restore it. So not knowing about the restoration fees would have given me a false sense of security and it might have been disastrous.
Regardless, it has good reliability properties.
Dollars to doughnuts that it doesn't (https://www.evga.com/warranty/power-supplies/), and likely has language confirming the opposite for liability reasons (it does). Commonplace for consumer gear.
Sure the good, and human thing to do would be to cut the guy a check, that's inarguably true. It's probably even a good business decision to, given that bad press is likely more expensive than a couple grand worth of drives. Legally though, they're probably in the clear.
I would bet dollars to donuts that they lost a lot more than the replacement cost would have been.
Here's a question: If you're going to change the pinout of the connector, which is proprietary anyway, why is the connector not keyed for the pinout it's supposed to use so you can't plug in the wrong cable?
Because it's cheaper to reuse the connectors they already have rather than to make a new one. /s
EDIT: Added a /s just to be on the safe side.
It's insane that PSU manufacturers still haven't standardised the PSU-end of modular PSUs to this day. Or at the very least, standardised within one manufacturers offerings.
But I suppose I know the reason, it's more profitable if people can't reuse cables but have to get new ones every time they get a new PSU...
And it is not like they could not be used in some other model later...
They could have easily rectified this by shipping new cables with the PSU and having sufficient warnings in multiple places.
You think there's no obligation here?
https://www.citizensadvice.org.uk/consumer/somethings-gone-w...
I'm sure that will totally fly in a court. Not one with real lawyers or jurists, but _a_ court.
This wasn't a mistake or freak accident. This was a reasonably foreseeable outcome of EVGA's decisions. They could have prevented it at multiple steps along the way. The only things the customer could have done to avoid this (i.e., replace the cables) they were directly advised not to do by EVGA.
Who could have prevent this happening, and how?
The end user could have prevented it by... not making use of the warranty he was entitled to? Ignoring EVGA's advice to keep his cables and instead replace them at his own expense without even knowing that the cables were in fact different?
EVGA could have prevented it by not sending him a PSU with a different pinout, by sending him the appropriate cables for the PSU he was sent, by providing a warning that the PSU was incompatible with previous cables and telling him to replace them, by...
Sort of sarcasm, sort of not.
Wow, the drive manufacturer's warranty most certainly wouldn't apply. There was no fault with the drives at all.
Very poor look for EVGA.
And I guess the lesson learned is you RMA all parts when you RMA a product.
But EVGA said that they won't return the cables if you send the power supply with the cables. From the article:
After contacting EVGA, he sent the unit (at his own expense) to EVGA to get the problem resolved. But in typical fashion, EVGA told the Reddit post to keep all his accessories and power supply cables as they wouldn't be returned if shipped. This is not an unusual practice for most power supply manufacturers.
I haven't had to RMA a power supply yet, but I buy Seasonics that are gold or better.
Not sure I can hear those coil whines though.
The sata power spec gives 3.3, 5, and 12 volt rails, but I believe most devices now don't use 3.3 or 5 volt rails at all because they want to use their own internal voltage regulators where power up scheduling is more predictable (ie. Ramp rates etc).
Given that, the drives probably only used 12v and gnd, any miswiring could only have been reverse polarity??
Never had an issue with seasonics and superflowers.
also, I wish the EU / GOV would take action to ask HDD companies to provide affordable, repairable services for these PCB problems.
However, EVGA swapped voltages around on sata cables without changing the keying on the proprietary connector. That is fire-the-engineers grade stupid and should probably mark their exit from the power supply market. It drastically lowers confidence that the rest of the product is sensibly put together.
Say you buy USB-C brick. How do you test that is outputs correct 9V/12V when PD or PPS requests it? Do you inspect it with thermal camera under max rated load?
World cannot function if every fault is blamed to lack of due diligence by end user
There is no reasonable test that would have saved him.
The new supply even had the same model number!
https://reddit.com/r/DataHoarder/comments/1bilja1/egva_power...
But in any case, I was never blaming the guy. I'm sure I would have blown up my SATA drives in exactly the same way. I'm only saying that from now on, the paranoid, like me will always check the voltages on a new power supply output pins before I use them. He did actually check them after he realized the problem. It doesn't take long to check. You don't need to send it out to a lab as was suggested.