RDIMMs have buffers on the memory lines to improve signal integrity, this is only helpful when there are many slots on the same memory controller channel. (ex. Servers often support 4 DIMMs per channel)
I think RDIMMs are available in higher capacities than UDIMMs. But the registering adds additional latency. So if you need the capacity, RDIMM is the way to go. But if you don't, you might prefer UDIMMs.
I bought a second hand first gen EPYC for my desktop. I’m not sure it was a great idea overall but it does work fine and I have 128gb full ECC, two M2 slots on the board and many many PCIe 3x lanes for anything else I want to add. The built in BMC is handy for remote access.
The 32 cores is pretty pointless, I very rarely use more than half. And it probably uses way to much power.
I was pleasantly surprised a few days ago to find Supermicro sells a 3U chassis that runs 8x Ryzen 7000 series CPUs supporting ECC. If one doesn’t need more than 128GB of RAM per system then they can get much higher clock speeds at a much lower power envelope than an EPYC CPU with an equivalent core count.
That should be a much higher power envelope, no? I'm struggling to think of a workload that would like eight 8/12/16 core systems on a network more than one 64 or 128 core chip on the grounds that latency is harder to find than bandwidth.
The performance per dollar of Ryzen 7950X is many times higher than of any Epyc or Threadripper CPUs, even after adding the costs of the motherboards, coolers, cases and PSUs.
The DRAM bandwidth per core is identical for Ryzen 7950X and for 96-core Genoa Epyc CPUs. On the other hand, the Epyc CPUs with high-core count have a better performance per watt.
So the initial cost for a cluster of Ryzen servers is many times less. Depending on the cost of electricity, if an Epyc server is used 24/7, after some years the expenses with it may become lower than with Ryzen.
If the server is used intermittently, the total cost of ownership may remain lower with Ryzen until the end of life.
The only certain advantages of Epyc are the ability of aggregating a higher amount of memory, especially if it is preferable to have it inside a single box, and the faster inter-core communication for applications that use all cores (as opposed to the case when the cores are partitioned between weakly-coupled applications, e.g. between different virtual machines).
The prices of the Epyc CPUs have increased a lot since their first generation until now.
With Zen 1, a server with any Ryzen would not have been competitive with a server with Epyc. Meanwhile, the ratio between the prices per core of Epyc and Ryzen has increased a lot, while the ratio between the performance per core of Epyc and Ryzen has decreased a lot, because the clock frequency of Ryzens has become much greater while that of Epycs has increased only a little.
These two evolutions combined have made that now the servers with Ryzen have become preferable to servers with Epyc in many cases.
AMD has realized that they no longer have a solution for cheap servers, so in theory they have introduced the Siena CPUs for this purpose. Nevertheless, those remain somewhat too expensive and moreover they are nowhere to be seen.
I envy you. A life with zero problems.... I think you're probably the only person in the universe so fortunate.
Here's the basic set of problems:
1) ECC is almost free if done at scale (e.g. every computer). The only reason it costs extra is that it is only done for high-paying customers and in limited volume.
2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-surgery, you potentially pay.
3) It's good for the economy. In either case, the cost savings of not having a call center rep say "Wait a minute. My computer just crashed." far outweighs the costs.
4) It's critical for resilience. Disasters (solar flares, EMP, etc.) are rare, but if every computer destroys data, we have a systemic problem.
5) The issue primarily is one of transparency. Most people don't realize it's an issue, and don't shop on it.
6) That's not to mention dual-market uses. E.g. if we need to ramp up military production, medical production, etc. next crisis, our infrastructure should be ready for it.
Classical economic theory states government interventions do well for disaster resilience, for externalized costs, and for lack of transparency. Those are all places where markets fail. That means this should be regulated. We should have error correction in all storage and in all computation.
Best intervention, though, isn't a ban but a tax. Sales tax should be on a sliding scale from 1-10%, with:
- 1% for parts / support / service manuals being available for some extended period
- 1% for all software being open-source
- 1% for all hardware being open
- 1% for use of standard, interchangeable parts (e.g. batteries, power connectors, etc.)
- 1% for environmental factors (CO2, toxic materials, etc.)
- 1% for reusable infrastructure (e.g. being able to repurpose manufacturing equipment for crisis use)
- 1% for workplace factors (e.g reasonable benefits, audits/compliance, non-discrimination for target populations like ex-convicts, etc.)
... and so on
Those sorts of gentle incentives accomplish the same thing via market mechanisms without limiting freedom. For a commodity product, a 10% edge far outweighs costs. If Apple wants to make a proprietary magnetic screen connector and save 1mm of space, a 1% extra tax won't even slow them down.
> 1) ECC is almost free if done at scale (e.g. every computer). The only reason it costs extra is that it is only done for high-paying customers and in limited volume.
This is simply not true. ECC means you need to store a minimum of 11% more bits, and at scale this means that you are paying a minimum of 11% more for memory.
Note that I am very pro ECC, and think that most computers should use ECC ram.
As I have also said above, +12.5% paid by the manufacturer of DIMM for memory chips may mean +10% in the price of DIMMs when they are not marked up excessively, like the DDR5 modules today.
That +10% may become +5% after adding CPU, cooler, MB, PSU, case and it may become +3% after adding SSDs, a high-quality monitor and other usual peripherals.
While at work I have seen plenty of problems with computers without ECC, for my desktops and servers and even for some laptops, when I could, I have used only ECC memory, for the last few decades.
That has never increased the cost as much as I have saved by choosing wisely other components in the computers that I have built myself.
The main problem has never been the extra cost, but the difficulty to find them exactly when I needed them, because at retail the availability of ECC modules may be sporadic and many distributors appear and disappear, so if I upgrade systems after some years I seldom can find the same source.
I don't think the price of ram grows linearly with its bits. Manufacturing, shipping, distribution etc. are major costs that are mostly fixed for a single stick I would imagine.
You are assuming that the raw cost of the chips are 100% of the cost of selling one to the consumer. Other costs are fixed like quality control/testing, marketing, delivery, overhead for the distributors profit, etc.
I'd also expect RMA costs to be lower since you can be sure which DIMM is bad, and that it's not CPU, CPU socket, motherboard, powersupply, etc.
>1) ECC is almost free if done at scale (e.g. every computer). The only reason it costs extra is that it is only done for high-paying customers and in limited volume.
A ECC DRAM module contains 9 chips rather than 8. I'm not sure how you can claim that's "almost free" unless you have a very loose definition of "almost free".
>2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-surgery, you potentially pay.
Sounds like a good reason to regulate medical devices and force them to undergo certification as well. Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
>3) It's good for the economy. In either case, the cost savings of not having a call center rep say "Wait a minute. My computer just crashed." far outweighs the costs.
If this is true, why do we need the government to step in? Why aren't businesses doing this themselves out of self interest?
>4) It's critical for resilience. Disasters (solar flares, EMP, etc.) are rare, but if every computer destroys data, we have a systemic problem.
ECC protects against solar flares and EMP? That's new.
>6) That's not to mention dual-market uses. E.g. if we need to ramp up military production, medical production, etc. next crisis, our infrastructure should be ready for it.
1. DRAM modules aren't made in the US, they're made in asian countries with cheap labor. If there's a military crisis, we'd be screwed regardless
2. As mentioned above, ECC DRAM modules are basically the same as non-ECC modules but with an extra chip. The only work to retool a line from non-ECC to ECC is to change the PCBs. Thus the supposed benefits is negligible, unless for whatever reason the military needs a massive amount of desktops.
Almost free means that the 12.5% extra cost for memory chips may mean 10% extra manufacturing cost for the memory modules and that may mean less than 5% of the total cost of a computer with 64 GB DRAM (not counting monitor, SSDs and other peripherals, which would drive the percentage to be much lower) and even less for computers having less memory.
> A ECC DRAM module contains 9 chips rather than 8. I'm not sure how you can claim that's "almost free" unless you have a very loose definition of "almost free".
I do. Let's do the math here. A typical computer has 8GB RAM. That's under $20 at my local retail store. We're talking a $2 price difference. That's with the methods used today. The overhead can go arbitrarily low if we adopt techniques used in nonvolatile storage and work in larger blocks of bits.
$2 is less than the cost of a tiny fraction of a crash.
> Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
A broader regulatory regime would certainly be good.
> If this is true, why do we need the government to step in? Why aren't businesses doing this themselves out of self interest?
Mostly, because businesses can focus on a finite number of things. A $20 benefit which costs $2 to implement stops making sense if it means your local clinic needs to have someone on-staff who can do an ROI computation on adopting ECC. It's cheaper to buy something sub-optimal.
Again, that's where centralization and regulation helps.
> ECC protects against solar flares and EMP? That's new.
That's literally what they're there for. ECC primarily protects against bit flips, which are the result of random radiation. Mild flares will flip a few bits which ECC will correct. Severe ones will flip a lot of bits, which ECC will detect, and the computer can shut down before destroying more data.
> 1. DRAM modules aren't made in the US, they're made in asian countries with cheap labor. If there's a military crisis, we'd be screwed regardless
I'm not sure what you mean by "modules," but DRAM PCBs are easy to assemble. They can be made in the US overnight. DRAM chips are primarily made in the US (Micron), Korea (Samsung), and Taiwan (TSMC).
> The only work to retool a line from non-ECC to ECC is to change the PCBs. Thus the supposed benefits is negligible, unless for whatever reason the military needs a massive amount of desktops.
If all you need is memory, that's true. If you also want motherboards and processors which will take those chips, that's harder. If you don't want bit flips in your computation, that's impossible without redesigning all CPUs. Academic CPUs can do error-correcting computation at nominal cost, but there isn't a market incentive to bring that to market.
>>2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-surgery, you potentially pay.
>Sounds like a good reason to regulate medical devices and force them to undergo certification as well. Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
Did you know that medical reports are frequently viewed on consumer or consumer-grade hardware? Or that there is software that is classified as a class II medical device that runs on iOS devices and commodity desktops? Or that class III devices like pacemakers now often use phone apps for things like managing telemetry? It's all well and good to make sure the hardware in the OR is reliable, but that's woefully insufficient.
The power consumption is higher by an amount that is not easily detectable.
The power consumed by the relatively few gates that generate and verify the ECC code is dwarfed by the power consumption needed to transport bits over the PCB traces.
The extra memory cell arrays for ECC bits add some power consumption for refreshing their content and the extra PCB traces for the ECC bits add some power consumption, but this extra power is in the milliwatt range, so it is negligible in comparison with the CPU and GPU power even in a small laptop, not only in a desktop.
Most gaming desktops use overclocked memory modules that have an extra power consumption many times greater than the DDR5-4800 or DDR5-5600 ECC UDIMMs.
The ECC UDIMMs never need heatsinks, while many of the commonly used overclocked DIMMs will overheat without heatsinks.
So the claim that using ECC will increase the power consumption is completely bogus. Something like using a power-save profile instead of a high-performance profile, for clock management, or using JEDEC timings for memory, instead of high-performance timings, would diminish the power consumption by orders of magnitude more than not using ECC memory.
It's actually a bit more bogus than that. If I can tolerate occasional bit errors -- which ECC permits -- I can reduce my power consumption by far more than the overhead of ECC.
Right now, I power rails set to where billions of bits to never flip. If I can tolerate a one-in-a-billion bit flip, my tolerances go up a lot, and power consumption goes down by a similar factor.
Properly implemented, ECC would likely cut power while increasing reliability.
"Properly implemented" only happens in volume. That sort of optimization makes no sense for the very limited volumes ECC has today. Neither do a lot of other optimizations.
Still, those sorts of tweaks are more important for battery-powered devices. I'm not sure the impact on server power consumption would be important. RAM is not power-hungry relative to a Xeon or an H100.
2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-> surgery, you potentially pay.
Given that software can also crash, I'd go with a different approach: Redundant Array of Inexpensive Computers. For most situations, RAIC1 is good enough.
It's all fun and games until your computer starts acting all weird, like my 3 year old box did, last week. After being befuddled for a couple of hours, I decided to run MemTest86 off of an Ubuntu installer and found memory errors[1].
The manufacturer has RMA'd the whole set of memory sticks, but ECC would've detected this earlier and I'd have saved a couple of hours of my time.
If you've used the computer for rendering things or running heavy duty analysis, could you trust your results now, if you know that the memory was faulty for some time? ECC isn't about making life easy when the memory goes bad, it's about allowing one to have more confidence in their system operating correctly.
So you think everyone should be forced to accept higher prices and lower battery life permanently because of something that cost you a few hours, once, that most people never encounter?
Devices sold should work to a certain expected level. Regulators could define (or even just judges decide in a specific case that can be applied more generally) what that level is. Losing bits randomly isn't within the range of what is expected. The more memory there is, the more likely it is going to happen. And at some point devices without ECC could simply be returned as faulty for a full refund. ECC would be required, without any explicit rule.
That is already how it works now… you can already return defective ram, ECC or not.
I am happy with the point in the price/reliability/power usage tradeoff that non-ECC ram represents. Why force me to have fewer choices and use something that is worse for my needs in order to solve a theoretical problem that I don’t care about?
To me, it sounds like "if it doesn't work 100.00% of the time, we cannot trust it and should forbid it", but the reality is that neither the world (and even internet) nor electricity nor healthcare, are "100.00% available/reliable".
In general don't worry that for very important calculations, they are run a couple of times, at least just to verify.
Well the trick is a process or your entire OS crashed, why? Was it the powersupply? Motherboard? CPU? DIMM? Which DIMM? Are you going to keep a complete PC of spares to replace piece by piece to see if the crash stops?
For such a minimal cost I'd rather have DIMM 3 died with a non-correctable error.
Not really. Some errors are expected, are you going to mandate 2 GPUs because the Memory could fail and it could show artifacts on the screen? Electronics failure is always a possibility with a small but nonzero probability. If your RAM is faulty, you can already return it. That does not mean that every device has to have ECC to prevent very improbable bitflips.
There's no reason why any newly created file system shouldn't have checksumming, and if grandmothers don't know what's happening underneath the hood it's no worse than them not knowing how virtual memory or the TCP/IP stack work.
Zero problems? Very unlikely. The thing about memory corruption is that it tends to create problems that most people don't recognize as memory corruption issues. A random BSOD or freeze. A corrupted file. A video glitch. A website/JS code fragment misbehaving. Etc.
There's also the chance that you think you had zero problems because you never got notified.
Some years ago I built a small N3150 based home server with 16GB RAM. All was fine. But there was this one process that would always act weird after running for a couple of days. It was a Python server with long-running WebSocket connections which I had configured to use `select()` instead of `epoll()`, and for some reason it would eat up file handles until it crashed or froze, can't remember what it was, very odd stuff. I think it was running inside a Docker container and that container then behaved very strangely.
Turns out that when I then made a MemTest86 it had faulty RAM. All the data which passed through that area of the RAM and got written to disk was assumed to be corrupted.
After replacing the RAM (RMA) and doing a MemTest on it for a day to ensure that it was OK, that problem never appeared again.
But keep in mind, this was just one specific process which showed an issue after running for a couple of days, among other long-running processes.
1 >> I’ve done so my whole life with zero problems.
2 >> There's also the chance that you think you had zero problems because you never got notified.
Both are likely true. I.e. Over my 40 years of dealing with computers ,it is likely there were errors which did not become problems.
I have about 20tb of unique files in my personal storage at this point (used to run photography business, plus personal videos and photos,plus movies and tv shows, plus actually important documents which are less than 3% by size).I have about 4-8 computers running around my home doing stuff (plus dozens of computing devices - phones, googles, whatnots).
I understand the overwhelming chance there were many errors over time in running computers or raid transfers or usb backups over decades. They just haven't manifested themselves as problems.
The fact that you're dealing with silent corruption does not make it any better, but in fact, much worse.
And yes, the corruption is there. DRAM is alarmingly unreliable. Because no ECC, we just do not notice it most of the time, or misattribute the problems it causes.
If you're unlucky you get into mode with ZFS where every scrub corrupts more and more data, while ZFS itself thinks that all is OK because it computes the checksum on the data it just corrupted while scrubbing.
You would be surprised how often the system behaves unpredictably and it is because of RAM.
i had multiple times faulty ram and the symptoms are wildly different.
Also, when you download large files which you can run checksun against such as steam games, it is pretty common to find minor corruption (this i verified across many computers)
Anecdotal non-evidence. You've never noticed or had specific evidence of silent corruption that probabilistically risked silent corruption to quiesced to permanent storage or caused unexplained storage. Perhaps you've never heard of domain bitsquatting but it was rampant before https:// became widespread. All network gear, servers, and client devices need ECC or parity on all buses and at higher level storage, compute, and networking protocols to ensure integrity. Retail consumer electronics manufacturers throw out features customers lack the expertise to understand the consequences of. ECC RAM would need to be enforced by government mandate because that's the only way things will ever change. It's a massive, still present security and data integrity hole that could be closed today by spending a trivial amount of money if it were implemented at large scale.
Not a fan of using regulatory hammer to forbid people from having different priorities than I do. Adding across the board cost to computers has consequences.
I've had way more problems with defective security and bugs from Microsoft than I have with ECC bit-flipping errors. I would rather have "regulators step in" and nullify the disclaimer about "no fitness for any particular purpose" clauses in software license agreements. If I buy a windows computer and it gets hacked, I should be able to take MSFT to court (small claims if need be) and sue for damages (if I can show them) due to their negligence. They market PCs as these great tools for running everything in your life, but first they want release for any responsibility on their part. Those terms do not agree.
The frequency of software bugs is what has allowed the hardware manufacturers to omit ECC from most computer products.
Every time when a program crashes or remains stuck or data corruption is discovered, the user automatically assumes that this is a software bug.
While it is indeed true that software bugs are really much more frequent than memory errors before the memory modules happen to become defective, in a computer without ECC the user can never know whether it has been a software bug or it was a hardware memory problem.
So many memory errors remain hidden by the abundant software bugs, which in most commercial programs, especially in those from Microsoft, almost never generate error messages from which the cause of the error can be determined.
I suppose they'll claim that it's what doesn't trap which is important, e.g. what can be (silently) corrected by ECC would normally result in silent corruption without ECC.
Like, in what way - you get a windows popup saying "hey, just detected a bit error, but it was corrected, all good"? Because if yes, you'd get such a popup daily, and I'm not sure what the actual value of it is. And I'm not being funny, I literally don't see the value of it(of the notification, not the correcting).
You don't get a popup. In Linux, it's logged in dmesg. In Windows I imagine it goes in the system log. The system keeps on running.
The value of the notification is that if most days you get nothing and suddenly 5000 corrected errors pop up, you know something is not quite right as of late. Maybe you're overdoing the overclocking. Maybe the module isn't sitting quite well in the slot. Maybe a solder joint is failing.
If you have a 2 bit memory error, the typical result is a BSOD/kernel oops.
Which is a good thing actually. It means you know for sure your RAM is bad, and it prevents any faulty data from being written anywhere. So rather than possibly propagating the error is just stopped.
I believe there have been systems that try to handle it more elegantly. Kill the process the memory belongs to if it's a non-essential one then permanently blacklist that block to ensure it's not used again.
> Like, in what way - you get a windows popup saying "hey, just detected a bit error, but it was corrected, all good"?
No I've set it up so I get an email.
> Because if yes, you'd get such a popup daily
I doubt that. There was a study done at Google that reported 8% of DIMMs encountered at least one memory error per year[0].
> I'm not sure what the actual value of it is.
I like my PC to be as deterministic as possible[1] so if my programs crash I don't want to have to turn it off and test the memory which takes hours[2].
On a computer configured correctly, the errors are logged and the user or administrator may be notified when they happen.
In my experience the most valuable benefit of ECC is the detection of aged memory modules.
All modern DRAM has a limited lifetime. Hopefully that lifetime is more than 5 years, but there are always some modules that may reach end-of-life much faster, especially on computers that are always on.
While on a new memory module it would be abnormal for memory errors to happen more frequently than a few times per year, on an aged module errors may happen many times per day, or even many times per hour.
I have encountered a few such cases that happened after several years of continuous using. Due to ECC I was notified about the errors and I could replace the offending modules before them becoming so bad as to guarantee data corruption.
Another reason is that even if ECC cannot prevent a RowHammer attack, it makes pretty certain that a RowHammer attack will be detected immediately (because the attacker cannot control the location of the flipped bits, and even if there are many errors, so they will be miscorrected, in most cases they will still be detected and logged and the administrator should always be instantly notified when there are two or more errors per day, as this will never happen normally).
My thought also when I first heard about RowHammer - how do you avoid triggering an ECC error?
It seemed like you'd need multiple bit flips, possibly of specific bits.
I had a stick of RAM go bad in an old Socket 775 motherboard, but since it was one of 6 sticks of RAM, it only affected the PC on rare occasions. That PC was flaky for years before I pulled out some RAM to upgrade and found the problem had been fixed.
When a similar problem occurred on a PC with ECC RAM, I got a warning message saying memory errors had been detected that repeated a few times, then I got a message saying one stick of RAM had been deactivated and that I could keep running. The Dell T3500 dropped from 24 GB to 20 GB and did not even need to be rebooted.
That is why I love ECC RAM and Xeon processors. I know if there is a problem and can wait to fix it.
I can buy used RAM and processors with confidence that the BIOS self rests will tell me what is good and what is bad.
Some devices with 13th gen Intel processors support in-band ECC. This basically hides a fraction of RAM from the OS and uses it for ECC.
Example devices are: LattePanda Sigma [0] and AsRock Industrial NUCS BOX-1360P [1].
Unfortunately they are quite expensive and enabling in-band ECC lowers performance significantly. So my next server rig will likely have an AMD PRO CPU instead.
I find it annoying that (a) this seems to be a hardware feature that is almost universally permanently disabled in firmware and (b) it's almost impossible to find out whether it is supported by any given product (neither of the two products you link provide any mention on the spec sheet).
This inertia holding ECC away from desktops is just so dumb. As if me doing banking, taxes and everything else on desktop is less important than some server hosting cat pictures. Or as if ECC would truly represent any meaningful part on higher end system total cost. Or as if I would even have the option of simply paying some premium to get ECC, when in practice desktop Xeon availability and selection are both extremely limited.
Of course the benefit of ECC can be debated. But it seems such a waste to bother debating it when Intel could just make the whole thing a non-issue by a flick of a switch.
The entire computing industry is based on trading-off the probability & effects of failure with cost. Why would you even bother with ECC when practically no one except maybe server farms with tens of 24h-on servers would notice a thing? Even a single micro-cent per device spent on improving testing would have a much more felt-through effect on the entire computing landscape.
For desktop computers most of the value of ECC comes from not corrupting your hard disk when a ram stick spontaneously goes bad, rather than protecting against once in a blue moon bit flips
ECC is kind of useless when a RAM stick "spontaneously goes bad", unless precisely you are talking about the situation where a single bit goes bad (which is, indeed, less than once in a blue moon).
It will spew warnings if correctly setup and may even be able to correct for the fault. Anything that highlights the problem before your unstable system silently corrupts files (and their backups) or hoses its filesystem is a godsend
This one does pervasive damage to software quality though. You get a bug report, it doesn't reproduce easily. The thought is always there: "maybe their hardware had a transient error?". If it did you have no bug to find. That's attractive.
If storage drives reported writes accurately and memory didn't occasionally silently corrupt, software falling over would be more likely to imply an error the developer is empowered to fix.
ECC RAM doesn't prevent _all_ hardware transient errors. It doesn't even prevent the majority of them.
But whenever you get a bug report, is really the first thing that comes to mind "maybe their hardware had a transient error?"? The amount of software vs hardware bugs is still like a thousand to one, and that's _even_ for companies which receive a disproportionate amount of hardware errors (like OS vendors). For regular software, I'd bet it's a couple million to one.
It's market segmentation. You want to sell expensive chips to people with money and cheap ones to people without. That's hard to do if the products seem to be equivalent. ECC availability is part of the marketing skill with which people buy Xeon branded processors.
This isn't inherently evil behaviour.
AMD is interesting in this regard. ECC on desktops. Lots of x86 chips have some integrated GPU. Lots of PCIe lanes. The idea seems to be biased much more towards build a great platform, as opposed to build a great profit margin.
Ecc is still segmented on the amd apu lineup, including desktop models (and for many years these were the only models with integrated graphics!), and there literally is no way for consumers to even buy some of the parts that AMD has segmented these features to. So there is not some moral imperative that AMD has undertaken here - AMD backs my "right" to ECC right up until I want to use it on a 5700G, or buy a 5750G Pro, which they won't even sell to me as a consumer!
(just like AMD also segmented based on SMT up through at least the ryzen 4000 series despite the fans bashing intel for the same, lol)
Moreover, with amd one increasingly has to worry about the platform lock, which is an absolutely obscene e-waste stream AMD has created to reduce the secondhand sales market. This isn’t limited to server CPUs or pro CPUs, notionally it affects all CPUs that might have ever passed through an oem system, including non-pro consumer CPUs etc.
The stated reason is security/antitheft, but the platform lock doesn't lock a CPU to a motherboard, it locks it to a brand. So if someone has physical access to your server rack and wants to steal the CPU, they just have to find a lenovo-branded or hp-branded chip to swap it out with... and there are rivers of those because locked chips are essentially treated as e-waste. What it does do very successfully is throw a whole cloud of confusion and pain over the secondhand market and drive these to be scrapped etc and push sales of new devices instead.
AMD could very easily have implemented a lock/unlock system using non-volatile storage or simple e-fuse even/odd counting (limited number of locks/unlocks, granted, but it carries with the CPU not the mobo, and how many systems is the average CPU used in?). They, naturally, chose the one that is permanent but bricks the cpu for everyone else, but which also doesn't really fully prevent CPU swapping attacks either. Plenty of e-fuses to spare for detecting memory overclocking though - we all know denying warranties comes first ;)
For the type of person who gets wound up about the moral imperative of ecc, that’s not exactly what you want to see from AMD either. First 2 Rs are "reduce" and "re-use", followed by "recycle". Like how about we don't tivoize the cpu in 2018 or whatever? Secondhand sales are just as much of a moral right and AMD is really having the old college try at taking that away, and frankly a lot more impactful to the average user. I strongly wish the EU would step in and ban this practice. This is not a good/healthy path, it would actually be really bad if everyone was doing this on their products.
Also, Intel has ECC support on the upper half of their range. Yes, it requires motherboard support, but you also have to pick and choose with AMD because only a handful of motherboards actually support/validate it anyway. Plus with AMD you have the fTPM issue (and very few of the premium creator boards/etc have external TPM headers). This is the reality as a homelab owner - you have to pick your hardware very carefully regardless of brand, a random B650 board probably is not going to report errors and may not even detect them. And while it may work on any given BIOS, it may stop working at any future point, whether intentionally or not.
I thought the APU chips also supported ECC though some motherboards don't. That could well be wrong though, the messaging seems quite muddied. In fairness my NAS is an intel i3 with ECC support so it's not a simple Xeon/not split there either.
I remember news stories about the platform lock but haven't kept up to date, do you know how that played out in practice?
My asrock board doesn't seem to have an option to burn out fuses to stop it working elsewhere but I am slightly less cavalier about changing settings knowing that might be in there.
ECC support is segmented to Pro APUs only. Which aren't even sold to consumers.
Yeah the lack of iGPU support (plex encoding, VMs, etc) pushed me to the 7100 last time I did a homeserver build. It's been muddled on intel's line forever - for a while it was only i3s (since they dropped the idea of a 2C xeon there was no reason to keep segmenting it) but then with alder lake they flipped it and it's on everything 12500 and above (except F skus). And while you do need a motherboard that supports it, W680 supports overclocking/XMP/etc - the tier of boards I'm looking at aren't materially different in cost (or actually intel often trends cheaper).
The platform lock has actually affected me quite badly trying to do a ROMED8-2T build, a massive number of the desirable CPUs are platform locked and unless they explicitly say unlocked it's always a question whether any given chip is actually unlocked or just doesn't say/doesn't know. Which is what I see as the whole point - get those machines into the scrapper (zero value means nobody bothers reselling) instead of people buying perfectly good older cpus and drawing extra life from them.
I am not thrilled about the idea of that bleeding into the consumer market at all. So far AMD hasn't gotten much traction with OEMs but that's changing over time, and I see every reason for AMD to incentivize OEM partners to use this functionality across their whole lineups - OEMs want cheaper chips and AMD can sell them cheaper because they know they are junk after the first sale. Why wouldn't ThinkStation SFF PCs etc come with locked processors at some point? AMD clearly has laid the groundwork and all the tooling is there and just is boiling the frog very cautiously.
By the time people see the impact in the consumer market it will be too late. And there are a lot more ThinkStations and Optiplexes sold than DIY PCs.
Torvalds went on his lengthy post to say, "The "modern DRAM is so reliable that it doesn't need ECC" was always a bedtime story for children that had been dropped on their heads a bit too many times. Yes, I'm pissed off about it. You can find me complaining about this literally for decades now. I don't want to say "I was right". I want this fixed, and I want ECC. And AMD did it. Intel didn't."
"When plotting the benchmarks with a statistically significant difference, there was just a handful of benchmarks within the 2~3% faster range when ECC was disabled. As shown earlier, the Graph500 HPC benchmark was the outlier with a ~8% difference toggling ECC. PostgreSQL and Hadoop were among the few real-world workloads with the 2~3% difference, which is a small price to pay for the benefits of ECC.
It's great that the desktop AMD Ryzen processors continue to support ECC memory while still it's ultimately up to the motherboard vendors whether ECC is to be supported. We're at least seeing more AMD Ryzen motherboards come to market with a server focus which in turn is great for allowing more motherboard options with ECC memory support. My testing with the ASRock Rack 1U4LW-B650/2L2T 1U server (ASRockRack B650D4U-2L2T/BCM motherboard) continues working out very well and hasn't yielded any platform troubles with all of my hundreds of hours of Linux testing thus far."
Running azrock mb with a ryzen 9 3900x and Samsung b die ram… it’s pile of crap compared to asus mobo with the same plugins… ecc, I don’t know… what’s a proper ecc test today and is it worth sacrificing 12% in performance?
I tested that ECC was working on my Zen 3 + ASRock B550 setup by gradually undervolting the RAM more and more, and running Memtester[0] each time until Linux reported corrected errors.
No memory capacity or memory throughput is lost in consumer motherboards when enabling ECC.
You just need to buy ECC UDIMM modules, which, especially for DDR5, are more expensive, up to 50% more expensive for DDR5, even if this price difference is not justified.
Moreover, the ECC modules are never factory overclocked, like most non-ECC modules, therefore ECC UDIMM modules faster than DDR5-4800 are hard to find. Nevertheless, that has not stopped some users to overclock them manually.
There are some embedded computers for industrial applications, with certain models of Intel Atom or Core CPUs, which support the so-called in-band ECC, which can be used with non-ECC memory, like LPDDR5. Only on those computers enabling ECC reduces both the available memory and the performance.
No mention of on die ECC so far so I'll mention it. It's used in modern RAM (DDR5, maybe some DDR4) to improve yields. A borderline chip may pass tests if occasional bit errors are corrected before being read out. This is transparent to the OS as there is no mechanism that I'm aware of to report fixed bits or unfixable errors.
I'm hopeful that this is going to help fix problems with occasional bit flips from cosmic rays and similar. Since it is used to keep marginal chips in the product stream, I'm not sure if it is really beneficial.
The Raspberry Pi 4B claims DDR4 ECC RAM and I'm pretty sure that's on die ECC.
I'd really prefer full blown ECC RAM and would pay a reasonable increase for it. I recently upgraded my desktop and sticking with ECC RAM would have cost hundreds of dollars more so I skipped it.
ECC without error reporting still won't save one from data corruption. Ultimately, I want correctness not continuous but faulty operation. On-die ECC helps with the latter.
Agreed. Reliability is the most important part of "reliability, availability and serviceability"[0]. Error correction is nice to have, but even simple parity checking is much better than nothing. I'd rather halt with a kernel panic than continue using corrupted data.
It really depends on the application. Halting rather than displaying a slightly corrupted graphics texture is probably not what gamers want. Swallowing an exception in a finite element analysis is probably not what engineers want.
It does depend on the application. And there's a whole lot of software for which the developers don't have complete enough understanding of everything downstream to really know the significance of incorrect output.
There are loads of different ways to "have ECC"... it isn't as binary as the post implies. Different methods can detect and correct different failures.
What's an affordable x86_64 NUC with ECC RAM? I remember getting frustrated trying to find one that I just abandoned the idea of making a home server altogether.
Please, I don't want to hear "you don't need ECC", I just want some recommendations.
112 comments
[ 2.9 ms ] story [ 178 ms ] threadIt seems RDIMM’s are mostly supported on higher end server boards but I haven’t seen a technical comparison of the two types.
The 32 cores is pretty pointless, I very rarely use more than half. And it probably uses way to much power.
https://www.supermicro.com/en/products/motherboard/h13sae-mf
The DRAM bandwidth per core is identical for Ryzen 7950X and for 96-core Genoa Epyc CPUs. On the other hand, the Epyc CPUs with high-core count have a better performance per watt.
So the initial cost for a cluster of Ryzen servers is many times less. Depending on the cost of electricity, if an Epyc server is used 24/7, after some years the expenses with it may become lower than with Ryzen.
If the server is used intermittently, the total cost of ownership may remain lower with Ryzen until the end of life.
The only certain advantages of Epyc are the ability of aggregating a higher amount of memory, especially if it is preferable to have it inside a single box, and the faster inter-core communication for applications that use all cores (as opposed to the case when the cores are partitioned between weakly-coupled applications, e.g. between different virtual machines).
The prices of the Epyc CPUs have increased a lot since their first generation until now.
With Zen 1, a server with any Ryzen would not have been competitive with a server with Epyc. Meanwhile, the ratio between the prices per core of Epyc and Ryzen has increased a lot, while the ratio between the performance per core of Epyc and Ryzen has decreased a lot, because the clock frequency of Ryzens has become much greater while that of Epycs has increased only a little.
These two evolutions combined have made that now the servers with Ryzen have become preferable to servers with Epyc in many cases.
AMD has realized that they no longer have a solution for cheap servers, so in theory they have introduced the Siena CPUs for this purpose. Nevertheless, those remain somewhat too expensive and moreover they are nowhere to be seen.
Otherwise, organizations such as Microsoft (windows readyness logo) or RISC-V (application profiles such as RVA23) could make it a requirement.
It is clear that we can't just rely on the hardware vendors to do it, and practically nobody should be running computers w/o ECC.
Why? I’ve done so my whole life with zero problems.
Here's the basic set of problems:
1) ECC is almost free if done at scale (e.g. every computer). The only reason it costs extra is that it is only done for high-paying customers and in limited volume.
2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-surgery, you potentially pay.
3) It's good for the economy. In either case, the cost savings of not having a call center rep say "Wait a minute. My computer just crashed." far outweighs the costs.
4) It's critical for resilience. Disasters (solar flares, EMP, etc.) are rare, but if every computer destroys data, we have a systemic problem.
5) The issue primarily is one of transparency. Most people don't realize it's an issue, and don't shop on it.
6) That's not to mention dual-market uses. E.g. if we need to ramp up military production, medical production, etc. next crisis, our infrastructure should be ready for it.
Classical economic theory states government interventions do well for disaster resilience, for externalized costs, and for lack of transparency. Those are all places where markets fail. That means this should be regulated. We should have error correction in all storage and in all computation.
Best intervention, though, isn't a ban but a tax. Sales tax should be on a sliding scale from 1-10%, with:
- 1% for parts / support / service manuals being available for some extended period
- 1% for all software being open-source
- 1% for all hardware being open
- 1% for use of standard, interchangeable parts (e.g. batteries, power connectors, etc.)
- 1% for environmental factors (CO2, toxic materials, etc.)
- 1% for reusable infrastructure (e.g. being able to repurpose manufacturing equipment for crisis use)
- 1% for workplace factors (e.g reasonable benefits, audits/compliance, non-discrimination for target populations like ex-convicts, etc.)
... and so on
Those sorts of gentle incentives accomplish the same thing via market mechanisms without limiting freedom. For a commodity product, a 10% edge far outweighs costs. If Apple wants to make a proprietary magnetic screen connector and save 1mm of space, a 1% extra tax won't even slow them down.
This is simply not true. ECC means you need to store a minimum of 11% more bits, and at scale this means that you are paying a minimum of 11% more for memory.
Note that I am very pro ECC, and think that most computers should use ECC ram.
That +10% may become +5% after adding CPU, cooler, MB, PSU, case and it may become +3% after adding SSDs, a high-quality monitor and other usual peripherals.
While at work I have seen plenty of problems with computers without ECC, for my desktops and servers and even for some laptops, when I could, I have used only ECC memory, for the last few decades.
That has never increased the cost as much as I have saved by choosing wisely other components in the computers that I have built myself.
The main problem has never been the extra cost, but the difficulty to find them exactly when I needed them, because at retail the availability of ECC modules may be sporadic and many distributors appear and disappear, so if I upgrade systems after some years I seldom can find the same source.
I'd also expect RMA costs to be lower since you can be sure which DIMM is bad, and that it's not CPU, CPU socket, motherboard, powersupply, etc.
A ECC DRAM module contains 9 chips rather than 8. I'm not sure how you can claim that's "almost free" unless you have a very loose definition of "almost free".
>2) Computer crashes cost lives. If e.g. your doctor's device has an issue mid-surgery, you potentially pay.
Sounds like a good reason to regulate medical devices and force them to undergo certification as well. Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
>3) It's good for the economy. In either case, the cost savings of not having a call center rep say "Wait a minute. My computer just crashed." far outweighs the costs.
If this is true, why do we need the government to step in? Why aren't businesses doing this themselves out of self interest?
>4) It's critical for resilience. Disasters (solar flares, EMP, etc.) are rare, but if every computer destroys data, we have a systemic problem.
ECC protects against solar flares and EMP? That's new.
>6) That's not to mention dual-market uses. E.g. if we need to ramp up military production, medical production, etc. next crisis, our infrastructure should be ready for it.
1. DRAM modules aren't made in the US, they're made in asian countries with cheap labor. If there's a military crisis, we'd be screwed regardless
2. As mentioned above, ECC DRAM modules are basically the same as non-ECC modules but with an extra chip. The only work to retool a line from non-ECC to ECC is to change the PCBs. Thus the supposed benefits is negligible, unless for whatever reason the military needs a massive amount of desktops.
I do. Let's do the math here. A typical computer has 8GB RAM. That's under $20 at my local retail store. We're talking a $2 price difference. That's with the methods used today. The overhead can go arbitrarily low if we adopt techniques used in nonvolatile storage and work in larger blocks of bits.
$2 is less than the cost of a tiny fraction of a crash.
> Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
A broader regulatory regime would certainly be good.
> If this is true, why do we need the government to step in? Why aren't businesses doing this themselves out of self interest?
Mostly, because businesses can focus on a finite number of things. A $20 benefit which costs $2 to implement stops making sense if it means your local clinic needs to have someone on-staff who can do an ROI computation on adopting ECC. It's cheaper to buy something sub-optimal.
Again, that's where centralization and regulation helps.
> ECC protects against solar flares and EMP? That's new.
That's literally what they're there for. ECC primarily protects against bit flips, which are the result of random radiation. Mild flares will flip a few bits which ECC will correct. Severe ones will flip a lot of bits, which ECC will detect, and the computer can shut down before destroying more data.
> 1. DRAM modules aren't made in the US, they're made in asian countries with cheap labor. If there's a military crisis, we'd be screwed regardless
I'm not sure what you mean by "modules," but DRAM PCBs are easy to assemble. They can be made in the US overnight. DRAM chips are primarily made in the US (Micron), Korea (Samsung), and Taiwan (TSMC).
https://www.micron.com/manufacturing-expansion
And not all crises are military.
> The only work to retool a line from non-ECC to ECC is to change the PCBs. Thus the supposed benefits is negligible, unless for whatever reason the military needs a massive amount of desktops.
If all you need is memory, that's true. If you also want motherboards and processors which will take those chips, that's harder. If you don't want bit flips in your computation, that's impossible without redesigning all CPUs. Academic CPUs can do error-correcting computation at nominal cost, but there isn't a market incentive to bring that to market.
>Sounds like a good reason to regulate medical devices and force them to undergo certification as well. Surely you wouldn't want the the computer to have ECC memory, but it decides to reboot for a 2 hour update?
Did you know that medical reports are frequently viewed on consumer or consumer-grade hardware? Or that there is software that is classified as a class II medical device that runs on iOS devices and commodity desktops? Or that class III devices like pacemakers now often use phone apps for things like managing telemetry? It's all well and good to make sure the hardware in the OR is reliable, but that's woefully insufficient.
The power consumed by the relatively few gates that generate and verify the ECC code is dwarfed by the power consumption needed to transport bits over the PCB traces.
The extra memory cell arrays for ECC bits add some power consumption for refreshing their content and the extra PCB traces for the ECC bits add some power consumption, but this extra power is in the milliwatt range, so it is negligible in comparison with the CPU and GPU power even in a small laptop, not only in a desktop.
Most gaming desktops use overclocked memory modules that have an extra power consumption many times greater than the DDR5-4800 or DDR5-5600 ECC UDIMMs.
The ECC UDIMMs never need heatsinks, while many of the commonly used overclocked DIMMs will overheat without heatsinks.
So the claim that using ECC will increase the power consumption is completely bogus. Something like using a power-save profile instead of a high-performance profile, for clock management, or using JEDEC timings for memory, instead of high-performance timings, would diminish the power consumption by orders of magnitude more than not using ECC memory.
Right now, I power rails set to where billions of bits to never flip. If I can tolerate a one-in-a-billion bit flip, my tolerances go up a lot, and power consumption goes down by a similar factor.
Properly implemented, ECC would likely cut power while increasing reliability.
"Properly implemented" only happens in volume. That sort of optimization makes no sense for the very limited volumes ECC has today. Neither do a lot of other optimizations.
Still, those sorts of tweaks are more important for battery-powered devices. I'm not sure the impact on server power consumption would be important. RAM is not power-hungry relative to a Xeon or an H100.
Given that software can also crash, I'd go with a different approach: Redundant Array of Inexpensive Computers. For most situations, RAIC1 is good enough.
The manufacturer has RMA'd the whole set of memory sticks, but ECC would've detected this earlier and I'd have saved a couple of hours of my time.
[1]: https://imgur.com/a/aRHWKxw
And folks experienced zero file corruption… until they started using ZFS checksums and regular scrubbing.
I am happy with the point in the price/reliability/power usage tradeoff that non-ECC ram represents. Why force me to have fewer choices and use something that is worse for my needs in order to solve a theoretical problem that I don’t care about?
In general don't worry that for very important calculations, they are run a couple of times, at least just to verify.
Citation needed, as this is unheard of in software we trust regardless to do banking or sign legally binding documents with.
For such a minimal cost I'd rather have DIMM 3 died with a non-correctable error.
Measures are in the works there, such as the switch to a simpler and cleaner ISA.
The value of simplicity is often overlooked; RISC-V will lead to higher quality software and hardware.
But I digress, as this has no weight in whether or not lack of ECC is acceptable.
* https://arstechnica.com/gadgets/2009/10/apple-abandons-zfs-o...
* https://web.archive.org/web/20091103163642/http://mail.opens...
There's no reason why any newly created file system shouldn't have checksumming, and if grandmothers don't know what's happening underneath the hood it's no worse than them not knowing how virtual memory or the TCP/IP stack work.
Some years ago I built a small N3150 based home server with 16GB RAM. All was fine. But there was this one process that would always act weird after running for a couple of days. It was a Python server with long-running WebSocket connections which I had configured to use `select()` instead of `epoll()`, and for some reason it would eat up file handles until it crashed or froze, can't remember what it was, very odd stuff. I think it was running inside a Docker container and that container then behaved very strangely.
Turns out that when I then made a MemTest86 it had faulty RAM. All the data which passed through that area of the RAM and got written to disk was assumed to be corrupted.
After replacing the RAM (RMA) and doing a MemTest on it for a day to ensure that it was OK, that problem never appeared again.
But keep in mind, this was just one specific process which showed an issue after running for a couple of days, among other long-running processes.
2 >> There's also the chance that you think you had zero problems because you never got notified.
Both are likely true. I.e. Over my 40 years of dealing with computers ,it is likely there were errors which did not become problems.
I have about 20tb of unique files in my personal storage at this point (used to run photography business, plus personal videos and photos,plus movies and tv shows, plus actually important documents which are less than 3% by size).I have about 4-8 computers running around my home doing stuff (plus dozens of computing devices - phones, googles, whatnots).
I understand the overwhelming chance there were many errors over time in running computers or raid transfers or usb backups over decades. They just haven't manifested themselves as problems.
And yes, the corruption is there. DRAM is alarmingly unreliable. Because no ECC, we just do not notice it most of the time, or misattribute the problems it causes.
See https://www.truenas.com/community/threads/ecc-vs-non-ecc-ram...
Also, when you download large files which you can run checksun against such as steam games, it is pretty common to find minor corruption (this i verified across many computers)
See also: PDF https://media.defcon.org/DEF%20CON%2021/DEF%20CON%2021%20pre...
https://cr.yp.to/hardware/ecc.html
Them not having ECC is insanity.
There are of course niche cases where memory reliability does not matter. Exceptions can be made for those few.
But think about the general case first. Got to protect the non-technical from non-ECC memory.
I guess the computing world has not survived up to this point then?
freedom TO vs freedom FROM
Every time when a program crashes or remains stuck or data corruption is discovered, the user automatically assumes that this is a software bug.
While it is indeed true that software bugs are really much more frequent than memory errors before the memory modules happen to become defective, in a computer without ECC the user can never know whether it has been a software bug or it was a hardware memory problem.
So many memory errors remain hidden by the abundant software bugs, which in most commercial programs, especially in those from Microsoft, almost never generate error messages from which the cause of the error can be determined.
Correct single bit errors and alert me to them.
The value of the notification is that if most days you get nothing and suddenly 5000 corrected errors pop up, you know something is not quite right as of late. Maybe you're overdoing the overclocking. Maybe the module isn't sitting quite well in the slot. Maybe a solder joint is failing.
If you have a 2 bit memory error, the typical result is a BSOD/kernel oops.
Which is a good thing actually. It means you know for sure your RAM is bad, and it prevents any faulty data from being written anywhere. So rather than possibly propagating the error is just stopped.
I believe there have been systems that try to handle it more elegantly. Kill the process the memory belongs to if it's a non-essential one then permanently blacklist that block to ensure it's not used again.
No I've set it up so I get an email.
> Because if yes, you'd get such a popup daily
I doubt that. There was a study done at Google that reported 8% of DIMMs encountered at least one memory error per year[0].
> I'm not sure what the actual value of it is.
I like my PC to be as deterministic as possible[1] so if my programs crash I don't want to have to turn it off and test the memory which takes hours[2].
[0] - https://www.cs.toronto.edu/~bianca/papers/sigmetrics09.pdf
[1] - https://blog.cskr.dev/posts/corrupt-backup/
[2] - https://www.youtube.com/watch?v=0itrM7t4l34
In my experience the most valuable benefit of ECC is the detection of aged memory modules.
All modern DRAM has a limited lifetime. Hopefully that lifetime is more than 5 years, but there are always some modules that may reach end-of-life much faster, especially on computers that are always on.
While on a new memory module it would be abnormal for memory errors to happen more frequently than a few times per year, on an aged module errors may happen many times per day, or even many times per hour.
I have encountered a few such cases that happened after several years of continuous using. Due to ECC I was notified about the errors and I could replace the offending modules before them becoming so bad as to guarantee data corruption.
Another reason is that even if ECC cannot prevent a RowHammer attack, it makes pretty certain that a RowHammer attack will be detected immediately (because the attacker cannot control the location of the flipped bits, and even if there are many errors, so they will be miscorrected, in most cases they will still be detected and logged and the administrator should always be instantly notified when there are two or more errors per day, as this will never happen normally).
When a similar problem occurred on a PC with ECC RAM, I got a warning message saying memory errors had been detected that repeated a few times, then I got a message saying one stick of RAM had been deactivated and that I could keep running. The Dell T3500 dropped from 24 GB to 20 GB and did not even need to be rebooted.
That is why I love ECC RAM and Xeon processors. I know if there is a problem and can wait to fix it.
I can buy used RAM and processors with confidence that the BIOS self rests will tell me what is good and what is bad.
Example devices are: LattePanda Sigma [0] and AsRock Industrial NUCS BOX-1360P [1].
Unfortunately they are quite expensive and enabling in-band ECC lowers performance significantly. So my next server rig will likely have an AMD PRO CPU instead.
[0] https://www.servethehome.com/lattepanda-sigma-review-the-ras... [1] https://www.anandtech.com/show/18732/asrock-industrial-nucs-...
Of course the benefit of ECC can be debated. But it seems such a waste to bother debating it when Intel could just make the whole thing a non-issue by a flick of a switch.
If storage drives reported writes accurately and memory didn't occasionally silently corrupt, software falling over would be more likely to imply an error the developer is empowered to fix.
But whenever you get a bug report, is really the first thing that comes to mind "maybe their hardware had a transient error?"? The amount of software vs hardware bugs is still like a thousand to one, and that's _even_ for companies which receive a disproportionate amount of hardware errors (like OS vendors). For regular software, I'd bet it's a couple million to one.
This isn't inherently evil behaviour.
AMD is interesting in this regard. ECC on desktops. Lots of x86 chips have some integrated GPU. Lots of PCIe lanes. The idea seems to be biased much more towards build a great platform, as opposed to build a great profit margin.
(just like AMD also segmented based on SMT up through at least the ryzen 4000 series despite the fans bashing intel for the same, lol)
Moreover, with amd one increasingly has to worry about the platform lock, which is an absolutely obscene e-waste stream AMD has created to reduce the secondhand sales market. This isn’t limited to server CPUs or pro CPUs, notionally it affects all CPUs that might have ever passed through an oem system, including non-pro consumer CPUs etc.
https://youtu.be/bFNJVaO9E-o?t=420
The stated reason is security/antitheft, but the platform lock doesn't lock a CPU to a motherboard, it locks it to a brand. So if someone has physical access to your server rack and wants to steal the CPU, they just have to find a lenovo-branded or hp-branded chip to swap it out with... and there are rivers of those because locked chips are essentially treated as e-waste. What it does do very successfully is throw a whole cloud of confusion and pain over the secondhand market and drive these to be scrapped etc and push sales of new devices instead.
AMD could very easily have implemented a lock/unlock system using non-volatile storage or simple e-fuse even/odd counting (limited number of locks/unlocks, granted, but it carries with the CPU not the mobo, and how many systems is the average CPU used in?). They, naturally, chose the one that is permanent but bricks the cpu for everyone else, but which also doesn't really fully prevent CPU swapping attacks either. Plenty of e-fuses to spare for detecting memory overclocking though - we all know denying warranties comes first ;)
For the type of person who gets wound up about the moral imperative of ecc, that’s not exactly what you want to see from AMD either. First 2 Rs are "reduce" and "re-use", followed by "recycle". Like how about we don't tivoize the cpu in 2018 or whatever? Secondhand sales are just as much of a moral right and AMD is really having the old college try at taking that away, and frankly a lot more impactful to the average user. I strongly wish the EU would step in and ban this practice. This is not a good/healthy path, it would actually be really bad if everyone was doing this on their products.
Also, Intel has ECC support on the upper half of their range. Yes, it requires motherboard support, but you also have to pick and choose with AMD because only a handful of motherboards actually support/validate it anyway. Plus with AMD you have the fTPM issue (and very few of the premium creator boards/etc have external TPM headers). This is the reality as a homelab owner - you have to pick your hardware very carefully regardless of brand, a random B650 board probably is not going to report errors and may not even detect them. And while it may work on any given BIOS, it may stop working at any future point, whether intentionally or not.
I remember news stories about the platform lock but haven't kept up to date, do you know how that played out in practice?
My asrock board doesn't seem to have an option to burn out fuses to stop it working elsewhere but I am slightly less cavalier about changing settings knowing that might be in there.
Yeah the lack of iGPU support (plex encoding, VMs, etc) pushed me to the 7100 last time I did a homeserver build. It's been muddled on intel's line forever - for a while it was only i3s (since they dropped the idea of a 2C xeon there was no reason to keep segmenting it) but then with alder lake they flipped it and it's on everything 12500 and above (except F skus). And while you do need a motherboard that supports it, W680 supports overclocking/XMP/etc - the tier of boards I'm looking at aren't materially different in cost (or actually intel often trends cheaper).
The platform lock has actually affected me quite badly trying to do a ROMED8-2T build, a massive number of the desirable CPUs are platform locked and unless they explicitly say unlocked it's always a question whether any given chip is actually unlocked or just doesn't say/doesn't know. Which is what I see as the whole point - get those machines into the scrapper (zero value means nobody bothers reselling) instead of people buying perfectly good older cpus and drawing extra life from them.
I am not thrilled about the idea of that bleeding into the consumer market at all. So far AMD hasn't gotten much traction with OEMs but that's changing over time, and I see every reason for AMD to incentivize OEM partners to use this functionality across their whole lineups - OEMs want cheaper chips and AMD can sell them cheaper because they know they are junk after the first sale. Why wouldn't ThinkStation SFF PCs etc come with locked processors at some point? AMD clearly has laid the groundwork and all the tooling is there and just is boiling the frog very cautiously.
By the time people see the impact in the consumer market it will be too late. And there are a lot more ThinkStations and Optiplexes sold than DIY PCs.
Torvalds went on his lengthy post to say, "The "modern DRAM is so reliable that it doesn't need ECC" was always a bedtime story for children that had been dropped on their heads a bit too many times. Yes, I'm pissed off about it. You can find me complaining about this literally for decades now. I don't want to say "I was right". I want this fixed, and I want ECC. And AMD did it. Intel didn't."
(via https://www.phoronix.com/news/Linus-Torvalds-ECC )
https://www.phoronix.com/review/amd-ryzen9-ddr5-ecc
"When plotting the benchmarks with a statistically significant difference, there was just a handful of benchmarks within the 2~3% faster range when ECC was disabled. As shown earlier, the Graph500 HPC benchmark was the outlier with a ~8% difference toggling ECC. PostgreSQL and Hadoop were among the few real-world workloads with the 2~3% difference, which is a small price to pay for the benefits of ECC. It's great that the desktop AMD Ryzen processors continue to support ECC memory while still it's ultimately up to the motherboard vendors whether ECC is to be supported. We're at least seeing more AMD Ryzen motherboards come to market with a server focus which in turn is great for allowing more motherboard options with ECC memory support. My testing with the ASRock Rack 1U4LW-B650/2L2T 1U server (ASRockRack B650D4U-2L2T/BCM motherboard) continues working out very well and hasn't yielded any platform troubles with all of my hundreds of hours of Linux testing thus far."
I wish more motherboards bothered to validate ECC working - it's a nice benefit in an affordable platform.
"It works" as in it boots, or "it works" as in it checks and reports single-bit errors and reports double-bit errors?
What OS? If Linux, are you using rasdaemon? Does it confirm that the appropriate EDAC drivers are loaded and working?
Or is that data only available in the BMC's event log?
For Linux support you need a very recent kernel - I forget how recent, but some reporting on it: https://www.phoronix.com/news/AMD-EDAC-Ryzen-7000-Series
[0] https://pyropus.ca./software/memtester/
The new Asrock Ryzen 7xxx motherboards unfortunately don't have that same ECC support.
You just need to buy ECC UDIMM modules, which, especially for DDR5, are more expensive, up to 50% more expensive for DDR5, even if this price difference is not justified.
Moreover, the ECC modules are never factory overclocked, like most non-ECC modules, therefore ECC UDIMM modules faster than DDR5-4800 are hard to find. Nevertheless, that has not stopped some users to overclock them manually.
There are some embedded computers for industrial applications, with certain models of Intel Atom or Core CPUs, which support the so-called in-band ECC, which can be used with non-ECC memory, like LPDDR5. Only on those computers enabling ECC reduces both the available memory and the performance.
I'm hopeful that this is going to help fix problems with occasional bit flips from cosmic rays and similar. Since it is used to keep marginal chips in the product stream, I'm not sure if it is really beneficial.
The Raspberry Pi 4B claims DDR4 ECC RAM and I'm pretty sure that's on die ECC.
I'd really prefer full blown ECC RAM and would pay a reasonable increase for it. I recently upgraded my desktop and sticking with ECC RAM would have cost hundreds of dollars more so I skipped it.
[0] https://en.wikipedia.org/wiki/Reliability,_availability_and_...
Please, I don't want to hear "you don't need ECC", I just want some recommendations.