“ECC availability matters a lot - exactly because Intel has been instrumental in killing the whole ECC industry with it's horribly bad market segmentation.”
Its.
There, I finally corrected Linus Torvalds in something. :))
Yeah I'm always annoyed with this kind of mistake. Especially as non-native speakers should know better than the native ones (which usually don't give a f.).
Now the point about internally doing ECC is an interesting one, could be a way out of this mess. And apparently ECC is more available in AMD land
The AMD parts all have the ECC feature but the platform support outside of EPYC may as well not exist. Most motherboards for the Ryzen segment don’t do it properly or don’t do it at all, some support it but aren’t capable of reporting events to the operating system which is dumb. Ryzen laptops don’t have it either.
Closest you can come to a nuc with ecc is I think a mini server equipped with one of the four-core i3 parts that have ecc.
Ah thanks for the real-world check on AMD Ryzen and ECC. Sad state of affairs really, when so many things are integrated in the SOC, why skimp on this...
As for NUCs, I thought there were some Atom chips with ECC but not in NUC form factor ? Something shiny from Logic Supply might do then...
As a non native speaker, my phone has both the Italian and English dictionary, when I write its it always auto corrects to it's as soon as I hit space and sometimes it gets unnoticed.
Yes it is. The problem is they dont really advertise it. I'm not certain but it might even be standard on AMD chips, but if they dont say so and board makers are also unclear, who knows...
For consumer motherboard OEMs, only AMD effectively has ECC support (Intel's has been so spotty and haphazard from product to product), and of AMD users, only a small number care about ECC.
So motherboard companies, being resource and time-starved as they are, don't make it a priority to address such a small user-base.
If Intel started shipping ECC on everything, it would go a long way towards shifting the market.
For a 2nd language speaker making these homophonic mistakes is actually a sign of fluency. It means that you just transcribe a mental flow of words instead of consciously constructing the language.
The first time I wrote "your" instead of "you're" in English I thought it was quite a milestone!
> For a 2nd language speaker making these homophonic mistakes is actually a sign of fluency.
I kinda disagree because while the homophony works in (spoken) English in written it stands as a sore thumb. So yeah you will make it if you only heard it but doesn't know the written form.
(And in their native language it's probably two unrelated words, so that might intensify the feeling of wrongness)
I mean, my native language is French where "your" is "ton" and "you're" is "tu es", yet it (rarely) happens that I mix them up in English. If I proofread I'll spot it almost every single time, but if I'm just typing my "stream of consciousness" my brain's speech-to-text module sometimes messes up.
meh, plenty of (intelligent!) native english speakers do not know all the canonical grammar rules. english contains a lot of what could be considered error correction bits, so it doesn't usually impede understanding. syntactically perfect english with weird/misused idioms (common among non-native speakers with lots of formal education) is harder to understand in my experience. I imagine this is true of most natural languages.
Yes. I noticed this. When I was younger, I thought how can you mix up 'their, they're, there' people you do this must be the opposite of smart. This lasted for 4 years living in an English speaking country....
As an “english as a second language” user, I can’t see myself writing e.g. “should of” instead of “should have”, however fluent I am. I think you don’t make that kind of typo unless you have learnt english before grammar.
I also wouldn't do this one, but that's because in my English accent I simply wouldn't pronounce them the same way. Also the word sequence "should of" is extremely uncommon in proper English, so it catches the eye more easily I think.
"You're/your", "their/they're", "its/it's" and the like are a different story, because I do pronounce those the same and they're all very common.
I've realized that when I'm engaged in the writing (angry or emotional in some way) I tend to commit more of these mistakes, even though I know the difference between "it's" and "its". Linus is always angry, so that probably makes him commit more orthographic mistakes.
He uses "do do" instead of "to do" which is a more obvious typo. Linus usually comes across as borderline arrogant, and deservedly so, but not necessarily perfect in his writing. I think it's an effective strategy to communicate his priorities and wrangle smart but easily intimidated folk "do do" what he believes is right!
Maybe he composed the message using a machine with non-ECC RAM and suffered a bit flip, which through some chain of events, led to the ' being added. Best to give him the benefit of doubt, I think!
It is. Swedish and English are both Germanic languages, so there are a lot of commonalities. Finnish is in a completely different language family. English and Swedish are more closely related to Persian and Hindi than to Finnish.
Yes. Swedish is also easy compared to English and French, the other two languages I've learned after early childhood. The only thing that makes it hard is that you never really have use for it and you're forced to learn it nevertheless here in Finland.
I'm happy to see people here on HN respect the difficulty of learning languages. Most foreigners that speak Finnish do it very poorly at first and even after decades they still sound like foreigners. But it shows huge respect to our small country for someone to make the effort, and we really appreciate it. I'm hoping other people see learning their own mother tongue the same way. Sure, most of us need English, but learning it well is still a huge task.
Linus must have English as his '1st' language now. For non-originally-native speaker mistakes like 'it's vs its', 'than vs then', etc. are pretty uncommon.
I guess this is what happens when someone first learns to speak the language, learning how to write in it only later on - as it often is the case with children.
I spent my preschool years in a multicultural environment and English was our lingua franca(ironically the school-mandated language was French), so I didn't properly learn contractions until grade school - same with similarly sounding words like "than vs then" and "your vs you're".
I've spent my whole life speaking multiple languages and this still trips me up every now and then, in fact quotes as such are a problem for me and I keep using them wrong, no idea why, it just won't register. So unless I slow down to 1/10th of my normal writing speed I will definitely make mistakes like that. Good we have proofreaders :)
It's because they have two different uses (three if you count nested quotes, but those aren't common and are pretty easy to figure out), contractions and possession, and they seemingly collide on words like "its" where you'd think it could mean either.
Not sure if you've already learned this (or if it helps), but English used to be declined, and its pronouns still are, e.g. they/their/them. That's why "its" isn't contracted; the possessive marker is already in the word.
In this particular case 'it's' can also be possessive although it's considered non-standard, so to be correct you can always treat it like a contraction of 'it is'.
I wish this was more of a cohesive argument. He says he thinks it’s important and points to row-hammer problems but doesn’t explain why. Probably because the audience it was written for already knows the arguments of why, but this is not the best argument.
If in doubt, get ECC. Do your own research on how it works and why. This post won’t explain it, just will blame Intel (probably rightfully so).
> We have decades of odd random kernel oopses that could never be explained and were likely due to bad memory. And if it causes a kernel oops, I can guarantee that there are several orders of magnitude more cases where it just caused a bit-flip that just never ended up being so critical.
It might be false, but I think it's a reasonable assumption.
To someone on HN who isn’t familiar with what ECC does that explains nothing about how ECC works and how it could have prevented these situations. Or how often they really happen.
The problem is that, if you don't have ECC to detect the errors, it's very hard to know what exactly caused a random, non-reproducible crash. Especially in kernel mode where there's little memory protection and basically any driver could be writing anywhere at any time.
I can understand Linus's frustration from that point of view: without ECC RAM when you get some super weird crash report where some pointer got corrupted for no apparent reason you can't be sure if it's was just a random bitflip or if it's actually hiding a bigger problem.
Pretty sure most memory test tools like memtest86 write the memory and then read it back shortly thereafter in relatively small blocks. This makes the window for errors to be introduced dramatically smaller. Most memory in a computer is not being continually rewritten under normal use.
If you manage to replicate bitflips every few days your RAM is broken.
It's the "once every other year" type of bitflip that's the problem. The proverbial "cosmic ray" hitting your DRAM and flipping a bit. That will be caught by ECC but it'll most likely remain a total mystery if it causes your non-ECC hardware to crash.
It isn't only cosmic rays. Regular old radiation can also cause it. I've read about a server that had many repeated problems and the techs replaced the entire motherboard at one point.
Then one of them brought in his personal Geiger counter and found the radiation coming off the steel in that rack case was significantly higher than background.
You may never know when the metal you use was recycled from something used to hold radioactive materials.
In the mean time I read here the rate to be around 1 bit flip per 1 GB per month. So in an 120 GB System that shd be about 1 flip every 6h. But then this number might be wrong.
Right. My point that TFA serves zero purpose to most people on here. Those that know how ECC works already know that it is a must have. Those that don't will learn very little from the post because it fails to explain what ECC is and why you need it aside from general statements about memory errors. It will reaffirm for those that know about what ECC RAM is that it's a good idea, but they already know it anyways. It reads a lot like an article about why vitamin C is a good thing.
> A large-scale study based on Google's very large number of servers was presented at the SIGMETRICS/Performance ’09 conference.[6] The actual error rate found was several orders of magnitude higher than the previous small-scale or laboratory studies, with between 25,000 (2.5 × 10−11 error/bit·h) and 70,000 (7.0 × 10−11 error/bit·h, or 1 bit error per gigabyte of RAM per 1.8 hours) errors per billion device hours per megabit. More than 8% of DIMM memory modules were affected by errors per year
I used to be pretty skeptical of ECC for consumer-grade hardware, mainly because I felt that I'd always prefer cheaper/more RAM over ECC RAM even if it meant that I'd get a couple of crash every year due to rogue bitflips. For servers it's a different story, but for a desktop I'm fine dealing with some instability for better performance.
But these days with the RAM density being so high and bitflipping attacks being more than a theoretical threat it seems like there's really no good reason not to switch to ECC everywhere.
Just Intel fucking over security by making ECC a non feature on consumer grade hardware - wouldn't be surprised if it was just a single bit flipped in a feature mask.
Well, with as common as a bunch of people in this thread seem to think bit flips are, it should just be a matter of time until that bit gets flipped on your cpu and activates the ecc feature.
That bit probably is either burned in or stored with the firmware in something more permanent than RAM. Modern RAM has the issue that it is optimized for capacity and speed to a point where state changes can leak into nearby bits.
- That bit is not stored in RAM but rather somewhere within CPU which can have different bit flip frequency characteristics.
- Even assuming this bit can be flipped, most computers wouldn't have ECC RAM inserted because well... the CPU doesn't support it.
- And even assuming you have ECC RAM inserted, this bit flipping while your computer is running will likely lead to CPU hang considering the CPU didn't fill in error correction bits and will lead CPU to think that pretty much every single byte is corrupt.
- And all in all, even if you did manage to turn on ECC bit, it would only work until the computer gets turned off.
I've looked into this before, personally a Xeon E 2288g is probably the most "consumer" cpu intel offers that has ECC, 8 cores and QuickSync support. Trying to get ECC + Quicksync is a real pain.
As others have replied, all the ASRock boards where I have ever checked the specifications do support ECC and also some ASUS boards support ECC, e.g. all ASUS workstation boards.
Because ECC means Error Correcting Code, by definition, any board that claims ECC support must actually correct the errors. The ECC codes used now, with 8 extra bits for each 64 data bits, correct any 1-bit error and detect any 2-bit errors.
Very old computers (25 years old, or more) used parity instead of ECC and they just detected any 1-bit error (and any errors with an odd number of flipped bits), without being able to correct the errors.
I was looking at getting the Xeon-based NUC recently and one of the reasons I decided against it was that ECC SO-DIMMs seem to be a really marginal product. If you want ECC, something that takes full-size DIMMs seems much easier to buy memory for.
For the price, it made more sense for me to buy an R630 and populate it with a few less expensive, higher capacity ECC RDIMMs. I don't really need ECC as a local feature, so this lets me run on the mobile I want.
Note that the price is mostly due to market segmentation, in your case most of it by the laptop vendor (of course some for Intel, but not that much compared to the laptop vendor)
Xeon with ECC are not that overpriced compared with similar Core without. Likewise, RAM sticks with ECC are cheap to produce (basically just one more chip to populate per side per module). Likewise soldered RAM would simply add maybe $10 or $20 of extra chips.
I simply care that my computer executes code perfectly. Let's settle on "one instance of unintended behaviour per hundred years" for that metric.
If it needs ECC memory to do that, then fit it with ECC memory. If there are other ways to achieve that (for example deeper dram cells to be more robust to cosmic rays) that's fine too.
Just meet the reliability spec - I don't care how.
Then you'll have to pay a huge primer for that privilege. I can assure you that your standard computer components are not rated for century-scale use.
That's why I've always been on the fence with this ECC thing. For servers it's vital because you need stability and security.
For desktops I think that for a long time it was fine without ECC. If I have to chose between having, say, 30% more RAM or avoid a potential crash once a year, I'll probably take the additional RAM.
The problem is that now these problem can be exploited by malicious code instead of just merely happening because of cosmic rays. That's the main argument in favour of ECC IMO, the rest is just a tradeoff to consider.
This. How many user documents have memory flip errors introduced that are never detected? Impossible to say, but it is not a small number given the world-wide use of DRAM. Most are in trivial and unimportant documents, but some aren't...
It can be a concern, that's true, but personally most of the stuff I edit end up checked into a git repository or something similar.
And I mean, we all spend all day editing test messages and comments and files on non-ECC hardware, yet bitflip-induced corruption is rare enough that I can't say that I've witnessed a single instance of it in my life, despite spending a good chunk of it looking at screens.
It's just not a problem that occurs in practice in my experience. If you're compiling the release build of a critical piece of software, you probably want ECC. If you're building the dev version of your webapp or writing an email to your boss, you'll probably survive without it.
Can make that statement with any certainty? My personal and family computers have crashed quite a few times, and have corrupted photoes and files, some of them are valuable (taxes, healthcare, etc. Personal computers have valuable data these days)
I couldn't tell, as a user, which if those corruptions and crashes were causes by bitflips. Could you?
> I can assure you that your standard computer components are not rated for century-scale use.
And that's probably not what GP asked for. There's a difference between guaranteeing an error rate of 1 error per century of use on average, and guaranteeing it over the course of an actual century. It might be okay to guarantee that error rate for only 5 years of uninterrupted use, and degrade after that. For instance:
Years 1- 5: 1 error per century.
Years 6-10: 3 errors per century.
Years 10-15: 10 errors per century.
Years 15-20: 20 errors per century.
Years 20-30: 1 error per *year*.
Years 30+ : the chip is broken.
Now, given how energy hungry and polluting the whole computer industry actually is, it might be a good idea to shoot for extreme durability and reliability anyway. Say, sustain 1 error per century, over the course of fifty years. It will be slower and more expensive, but at least it won't burn the planet as fast as our current electronics.
In "theory" it needs ECC because you must also protect the link between the CPU and the RAM. So with ECC fully in DRAM but no protection on the bus, you risk some errors during the transfer. However maybe this kind of errors are rare enough so that you would have less than one per century. It probably depends on the motherboard design and fabrication quality though, and the environment where it is used.
ECC memory is predominantly used in servers where failure absolutely must be identified and logged. The desktop market to a lesser extent due to lack of mission critical tasks being run from there.
There are situations though, where you’re working on a document and the documents “save” format is a memory dump. Corruption for things of that type (Adobe RAW for example) would remove data.
It might present itself as a 1pixel colour difference, but it could be more damaging (incorrect finances, in accounting software for example). Software trusts memory; but memory can lie.
Well maybe. Rather than having to trust memory completely, it would just be better to use a binary format where each bit is verifiable so then at least a single bit flip would be immediately obvious. For example, a bit flip in a TLS session causes the whole session to fail rather than a random page element to change.
Not really. Gray codes are designed so that if you're counting correctly, only one bit flips at a time. But if you flip the wrong bit by accident, you'll end up with a completely different number, no way to tell a problem happened.
Right, exactly. TCP protects us from data-corruption in network streams, and ECC protects us from data-corruption in RAM. I doubt any sort of software solution could practically compete against hardware ECC, even if it could be done it would presumably be disastrous for performance.
Those corner cases might occur rarely but are probably inconsequential given rate of occurrence versus rate of criticalness - it probably doesn't justify the markup for most. In a data center you're processing millions of transactions per minute so occurrence is much more impactful.
I would EASILY pay 12.5% more (that's the bit overhead) for memory that actually works.
If my data is fine being corrupted to save 12.5% on RAM costs, then why am I even bothering processing the data? Apparently it's worthless.
People today weigh the cost of maybe 16 vs 32GB on a mid-tier desktop. ~doubling the cost for twice the RAM. Yes, paying 12.5% more for ECC RAM is a no-brainer.
Why does it matter if it =HAD= checksum, the numbers would have been altered prior to save. It means you store one but you get two when read later. If the format calculates immediate checksums on blocks it'd detect memory corruption at best. The extreme downside is that such a part is untestable under normal conditions, hard to maintain, and it costs more than the ECC in development.
Perhaps consumer-grade software that needs guarantees of correctness should be using error correction in software. For example, database records for financial software, DNS, e-mail addresses, etc.
I beg this, every time this conversation comes up it’s the same answer “I don’t see a problem”.
It’s so easy to chalk these kind of errors to other issues, a little corruption here, a running program goes bezerk there- could be a buggy program or a little accidental memory overwrite. Reboot will fix it.
But I ran many thousands of physical machines, petabytes of RAM, I tracked memory flip errors and they were _common_; common even in: less dense memory, in thick metal enclosures surrounded by mesh. Where density and shielding impacts bitflips a lot.
My own experience tracking bitflips across my fleet led me to buy a Xeon laptop with ECC memory (precision 5520) and it has (anecdotally) been significantly more reliable than my desktop.
Yeah, it's real obnoxious of Intel to silo ECC support off into the Xeon line, isn't it? I switched to ECC memory in 2013 or 2014 with a Xeon E3 (fundamentally a Core i7 without the ECC support fused off) and of course a Xeon-supporting motherboard (with weird "server board" quirks: e.g., no on-board sound device).
I love that AMD doesn't intentionally break ECC on its consumer desktop platforms and upgraded to the Threadripper in 2017.
Yeah, that appears to be a BGA-packaged processor designed to be permanently soldered to the board of some embedded device, not something that you can install in your desktop at all. I'm not sure why Intel decided to brand their embedded processors with ECC as i3, though I suspect the reason this range exists at all is because companies were going with competitors like AMD instead due to their across-the-board ECC support.
I've considered using an AMD CPU instead of Intel's Xeon on the primary desktop computer, but even low-end Ryzen Threadripper CPUs have TDP of 180W, which is a bit higher than I'd like. And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those, so one won't be able to return/replace a CPU if it doesn't work with ECC memory, AIUI, making it risky. Though I don't know how common it is for ECC to not be handled properly in an otherwise functioning CPU; are there any statistics or estimates around?
They don’t. They just measure it differently than AMD. Intel measures at base clock, but AMD measures at sustained max clock IIRC. It’s definitely deceptive, but it’s not a lie as long as Intel tells you (which they do).
Intel's TDP numbers are at best an indicator of which product segment a chip falls into. They are wildly inaccurate and unreliable indicators of power draw under any circumstance. For example, here's a "58W" TDP Celeron that can't seem to get above 20W: https://twitter.com/IanCutress/status/1345656830907789312
And on the flip side, if you're building a desktop PC with a more high-end Intel processor, you will usually have to change a lot of motherboard firmware settings to get the behavior to resemble Intel's own recommendations that their TDP numbers are supposedly based on. Without those changes, lots of consumer retail motherboards default to having most or all of the power limits effectively disabled. So out of the box, a "65W" i7-10700 and a "125W" i7-10700K will both hit 190-200W when all 8 cores/16 threads are loaded.
If a metric can in practice be off by a factor of three in either direction, it's really quite useless and should not be quantified with a scientific unit like Watts.
There's been a lot of misinformation spread about what TDP means for modern CPUs. In Intel's case TDP is the steady state power consumption of the CPU in its default configuration while executing a long running workload. Long meaning more than a minute or two. The CPU implements this by keeping an exponentially weighted moving average (EWMA) of the CPU's power consumption. The CPU will modulate its frequency to keep this moving average at-or-below the TDP.
One consequence of using a moving average is that if the CPU has been idle for a long time then starts running a high power workload instantaneous power consumption can momentarily exceed the TDP while the average catches up. This is often misleadingly referred to as "turbo mode" by hardware review sites. It's not a mode, there's no state machine at work here, it's just a natural result of using a moving average. The use of EWMA is meant to model the heat capacity of the cooling solution. When the CPU has been idle for a while and the heatsink is cool, the CPU can afford to use more power while the heatsink warms up.
Another factor which confuses things is motherboard firmware disabling power limits without the user's knowledge. Motherboards marketed to enthusiasts often do this to make the boards look better in review benchmarks. This is where a lot of the "Intel is lying" comes from, but it's really the motherboard manufacturers being underhanded.
The situation on the AMD side is of course a bit different. AMD's power and frequency scaling is both more complex and much less documented than Intel's so it's hard to say exactly what the CPU is doing. What is known is that none of the actual power limits programmed into the CPU align with the TDP listed in the spec. In practice the steady state power consumption of AMD CPUs under load is typically about 1.35x the TDP.
Unlike Intel, firmware for AMD motherboards does not mess with the CPU's power limit settings unless the user does so explicitly. Presumably this is because AMD's CPU warranty is voided by changing those settings, while Intel's is not.
Intel and AMD absolutely have "turbo mode" and market such features using the term "turbo". It might just be a result of the weighted moving average, but the term "turbo" isn't something reviewers just made up out of nowhere.
Nah. Both brands pull more than TDP when boosting at max, AMD desktop processors will pull up to 30% above the specified TDP for an indefinite period of time (they call this number the "PPT" instead, but they need to go higher than TDP to hit full boost, and PPT is the number that governs that).
Intel mobile processors actually obey TDP better than AMD processors do - Tiger Lake has a hard limit, when you configure a 15W TDP then it really is 15W steady-state once boost expires, while AMD mobile products will pull up to 50% more than configured in steady-state operation. (the gap is larger than desktop)
"the brands measure it differently" is sort of theoretically true but not in the sense people think, and in practice it is not true.
On AMD it is literally just a number they pick that goes into the boost algorithm. Robert Hallock did some dumb handwavy shit about how it's measured with some delta-t above ambient with a reference cooler but the fact is that the chip itself basically determines how high it'll boost based on the number they configure, so that is a self-fulfilling prophecy, the delta-t above ambient is dependent on the number they configure the chip to run at.
In practice: what's the difference between a 3600 and a 3600X? One is configured with a TDP of 65W and one is configured with a TDP of 95W, the latter lets you boost higher and therefore it clocks higher. Configure them both to a 65W PPT limit and they will boost to pretty much the same place.
Intel nominally states that it's measured as a worst-case load at base clocks, something like Prime95 that absolutely nukes the processor (and even then many processors do not actually hit it). But really it is also just a number that they pick. The number has shifted over time, previously they used to undershoot a lot, now they tend to match the official TDP. It's not an actual measurement, it's just a "power category" that they classify the processors as, it's informed by real numbers but it's ultimately a human decision which tier they put them in.
So in practice, for both brands, it is just a number they pick. They have different theoretical methods for getting there but ultimately the marketing department looks at where the clocks would put them and pick a power number that they think represents that. It is not, in practice, a pure measurement from either brand, it is just a "category" they use.
Real-world you will always boost above base clocks on both brands at stock TDP, at least on real-world loads. You won't hit full boost on either brand without exceeding TDP, the "AMD measures at full boost" is categorically false despite the fact that it's commonly repeated. AMD PPT lets them boost above the official TDP for an unlimited period of time, they cannot run full boost when limited to official TDP.
> but even low-end Ryzen Threadripper CPUs have TDP of 180W, which is a bit higher than I'd like.
Why does it matter? It doesn't idle that high; it only goes that high of you're using it flat out, in which case the extra power usage is justified because it's giving that much more performance over a 100 W TDP CPU. Now I totally get it if you don't want to go Threadripper just for ECC because it's more expensive, but max power draw, which you don't even have to use? I've never seen anyone shop a desktop CPU by TDP, rather than by performance and price.
I prefer to pick PSU and fans (for both CPU and chassis) that can handle it comfortably (preferably while staying silent and with some reserve) with maximum TDP in mind, and given that I don't need that many cores or high clock speed either, a powerful CPU with high TDP is undesirable because it just makes picking other parts harder. I've mentioned TDP explicitly because I wouldn't mind if it was a (possibly even high-end) Threadripper that somehow didn't produce as much heat. Although price also matters, indeed.
>> I've never seen anyone shop a desktop CPU by TDP, rather than by performance and price.
Oh oh, me! Back in the day I bought a 65W CPU for a system that could handle a 90W. I wanted quiet and figured that would keep fan noise down at a modest performance penalty. It should also last longer, being the same design but running cooler. I ran that from 2005 until a few years ago (it still run fine but is in storage).
Planning to continue this strategy. I suspect it's common among SFF enthusiasts.
The Intel Nuc and Mac mini are good examples of this - however the Nuc doesn’t have its psu inside, it’s a brick. Great for fixing failures, horrible in general as a built in psu is so much tidier.
Hm... My 2013 NUC in fanless Akasa enclosure runs 24/7 on a 6W CPU, I recently looked at the options, and the 2019 6W offering changes little in performance. Yes, memory got faster, but that's it.
My passive-cooled desktop is also running a slightly trottled down 65W CPU.
So yes, there are people who choose there hardware by TDP.
When looking for a CPU for a server that sits in my living room, I went down the thought process of getting a low tdp. I don't have a quote, but I seem to remember coming to the conclusion that tdp is the max temp threshold, not the consistent power draw. If you have a computer idling I believe you won't see a difference in temp between cpus, but you will have the performance when you need it.
These days, a quiet, pwm fan with good thermal paste (and maybe some linux CPU throttling) more than achieves my needs for a "silent" pc 99% of the time.
I would love to be told my above assumptions are wrong if they are.
Yah-- one should look at performance within a given power envelope. Being able to dissipate more and then either end up with the fan running or the processor throttling back somewhat is good, IMO.
The worst bit is, AMD and Intel define TDP differently-- neither is the maximum power the processor can draw-- though Intel is far more optimistic.
On AMD, with Ryzen Master, you can set the TDP-envelope of the processor to what you want. Then the boost/frequency/voltage envelope it chooses to operate in under sustained load is different.
IMO, shopping by performance/watt makes sense. Shopping by TDP doesn't. (Especially since there is no comparing the AMD and Intel TDP numbers as they're defined differently; neither is the maximum the processor can draw, and Intel significantly exceeds the specified TDP on normal workloads).
TDP matters a fair bit in SFF(Small Form Factor) PCs. For instance the 3700x is a fantastic little CPU since it has a 65W TDP but pretty solid performance.
In a sandwich style case you're usually limited to low profile coolers like Noctua L9i/L9a since vertical height is pretty limited.
Performance/watt matters. You can just set TDP to what you want with throttling choices.
If you want a 45W TDP from the 3700X, you can just pop into Ryzen Master and ask for a 45W TDP. Boom, you're running in that envelope.
I think shopping based on TDP is not the best, because it's not comparable between manufacturers and because it's something you can effectively "choose".
How do you do that? Is it a setting in the bios? Or can it be done runtime? If so, how? It sounds interesting if I can run a beefy rig as a power efficient device, for always-on scenarios, and then boost it when I need.
> How do you do that? Is it a setting in the bios? Or can it be done runtime?
On AMD, it's a utility you run. I believe you may require a reboot to apply it. On some Intel platforms, it's been settings in the BIOS.
> It sounds interesting if I can run a beefy rig as a power efficient device, for always-on scenarios, and then boost it when I need.
This is what the processor is doing internally anyways. It throttles voltage and frequency and gates cores based on demanded usage. Changing the TDP doesn't change the performance under a light-to-moderate workload scenario at all.
Ryzen Master lets you change some of the tuning for the choices it makes about when and how aggressively to boost, though, too.
Ryzen Master doesnt seem to be available for linux so you end up with bunch of unnofficial hacks that may or may not work. I run sff setup myself, originally wanted to get 3600 but it was out of stock, and the next tdp friendly processor was 3700x.
That's an annoyance, but on Linux you have infinite more control of thermal throttling and you can get whatever thermal behavior you want. Thermald has been really good on Intel, and now that Google contributed RAPL support you can get the same benefits on AMD-- pick exactly your power envelope and thermal limits.
> Yeah but can I get a metric ton of benchmarks at that 45w setpoint?
Yup, they're out there.
> I don't really see the reason in paying for a 100w TDP premium if I'm just going to scale it down to 65w.
You might want the core count or peak performance for the very short term. When I was looking, running 65W parts in the 45W envelope was only about a 7% penalty, so you get a bunch more performance/watt.
Back when my daily driver was a Core 2 laptop, someone told me that capping the clock frequency would make it unusable.
As a petty "Take that", I dropped the max frequency from 2.0 GHz to 1.0 GHz. I ran a couple benchmarks to prove the cap was working, and then just kept it at 1.0 for a few months, to prove my point.
It made a bigger difference on my ARM SBC, where I tried capping the 1,000 MHz chip to 200 or 400 MHz. That chip was already CPU-bound for many tasks and could barely even run Firefox. Amdahl's Law kicked in - Halving the frequency made _everything_ twice as slow, because almost everything was waiting on the CPU.
The funny thing is, on modern processors-- throttling TDP only affects when running flat out all-core workloads. A subset of cores can still boost aggressively, and you can run all-core max-boost for short intervals.
And the relationship between power and performance isn't linear as processor voltages climb trying to squeeze out the last bit of performance.
So if you want to take a 105W CPU and ask it to operate in a 65W envelope, you're not giving up even 1/3rd of peak performance, and much less than that of typical performance.
AMD Ryzen 9 5950X: 20.6W for a single core at 5050MHz, 49W for the whole package. (And it’s generally the package figure that you care about.)
AMD Ryzen 9 5900X: 17.9W/54W at 4875MHz.
AMD Ryzen 7 5800X: 17.3W/37W at 4825MHz.
AMD Ryzen 5 5600X: 11.8W/28W at 4650MHz (though the highest core reading is 13W, at three cores loaded).
You’re both correct: by simply restricting that power envelope by 40%, you shed a lot less multi-threaded performance than people realise, and no single-threaded performance.
Look at the 5950X figures, and you observe that at about 120W, it can run 6 cores at 4,650MHz (27,900 core–MHz), or 16 cores at 3,775MHz (60,400 core–MHz).
Expressed one way: by dropping the frequency by 20%, power per watt increased by around 2.7×.
Expressed another way: let’s skip a 65W envelope—put this particular 105W chip in a 40W envelope and you lose only 20% of your six-cores performance. Seriously. But I’m not sure what the curve would look like if you load all 16 cores at a 40W envelope, what speed they’d be going at.
(But do remember that “TDP” is a bit of a mess as a concept, and that we’re depending on non-core power consumption being generally fairly consistent regardless of load.)
> I've never seen anyone shop a desktop CPU by TDP, rather than by performance and price.
That's me. When I start to plan for a new system, I select the processor first and read its thermal design guidelines (Intel used to have nice load vs. max temp graphs in their docs) and select every component around it for sustained max load.
This results in a more silent system for idle and peace of mind for loading it for extended duration.
If my only interest would be ECC, I might do that but, I develop scientific software for research purposes. I need every bit of performance from my system.
In my case loading means maxing out all cores and extended period of time can be anything from five minutes to hours.
The problem is-- you can't compare the TDP nor even the system cooling design guidelines between AMD and Intel.
Both are optimistic lies, but-- if you look at the documents it looks like currently AMD needs more cooling, but actually dissipates less power in most cases and definitely has higher performance/watt.
> The problem is-- you can't compare the TDP nor even the system cooling design guidelines between AMD and Intel.
Doesn't matter for me since I'm not interested in comparing them.
> Both are optimistic lies, but-- if you look at the documents it looks like currently AMD needs more cooling, but actually dissipates less power in most cases and definitely has higher performance/watt.
I'm aware of the situation, and I always inflate the numbers 10-15% to increase headroom in my systems. The code I'm running is not a most case code. A FPU heavy, "I will abuse all your cores and memory bandwidth" type, heavily optimized scientific software. I can sometimes hear that my system is swearing at me for repeatedly running for tests.
I don't like to add this paragraph but, I'm one of the administrators of one of the biggest HPC clusters in my country. I know how a system can surpass its TDP or how can CPU manufacturers skew this TDP numbers to fit in envelopes. We make these servers blow flames from their exhausts.
Built a NAS. My #1 concern for choosing CPU was TDP. This machine is on 24/7 and power use is a primary concern where I live because electricity is NOT cheap.
This is a poor way to make the choice. TDP is supposed to specify the highest power you can get the processor to dissipate, not typical or idle use. And since different manufacturers specify TDP differently, you can't even compare the number.
Performance/watt metrics and idle consumption would have been a far better way to make this choice.
If you have a choice between A) something that can dissipate 65W peak for 100 units of performance, but would dissipate 4W average under your workload, and B) something that can dissipate 45W peak for 60 units of performance, but would dissipate 4.5W under your workload... I'm not sure why you'd ever pick B.
Is there a metric to look for to understand what power consumption is at "idle" or something close to that? That is what confuses me. I don't want to spend a lot of money on something that will be always on, and usually idling, and finding that its power usage is way higher than I thought. But perhaps there is a metric that tells that. I have not looked closely at it.
Also, even though the CPU may draw less, can still the power supply waste more, just because it is beefy? Comparing with a sports car, they have great performance, but also use more gas in ordinary traffic? Can a computer be compared with that?
> Is there a metric to look for to understand what power consumption is at "idle" or something close to that? That is what confuses me. I don't want to spend a lot of money on something that will be always on, and usually idling, and finding that its power usage is way higher than I thought.
Community benchmarks, from Tom's Hardware, etc.
The vendor numbers are make believe-- you can't use them for power supply sizing or for thermal path sizing. If you look at the cited TDP numbers today-- it can be misleading-- e.g. often Intel 45W TDP parts use more power at peak than AMD 65W parts.
On modern systems, almost none of the idle consumption is the processor. The power supply's idle use and motherboard functions dominate.
> Also, even though the CPU may draw less, can still the power supply waste more, just because it is beefy?
Yes, having to select a larger power supply can result in more idle consumption, though this is more of a problem on the very low end.
If the concern is about heat and noise, the cooling system is the most important factor, oddly I don't see it even mentioned.
Get a huge cooler like Noctua d14, and you pc becomes silent. It lasts forever, requires no maintenance, a good investment.
If you are adventurous, watercooling is even better, but its a can of worms I decided I'd rather live without - possibility of leaks and cost make it harder to justify
Regarding verification. There is a debian package called edac-utils. As I recall you overclock your RAM and run your system at load in order to generate failures.
Looking back at my notes, the output of journalctl -b tells should say something like, "Node 0: DRAM ECC enabled."
Then 'edac-ctl --status' should tell you that drivers are loaded.
Then you run 'edac-util -v' to report on what it has seen,
mc0: 0 Uncorrected Errors with no DIMM info
mc0: 0 Corrected Errors with no DIMM info
mc0: csrow2: 0 Uncorrected Errors
mc0: csrow2: mc#0csrow#2channel#0: 0 Corrected Errors
mc0: csrow2: mc#0csrow#2channel#1: 0 Corrected Errors
mc0: csrow3: 0 Uncorrected Errors
mc0: csrow3: mc#0csrow#3channel#0: 0 Corrected Errors
mc0: csrow3: mc#0csrow#3channel#1: 0 Corrected Errors
edac-util: No errors to report.
All AMD CPUs with integrated memory controllers support ECC. The CPU also exposes an interface usable by the operating system to verify ECC works - the same interface is used to provide monitoring of memory fault data provided by ECC.
They aren't tested on it, so it's possible to get a dud, but it's minuscule chance that isn't worth bothering.
Now, to actual issues you can encounter: motherboards
The problem is that ECC means you need to have, iirc, 8 more data lines between CPU and memory module, which of course mean more physical connections (don't remember how many right now). Those also need to be properly done and tested, and you might encounter a motherboard where it wasn't done. Not sure how common, unfortunately.
Another issue is motherboard firmware. Even though AMD supplies the memory init code, the configuration can be tweaked by motherboard vendor, and they might simply break ECC support accidentally (even by something as simple as making a toggle default to false then forgot to expose it in configuration menu).
Those are the two issues you can encounter.
The difference with AFAIK Threadripper PRO, and EPYC, is that AMD includes ECC in its test and certification programs for it, which kind of enforces support.
> Another issue is motherboard firmware. Even though AMD supplies the memory init code, the configuration can be tweaked by motherboard vendor, and they might simply break ECC support accidentally (even by something as simple as making a toggle default to false then forgot to expose it in configuration menu).
I think some Gigabyte boards are infamous for this in certain circle
OTOH: Gigabyte might have a Threadripper PRO motherboard (WRX80 chipset) coming out in the future
Gigabyte is also infamous for trying to claim that they implemented UEFI by dropping a build of DUET (UEFI that boots on top of BIOS, used for early development) into BIOS image...
On Windows, to check if ECC is working, run the command 'wmic memphysical get memoryerrorcorrection':
PC C:\> wmic memphysical get memoryerrorcorrection
MemoryErrorCorrection
6
SuperUser has a convenient decoder[1], but modern systems will report "6" here if ECC is working.
When Windows detects a memory error, it will record it in the system event log, under the WHEA source. As a side note, this is also how memory errors within the CPU's caches are reported under Windows.
All desktop Ryzen CPUs without integrated GPU, i.e. with the exception of APUs, support ECC.
You must check the specifications of the motherboard to see if ECC memory is supported.
As a rule, all ASRock MBs support ECC and also some ASUS MBs support ECC, e.g. all ASUS workstation motherboards.
I have no experience with Windows and Ryzen, but I assume that ECC should work also there.
With Linux, you must use a kernel with all the relevant EDAC options enabled, including CONFIG_EDAC_AMD64.
For the new Zen 3 CPUs, i.e. Ryzen 5xxx, you must use a kernel 5.10 or later, for ECC support.
On Linux, there are various programs, e.g. edac-utils, to monitor the ECC errors.
To be more certain that the ECC error reporting really works, the easiest way is to change the BIOS settings to overclock the memory, until memory errors appear.
I didn't consider embedded CPUs (I guess that's about an embedded EPYC, not a server one), those look neat. But there's no official ECC support (i.e., it's similar to Ryzen CPUs), is there?
Edit: as detaro mentioned in the reply, there is, and here's the source [0] -- that's what they mean by "RAS" on promotional pages [1]. That indeed looks like a nice option.
RAS covers more than just DRAM, but yes. Historically, the reporting interface is called MCA (Machine Check Architecture) / MCE. I think both AMD and Intel have extensions with other names, but MCA/MCE points you in the right direction.
what kind of machine is that? Been vaguely looking for one a while back, and everything seemed difficult to get (since the main target is large-volume customers I guess)
Sorry, I didn't see this earlier. It's Supermicro board with an Epyc 3201 (8c/8t), base frequency is ~1.5 Ghz. There's no CPU fan, so it can get quite hot. It throttles at 94 degrees celcius, and it will hit that if more than around 4-6 cores are being fully used. However there are options in the bios to disable some of the cores. In my case I've disabled 4 of the cores and the temperature is much more reasonable now.
I went through this about a year ago, to build a low-TDP ECC workstation. I do not have stats on failure rates, just this anecdotal experience. Asrock and Asus seem to be the boards to get. For RAM, I got two sticks of Samsung M391A4G43MB1, and verified. The advice I remember from the forums was to stick to unbuffered ram (UDIMMS).
The TDP on EPYC chips is a lot higher. I think of Threadripper as mid-tier, and EPYC as the high-end. Ryzen is remarkable because you can buy new equipment with ECC, at consumer prices. I am hazy, but don't think that has been possible since the 386 era ('parity ram').
EPYC TDP ranges from "a lot lower" (35W embedded, 120W regular) up to "comparable with" (180-240W, a single 280W model) relative to Threadripper (180-250W last gen; current gen is all 280W). It's definitely not a lot higher on the Epyc side.
> I think of Threadripper as mid-tier, and EPYC as the high-end.
This oversimplifies to the point of not being a useful intuition (or is arguably even incorrect). Threadripper is a SKU with a moderate number of cores at relatively high clocks; (high-power) EPYC SKUs have a lot of very efficient cores running at lower clocks. They both have a niche, but Threadripper has unambiguously better single-core performance due to the ~20% higher clocks. And single-core IPC still matters in many applications (to oversimplify: Amdahl's law; but also, latency-sensitive applications).
I don't understand. Whatever the TDP of Intel processors, you are straight up getting less bang for watt given their ancient process. Same reason smartphones burst to high clocks and power; getting the task done faster is on average much more efficient.
> I've considered using an AMD CPU instead of Intel's Xeon on the primary desktop computer, but even low-end Ryzen Threadripper CPUs have TDP of 180W, which is a bit higher than I'd like.
Any apples-to-apples comparable Intel CPU will have comparable power use. The difficulty is that Intel didn't really have anything like Threadripper — their i9 series was the most comparable (high clocks and moderate core counts), but i9 explicitly did not support ECC memory, nullifying the comparison.
You're looking at 2950X, probably? That's a Zen+ (previous gen) model. 16 core / 32 thread, 3.5 GHz base clock, launched August 2018.
Comparable Intel Xeon timeline is Coffee Lake at the latest, Kaby lake before that. As far as I can tell, no Kaby Lake nor Coffee Lake Xeons even have 16 cores.
The closest Skylake I've found is an (OEM) Xeon Gold 6149: 16/32 core/thread, 3.1 GHz base clock, 205W nominal TDP (and it's a special OEM part, not available for you). The closest buyable part is probably Xeon Gold 6154 with 18/36 core/threads, 3GHz clock, and 200W nominal TDP.
Looking at i9 from around that time, you had Skylake-X and a single Coffe Lake-S (i9-9900K). 9900K only has 8 cores. The Skylake i9-9960X part has 16/32 cores/threads, base clock of 3.1GHz, and a nominal TDP of 165W. That's somewhat comparable to the AMD 2950X, ignoring ECC support.
Another note that might interest you: you could run the Threadripper part at substantially lower power by sacrificing a small amount of performance, if thermals are the most important factor and you are unwilling to trust Ryzen ECC: http://apollo.backplane.com/DFlyMisc/threadripper.txt
Or just buy an Epyc, if you want a low-TDP ECC-definitely-supported part: EPYC 7302P has 16/32 cores, 3GHz base clock, and 155W nominal TDP. EPYC 7282 has 16/32 cores, 2.8 GHz base, and 120W nominal TDP. These are all zen2 (vs 2950X's zen+) and will outperform zen+ on a clock-for-clock basis.
> And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those, so one won't be able to return/replace a CPU if it doesn't work with ECC memory, AIUI, making it risky.
If your vendor won't accept defective CPU returns, buy somewhere else.
> Though I don't know how common it is for ECC to not be handled properly in an otherwise functioning CPU; are there any statistics or estimates around?
ECC support requires motherboard support; that's the main thing to be aware of shopping for Ryzen ECC setups. If the board doesn't have the traces, there's nothing the CPU can do.
> And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those
ECC isn't validated by AMD for AM4 Ryzen models, but it's present and supported if the motherboard also supports it. Many motherboards have ECC support (the manual will say for sure), and a handful of models even explicitly advertise it as a feature.
I have a Ryzen 9 3900X on an ASRock B450M Pro4 and 64 GB of ECC DRAM, and ECC functionality is active and working.
More like "The feature is present in silicon but motherboard makers are not required to turn it on". At the end of the day, ECC support does require extra copper traces in the PCB and some low end models may deliberately choose to skip them, thus the expectation has to be managed.
IMO, "validated" is intentionally wishy-washy and mostly means that AMD would prefer it if enterprises paid them more money by buying EPYC (or Ryzen Pro) parts instead of consumer Ryzen parts. Much like how Intel prefers selling higher-margin Xeons over Core i5. It's market segmentation, but friendlier to consumers than Intel's approach.
My (largely unfounded) understanding is that this means they don't don't run the consumer chip configurations though a battery of compatibility tests with with various memory modules on a reference motherboard. My understanding is that they run these steps for each stepping or significant process change for their.
Secondly, it probably also means that they do not include tests for this functionality when they perform the final tests against each fully assembled chip. I'd expect that a jtag boundary scan does verify that the bond wires are in place and work, but no functional tests of ECC are run on each processor in the consumer configuration.
The net result is that with a compatible motherboard and memory, ECC almost certainly works (since the memory controller is the same as in the supported model) but AMD does not officially guarantee it. It is much like overclocking. The functionality is present, and it should work, and most likely does, but AMD accepts no responsibility if it does not, since they don't formally test for it.
> one won't be able to return/replace a CPU if it doesn't work with ECC memory
I don't know where you live, but around here, (if you buy new?), the vendor MUST take back items up to 15 days after they were delivered, for ANY reason.
So, as long as you synchronize your buying of CPU, RAM, (motherboard), you should be fine.
You can use BIOS settings to change the TDP to whatever you like, with substantially higher efficiency if you are under-powering and substantially lower efficiency if you are over-powering.
While it's true that Intel only has ECC support on Xeon (and several other chips targeted at the embedded market) it's not true that ECC is supported well on AMD.
We only use Xeons on developer desktops and production machines here precisely because of ECC. It's about 1 bit flip/month/gigabyte. That's too much risk when doing something critical for a client.
ECC is supported on most Ryzen models[1], as long as the motherboard supports it. In fact, ASUS and ASRock (possibly others) have Ryzen motherboards designed for workstation/server use where ECC support is specifically advertised.
[1] The only exception is the Ryzen CPUs with integrated graphics.
Depends what you mean by supported. Semi-offically:
ECC is not disabled. It works, but not validated for our consumer client platform.
Validated means run it through server/workstation grade testing. For the first Ryzen processors, focused on the prosumer / gaming market, this feature is enabled and working but not validated by AMD. You should not have issues creating a whitebox homelab or NAS with ECC memory enabled.
AMD may claim not to validate ECC on Ryzen, but it's working well enough for major motherboard vendors to market Ryzen motherboards with ECC advertised as a feature.
ECC support not being "validated," for all practical purposes, simply means that board vendors can advertise a board lacking ECC support as compatible with AMD's AM4 platform, without getting a nasty letter from AMD's lawyers.
Yes there is a risk to buy a Ryzen CPU with non-functional ECC.
However, I use only computers with ECC, previously only Xeons, but in the last years I have replaced many of them with Ryzens, all of which work OK with ECC memory.
When having to choose between a very small risk of losing the price of a CPU and having to use for sure, during many years, an Intel CPU with half of the AMD speed, the choice was very obvious for me.
Your quote is for consumer platforms (Ryzen) only; GP's statement was that ECC is not well-supported on AMD at all, which is obviously false (EPYC, Threadripper).
It's an unsupported configuration and it's not tested.
The latter is a big problem, one of the extreme-OC guys (Buildzoid) who interacts frequently with the OEMs (as he is pushing their stuff to the limit and he frequently needs their help) has commented that AMD has a really bad problem with their BIOS teams. The AGESA firmware (the low-level code that the processor actually runs) is buggy as all hell at a firmware level and the OEMs are forced to patch around it in BIOS, but the AGESA firmware also has a massive problem with code churn, so these BIOS fixups basically stop working all the time. And the driver teams at a lot of OEMs are literally one person, so there isn't enough staffing there to test everything all the time. Long and short of it is: stuff breaks in AMD BIOSs, constantly, and they don't notice it.
This is obviously a huge problem when ECC is not an officially supported feature, because it means nobody is testing it! You might update your BIOS (as you frequently have to do with AMD machines) and suddenly ECC stops working, it might be running ECC in non-ECC mode and no longer correcting errors. Or it might have screwed up reporting them to the OS.
The server/workstation boards are the only ones you should be trusting Ryzen with ECC usage on.
Yeah, the iMac Pro has the Xeon W and ECC. T'would be nice if the Apple Silicon MacBook Pro had it. There's not much of a reason to pay for the Pro over the Air. But like Linus, I'm going to blame Intel for this situation in the market. Maybe Apple will strike out on its own with Apple Silicon but since their dominant use case is phones, I'll not hold my breath.
Unless something weird happens, the next generation of the Apple M-line will use LPDDR5 memory instead of the LPDDR4X used in the Apple M1. While it probably won't support error correction monitoring, LPDDR5 has built in error correction that silently corrects single bit flips. That alone should be a huge reliability improvement.
LPDDR5 will enable some much needed level of error correction in a metric ton of other future SoC designs too. I look forward to the future Raspberry Pi with built in error correction capabilities.
Yes, this applies to both DDR5 and LPDDR5. Leaks indicate that DDR5 CPUs and motherboards by Intel and AMD are not going to be out this year though, at least not on the desktop.
There is no guarantee of state at the quantum level ... just a high-degree of assurance in a state. After 40 years in the electronics, optics, software business, I've learned that there is absolutely the possibility for unexplained "blips".
Newer linux have replaced mcelog with edac-util. I think most shops operating systems at that scale are getting their ECC errors out of band with IPMI SEL, though.
When you say bitflips were "common" on thousands of physical machines, does that mean you observed thousands of bitflips?
Otherwise, I would think that an unlikely event becoming 1000x more likely by sheer numbers would have warped your perception.
I believe that hardware reliability is mostly irrelevant, because software reliability is already far worse. It doesn't matter whether a bitflip (unlikely) or some bug (likely) causes a node to spuriously fail, what matters is that this failure is handled gracefully.
Another comment[1] mentioned 1 bitflip per gigabyte per month. If you have a lot of RAM, that's rather a lot.
> It doesn't matter whether a bitflip (unlikely) or some bug (likely) causes a node to spuriously fail
Except that a bitflip can go undetected. It may crash your software or system, but it also may simply leak errors into your data, which can be far more catastrophic.
The point is that you can't prevent failure by just buying something. You have to deal with the fact that failure can not be prevented.
In other words, if a single defective DIMM somewhere in your deployment is causing catastraphic failure, your mistake was not buying the wrong RAM modules. Your mistake was relying on a single point of failure for mission critical data.
People might not be preventing failure by using ECC, but they significantly decrease the likelihood of having to deal with hard-to-debug problems caused by bitflips.
I have a server with 384GB of RAM sitting next to me, and in two months of uptime with the only shielding being a pile of IBM iron on top of it, it has detected a grand total of 0 errors.
This is, of course, an anecdote rather than data, but 0 is different enough from the expected 768 that it makes me doubt that statistic.
It depends where the failure happens. Sometimes you really lose the “failure in the wrong place” lottery. For example, in a domain name: http://dinaburg.org/bitsquatting.html
Ya, I'm not buying that biyflips are a problem. Or maybe modern software can correct better for this? Because I use my desktop all day everyday running tons of software on 64 gb of ram and I don't get errors or crashes often enough to remember ever having one.
Nobody denies that bitflips happen. On the whole, you fail to make a case that preventing bitflips is the solution to a problem. Bitsquatting is not a real problem, it's a curiosity.
As for the case of bitflips killing someone: Bitflips are not the root cause here. The root cause is that somebody engineered something life-critical that mistakenly assumed hardware can not fail. Bitflips are just one of many reasons for hardware failure.
So those systems didn't fail when a bitflip happened?
> The root cause is that somebody engineered something life-critical that mistakenly assumed hardware can not fail.
The systems I am aware of were designed with bitflips in mind. NO software can handle arbitrary amounts of bitflips. ALL software designed to mitigate bitflips only lower the odds via various forms of redundancy. (For context, I've written code for NASA, written a few proposals on making things more radiation hardened, and my PhD thesis was on a new class of error correcting codes - so I do know a little about making redundant software and hardware specifically designed to mitigate bitflips).
By claiming a bitflip didn't kick off the problems, and trying to push the cause elsewhere, you may as well blame all of engineering for making a device that can kill on failure.
So your argument is a red herring
>On the whole, you fail to make a case that preventing bitflips is the solution to a problem
Yes, had those bitflips been prevented, or not happened, those fatalities would not have happened.
>Ya, I'm not buying that biyflips are a problem.
If bitflips are not a problem then we don't need ECC ram (or ECC almost anything!) which is clearly used a lot. So bitflips are enough of a problem that a massively widespread technology is in place to handle precisely that problem.
I guess you've never written a program and watched bits flip on computers you control? You should try it - it's a good exercise to see how often it does happen.
I guess you define something being a problem differently than I or the ECC ram industry do.
> So those systems didn't fail when a bitflip happened?
I didn't say that. I'm saying that the root cause (as in "root cause analysis") is not the bitflip. Designating the bitflip as the root cause is like analyzing your drunk driving accident and concluding that the root cause must be ethanol, rather than your drinking habits.
> The systems I am aware of were designed with bitflips in mind. NO software can handle arbitrary amounts of bitflips. ALL software designed to mitigate bitflips only lower the odds via various forms of redundancy.
Of course, and I'm not actually arguing that adding in ECC is completely worthless to that effect, though it is close to worthless. Luckily, ECC is quite cheap, if not free, so throwing it in there makes sense.
However, suppose ECC would increase the cost by several magnitudes, would it still be worth it? Obviously not. Redundancy alone reduces the probability of spurious failure by several magnitudes, and simply increasing redundancy would be far cheaper than adding in ECC.
> If bitflips are not a problem then we don't need ECC ram (or ECC almost anything!) which is clearly used a lot. So bitflips are enough of a problem that a massively widespread technology is in place to handle precisely that problem.
My point is that bitflips either don't really matter, in case data integrity is not mission critical, or they don't actually solve the problem, in case data integrity is mission critical.
If you have solved the problem of data integrity through redundancy, then ECC doesn't make much of a difference anymore. If you haven't solved the problem, then ECC will only prevent a vanishingly small subset of disasters that are awaiting you.
> I guess you've never written a program and watched bits flip on computers you control? You should try it - it's a good exercise to see how often it does happen.
I don't care how often it happens. I care about the odds of a bitflip causing an actual problem. If a computer crashes, that's okay, it'll reboot. If any data were to be corrupted, it would most likely happen at the disk level and not the DRAM level.
> I guess you define something being a problem differently than I or the ECC ram industry do.
Of course, somebody who sells ECC RAM will want to convince you that ECC actually solves a real problem. The same can be said about the nutritional supplement industry, or many other industries that rely on make-belief.
> If you have solved the problem of data integrity...
As above, this is not a binary, black and white thing, but you keep presenting it as such. It's probabilistic, and higher protection is not free - the tradeoff is engineering.
> Redundancy alone reduces the probability of spurious failure by several magnitudes
ECC "alone reduces the probability of spurious failure by several magnitudes". That's why it is used.
Naive redundancy ignores almost a century of better method form forward error correcting codes. I have a feeling your idea of redundancy is having multiple exact copies of a system or data and having them vote, which is a terribly expensive way to do data protection when there are vastly better methods.
>Of course, somebody who sells ECC RAM will want to convince you that ECC actually solves a real problem. The same can be said about the nutritional supplement industry, or many other industries that rely on make-belief.
And we're done. If you don't think ECC helps a real problem then I see why you don't understand bitflip causing problems. Good luck.
> As above, this is not a binary, black and white thing, but you keep presenting it as such. It's probabilistic, and higher protection is not free - the tradeoff is engineering.
The actual problem is binary. You either solved it, or you didn't. ECC is "free", but it doesn't actually solve the problem. Actually solving the problem requires engineering.
Of course there's a probabilistic element to it, but the problem is to drive the probability of failure to "vanishingly small". The utility of adding or removing a vanishingly small constant to another vanishingly small constant is vanishingly small. This is what ECC does for you.
> ECC "alone reduces the probability of spurious failure by several magnitudes". That's why it is used.
ECC reduces the probability of spurious failure due to bitflips in DRAM by several magnitudes. However, spurious failure can occur for so many more reasons that the bitflip issue becomes a vanishingly small part.
> I have a feeling your idea of redundancy is having multiple exact copies of a system or data and having them vote, which is a terribly expensive way to do data protection when there are vastly better methods.
As you know, having worked for NASA, this is the right choice under certain circumstances. If there are lives on the line and you have a choice between "not solving a problem" and "a terribly expensive solution", you should go with the latter.
> If you don't think ECC helps a real problem then I see why you don't understand bitflip causing problems.
ECC does not solve the problem of data integrity. If you actually solve the problem of data integrity, you will find that ECC becomes effectively redundant. Do we not fundamentally agree on this? If so, why not?
That's not to say ECC is entirely useless from an administrative standpoint. It makes DRAM bitflips one less thing to worry about. One less thing out of thousands of things. Commensurately, the cost of ECC in a given deployment, like its utility, is vanishingly small.
On a single computer with a large memory, e.g. 32 GB or more, the time between errors can be of a few months, if you are lucky to have good modules. Moreover, some of the errors will have no effect, if they happened to affect free memory.
Nevertheless, anyone who uses the computer for anything else besides games or movie watching, will greatly benefit from having ECC memory, because that is the only way to learn when the memory modules become defective.
Modern memories have a shorter lifetime than old memories and very frequently they begin to have bit errors from time to time long before breaking down completely.
Without ECC, you will become aware that a memory module is defective only when the computer crashes or no longer boots and severe data corruption in your files could have happened some months before that.
For myself, this was the most obvious reason why ECC was useful, because I was able in several cases to replace memory modules that began to have frequent correctable errors, after many years with little or no errors, without losing any precious data and without downtime.
The good modules bit is important. I'm told by some colleagues that most of the bit flips are from alpha particles from the ram casings surprisingly enough.
Plus, ECC RAM is so accessible these days thanks to Ryzen. All the asrock mobos, from the lowest end to the highest end, advertise official support for it.
The libraries we maintain (1) are responsible for a non-trivial part of Facebook's overall compute footprint, (2) should basically never fail of their own accord, and (3) have pretty good error monitoring. So my team is operating what is effectively (among other things) a very sensitive detector for hardware failure.
And indeed we see examples all the time of blobs that fail to decompress, and usually when we dig in we find that the blob is only a single bit-flip away from a blob that decompresses successfully into a syntactically correct message. I can't share numbers, but, off the top of my head, I think it's the largest source of failures we see. It happens frequently enough that I wrote a tool to automate checking [0].
So yes. It happens. Pretty frequently, in the sense that if you're doing xillions of operations a day, a one-in-a-xillion failure happens all the time.
Were you around for enough DRAM generations to notice an effect of DRAM density / cell-size on reported ECC error rate?
I’ve always believed that, ECC aside, DRAM made intentionally with big cells would be less prone to spurious bit-flips (and that this is one of the things NASA means when they talk about “radiation hardening” a computer: sourcing memory with ungodly-large DRAM cells, willingly trading off lower memory capacity for higher per-cell level-shift activation-energy.)
If that’s true, then that would mean that the per-cell error rate would have actually been increasing over the years, as DRAM cell-size decreased, in the same way cell-size decrease and voltage-level tightening have increased error rate for flash memory. Combined with the fact that we just have N times more memory now, you’d think we’d be seeing a quadratic increase in faults compared to 40 years ago. But do we? It doesn’t seem like it.
I’ve also heard a counter-effect proposed, though: maybe there really are far more “raw” bit-flips going on — but far less of main memory is now in the causal chain for corrupting a workload than it used to be. In the 80s, on an 8-bit micro, POKEing any random address might wreck a program, since there’s only 64k addresses to POKE and most of the writable ones are in use for something critical. Today, most RAM is some sort of cache or buffer that’s going to be used once to produce some ephemeral IO effect (e.g. the compressed data for a video frame, that might decompress incorrectly, but only cause 16ms of glitchiness before the next frame comes along to paper over it); or, if it’s functional data, it’s part of a fault-tolerant component (e.g. a TCP packet, that’s going to checksum-fail when passed to the Ethernet controller and so not even be sent, causing the client to need to retry the request; or, even if accidentally checksums correctly, the server will choke on the malformed request, send an error... and the client will need to retry the request. One generic retry-on-exception handler around your net request, and you get memory fault-tolerance for free!)
If both effects are real, this would imply that regular PCs without ECC should still seem quite stable — but that it would be a far worse idea to run a non-ECC machine as a densely-packed multitenant VM hypervisor today (i.e. to tile main memory with OS kernels), than it would have been ~20 years ago when memory densities were lower. Can anyone attest to this?
(I’d just ask for actual numbers on whether per-cell per-second errors have increased over the years, but I don’t expect anyone has them.)
I think it's been quadratic with a pretty low contribution from the order 2 term.
Think of the number of events that can flip a bit. If you make bits smaller, you get a modestly larger number of events in a given area capable of flipping a bit, spread across a larger number of bits in that area.
That is, it's flip event rate * memory die area, not flip event rate * number of memory bits.
In recent generations, I understand it's even been a bit paradoxical-- smaller geometries mean less of the die is actual memory bits, so you can actually end up with fewer flips from shrinking geometries.
And sure, your other effect is true: there's a whole lot fewer bitflips that "matter". Flip a bit in some framebuffer used in compositing somewhere-- and that's a lot of my memory-- and I don't care.
Sorry, I don't have the numbers you asked for. But afaik one other effect is that "modern" semiconductor processes like FinFET and Fully-Depleted Silicon-on-Insulator are less prone to single event upsets and especially result in only a single bit flipping and no drain of a whole region of transistors from a single alpha particle.
I don't know if ECC is that important, but reliability of RAM (or any storage) feels pretty crazy to me.
128GB being refreshed every second for a month error requires that the per-bit refresh process has a reliability of 99.9999999999999999% to be flawless.
Considering we are dealing with quantum effects (which are inherently probabilistic), I wouldn't trust myself to design anything like that.
Now back to ECC, I'll probably be corrected, but I don't think ECC helps gain more than two order of magnitudes, so we still need incredibly reliable RAM. If we move to ECC RAM by default everywhere, aren't we simply going to get less reliable RAM at the end?
A system on Earth, at sea level, with 4 GB of RAM has a 96% percent chance of having a bit error in three days without ECC RAM. With ECC RAM, that goes down to 1.67e-10 or about one chance in six billions.
So I'd say ECC is not only important but insanely impactful. There's a reason why many organizations don't even want to hear about getting rigs with non-ECC memory.
I like when people back up their claims with numbers, but would you mind describing roughly what that 96% probability of error is based upon?
I understand altitude has some kind of proportionality to cosmic ray exposure, and number of bits will multiply the probability of an error.. I'm presuming there is also an inherent error rate to DRAM separate from environment. But what are those numbers.
Looks like the calculation was revised [0] after criticism:
> Under these assumptions, you'll have to wait about 33 to 600 days to get a 96% chance of getting a bit error.
What's more worrying is the variance, the above calculation is based on expected well behaved DRAM.. yet some computers just seem to have manufacturing defects that make the incidence of errors high enough to be a regular problem.
:D yes, although I expect you would need either a prohibitively large quantity of memory or a extremely slow rate of change in altitude to effectively measure it.
That’s a very interesting idea, and I think you totally could. You run some benchmarks, measure the bit flips, and after enough runs you’d be able to say with a degree of confidence what your altitude is. I wonder though what accuracy could be achieved with this?
How did you run those tests? From what I understand on the topic, for your results to be statistically significant you need at least hundreds of machines and very rigid testing methodology.
As someone who also ran a similar test myself and haven't seen a bit flip, I'm also skeptical of the 96% figure.
I'm too lazy to run the exact numbers right now, but with "4 GB, 96% percent chance, three days" as the hypothesis, I think you'll find that an experimental result of "8 GB, 0% chance, 14 days" is highly statistically significant.
Edit: rough back of napkin estimate - you're not seeing an event in roughly 10x trials (2x number of bits and ~5x number of days). Given hypothesis is true your experimental result has a probability of (1-0.96)^10 = very very small. Conclusion: hypothesis is false.
The 96% figure comes from Google and was obtained in a large scale experiment over many months.
I've been in this business long enough to have witnessed adverse effects of cosmic rays an non-ECC memory multiple times myself. I don't think you're sample gets anywhere near statistical significance. Not mentioning testing methodology.
My anecdotal evidence is far from rigorous, but the Google data from ten years ago doesn't match up with my experience running thousands of ECC enabled servers up to a few years ago. Their rates seem a lot higher than what my servers experienced; we would page on any ram errors, correctable or not (uncorrectable would halt the machine, so we would have to inspect the console to confirm; when we knowingly tried machines with uncorrectable errors after a halt, they nearly all failed again within 24 hours, so those we didn't inspect the console of probably were counted on their second failure), and while there were pages from time to time, it felt like a lot less than 8% of the machine having a
There's a lot of variables that go into RAM errors, including manufacturing quality and condition of the ram, the dimm, the dimm slot, the motherboard generally, the power supply, the wiring, and the temperature of all of those. Google was known for cost cutting in their servers, especially early on; so I wouldn't be surprised if some of that resulted in higher bitflip rate than running in commercially available servers. Things like running bare motherboards, supported only on the edges cause excess strain and can impact resistance and capacitance of traces on the board (and in extreme cases, break the traces).
No it doesn't. You're assuming an even distribution of errors, which is very much not the case.
Google found that the average number of errors is around that range, but they also found that only one third of their servers had any errors in a year.
So, I hear what you are saying. But, on the other hand, I have been using 2 non-ECC desktops for a workstation/server for the past ~6 years.. and I would be hard pressed to come up with a single situation where either of the machines randomly crashed or applications did anything 'unexpected' (to my knowledge, of course).
My point is, when you say there is a "96% chance of having an error in THREE DAYS", one would EXPECT to be having issues like.. all the time? So I'm not disagreeing with you, but with the amount of non-ECC machines all over the world and how insanely stable modern machines are, it still seems like a very low risk.
Now of course I agree that if you want to take every precaution, go ECC, but simple observation prove that this "problem" can't be as bad as the numbers are saying.
Your questions are perfectly valid. It's just that out of all the random bit flips that happen over a period of time on a non-ECC platform only a miniscule percentage will manifest to you in any noticeable way.
"33 to 600 days to get a 96% chance of getting a bit error."
Still, it seems way too high. I guess anyone with ECC RAM could confirm that they are getting those sort of recovered error rates?
RAM is not as reliable as you think. Some ECC memory hardly ever finds an error, some machines see them at a very consistent rate, e.g. 50 errors per TB-day. That would translate to 1-2 errors per day in a 32 GB PC. Without ECC you cannot know in which bucket you are.
You don't need to look at kernel crashes to speculate about bus and memory errors -- just check the logs on a few systems that do have ecc. Pretty soon you'll see correctable errors being reported.
I don’t know much about this topic, but is it possible that ECC memory is more prone to single bit errors than non-ECC memory because there is less pressure on companies to minimize such errors? If this were the case, it would skew the data.
There are 12.5% more memory cells for a given module size, which equals more targets to possibly be flipped by cosmic rays. It’s not crazy to think that modules of equivalent quality (same brand, same chip part numbers) would experience a greater incidence of that kind of single bit flip (which would be corrected on the ECC modules). If a manufacturer were shipping chips prone to bit flipping because of slightly radioactive packaging, as happened at times in the past, you might see something similar.
But you’ve got it backwards about the incentives. A manufacturer has less incentive to deliberately ship a defective part in the case of ECC modules. If the modules consistently log ECC errors, they can easily be identified and returned under warranty to the manufacturer. A consumer is much less likely to identify an intermittent problem with a non-ECC part.
I am trying to get a laptop with dual NVMe (for ZFS) and ECC RAM. I can't get that, at all - even without the other fancy things I would like such as a 4k OLED with pen/touchscreen.
In 2020, even the Dell XPS stopped shipping OLED (goodbye dear 7390!)
I will gladly give my money to anyone who sells AMD laptop with ECC. Hopefully, it will show there's demand for "high end yet non bulky laptops"
Lenovo P53 has 3 NVMe slots, 4k OLED with touchscreen (and optional pen) and up to 128GB ECC RAM if you choose the Xeon processor. It's big and heavy, but it exists.
I hope AMD will create a better market for the ECC laptop memory (right now it's hard to find + expensive).
I always have that conversation when ZFS comes up. Some peoples think ZFS NEEDS ECC, but in fact ZFS needs ECC much as every single one FS in Linux. And every single reliable Machine needs ECC.
There were some great follow up talks as well! It turns out a viable attack vector was also MX records. And there was the guy who registered kremlin.re ( versus kremlin.ru ).
miclosoft.com is only one bit away from microsoft.com. Used to see these problems all the time when I worked on gmail.
At Google even with ECC everywhere there wasn't enough systematic error detection and correction to prevent the global database of monitoring metrics from filling up with garbage. /rpc/server/count was supposed to exist but also in there would be /lpc/server/count and /rpc/sdrver/count and every other thing. Reminded me daily of the terrors of flipped bits.
Ahaha. Reminds me of when I worked there. One day a large service tanked in some datacenter because BigTable replication in that location just stopped. Digging in, it turned out the BigTable should have been replicating from YQ but had started trying to use QQ instead, which didn't exist. Q being one bit away from Y. Or it was something like that, I don't remember exactly. There'd been a bit flip in the exact part of memory that contained the name of the database cluster to replicate from!
As long as we're swapping war stories there was once a frontend server at that company which died of bit flip related causes but just before it did it managed to charge some product group for 2^63 bytes worth of network traffic in the internal quota system, which set off every budgetary alarm that capacity planners had.
ECC matters, even on the desktop, it's not even a discussion, to me.
If you think it doesn't matter: how do you know? If you don't run with ECC memory, you'll never know if memory was corrupted (and recovered).
That blue screen, that sudden reboot, that program crashing. That corrupted picture of your kid.
Who knows.
I'll tell you, who knows. God damn every sysadmin (or the modern equivalent) can tell you how often they get ECC errors. And at even a small scale you'll encounter them. I have, on servers and even on an SAN Storage controller, for crying out loud.
If you care about your data, use ECC memory in your computers.
I've got nearly 30 years of experience and not once has non ECC memory lead to corruption. Maybe a crash, maybe a panic, maybe a kernel dump...
But.. in all my time operating servers over 3 decades, it's always been bad drivers, bad code and problematic hardware that's caused most of my headaches.
Have i seen ECC error correction in logs? yeah.. I don't advocate against it but, i've found for most people you design around multiple failure scenarios more than you design around preventing specific ones.
Take the average web app - you run it on 10 commodity systems and distribute the load.. if one crashes, so what. Chances are, a node will crash for many more reasons other than memory issues.
If you have an app that requires massive amounts of ram or you do put all of your begs in one basket, then ECC makes sense...
I just know i like going horizontal and I avoid vertical monoliths.
Crashes might not matter, but silent data corruption does. The owner/user of that data will care when they eventually discover that it at some point mysteriously got corrupted.
Corruption can happen for any number of reasons. ECC doesn't correct for human error, protocol error, coding error, logic error, type error so on and so forth.
Again, I don't advocate NOT using ECC, but i'd say in complex systems, never assume ECC alone is enough... and if ECC becomes your champion cause, how could you enforce it through every device that touches data, provides data, consumes data or injects data?
Complex systems require constant probing. You always probe. If you stop probing to look for errors because you assumed ECC fixed it for you, then you're probably at more risk of corruption vs accepting the fact there will be an infinite risk of error and being defensive against all types.
I know what it does, but I still don’t care (so long as it costs money or even 1% performance).
It’s a tradeoff between money/performance and the frequency of crashes, corruption etc.
Bit rot is just one of many threats to my data. Backups take care of that as well as other threats like theft, fire, accidental deletion.
This is similar to my reasoning around the recent side channel attacks on intel CPUs. If I had a choice I’d like to run with max performance without the security fixes even though it would be less secure. Not because I don’t care about security but because 1% or 5% perf is a lot and I’d rather simply avoid doing anything security critical on the machine entirely than take that hit.
> Bit rot is just one of many threats to my data. Backups take care of that as well as other threats like theft, fire, accidental deletion.
No, that's the big mistake people make: backups just backup bit-rotted data, until it is too late and the last good version is rotated out and lost forever.
I’m aware. But the risk is extremely small (and 99.9% of important data is not created on the machine but goes directly from e.g iOS camera to backup).
My desktop machine is basically a gaming rig with disposable data. Hence the “performance over integrity”.
I also never rotate anything out. Every version of everything is in the backups. Storage is that cheap these days.
...unlike all the accountant machines which are just standard desktop grade ones that have invoices entered from their keyboards.
>Hence the “performance over integrity”.
If anything ECC would be better for performance, as it'd allow to clock memory higher. It's a mystery how Intel has managed to convince people ECC = enterprise/server market. The real cost of ECC is 1/8 more memory (and datatraces on the motherboard)... and if anything, virtually all intel cpus have the needed support/transistors in the memory controller, it's just desktop variants have that part fused off on purpose.
That's more a hypothetical "if ECC was common enough that the products would exist" scenario.
The current status is that it mostly doesn't exist outside OEM parts. Searching an online retailer for ECC brings me hundreds of results for "non-ECC" RAM. When they occasionally have 1-2 products with ECC, they aren't as aggressively binned as the non-ECC sticks. They'll have higher latencies and lower clock frequencies. Basically: you'll need to accept gambling on being able to overclock your ECC RAM.
ECC could have been common, if there was a meaningful CPU support for. Intel basically did - "you need Xeon to do exactly what an i7 does + ECC, it just costs 5 times more... We just fused the ECC part of memory controller of i7, so buy Xeon."
The entire discussion is about why ECC is not common and why ECC matters. There is no technical reason for ECC to be uncommon aside purposed market segmentation.
Of course, there is no intrinsic availability of ECC udimms for the retail market currently, however that does mean ECC has no use or benefits for a small extra production cost.
Completely agree. I'm not avoding it because I don't think it' s worth it in theory. It's because it'snot worth it in practice, right now. Because you basically either have to buy something slower, or can't buy it at all.
I have had in the past encountered an issue where line card was stripping exactly one bit of address data. Don't know of the follow up investigation, but it probably wasn't TCAM
Consumer awareness about ECC needs to be better, with recent security implications I simply can't understand why more motherboard manufacturers don't support it on AMD. Intel of course is all to blame on the blue side, I stopped buying their overpriced Xeons because of this.
The industry has convinced the average user of consumer hardware that PPA (Power,Performance,Area) is all that needs to get better with generational improvements. Hoping that the concerning aspects of security and reliability that have come to light in the recent past changes this.
I would also add that Row Hammer Attacks are much harder on ECC.
When I first tried to replicate the row hammer attack I was not getting any results. Turns out I was doing this on ECC. On non ECC memory the same test easily replicated the row hammer attack.
On die ECC is great for increasing reliability, if all else is equal, but if it doesn't report to the memory controller, and if the memory controller doesn't report to the OS, I think it will be worse than status quo, because all else won't be equal. With no feedback, systems are going to continue to run on the edge, but now detectable failures will all be multi-bit; because single bit errors are hidden.
Huh? Why would the memory controller not be updated accordingly? Also I have no idea about Linux or Mac, but Windows has had ECC support and active management for decades?
Normally, ECC has meant just the DIMM stores some extra bits, and the memory controller itself implements ECC-- writing the extra parity, and recovering when errors emerge (and halting when non-recoverable errors happen).
DDR5 includes on-die ECC, where the RAM fixes the errors before sending them over the memory bus.
This means if the bus between the processor and ram corrupts the bits-- tough luck, they're still corrupted. And it's unclear whether we're going to get the quality of memory error reporting that we're used to or get the desired halt-on-non-recoverable error behavior (I've not been able to obtain/read the DDR5 specification as yet).
It's part of the firmware first trend of fixing things at the firmware level before reporting problems up the stack. This makes it a real nightmare for systems integrators to do root cause analysis.
I always wondered why isn't ECC built into the memory controller, the same hardware that runs the bus into L3 or the page mapper could checksum groups of cachelines.
It seems redundant to have every module come with its own checking hardware.
AFAIK it is built into the memory controller, at least for ECC UDIMM. There's an extra DRAM chip on the module for parity (generally 8+1), but it is memory controllers responsibility to utilize it (that's why not all CPUs support ECC)
ECC is a function of memory controller, not memory, on current systems. There's also usually some form of ECC on whatever passes for system bus, and internal caches have ECC as well.
For memory controller, parity/ECC/chipkill/RAIM usually involved simply adding additional memory planes to store correction data. I believe the rare exceptions are fully buffered memories where you have effectively separate memory controller on each module (or add-in card with DIMMs)
As someone who has had to read thousands of random game crash reports from all over the interwebs (you know when Windows says you might want to send that crash log? like that), I totally agree.
Of all the things to be worried about, like OS bugs, bad hardware configuration, etc. bad memory is one of those really troubling things. You look at the code and say "it's can't make it here, because this was set" but when you can't trust your memory you can't trust anything.
And as the timeline goes to infinity, you may also get one of these reports and be asked to fix it... good luck.
It would be interesting to see whether there is a correlation between solar activity and game crashes — which in turn may provide an indication whether crashes are due to bugs or bit flips.
As long as someone is tracking crash times/locations, the data is definitely there. Working tech support for major ISPs, there were some “surprising” large-queue days that correlated with solar flares.
One of the tire pressure sensors in my car tires had a bit flip a couple years ago and I had to reprogram it's ID. Luckily it was a subaru, so only a light came on in the dash.
My old Honda crv however would turn traction control on if your pressure was low - which worked by applying brakes to wheels that were slipping. If you were going up a slippery hill you would soon have no power, sliding backwards nearly off the road in nowhere West Virginia on the way to a ski resort.
I have a TPMS programming tool (ATEQ QuickSet) and reader (Autel TS401), because I like to swap my winter / summer tires on my own. The TPMS light came on one day and inflating tires didn't help - I used the reader and found that one sensor's ID had changed. When I compared the ID (it was in hex) to the last programming - it was a single bit off. I couldn't reprogram the sensor itself, but I was able to update the ECU with the changed ID using the ATEQ.
I live in Denver but spend a lot of time skiing around 11k feet, maybe the higher elevation means more radiation.
Similar story, we saw that one particular IP address in a public cloud network had a 3% TLS handshake error rate. We diverted traffic and then analyzed with wireshark. We found one particular bit was being pulled low (i.e. 0 -> 0 and 1 -> 0). HTTP connections didn't notice but TLS checksum verifications would randomly fail. Had a hell of a time convincing the cloud provider they had a hardware fault- turned out to be a bug which disabled ECC on some of their hardware.
Aside: I'm surprised you got a TPMS programming tool instead of a set of steelies. Big wheels? Multiple winter vehicles?
I have 2 cars. I like the TPMS to work since I've had 3 nails in tires in 4 years (newer construction area). Also the TPMS light in my impreza is almost as bright as the sun.
IIRC higher altitudes definitely register more radiation, the latitude effect being relatively minor in comparison.
Interestingly enough, average background radiation is actually the highest in the US exactly where you are[0] and I've seen comments (on a few websites talking about it) ranging from "the Rockies are loaded in Uranium" to "what about the Rocky Flats nuclear weapons site". Perhaps you're exactly in the right place to be an advocate for ECC :)
It is incomprehensible that there are still NAS devices being sold without ECC support.
Synology took a step in the right direction to offer prosumer devices with ECC but it is not really advertised as such. It is actually difficult to find which do have ECC and which ones don't.
>Synology took a step in the right direction to offer prosumer devices with ECC
I just look it up because if it was true it would have been news to me. Synology have been known to be stingy with Hardware Spec. But none of what I called Prosumer, the Plus Series have ECC memory by default. And there are "Value" and "J" Series below that.
Edit: Only two model from the new xx21 series using AMD Ryzen V has ECC memory by default.
Aye. I have an assert in the code that fronts a very pedantic test of the context. In all cases when this assert was tripped (and reported) an overnight memtest86 test surfaced RAM issues.
- Edit -
Also, bit flips in the non-ECC memory are _the_ cause of the "bitrot" phenomenon. That is when you write out X to a storage device, but you get Y when you read it back. A common explanation is that the corruption happens _at rest_. However all drives from the last 30+ years have FEC support, so in reality the only way a bit rot can happen is if the data is damaged _in transit_, while in RAM, on the way to/from the storage media.
So, if you ever decide if to get an ECC RAM, get it. It's very much worth it.
> Also, bit flips in the non-ECC memory are _the_ cause of the "bitrot" phenomenon. That is when you write out X to a storage device, but you get Y when you read it back. A common explanation is that the corruption happens _at rest_. However all drives from the last 30+ years have FEC support, so in reality the only way a bit rot can happen is if the data is damaged _in transit_, while in RAM, on the way to/from the storage media.
Problems can definitely happen in the IO controller, RAID controller, cable, and disk controller. AFAIK all of these were seen and motivations for the existence of ZFS. One of their biggest insistence was that drives are universally lying bastards and should not be trusted any further than they can be thrown.
See "Your computer is broken". They essentially inserted a stress test into the game that verified if the hardware was still doing calculations correctly, and if not, inform the user.
For which bit flips are even more relevant: EdDSA has this nasty tendency of leaking the private key if the wrong bits are flipped (there are papers on fault injection attacks). People who sign lots of stuff all the time, say Let's Encrypt, could conceivably gain some piece of mind with ECC.
(Note: EdDSA is still much much better than ECDSA, most notably because it's easier to implement correctly.)
Which means, assuming google is running very large machines with lots of memory that one might expect a single correctable error once every 6-10 years on your average workstation of small server. That's generously assuming your workstation has 1/3 as much memory as the average google server.
Google does not use very large or even large machines for most of their fleet. You can quickly see in the paper this is for 1, 2, and 4 GB RAM machines (in 2006-2008).
These were state of the art 13 years ago. It’s not safe to extrapolate from this paper that they aren’t using servers having significantly more memory today. Thirteen years is more than 2 full depreciation intervals.
With a single bit flip on 8% of the dimms you only need 12.5 dimms in your workstation to have one bit flip every year. Not everyone has that much dimms, but at least 4 is pretty normal. So in average every 3 years for every workstation.
But i don‘t know how relevant these metrics from 2009 are. Did memory got better or worse compared to 2009 for bit flips?
ECC works if done right. Accessing a memory location can fix bit-flips (ECC is a 'correcting' code). But systems that don't regularly visit every memory location, can accumulate risk. Those dark corners of RAM can eventually get double-bit errors and be uncorrectable. So an OS might 'wash' RAM during idle moments, reading every location in a round-robin manner to get ECC to kick in and auto-correct. Doesn't matter how fast (1M every hour or whatever) as long as somehow ECC has a chance to work.
Interesting, similar to scrubbing raid arrays. How often do those double bitflips appear though? You'd have to have a pretty long running server for that to be a problem, no?
A double-bit error in many cases is fine. If the error is at least detectable at the time of a read, your protection worked. What's scary is a triple-flip event. Most of those will still look like corrupted data, but if it happens to flip into looking like a fixable, single-bit error, you're out of luck and won't even know it.
> Most of those will still look like corrupted data,
Not if you're using a typical 72-bit SECDED code[0].
You have two error indicators: a summary parity bit (even number of errors: 0,2,etc vs odd number of errors: 1,etc), and a error index: 0 for no errors, or the bitwise xor of the locations each bit error.
For a triple error at bits a,b, and c, you'll have summary parity of 1 (odd number of errors, assumed to be 1), and a error index of a^b^c, in the range 0..127, of which 0..71[1] (56.25%, a clear albeit not overwhelming majority) will correspond to legitimate single-bit errors.
1: or 72 out of 128 anyway; the active bits might not all be assigned contiguous indexes starting from zero, but it doesn't change the probability and it's simpler to analyse if summary is bit 0 and index bit i is substrate bit 2^i.
Actually, a,b,c are also sampled from the range of valid bit indexes (not uniformly on 0..127), so you might be able to pick a cardinality-72 subset of 0..127 such that random a^b^c is disproportionately likely to fall outside that subset (and thus get diagnosed as not a valid single-bit error correction). I don't know that any existing ECC implementations actually do that, though.
Edit: did some cursory testing and using indexes 0..71 actually catches only ~24.04% (86016/357840) of triple-bit errors, compared the theoretical 43.75% (156555/357840, I think?) from a random error index. So "doesn't change the probability" is just completely wrong given that a,b,c are randomly chosen from the 72 substrate bits, rather than from 128 possible 7-bit indexes.
Oddly enough, testing random selections of 72 valid indexes (eg 74773982'EBD0D35C'C5BEB2D8'C3FE9C5E, where set bits correspond to used indexes) actually gives slightly better results than theory (44.97% for that one, 44.65% is the lowest in the last dozen or so), which is somewhat interesting, but I still haven't found any bit assignment that gives better than 50% (178920/357840) catchment of triple bit errors.
565 comments
[ 39.7 ms ] story [ 1345 ms ] threadIts.
There, I finally corrected Linus Torvalds in something. :))
Now the point about internally doing ECC is an interesting one, could be a way out of this mess. And apparently ECC is more available in AMD land
Closest you can come to a nuc with ecc is I think a mini server equipped with one of the four-core i3 parts that have ecc.
As for NUCs, I thought there were some Atom chips with ECC but not in NUC form factor ? Something shiny from Logic Supply might do then...
Yes it is. The problem is they dont really advertise it. I'm not certain but it might even be standard on AMD chips, but if they dont say so and board makers are also unclear, who knows...
For consumer motherboard OEMs, only AMD effectively has ECC support (Intel's has been so spotty and haphazard from product to product), and of AMD users, only a small number care about ECC.
So motherboard companies, being resource and time-starved as they are, don't make it a priority to address such a small user-base.
If Intel started shipping ECC on everything, it would go a long way towards shifting the market.
The first time I wrote "your" instead of "you're" in English I thought it was quite a milestone!
I kinda disagree because while the homophony works in (spoken) English in written it stands as a sore thumb. So yeah you will make it if you only heard it but doesn't know the written form.
(And in their native language it's probably two unrelated words, so that might intensify the feeling of wrongness)
"You're/your", "their/they're", "its/it's" and the like are a different story, because I do pronounce those the same and they're all very common.
I'm happy to see people here on HN respect the difficulty of learning languages. Most foreigners that speak Finnish do it very poorly at first and even after decades they still sound like foreigners. But it shows huge respect to our small country for someone to make the effort, and we really appreciate it. I'm hoping other people see learning their own mother tongue the same way. Sure, most of us need English, but learning it well is still a huge task.
I spent my preschool years in a multicultural environment and English was our lingua franca(ironically the school-mandated language was French), so I didn't properly learn contractions until grade school - same with similarly sounding words like "than vs then" and "your vs you're".
It's because they have two different uses (three if you count nested quotes, but those aren't common and are pretty easy to figure out), contractions and possession, and they seemingly collide on words like "its" where you'd think it could mean either.
Not sure if you've already learned this (or if it helps), but English used to be declined, and its pronouns still are, e.g. they/their/them. That's why "its" isn't contracted; the possessive marker is already in the word.
The ambiguity makes it hard for me to say. "It's the best" vs "its is the best"
It doesn't feel right dropping the is
If in doubt, get ECC. Do your own research on how it works and why. This post won’t explain it, just will blame Intel (probably rightfully so).
> We have decades of odd random kernel oopses that could never be explained and were likely due to bad memory. And if it causes a kernel oops, I can guarantee that there are several orders of magnitude more cases where it just caused a bit-flip that just never ended up being so critical.
It might be false, but I think it's a reasonable assumption.
I can understand Linus's frustration from that point of view: without ECC RAM when you get some super weird crash report where some pointer got corrupted for no apparent reason you can't be sure if it's was just a random bitflip or if it's actually hiding a bigger problem.
It's the "once every other year" type of bitflip that's the problem. The proverbial "cosmic ray" hitting your DRAM and flipping a bit. That will be caught by ECC but it'll most likely remain a total mystery if it causes your non-ECC hardware to crash.
Then one of them brought in his personal Geiger counter and found the radiation coming off the steel in that rack case was significantly higher than background.
You may never know when the metal you use was recycled from something used to hold radioactive materials.
Hi alphabet lawyers.
They argued that 'Google' has now become a verb meaning 'to search the Internet for' and as such alphabet should have the name taken away.
> A large-scale study based on Google's very large number of servers was presented at the SIGMETRICS/Performance ’09 conference.[6] The actual error rate found was several orders of magnitude higher than the previous small-scale or laboratory studies, with between 25,000 (2.5 × 10−11 error/bit·h) and 70,000 (7.0 × 10−11 error/bit·h, or 1 bit error per gigabyte of RAM per 1.8 hours) errors per billion device hours per megabit. More than 8% of DIMM memory modules were affected by errors per year
But these days with the RAM density being so high and bitflipping attacks being more than a theoretical threat it seems like there's really no good reason not to switch to ECC everywhere.
Not all CPUs support ECC however.
- That bit is not stored in RAM but rather somewhere within CPU which can have different bit flip frequency characteristics.
- Even assuming this bit can be flipped, most computers wouldn't have ECC RAM inserted because well... the CPU doesn't support it.
- And even assuming you have ECC RAM inserted, this bit flipping while your computer is running will likely lead to CPU hang considering the CPU didn't fill in error correction bits and will lead CPU to think that pretty much every single byte is corrupt.
- And all in all, even if you did manage to turn on ECC bit, it would only work until the computer gets turned off.
https://rog.asus.com/forum/showthread.php?112750-List-Asus-M...
Circa Zen1 launch, ASRock claimed all of their consumer boards would support ECC.
Because ECC means Error Correcting Code, by definition, any board that claims ECC support must actually correct the errors. The ECC codes used now, with 8 extra bits for each 64 data bits, correct any 1-bit error and detect any 2-bit errors.
Very old computers (25 years old, or more) used parity instead of ECC and they just detected any 1-bit error (and any errors with an odd number of flipped bits), without being able to correct the errors.
It's faster too.
Bit flips happen and are real. I really wish ECC was plentiful and not brutally expensive!
It is filled to the gunwales with ECC RAM.
Cost him the equivalent of $7k or so. Eeek.
Finding a memory upgrade seems difficult though.
0: https://www.lenovo.com/us/en/laptops/thinkpad/thinkpad-p/Thi...
I'm guessing you won't find any.
Xeon with ECC are not that overpriced compared with similar Core without. Likewise, RAM sticks with ECC are cheap to produce (basically just one more chip to populate per side per module). Likewise soldered RAM would simply add maybe $10 or $20 of extra chips.
If it needs ECC memory to do that, then fit it with ECC memory. If there are other ways to achieve that (for example deeper dram cells to be more robust to cosmic rays) that's fine too.
Just meet the reliability spec - I don't care how.
That's why I've always been on the fence with this ECC thing. For servers it's vital because you need stability and security.
For desktops I think that for a long time it was fine without ECC. If I have to chose between having, say, 30% more RAM or avoid a potential crash once a year, I'll probably take the additional RAM.
The problem is that now these problem can be exploited by malicious code instead of just merely happening because of cosmic rays. That's the main argument in favour of ECC IMO, the rest is just a tradeoff to consider.
And I mean, we all spend all day editing test messages and comments and files on non-ECC hardware, yet bitflip-induced corruption is rare enough that I can't say that I've witnessed a single instance of it in my life, despite spending a good chunk of it looking at screens.
It's just not a problem that occurs in practice in my experience. If you're compiling the release build of a critical piece of software, you probably want ECC. If you're building the dev version of your webapp or writing an email to your boss, you'll probably survive without it.
I couldn't tell, as a user, which if those corruptions and crashes were causes by bitflips. Could you?
And that's probably not what GP asked for. There's a difference between guaranteeing an error rate of 1 error per century of use on average, and guaranteeing it over the course of an actual century. It might be okay to guarantee that error rate for only 5 years of uninterrupted use, and degrade after that. For instance:
Now, given how energy hungry and polluting the whole computer industry actually is, it might be a good idea to shoot for extreme durability and reliability anyway. Say, sustain 1 error per century, over the course of fifty years. It will be slower and more expensive, but at least it won't burn the planet as fast as our current electronics.It might present itself as a 1pixel colour difference, but it could be more damaging (incorrect finances, in accounting software for example). Software trusts memory; but memory can lie.
That’s dangerous.
If you want to detect a bit flip, use parity.
Or after it's been checked. (time-of-check vs time-of-use)
I guess in theory some software could produce signed data in CPU cache, and "commit" it to RAM as a verified block.
But the overhead would be enormous. Would you slow down your CPU by half in order to not pay 12.5% more for RAM?
Hmm, I wonder what SGX and similar do about this.
If my data is fine being corrupted to save 12.5% on RAM costs, then why am I even bothering processing the data? Apparently it's worthless.
People today weigh the cost of maybe 16 vs 32GB on a mid-tier desktop. ~doubling the cost for twice the RAM. Yes, paying 12.5% more for ECC RAM is a no-brainer.
https://eclecticlight.co/2020/12/09/what-happens-when-an-m1-...
It’s so easy to chalk these kind of errors to other issues, a little corruption here, a running program goes bezerk there- could be a buggy program or a little accidental memory overwrite. Reboot will fix it.
But I ran many thousands of physical machines, petabytes of RAM, I tracked memory flip errors and they were _common_; common even in: less dense memory, in thick metal enclosures surrounded by mesh. Where density and shielding impacts bitflips a lot.
My own experience tracking bitflips across my fleet led me to buy a Xeon laptop with ECC memory (precision 5520) and it has (anecdotally) been significantly more reliable than my desktop.
I love that AMD doesn't intentionally break ECC on its consumer desktop platforms and upgraded to the Threadripper in 2017.
https://ark.intel.com/content/www/us/en/ark/compare.html?pro...
https://ark.intel.com/content/www/us/en/ark/products/208074/...
Problem is this processor is an Embedded processor so probably not for us
> Industrial Extended Temp, Embedded Broad Market Extended Temp
My understanding is Intel does not support ECC on the desktop unless you pay extra.
https://ark.intel.com/content/www/us/en/ark/products/199280/...
0: https://ark.intel.com/content/www/us/en/ark/products/199281/...
1: https://ark.intel.com/content/www/us/en/ark/products/134886/...
If they gave it some other name, it would be only misleading. Calling it TDP is a lie.
May be we should use a new term for it, something like iTDP.
And on the flip side, if you're building a desktop PC with a more high-end Intel processor, you will usually have to change a lot of motherboard firmware settings to get the behavior to resemble Intel's own recommendations that their TDP numbers are supposedly based on. Without those changes, lots of consumer retail motherboards default to having most or all of the power limits effectively disabled. So out of the box, a "65W" i7-10700 and a "125W" i7-10700K will both hit 190-200W when all 8 cores/16 threads are loaded.
If a metric can in practice be off by a factor of three in either direction, it's really quite useless and should not be quantified with a scientific unit like Watts.
One consequence of using a moving average is that if the CPU has been idle for a long time then starts running a high power workload instantaneous power consumption can momentarily exceed the TDP while the average catches up. This is often misleadingly referred to as "turbo mode" by hardware review sites. It's not a mode, there's no state machine at work here, it's just a natural result of using a moving average. The use of EWMA is meant to model the heat capacity of the cooling solution. When the CPU has been idle for a while and the heatsink is cool, the CPU can afford to use more power while the heatsink warms up.
Another factor which confuses things is motherboard firmware disabling power limits without the user's knowledge. Motherboards marketed to enthusiasts often do this to make the boards look better in review benchmarks. This is where a lot of the "Intel is lying" comes from, but it's really the motherboard manufacturers being underhanded.
The situation on the AMD side is of course a bit different. AMD's power and frequency scaling is both more complex and much less documented than Intel's so it's hard to say exactly what the CPU is doing. What is known is that none of the actual power limits programmed into the CPU align with the TDP listed in the spec. In practice the steady state power consumption of AMD CPUs under load is typically about 1.35x the TDP.
Unlike Intel, firmware for AMD motherboards does not mess with the CPU's power limit settings unless the user does so explicitly. Presumably this is because AMD's CPU warranty is voided by changing those settings, while Intel's is not.
https://www.intel.com/content/www/us/en/architecture-and-tec...
https://www.amd.com/en/technologies/turbo-core
Intel mobile processors actually obey TDP better than AMD processors do - Tiger Lake has a hard limit, when you configure a 15W TDP then it really is 15W steady-state once boost expires, while AMD mobile products will pull up to 50% more than configured in steady-state operation. (the gap is larger than desktop)
https://images.anandtech.com/doci/16084/Power%20-%2015W%20Co...
"the brands measure it differently" is sort of theoretically true but not in the sense people think, and in practice it is not true.
On AMD it is literally just a number they pick that goes into the boost algorithm. Robert Hallock did some dumb handwavy shit about how it's measured with some delta-t above ambient with a reference cooler but the fact is that the chip itself basically determines how high it'll boost based on the number they configure, so that is a self-fulfilling prophecy, the delta-t above ambient is dependent on the number they configure the chip to run at.
In practice: what's the difference between a 3600 and a 3600X? One is configured with a TDP of 65W and one is configured with a TDP of 95W, the latter lets you boost higher and therefore it clocks higher. Configure them both to a 65W PPT limit and they will boost to pretty much the same place.
Intel nominally states that it's measured as a worst-case load at base clocks, something like Prime95 that absolutely nukes the processor (and even then many processors do not actually hit it). But really it is also just a number that they pick. The number has shifted over time, previously they used to undershoot a lot, now they tend to match the official TDP. It's not an actual measurement, it's just a "power category" that they classify the processors as, it's informed by real numbers but it's ultimately a human decision which tier they put them in.
So in practice, for both brands, it is just a number they pick. They have different theoretical methods for getting there but ultimately the marketing department looks at where the clocks would put them and pick a power number that they think represents that. It is not, in practice, a pure measurement from either brand, it is just a "category" they use.
Real-world you will always boost above base clocks on both brands at stock TDP, at least on real-world loads. You won't hit full boost on either brand without exceeding TDP, the "AMD measures at full boost" is categorically false despite the fact that it's commonly repeated. AMD PPT lets them boost above the official TDP for an unlimited period of time, they cannot run full boost when limited to official TDP.
Why does it matter? It doesn't idle that high; it only goes that high of you're using it flat out, in which case the extra power usage is justified because it's giving that much more performance over a 100 W TDP CPU. Now I totally get it if you don't want to go Threadripper just for ECC because it's more expensive, but max power draw, which you don't even have to use? I've never seen anyone shop a desktop CPU by TDP, rather than by performance and price.
Oh oh, me! Back in the day I bought a 65W CPU for a system that could handle a 90W. I wanted quiet and figured that would keep fan noise down at a modest performance penalty. It should also last longer, being the same design but running cooler. I ran that from 2005 until a few years ago (it still run fine but is in storage).
Planning to continue this strategy. I suspect it's common among SFF enthusiasts.
My passive-cooled desktop is also running a slightly trottled down 65W CPU.
So yes, there are people who choose there hardware by TDP.
These days, a quiet, pwm fan with good thermal paste (and maybe some linux CPU throttling) more than achieves my needs for a "silent" pc 99% of the time.
I would love to be told my above assumptions are wrong if they are.
The worst bit is, AMD and Intel define TDP differently-- neither is the maximum power the processor can draw-- though Intel is far more optimistic.
IMO, shopping by performance/watt makes sense. Shopping by TDP doesn't. (Especially since there is no comparing the AMD and Intel TDP numbers as they're defined differently; neither is the maximum the processor can draw, and Intel significantly exceeds the specified TDP on normal workloads).
In a sandwich style case you're usually limited to low profile coolers like Noctua L9i/L9a since vertical height is pretty limited.
If you want a 45W TDP from the 3700X, you can just pop into Ryzen Master and ask for a 45W TDP. Boom, you're running in that envelope.
I think shopping based on TDP is not the best, because it's not comparable between manufacturers and because it's something you can effectively "choose".
On AMD, it's a utility you run. I believe you may require a reboot to apply it. On some Intel platforms, it's been settings in the BIOS.
> It sounds interesting if I can run a beefy rig as a power efficient device, for always-on scenarios, and then boost it when I need.
This is what the processor is doing internally anyways. It throttles voltage and frequency and gates cores based on demanded usage. Changing the TDP doesn't change the performance under a light-to-moderate workload scenario at all.
Ryzen Master lets you change some of the tuning for the choices it makes about when and how aggressively to boost, though, too.
I don't really see the reason in paying for a 100w TDP premium if I'm just going to scale it down to 65w.
Yup, they're out there.
> I don't really see the reason in paying for a 100w TDP premium if I'm just going to scale it down to 65w.
You might want the core count or peak performance for the very short term. When I was looking, running 65W parts in the 45W envelope was only about a 7% penalty, so you get a bunch more performance/watt.
As a petty "Take that", I dropped the max frequency from 2.0 GHz to 1.0 GHz. I ran a couple benchmarks to prove the cap was working, and then just kept it at 1.0 for a few months, to prove my point.
It made a bigger difference on my ARM SBC, where I tried capping the 1,000 MHz chip to 200 or 400 MHz. That chip was already CPU-bound for many tasks and could barely even run Firefox. Amdahl's Law kicked in - Halving the frequency made _everything_ twice as slow, because almost everything was waiting on the CPU.
And the relationship between power and performance isn't linear as processor voltages climb trying to squeeze out the last bit of performance.
So if you want to take a 105W CPU and ask it to operate in a 65W envelope, you're not giving up even 1/3rd of peak performance, and much less than that of typical performance.
AMD Ryzen 9 5950X: 20.6W for a single core at 5050MHz, 49W for the whole package. (And it’s generally the package figure that you care about.)
AMD Ryzen 9 5900X: 17.9W/54W at 4875MHz.
AMD Ryzen 7 5800X: 17.3W/37W at 4825MHz.
AMD Ryzen 5 5600X: 11.8W/28W at 4650MHz (though the highest core reading is 13W, at three cores loaded).
You’re both correct: by simply restricting that power envelope by 40%, you shed a lot less multi-threaded performance than people realise, and no single-threaded performance.
Look at the 5950X figures, and you observe that at about 120W, it can run 6 cores at 4,650MHz (27,900 core–MHz), or 16 cores at 3,775MHz (60,400 core–MHz).
Expressed one way: by dropping the frequency by 20%, power per watt increased by around 2.7×.
Expressed another way: let’s skip a 65W envelope—put this particular 105W chip in a 40W envelope and you lose only 20% of your six-cores performance. Seriously. But I’m not sure what the curve would look like if you load all 16 cores at a 40W envelope, what speed they’d be going at.
(But do remember that “TDP” is a bit of a mess as a concept, and that we’re depending on non-core power consumption being generally fairly consistent regardless of load.)
That's me. When I start to plan for a new system, I select the processor first and read its thermal design guidelines (Intel used to have nice load vs. max temp graphs in their docs) and select every component around it for sustained max load.
This results in a more silent system for idle and peace of mind for loading it for extended duration.
You can passively cool threadrippers if you underclock them enough and have good ventilation in case.
In my case loading means maxing out all cores and extended period of time can be anything from five minutes to hours.
Both are optimistic lies, but-- if you look at the documents it looks like currently AMD needs more cooling, but actually dissipates less power in most cases and definitely has higher performance/watt.
Doesn't matter for me since I'm not interested in comparing them.
> Both are optimistic lies, but-- if you look at the documents it looks like currently AMD needs more cooling, but actually dissipates less power in most cases and definitely has higher performance/watt.
I'm aware of the situation, and I always inflate the numbers 10-15% to increase headroom in my systems. The code I'm running is not a most case code. A FPU heavy, "I will abuse all your cores and memory bandwidth" type, heavily optimized scientific software. I can sometimes hear that my system is swearing at me for repeatedly running for tests.
I don't like to add this paragraph but, I'm one of the administrators of one of the biggest HPC clusters in my country. I know how a system can surpass its TDP or how can CPU manufacturers skew this TDP numbers to fit in envelopes. We make these servers blow flames from their exhausts.
Performance/watt metrics and idle consumption would have been a far better way to make this choice.
If you have a choice between A) something that can dissipate 65W peak for 100 units of performance, but would dissipate 4W average under your workload, and B) something that can dissipate 45W peak for 60 units of performance, but would dissipate 4.5W under your workload... I'm not sure why you'd ever pick B.
Also, even though the CPU may draw less, can still the power supply waste more, just because it is beefy? Comparing with a sports car, they have great performance, but also use more gas in ordinary traffic? Can a computer be compared with that?
Community benchmarks, from Tom's Hardware, etc.
The vendor numbers are make believe-- you can't use them for power supply sizing or for thermal path sizing. If you look at the cited TDP numbers today-- it can be misleading-- e.g. often Intel 45W TDP parts use more power at peak than AMD 65W parts.
On modern systems, almost none of the idle consumption is the processor. The power supply's idle use and motherboard functions dominate.
> Also, even though the CPU may draw less, can still the power supply waste more, just because it is beefy?
Yes, having to select a larger power supply can result in more idle consumption, though this is more of a problem on the very low end.
Get a huge cooler like Noctua d14, and you pc becomes silent. It lasts forever, requires no maintenance, a good investment.
If you are adventurous, watercooling is even better, but its a can of worms I decided I'd rather live without - possibility of leaks and cost make it harder to justify
What specific configurations (CPU, MB, RAM) are known to work?
Let's say I have a Ryzen system, how can I check if ECC really works? Like, can I see how many bit flips got corrected in, say, last 24h?
*not officially, and the memory controller provides no report for 'fixed' errors.
0: http://www.asrock.com/mb/AMD/X570%20Taichi/
Looking back at my notes, the output of journalctl -b tells should say something like, "Node 0: DRAM ECC enabled."
Then 'edac-ctl --status' should tell you that drivers are loaded.
Then you run 'edac-util -v' to report on what it has seen,
You can also use memtest86+ for this, although I don't recall if it requires specific configuration for ECC testing.
They aren't tested on it, so it's possible to get a dud, but it's minuscule chance that isn't worth bothering.
Now, to actual issues you can encounter: motherboards
The problem is that ECC means you need to have, iirc, 8 more data lines between CPU and memory module, which of course mean more physical connections (don't remember how many right now). Those also need to be properly done and tested, and you might encounter a motherboard where it wasn't done. Not sure how common, unfortunately.
Another issue is motherboard firmware. Even though AMD supplies the memory init code, the configuration can be tweaked by motherboard vendor, and they might simply break ECC support accidentally (even by something as simple as making a toggle default to false then forgot to expose it in configuration menu).
Those are the two issues you can encounter.
The difference with AFAIK Threadripper PRO, and EPYC, is that AMD includes ECC in its test and certification programs for it, which kind of enforces support.
I think some Gigabyte boards are infamous for this in certain circle
OTOH: Gigabyte might have a Threadripper PRO motherboard (WRX80 chipset) coming out in the future
When Windows detects a memory error, it will record it in the system event log, under the WHEA source. As a side note, this is also how memory errors within the CPU's caches are reported under Windows.
[1] https://superuser.com/questions/893560/how-do-i-tell-if-my-m...
You must check the specifications of the motherboard to see if ECC memory is supported.
As a rule, all ASRock MBs support ECC and also some ASUS MBs support ECC, e.g. all ASUS workstation motherboards.
I have no experience with Windows and Ryzen, but I assume that ECC should work also there.
With Linux, you must use a kernel with all the relevant EDAC options enabled, including CONFIG_EDAC_AMD64.
For the new Zen 3 CPUs, i.e. Ryzen 5xxx, you must use a kernel 5.10 or later, for ECC support.
On Linux, there are various programs, e.g. edac-utils, to monitor the ECC errors.
To be more certain that the ECC error reporting really works, the easiest way is to change the BIOS settings to overclock the memory, until memory errors appear.
Edit: as detaro mentioned in the reply, there is, and here's the source [0] -- that's what they mean by "RAS" on promotional pages [1]. That indeed looks like a nice option.
[0] https://www.amd.com/system/files/documents/updated-3000-fami...
[1] https://www.amd.com/en/products/embedded-epyc-3000-series
There are computers in the Intel NUC form factor, with ECC support (e.g. with Ryzen V2718), e.g from ASRock Industrial.
Found some here -- bottom of the EPYC product line starts at $2849 ...!
https://www.velocitymicro.com/wizard.php?iid=337
EPYC TDP ranges from "a lot lower" (35W embedded, 120W regular) up to "comparable with" (180-240W, a single 280W model) relative to Threadripper (180-250W last gen; current gen is all 280W). It's definitely not a lot higher on the Epyc side.
> I think of Threadripper as mid-tier, and EPYC as the high-end.
This oversimplifies to the point of not being a useful intuition (or is arguably even incorrect). Threadripper is a SKU with a moderate number of cores at relatively high clocks; (high-power) EPYC SKUs have a lot of very efficient cores running at lower clocks. They both have a niche, but Threadripper has unambiguously better single-core performance due to the ~20% higher clocks. And single-core IPC still matters in many applications (to oversimplify: Amdahl's law; but also, latency-sensitive applications).
Any apples-to-apples comparable Intel CPU will have comparable power use. The difficulty is that Intel didn't really have anything like Threadripper — their i9 series was the most comparable (high clocks and moderate core counts), but i9 explicitly did not support ECC memory, nullifying the comparison.
You're looking at 2950X, probably? That's a Zen+ (previous gen) model. 16 core / 32 thread, 3.5 GHz base clock, launched August 2018.
Comparable Intel Xeon timeline is Coffee Lake at the latest, Kaby lake before that. As far as I can tell, no Kaby Lake nor Coffee Lake Xeons even have 16 cores.
The closest Skylake I've found is an (OEM) Xeon Gold 6149: 16/32 core/thread, 3.1 GHz base clock, 205W nominal TDP (and it's a special OEM part, not available for you). The closest buyable part is probably Xeon Gold 6154 with 18/36 core/threads, 3GHz clock, and 200W nominal TDP.
Looking at i9 from around that time, you had Skylake-X and a single Coffe Lake-S (i9-9900K). 9900K only has 8 cores. The Skylake i9-9960X part has 16/32 cores/threads, base clock of 3.1GHz, and a nominal TDP of 165W. That's somewhat comparable to the AMD 2950X, ignoring ECC support.
Another note that might interest you: you could run the Threadripper part at substantially lower power by sacrificing a small amount of performance, if thermals are the most important factor and you are unwilling to trust Ryzen ECC: http://apollo.backplane.com/DFlyMisc/threadripper.txt
Or just buy an Epyc, if you want a low-TDP ECC-definitely-supported part: EPYC 7302P has 16/32 cores, 3GHz base clock, and 155W nominal TDP. EPYC 7282 has 16/32 cores, 2.8 GHz base, and 120W nominal TDP. These are all zen2 (vs 2950X's zen+) and will outperform zen+ on a clock-for-clock basis.
> And though ECC is not disabled in Ryzen CPUs, AFAIK it's not tested in (or advertised for) those, so one won't be able to return/replace a CPU if it doesn't work with ECC memory, AIUI, making it risky.
If your vendor won't accept defective CPU returns, buy somewhere else.
> Though I don't know how common it is for ECC to not be handled properly in an otherwise functioning CPU; are there any statistics or estimates around?
ECC support requires motherboard support; that's the main thing to be aware of shopping for Ryzen ECC setups. If the board doesn't have the traces, there's nothing the CPU can do.
ECC isn't validated by AMD for AM4 Ryzen models, but it's present and supported if the motherboard also supports it. Many motherboards have ECC support (the manual will say for sure), and a handful of models even explicitly advertise it as a feature.
I have a Ryzen 9 3900X on an ASRock B450M Pro4 and 64 GB of ECC DRAM, and ECC functionality is active and working.
Secondly, it probably also means that they do not include tests for this functionality when they perform the final tests against each fully assembled chip. I'd expect that a jtag boundary scan does verify that the bond wires are in place and work, but no functional tests of ECC are run on each processor in the consumer configuration.
The net result is that with a compatible motherboard and memory, ECC almost certainly works (since the memory controller is the same as in the supported model) but AMD does not officially guarantee it. It is much like overclocking. The functionality is present, and it should work, and most likely does, but AMD accepts no responsibility if it does not, since they don't formally test for it.
I don't know where you live, but around here, (if you buy new?), the vendor MUST take back items up to 15 days after they were delivered, for ANY reason.
So, as long as you synchronize your buying of CPU, RAM, (motherboard), you should be fine.
We only use Xeons on developer desktops and production machines here precisely because of ECC. It's about 1 bit flip/month/gigabyte. That's too much risk when doing something critical for a client.
That's not true. There are Core i3, Atom, Celeron, and Pentium SKUs with ECC. E.g. the Core i3-9300
https://en.wikichip.org/wiki/intel/core_i3/i3-9300
That's an extreme claim. Why do you say so?
ECC is supported on most Ryzen models[1], as long as the motherboard supports it. In fact, ASUS and ASRock (possibly others) have Ryzen motherboards designed for workstation/server use where ECC support is specifically advertised.
[1] The only exception is the Ryzen CPUs with integrated graphics.
ECC is not disabled. It works, but not validated for our consumer client platform.
Validated means run it through server/workstation grade testing. For the first Ryzen processors, focused on the prosumer / gaming market, this feature is enabled and working but not validated by AMD. You should not have issues creating a whitebox homelab or NAS with ECC memory enabled.
https://old.reddit.com/r/Amd/comments/5x4hxu/we_are_amd_crea...
ECC support not being "validated," for all practical purposes, simply means that board vendors can advertise a board lacking ECC support as compatible with AMD's AM4 platform, without getting a nasty letter from AMD's lawyers.
However, I use only computers with ECC, previously only Xeons, but in the last years I have replaced many of them with Ryzens, all of which work OK with ECC memory.
When having to choose between a very small risk of losing the price of a CPU and having to use for sure, during many years, an Intel CPU with half of the AMD speed, the choice was very obvious for me.
The latter is a big problem, one of the extreme-OC guys (Buildzoid) who interacts frequently with the OEMs (as he is pushing their stuff to the limit and he frequently needs their help) has commented that AMD has a really bad problem with their BIOS teams. The AGESA firmware (the low-level code that the processor actually runs) is buggy as all hell at a firmware level and the OEMs are forced to patch around it in BIOS, but the AGESA firmware also has a massive problem with code churn, so these BIOS fixups basically stop working all the time. And the driver teams at a lot of OEMs are literally one person, so there isn't enough staffing there to test everything all the time. Long and short of it is: stuff breaks in AMD BIOSs, constantly, and they don't notice it.
This is obviously a huge problem when ECC is not an officially supported feature, because it means nobody is testing it! You might update your BIOS (as you frequently have to do with AMD machines) and suddenly ECC stops working, it might be running ECC in non-ECC mode and no longer correcting errors. Or it might have screwed up reporting them to the OS.
The server/workstation boards are the only ones you should be trusting Ryzen with ECC usage on.
LPDDR5 will enable some much needed level of error correction in a metric ton of other future SoC designs too. I look forward to the future Raspberry Pi with built in error correction capabilities.
Looks like `mcelog --client` might be a starting place? Feed that into your metrics pipeline and alert on it like anything else...
Otherwise, I would think that an unlikely event becoming 1000x more likely by sheer numbers would have warped your perception.
I believe that hardware reliability is mostly irrelevant, because software reliability is already far worse. It doesn't matter whether a bitflip (unlikely) or some bug (likely) causes a node to spuriously fail, what matters is that this failure is handled gracefully.
> It doesn't matter whether a bitflip (unlikely) or some bug (likely) causes a node to spuriously fail
Except that a bitflip can go undetected. It may crash your software or system, but it also may simply leak errors into your data, which can be far more catastrophic.
[1] https://news.ycombinator.com/item?id=25623206
In other words, if a single defective DIMM somewhere in your deployment is causing catastraphic failure, your mistake was not buying the wrong RAM modules. Your mistake was relying on a single point of failure for mission critical data.
This is, of course, an anecdote rather than data, but 0 is different enough from the expected 768 that it makes me doubt that statistic.
Google and read up - it is a problem, has killed people, has thrown election results, and much more.
It's such a common problem than bitsquatting is a real thing :)
Want to do an experiment? Pick a bitsquatted domain for a common site, and see how often you get hits.
https://en.wikipedia.org/wiki/Bitsquatting
As for the case of bitflips killing someone: Bitflips are not the root cause here. The root cause is that somebody engineered something life-critical that mistakenly assumed hardware can not fail. Bitflips are just one of many reasons for hardware failure.
So those systems didn't fail when a bitflip happened?
> The root cause is that somebody engineered something life-critical that mistakenly assumed hardware can not fail.
The systems I am aware of were designed with bitflips in mind. NO software can handle arbitrary amounts of bitflips. ALL software designed to mitigate bitflips only lower the odds via various forms of redundancy. (For context, I've written code for NASA, written a few proposals on making things more radiation hardened, and my PhD thesis was on a new class of error correcting codes - so I do know a little about making redundant software and hardware specifically designed to mitigate bitflips).
By claiming a bitflip didn't kick off the problems, and trying to push the cause elsewhere, you may as well blame all of engineering for making a device that can kill on failure.
So your argument is a red herring
>On the whole, you fail to make a case that preventing bitflips is the solution to a problem
Yes, had those bitflips been prevented, or not happened, those fatalities would not have happened.
>Ya, I'm not buying that biyflips are a problem.
If bitflips are not a problem then we don't need ECC ram (or ECC almost anything!) which is clearly used a lot. So bitflips are enough of a problem that a massively widespread technology is in place to handle precisely that problem.
I guess you've never written a program and watched bits flip on computers you control? You should try it - it's a good exercise to see how often it does happen.
I guess you define something being a problem differently than I or the ECC ram industry do.
I didn't say that. I'm saying that the root cause (as in "root cause analysis") is not the bitflip. Designating the bitflip as the root cause is like analyzing your drunk driving accident and concluding that the root cause must be ethanol, rather than your drinking habits.
> The systems I am aware of were designed with bitflips in mind. NO software can handle arbitrary amounts of bitflips. ALL software designed to mitigate bitflips only lower the odds via various forms of redundancy.
Of course, and I'm not actually arguing that adding in ECC is completely worthless to that effect, though it is close to worthless. Luckily, ECC is quite cheap, if not free, so throwing it in there makes sense.
However, suppose ECC would increase the cost by several magnitudes, would it still be worth it? Obviously not. Redundancy alone reduces the probability of spurious failure by several magnitudes, and simply increasing redundancy would be far cheaper than adding in ECC.
> If bitflips are not a problem then we don't need ECC ram (or ECC almost anything!) which is clearly used a lot. So bitflips are enough of a problem that a massively widespread technology is in place to handle precisely that problem.
My point is that bitflips either don't really matter, in case data integrity is not mission critical, or they don't actually solve the problem, in case data integrity is mission critical.
If you have solved the problem of data integrity through redundancy, then ECC doesn't make much of a difference anymore. If you haven't solved the problem, then ECC will only prevent a vanishingly small subset of disasters that are awaiting you.
> I guess you've never written a program and watched bits flip on computers you control? You should try it - it's a good exercise to see how often it does happen.
I don't care how often it happens. I care about the odds of a bitflip causing an actual problem. If a computer crashes, that's okay, it'll reboot. If any data were to be corrupted, it would most likely happen at the disk level and not the DRAM level.
> I guess you define something being a problem differently than I or the ECC ram industry do.
Of course, somebody who sells ECC RAM will want to convince you that ECC actually solves a real problem. The same can be said about the nutritional supplement industry, or many other industries that rely on make-belief.
Yes, that is clear.
> If you have solved the problem of data integrity...
As above, this is not a binary, black and white thing, but you keep presenting it as such. It's probabilistic, and higher protection is not free - the tradeoff is engineering.
> Redundancy alone reduces the probability of spurious failure by several magnitudes
ECC "alone reduces the probability of spurious failure by several magnitudes". That's why it is used.
Naive redundancy ignores almost a century of better method form forward error correcting codes. I have a feeling your idea of redundancy is having multiple exact copies of a system or data and having them vote, which is a terribly expensive way to do data protection when there are vastly better methods.
>Of course, somebody who sells ECC RAM will want to convince you that ECC actually solves a real problem. The same can be said about the nutritional supplement industry, or many other industries that rely on make-belief.
And we're done. If you don't think ECC helps a real problem then I see why you don't understand bitflip causing problems. Good luck.
The actual problem is binary. You either solved it, or you didn't. ECC is "free", but it doesn't actually solve the problem. Actually solving the problem requires engineering.
Of course there's a probabilistic element to it, but the problem is to drive the probability of failure to "vanishingly small". The utility of adding or removing a vanishingly small constant to another vanishingly small constant is vanishingly small. This is what ECC does for you.
> ECC "alone reduces the probability of spurious failure by several magnitudes". That's why it is used.
ECC reduces the probability of spurious failure due to bitflips in DRAM by several magnitudes. However, spurious failure can occur for so many more reasons that the bitflip issue becomes a vanishingly small part.
> I have a feeling your idea of redundancy is having multiple exact copies of a system or data and having them vote, which is a terribly expensive way to do data protection when there are vastly better methods.
As you know, having worked for NASA, this is the right choice under certain circumstances. If there are lives on the line and you have a choice between "not solving a problem" and "a terribly expensive solution", you should go with the latter.
> If you don't think ECC helps a real problem then I see why you don't understand bitflip causing problems.
ECC does not solve the problem of data integrity. If you actually solve the problem of data integrity, you will find that ECC becomes effectively redundant. Do we not fundamentally agree on this? If so, why not?
That's not to say ECC is entirely useless from an administrative standpoint. It makes DRAM bitflips one less thing to worry about. One less thing out of thousands of things. Commensurately, the cost of ECC in a given deployment, like its utility, is vanishingly small.
See also this comment above: https://news.ycombinator.com/item?id=25623764
I can't give my source, but its far higher than most people think. Just pay the money.
Nevertheless, anyone who uses the computer for anything else besides games or movie watching, will greatly benefit from having ECC memory, because that is the only way to learn when the memory modules become defective.
Modern memories have a shorter lifetime than old memories and very frequently they begin to have bit errors from time to time long before breaking down completely.
Without ECC, you will become aware that a memory module is defective only when the computer crashes or no longer boots and severe data corruption in your files could have happened some months before that.
For myself, this was the most obvious reason why ECC was useful, because I was able in several cases to replace memory modules that began to have frequent correctable errors, after many years with little or no errors, without losing any precious data and without downtime.
The libraries we maintain (1) are responsible for a non-trivial part of Facebook's overall compute footprint, (2) should basically never fail of their own accord, and (3) have pretty good error monitoring. So my team is operating what is effectively (among other things) a very sensitive detector for hardware failure.
And indeed we see examples all the time of blobs that fail to decompress, and usually when we dig in we find that the blob is only a single bit-flip away from a blob that decompresses successfully into a syntactically correct message. I can't share numbers, but, off the top of my head, I think it's the largest source of failures we see. It happens frequently enough that I wrote a tool to automate checking [0].
So yes. It happens. Pretty frequently, in the sense that if you're doing xillions of operations a day, a one-in-a-xillion failure happens all the time.
[0] https://github.com/facebook/zstd/tree/dev/contrib/diagnose_c...
I’ve always believed that, ECC aside, DRAM made intentionally with big cells would be less prone to spurious bit-flips (and that this is one of the things NASA means when they talk about “radiation hardening” a computer: sourcing memory with ungodly-large DRAM cells, willingly trading off lower memory capacity for higher per-cell level-shift activation-energy.)
If that’s true, then that would mean that the per-cell error rate would have actually been increasing over the years, as DRAM cell-size decreased, in the same way cell-size decrease and voltage-level tightening have increased error rate for flash memory. Combined with the fact that we just have N times more memory now, you’d think we’d be seeing a quadratic increase in faults compared to 40 years ago. But do we? It doesn’t seem like it.
I’ve also heard a counter-effect proposed, though: maybe there really are far more “raw” bit-flips going on — but far less of main memory is now in the causal chain for corrupting a workload than it used to be. In the 80s, on an 8-bit micro, POKEing any random address might wreck a program, since there’s only 64k addresses to POKE and most of the writable ones are in use for something critical. Today, most RAM is some sort of cache or buffer that’s going to be used once to produce some ephemeral IO effect (e.g. the compressed data for a video frame, that might decompress incorrectly, but only cause 16ms of glitchiness before the next frame comes along to paper over it); or, if it’s functional data, it’s part of a fault-tolerant component (e.g. a TCP packet, that’s going to checksum-fail when passed to the Ethernet controller and so not even be sent, causing the client to need to retry the request; or, even if accidentally checksums correctly, the server will choke on the malformed request, send an error... and the client will need to retry the request. One generic retry-on-exception handler around your net request, and you get memory fault-tolerance for free!)
If both effects are real, this would imply that regular PCs without ECC should still seem quite stable — but that it would be a far worse idea to run a non-ECC machine as a densely-packed multitenant VM hypervisor today (i.e. to tile main memory with OS kernels), than it would have been ~20 years ago when memory densities were lower. Can anyone attest to this?
(I’d just ask for actual numbers on whether per-cell per-second errors have increased over the years, but I don’t expect anyone has them.)
Think of the number of events that can flip a bit. If you make bits smaller, you get a modestly larger number of events in a given area capable of flipping a bit, spread across a larger number of bits in that area.
That is, it's flip event rate * memory die area, not flip event rate * number of memory bits.
In recent generations, I understand it's even been a bit paradoxical-- smaller geometries mean less of the die is actual memory bits, so you can actually end up with fewer flips from shrinking geometries.
And sure, your other effect is true: there's a whole lot fewer bitflips that "matter". Flip a bit in some framebuffer used in compositing somewhere-- and that's a lot of my memory-- and I don't care.
Screen, Wi-Fi, and to a much lesser extent (unless under load) the CPU are the most major culprits of low battery life.
https://media-www.micron.com/-/media/client/global/documents...
Now back to ECC, I'll probably be corrected, but I don't think ECC helps gain more than two order of magnitudes, so we still need incredibly reliable RAM. If we move to ECC RAM by default everywhere, aren't we simply going to get less reliable RAM at the end?
So I'd say ECC is not only important but insanely impactful. There's a reason why many organizations don't even want to hear about getting rigs with non-ECC memory.
I understand altitude has some kind of proportionality to cosmic ray exposure, and number of bits will multiply the probability of an error.. I'm presuming there is also an inherent error rate to DRAM separate from environment. But what are those numbers.
http://lambda-diode.com/opinion/ecc-memory#:~:text=A%20syste....
[edit]
Looks like the calculation was revised [0] after criticism:
> Under these assumptions, you'll have to wait about 33 to 600 days to get a 96% chance of getting a bit error.
What's more worrying is the variance, the above calculation is based on expected well behaved DRAM.. yet some computers just seem to have manufacturing defects that make the incidence of errors high enough to be a regular problem.
[0] http://lambda-diode.com/opinion/ecc-memory-2
I'm too lazy to run the exact numbers right now, but with "4 GB, 96% percent chance, three days" as the hypothesis, I think you'll find that an experimental result of "8 GB, 0% chance, 14 days" is highly statistically significant.
Edit: rough back of napkin estimate - you're not seeing an event in roughly 10x trials (2x number of bits and ~5x number of days). Given hypothesis is true your experimental result has a probability of (1-0.96)^10 = very very small. Conclusion: hypothesis is false.
There's a lot of variables that go into RAM errors, including manufacturing quality and condition of the ram, the dimm, the dimm slot, the motherboard generally, the power supply, the wiring, and the temperature of all of those. Google was known for cost cutting in their servers, especially early on; so I wouldn't be surprised if some of that resulted in higher bitflip rate than running in commercially available servers. Things like running bare motherboards, supported only on the edges cause excess strain and can impact resistance and capacitance of traces on the board (and in extreme cases, break the traces).
No it doesn't. You're assuming an even distribution of errors, which is very much not the case.
Google found that the average number of errors is around that range, but they also found that only one third of their servers had any errors in a year.
My point is, when you say there is a "96% chance of having an error in THREE DAYS", one would EXPECT to be having issues like.. all the time? So I'm not disagreeing with you, but with the amount of non-ECC machines all over the world and how insanely stable modern machines are, it still seems like a very low risk.
Now of course I agree that if you want to take every precaution, go ECC, but simple observation prove that this "problem" can't be as bad as the numbers are saying.
Most will escape your attention.
"33 to 600 days to get a 96% chance of getting a bit error." Still, it seems way too high. I guess anyone with ECC RAM could confirm that they are getting those sort of recovered error rates?
But you’ve got it backwards about the incentives. A manufacturer has less incentive to deliberately ship a defective part in the case of ECC modules. If the modules consistently log ECC errors, they can easily be identified and returned under warranty to the manufacturer. A consumer is much less likely to identify an intermittent problem with a non-ECC part.
I am trying to get a laptop with dual NVMe (for ZFS) and ECC RAM. I can't get that, at all - even without the other fancy things I would like such as a 4k OLED with pen/touchscreen.
In 2020, even the Dell XPS stopped shipping OLED (goodbye dear 7390!)
I will gladly give my money to anyone who sells AMD laptop with ECC. Hopefully, it will show there's demand for "high end yet non bulky laptops"
I hope AMD will create a better market for the ECC laptop memory (right now it's hard to find + expensive).
Unfortunately, Lenovo is not selling the P53 anymore, which is exactly why I say I can't get that even in a "bulky" version.
Basically you can register domains using small bit differences for domains and start getting email and such for that domain
If I recall correctly the example given was a variation of microsoft.com
All because so much equipment doesn't use ECC
At Google even with ECC everywhere there wasn't enough systematic error detection and correction to prevent the global database of monitoring metrics from filling up with garbage. /rpc/server/count was supposed to exist but also in there would be /lpc/server/count and /rpc/sdrver/count and every other thing. Reminded me daily of the terrors of flipped bits.
As long as we're swapping war stories there was once a frontend server at that company which died of bit flip related causes but just before it did it managed to charge some product group for 2^63 bytes worth of network traffic in the internal quota system, which set off every budgetary alarm that capacity planners had.
If you think it doesn't matter: how do you know? If you don't run with ECC memory, you'll never know if memory was corrupted (and recovered).
That blue screen, that sudden reboot, that program crashing. That corrupted picture of your kid.
Who knows.
I'll tell you, who knows. God damn every sysadmin (or the modern equivalent) can tell you how often they get ECC errors. And at even a small scale you'll encounter them. I have, on servers and even on an SAN Storage controller, for crying out loud.
If you care about your data, use ECC memory in your computers.
But.. in all my time operating servers over 3 decades, it's always been bad drivers, bad code and problematic hardware that's caused most of my headaches.
Have i seen ECC error correction in logs? yeah.. I don't advocate against it but, i've found for most people you design around multiple failure scenarios more than you design around preventing specific ones.
Take the average web app - you run it on 10 commodity systems and distribute the load.. if one crashes, so what. Chances are, a node will crash for many more reasons other than memory issues.
If you have an app that requires massive amounts of ram or you do put all of your begs in one basket, then ECC makes sense...
I just know i like going horizontal and I avoid vertical monoliths.
Crashes might not matter, but silent data corruption does. The owner/user of that data will care when they eventually discover that it at some point mysteriously got corrupted.
The real killer is data corruption. Houw would you even begin to know that data is corrupted until it is too late?
Again, I don't advocate NOT using ECC, but i'd say in complex systems, never assume ECC alone is enough... and if ECC becomes your champion cause, how could you enforce it through every device that touches data, provides data, consumes data or injects data?
How do you know?
It’s a tradeoff between money/performance and the frequency of crashes, corruption etc.
Bit rot is just one of many threats to my data. Backups take care of that as well as other threats like theft, fire, accidental deletion.
This is similar to my reasoning around the recent side channel attacks on intel CPUs. If I had a choice I’d like to run with max performance without the security fixes even though it would be less secure. Not because I don’t care about security but because 1% or 5% perf is a lot and I’d rather simply avoid doing anything security critical on the machine entirely than take that hit.
No, that's the big mistake people make: backups just backup bit-rotted data, until it is too late and the last good version is rotated out and lost forever.
My desktop machine is basically a gaming rig with disposable data. Hence the “performance over integrity”.
I also never rotate anything out. Every version of everything is in the backups. Storage is that cheap these days.
...unlike all the accountant machines which are just standard desktop grade ones that have invoices entered from their keyboards.
>Hence the “performance over integrity”.
If anything ECC would be better for performance, as it'd allow to clock memory higher. It's a mystery how Intel has managed to convince people ECC = enterprise/server market. The real cost of ECC is 1/8 more memory (and datatraces on the motherboard)... and if anything, virtually all intel cpus have the needed support/transistors in the memory controller, it's just desktop variants have that part fused off on purpose.
The current status is that it mostly doesn't exist outside OEM parts. Searching an online retailer for ECC brings me hundreds of results for "non-ECC" RAM. When they occasionally have 1-2 products with ECC, they aren't as aggressively binned as the non-ECC sticks. They'll have higher latencies and lower clock frequencies. Basically: you'll need to accept gambling on being able to overclock your ECC RAM.
The entire discussion is about why ECC is not common and why ECC matters. There is no technical reason for ECC to be uncommon aside purposed market segmentation.
Of course, there is no intrinsic availability of ECC udimms for the retail market currently, however that does mean ECC has no use or benefits for a small extra production cost.
[1]: https://twitter.com/catfish_man/status/1335373029245775872?l...
The industry has convinced the average user of consumer hardware that PPA (Power,Performance,Area) is all that needs to get better with generational improvements. Hoping that the concerning aspects of security and reliability that have come to light in the recent past changes this.
https://www.tomshardware.com/reviews/ecc-memory-ram-glossary...
When I first tried to replicate the row hammer attack I was not getting any results. Turns out I was doing this on ECC. On non ECC memory the same test easily replicated the row hammer attack.
https://en.wikipedia.org/wiki/Row_hammer
https://www.anandtech.com/show/15912/ddr5-specification-rele...
And now you have 8 bits of ecc per 32 data versus older DDR having 8 bits of ecc per 64 data. Hence the cost for dimm-wide ecc is going up.
DDR5 includes on-die ECC, where the RAM fixes the errors before sending them over the memory bus.
This means if the bus between the processor and ram corrupts the bits-- tough luck, they're still corrupted. And it's unclear whether we're going to get the quality of memory error reporting that we're used to or get the desired halt-on-non-recoverable error behavior (I've not been able to obtain/read the DDR5 specification as yet).
It seems redundant to have every module come with its own checking hardware.
For memory controller, parity/ECC/chipkill/RAIM usually involved simply adding additional memory planes to store correction data. I believe the rare exceptions are fully buffered memories where you have effectively separate memory controller on each module (or add-in card with DIMMs)
https://media-www.micron.com/-/media/client/global/documents...
When the value add feature becomes a necessity, it’s not a value add any more.
Of all the things to be worried about, like OS bugs, bad hardware configuration, etc. bad memory is one of those really troubling things. You look at the code and say "it's can't make it here, because this was set" but when you can't trust your memory you can't trust anything.
And as the timeline goes to infinity, you may also get one of these reports and be asked to fix it... good luck.
My old Honda crv however would turn traction control on if your pressure was low - which worked by applying brakes to wheels that were slipping. If you were going up a slippery hill you would soon have no power, sliding backwards nearly off the road in nowhere West Virginia on the way to a ski resort.
I live in Denver but spend a lot of time skiing around 11k feet, maybe the higher elevation means more radiation.
Aside: I'm surprised you got a TPMS programming tool instead of a set of steelies. Big wheels? Multiple winter vehicles?
Interestingly enough, average background radiation is actually the highest in the US exactly where you are[0] and I've seen comments (on a few websites talking about it) ranging from "the Rockies are loaded in Uranium" to "what about the Rocky Flats nuclear weapons site". Perhaps you're exactly in the right place to be an advocate for ECC :)
[0] http://www.radiationnetwork.com/
https://forums.factorio.com/viewtopic.php?p=405060#p405060
It is incomprehensible that there are still NAS devices being sold without ECC support.
Synology took a step in the right direction to offer prosumer devices with ECC but it is not really advertised as such. It is actually difficult to find which do have ECC and which ones don't.
I just look it up because if it was true it would have been news to me. Synology have been known to be stingy with Hardware Spec. But none of what I called Prosumer, the Plus Series have ECC memory by default. And there are "Value" and "J" Series below that.
Edit: Only two model from the new xx21 series using AMD Ryzen V has ECC memory by default.
- Edit -
Also, bit flips in the non-ECC memory are _the_ cause of the "bitrot" phenomenon. That is when you write out X to a storage device, but you get Y when you read it back. A common explanation is that the corruption happens _at rest_. However all drives from the last 30+ years have FEC support, so in reality the only way a bit rot can happen is if the data is damaged _in transit_, while in RAM, on the way to/from the storage media.
So, if you ever decide if to get an ECC RAM, get it. It's very much worth it.
Problems can definitely happen in the IO controller, RAID controller, cable, and disk controller. AFAIK all of these were seen and motivations for the existence of ZFS. One of their biggest insistence was that drives are universally lying bastards and should not be trusted any further than they can be thrown.
See "Your computer is broken". They essentially inserted a stress test into the game that verified if the hardware was still doing calculations correctly, and if not, inform the user.
(Note: EdDSA is still much much better than ECDSA, most notably because it's easier to implement correctly.)
But i don‘t know how relevant these metrics from 2009 are. Did memory got better or worse compared to 2009 for bit flips?
Not if you're using a typical 72-bit SECDED code[0].
You have two error indicators: a summary parity bit (even number of errors: 0,2,etc vs odd number of errors: 1,etc), and a error index: 0 for no errors, or the bitwise xor of the locations each bit error.
For a triple error at bits a,b, and c, you'll have summary parity of 1 (odd number of errors, assumed to be 1), and a error index of a^b^c, in the range 0..127, of which 0..71[1] (56.25%, a clear albeit not overwhelming majority) will correspond to legitimate single-bit errors.
0: https://en.wikipedia.org/wiki/Hamming_code#Hamming_codes_wit...
1: or 72 out of 128 anyway; the active bits might not all be assigned contiguous indexes starting from zero, but it doesn't change the probability and it's simpler to analyse if summary is bit 0 and index bit i is substrate bit 2^i.
Actually, a,b,c are also sampled from the range of valid bit indexes (not uniformly on 0..127), so you might be able to pick a cardinality-72 subset of 0..127 such that random a^b^c is disproportionately likely to fall outside that subset (and thus get diagnosed as not a valid single-bit error correction). I don't know that any existing ECC implementations actually do that, though.
Edit: did some cursory testing and using indexes 0..71 actually catches only ~24.04% (86016/357840) of triple-bit errors, compared the theoretical 43.75% (156555/357840, I think?) from a random error index. So "doesn't change the probability" is just completely wrong given that a,b,c are randomly chosen from the 72 substrate bits, rather than from 128 possible 7-bit indexes.
Oddly enough, testing random selections of 72 valid indexes (eg 74773982'EBD0D35C'C5BEB2D8'C3FE9C5E, where set bits correspond to used indexes) actually gives slightly better results than theory (44.97% for that one, 44.65% is the lowest in the last dozen or so), which is somewhat interesting, but I still haven't found any bit assignment that gives better than 50% (178920/357840) catchment of triple bit errors.
https://www.intel.com/content/dam/www/public/us/en/documents...
However I don't remember if there are provisions for ECC checking in case there are some dedicated refresh commands. I hope so, but I'm not sure.