This seems like deflection to me. There’s no reproducer, no technical info, no objective facts besides what is basically a meta study that lumps their code quality with intel’s hardware correctness as one metric.
If it’s as they say, switching to AMD shouldn’t lead to 100x fewer crashes, it should lead to no crashes.
Lumping both 13th and 14th gen hardware together makes it harder to take the claims as easily at face value. 13th gen have been around long enough that you’d think they’d be able to dig up a kernel mailing list bug report or some other corroboration of their accusation.
I clicked expecting basically an undocumented errata with specific, 100% reproducible instructions and a deep dive into the internal architecture changes that could have caused this. I’m tempted to flag the submission, honestly. Claims of such magnitude demand at least some baseline evidence.
All computers have a baseline crash rate due to environmental factors such as overheating, cosmic rays, low bit error rates of modern hardware, etc, etc. My 5950X has never crashed, but I bet over a large enough population the crash rate would be measurable and consistent from game to game.
I used to be a hard core Intel guy through my career in IT and systems, but man my AM4 has been a super great system, the 5600x was stable, then upgraded to the 5950x to extend the system and it made me a true AMD fanboy overnight.
If they were the only ones noticing, I might agree. But there are some groups I trust to know their hardware inside and out, and one of those is RadGames[1].
> If it’s as they say, switching to AMD shouldn’t lead to 100x fewer crashes, it should lead to no crashes.
No, this does not follow, at all. Few things in life are monocausal and systems (here the aggregate of hardware and software) tend to have more than one bug. If one bug dominates and you remove it then of course you're still left with residuals.
It doesn't have to be intel's fault though. It could be mobo vendors defaulting to unsafe settings.
> It doesn't have to be intel's fault though. It could be mobo vendors defaulting to unsafe settings
Sure. However workstation and server class motherboards have a much lower risk of not following Intel recommendations on voltages and clock speeds.
The error rate on the workstation boards are pretty high, on the order of a failure per week for 50% of workstations they were collecting telemetry from.
The strangest part of the whole thing is how they claim that the CPUs will work fine initially, but then degrade over time in a deterministic way. How is nobody else seeing this? Are these guys executing code paths that are normally not used? It sounds a bit like their code is buggy in a strange way that doesn't cause problems on AMD but does on later generation Intel, but that wouldn't explain why code would work initially but fail later unless maybe the problem was brought about by a microcode update from Intel?
I fully agree that demanding Intel recall all of the CPUs at this point is premature. They've done nowhere near enough research to pin the problem on the processor and not their code.
Well, ... they are. This has been being discussed for a while. It came up a bit ago, many people were seeing it in games with a weird error about low VRAM which apparently wound up being linked to Intel's 13th and 14th gen processors. Exactly why is anyone's guess.
Hmmm, I think this is unlikely. For one thing, most people's power supplies are quite wildly overspecced in gaming rigs since its genuinely pretty hard to find good small ATX PSUs these days. For another, even crummy low-quality PSUs tend to mostly work pretty well for a reasonably long time in most cases, so it'd be weird to see this so much in high-end gaming rigs with brand new CPUs.
I also don't know exactly what failure mode you are suggesting with this (voltage rail out of spec?) but I think unless it starts feeding too much voltage into the CPU it is probably unlikely to be able to cause damage. (Instability, sure.)
It also doesn't explain why Intel's 12th generation CPUs are still running strong, along with most of AMD's lineup, by comparison. Don't get me wrong, every SKU of CPU has its issues, but all of the major issues with other CPU lines we have explanations for; the issues with Intel Raptor Lake are thusfar unexplained. Intel has blamed motherboard manufacturers quite extensively, but so far the evidence of this is pretty uncompelling, because the failures can apparently also be seen in abnormally high numbers on motherboards that never had unreasonable configurations.
Having built a few power management circuits myself: this is not how PSUs degrade, at least not in the first 5 years, and generally not "ever" in most cases.
The PSU (notably in this case the M/B VRM, not the ATX PSU) can't really trigger brownouts. 1.05V is 1.05V, and if asked to do so, the PSU will deliver 1.05V. It will just get hotter while doing so over time (and specifically: capacitors degrading.) The voltage references are quite stable over time, and so are the window comparators that ensure "1.05V" is between, say, 1.035V and 1.065V. (Voltage references have essentially continuous engineering history back to the 50s≈60s, it's a "solved" problem.)
Varies by brand, some use better parts. I've definitely seen file servers that won't boot after 3 years, then I pull a few drives to lower the power needed and it boots. After another year or two I have to pull more drives.
Even with generously specced power supplies, server class hardware, redundant power supplies, and enough smarts to stage the drive spin ups to avoid a large peak inrush current when 16 drives try to spin up at once.
But GP's explanation still holds. The PSU will give the reference voltage, while being less and less efficient. Unable to deliver proper voltage (because it becomes less efficient with time), will not deliver any, until you remove the drives and it can deliver it? Did I misunderstand something ?
Now I'm thinking of these HPE over-over-over-specced PSUs I run and y'all got me worried.
Sort of. It's not just efficiency that's lost, but peak power. But if peak power was causing the CPU problems it wouldn't get better with a replacement CPU.
Sure power supplies provide accurate power ... till they can't. Unlike peak power that starts to degrade from new, the voltages will hold as long as the PS can manage it.
A bit sardonic, but if peak power is the problem, your PSU better be shutting off, since the alternative is that it'll catch fire at some point. (And it normally will shut off, since it has thermal protections. Those are boolean, not somehow lowering output power.)
And again, the ATX PSU doesn't really matter here, the M/B VRM does. Even if the ATX PSU has a failure mode where it delivers 10V instead of 12V — you can still make 1.05V out of 10V. Decoupling is spread all the way too, so short peaks don't explain a lot either.
Lastly, it's of course possible the VRM is misdesigned. But this issue is seen across mainboards from at least 3 vendors… what is the likelihood of all of them getting it wrong in a very similar way? The primary way for that to happen is if they are all implementing Intel's reference design, in which case it's Intel's fault again anyway and we've looped back to the origin of the discussion…
As for everyone getting the same thing wrong being unlikely, seeing how much 'inspiration' there is in the space, especially anything helping the race to the bottom on price... it's not unseen, as in a whole generation of motherboards caps blowing or leaking, bad TPM implementations... not saying your overall point is wrong but this part I don't know
The problem description says it happens gradually over time, but affects roughly 100% of the machines client and server alike. I have a hard time believing that everybody has underspecced and failing power supplies, especially on their servers.
that's just alderon_matt's assertion. wendell and GN have triaged this to approximately 10-25% of units, seems pretty consistent acros a whole bunch of different environments.
The alderon_matt thing is actually carefully worded to say affected machines invariably degrade... but they don't say how many machines are affected. It's deceptively worded, intended to be read the way you read it, but actually not saying that 100% of machines are affected either.
at the end of the day game developers are just people too, and they're equally frustrated with the process/not willing to cut intel slack. but like, this is shades of the Ryzenfall thing (which, remember, contrary to early takes did have several real severity 9.0 issues behind it, including a PSP jailbreak and a UEFI module signing bypass, as well as memory encryption bypass and some other goodies - AMD doesn't issue AGESA patches for "root password lets you do root things"). People have moved past the facts and into the "trying to generate noise to get attention" on levels that are not supported by the facts.
the Alderon Games assertion, specifically, is very aggressively/deceptively worded and is not supported by the broader facts, nor does he have any real idea what can or can't be patched. as wendell notes - all kinds of "unfixable" hardware bugs are indeed fixed in microcode, routinely. that's why it's there, because of the pentium FOOF bug.
this isn't to say there isn't a degradation problem, there clearly is something (or more than one something) going on. but the alderon games press release is very over-the-top compared to all the rest of the evidence. GN is pointing at 10-25% and this is very consistent across a number of different operators, boards, environments, etc. That's the number Dell is getting in their validation (depending on the department/how stringent they validate), that's the number Wendell is seeing in operation in datacenter error reports. And they're thinking possibly some kind of fab defect affecting those units, basically (oxidation of the vias).
Sure, that might explain it, except when people get replacements things are improved, but then start getting worse. Some even are on their 3rd CPU.
What's making news is that Intel is starting (in at least one case) to deny RMA replacements after the first, despite shipping defective CPUs as replacements.
> Are these guys executing code paths that are normally not used?
Yes: games are notorious for having poor multithreading and hitting one core with very high load. And unlike servers where you're rarely the only workload, the rest of the CPU will be rather idle on a desktop/gaming system. This pushes the CPU to a much more "imbalanced" mode (putting all the power and heat into a very small area) than is common elsewhere.
I have these issues on a dedicated Postgres server running Linux and literally nothing else. It’s not just games, it’s just a lot of gamers buy these chips.
> If it’s as they say, switching to AMD shouldn’t lead to 100x fewer crashes, it should lead to no crashes.
No?
There’s always multiple sources of crashes. Switching from Intel may eliminate 100% of one crash source, but not 100% of all crashes.
One of my favorite stories is that ArenaNet once got Guild Wars servers so stable that any crash could be reliably attributed to hardware failure. Usually memory IIRC.
Crashes caused by hardware failures look very different from crashes caused by a buggy codebase. 80% of code bugs will have very unisonous crash reports. The other 20% of code bugs — memory corruption and race conditions, notably — have more random crash reports, but are still identifiable as such. Random hardware failures generally cause crash reports that look like someone put your bits in a mixer and made a smoothie.
(NB: random hardware failures like we're talking about here. CPU design/logic erratas can and do look like code bugs; but that's not what's being discussed here.)
Yeah, I'm super curious here. It's a very bold claim. It's possible that they've got some kind of data race/memory barrier/other threading issue that's more likely to get tickled by Intel processors than AMD processors. Hell, it could even be a compiler bug that generates incorrect assembly that trips more often on Intel processors.
There is no such thing as "incorrect assembly", modern CPUs don't have undefined instructions like the 6502 did. Either it gets executed deterministically, or you get an #UD exception.
That's not quite true, though. While there are no singular undefined instructions, there are sequences of instructions that can result in undesired/undefined behavior. Read through the published errata for modern CPUs, there will be a large number of combinations of instructions/actions that should be avoided entirely or modified to avoid misbehavior.
> Read through the published errata for modern CPUs
I totally agree, and this entire HN post is about the fact that these Intel CPUs probably have some yet-undiscovered erratas. Best theory at this point is that it's power/thermal related though.
> there will be a large number of combinations of instructions/actions that should be avoided
As a matter of fact, reading those erratas is part of my job, and no, it is very rare for such erratas to have the effect you're implying. Rare enough that this issue that is seen here would already have been matched against an errata.
I don't see anything in Intel's current errata — other than their in-progress responses to these reports — that could explain this. Here's the errata sheet for 13th/14th gen: https://edc.intel.com/content/www/us/en/design/products/plat... … do you see anything?
If you want to be pedantic, how about "incorrectly translating code into assembly such that two statements that should be separated by a sequence point in the C memory model can interfere with each other, but the implementations before this generation of Intel processor wouldn't reorder microcode execution such that it was actually a problem in practice".
> statements that should be separated by a sequence point in the C memory model can interfere with each other,
That also doesn't exist on x86 since it is a TSO platform.
The other thing is that even in the extremely unlikely case that some new behavior in 13th/14th gen Intel CPUs is exposing a bug in either code or compiler that has just been lurking there for who knows long, and doesn't affect any other CPU at all…
… with the way the world works, it'd still be a CPU bug, even if there was a document somewhere that says the CPU is right. x86 is shoved down the compatibility road so far, it just doesn't matter what some paper says if you break compat. (For user space at least, kernel is not quite there.)
> Lumping both 13th and 14th gen hardware together makes it harder to take the claims as easily at face value. 13th gen have been around long enough that you’d think they’d be able to dig up a kernel mailing list bug report or some other corroboration of their accusation.
This is pretty damn reasonable actually. The 14th gen Intel desktop hardware is for all intents and purposes, essentially the same as the 13th gen hardware. It behaves nearly identically, it benchmarks nearly identically.
I agree there should be something more specific but in the resources links they provide there is an epic games instructions page [1] to do msinfo for people getting crashes for these specific CPUs, so its no just them saying it.
Based on Fortnite’s suggested fix in their help, it sounds like there there is some flaw in the Intel hardware that makes it crash later in life when using the shipping voltage, which is maybe its most important parameter. When that is forced to be higher the part gets much hotter and performance per watt declines.
You can't necessarily publish a 100% repro for glitchy CPUs. It's stochastic both for whether it happens and whether it's detectable or silent corruption. My glitchy 14900K fails with illegal instruction, or sometimes machine checks. But who can say what else is happening that wasn't detected? I can definitely tell you that at least my instance of the 14900K cannot complete a `bazel test` of Abseil without crashing.
The links that are included shed more light on things. The write-up linked is a little light on details, and I also expected an actual "flaw". What the other links show is a series of anecdotal reports that the k-series intel processors have been found by end-users to become unstable, and that setting more conservative power management settings in the BIOS helps these chips regain stability.
The argument is that Intel is selling chips that at their out-of-the-box recommended use patterns can wear out extra quickly, and by default are over-volted. So it's not a "flaw" so much as not having a great engineering safety margin for premature wear.
the same series of videos that launched this, also confirm that chips like 13700T are degrading (or at least crashing). So it’s not nearly so simple as “too much power/voltage” - this is happening in 35W chips.
There are at least 2 causes and failures modes, intel has already confirmed eTVB=off was a problem and could cause degradation due to excessive heat. The other suspect right now is the ring bus degrading, perhaps.
> This seems like deflection to me. There’s no reproducer, no technical info, no objective facts besides what is basically a meta study that lumps their code quality with intel’s hardware correctness as one metric.
It's just laziness. The RAD game tools statement/article[1] has much more detail. I'm not sure to what degree the people behind the current article verified that they're seeing the same issue, but there is definitely a issue (apparently power/scaling/temperature related).
I’ve had extensive problems with the 13th gen, nearly all of the reporting I’m seeing is what I’ve experienced. Including weird NVME issues that I couldn’t figure out until I saw reports of it being related to these issues.
> Lumping both 13th and 14th gen hardware together makes it harder to take the claims as easily at face value. 13th gen have been around long enough that you’d think they’d be able to dig up a kernel mailing list bug report or some other corroboration of their accusation.
First of all the 13th and 14th gen are nearly the same CPUs. Slight differences in clock, TDPs, and core counts, but all using the identical raptor lake cores. Intel's been widely ridiculed for calling them 14th gen when there's no difference in the cores.
There have been various posts about these stability problems going back months on numerous forums. Often things like BSOD, not enough VRAM errors, decompression errors, failure to load game assets, which looks like a NVMe error, etc.
Part of the reason it's flying under the radar is a combination of factors. It's not all chips, it's not a specific instruction, and the behavior changes over time by getting worse. Sadly if a game or OS crashes nobody is terribly surprised, which can make hardware problems less obvious.
From what I can tell there Intel clock/voltage/temp curves were pushed, trying to be competitive with AMD despite a disadvantage Intel has with their 10nm fabs. Not only does this cause errors, but it causes damage, so the error rate increases over time.
The best source found seems to be the telemetry built into unreal engine that provides an unbiased report on crashes (but not hangs, BSoD aren't reported). Sure some gamers overclock, have poor airflow, poorly applied headsink goo, etc. But I found the following pretty compelling since it's based on workstation hardware, without overclocking, using the W series workstation chipset not the Z series consumer chipset which allows overclocking:
In a test population of more than 210 W680-based systems, 47.1% of these systems experience at least one incident of instability over a 168 hour test window. This distribution is the same to within 0.4% between Asus brand W680 and Supermicro W680 based boards.
For people running these boards on the server side they actually are charging more because of support issues related to CPU replacements.
Certainly a failure of 50% per week per node is crazy high and unacceptable on workstation class hardware.
>If it’s as they say, switching to AMD shouldn’t lead to 100x fewer crashes, it should lead to no crashes.
As many others have pointed out, this is baloney. Computers are not perfect machines and even if they were, gathering aggregate user data is also imperfect.
> I clicked expecting basically an undocumented errata with specific, 100% reproducible instructions and a deep dive into the internal architecture changes that could have caused this.
I don't know why you would expect that. You know what an intermittent bug is, this feels like disingenuous reasoning. Furthermore, there are many other people who are noticing this, documenting it, and attempting to report on it.
Installed a BIOS update, and now I am using "Intel Default Settings" with the "Extreme" profile. I bought a 13900K because it was cheaper than a 13900, not because I needed to overclock. Hope this was helpful to someone else.
I have my 14900K power capped at 225W/125W using RAPL and that seems to have stabilized it. I don't game with it, but it was unstable under all-core C++ compilation workloads. With these settings it is essentially a 14900. While trying to stabilize it I wasted a lot of time, bought a new motherboard, and bought all new DIMMs. I feel that if Intel can fix it they should provide owners with working replacements, a refund, and a goodwill discount on a future Intel product.
Original Pentium, similar. I didn't remember the exact details, but they tried to make owners happy. After initially bungling the response, they replaced CPUs by mail and handed out rebate coupons for retail Intel junk.
This issue is specifically affecting Intel's K-series line, which are engineered to run up against the CPU's thermal limits. Basically, as long as there is load on the CPU and the CPU is running below 100°C, it keeps cranking up the frequency and voltage until it hits that barrier. I am quite sure that this behavior is what is causing CPU lifespan issues as pumping a theoretically unlimited amount of volts into a CPU can't be healthy. AMD is doing this performance hack on their newest AM5 Ryzen CPUs as well.
> Intel® Thermal Velocity Boost (Intel® TVB) is a feature that opportunistically and automatically increases clock frequency above single-core and multi-core Intel® Turbo Boost Technology frequencies based on how much the processor is operating below its maximum temperature and whether turbo power budget is available
TVB specifically is exclusive to the i9 product line, but K SKUs in the i7 line have also experienced a heightened occurrence of stability issues.
So you kind of have to go all in on Turbo boost 3.0 as the issue and then explain why the non-K variants of the i7 and i9 don't seem to have the same frequency of occurrence.
You may want to take a look at some of the sleuthing Wendell over on Level1Techs did.
He was able to identify a large group of CPUs used in W680 chipset motherboards (i.e. using stock Intel power curves, never seeing a lick of overclocking/overvolting) which exhibited issues.
It would seem enthusiast ricing is, at best, an aggravating factor to some other underlying cause.
It isn't "overclocking" if you conform to the Intel specification.
What you are getting at is Turbo boost, which is not a new thing. We can split hairs over the incremental enhancements to Turbo boost over the years of course.
Oodle, a very optimized texture compression library used by Unreal, noted some problems months ago.
It is a bit odd that after months of this being well-known problem Intel still hasn't released a root cause. Is it an instruction sequence? Is their code subtly wrong in a way that works sometimes on these chips but always on other chips? Are some processors defective?
I might be out in left field here but I think there may not be a person who currently fully understands the entire workings of modern intel chips who can then identify a root cause of the issue.
There may also be some AI chip design shenanigans going on as well, which would further exacerbate the human comprehension difficulty scale.
Wow, I would not make a public statement like this without narrowing down the issue and making a replication case.
Given what little insight they have now, the crash-triggering fault could just as likely be in the peculiarities of their own code, some flaw in their build toolchain, some flaw in a runtime library, etc
With all respect to the folks at this studio, this statement reads like a junior dev throwing up their arms and blaming others over a bug they can't wrap their head around.
Even if they have a hunch that it's a CPU bug and can't afford to prove it, a more professional approach would be to perform the hardware change to AMD and toss out a more curious and less authoritative statement -- "Anybody else seeing more crashes on these processors? Have you figured out what from?" vs "These are broken and should be recalled".
And there could certainly be a CPU issue here. My critique is mostly about them trying to make such a strong statement about it with little specific evidence, despite not needing to make a statement at all.
As someone who used to spec data center hardware I assure you that AMD is going to have a series of kick ass quarters. We’ve had a slight preference for Intel for a decade but this probably is enough to flip 50% of large scale cpu purchases for the next year.
We're not a huge company, but my employer has moved to only buying AMD a while back (a year ago or so, it was a gradual shift but it is complete now). They're just less (costly) hassle.
This was widely covered by the tech press back in April and was common knowledge among gaming enthusiasts much earlier. Intel has been working with motherboard suppliers to leave aggressive overclocking features turned on, and it's also pretty clear they're not correctly binning their chips. For a while now Intel hasn't been "innovating", just turning up the wick. They're clearly lagging in R&D while also trying to chase the very tippy-top of gaming benchmarks which makes no sense, even for world-class competitive FPS players.
Nvidia's been flat out telling people complaining about games crashing on 13th and 14th gen CPUs to 'talk to Intel':
If you overlooked Ryzen because of issues like memory latency in the early days, I encourage looking at them again. If you do your homework, you can find motherboards that support ECC ram to varying degrees (ie from "it'll run but with ECC disabled" to "error correcting but doesn't report it" to "full support, just not listed as such".) I believe AM4-socket ECC support is much more prevalent, if that's important to you.
Familiarize yourself with AMD's Precision Boost Overdrive, which can be used for overclocking and/or undervolting. Unlike AMD's graphics cards (which are space heaters, though the current 7xxx series is an improvement), Ryzen processors are fairly efficient in stock form; more so if you do even a bit of mild undervolting.
Don't waste your time with the stock coolers supplied with any of the chips. They'll keep the chip cool enough, but are noisy as hell compared to a well-performing $30 dual-fan air cooler (GamersNexus has found several in their reviews.)
Keep your BIOS up to date for AGESA (AMD microcode) improvements, and keep your AMD chipset drivers up to date as well.
they are using the FPU heavily - in my experience an FPU design can be prone to gnarly and unpredictable bugs that are related to certain combinations of bit patterns (eg hammering all ‘1’s to all ‘0’s) in successive pipe stages which my be issued in an out of order sequence - it is very hard to diagnose and correct such failures and impossible to screen for them on a production tester
> I clicked expecting basically an undocumented errata with specific, 100% reproducible instructions and a deep dive into the internal architecture changes that could have caused this.
That's a really high bar for a non-Intel investigation. The Pentium III 1.13GHz issue 24 years ago had 100% reproduceable instruction, but even that didn't include any deep dive into the internal architecture because that just isn't information that people have.
I've had this issue come up twice now on one server, at first i was perplexed as i simply couldn't imagine the nvme drive causing these errors. Could it be related to the 13th gen issues?
https://i.imgur.com/l6vQFum.jpeg
Not sure. But I have recently discovered that some NVMe drives do support self-tests and I was able to start and monitor the test from smartmontools, may be worth looking in to an extended self test.
89 comments
[ 3.2 ms ] story [ 85.1 ms ] threadIf it’s as they say, switching to AMD shouldn’t lead to 100x fewer crashes, it should lead to no crashes.
Lumping both 13th and 14th gen hardware together makes it harder to take the claims as easily at face value. 13th gen have been around long enough that you’d think they’d be able to dig up a kernel mailing list bug report or some other corroboration of their accusation.
I clicked expecting basically an undocumented errata with specific, 100% reproducible instructions and a deep dive into the internal architecture changes that could have caused this. I’m tempted to flag the submission, honestly. Claims of such magnitude demand at least some baseline evidence.
[1] https://www.radgametools.com/oodleintel.htm
No, this does not follow, at all. Few things in life are monocausal and systems (here the aggregate of hardware and software) tend to have more than one bug. If one bug dominates and you remove it then of course you're still left with residuals.
It doesn't have to be intel's fault though. It could be mobo vendors defaulting to unsafe settings.
Sure. However workstation and server class motherboards have a much lower risk of not following Intel recommendations on voltages and clock speeds.
The error rate on the workstation boards are pretty high, on the order of a failure per week for 50% of workstations they were collecting telemetry from.
I fully agree that demanding Intel recall all of the CPUs at this point is premature. They've done nowhere near enough research to pin the problem on the processor and not their code.
Well, ... they are. This has been being discussed for a while. It came up a bit ago, many people were seeing it in games with a weird error about low VRAM which apparently wound up being linked to Intel's 13th and 14th gen processors. Exactly why is anyone's guess.
https://www.tomshardware.com/pc-components/cpus/nvidia-blame...
Level1Techs is relatively well-trusted, and they seem to have some contacts that can also corroborate these issues.
https://www.youtube.com/watch?v=QzHcrbT5D_Y
500 watt power supply becomes less efficient over time, now it's not quite 500 watts, parts of the chip brown out. Undefined behavior.
I also don't know exactly what failure mode you are suggesting with this (voltage rail out of spec?) but I think unless it starts feeding too much voltage into the CPU it is probably unlikely to be able to cause damage. (Instability, sure.)
It also doesn't explain why Intel's 12th generation CPUs are still running strong, along with most of AMD's lineup, by comparison. Don't get me wrong, every SKU of CPU has its issues, but all of the major issues with other CPU lines we have explanations for; the issues with Intel Raptor Lake are thusfar unexplained. Intel has blamed motherboard manufacturers quite extensively, but so far the evidence of this is pretty uncompelling, because the failures can apparently also be seen in abnormally high numbers on motherboards that never had unreasonable configurations.
The PSU (notably in this case the M/B VRM, not the ATX PSU) can't really trigger brownouts. 1.05V is 1.05V, and if asked to do so, the PSU will deliver 1.05V. It will just get hotter while doing so over time (and specifically: capacitors degrading.) The voltage references are quite stable over time, and so are the window comparators that ensure "1.05V" is between, say, 1.035V and 1.065V. (Voltage references have essentially continuous engineering history back to the 50s≈60s, it's a "solved" problem.)
Even with generously specced power supplies, server class hardware, redundant power supplies, and enough smarts to stage the drive spin ups to avoid a large peak inrush current when 16 drives try to spin up at once.
Now I'm thinking of these HPE over-over-over-specced PSUs I run and y'all got me worried.
Sure power supplies provide accurate power ... till they can't. Unlike peak power that starts to degrade from new, the voltages will hold as long as the PS can manage it.
And again, the ATX PSU doesn't really matter here, the M/B VRM does. Even if the ATX PSU has a failure mode where it delivers 10V instead of 12V — you can still make 1.05V out of 10V. Decoupling is spread all the way too, so short peaks don't explain a lot either.
Lastly, it's of course possible the VRM is misdesigned. But this issue is seen across mainboards from at least 3 vendors… what is the likelihood of all of them getting it wrong in a very similar way? The primary way for that to happen is if they are all implementing Intel's reference design, in which case it's Intel's fault again anyway and we've looped back to the origin of the discussion…
https://www.youtube.com/watch?v=gTeubeCIwRw
similarly, the oodle maintainers only found it to affect a small % of machines (but reliably, of course).
https://news.ycombinator.com/item?id=39486930
The alderon_matt thing is actually carefully worded to say affected machines invariably degrade... but they don't say how many machines are affected. It's deceptively worded, intended to be read the way you read it, but actually not saying that 100% of machines are affected either.
at the end of the day game developers are just people too, and they're equally frustrated with the process/not willing to cut intel slack. but like, this is shades of the Ryzenfall thing (which, remember, contrary to early takes did have several real severity 9.0 issues behind it, including a PSP jailbreak and a UEFI module signing bypass, as well as memory encryption bypass and some other goodies - AMD doesn't issue AGESA patches for "root password lets you do root things"). People have moved past the facts and into the "trying to generate noise to get attention" on levels that are not supported by the facts.
https://www.youtube.com/watch?v=QuqefIZrRWc
the Alderon Games assertion, specifically, is very aggressively/deceptively worded and is not supported by the broader facts, nor does he have any real idea what can or can't be patched. as wendell notes - all kinds of "unfixable" hardware bugs are indeed fixed in microcode, routinely. that's why it's there, because of the pentium FOOF bug.
this isn't to say there isn't a degradation problem, there clearly is something (or more than one something) going on. but the alderon games press release is very over-the-top compared to all the rest of the evidence. GN is pointing at 10-25% and this is very consistent across a number of different operators, boards, environments, etc. That's the number Dell is getting in their validation (depending on the department/how stringent they validate), that's the number Wendell is seeing in operation in datacenter error reports. And they're thinking possibly some kind of fab defect affecting those units, basically (oxidation of the vias).
https://youtu.be/5KHCLBqRrnY?t=2280
What's making news is that Intel is starting (in at least one case) to deny RMA replacements after the first, despite shipping defective CPUs as replacements.
Yes: games are notorious for having poor multithreading and hitting one core with very high load. And unlike servers where you're rarely the only workload, the rest of the CPU will be rather idle on a desktop/gaming system. This pushes the CPU to a much more "imbalanced" mode (putting all the power and heat into a very small area) than is common elsewhere.
No?
There’s always multiple sources of crashes. Switching from Intel may eliminate 100% of one crash source, but not 100% of all crashes.
One of my favorite stories is that ArenaNet once got Guild Wars servers so stable that any crash could be reliably attributed to hardware failure. Usually memory IIRC.
(NB: random hardware failures like we're talking about here. CPU design/logic erratas can and do look like code bugs; but that's not what's being discussed here.)
I totally agree, and this entire HN post is about the fact that these Intel CPUs probably have some yet-undiscovered erratas. Best theory at this point is that it's power/thermal related though.
> there will be a large number of combinations of instructions/actions that should be avoided
As a matter of fact, reading those erratas is part of my job, and no, it is very rare for such erratas to have the effect you're implying. Rare enough that this issue that is seen here would already have been matched against an errata.
I don't see anything in Intel's current errata — other than their in-progress responses to these reports — that could explain this. Here's the errata sheet for 13th/14th gen: https://edc.intel.com/content/www/us/en/design/products/plat... … do you see anything?
That also doesn't exist on x86 since it is a TSO platform.
The other thing is that even in the extremely unlikely case that some new behavior in 13th/14th gen Intel CPUs is exposing a bug in either code or compiler that has just been lurking there for who knows long, and doesn't affect any other CPU at all…
… with the way the world works, it'd still be a CPU bug, even if there was a document somewhere that says the CPU is right. x86 is shoved down the compatibility road so far, it just doesn't matter what some paper says if you break compat. (For user space at least, kernel is not quite there.)
Edit: 14th gen top end parts are basically 13th gen parts with different binning and higher clocks.
https://www.youtube.com/watch?v=QzHcrbT5D_Y
This is pretty damn reasonable actually. The 14th gen Intel desktop hardware is for all intents and purposes, essentially the same as the 13th gen hardware. It behaves nearly identically, it benchmarks nearly identically.
[1] https://www.epicgames.com/help/en-US/c-Category_Fortnite/c-F...
He's reached out to folk who gather game telemetry and got some interesting data to play with.
Based on Fortnite’s suggested fix in their help, it sounds like there there is some flaw in the Intel hardware that makes it crash later in life when using the shipping voltage, which is maybe its most important parameter. When that is forced to be higher the part gets much hotter and performance per watt declines.
The argument is that Intel is selling chips that at their out-of-the-box recommended use patterns can wear out extra quickly, and by default are over-volted. So it's not a "flaw" so much as not having a great engineering safety margin for premature wear.
There are at least 2 causes and failures modes, intel has already confirmed eTVB=off was a problem and could cause degradation due to excessive heat. The other suspect right now is the ring bus degrading, perhaps.
It's just laziness. The RAD game tools statement/article[1] has much more detail. I'm not sure to what degree the people behind the current article verified that they're seeing the same issue, but there is definitely a issue (apparently power/scaling/temperature related).
[1] https://www.radgametools.com/oodleintel.htm
I’m done with Intel.
First of all the 13th and 14th gen are nearly the same CPUs. Slight differences in clock, TDPs, and core counts, but all using the identical raptor lake cores. Intel's been widely ridiculed for calling them 14th gen when there's no difference in the cores.
There have been various posts about these stability problems going back months on numerous forums. Often things like BSOD, not enough VRAM errors, decompression errors, failure to load game assets, which looks like a NVMe error, etc.
Part of the reason it's flying under the radar is a combination of factors. It's not all chips, it's not a specific instruction, and the behavior changes over time by getting worse. Sadly if a game or OS crashes nobody is terribly surprised, which can make hardware problems less obvious.
From what I can tell there Intel clock/voltage/temp curves were pushed, trying to be competitive with AMD despite a disadvantage Intel has with their 10nm fabs. Not only does this cause errors, but it causes damage, so the error rate increases over time.
The best source found seems to be the telemetry built into unreal engine that provides an unbiased report on crashes (but not hangs, BSoD aren't reported). Sure some gamers overclock, have poor airflow, poorly applied headsink goo, etc. But I found the following pretty compelling since it's based on workstation hardware, without overclocking, using the W series workstation chipset not the Z series consumer chipset which allows overclocking:
For people running these boards on the server side they actually are charging more because of support issues related to CPU replacements.Certainly a failure of 50% per week per node is crazy high and unacceptable on workstation class hardware.
>If it’s as they say, switching to AMD shouldn’t lead to 100x fewer crashes, it should lead to no crashes.
As many others have pointed out, this is baloney. Computers are not perfect machines and even if they were, gathering aggregate user data is also imperfect.
> I clicked expecting basically an undocumented errata with specific, 100% reproducible instructions and a deep dive into the internal architecture changes that could have caused this.
I don't know why you would expect that. You know what an intermittent bug is, this feels like disingenuous reasoning. Furthermore, there are many other people who are noticing this, documenting it, and attempting to report on it.
I've ran a 13900K from launch, no issues.
Intel has a Pretty Big Problem (by Level1Techs): https://www.youtube.com/watch?v=QzHcrbT5D_Y
Intel's CPUs Are Failing, ft. Wendell of Level1 Techs (by Gamers Nexus): https://www.youtube.com/watch?v=oAE4NWoyMZk
https://www.intel.com/content/www/us/en/support/articles/000...
So you kind of have to go all in on Turbo boost 3.0 as the issue and then explain why the non-K variants of the i7 and i9 don't seem to have the same frequency of occurrence.
He was able to identify a large group of CPUs used in W680 chipset motherboards (i.e. using stock Intel power curves, never seeing a lick of overclocking/overvolting) which exhibited issues.
It would seem enthusiast ricing is, at best, an aggravating factor to some other underlying cause.
If the latter, I am afraid I don't see the point you are hinting at and need more clarification.
What you are getting at is Turbo boost, which is not a new thing. We can split hairs over the incremental enhancements to Turbo boost over the years of course.
It is a bit odd that after months of this being well-known problem Intel still hasn't released a root cause. Is it an instruction sequence? Is their code subtly wrong in a way that works sometimes on these chips but always on other chips? Are some processors defective?
There may also be some AI chip design shenanigans going on as well, which would further exacerbate the human comprehension difficulty scale.
Given what little insight they have now, the crash-triggering fault could just as likely be in the peculiarities of their own code, some flaw in their build toolchain, some flaw in a runtime library, etc
With all respect to the folks at this studio, this statement reads like a junior dev throwing up their arms and blaming others over a bug they can't wrap their head around.
Even if they have a hunch that it's a CPU bug and can't afford to prove it, a more professional approach would be to perform the hardware change to AMD and toss out a more curious and less authoritative statement -- "Anybody else seeing more crashes on these processors? Have you figured out what from?" vs "These are broken and should be recalled".
Idiot: There's something wrong with this CPU
Midwit: Nooo it's never the CPU if you think it's the CPU you must be a junior dev
Genius: There's something wrong with this CPU
And there could certainly be a CPU issue here. My critique is mostly about them trying to make such a strong statement about it with little specific evidence, despite not needing to make a statement at all.
Nvidia's been flat out telling people complaining about games crashing on 13th and 14th gen CPUs to 'talk to Intel':
https://www.techspot.com/news/102611-nvidia-advises-crash-pr...
If you overlooked Ryzen because of issues like memory latency in the early days, I encourage looking at them again. If you do your homework, you can find motherboards that support ECC ram to varying degrees (ie from "it'll run but with ECC disabled" to "error correcting but doesn't report it" to "full support, just not listed as such".) I believe AM4-socket ECC support is much more prevalent, if that's important to you.
Familiarize yourself with AMD's Precision Boost Overdrive, which can be used for overclocking and/or undervolting. Unlike AMD's graphics cards (which are space heaters, though the current 7xxx series is an improvement), Ryzen processors are fairly efficient in stock form; more so if you do even a bit of mild undervolting.
Don't waste your time with the stock coolers supplied with any of the chips. They'll keep the chip cool enough, but are noisy as hell compared to a well-performing $30 dual-fan air cooler (GamersNexus has found several in their reviews.)
Keep your BIOS up to date for AGESA (AMD microcode) improvements, and keep your AMD chipset drivers up to date as well.
That's a really high bar for a non-Intel investigation. The Pentium III 1.13GHz issue 24 years ago had 100% reproduceable instruction, but even that didn't include any deep dive into the internal architecture because that just isn't information that people have.
I don't think I should have issues?