>If an existing session has the same ID as the new number, WireGuard asks RDRAND for another "random" number, checks it for uniqueness, and so on. Since RDRAND on my system—and any non-microcode-updated Ryzen 3000 system—always returned 0xFFFFFFFF no matter what, that means infinite loop.
I'm pretty surprised that so much software seems to rely on external (be it from the CPU, or the OS) random number generation directly instead of simply using a software PRNG and seeding it with external sources (and maybe additional creative sources if desired). That seems like a way more predictable system to me, in the sense that at least you get guaranteed uniformly distributed numbers, even if the external seeds are not uniformly distributed at all.
My understanding is that implementing a PRNG in software results in a very small entropy pool. At the OS level, it can collect entropy from a vast number of sources, including things that an application won't have access to, which is why the OS exposes the RNG to applications. It also means maintaining such a PRNG in your software. Basically, it's the same old dependency argument again - there is nothing inherently wrong with using the tools the environment provides for you rather than building your own, but it does mean that the tools have to work properly. This is especially critical for cryptographically-secure PRNGs - those are things you really do not want to be maintaining yourself if you can access a high-quality source of random data, but again, if the PRNG doesn't work, you're in deep trouble (see for example Yubico's broken PRNG chips on the Yubikey 4). Hardware PRNGs on the CPU itself were supposed to dramatically improve the state of random data provisioning, but when bugs like this hit, it shows the weakness of depending on the stack beneath.
> This is especially critical for cryptographically-secure PRNGs - those are things you really do not want to be maintaining yourself
My conclusion would be exactly the opposite: If your RNG is so important that it has to be cryptographically secure, you owe it to your users to put in the time and effort of maintaining a proper implementation yourself, or at the very least use an open source library that provides this functionality in software. Otherwise you're always going to be at the mercy of a potentially misbehaving environment.
In terms of entropy, you don't really need to "maintain a pool" for CSPRNGs. You either have enough entropy to feed it with, or you don't. Once it is properly seeded, you can squeeze as many random bits out of it as you want (or at least, as many as anyone would ever reasonably need). It's really no different from a stream cipher, the key is the seed, and you're just encrypting zeroes. You don't need to suddenly get another randomly generated key after encrypting 100 MiB to encrypt the next 100 MiB securely.
Another great thing about entropy is that you can't reduce it. Which is why you really don't have to spend any time thinking about whether a particular entropy source is well behaved or uniformly distributed or anything like that at all. You just have to be certain that you have overall enough entropy that nobody can guess the entire seed. So anything the OS can give you? Dump it in there. Any kind of user interaction? Dump it in there. The time? CPU jitter? Network jitter? Just put it all in there. 100 MiBs of 0s? You know what why not, just put it on top because you literally can't make it worse, only better.
To be clear, you only need some random bytes to seed your cryptographic PRNG. This should of course be gathered from the OS but after that you only need to reseed once in a blue moon. Of course you shouldn't write and maintain a CPRNG yourself but there are many widely used, maintained and scrutinized libraries for this purpose.
For example, seeding ChaCha with 256bits will give you 1 ZiB of output before cycling. That should keep you going for awhile.
Because your intuition about the ease of building random number generators is wrong, especially if they need to be cryptographically secure. I trust my OS’ RNG functions[1] far more than I trust third party code to be right, and I trust it a million times more than I trust myself to get it right. Trusting a small core of validated code is far more likely to be correct in the long term.
Wireguard’s mistake was not using what the OS gave them; I’m not sure why they didn’t.
WireGuard is a kernel module, it already uses what the kernel gave it, which is get_random_u32(). Note that this is NOT used for anything crypto related, it's just used in the hashtable code.
No, building random number generators is easy; the reason arc4random was (it's been fixed) insecure was because arc4 was insecure.
If you have a secure stream cipher, you have a secure RNG. Not "can build", they're the same thing. If you only have block ciphers, the security proof is less trivial, but in practice "decrypt /dev/zero" still works.
If you don't have a secure (symmetric) cipher, you have bigger crypographic problems than bad random numbers.
And again, like before, the reaction to this is just hilarious to me. "It's a kernel bug." - Yeah, sure, but your users don't care, and your users shouldn't have to care. Because this an easy problem to fix, and if you're worth your salt as a developer you should fix it. Evidently Wireguard isn't worth it's salt.
That just changes the issue from "how do I get good quality random numbers" to "how do I get a good quality seed". Yes, less data is needed for just a seed, but this is not a perfect solution for every case either.
>That just changes the issue from "how do I get good quality random numbers" to "how do I get a good quality seed".
This is completely true, but there is a property that your random numbers have to have that your seed doesn't need in any way: Being uniformly distributed.
If you need a random 64 bit number, you want every single bit in that number to have a 50% chance of being 0 or 1. This requirement doesn't exist at all for seed data, the entropy of all your seed data together just needs to be large enough for it to be computationally infeasible to calculate it from the outside.
I would say that is a pretty big difference. Also, if you just need uniformly distributed random IDs but those IDs aren't secret and don't need to be unpredictable, you don't even need a seed. Or at least, you can just use a constant as a seed. No interaction with the outside world needed at all.
Depends on what you're looking for. Often you don't want 'guaranteed uniformly distributed numbers', you want random numbers. The reason for using hardware is that some processors can now provide greater (at least theoretically) entropy and/or greater performance in doing so than a software-only PRNG.
WireGuard uses the correct kernel function, get_random_u32(). WireGuard does not use RdRand directly. WireGuard instead uses the proper kernel function for asking for a unsigned 32-bit number. The kernel implements get_random_u32() under the hood using a variety of backends for it. One of them is RdRand.
Other facilities in the kernel, such as ASLR, also use get_random_u32().
Many things in the kernel use get_random_u32(). That's the proper function to use.
When presented with this bug, the upstream kernel maintainers chose not to fix get_random_u32(), due to the availability (?) of microcode updates for AMD chips. That's not my decision. WireGuard is just a mere consumer of get_random_u32(), like all other modules. This is an upstream kernel bug.
Wow that’s quite the bug. I’m surprised systemd uses RDRAND since I remember a big scare related to unauditable RNGs and government backdoors around the time that RDRAND and RDSEED came into existence. The thing is, they’re technically auditable. You just need an SEM and an astounding amount of time to reverse engineer a core to the point that you can identify and understand the RNG circuitry, beginning to end.
Edit: There’s some interesting discussion in the Wikipedia article’s talk page’s criticism section.
I always found that worry quite weird, these are processors that already have things like System Management Mode that takes control away from the kernel and runs some unknown code at an arbitrary point in the execution. Worrying about a subverted RNG and trusting the rest of the CPU will just create a load of busywork that does nothing to keep you more secure.
System Management Mode is the feature that does things like getting your fans to spin up when the processor runs hot. Pretty critical when you put it like that. Yes, that requires "tak[ing] control away from the kernel and running some unknown code at an arbitrary point in the execution". We could do away with such things entirely, but then our kernels would look like coreboot code, and a comparatively trivial kernel bug could permabrick your system. Not a very sensible tradeoff!
I think they're more worried about the untrusted/unauditable part of it. I suspect most people that dislike the ME would be perfectly fine with it if Intel released the code for it and allowed users to build and upload their own copy of the ME code.
CPUs have been executing arbitrary code that you didn't load for more than a decade now. All Intel CPUs run a ring -3 supervisor core based on Minix, on AMD it's based on ARM TrustZone.
If those supervisor cores want, they can literally just tamper with the entropy in your pool directly, they don't need to go through the charade of feeding it tampered entropy and then trying to un-mix it later.
> If those supervisor cores want, they can literally just tamper with the entropy in your pool directly
Go ahead and try to tamper with that pool. On all existing and future kernel versions. After undoing kASLR.
Not arguing, that it is impossible, but malware writers usually choose easier paths. When Samsung backdoored their phones [1], they didn't make a pure TrustZone rootkit. Instead they created a companion app, acting as helper for performing high-level tasks within OS bounds. Intel ME rootkit has it's own network stack and can use Intel network card, because it is both easier and safer than trying to interact with constantly updating OS, written by different people.
A rootkit, trying to perform any kind of complex interactions with host OS, may be exploited and taken advantage of, — for example see bugs [2] in AMD's PSP, that allowed host OS to take over by giving it specially crafted certificate.
The way that SystemD uses RDRAND is not necessarily security-critical. It is used to create seeds for hashing functions for hashmaps. If someone controls RDRAND, they might be able to do a denial of service attack, but that's about it.
I'm surprised WireGuard asks RDRAND directly. Isn't there a facility inside the Linux to get random numbers ?
OpenBSD conveniently provides arc4random() in its libc for applications to use, and the same function is available for kernel components (obviously one needs to include different headers).
Linux provides /dev/random and /dev/urandom to all applications as well as relevant API calls, but the quality of those is always in doubt (especially with urandom, which recycles the entropy pool when it gets too low). CPU hardware PRNGs are supposed to be much higher quality, when they work. So it's all about depending on the CPU to actually do what it's advertising - the CPUID flag advertises a high-quality hardware RNG, but then the firmware bug completely nurfs it.
> So it's all about depending on the CPU to actually do what it's advertising
I know I'm nitpicking but I disagree with that statement.
There's a kernel option called CONFIG_RANDOM_TRUST_CPU which you can set to false. So if for some reason, even if it's a bad one, you don't want your random numbers generated by your cpu then that's that. End of discussion. (In theory, not sure if rdrand if trapable)
I get that you're skeptical about the quality of what's provided in /dev/(u)random because in most cases it's true. Should I ever feel the need to hookup a hw random generator then I hope programs would use that one instead of guessing they can do better by calling rdrand.
I wasn't aware of this kernel flag (nor am I surprised it exists), seems like it would be useful to the article's author. However, by default it's left enabled, which allows the kernel to automatically set up the environment based on what it can glean from the hardware. So my comment was in regards to the machine without intervention.
From cursory look at the code wireguard relies on Linux's get_random_whatever() internal APIs, which are kernel-space interface to same PRNG as userspace (u)random/get_random()/get_entropy(). It even seems that there is not a single line of x86_64 assembly in the whole WireGuard.
On the other hand it is somewhat worrisome that brokenness of RDRAND can affect output of Linux's RNG that much.
Alright, so that would mean that it's not WireGuard that's broken because it's trying to be more picky about the RNG, but rather the Linux CSPRNG that's broken when RDRAND is misbehaving. As you said, worrisome.
The problem is that the "right kernel API" (get_random_u32) will return whatever the hardware-provided RNG returns if it's available[1].
There is assembly in WireGuard for the crypto primitive implementations, but those are generated by scripts and are based on either formally-proven implementations or highly-vetted ones.
WireGuard uses the correct kernel function, get_random_u32(). WireGuard does not use RdRand directly. WireGuard instead uses the proper kernel function for asking for a unsigned 32-bit number. The kernel implements get_random_u32() under the hood using a variety of backends for it. One of them is RdRand.
Other facilities in the kernel, such as ASLR, also use get_random_u32().
Many things in the kernel use get_random_u32(). That's the proper function to use.
When presented with this bug, the upstream kernel maintainers chose not to fix get_random_u32(), due to the availability (?) of microcode updates for AMD chips. That's not my decision. WireGuard is just a mere consumer of get_random_u32(), like all other modules. This is an upstream kernel bug.
So says a maintainer of WireGuard. HN is beautiful sometimes.
If RANDOM_TRUST_CPU is disabled, that will stop the kernel function from using RDRAND and avoid this issue for anyone using the ‘get_random_u32()’ function?
No. get_random_u32() simply returns result of RDRAND if RDRAND is available regardless of any runtime configuration. For me that is pretty significant issue, because it is documented as being based on separate kernel-only CSPRNG with somewhat specific security assumptions (complete with pretty large discussion in comments of random.c as to why would anybody want that weird thing)
If you know that get_random_u32() is catastrophically broken on some systems, it is your decision to not mitigate for that. Especially if your app's critical functionality relies on said function. At the very least the user should know their VPN is insecure or something.
As noted by others, linux does provide a similar function, get_random_u32(). However, it may depend exclusively on rdrand, instead of simply using rdrand as an input to another generator. (arc4random() always stirs and mixes, regardless of what hardware sources are available.)
It's interesting that it relies on thermal noise to generate random numbers. Would this mean, theoretically, that the more stable the thermals of the system are, the less entropy, and the less random the numbers generated?
Johnson Nyquist noise is measured from resistors, and is therefore present in every circuit. Shot noise however, is more evident in transistors.
------------
I don't know if the AMD circuit is Shot-noise or Johnson Nyquist noise. But its important to remember that there are many sources of noise / randomness in normal circuits.
The funny thing about true-random number generators: noise sources are all over the place! A beginner who plays with transistors will almost immediately "discover" a form of noise, and be forced to minimize it.
With regard to "true random" circuits (of which there are many, many different kinds), they all isolate a particular noise, and then amplify it. You have to isolate a particular form of noise if you want to get a good entropy estimation.
EDIT: It seems both shot noise and Johnson-Nyquist noise are white-noise, so maybe it doesn't matter. White noise + white noise should result in white noise, but its been a long time since I've taken this circuit class...
Thermal noise is a very high source of true randomness. Thermal heat, at absurd amplification levels, fundamentally is asking the question: "How fast are the molecules moving at this location at point in time?"
Which will affect the voltage and current on the line, which will affect the timing of the circuits, which will be amplified into uniform noise through some mechanism, generating true randomness. After all, we can't predict the location and velocity of molecules! Heisenberg uncertainty principle!
By thermal noise, they don't mean the temperature going up and down with CPU load. Thermal noise comes from atoms jiggling, where the average kinetic energy of an atom is proportional to absolute temperature. If you put a sensitive-enough amplifier across a resistor passing current, it will detect individual electrons emerging from the maze of charges they just went through. It is very, very random.
Yup, this killed a weekend of mine as well. I'm surprised this isn't a footnote of every conversation about AMD here. Luckily for me the fix was available from my motherboard manufacturer about a week after I bought the chip.
Also, the Ryzen 3 seems to be picky in dealing with RAM chips. I couldn't boot my computer with all slots in my motherboard filled. I ended up having to dial down the speed of my RAM in the BIOS as well as manually set some other settings.
Also, certain motherboards require an updated BIOS before it can boot off a Ryzen 3. In my case, I had the store I bought the chip from install the chip into the mobo so that they could demonstrate that it would boot up and I wouldn't have to return anything. It took them hours to figure this out and I had to tell them how to fix the problem after arriving because I had previously run across mention of the problem. The fix was to install a Ryzen 2, enter the BIOS, install the upgrade and then go ahead with installing the Ryzen 3 chip.
What a PITA that was. It put me off building my own system. Next time, I think I'll get a Dell or something.
I was under the impression that it was widely known that X470 motherboards would only support the newer Ryzen CPUs with a BIOS update. This is very often the case for motherboard chipsets that are forward-compatible with newer CPUs. As far as memory goes, you do have to pay attention to what memory the CPU supports, it's not like any DDR4 modules will _just work_ with any motherboard/CPU combo. I just built a ryzen desktop, and everything was pretty much turn-key, as I selected the parts which with my limited amount of knowledge had the highest probability of working together nicely.
This was in the Philippines. I wasn't surprised they ran into issues and I knew right away what the issue was.
I understand you can't just get any memory. The memory I purchased was reasonably within the required specs and a solid brand. The memory wasn't on the "tested list" but on a relatively new chip that's a small list and in the Philippines, I had a limited choice. This is also a wide problem as I found loads of people who had the same issues. I was able to fix it though, and I have my original RAM installed. If there's a secret sauce to picking RAM other than being on this tested list, it's not widely known. Just finding a thread where someone could explain the problem and the fix was difficult.
I guess this is no different from any newly released thing. Better to wait until the quirks get worked out.
In your case, I think it might very well just be the fact that this is a completely new platform and all the kinks haven't been straightened out just yet. And yeah, being in a smaller market must suck due to the limited choice :/
Having said that, getting ahold of the 3900x in the UK made me feel like I was too participating in a very limited market, situated somewhere in the middle of the Pacific ocean.
Out of curiosity, what speed were you trying to boot at? I had an issue getting 4 sticks to run at 3600 MHz 1:1. I got it running at 3200 but this wasnt because of finicky ram it was because memory controller couldn't handle 4 sticks at that high a speed. In hindsight this was even hardcoded into the bios as it tried to set FCLK at 1600 with 4 sticks when the XMP profile was loaded (3600), with only 2 sticks it auto loaded 1800 FCLK.
It's Kingston DDR4-2666 (8x4) and I was just trying to run as is. I'm in the Philippines and limited in what I could get my hands on. No boot with 4 slots filled. I could only boot with 2 slots filled. I had to do 2 things. I set the RAM to 2400 and I ran CPUZ, took down the timings from the SPD tab and then set the lowest common manually in the BIOS. I'm now able to boot normally most of the time and the system is solid. Occasionally the system won't POST, but a restart solves that.
That's still well below the 4-slot JEDEC max on Ryzen 3000 (2933Mhz), you should definitely be able to boot with that combination. Sounds like either faulty RAM or a bad memory controller, I would try to RMA the RAM or the CPU.
Many newer AM4 motherboards can update BIOS without a processor installed. I recently had to do the same processor shuffle with an older motherboard I purchased because I need legacy PCI slots. It wasn't too painful because the motherboard was a cheap open box special, most likely because someone else couldn't get it to run a newer Ryzen.
It causes systemd to blow up, but you could try doing your microcode update from initrd. Or just get rid of systemd, run your microcode update, and make some Halloween jello shots to take to the no-systemd party (it's going to be even crazier, this year).
If the kernel itself doesn't boot, installing the microcode package may not help. It only helps if the problem only occurs after the point at which the kernel loads the microcode. Hopefully that's early enough for most cases.
Indeed. At least the Intel driver does it so early in the boot process that the "microcode updated" message is literally the first line in dmesg on a couple of systems I've bothered to check.
He could boot. His problem was lockups after boot with wireguard directly calling RDRND & looping on non-random random numbers rather than using an API to get random numbers.
I believe AMD made it policy to stop providing those updates separately and for users to get them via BIOS updates, though I don't have a reference for it right now.
There appears to be an AMD microcode update [1] in the kernel's linux-firmware repo as recently as 1 week ago, so I don't think this is true? And it's an update for "processor family 17h", which is Zen/Zen2, so it's not like this is only for old CPUs.
The bug in question does seem to have stopped some Linux systems from booting, which would of course prevent the update.
Although in the author's case, he seems to be waiting on a BIOS fix because the microcode fix doesn't actually work. In the updates he mentions being told "...that the amd64-microcode package... would fix the issue. This, too, is not the case; the amd64-microcode and intel-microcode packages are both installed by default on all Ubuntu 19.10 systems, including the one I'm experiencing the RDRAND failure on."
It's good news that microcode fixes generally don't demand BIOS updates, but in practice this one seems to be getting fixed that way before it's fixed by the manufacturer.
I have a 3700x system (but running Windows) at home.
Check with the motherboard vendor first? My board¹ has received three or four updates since I got it (which .. is an entirely different can of worms and doesn't exactly inspire confidence yet). At least they do seem to propagate things "quickly" I guess..
The bug is in the initialization part of the microcode and can only be patched during CPU initialization by the bios. Early microcode loading by the kernel is too late.
In the worst-case scenario of a lack of motherboard-update, to make things work during boot and later in Linux would it be enough to disable RANDOM_TRUST_CPU in the kernel source config ( https://www.phoronix.com/scan.php?page=news_item&px=Linux-4.... ) (and/or using a kernel parameter in Grub), or would that make it unavailable only to the kernel itself but still available to other programs (e.g. systemd, wireguard, etc...)? If #2, are there any other parameters to completely disable it to then make programs fallback to kernel-based sources of entropy (disk activity, network, etc) to generate their random nbrs?
get_random_u32() which is the function in question uses RDRAND if ut is available regardless of what RANDOM_TRUST_CPU says. It is meant of fast source of “good enough entropy” for short lived secrets. Short circuiting to RDRAND arguably breaks the API contract, but when RDRAND actually produces random-ish values it is probably good enough.
>When there's a bug in the CPU microcode, you're at the mercy of your motherboard vendor to release a new system BIOS that will update it for you—you can't just go to some download link at AMD and apply a fix yourself.
I think this is the real kicker, and might represent one of the next major fronts in the security struggle. It's a little different from the debates happening right now about support periods, that at least has clear economic implications. It's one thing to argue about whether a product should still be supported at all. But it's quite another when something is being supported, and does in fact have a patch available, yet many owners still can not apply it anyway. That seems like an avoidable failure, and something worth considering legislation around. The industry could and should have more standardized methods and requirements to make sure that any patches that are created do make it out to product owners quickly and universally, there just hasn't been consistent motivation.
> When there's a bug in the CPU microcode, you're at the mercy of your motherboard vendor to release a new system BIOS that will update it for you—you can't just go to some download link at AMD and apply a fix yourself.
At least on Linux, that's not true. Intel publishes microcode updates on Github [1] (and distros package it) and AMD has it upstreamed in linux-firmware [2], so you don't have to rely on motherboard vendors at all.
Microsoft also distributes microcode updates through Windows Update. Although, on a second look, it seems like all the microcode updates they've made available were only security-related (Spectre and co) Intel patches.
Why would anyone use rdrand directly? Seems like user space applications should use getrandom() or /dev/urandom and the kernel should use rdrand as a complementary random number source in its random number generator.
No user space program should need to use rdrand directly at all.
I believe Wireguard is implemented in the kernel. I'd argue there should be a kernel-wide wrapper function for this sort of thing, or if that already exists, Wireguard should probably use it.
That doesn't explain why systemd uses it, of course.
WireGuard does use the right wrapper -- get_random_u32(). The issue is that the implementation will just use whatever the architecture-provided randomness source provides if it's available[1]. That's the real bug.
The problem with creating such wrapper function is that someone like systemd/Wireguard developers will doubtlessly exploit it to drain entropy pool (whatever that means), at which point kernel drivers may start locking up, waiting for more entropy to appear.
In comparison get_random_u32() is safe to call at any point — including early boot — and does not affect global entropy pool. At worst it may return low-quality numbers, but that can be easily fixed by running your own peudo-random generator on top of it (which is a good idea anyway because you don't want your kernel module to contend with other parties for RNG ownership).
Indeed, because RDSEED is actually what most people want anyway.
Its an assembly instruction that gets the job done. People should be expecting that the assembly instructions of their CPUs work as intended. No different than using AVX-intrinsics or hand-crafted assembly in x264 / x265 code.
In any case, RDSEED is the assembly instruction for gathering entropy (aka: setting a random number generator should use RDSEED), while RDRAND is an older assembly instruction for purely getting a cryptographic random number. Its slightly different amounts of entropy involved in RDSEED vs RDRAND. So this is a very subtle issue that requires a lot of understanding of the x86 assembly instruction set.
But if you understand these details, then by golly you should use the instructions!
systemd may be running in an environment where /dev/urandom isn't available, and getrandom() will either block or return nothing depending on whether the entropy pool has been initialised so you still need a fallback if you're working in the early boot process.
> When theRDRAND bug in Ryzen 3000 first surfaced back in June, Linux users widely reported that their entire Ryzen 3000-powered systems wouldn't boot. The failure to boot was due to systemd's use of RDRAND—and it wasn't systemd's first clash with AMD and a buggy random-number generator, unfortunately.
Then why are half the posts on Hacker News about how wonderful Ryzen is and how Intel is all but dead, etc.?
HN has a pretty vocal contingent that won't allow systemd on their systems, so they wouldn't have noticed?
Also, this issue was discussed here when it was noticed shortly after release. AMD acknowledged the issue and released a fix pretty quickly, subject to microcode updates don't necessarily make it out to all motherboards very rapidly.
WireGuard uses the correct kernel function, get_random_u32(). WireGuard does not use RdRand directly. WireGuard instead uses the proper kernel function for asking for a unsigned 32-bit number. The kernel implements get_random_u32() under the hood using a variety of backends for it. One of them is RdRand.
Other facilities in the kernel, such as ASLR, also use get_random_u32().
Many things in the kernel use get_random_u32(). That's the proper function to use.
When presented with this bug, the upstream kernel maintainers chose not to fix get_random_u32(), due to the availability (?) of microcode updates for AMD chips. That's not my decision. WireGuard is just a mere consumer of get_random_u32(), like all other modules. This is an upstream kernel bug.
I imagine, that ASLR uses get_random_u32(), because it is needed during earliest phases in boot, before random pool is initialized. It might not necessarily be the proper function for making kernel random numbers — rather such function might not exist in Linux yet.
Any machine that has a CCD image sensor has an excellent source of high-quality random noise, in the least-significant bit of each pixel.
Each pixel will have a bias, but x-oring lots of them together gets nice, unbiased random bits to stir into your pool.
If it doesn't have a CCD, maybe it has a microphone or ambient-light sensor. That yields fewer total bits, but usually enough.
Maybe it has two, or all three. The advantage of mixing randomness from multiple sources is that you don't need to trust them all. If one starts feeding you FFFF..., the output has just a little less entropy, not none.
My guess for why AMD produces FFFFF... is that a Spook Mode was activated by accident. If so, it was supposed to switch to that only on command from the "management engine", the wired-in exploit every big CPU has. Maybe somebody deep within AMD wanted to ensure that we would not trust our RDRAND instruction overmuch.
> Readers also suggested that if the amd64-microcode package were installed, it would fix the issue. This, too, is not the case; the amd64-microcode and intel-microcode packages are both installed by default on all Ubuntu 19.10 systems, including the one I'm experiencing the RDRAND failure on. I contacted AMD and asked representatives to check the status of that package and see if anything needs to (or can) be done about updating it.
Can anyone comment why the AMD microcode update doesn't seem to actually fix this bug? I've seen plenty of people commenting that you don't need a bios update and that it should, however the author seems to have tried that.
It's CPU initialisation microcode, it has to be put in a bios update for your motherboard. Specifically it's called AEGSA https://en.m.wikipedia.org/wiki/AGESA
Is there an upstream source for this article? There's no mention of the actual versions of the Linux kernel, linux-firmware, microcode (either bios or applied by the kernel), etc. Which makes it seem less thoroughly investigated since I could easily see Ubuntu having something outdated.
I just got my 3900x a couple week ago and I don't have this bug according to the test code provided in the article.
It was patched for almost all systems almost immediately like he said. I suspect the reason he still had the bug was due to the Asrock Rack X470D4U motherboard being an older generation board, and also a micro atx server board of all things. They might have only sold a couple hundred of them total in a niche like that so it's not too surprising it would be a low priority on the bios update list.
I also find it kind of funny that he calls Asrock Asus every single time he mentions them in the article. If he was trying to install an Asus bios on his Asrock mobo then he's got bigger problems than this bug.
You would definitely be fine as long as you're using an updated motherboard with at least AGESA 1003ABB, which would be any x570 board and likely the vast majority of x470s by now.
Hmm, I see 1.0.0.3abba was released for that board on 9-18-2019. That should include the fix, although I'm not an expert, maybe it isn't there for some reason?
My Lenovo E485 with Ryzen 2700U processor had the same issue - the BIOS update for that was provided in August, I believe, though I only installed it this past weekend.
It was a very frustrating thing to happen, since I was super excited to be back on AMD for my laptop.
I don't have any insights to add here but I did just build a new PC this weekend with an Asus X570 MoBo a 3900x, and 32GB of 3600Mhz memory w/ Ubuntu 19.04 and everything worked as expected. I haven't even updated the BIOS yet. I did spend all day Sunday getting the Nvidia/CUDA drivers configured for Tensorflow, but that's a different story.
I just hope people don't think that every system build will deal with these problems.
>When there's a bug in the CPU microcode, you're at the mercy of your motherboard vendor to release a new system BIOS that will update it for you—you can't just go to some download link at AMD and apply a fix yourself.
Since when? On Linux anyway microcode can be installed just fine as a package, often bundled in kernel updates.
"My CPU still thought 0xFFFFFFFF was the randomest number ever, always, no matter what."
"What if two years later, I was still vulnerable to stack-smashing that I shouldn't have been, due to ASLR that wasn't actually randomizing?"
An excellent point... IF ASLR is dependent on RDRAND, and IF RDRAND always returns 0xFFFFFFFF, THEN ASLR will not work, and addresses will be laid out in memory deterministically rather than randomly... which is a big no-no for security...
A future OS would check the RNG's it uses for any security feature, on startup, and upon failure would stop, log the problem, advise the user, and ask them how they want to proceed...
99% of users would click "Ignore and don't ask me again." If that option is not available they would tell their friends how stupid their OS was and that it ruined their weekend.
The OS needs to paper over known defects, if consumers are to be kept sane.
DOS had abort, retry, fail (https://en.wikipedia.org/wiki/Abort,_Retry,_Fail%3F) when it encountered an issue; most BIOSes will stop if they can't detect the keyboard or if a memory test failed (it typically can be overriden), Windows has an Advanced Boot Options screen that automatically comes up (such that the user can select "Safe Mode") if/when the computer crashed previously...
I'm not saying you're wrong; I'm just saying, hey, there's "prior art" for this... <g>
Too bad that all the benefits of running an open source OS are lost because it's too arduous for regular people to recompile their OS and software. At least the BSDs make it trivial. When you give up freedom for "ease of use", you get what people give you, and you have no real options.
I'm sorry, but I really don't feel bad for this person.
Also, it's disingenuous to say that only "some" (the author uses that word) motherboards have updated microcode. "Most" would be more accurate, and "pretty much all, with few exceptions" is actually closer to the truth. Even my el cheapo A320 chipset motherboard already has AGESA 1.0.0.3 ABBA.
140 comments
[ 3.2 ms ] story [ 230 ms ] threadI'm pretty surprised that so much software seems to rely on external (be it from the CPU, or the OS) random number generation directly instead of simply using a software PRNG and seeding it with external sources (and maybe additional creative sources if desired). That seems like a way more predictable system to me, in the sense that at least you get guaranteed uniformly distributed numbers, even if the external seeds are not uniformly distributed at all.
Also, it is very easy to shoot yourself in the foot when being "creative" with random numbers, especially in the domain of security.
My conclusion would be exactly the opposite: If your RNG is so important that it has to be cryptographically secure, you owe it to your users to put in the time and effort of maintaining a proper implementation yourself, or at the very least use an open source library that provides this functionality in software. Otherwise you're always going to be at the mercy of a potentially misbehaving environment.
In terms of entropy, you don't really need to "maintain a pool" for CSPRNGs. You either have enough entropy to feed it with, or you don't. Once it is properly seeded, you can squeeze as many random bits out of it as you want (or at least, as many as anyone would ever reasonably need). It's really no different from a stream cipher, the key is the seed, and you're just encrypting zeroes. You don't need to suddenly get another randomly generated key after encrypting 100 MiB to encrypt the next 100 MiB securely.
Another great thing about entropy is that you can't reduce it. Which is why you really don't have to spend any time thinking about whether a particular entropy source is well behaved or uniformly distributed or anything like that at all. You just have to be certain that you have overall enough entropy that nobody can guess the entire seed. So anything the OS can give you? Dump it in there. Any kind of user interaction? Dump it in there. The time? CPU jitter? Network jitter? Just put it all in there. 100 MiBs of 0s? You know what why not, just put it on top because you literally can't make it worse, only better.
For example, seeding ChaCha with 256bits will give you 1 ZiB of output before cycling. That should keep you going for awhile.
A lot of PRNG are now implemented as the output of stream ciphers or block cipher in counter mode:
* https://en.wikipedia.org/wiki/Fortuna_(PRNG)
So 128 bits is all that is needed to get going.
Re-key every so often to ensure forward security in case there is a kernel-level compromise.
With AES-NI instructions in most CPUs, several GB/s can be achieved.
https://wireguard.com
Wireguard’s mistake was not using what the OS gave them; I’m not sure why they didn’t.
[1]: https://man.openbsd.org/arc4random.3
If you have a secure stream cipher, you have a secure RNG. Not "can build", they're the same thing. If you only have block ciphers, the security proof is less trivial, but in practice "decrypt /dev/zero" still works.
If you don't have a secure (symmetric) cipher, you have bigger crypographic problems than bad random numbers.
And again, like before, the reaction to this is just hilarious to me. "It's a kernel bug." - Yeah, sure, but your users don't care, and your users shouldn't have to care. Because this an easy problem to fix, and if you're worth your salt as a developer you should fix it. Evidently Wireguard isn't worth it's salt.
This is completely true, but there is a property that your random numbers have to have that your seed doesn't need in any way: Being uniformly distributed.
If you need a random 64 bit number, you want every single bit in that number to have a 50% chance of being 0 or 1. This requirement doesn't exist at all for seed data, the entropy of all your seed data together just needs to be large enough for it to be computationally infeasible to calculate it from the outside.
I would say that is a pretty big difference. Also, if you just need uniformly distributed random IDs but those IDs aren't secret and don't need to be unpredictable, you don't even need a seed. Or at least, you can just use a constant as a seed. No interaction with the outside world needed at all.
Gee what if instead when it hit a threshold, it disabled the feature and called BUG()?
Other facilities in the kernel, such as ASLR, also use get_random_u32().
Many things in the kernel use get_random_u32(). That's the proper function to use.
When presented with this bug, the upstream kernel maintainers chose not to fix get_random_u32(), due to the availability (?) of microcode updates for AMD chips. That's not my decision. WireGuard is just a mere consumer of get_random_u32(), like all other modules. This is an upstream kernel bug.
Edit: There’s some interesting discussion in the Wikipedia article’s talk page’s criticism section.
https://en.wikipedia.org/wiki/Talk:RdRand#Criticism
We already know that parties like the NSA have intentionally backdoored RNGs that made it into production use.
If those supervisor cores want, they can literally just tamper with the entropy in your pool directly, they don't need to go through the charade of feeding it tampered entropy and then trying to un-mix it later.
It is far less risky detection wise to just tamper with the RNG directly.
Go ahead and try to tamper with that pool. On all existing and future kernel versions. After undoing kASLR.
Not arguing, that it is impossible, but malware writers usually choose easier paths. When Samsung backdoored their phones [1], they didn't make a pure TrustZone rootkit. Instead they created a companion app, acting as helper for performing high-level tasks within OS bounds. Intel ME rootkit has it's own network stack and can use Intel network card, because it is both easier and safer than trying to interact with constantly updating OS, written by different people.
A rootkit, trying to perform any kind of complex interactions with host OS, may be exploited and taken advantage of, — for example see bugs [2] in AMD's PSP, that allowed host OS to take over by giving it specially crafted certificate.
[1]: https://www.zdnet.com/article/backdoor-in-samsung-galaxy-dev...
[2]: https://seclists.org/fulldisclosure/2018/Jan/12
OpenBSD conveniently provides arc4random() in its libc for applications to use, and the same function is available for kernel components (obviously one needs to include different headers).
I know I'm nitpicking but I disagree with that statement.
There's a kernel option called CONFIG_RANDOM_TRUST_CPU which you can set to false. So if for some reason, even if it's a bad one, you don't want your random numbers generated by your cpu then that's that. End of discussion. (In theory, not sure if rdrand if trapable)
I get that you're skeptical about the quality of what's provided in /dev/(u)random because in most cases it's true. Should I ever feel the need to hookup a hw random generator then I hope programs would use that one instead of guessing they can do better by calling rdrand.
There's been a bit of drama around it:
* https://news.ycombinator.com/item?id=21114524 (Oct 2)
* https://news.ycombinator.com/item?id=21290306 (Oct 19)
On the other hand it is somewhat worrisome that brokenness of RDRAND can affect output of Linux's RNG that much.
There is assembly in WireGuard for the crypto primitive implementations, but those are generated by scripts and are based on either formally-proven implementations or highly-vetted ones.
[1]: https://elixir.bootlin.com/linux/v5.3.6/source/drivers/char/...
Other facilities in the kernel, such as ASLR, also use get_random_u32().
Many things in the kernel use get_random_u32(). That's the proper function to use.
When presented with this bug, the upstream kernel maintainers chose not to fix get_random_u32(), due to the availability (?) of microcode updates for AMD chips. That's not my decision. WireGuard is just a mere consumer of get_random_u32(), like all other modules. This is an upstream kernel bug.
So says a maintainer of WireGuard. HN is beautiful sometimes.
If RANDOM_TRUST_CPU is disabled, that will stop the kernel function from using RDRAND and avoid this issue for anyone using the ‘get_random_u32()’ function?
https://en.wikipedia.org/wiki/Johnson–Nyquist_noise
https://en.wikipedia.org/wiki/Shot_noise
Johnson Nyquist noise is measured from resistors, and is therefore present in every circuit. Shot noise however, is more evident in transistors.
------------
I don't know if the AMD circuit is Shot-noise or Johnson Nyquist noise. But its important to remember that there are many sources of noise / randomness in normal circuits.
The funny thing about true-random number generators: noise sources are all over the place! A beginner who plays with transistors will almost immediately "discover" a form of noise, and be forced to minimize it.
With regard to "true random" circuits (of which there are many, many different kinds), they all isolate a particular noise, and then amplify it. You have to isolate a particular form of noise if you want to get a good entropy estimation.
EDIT: It seems both shot noise and Johnson-Nyquist noise are white-noise, so maybe it doesn't matter. White noise + white noise should result in white noise, but its been a long time since I've taken this circuit class...
Which will affect the voltage and current on the line, which will affect the timing of the circuits, which will be amplified into uniform noise through some mechanism, generating true randomness. After all, we can't predict the location and velocity of molecules! Heisenberg uncertainty principle!
Also, the Ryzen 3 seems to be picky in dealing with RAM chips. I couldn't boot my computer with all slots in my motherboard filled. I ended up having to dial down the speed of my RAM in the BIOS as well as manually set some other settings.
Also, certain motherboards require an updated BIOS before it can boot off a Ryzen 3. In my case, I had the store I bought the chip from install the chip into the mobo so that they could demonstrate that it would boot up and I wouldn't have to return anything. It took them hours to figure this out and I had to tell them how to fix the problem after arriving because I had previously run across mention of the problem. The fix was to install a Ryzen 2, enter the BIOS, install the upgrade and then go ahead with installing the Ryzen 3 chip.
What a PITA that was. It put me off building my own system. Next time, I think I'll get a Dell or something.
I understand you can't just get any memory. The memory I purchased was reasonably within the required specs and a solid brand. The memory wasn't on the "tested list" but on a relatively new chip that's a small list and in the Philippines, I had a limited choice. This is also a wide problem as I found loads of people who had the same issues. I was able to fix it though, and I have my original RAM installed. If there's a secret sauce to picking RAM other than being on this tested list, it's not widely known. Just finding a thread where someone could explain the problem and the fix was difficult.
I guess this is no different from any newly released thing. Better to wait until the quirks get worked out.
Having said that, getting ahold of the 3900x in the UK made me feel like I was too participating in a very limited market, situated somewhere in the middle of the Pacific ocean.
Motherboard is TUF B450M-PLUS gaming.
Chip is Ryzen 3600.
That's just... not true. Is the article wrong, or was the rdrand issue actually bios-specific rather than in CPU microcode?
Edit: Whoever downvoted this - ucode update is supported by both Intel and AMD for quite a while. You certainly can just download it: https://github.com/platomav/CPUMicrocodes/tree/master/AMD
On Ubuntu, he'd want to use the amd64-microcode package: https://launchpad.net/ubuntu/+source/amd64-microcode
Which a live boot, you can install packages such as microcode updates.
You'll need to compile a kernel which does not depend on the RDRAND instruction as a source of randomness.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin...
Although in the author's case, he seems to be waiting on a BIOS fix because the microcode fix doesn't actually work. In the updates he mentions being told "...that the amd64-microcode package... would fix the issue. This, too, is not the case; the amd64-microcode and intel-microcode packages are both installed by default on all Ubuntu 19.10 systems, including the one I'm experiencing the RDRAND failure on."
It's good news that microcode fixes generally don't demand BIOS updates, but in practice this one seems to be getting fixed that way before it's fixed by the manufacturer.
That's as well my understanding of the situation => does anybody have more infos?
I was really about to upgrade from an i7-8700 to a Ryzen 3700X but after reading this I froze and now I feel totally insecure... .
EDIT: was answered by "iforgotpassword"
①: https://www.gigabyte.com/Motherboard/X570-GAMING-X-rev-10/su...
I think this is the real kicker, and might represent one of the next major fronts in the security struggle. It's a little different from the debates happening right now about support periods, that at least has clear economic implications. It's one thing to argue about whether a product should still be supported at all. But it's quite another when something is being supported, and does in fact have a patch available, yet many owners still can not apply it anyway. That seems like an avoidable failure, and something worth considering legislation around. The industry could and should have more standardized methods and requirements to make sure that any patches that are created do make it out to product owners quickly and universally, there just hasn't been consistent motivation.
At least on Linux, that's not true. Intel publishes microcode updates on Github [1] (and distros package it) and AMD has it upstreamed in linux-firmware [2], so you don't have to rely on motherboard vendors at all.
[1] https://github.com/intel/Intel-Linux-Processor-Microcode-Dat...
[2] https://git.kernel.org/pub/scm/linux/kernel/git/firmware/lin...
No user space program should need to use rdrand directly at all.
That doesn't explain why systemd uses it, of course.
[1]: https://elixir.bootlin.com/linux/v5.3.6/source/drivers/char/...
In comparison get_random_u32() is safe to call at any point — including early boot — and does not affect global entropy pool. At worst it may return low-quality numbers, but that can be easily fixed by running your own peudo-random generator on top of it (which is a good idea anyway because you don't want your kernel module to contend with other parties for RNG ownership).
Indeed, because RDSEED is actually what most people want anyway.
Its an assembly instruction that gets the job done. People should be expecting that the assembly instructions of their CPUs work as intended. No different than using AVX-intrinsics or hand-crafted assembly in x264 / x265 code.
In any case, RDSEED is the assembly instruction for gathering entropy (aka: setting a random number generator should use RDSEED), while RDRAND is an older assembly instruction for purely getting a cryptographic random number. Its slightly different amounts of entropy involved in RDSEED vs RDRAND. So this is a very subtle issue that requires a lot of understanding of the x86 assembly instruction set.
But if you understand these details, then by golly you should use the instructions!
Then why are half the posts on Hacker News about how wonderful Ryzen is and how Intel is all but dead, etc.?
Also, this issue was discussed here when it was noticed shortly after release. AMD acknowledged the issue and released a fix pretty quickly, subject to microcode updates don't necessarily make it out to all motherboards very rapidly.
Other facilities in the kernel, such as ASLR, also use get_random_u32().
Many things in the kernel use get_random_u32(). That's the proper function to use.
When presented with this bug, the upstream kernel maintainers chose not to fix get_random_u32(), due to the availability (?) of microcode updates for AMD chips. That's not my decision. WireGuard is just a mere consumer of get_random_u32(), like all other modules. This is an upstream kernel bug.
https://spectrum.ieee.org/computing/hardware/behind-intels-n...
https://www.hotchips.org/wp-content/uploads/hc_archives/hc23...
NB: RDSEED was added at a later date, some people didn't like that Intel used AES for conditioning, they wanted the raw bits.
Each pixel will have a bias, but x-oring lots of them together gets nice, unbiased random bits to stir into your pool.
If it doesn't have a CCD, maybe it has a microphone or ambient-light sensor. That yields fewer total bits, but usually enough.
Maybe it has two, or all three. The advantage of mixing randomness from multiple sources is that you don't need to trust them all. If one starts feeding you FFFF..., the output has just a little less entropy, not none.
My guess for why AMD produces FFFFF... is that a Spook Mode was activated by accident. If so, it was supposed to switch to that only on command from the "management engine", the wired-in exploit every big CPU has. Maybe somebody deep within AMD wanted to ensure that we would not trust our RDRAND instruction overmuch.
Can anyone comment why the AMD microcode update doesn't seem to actually fix this bug? I've seen plenty of people commenting that you don't need a bios update and that it should, however the author seems to have tried that.
( see the alt text on https://xkcd.com/221/ )
https://dilbert.com/strip/2001-10-25
I just got my 3900x a couple week ago and I don't have this bug according to the test code provided in the article.
I'm running:
> Archlinux linux-5.3.7
> Linux-firmware: 20191022.2b016af-1
> Microcode: v2.2 (patch_level=0x08701013)
> MB: ASUS PRIME x570-P. bios=1201
I also find it kind of funny that he calls Asrock Asus every single time he mentions them in the article. If he was trying to install an Asus bios on his Asrock mobo then he's got bigger problems than this bug.
EDIT: looks like it's out of beta now - awesome! Thanks.
It was a very frustrating thing to happen, since I was super excited to be back on AMD for my laptop.
There's a good related Dilbert too: https://dilbert.com/strip/2001-10-25
I just hope people don't think that every system build will deal with these problems.
Since when? On Linux anyway microcode can be installed just fine as a package, often bundled in kernel updates.
"My CPU still thought 0xFFFFFFFF was the randomest number ever, always, no matter what."
"What if two years later, I was still vulnerable to stack-smashing that I shouldn't have been, due to ASLR that wasn't actually randomizing?"
An excellent point... IF ASLR is dependent on RDRAND, and IF RDRAND always returns 0xFFFFFFFF, THEN ASLR will not work, and addresses will be laid out in memory deterministically rather than randomly... which is a big no-no for security...
A future OS would check the RNG's it uses for any security feature, on startup, and upon failure would stop, log the problem, advise the user, and ask them how they want to proceed...
The OS needs to paper over known defects, if consumers are to be kept sane.
I'm not saying you're wrong; I'm just saying, hey, there's "prior art" for this... <g>
I'm sorry, but I really don't feel bad for this person.
Also, it's disingenuous to say that only "some" (the author uses that word) motherboards have updated microcode. "Most" would be more accurate, and "pretty much all, with few exceptions" is actually closer to the truth. Even my el cheapo A320 chipset motherboard already has AGESA 1.0.0.3 ABBA.
> I'm sorry, but I really don't feel bad for this person.
And this is why Linux will never take off as a consumer OS.