"If N = 300, even a 256-bit seed arbitrarily precludes all but an unknown, haphazardly selected, non-random, and infinitesimally small fraction of permissible assignments. This introduces enormous bias into the assignment process and makes total nonsense of the p-value computed by a randomization test."
The first sentence is obviously true, but I'm going to need to see some evidence for "enormous bias" and "total nonsense". Let's leave aside lousy/little/badly-seeded PRNGs. Are there any non-cryptographic examples in which a well-designed PRNG with 256 bits of well-seeded random state produces results different enough from a TRNG to be visible to a user?
I think your intuition comes from the assumption that the experimental subjects are already coming to you in a random order. If that's the case, then you might as well assign the first half to control and the second half to treatment. To see the problem with poor randomization, you have to think about situations where there is (often unknown) bias or correlations in the order of the list that you're drawing from to randomize. Say you have an ordered list of 10 numbers, assigned 5 and 5 to control and (null) treatment groups. There are 252 assignments, which in theory should be equally likely. Assuming they all give different values of your statistic, you'll have 12 assignments with p <= .0476. If, say, you do the assignment from ~~a 256~~ an 8 bit random number such that 4 of the possible assignments are twice as likely as the others under your randomization procedure, the probability of getting one of those 12 assignments something between .0469 and .0625, depending whether the more-likely assignments happen to be among the 12 most extreme statistics, which is a difference of about 1/3 and could easily be the difference between "p>.05" and "p<.05". Again, if you start with your numbers in a random order, then this doesn't matter -- the biased assignment procedure will still give you a random assignment, because each initial number will be equally likely to be among the over-sampled or under-sampled ones.
Also worth noting that the situations where this matters are usually where your effect size is fairly small compared to the unexplained variation, so a few percent error in your p-value can make a difference.
It reminded me an experiment where each subject was presented with a pseudorandomised sequence of trials, only that, unknown to the researchers, every time the experiment was running the same (default) seed was used, which resulted in all subjects being presented to the same "random" sequence of trials.
I'm no statistician, but the part about halfway through that says not to use PRNGs for random assignment into bins seems wrong to me?
Sure I can understand why for a research trial you might want just want to be totally safe and use a source of true randomness, but for all practical purposes a decent PRNG used for sorting balls into buckets is totally indistinguishable from true randomness is it not?
I was half expecting this to have been written a few decades ago when really bad PRNGs were in common usage, but the article seems to be timestamped 2025.
Yes, there are theoretical issues with assuming PRNGs are truly random. However, there are also theoretical issues with assuming that Newton's law of universal gravitation is true.
I am confident is saying that more experiments have gone wrong due to not considering relativity, than have gone wrong due to the proper usage of a statistically sound (even if not cryptographically so) PRNG.
I also suspect that both classes of errors are dwarfed by improper usage or non sound PRNGs.
Newton’s gravity vs Relativity is a matter of precision. Newtonian mechanics is a limiting case of general relativity that works excellently within known bounds. PRNGs, by contrast, can fail categorically, not just in precision. A PRNG with subtle correlations doesn’t just give you a slightly less accurate answer, it can produce systematically biased results that look perfectly fine until they don’t. The failure modes are qualitatively different.
The observation that it is impossible to ennumerate 300 bits of assignments with a 256-bit random generator is a very clever observation. Just like almost all assignments are balanced (300 vs. 295 bits), I wonder if there's a similar equipartition property argument that 256 bits will generate the vast majority of the outcomes, but my Shannon-fu is too weak to tell.
Maybe that's the problem? With 256 bits we will only get the "typical" assignments and not the edge cases which are the ones that are important for randomisation tests?
I read this and came away a bit sheepish not really grasping the significance of extreme focus on PRNG and entropy for basic things. Glad to see the rest of the comments agreeing. "What every experimenter must know"...
One advantage of PRNG assignment is that you can reproduce your assignment procedure whereas this is impossible with true RNG. If you use true RNG how could you ever prove to anyone that that was what you actually did?
The thesis of this piece is effectively "causality is the inevitable destination of science" paired with the corrolary that "correlation is not causation". Statistics is a field with very limited insights. Once you apply causality to the mathematics of statistics (i.e. Pearl's do-calculus) you can actually make claims about the informational content expressed through recording events and facts of reality.
> After a proud surgeon delivered a lecture on a procedure he’d performed many times, a medical student asked whether half of eligible patients had been set aside as a baseline for comparison.
> “Of course not!” thundered the surgeon. “That would have doomed half of the patients to death!”
> Stunned silence filled the lecture hall, and the student softly asked,
I’m pretty sure that one of the fundamental premises of this argument is not correct, but I haven’t done mathematical statistics so I would defer to anyone who has and says I’m wrong. The author expresses this in a variety of forms throughout the article, but for instance
> Significance tests are meaningful and valid if AND ONLY IF assignment was
properly randomized.
OK I agree with the “if” but not the “only if” part of this implication. My intuition is that significance tests are still meaningful if a non-random method of assignment is chosen, but the bias introduced by the method is not in any way correlated with the observations under study.
This leads to the whole thing the author gets into about PRNGs not being valid for assignment. But a much simpler and more deterministic method of sampling is stratified sampling, which is used all over the place for various types of statistical experiments. The random entropy of a stratified sample is nowhere near enough to be fully random, yet I haven’t seen anyone claim that all experiments done using stratified sampling have invalid or meaningless p-values.
Basically stratified sampling works as follows. Say you want to study the effects of age and gender on income. One way is to just study everyone, record their age, gender and income and do your study. But that gets pretty unweildy and you may not be able to get accurate data over the whole population. What you can do instead is get a list of people, sort them by age and gender and pick a size that’s convenient. So say you have capacity to analyse a sample 1/20th the size of the total. Cool. Then you just take every 20th person on your list and that’s your sample. By the magic of stratified sampling, because your list is sorted by age and gender, the sample will more or less have the same proportions as the total population with respect to age and gender.
Not random, but used all over the place for statistical studies.
It should be noted that while the article recommends TRNGs with radioactive substances, those are completely unnecessary.
Any high-gain electronic amplifier whose input is connected to a resistor, or for a higher signal level, to a diode, will produce copious amounts of random noise at its output. If the output is converted with a comparator to digital, you have a good source of random bits.
Older people can remember the random audio or video noise of ancient radio receivers or TV sets, when they were tuned outside a correct channel.
In the past, there were easily available analog TV tuners for PCs, which could be used as noise sources. Nowadays, it is still possible to use the microphone input of a PC and connect to it an external analog noise source with a negligible cost, made on a little PCB with a small amplifier IC, e.g. an operational amplifier or an audio amplifier. The microphone input of PCs provides a weak 5 V supply, which would be enough to power a noise source.
The only problem that exists with any analog noise source is that the imperfections of analog-to-digital conversion, e.g. the fluctuations of comparator thresholds, due to temperature or age, can cause a bias in the random bits, so they do not have a truly uniform distribution. This problem also exists with radioactive sources.
Thus the bits must be processed to remove any bias. There are more sophisticated methods, but even the brute force method of using a one-way hash function, e.g. one of the SHA-3 or SHA-2 variants, to hash the bits and produce a uniform random number, is good enough.
Most modern CPUs, since Intel Ivy Lake, have instructions for providing true random numbers. However those are less useful outside their main intended application, of providing temporary session keys for TLS or other network protocols.
Some CPUs, especially from AMD, had bugs in their RNG, so they provided bad values. Even where the TRNG is good, the output passes through AES-128. Its state is small so you may encounter the problems mentioned in the parent article, of unreachable parts of the solution space that you are investigating in some Monte Carlo simulation.
It is preferable to be able to choose yourself the bias-removing method, to be able to use a hashing method with a much greater state.
What I personally would like to see is some kind of quantization of how the biases that the author talks about (such as insufficient seed volume of a PRNG) affects computed p-values. Specifically, why there must no "cancellation of errors" happen? So far, IIUC, the author only shows theoretical possibility of errors, but what's more interesting is a real effect. When it all boils down to a p-value being less than a certain threshold (choosing which is another pita), it might not matter whether a true p-value is within, say, 2^-16 from the computed.
18 comments
[ 20.2 ms ] story [ 102 ms ] threadIt's a lot easier to use diodes (light emitting and otherwise).
The first sentence is obviously true, but I'm going to need to see some evidence for "enormous bias" and "total nonsense". Let's leave aside lousy/little/badly-seeded PRNGs. Are there any non-cryptographic examples in which a well-designed PRNG with 256 bits of well-seeded random state produces results different enough from a TRNG to be visible to a user?
Also worth noting that the situations where this matters are usually where your effect size is fairly small compared to the unexplained variation, so a few percent error in your p-value can make a difference.
Sure I can understand why for a research trial you might want just want to be totally safe and use a source of true randomness, but for all practical purposes a decent PRNG used for sorting balls into buckets is totally indistinguishable from true randomness is it not?
I was half expecting this to have been written a few decades ago when really bad PRNGs were in common usage, but the article seems to be timestamped 2025.
Yes, there are theoretical issues with assuming PRNGs are truly random. However, there are also theoretical issues with assuming that Newton's law of universal gravitation is true.
I am confident is saying that more experiments have gone wrong due to not considering relativity, than have gone wrong due to the proper usage of a statistically sound (even if not cryptographically so) PRNG.
I also suspect that both classes of errors are dwarfed by improper usage or non sound PRNGs.
Maybe that's the problem? With 256 bits we will only get the "typical" assignments and not the edge cases which are the ones that are important for randomisation tests?
> “Of course not!” thundered the surgeon. “That would have doomed half of the patients to death!”
> Stunned silence filled the lecture hall, and the student softly asked,
> “Which half?”
damn that was good
> Significance tests are meaningful and valid if AND ONLY IF assignment was properly randomized.
OK I agree with the “if” but not the “only if” part of this implication. My intuition is that significance tests are still meaningful if a non-random method of assignment is chosen, but the bias introduced by the method is not in any way correlated with the observations under study.
This leads to the whole thing the author gets into about PRNGs not being valid for assignment. But a much simpler and more deterministic method of sampling is stratified sampling, which is used all over the place for various types of statistical experiments. The random entropy of a stratified sample is nowhere near enough to be fully random, yet I haven’t seen anyone claim that all experiments done using stratified sampling have invalid or meaningless p-values.
Basically stratified sampling works as follows. Say you want to study the effects of age and gender on income. One way is to just study everyone, record their age, gender and income and do your study. But that gets pretty unweildy and you may not be able to get accurate data over the whole population. What you can do instead is get a list of people, sort them by age and gender and pick a size that’s convenient. So say you have capacity to analyse a sample 1/20th the size of the total. Cool. Then you just take every 20th person on your list and that’s your sample. By the magic of stratified sampling, because your list is sorted by age and gender, the sample will more or less have the same proportions as the total population with respect to age and gender.
Not random, but used all over the place for statistical studies.
Any high-gain electronic amplifier whose input is connected to a resistor, or for a higher signal level, to a diode, will produce copious amounts of random noise at its output. If the output is converted with a comparator to digital, you have a good source of random bits.
Older people can remember the random audio or video noise of ancient radio receivers or TV sets, when they were tuned outside a correct channel.
In the past, there were easily available analog TV tuners for PCs, which could be used as noise sources. Nowadays, it is still possible to use the microphone input of a PC and connect to it an external analog noise source with a negligible cost, made on a little PCB with a small amplifier IC, e.g. an operational amplifier or an audio amplifier. The microphone input of PCs provides a weak 5 V supply, which would be enough to power a noise source.
The only problem that exists with any analog noise source is that the imperfections of analog-to-digital conversion, e.g. the fluctuations of comparator thresholds, due to temperature or age, can cause a bias in the random bits, so they do not have a truly uniform distribution. This problem also exists with radioactive sources.
Thus the bits must be processed to remove any bias. There are more sophisticated methods, but even the brute force method of using a one-way hash function, e.g. one of the SHA-3 or SHA-2 variants, to hash the bits and produce a uniform random number, is good enough.
Most modern CPUs, since Intel Ivy Lake, have instructions for providing true random numbers. However those are less useful outside their main intended application, of providing temporary session keys for TLS or other network protocols.
Some CPUs, especially from AMD, had bugs in their RNG, so they provided bad values. Even where the TRNG is good, the output passes through AES-128. Its state is small so you may encounter the problems mentioned in the parent article, of unreachable parts of the solution space that you are investigating in some Monte Carlo simulation.
It is preferable to be able to choose yourself the bias-removing method, to be able to use a hashing method with a much greater state.
What I personally would like to see is some kind of quantization of how the biases that the author talks about (such as insufficient seed volume of a PRNG) affects computed p-values. Specifically, why there must no "cancellation of errors" happen? So far, IIUC, the author only shows theoretical possibility of errors, but what's more interesting is a real effect. When it all boils down to a p-value being less than a certain threshold (choosing which is another pita), it might not matter whether a true p-value is within, say, 2^-16 from the computed.