I think the explanation of entropy's blind spot is a bit off. It's not actually a problem for entropy if something is generated by a rule, you can calculate entropy for things like the continuous fractions for instance, with an easy rule to generate them for any particular number. Likewise for decimal expansions.
The real blind spot is that entropy is meaningless for a specific sequence, you can't really ask about the entropy of pi if you don't have a theory for how the numbers are generated. Sure if it is pick a uniformly random real number between 0 and 10 then both files have equivalent entropy, but sending pi is also vanishingly unlikely.
There's actually a more subtle way in which this is a blind spot, which takes a bit more machinery. You can define entropy for an ergodic system, which could be considered a kind of mathematical RNG. Now as it turns out this provides a way to generate something almost equivalent to a particular distribution except that this argument only holds for most starting points not all. A direct example would be how pi generates a perfectly fine random distribution of digits (we think) but something like 1/3 does not.
The thing that frustrates me about this argument is that there is no shortest program that produces pi. You need a computer to run it, which is massive non compressed data, or a human to calculate stuff, an uncountable amount of entropy.
I see that the irrational pi has a smooth distribution of digits and a file full of zeroes is compressible, but they are both sort of magically part of a world that does not run programs and thus not quite different in a practical sense.
Lately I’ve felt Kolmogorov complexity is an unfair measurement because it takes for granted your underlying programming language as treats it as zero cost. In theory you could create a custom language and embed the program as data and “compress” a large random sequence with a better Kolmogorov complexity for that specific language than Pi, simply by not exposing the ability in the language to even work with Pi. I think what’s maybe more interesting is when you take into account the work of Dr. Futamura and the idea of Jones Optimality and view things through that lens.
The choice of pi as a number whose sequence of digits is random is a bit of a weak point of the argument in the post. It is not even known whether every digit 0-9 appears infinitely many times in pi (a weak version of the normal number conjecture). So we do not really know (but strongly believe) that the sequence of pi digits would truly appear random. This of course does not detract from beautiful arguments and the general ideas in this post.
This is under-taught in comp sci so I'll say it in all caps for visibility. THERE IS NO KNOWN GENERIC WAY TO MEASURE INFORMATION ENTROPY!!!!
Seriously. There isn't. You might say 'but measure the number of 1's vs 0's' and i'll just reply with '101010101010 repeated'. So you up it and start saying ok maybe measure frequencies of pairwise bits then and i just reply by cycling 00,01,10,11 endlessly etc. Frequency counts and any other measure you can think of doesn't actually work.
The root definition will state it's just based on the properties of the prediction but... what model do you use for the prediction? This gets to the heart of the matter;
The measure of information entropy itself is actually equivalent to creating on omniscient oracle. It requires being able to ask the question "Oh great oracle, what's the shortest program that can reproduce X"? or equivalently "What's the probability of this data assuming i always have the most perfect model to predict it?". You then calculate entropy based on that oracles answer.
Kolmorogorov, AI and information entropy are all pointing to the same thing. The reason people get confused by the fact that the frequencies of base 10 digits is the same for digits of pi and a truly random sequence is just because the counts of base 10 digits isn't a measure of entropy at all. Measuring entropy literally requires an omniscient oracle.
I tried to make a codebook of everything. You can get surprisingly good compression by assigning words and phrases to numbers.
My particular attempt had phrases up to 12 words.
No grammar or anything...to test the program, when I had a random thought, I tested to see if I could encode that thought. I could and on average the thought/message was compressed by 2. There was very few times where I wasent able to encode the idea into the available building blocks or language model of the database.
I don’t understand the claim that Kolmogorov lower bounds are uncomputable. Three digit strings are unlikely to run as a program so surely 3 is a lower bound on Kolmogorov of the first million digits of pi, no? To tighten bound, one would need to try out every 4-digit program, every 5-digit program, and so on. Either these produce the first million digits of pi or they don’t when the program is run, so each possibility is checkable in finite time, no? There must be something I’m missing here.
Overall an interesting blog post, but the author makes this claim several times:
> So you can never close the case on even one candidate, let alone all of them at once, which means the lower bound stays sealed.
Which is simply not true. The specifics depends on what language you use, but let's take Turing machines as an example. Many candidates are easily disproven: ones that halt immediately, ones that have no rule for writing a "1", ones that have no rule for halting and so on. It's impossible to write a program that can handle every possible candidate though.
And if your string is short enough, let's say the string you are investigating is "0011" then you CAN sit down and check all Turing machines of size 4 and lower. This is essentially what the https://bbchallenge.org/ project did.
For Turing machines we have determined if they halt or not for all machines up to 5 states. It is very computationally hard to push this limit further, but we don't know where the theoretical limit is. We have some large upper bound where a Turing machine computes something that we know is independent of ZFC (see Scott Aaronson for more details), but many suspect that the limit is way lower.
Kolomogorov complexity as an objective measure of the information contained in a string just doesn't work. It depends entirely on the language used for the program, and for any string, in some language, the complexity is zero.
11 comments
[ 2.4 ms ] story [ 34.3 ms ] threadThe real blind spot is that entropy is meaningless for a specific sequence, you can't really ask about the entropy of pi if you don't have a theory for how the numbers are generated. Sure if it is pick a uniformly random real number between 0 and 10 then both files have equivalent entropy, but sending pi is also vanishingly unlikely.
There's actually a more subtle way in which this is a blind spot, which takes a bit more machinery. You can define entropy for an ergodic system, which could be considered a kind of mathematical RNG. Now as it turns out this provides a way to generate something almost equivalent to a particular distribution except that this argument only holds for most starting points not all. A direct example would be how pi generates a perfectly fine random distribution of digits (we think) but something like 1/3 does not.
I see that the irrational pi has a smooth distribution of digits and a file full of zeroes is compressible, but they are both sort of magically part of a world that does not run programs and thus not quite different in a practical sense.
Just my thoughts and sorry for the confusion.
Seriously. There isn't. You might say 'but measure the number of 1's vs 0's' and i'll just reply with '101010101010 repeated'. So you up it and start saying ok maybe measure frequencies of pairwise bits then and i just reply by cycling 00,01,10,11 endlessly etc. Frequency counts and any other measure you can think of doesn't actually work.
The root definition will state it's just based on the properties of the prediction but... what model do you use for the prediction? This gets to the heart of the matter;
The measure of information entropy itself is actually equivalent to creating on omniscient oracle. It requires being able to ask the question "Oh great oracle, what's the shortest program that can reproduce X"? or equivalently "What's the probability of this data assuming i always have the most perfect model to predict it?". You then calculate entropy based on that oracles answer.
Kolmorogorov, AI and information entropy are all pointing to the same thing. The reason people get confused by the fact that the frequencies of base 10 digits is the same for digits of pi and a truly random sequence is just because the counts of base 10 digits isn't a measure of entropy at all. Measuring entropy literally requires an omniscient oracle.
I tried to make a codebook of everything. You can get surprisingly good compression by assigning words and phrases to numbers.
My particular attempt had phrases up to 12 words.
No grammar or anything...to test the program, when I had a random thought, I tested to see if I could encode that thought. I could and on average the thought/message was compressed by 2. There was very few times where I wasent able to encode the idea into the available building blocks or language model of the database.
https://i.postimg.cc/WpBcB4HK/IMG-20260510-103853.jpg
> So you can never close the case on even one candidate, let alone all of them at once, which means the lower bound stays sealed.
Which is simply not true. The specifics depends on what language you use, but let's take Turing machines as an example. Many candidates are easily disproven: ones that halt immediately, ones that have no rule for writing a "1", ones that have no rule for halting and so on. It's impossible to write a program that can handle every possible candidate though.
And if your string is short enough, let's say the string you are investigating is "0011" then you CAN sit down and check all Turing machines of size 4 and lower. This is essentially what the https://bbchallenge.org/ project did.
For Turing machines we have determined if they halt or not for all machines up to 5 states. It is very computationally hard to push this limit further, but we don't know where the theoretical limit is. We have some large upper bound where a Turing machine computes something that we know is independent of ZFC (see Scott Aaronson for more details), but many suspect that the limit is way lower.
See https://forwardscattering.org/page/Kolmogorov%20complexity
And no, the invariance theorem doesn't save you.