I'm a pretty confused on how this all ties together. Are these these deep dives on examples on information theory? Would love some perspective that knows more on the topic.
It's an introduction to the fundamental concepts of information theory (entropy, KL divergence, conditional entropy), using 20 questions as an ongoing source of example applications and interpretations of these concepts. The author also sprinkles in various interpretations coming from their own research focus (the social sciences).
Definitely not a deep dive. The details can probably be found by chasing down the citations.
I thought it was ironic, because it starts out almost like a children's story and is very approachable. I'd rather read this than a textbook, that's for sure.
You know the book series 'Computers for Dummies?' It's a play on that. See also Zytrax's 'X for Rocket Scientists' which covers complicated stuff like LDAP and DNS and explains why they are the way they are.
I didn't even make the "for dummies" connection! The name is a huge fail but it looks like the author meant it well, perhaps someone should tell him that it comes across as intellectual elitism and exclusionary to people who perhaps feel a little insecure reading material on the internet that is associated with fancy universities and graduate schools.
For another perspective of "the study of information in the social world", check out "Variational ecology and the physics of sentient systems" (https://www.sciencedirect.com/science/article/pii/S157106451...). Also based on information theory, released just a few months after op paper.
Why do they use the word Release? Release of entropy suddenly drops into the picture when talking abt ice to water. Why not just say info/entropy/uncertainty of the system is increasing or decreasing?
This subject is new to me but I like the 20 questions analogy.
Thinking of probability distribution of N words as a measure of info in someone's mind is interesting.
Disciplined minds that have certain concepts on their finger tips clearly have spent time and effort producing that distribution.
True, it does seem a bit weird to use "release" in reference to entropy. I've googled around a bit and can't find a similar phrasing anywhere. A lot of places talk about "energy" being released into the surroundings, and this article talks about using the entropy of the surroundings to predict whether or not ice will melt: http://www.4college.co.uk/a/O/entsurr.php .
This is the opposite of pdf articles on HN. Many are titled with something like "beginners" or "for everyone", then I open up a Latex formatted document that is impenetrable from the first page using math symbols I have never seen before. this is the opposite, much easier to understand and not loaded up with mathematics symbol vomit all over the page.
Super-pleased to see the attention to this little piece. It gets updated from time to time, and it's always useful to see comments. I appreciate it.
Some responses:
1. hos234 asks "Why do they use the word Release? Release of entropy suddenly drops into the picture when talking abt ice to water. Why not just say info/entropy/uncertainty of the system is increasing or decreasing?"
It can be useful (or, at the very least, fun) to think of uncertainty as a fluid that moves through a physical system. Imagine an air-conditioner, for example; it is doing work, but not changing the overall energy of the system (at least in approximation and if it is working very slowly). It's just separating out particles.
2. I called it "for intelligent people" because I wanted to emphasize that just because you don't know what information theory is, or are not particularly mathy, it doesn't mean the author (me) thinks you're dumb. The hope was to counteract the usual physics-style arrogance that xkcd captures in "why does your field need a journal?" https://xkcd.com/793/ And yes, per jldugger, it was a play on the "for Dummies" series.
Ironically for me, the humanities people that it was aimed at often took it amiss. Dan Dennett told me as much. throwawayjava thank you for the suggestion; I'll use it in a footnote in the next iteration!
3. if you liked "information theory for intelligent people", you'll LOVE "Bayesian reasoning for intelligent people", the shocking prequel featuring characters that will ruin everything for you and plus be mildly racist? http://santafe.edu/~simon/br.pdf
Ironically for me, the humanities people that it was aimed at often took it amiss. Dan Dennett told me as much. throwawayjava thank you for the suggestion; I'll use it in a footnote in the next iteration!
Yes, I thought this article was a very well written introduction, but I held back on sharing this link with some non-math friends because I didn't want them to feel dumb if they find it difficult to get through.
20 comments
[ 0.28 ms ] story [ 44.1 ms ] threadDefinitely not a deep dive. The details can probably be found by chasing down the citations.
"Here's a treatment that will be useful for any smart-enough person [even if they aren't a phd in math/theoretical CS]".
Perhaps a better title could be "for Any Intelligent Person".
I would personally suggest a title "for curious people".
This subject is new to me but I like the 20 questions analogy.
Thinking of probability distribution of N words as a measure of info in someone's mind is interesting.
Disciplined minds that have certain concepts on their finger tips clearly have spent time and effort producing that distribution.
It's hard to imagine anyone improving on it.
http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...
Some responses:
1. hos234 asks "Why do they use the word Release? Release of entropy suddenly drops into the picture when talking abt ice to water. Why not just say info/entropy/uncertainty of the system is increasing or decreasing?"
It can be useful (or, at the very least, fun) to think of uncertainty as a fluid that moves through a physical system. Imagine an air-conditioner, for example; it is doing work, but not changing the overall energy of the system (at least in approximation and if it is working very slowly). It's just separating out particles.
The more advanced version is Maxwell's Demon; Rolf Landauer at IBM did the full analysis of this system, which you can think of as creating useful energy by "consuming" a pristine hard-drive (or, equivalently, dumping entropy into it.) http://www.w2agz.com/Library/Limits%20of%20Computation/Landa... or https://www.sciencedirect.com/science/article/pii/0375960196...
2. I called it "for intelligent people" because I wanted to emphasize that just because you don't know what information theory is, or are not particularly mathy, it doesn't mean the author (me) thinks you're dumb. The hope was to counteract the usual physics-style arrogance that xkcd captures in "why does your field need a journal?" https://xkcd.com/793/ And yes, per jldugger, it was a play on the "for Dummies" series.
Ironically for me, the humanities people that it was aimed at often took it amiss. Dan Dennett told me as much. throwawayjava thank you for the suggestion; I'll use it in a footnote in the next iteration!
3. if you liked "information theory for intelligent people", you'll LOVE "Bayesian reasoning for intelligent people", the shocking prequel featuring characters that will ruin everything for you and plus be mildly racist? http://santafe.edu/~simon/br.pdf
Yes, I thought this article was a very well written introduction, but I held back on sharing this link with some non-math friends because I didn't want them to feel dumb if they find it difficult to get through.
https://colah.github.io/posts/2015-09-Visual-Information/
his blog is pretty cool. he is original.
Much reading to do on his blog :-)