You can substitute current buzzwords (AI, ML) for Information Theory and it reads as fresh as ever:
> INFORMATION theory has, in the last few years,
become something of a scientific bandwagon.
> It will be all
too easy for our somewhat artificial prosperity to
collapse overnight when it is realized that the use of a
few exciting words like information, entropy, redundancy,
do not solve all our problems.
> What can be done to inject a note of moderation in
this situation? In the first place, workers in other
fields should realize that the basic results of the
subject are aimed in a very specific direction, a
direction that is not necessarily relevant to such
fields as psychology, economics, and other social
sciences. Indeed, the hard core of information theory
is, essentially, a branch of mathematics, a strictly
deductive system. A thorough understanding of the
mathematical foundation and its communication
application is surely a prerequisite to other applications.
To the extent that we can do this simple word substitution, it should probably make us less impressed, not more.
Basically only the most generic points that you can make about any new technology (e.g. this will not solve every single problem, this is commonly misunderstood and misapplied) will still make sense. We should focus our attention on the specific claims made about the specific new technology being critiqued, rather than the things someone can always say.
(This observation is actually an application of information theory - if the points being made are 100% predictable then they contribute no negentropy :)
Thing is, reasonably smart people fall for this same mistake over and over.
I think Claude hits on the issue here - it is the general unwillingness to go back to the actual math and work through the narrowness of its implications.
Instead we start with a popular prose explanation and then argue our points from our understanding of that.
We barely notice when we are using the ambiguity and wiggle room in the prose explanation to argue something that is unsupported by the math.
This is awesome. "Seldom do more than a few of nature’s secrets give way at one time."
Resonates with the application of (to practitioners) blackbox ML techniques to other domains. I saw an economist stuff an XGBoost model into a talk about demand elasticity a couple years ago. I think there can be insight generated by casting a wide net in applying new methods, but the really fundamental breakthroughs seem to be generated by careful and informed application of the scientific method.
10 comments
[ 3.4 ms ] story [ 32.7 ms ] thread> INFORMATION theory has, in the last few years, become something of a scientific bandwagon.
> It will be all too easy for our somewhat artificial prosperity to collapse overnight when it is realized that the use of a few exciting words like information, entropy, redundancy, do not solve all our problems.
> What can be done to inject a note of moderation in this situation? In the first place, workers in other fields should realize that the basic results of the subject are aimed in a very specific direction, a direction that is not necessarily relevant to such fields as psychology, economics, and other social sciences. Indeed, the hard core of information theory is, essentially, a branch of mathematics, a strictly deductive system. A thorough understanding of the mathematical foundation and its communication application is surely a prerequisite to other applications.
Basically only the most generic points that you can make about any new technology (e.g. this will not solve every single problem, this is commonly misunderstood and misapplied) will still make sense. We should focus our attention on the specific claims made about the specific new technology being critiqued, rather than the things someone can always say.
(This observation is actually an application of information theory - if the points being made are 100% predictable then they contribute no negentropy :)
I think Claude hits on the issue here - it is the general unwillingness to go back to the actual math and work through the narrowness of its implications.
Instead we start with a popular prose explanation and then argue our points from our understanding of that.
We barely notice when we are using the ambiguity and wiggle room in the prose explanation to argue something that is unsupported by the math.
Resonates with the application of (to practitioners) blackbox ML techniques to other domains. I saw an economist stuff an XGBoost model into a talk about demand elasticity a couple years ago. I think there can be insight generated by casting a wide net in applying new methods, but the really fundamental breakthroughs seem to be generated by careful and informed application of the scientific method.
https://oikosjournal.files.wordpress.com/2011/09/elias1958ir...
"Information Theory, Photosynthesis and Religion”