What is a model anyways? There are so many answers to say you that. The models are almost the same models, but at a different abstraction away from the original experienced in reality.
"the sciences" is very broad. in biology there are established methods for establishing causality (i.e. Koch's postulates, etc), and even then conclusions are generally qualified. not sure about the other fields, but I wish they had more concrete and recent examples of what they are talking about. this was painful to even skim.
also for some reason i cant click on anyting on the site or select text?
Looking at the paper, the core message is 'that even scientists harbor the illusion of understanding more than they actually do'.
In reality, science operates much like a mental model. The paper argues that just because a model predicts future values more accurately, it doesn't mean the model explains the actual causal structure. Yet, the fact that outcomes fall within the predicted range reinforces the illusion that one has truly 'understood' it.
This reminds me of the statistician's aphorism: 'All models are wrong, but some are useful.' Science itself, in a way, is a mental model—a simplification created for humans because the world is a complex system that is cognitively impossible to fully comprehend. Within that framework, certain facts reinforce the mental model, while others weaken it. While mental models vary from person to person, in a broad sense, we are commonly taught to view the macroscopic world through the Newtonian model and the microscopic world through the quantum mechanics model.
Reading this makes me reconsider what 'understanding' truly means. I believe the starting point of genuine understanding is acknowledging that perfect prediction is ultimately impossible, and that when viewing the world through our mental models, what matters is defining what we consider to be acceptable 'lossy information' (or information we can afford to lose)
This is a classic case of overthinking.
Induction should not yield new knowledge because nothing new is discovered, but it does.
Deduction likewise also cannot establish new knowledge, yet it does.
Empirical science is flawed on extremely many levels but it works because on average, over time, many converging observations can build refined and accurate causal theories.
It’s a matter of practicality that things cannot be proven fully. Judging from the state of modern medicine, engineering and the sciences, the system works ok regardless
I don’t think so. The point is that really nothing can be understood. This has been known since ancient times. But in practical terms it doesn’t matter. It doesn’t matter that you truly understand e.g explosions if your goal is to blow stuff up. You can drive this argument in circles like a child saying “but why” and you will discover that science is built on axioms, just things we kind of believe. They work well enough. Is it true knowledge? Probably not in the idealistic sense. But then again that just has no practical value. In the end science is a tool and not a means in itself, if you ask me.
This is kind of interesting, but I predict that it pleases almost nobody. Philosophy of science types will be kind of annoyed at the preoccupation with statistics, ML people will be annoyed at too much philosophy of science, etc.
I totally support a goal to get those groups talking more but something tighter is probably better. And why isn't it tighter? Without big original contributions, the goal does seem to be a survey
Like thinking LLMs aren’t magic* because you utter “it’s just predicting the next token!” I’d argue, only slightly tongue in cheek, that thinking of LLMs as magical leads to more effective use than the predicting-next-token explanation.
See also Frank Keil’s “illusion of explanatory depth.”
* magic not as “unreal,” but in the classical conception of a living magic world where mental intentions can manifest physical realities
This applies way beyond the sciences. So many people now think they understand something because they can prompt an AI to give them the right answer. Thats literally this same illusion just with a new interface on top. Getting correct outputs isnt understanding.
> It is the writer's experience that new degrees of comprehension are always and only consequent to ever-renewed review of the spontaneously rearranged inventory of significant factors. This awareness of the processes leading to new degrees of comprehension spontaneously motivates the writer to describe over and over again what—to the careless listener or reader—might seem to be tiresome repetition, but to the successful explorer is known to be essential mustering of operational strategies from which alone new thrusts of comprehension can be successfully accomplished.
R. Buckminster Fuller – Synergetics: Explorations in the Geometry of Thinking
> Delusional interpretation is a false deduction drawn from an accurate perception. The subject perceives correctly, but reasons wrongly; in him, judgment is impaired by affective disturbance, while the senses remain normal.
> Delusion progresses by accumulation, radiation, and extension; its richness is inexhaustible. The plan of the edifice does not change, but its proportions keep increasing.
> Every new fact, however insignificant, is immediately incorporated into the delusional system, where it becomes a fresh piece of evidence. The patient lives in a state of perpetual suspicion, searching everywhere for guiding threads, clues, correlations.
> Interpreters are not hallucinated subjects; they are logicians gone astray. Their point of departure is an intuition or a false belief, but the consequences they draw from it follow one another with an apparent rigor that often deceives the superficial observer. It is order within madness, logic in the service of the absurd.
> The need to write, graphomania, is in many interpreters a major symptom. They accumulate immense files, endless memoirs, interminable correspondences, in which every detail of their existence is dissected, analyzed, turned over and over, in order to bring to light what they believe to be the truth.
Sérieux & Capgras — Reasoning Madness: The Delusion of Interpretation
> The madman is, rather, the free man: the one who does not allow himself to be chained by the false appearances of common reality. Delusion is not an insult to logic; it is logic driven to exasperation. The paranoiac is a tireless translator, a man who spends his life deciphering the signs of the world in order to find in them the key to his own destiny. Far from being chaos, psychosis is an attempt at rigor, a complete theory that the subject constructs in order to account for his own genesis and his place before the Other. The risk of madness is measured by the very attraction of the identifications through which man alienates his freedom.
> following Fontenelle, I surrendered myself to that fantasy of holding my hand full of truths, the better to close it over them. I confess the ridiculousness of it, because it marks the limits of a being at the very moment when he is about to bear witness. Must one denounce here some failure in what the movement of the world demands of us, if speech was offered to me once again, at the very moment when it became clear even to the least perceptive that, once again, the infatuation of power had only served the cunning of Reason? I leave it to you to judge how my inquiry may suffer from it.
It's funny when you think you know something pretty much thoroughly. Then you learn a bit more and realize that your understanding was a bit simplified, had gaps or there was a whole other level to it.
The feeling is a strange mixture of disappointment, awe, annoyance and excitement.
More predictive power is always a good goal, full stop. This is orthogonal to whether the model producing prediction helps with "understanding" directly. Predictability encodes understanding in a strict information theoretic sense, regardless of our ability as humans to access that understanding.
But in some cases it is not good enough. If you look for a better explanation and chose gradient descent as your strategy, then you'll come to a local maximum eventually, but not for another explanation.
Arguably, it is hard to look for better explanation if the current one doesn't have a backtrack of failed predictions. One of the possible ways out of this situation is to search for the predictions that fail.
But what I want to say is explanations are not just for prediction. They are needed to build a mental model that then can drive the research. And new model can be built (theoretically) from the first principles. I can't find clean examples for it though. If we look at Einstein for example, he started with a failure to predict. But what he came up at first was Special Relativity which failed utterly with the gravity. Einstein spent like 10 years rewriting gravity to make it work with SR? Failed predictions of his new shiny theory didn't stop him, and it is considered to be good.
> Predictability encodes understanding in a strict information theoretic sense, regardless of our ability as humans to access that understanding.
But it doesn't necessary implies the possibility to move forward. I'm not sure if an analogy with compressed data is a good one, but you don't work with compressed data, you unpack it, and maybe unpack some more and convert to a very inefficient format with regard to the disk space used.
Compressed theory is good to apply it as is, but to refine it you should probably prefer something else.
It's not arguing that predictive power is bad. Just that people often mistakenly believe some phenomenon is understood more deeply than it really is, because a model can fit data and generate accurate predictions.
got me at "Most often scientists believe they understand more than they do, making their belief an illusion."
but why is it still bothering me?
1. feels unfalsifiable in spirit
2. somewhat restates "all models are wrong, but some are useful" less cleanly
3. doesn't really offer like, what can we do as science people? tomorrow morning perspective
> Many people know of Simpson’s through simple examples. One was UC Berkeley admissions showing that individual departments admitted more women than men but the university as a whole had admitted more men than women (Bickel et al., 1975).
This only seems possible if students can be admitted to more than one department.
The authors of this paper have not studied what historians and philosophers of science have written. They just use 'induction', 'validity', etc. They reinvent the wheel. They write "Of course the validity of that induction depends on a host of other assumptions.". Duhem-Quine thesis is better than this way of formulation, as the latter doesn't use 'validity'.
If authors ever come to this forum, please read Duhem-Quine thesis, over/under determination, inference to the best explanation, Goodman's paradox, also how various theories in philosophy of sciences: from Popper to Kuhn, Lakatos, Laudan, etc.
Who is this for? This section, which is fairly close to a central thesis, contains no citations:
> Illusions of understanding can take several (overlapping) forms. Some that are commonly encountered are: (1) Illusions of explanatory depth (we think we personally understand things in more detail than we do). (2) Illusions of explanatory completeness (even if we don’t think we fully understand it ourselves, we think the best experts do). (3) Illusions resulting from understanding something other than the goal (e.g. we believe we understand the formation of memories because we understand the anatomy of the brain site, the hippocampus, that is needed for such learning). (4) Illusions due to simple statements giving a feeling of insight (such as when tautological statements seem insightful because they are framed in a reductionist manner). (5) Illusions (as described earlier) that one understands the cause of phenomena because there exists a model or procedure that predicts well. (6) Illusions of causal strength (attending to an observed relation makes one believe the causal connection is stronger than it is). (7) illusions that one can describe causes simply. (8) Illusions by the explainer that the recipient understands what the communicator intends. (9) Illusions by the recipient of an explanation that the communicator understands well and that the explanation is correct and complete.
Are we to infer that these observations are unsupported by evidence? Are we to assume that the research work is so poorly constructed that they did not do research to find evidence of the existence of the classifications in existing research?
This is fascinating. It's so funny how everything flows back to philosophy... What is understanding? What is knowledge? Can we know what the truth is, and does it exist?
23 comments
[ 3.6 ms ] story [ 52.5 ms ] thread"the sciences" is very broad. in biology there are established methods for establishing causality (i.e. Koch's postulates, etc), and even then conclusions are generally qualified. not sure about the other fields, but I wish they had more concrete and recent examples of what they are talking about. this was painful to even skim.
also for some reason i cant click on anyting on the site or select text?
In reality, science operates much like a mental model. The paper argues that just because a model predicts future values more accurately, it doesn't mean the model explains the actual causal structure. Yet, the fact that outcomes fall within the predicted range reinforces the illusion that one has truly 'understood' it.
This reminds me of the statistician's aphorism: 'All models are wrong, but some are useful.' Science itself, in a way, is a mental model—a simplification created for humans because the world is a complex system that is cognitively impossible to fully comprehend. Within that framework, certain facts reinforce the mental model, while others weaken it. While mental models vary from person to person, in a broad sense, we are commonly taught to view the macroscopic world through the Newtonian model and the microscopic world through the quantum mechanics model.
Reading this makes me reconsider what 'understanding' truly means. I believe the starting point of genuine understanding is acknowledging that perfect prediction is ultimately impossible, and that when viewing the world through our mental models, what matters is defining what we consider to be acceptable 'lossy information' (or information we can afford to lose)
I totally support a goal to get those groups talking more but something tighter is probably better. And why isn't it tighter? Without big original contributions, the goal does seem to be a survey
;).
See also Frank Keil’s “illusion of explanatory depth.”
* magic not as “unreal,” but in the classical conception of a living magic world where mental intentions can manifest physical realities
R. Buckminster Fuller – Synergetics: Explorations in the Geometry of Thinking
> Delusional interpretation is a false deduction drawn from an accurate perception. The subject perceives correctly, but reasons wrongly; in him, judgment is impaired by affective disturbance, while the senses remain normal.
> Delusion progresses by accumulation, radiation, and extension; its richness is inexhaustible. The plan of the edifice does not change, but its proportions keep increasing.
> Every new fact, however insignificant, is immediately incorporated into the delusional system, where it becomes a fresh piece of evidence. The patient lives in a state of perpetual suspicion, searching everywhere for guiding threads, clues, correlations.
> Interpreters are not hallucinated subjects; they are logicians gone astray. Their point of departure is an intuition or a false belief, but the consequences they draw from it follow one another with an apparent rigor that often deceives the superficial observer. It is order within madness, logic in the service of the absurd.
> The need to write, graphomania, is in many interpreters a major symptom. They accumulate immense files, endless memoirs, interminable correspondences, in which every detail of their existence is dissected, analyzed, turned over and over, in order to bring to light what they believe to be the truth.
Sérieux & Capgras — Reasoning Madness: The Delusion of Interpretation
> The madman is, rather, the free man: the one who does not allow himself to be chained by the false appearances of common reality. Delusion is not an insult to logic; it is logic driven to exasperation. The paranoiac is a tireless translator, a man who spends his life deciphering the signs of the world in order to find in them the key to his own destiny. Far from being chaos, psychosis is an attempt at rigor, a complete theory that the subject constructs in order to account for his own genesis and his place before the Other. The risk of madness is measured by the very attraction of the identifications through which man alienates his freedom.
> following Fontenelle, I surrendered myself to that fantasy of holding my hand full of truths, the better to close it over them. I confess the ridiculousness of it, because it marks the limits of a being at the very moment when he is about to bear witness. Must one denounce here some failure in what the movement of the world demands of us, if speech was offered to me once again, at the very moment when it became clear even to the least perceptive that, once again, the infatuation of power had only served the cunning of Reason? I leave it to you to judge how my inquiry may suffer from it.
Lacan — Remarks on Psychic Causality
The feeling is a strange mixture of disappointment, awe, annoyance and excitement.
But in some cases it is not good enough. If you look for a better explanation and chose gradient descent as your strategy, then you'll come to a local maximum eventually, but not for another explanation.
Arguably, it is hard to look for better explanation if the current one doesn't have a backtrack of failed predictions. One of the possible ways out of this situation is to search for the predictions that fail.
But what I want to say is explanations are not just for prediction. They are needed to build a mental model that then can drive the research. And new model can be built (theoretically) from the first principles. I can't find clean examples for it though. If we look at Einstein for example, he started with a failure to predict. But what he came up at first was Special Relativity which failed utterly with the gravity. Einstein spent like 10 years rewriting gravity to make it work with SR? Failed predictions of his new shiny theory didn't stop him, and it is considered to be good.
> Predictability encodes understanding in a strict information theoretic sense, regardless of our ability as humans to access that understanding.
But it doesn't necessary implies the possibility to move forward. I'm not sure if an analogy with compressed data is a good one, but you don't work with compressed data, you unpack it, and maybe unpack some more and convert to a very inefficient format with regard to the disk space used.
Compressed theory is good to apply it as is, but to refine it you should probably prefer something else.
This only seems possible if students can be admitted to more than one department.
If authors ever come to this forum, please read Duhem-Quine thesis, over/under determination, inference to the best explanation, Goodman's paradox, also how various theories in philosophy of sciences: from Popper to Kuhn, Lakatos, Laudan, etc.
> Illusions of understanding can take several (overlapping) forms. Some that are commonly encountered are: (1) Illusions of explanatory depth (we think we personally understand things in more detail than we do). (2) Illusions of explanatory completeness (even if we don’t think we fully understand it ourselves, we think the best experts do). (3) Illusions resulting from understanding something other than the goal (e.g. we believe we understand the formation of memories because we understand the anatomy of the brain site, the hippocampus, that is needed for such learning). (4) Illusions due to simple statements giving a feeling of insight (such as when tautological statements seem insightful because they are framed in a reductionist manner). (5) Illusions (as described earlier) that one understands the cause of phenomena because there exists a model or procedure that predicts well. (6) Illusions of causal strength (attending to an observed relation makes one believe the causal connection is stronger than it is). (7) illusions that one can describe causes simply. (8) Illusions by the explainer that the recipient understands what the communicator intends. (9) Illusions by the recipient of an explanation that the communicator understands well and that the explanation is correct and complete.
Are we to infer that these observations are unsupported by evidence? Are we to assume that the research work is so poorly constructed that they did not do research to find evidence of the existence of the classifications in existing research?