This is an old post but it reminds me of the idea that the blogosphere of the last decade really has moved to arxiv instead. A lot less informal but for technical nuggets of insight that's really the best place with the best culture right now in 2022.
> The hunt for publishable p-values is nearly always fruitful. If one
cannot find a publishable p-value in one’s data—with the freedom to pick
and choose models and test statistics, to engage in “sub-group” and sequential analysis, and so on—then one is being lazy.
> P-values can and are used to
prove anything and everything. The sole limitation is the imagination of the
researcher. Fleeting exposure to a 72 × 45 pixel image of the American flag
turns one into a Republican; walking through a door (an “event segmentation”) damages your memory; Keynesian theory is wrong; Keynesian theory
is right; selenium causes cancer; selenium cures cancer. This list could (and
will) go on in perpetuity.
Sadly true and well written. However, the author is over certain that using Bayesian statistics will make things better.
Right, people can just go ahead with uniform priors and put significance stars next to credible intervals. But even then, there are a number of benefits, such as even stating CIs in the first place and needing to encounter some literature that's more advanced than clicking "run test" in a readymade GUI.
Maybe not, but the author is pointing at a much broader issue. Reseaerchers are pressured to do as much as possible with as little as possible. So you end up with salad bowl studies that are a mile wide and an inch deep, trying to do everything without even a pilot study to figure out what a minimum sample size might be.
It's fucking weird. If you're not wearing a military uniform, with rank, you're a civilian. This includes all scientists, except some folks at the NOAA and Public Health Service Commissioned Corps maybe.
This guy is full of himself: “we know par exp ́eriences nombreuses at funestes they are apt to misuse and misinterpret them”
It reminds me when I learned about Tyndale translating the Latin Bible into English. There was a pushback in creating a commoner’s version of the Bible where the uneducated would be able to read the Bible for themselves. The Catholic Church was later involved in his death.
Yeah, also this is not latin but french, with a typo even (i believe): should be "par expériences nombreuses et funestes" ("et" not "at"). (translation: by numerous and disastrous experiments)
He is full of himself because he uses a bit of French in an English text? I don't understand how that works. It's a common literary technique to do this and it is not without place in scholarly articles.
Just because a text is not pandering to the lowest common denominator with respect to literacy, it's not "full of [itself]".
It has the form of a scholarly article, but the author has no credentials, no institution, doesn't cite any references and to my knowledge has not submitted the article for any kind of peer review.
This is effectively Some Dude On Twitter but on Arxiv.
I like to use the term "misleading precision" because it's not just frequentism and not just non-statisticians and not just science.
That said, every step that forces everybody to acknowledge uncertainty is good. Given our progress e.g. in getting people to recognize how important power is (e.g. since Cohen, 1992 at the latest), I'm very pessimistic.
But while we may not be able to get everyone on board the bayes train, we can at least force people to openly show that they don't care about good statistics.
Bayesian methods have a lot going for them, but you need some way of checking whether the methods are giving sensible answers. Andrew Gelman (well know professor, blogger, and author of Bayesian Data Analysis) relies on simulation, which is a pretty Frequentist thing to do!
Bayesian and Frequentist methods are not nearly as at odds as posts like this suggest. Frequentism is mostly about how methods should be evaluated. Bayesianism is mostly about how to incorporate different sources of information. You can assess the Frequentist properties of Bayesian methods!
While I agree, the most important benefits of bayesian methods - in my opinion - are forcing people to consider (1) uncertainty and (2) effect sizes. These two things alone are a fundamental difference and a major step forward.
At the risk of a no-true-Scotsman argument, I want to point out the difference between Frequentism as practiced by statisticians and non-statisticians. Non-statisticians will simply say that a result was stat sig, without specifying even the corresponding p-value, let alone an effect size point estimate or confidence interval.
I always report confidence intervals front and center, and bring in point estimates and p values as supporting characters. And of course discussing to what extent the study design supports causal conclusions.
I also don't like when it's framed as an "either-or" dichotomy.
It's like a programming language - there is no "best" one in general, you just have to learn how to choose the best tool for the job at hand, understanding each tool's strengths and weaknesses. It's not that mysterious.
it may not be a personally acceptable, but there is a real world divide where a large subset of people pretty much only accept frequentism. This is often the same group publishing under powered, low n, studies.
The biggest issue with frequentism is the assumptions. I can't rattle them off like I used to be able to, but almost every real world scenario where statistics are useful are going to violate some of them, and yet frequentists will simply carry on.
It's really a cultural issue around 'correctness', and frequentism is often reduced to an appeal to authority.
Almost all modern Bayesian models rely heavily on simulation. This is mostly because it's a very effective way to do numerical integration, and almost all non-trivial Bayesian models involve integrals without closed form solutions.
Nonsense. Frequentist statistics is (relatively) easy to understand. If people can't be trusted with it, they certainly can't be trusted with Bayesian statistics.
This is more of an argument not to teach anyone statistics at all, other than some unspecified elite that the author feels qualified to learn it.
By the way, the author has some interesting obsessions:
Let me introduce you to Gerd Gigerenzer's paper "The Null Ritual", in which he brilliantly shows that almost nobody understands the rituals of frequentist statistics [1].
"How many students and teachers noticed that all of the statements were wrong? As Figure 1 shows, none of the students did. Every student endorsed one or more of the illusions about the meaning of a p-value. One might think that these students lack the right genes for statistical thinking and are stubbornly resistant to education. A glance at the performance of their teachers, however, indicates that wishful thinking might not be entirely their fault. Ninety percent of the professors and lecturers also had illusions, a proportion almost as high as among their students. Most surprisingly, 80% of the statistics teachers shared illusions with their students."
I don't disagree that many people struggle to properly interpret frequentist statistics. But adopting a Bayesian perspective isn't a magic talisman. The number of people who are competent Bayesians but don't understand frequentist statistics is zero, and no pedagogical changes will improve that.
The problem with frequentism and the null ritual is that it makes statistics easier (just do this one test, read this one number) and renders some kinds of mistakes somewhat harder (publishing a single false positive).
At the same time, it makes some bad mistakes much easier, most notably ignoring power, false positives, the garden of forking paths, type S errors and publication bias.
The inherent problem is that the null model is not what people assume it is, and the method (or at least the established canon of approaches) don't make you think about it.
If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them.
This means that even non-competent bayesians probably have a better contribution to cumulative science.
Frequentist statistics does not force you to accept any hypothesis test with a result p < 0.05 as definitive proof of something. It does not forbid considering prior probability of a result. It just doesn't formalize the consideration of prior probabilities because it is hard to distill this consideration into a formal recipe.
Everyone agrees that Bayes' Rule is valid and important, the question is when and how best to use it.
> If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them.
You're not forced to do those things well. Any scientific method can be cargo culted.
Yes I think Gelman is doing great work, as is e.g. McElreath [1].
The problem that I care about is not whether frequentist statistics can be taught and used well. They can, and I try in my teaching to do so.
The problem is that empirically, frequentist statistics is a fig leaf for a ton of extremely problematic work. And pushing bayesian thinking is currently our best chance to fix this, because it's easier to do a shift in the mental framework than to fix the perception of an existing framework.
I agree that more Bayesian thinking is needed. But I suspect that pushing the technicalities of Bayesian analysis (MCMC etc) would perversely lead to even more cargo-culting, as people would struggle with the technicalities and look desperately for quick fixes.
Bayesian methodology benefits from having relatively much more statistically sophisticated practitioners, which leads to an optimism bias when we imagine how it would scale up.
Perhaps you don't see it this way, but I take our eventual convergence as a sign that we don't actually disagree. It's rather that the problem we're starting from is different. Thanks for taking your time for this debate!
I guess it depends on the context in which it is applied. It is anecdotal, but in the industries where I worked, people trying to introduce Bayesian statistics did not have a higher chance of their results being interpreted properly. If you do A/B testing and don't do pre-registration or power analysis, what are the chances that you can/will be able to explain the nuances of probabilistic reasoning ?
I agree w/ the parent poster than the fundamental issue is probability: if you are talking w/ people w/o background in stats, you will have a really hard time to go beyond a true/false statement.
Moreover, one of the most effective (in $ terms) application of statistics in recent times is A/B testing. While you can do "Bayesian A/B testing", the basic methodology is fundamentally frequentist. Mistakes there can be hedged through better tooling / UX (to avoid peeking, etc.), as effectively as using Bayesian statistics.
Hum... You are basically equating frequentism with the null ritual. The article itself is basically equating frequentism with the null ritual.
Ok, that explains why people bother spend so much energy badmouthing frequentism, but it's a very bad framing anyway, bothering a lie. Frequentism is not the null ritual. In fact, it's almost completely compatible with bayesianism, the one large difference being the freedom to set priors before doing your analysis.
If the article was titled "It's time to stop teaching the null ritual to scientists", nobody would even disagree.
> Frequentism is not the null ritual. In fact, it's almost completely compatible with bayesianism, the one large difference being the freedom to set priors before doing your analysis.
This may be true of "frequentism" in a strict sense, but many statistical methods in common use that are often described as such (including, arguably, NHST) are not consistent with reasonable versions of the likelihood principle https://en.wikipedia.org/wiki/Likelihood_principle . In a sense, one might feasibly argue that these methods are not even properly frequentist.
No one said it’s a magic talisman. But Bayesian probability is far more in line with most peoples intuitive understanding about probability and statistics than frequentist probability.
There will always be no instances where individuals err in the way they apply statistical methods. But that doesn’t mean that there is no value in moving the “default” in statistics to a more intuitive methodology from one that is so obtuse that it’s common for relatively advanced practitioners to stumble over it
The problem is deeper than people just struggling with frequentist statistics. It's difficult IMHO to actually reason properly about frequentist results without also having a grounding in Bayesian statistics. For example, in the Gigerenzer et al. paper cited above, to properly understand why statement #6 is false, a person has to understand that they've confused p(D|H0) with 1 – p(D). That's why it's not surprising IMHO that cookbook approaches to frequentist statistics without a nontrivial Bayesian component, as are often found in social science-focused stats courses, inevitably end up with limited or skewed comprehension.
The difference between Bayesian and frequentist statistics is that they answer different questions. People are often more interested in the questions that Bayesian statistics answer, and so they often misinterpret frequentist results to also answer those questions.
It's possible that it can happen the other way around as well, but my impression is that it happens less often.
Well, the "article" is mostly an opinion piece, it could as well be a Twitter thread. So, at least, there is some value when evaluating a personal opinion in the context of other opinions expressed publicly by the author. I'm not saying this is the case, but I don't think is invalid either.
Of course it’s invalid. Either there is something about the substance of the article that is debatable or there isn’t. Other opinions are completely irrelevant.
If I write an article about the benefits of X drug and have expressed publicly by other channels my support for the company that produces such drug, I think is relevant for the discussion of the article I just wrote. That's why I said that may be it is not the case here, as I my self don't see any relationship between the linked tweets and the article. But I don't think is to crazy to think that two related opinions can be analyzed together regardless of the channels of distribution.
If you write an article about the benefits of X drug, you have presumably made arguments that can or can not be supported with available evidence. Your other interactions with the company producing the drug are irrelevant.
Browsing his personal website reveals frequent mistakes in science, statistics, and probability theory. I don’t think I’ll spend any time on this paper.
None of those twitter threads have anything to do with statistics pedagogy at all. What sort of irrelevant, ad hominem argument are trying to make here?
The twitter links you cite actually make me respect the author more, it shows for example he's a contrarian in a situation where that would cost him a lot, an attitude that I highly respect.
Alas, the article is badly written and unprofessionally laid-out, and, ironically given the title, isn't written with a non-statistician in mind.
There's a whole cottage industry of audience-captured contrarians who rely on "contrarian trust" to sell you the opposite of whatever the Establishment is selling.
It's not any more thoughtful, it's just the opposite.
There's a not-so-subtle failure mode where you go from "Some of those against the Establishment are cranks" to "All of those who go against the Establishment are cranks", it feels more clever and satisfying than "I really really like the Establishment and can't tolerate it or its opinons being wrong", but it's indistinguishable in input-output behaviour.
If you are talking about his statistics opinions, I don't understand enough statistics to know if it's right or wrong or not even wrong. But his other claims gives him a substantial respect-boost regardless of whether he's right or wrong. I simply like contrarians more.
This is all uninteresting to me because contrarian versus Establishment is a pretty poor indicator for truth. Leaning on that is like the famous drunkard leaning on the lamppost, more for support than illumination.
The yellow flag in those tweets is the hyperventilating culture warrior attitude. It speaks poorly of ability to humbly and dispassionately analyze topics, regardless of which side it comes from.
>contrarian versus Establishment is a pretty poor indicator for truth
False. It's a pretty reliable proxy if the Establishment is known or plausibly suspected to benefit from falsehood and\or misrepresent truth. If you have a reliable predictor of falsehood, then something which always opposes it is a reliable predictor of truth. I don't know if the statistics Establishment is one such example (probably not), but the trans-activism and war-media Establishments certainly Are reliable predictors of falsehood, and have been shown to be on multiple occasions.
That's why I like the author's opinions on those topics more than on statistics, even if I'm not necessarily certain of their truth. He's going in the rough general direction of truth.
>hyperventilating culture warrior attitude. It speaks poorly of ability to humbly and dispassionately analyze topics, regardless of which side it comes from.
False Again. Plenty of smart people have this attitude, the notorious example off the top of my head is Nassim Nicholas Taleb, incredibly arrogant and quick to ungraciously fail into combat under criticism - yes - but do you actually want to take him one-on-one in statistics? For a more distant example, see Newton's hilarious pettiness in dealing with rivals. Your heuristic yellow flag would have lead you to dismiss him faster than you can say "Universal Gravitation", and we both know you would be utterly and irredeemably wrong.
> Frequentist statistics is (relatively) easy to understand.
Easy to misunderstand. NHST creates false beliefs, and that's even when you do everything correctly. If you accept NHST, you believe in "psi", because Bem has proven it.
Regardless of the pros and cons of Bayesian methods, here is what I believe is needed:
- Pre-register all studies, declaring sample sizes and power analysis.
- Report results regardless of outcome. Eliminate the "we only publish stat sig results" baloney.
- Report confidence/credible intervals, adjusting for multiple comparisons as appropriate. Plot the posterior distribution of the effect size if appropriate.
- Publish all data and code.
- Provide funding for duplicating important studies.
100% this. It's not even hard today, just a cultural shift.
I'd add one additional technique: Specification curve analyses. Bonferroni etc. alone won't help against systematic bias in a field and/or misconduct, and specification curves are easy to do, e.g. with specR [1].
I don't know how often this is practical, but here's an alternative view: preregistration is unimportant when you are looking for enormous effect sizes, and you should do that if you can:
Hoeffding's inequality, Chebyshev's inequality, and Chernoff's inequality are broadly applicable, and are therefore less likely to be misused. They also don't require philosophical assumptions about subjective probabilities.
In the theory of Multi-Armed Bandit algorithms, compare the Bayesian approach (Thompson sampling) with the Concentration Inequality approach (the UCB algorithm). The former assumes that payoffs are in the 2-element set {0,1}, while the latter allows payoffs to be in the interval [0,1].
I think Bayesian is cool, but it imposes a big burden with choosing sensible priors.
Provide details. I don't know what "excessive" means here. They're going to be at least a bit conservative because they apply to more situations, but that's obvious.
If we meet someone identifying as a frequentest then we probably met someone who has thought deeply about philosophy of statistics. By contrast, if we meet someone who identifies as Bayesian, we probably met someone who read a blog post. Bayes theorem tells us, that this doesn't tell us anything about the underlying positions on philosophy of statistics.
So far I have observed that statistics as a whole are useless for decision support.
They are definitely useful as part of bigger models that we cannot explicitly express them in multiple fields. Physics, chemistry, ML, they all utilize statistics to make useful models and that is great.
The issue is that decision making is a singular point, and statistics are very good at not taking any responsibility for their bad predictions.
If you make a decision with an expected outcome the statistician will come out swinging. If it was the unexpected outcome then you are faced with “Well you were unlucky, that’s statistics”.
Thinking inductively is prone to error. This doesn't change if you choose to use a Bayesian approach. If you don't want error, first principles thinking is much more accurate. Some people discover it and think they've discovered the path to thinking without error, but actually first principles don't lead to no error - they lead to a deep and fundamental humility.
I find the greatest proponents of Bayesian thinking to be in the rationalist community. So to get across just how wrong everyone is about just about everything pretty much always consider that the pejorative lessons on predictable irrationality and cognitive bias are delusions that come about through the flaws of statistical reasoning. From first principles we know that superhuman chess AI are also predictably irrational and cognitively biased, but that this isn't actually irrational or biased so as to desire us to abandon the choice. Even in chess - much simpler than reality - the correct solution was too large to fit within our universe.
It is difficult to articulate the magnitude of this humility. We couldn't fit the chess solution into our universe, but how complex is chess relative to other problems? If the complexity of chess was a grain of sand or even a single atom than it would be an understatement to claim that the more general problems are only as large as our universe.
I tend to think that the problem that underlies all of this is the old question for a sound inductive method, or a deductive method that can replace it.
Many Bayesian theoreticists are already named in the comments, for a frequentist view that is well developed and quite strong (i.e. not a strawman) I want to suggest Deborah Mayo’s “Error and the Growth of Empirical Knowledge”.
This is related in the following sense to the article: I think the solution for the named ettors may be to stop teaching as if either Bayesianism or Frequentism were set truths and all statistical foundation problems anyway solved, and giving more info on what evidence is and why we think this specific set of tools will help us to prove anything at all.
Bayesianism comes with it’s own set of problems, starting with the philosophical underpinnings of it’s flavor of probability, to the often researched theory that humans and scientists are actually not Bayesian thinkers by birth (Giere, Kahneman, Tversky).
82 comments
[ 1.9 ms ] story [ 163 ms ] threadHeh. So people noticed.
> P-values can and are used to prove anything and everything. The sole limitation is the imagination of the researcher. Fleeting exposure to a 72 × 45 pixel image of the American flag turns one into a Republican; walking through a door (an “event segmentation”) damages your memory; Keynesian theory is wrong; Keynesian theory is right; selenium causes cancer; selenium cures cancer. This list could (and will) go on in perpetuity.
Sadly true and well written. However, the author is over certain that using Bayesian statistics will make things better.
Oh. How we all wish the word civilian here din't mean scientists in every field publishing in every journal.
And how I wish my reviewers didn't force people to do this :(
Civilian is someone who's never been in the armed forced and therefore ignorant of how things really work in the military and in battle.
So it's basically a metaphor for a someone who's never studied statistics in depth and therefore ignorant of how statistic really work.
It reminds me when I learned about Tyndale translating the Latin Bible into English. There was a pushback in creating a commoner’s version of the Bible where the uneducated would be able to read the Bible for themselves. The Catholic Church was later involved in his death.
Just because a text is not pandering to the lowest common denominator with respect to literacy, it's not "full of [itself]".
It has the form of a scholarly article, but the author has no credentials, no institution, doesn't cite any references and to my knowledge has not submitted the article for any kind of peer review.
This is effectively Some Dude On Twitter but on Arxiv.
That said, every step that forces everybody to acknowledge uncertainty is good. Given our progress e.g. in getting people to recognize how important power is (e.g. since Cohen, 1992 at the latest), I'm very pessimistic.
But while we may not be able to get everyone on board the bayes train, we can at least force people to openly show that they don't care about good statistics.
Bayesian and Frequentist methods are not nearly as at odds as posts like this suggest. Frequentism is mostly about how methods should be evaluated. Bayesianism is mostly about how to incorporate different sources of information. You can assess the Frequentist properties of Bayesian methods!
Larry Wasserman wrote a great post about this topic here: https://normaldeviate.wordpress.com/2012/11/17/what-is-bayes...
I always report confidence intervals front and center, and bring in point estimates and p values as supporting characters. And of course discussing to what extent the study design supports causal conclusions.
It's like a programming language - there is no "best" one in general, you just have to learn how to choose the best tool for the job at hand, understanding each tool's strengths and weaknesses. It's not that mysterious.
The biggest issue with frequentism is the assumptions. I can't rattle them off like I used to be able to, but almost every real world scenario where statistics are useful are going to violate some of them, and yet frequentists will simply carry on.
It's really a cultural issue around 'correctness', and frequentism is often reduced to an appeal to authority.
This is more of an argument not to teach anyone statistics at all, other than some unspecified elite that the author feels qualified to learn it.
By the way, the author has some interesting obsessions:
https://twitter.com/FamedCelebrity/status/155513494784911360...
https://twitter.com/FamedCelebrity/status/155516247586318336...
https://twitter.com/JohnZmirak/status/1554969385860796420?s=... (retweet)
"How many students and teachers noticed that all of the statements were wrong? As Figure 1 shows, none of the students did. Every student endorsed one or more of the illusions about the meaning of a p-value. One might think that these students lack the right genes for statistical thinking and are stubbornly resistant to education. A glance at the performance of their teachers, however, indicates that wishful thinking might not be entirely their fault. Ninety percent of the professors and lecturers also had illusions, a proportion almost as high as among their students. Most surprisingly, 80% of the statistics teachers shared illusions with their students."
[1] http://library.mpib-berlin.mpg.de/ft/gg/GG_Null_2004.pdf
The problem with frequentism and the null ritual is that it makes statistics easier (just do this one test, read this one number) and renders some kinds of mistakes somewhat harder (publishing a single false positive).
At the same time, it makes some bad mistakes much easier, most notably ignoring power, false positives, the garden of forking paths, type S errors and publication bias.
The inherent problem is that the null model is not what people assume it is, and the method (or at least the established canon of approaches) don't make you think about it.
If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them.
This means that even non-competent bayesians probably have a better contribution to cumulative science.
Frequentist statistics does not force you to accept any hypothesis test with a result p < 0.05 as definitive proof of something. It does not forbid considering prior probability of a result. It just doesn't formalize the consideration of prior probabilities because it is hard to distill this consideration into a formal recipe.
Everyone agrees that Bayes' Rule is valid and important, the question is when and how best to use it.
> If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them.
You're not forced to do those things well. Any scientific method can be cargo culted.
The problem that I care about is not whether frequentist statistics can be taught and used well. They can, and I try in my teaching to do so.
The problem is that empirically, frequentist statistics is a fig leaf for a ton of extremely problematic work. And pushing bayesian thinking is currently our best chance to fix this, because it's easier to do a shift in the mental framework than to fix the perception of an existing framework.
[1] https://xcelab.net/rm/statistical-rethinking/
Bayesian methodology benefits from having relatively much more statistically sophisticated practitioners, which leads to an optimism bias when we imagine how it would scale up.
I agree w/ the parent poster than the fundamental issue is probability: if you are talking w/ people w/o background in stats, you will have a really hard time to go beyond a true/false statement.
Moreover, one of the most effective (in $ terms) application of statistics in recent times is A/B testing. While you can do "Bayesian A/B testing", the basic methodology is fundamentally frequentist. Mistakes there can be hedged through better tooling / UX (to avoid peeking, etc.), as effectively as using Bayesian statistics.
Ok, that explains why people bother spend so much energy badmouthing frequentism, but it's a very bad framing anyway, bothering a lie. Frequentism is not the null ritual. In fact, it's almost completely compatible with bayesianism, the one large difference being the freedom to set priors before doing your analysis.
If the article was titled "It's time to stop teaching the null ritual to scientists", nobody would even disagree.
This may be true of "frequentism" in a strict sense, but many statistical methods in common use that are often described as such (including, arguably, NHST) are not consistent with reasonable versions of the likelihood principle https://en.wikipedia.org/wiki/Likelihood_principle . In a sense, one might feasibly argue that these methods are not even properly frequentist.
There will always be no instances where individuals err in the way they apply statistical methods. But that doesn’t mean that there is no value in moving the “default” in statistics to a more intuitive methodology from one that is so obtuse that it’s common for relatively advanced practitioners to stumble over it
It's possible that it can happen the other way around as well, but my impression is that it happens less often.
> https://twitter.com/FamedCelebrity/status/155513494784911360...
> https://twitter.com/FamedCelebrity/status/155516247586318336...
> https://twitter.com/JohnZmirak/status/1554969385860796420?s=... (retweet)
None of those twitter threads have anything to do with statistics pedagogy at all. What sort of irrelevant, ad hominem argument are trying to make here?
Alas, the article is badly written and unprofessionally laid-out, and, ironically given the title, isn't written with a non-statistician in mind.
It's not any more thoughtful, it's just the opposite.
If you are talking about his statistics opinions, I don't understand enough statistics to know if it's right or wrong or not even wrong. But his other claims gives him a substantial respect-boost regardless of whether he's right or wrong. I simply like contrarians more.
The yellow flag in those tweets is the hyperventilating culture warrior attitude. It speaks poorly of ability to humbly and dispassionately analyze topics, regardless of which side it comes from.
False. It's a pretty reliable proxy if the Establishment is known or plausibly suspected to benefit from falsehood and\or misrepresent truth. If you have a reliable predictor of falsehood, then something which always opposes it is a reliable predictor of truth. I don't know if the statistics Establishment is one such example (probably not), but the trans-activism and war-media Establishments certainly Are reliable predictors of falsehood, and have been shown to be on multiple occasions.
That's why I like the author's opinions on those topics more than on statistics, even if I'm not necessarily certain of their truth. He's going in the rough general direction of truth.
>hyperventilating culture warrior attitude. It speaks poorly of ability to humbly and dispassionately analyze topics, regardless of which side it comes from.
False Again. Plenty of smart people have this attitude, the notorious example off the top of my head is Nassim Nicholas Taleb, incredibly arrogant and quick to ungraciously fail into combat under criticism - yes - but do you actually want to take him one-on-one in statistics? For a more distant example, see Newton's hilarious pettiness in dealing with rivals. Your heuristic yellow flag would have lead you to dismiss him faster than you can say "Universal Gravitation", and we both know you would be utterly and irredeemably wrong.
Easy to misunderstand. NHST creates false beliefs, and that's even when you do everything correctly. If you accept NHST, you believe in "psi", because Bem has proven it.
- Pre-register all studies, declaring sample sizes and power analysis.
- Report results regardless of outcome. Eliminate the "we only publish stat sig results" baloney.
- Report confidence/credible intervals, adjusting for multiple comparisons as appropriate. Plot the posterior distribution of the effect size if appropriate.
- Publish all data and code.
- Provide funding for duplicating important studies.
I'd add one additional technique: Specification curve analyses. Bonferroni etc. alone won't help against systematic bias in a field and/or misconduct, and specification curves are easy to do, e.g. with specR [1].
[1] https://github.com/masurp/specr
[edit] you can get posteriors with bootstrapping etc. - it's not precisely the same but better than nothing.
https://slimemoldtimemold.com/2022/07/21/on-the-hunt-for-gin...
Hoeffding's inequality, Chebyshev's inequality, and Chernoff's inequality are broadly applicable, and are therefore less likely to be misused. They also don't require philosophical assumptions about subjective probabilities.
In the theory of Multi-Armed Bandit algorithms, compare the Bayesian approach (Thompson sampling) with the Concentration Inequality approach (the UCB algorithm). The former assumes that payoffs are in the 2-element set {0,1}, while the latter allows payoffs to be in the interval [0,1].
I think Bayesian is cool, but it imposes a big burden with choosing sensible priors.
Is a linear model 'better' than a regression developed using a tree method allowed to run over hundreds of parameters?
The amount of time I've been forced to take explaining random forest, and why we can trust it's R2 more than we can a linear regression is ridiculous.
They are definitely useful as part of bigger models that we cannot explicitly express them in multiple fields. Physics, chemistry, ML, they all utilize statistics to make useful models and that is great.
The issue is that decision making is a singular point, and statistics are very good at not taking any responsibility for their bad predictions.
If you make a decision with an expected outcome the statistician will come out swinging. If it was the unexpected outcome then you are faced with “Well you were unlucky, that’s statistics”.
I find the greatest proponents of Bayesian thinking to be in the rationalist community. So to get across just how wrong everyone is about just about everything pretty much always consider that the pejorative lessons on predictable irrationality and cognitive bias are delusions that come about through the flaws of statistical reasoning. From first principles we know that superhuman chess AI are also predictably irrational and cognitively biased, but that this isn't actually irrational or biased so as to desire us to abandon the choice. Even in chess - much simpler than reality - the correct solution was too large to fit within our universe.
It is difficult to articulate the magnitude of this humility. We couldn't fit the chess solution into our universe, but how complex is chess relative to other problems? If the complexity of chess was a grain of sand or even a single atom than it would be an understatement to claim that the more general problems are only as large as our universe.
'The "Null Ritual"', Marc Green https://www.visualexpert.com/Resources/nullritual.html