Depends how you model it. I would say not taking into account the publication bias if you have ways to measure or at least estimate its distribution is a bigger mistake to make.
This. Bayesian assumptions are explicit and you'll be forced to defend your priors. Frequentist assumptions like homoscedasticity are implicit and rarely challenged.
I wouldn't be surprised if many people using frequentist methods have no idea what assumptions need to be true for the methods to be valid. In a Bayesian framework you can't avoid the assumptions
That's why it's for constructing a prior that you change later; your study itself should update the prior in the direction of being more correct. But it's not like you can do better than "the best available prior" (by definition).
I've never heard anyone promote frequentist statistics. I've only ever heard Bayesians on the 'offense' talking about why their framework is better. To be clear I am someone who doesn't really understand the faultlines.
Indeed, when I took statistics in college, nearly 40 years ago, we proved Bayes theorem, and worked through several applications, so it could hardly have been regarded as controversial.
The controversy is decades old and largely resolved. Both methods work well on large enough data sets, Bayesian methods are far superior on small datasets
This statement doesn’t really make sense to me. A frequentist approach assumes a random (or noisy) underlying process and characterizes its probability. A Bayesian approach assumes a fixed but unknown process and updates the probability a given model matches that reality.
It is ultimately the same math, if you are comparing apples to apples. Just different ways of looking at a problem. Sometimes one is better suited than the other.
It’s like as if physicists were arguing over whether Cartesian or Polar coordinates were better. It’s the same damn physics, just expressed differently. In some problems one approach is easier to work with than the other, and can even make seemingly intractable problems solvable. But that doesn’t mean the other approach was “wrong.”
In many cases yes. A simple example where this is not true is if your populations have different variances. T tests are no longer valid but heterogenous variances can be modeled with a Bayesian approach
Bayes theorem is fairly elementary probability. It isn’t the part that is questioned in Bayesian statistics. Someone disagreeing with Bayes theorem would be like disagreeing with addition.
The controversial part is the methodology for statistical analysis built on top of it (e.g., you need a prior but where did that come from?)
This advantage can't be understated. Researchers, at least in psychology, almost never seem to care about the magnitude of an effect. It's simply enough to show that some effect happens in some direction. For this (generally acceptable) purpose, frequentist stats are great.
A statistically significant result without something like a Hedges G (or Cohens D if comparing means) measure would be called out by even a moderately attentive reviewer as a matter of course (especially if small sample sizes are involved), and many reputable journals actually require effect sizes to be stated, so you run the risk of being desk rejected for leaving it out.
For leaving it out perhaps, but the result is publishable even if the effect is small... where for significant you usually must p-hack your way to a standard threshold.
It's not entirely unreasonable, as often extra variables suppress the effect size. E.g. the effect is substantial in people with some genetics, insignificant otherwise. The fact that there is an effect at all makes it interesting as a starting point for further exploration e.g. to determine the mechanism or find the extra factors needed to make the effect significant.
> This advantage can't be understated. Researchers, at least in psychology, almost never seem to care about the magnitude of an effect. It's simply enough to show that some effect happens in some direction
That's not an advantage, that's one of the primary reasons psychology is among the worst subjects hit by the replication crisis.
A cheap shot that misunderstands the so called "crisis" in psychology, which almost comes entirely from the fact that brains are non randomly exchangeable, unique, samples differ, the brain is complex, multiple ways to solve problems, and sussing it all out is complex as hell.
If they are at least as knowledgeable as you, who correctly describes how messy you can expect the data to be, then you should be more concerned with effect size, because small effects are almost certainly just "noise". Instead, sensationalism about miniscule effects that vanish on replication attempts abound.
Sorry for this very slow response, but I actually disagree. I don’t think effect sizes are usually meaningful beyond the context of a specific experiment.
For example, suppose you are researching the effect of an emotional/negative picture on how a participant makes decision in an economic game. Here, you may think “a big effect size implies this study will have a large practical relevance.” However, following typical statistical designs, an effect size may be largely determined by just how many trials the experiment used for each participant. This is also not the only factor that’s relevant. For instance, it’s debatable how the emotion induced in the task compares to the intensity of real life emotional situations. With these factors in mind, the effect size really doesn’t tell you much that can be applied outside the laboratory.
Maybe in some non-experimental sub-fields (e.g. personality psychology) effect sizes are more meaningful…
Frequentist statistics is largely based on the maximum likelihood principle, which is a proper Bayesian analysis whenever the prior is a flat distribution. And since inference is mostly done on model parameters in frequentist statistics, the distribution you assume for them will also depend on how you parameterize the model. So it can often be justified as a convenient approximation to a "properly" Bayesian result.
I've probably heard 1000 Bayesians rant about the alleged Frequentist consensus and 0 Frequentists complain about Bayesian analysis. I'd need some pretty steep priors to interpret this as anything other than the academic equivalent of "one WEIRD trick THEY don't want you to know" marketing.
Bayesian statistics is sound but I suspect it's often just used to justify biases. It is technically valid to use a prior and de facto never update it, because you know I'll get around to updating my prior next week, or... eventually, cough cough let's be honest, never
I don't know. I've published Bayesian methodology papers and reviewed them and although I might call myself an objective Bayesianist maybe, Bayesianism has its own implicit biases. In the very least, I think it can take advantage of implicit biases people have in interpreting analyses.
One issue is that Bayes estimates are almost always produced, even if no information is coming from empirical data, and all the information is coming from a prior. So it's possible to produce results heavily influenced by the prior with Bayesian estimation that with frequentist methods would fail completely because of lack of identification of the model, sending a strong signal that something is wrong. This can all be sussed out with Bayesian methods but people often don't do it.
Another more subtle issue is people aren't quite aware of how a prior can deviate from "maximal conservativism". Sometimes, for example, depending on the model, a very flat prior is actually not conservative, and is overweighting tails.
There's other examples too. Basically, yes, Bayesianism forces you to be explicit with your biases, but people are really bad at interpreting the actual impact of those biases in a formal Bayesian framework, or at least, aren't any better at it than with frequentist methods that are available.
If you approach statistical inference from the perspective of accuracy (as the linked paper seems to do) Bayesianism is better to the extent the priors are accurate. This is true a lot of the time empirically, but it does lead to a kind of tautology, in that you're doing the analysis because you don't really know what "truth" is. So if you're right in your priors, Bayesianism is accurate, but then you didn't really need new data as much in the first place; if you're wrong, it's more biased. Basically in the bias-variance tradeoff, Bayesianism makes a bet on reduced variance assuming that the resulting bias will be small enough.
Philosophically, though, there's a completely different argument, which is one of competitive fairness. You might say this doesn't matter, but consider consequential decisions, like hiring or admissions decisions: if someone was making a prediction about you, would you want them to use a strong prior, or something that's maximally conservative and fair?
This philosophy leads to frequentism basically.
My preference is to be maximally conservative in a Bayesian framework, which leads to reference priors, which are often flat in many canonical situations, which is basically frequentism. In other situations you might have a different kind of prior.
To me the linked paper is pretty interesting and makes a good point. On the other hand, I'd rather not make any assumptions about a new result based on past studies on other effects. I'd rather just collect lots of diverse real data and meta-analyze it. There's no substitute for data -- and that includes priors.
Every single statistics class I've ever taken or been sent to has been 95% frequentist with a rushed Bayesian digression near the end. I've submitted papers with Bayesian work and the reviewer asked for p-values and would not budge. I gave him his bloody p-values, because I could not afford not to get the paper published that early in my career.
The "more complicated" version of frequentist statistics is called robust statistics. There's most likely a way to rephrase your favorite Bayesian analysis so as to make it fully kosher from a "robust+frequentist" point of view, even keeping the math unchanged. It just goes to show how silly the "controversy" is.
Bayesian statistics is fundamentally less complicated than Frequentist statistics since everything can be derived from a very simple set of first principles, rather than complex frameworks of ad hoc testing methodologies.
> Bayesian statistics is fundamentally less complicated than Frequentist statistics [...]
I broadly agree with you, but I'm wondering if you would reconsider your qualification as "less complicated" if you consider beginner learners. E.g. someone who knows basic descriptive statistics and probability theory, and is making first contact with inferential statistics. Specifically, assume a learner who knows what an integral is, but is far from proficient with it (UGRAD student, not a GRAD student).
I was reading this paper[1] recently, which highlights two difficulties of teaching Bayesian stats: 1) the mathematical complexity of understanding conditional probability distributions, and 2) the lack of well defined, broadly accepted conventions for what priors to use in specific data analysis scenarios.
I think a computational approach to prob theory could mitigate 1), but 2) remains a problem—the freedom to choose priors, is also a burden...
For context, I'm not taking sides in the frequentist-vs-Bayesian debate, but coming from a practical problem: I'm working on a INTRO TO STATS book right now, and I would like to include a chapter on Bayesian statistics, but I'm not sure what specific analyses to showcase and recommend as "standard" approaches. Unlike the canonical t-tests and ANOVAs of frequentists, Bayesian statistics doesn't have canonical procedures (that I know of) that I can wholeheartedly recommend as "must know" and ready to use broadly.
Maybe someone here might have suggestions?
The closest thing that comes to mind is "Bayes factors," which has some traction (usage), but apparently they have lots of problems and limitations too, cf. https://www.youtube.com/watch?v=MqeWpR6S4XA
Maybe you would find it useful to read a textbook on bayesian stats for inspiration. I can recommend Richard McElreath's "Statistical Rethinking" which makes it very clear how inflexible it is to just know recipes like t-tests or anovas.
The canonical approach is to build a generative model with a parameter (or multiple for ~anova) that codes for the difference between groups and do inference on that parameter of interest. Most of the recipes taught in statistics classes can be modelled as a regression of some kind (this counts for frequentist stats too, see https://lindeloev.github.io/tests-as-linear/ ). Some advocate to do that inference with bayes factors. Others, like discussed elsewhere in this thread, advocate combining the resulting posterior with a cost/value function, but either way the lesson is that there is less focus on "t-test-vs-anova" because they're the same thing anyways.
Thanks for the "Statistical Rethinking" recommendation. I had watched some of McElreath's lectures so I knew of the book, but next year I think I'm finally going to read it from end to end and follow along the 2024 course schedule: https://github.com/rmcelreath/stat_rethinking_2024#calendar-...
Yes, because Frequentists won a long time ago. And are responsible for producing garbage science ever since.
Of course the winners don't rant about anything. But any time we probe the consequences of Frequentist statistics they turn out to be horrific for science, our health, and our planet.
"Inverse probability" is just what statistical inference was called before frequentist ideas were introduced. It was in the second half of the last century when the "Bayesian" label was introduced - amusingly in opposition to the much newer "classical" methods.
I'd identify a larger problem, which is teaching anything as a bunch of recipes.
My college offered stats in two tracks: There was a one-semester course for science majors, which was mostly plugging numbers into formulas. The course was utterly baffling for most of the students who took it.
There was also a two-semester course for math majors. It was mainly about proofs, but also had time to go into more depth. You have to know the assumptions underlying a formula, if you're expected to prove it. ;-) But it was only taken by the math majors.
I’m not sure that teaching mathstats more broadly is the answer to frequentist approaches being used by default, or to the recipe problem. I did the whole mathstats thing as a student and spent a bunch of time doing calculus around things like maximum likelihood estimators or unbiased estimators. I don’t think it’s very useful for the application of statistics in practice. I would recommend Statistical Rethinking by Richard McElreath as a course in Bayesian statistics targeted at scientists who want to extract correct, useful conclusions from their experiments, which doesn’t follow the “recipe book” pattern.
I'm not sure of that either. For one thing, the students would be blown away.
One thing we didn't have when I took stats was computers. I graduated from high school in 1982, and the colleges in my state were just beginning to get computers. I wonder if stats could be approached differently if it could start with nothing but data -- lots of it -- and graphing tools. This could even happen pre-college.
I don’t know if it’s the “answer”, but it followed a pretty simple logistic curve for our curriculum at uni
First course was almost purely frequentist
Second was half and half with several weeks approaching the frequentist bayesian approach from both sides
Our third stats course was mostly bayesian though self directed as to which point of view was “most appropriate” for the formulated problem sets. I remember roughly half the cohort used a frequentist perspective on a fairly obvious (but not explicit) bayesian assignment and were severely punished by the course coordinator (fairly for a late year subject tbh)
But I don't see p-values as connected to frequentist statistics. Rather, p-values are a concept that applies to the problem of hypothesis testing, which both frequentist and bayesian statistics should apply to, right? And in both cases you would use a p-value. The unique thing that Bayesian statistics can do is fit parameters can calculate uncertainty in those parameters, which does not contradict any of the concepts of hypothesis testing.
A p-value is the frequency you’d expect to see a sample with an effect this size or greater, given the null hypothesis. It’s inherently a frequentist approach to inference.
You can squish it into Bayes by considering it a uniform prior on the real number line that you never update, but you’re not really doing things “in the spirit” of Bayes, then.
What is a “prior” in the context of hypothesis testing? That seems to me to be a concept used in parameter estimation, with Bayes rule. But hypothesis testing is not parameter estimation.
As the linked post points out, you can regularize a p-value based on prior experiments.
IMO, null hypothesis testing is way overused. We should be quantifying the comparison of the null versus alternative hypotheses. People already compare them. Might as well bring some math into it.
In Bayesian terms there is no distinction between hypothesis testing and parameter estimation.
If your hypothesis is that A is greater than B, then you're test boils down to "The probability that A is greater than B", which is arrived at through parameter estimation via Bayes' Theorem.
You can consider the distribution of estimates for a parameter to represent a space of possible hypothesis for the true value of that parameter and their absolutely or relative likelihood based on the information available.
The also has the pleasant consequence that hypothesis testing using Bayesian methods nearly always is closer to the actual question someone wants to answer rather than replying with confusing statements such as "the probability of rejecting the null hypothesis", which contains an implicit double negative.
> If your hypothesis is that A is greater than B, then you're test boils down to "The probability that A is greater than B", which is arrived at through parameter estimation via Bayes' Theorem.
but even in this situation you would need a threshold for decision making, i.e. a p-value.
"p-value" as a term already has a definition -- one that imposes a frequentist perspective -- so you'd have to come up with a new term to define a Bayesian analogue of that concept.
I fail to understand why the terminology matters. If you need to make a decision then there will be a threshold in the probability that will determine it. It doesn’t matter if you call it p values or anything else.
I'm not understanding your confusion. Different concepts require different words to distinguish them in conversation. p-value is a name, not merely a descriptor, of one possible way to define a threshold. Other ways of defining thresholds get different names so that everyone knows what's being discussed.
Bob bob bob'bob bob bob bob bob, bob bob bob bob bob "Bob".
I'm curious how much professional statistical experience you have? In my experience probabilities alone are very rarely used and distributions of beliefs are combined with distributions of value so you can choose optimal risk/reward scenarios.
Bayesian decision making typically involves combining a cost function with your posterior distribution.
It's worth pointing out that this is typically not possible in most frequentist frameworks since they work in double negative style assertions and arbitrary threshold rather than modeling the problem itself.
While you can map Bayesian approaches to frequentists tests, this is usually a misunderstanding of Bayesian methods coming from a purely frequentist background. Bayesian analysis is fundamentally more flexible since it just the application of the sum and product rules (you don't even need Bayes' theorem since it can be trivially derived from these) as well as corresponding cost/value functions.
I know this reply is a bit late, but, at least in my professional experience, Bayesian decision making typically does not involve just a probability of an event (in which thresholds are more or less arbitrary, including p-values) but involves mapping the probably to a cost/value in which the threshold isn't determined by arbitrary probability threshold but died to business value.
In the most common situations this typically means the expected value well exceeds the expected cost and you're comfortable with the risk if you're wrong.
> What is a “prior” in the context of hypothesis testing?
A prior could be any previous knowledge that you have on the subject, or anything about the world.
Let me give you a silly example. Imagine I do an experiment, on if person A can see the future, and predict coin flips.
Now, imagine that I do this experiment, and lo and behold it passes the 5% threshold P value!
Remember what a P value is. It is simply the percent chance that the results that you got were by chance.
So, if I do that experiment, and it actually gets at the "P value" of 5%, which means that there is a 5% chance of it happening by chance, do you know believe that Person A can see the future?
I wouldn't. I would need a much lower P value to start believing that.
Thats all a prior is. It is using the fact that "Person A can see the future!" is an extraordinary claim. And for that, we require extraordinary evidence. Much better evidence than a mere "5% that it happened by chance". P
> But I don't see p-values as connected to frequentist statistics.
P-values are fundamentally Frequentist because this framework see that observed data as random, whereas Bayesian statistics believe our observations are the only thing that is actually known, and all the other parameters are what is random in our experiments. That is: the data is the only part of an experiment that is real, everything else that we can't directly observe is something we must hypothesize about.
I went to a week-long Probabilistic AI seminar, and I got the impression there was an era when enough stats and math professors did not take a Bayesian framework seriously, that there was a risk someone's thesis or dissertation defense would be derailed by someone on faculty.
It's worth pointing out that the vast majority of "Bayesians" you see talking shit have no actual training in bayesian statistics, or any statistics training beyond the college-intro level.
> I've never heard anyone promote frequentist statistics.
At least in my undergrad education, I took two dedicated statistics classes and many more domain science classes that used statistics, all of which were so steeped in the frequentist paradigm that frequentist vs Bayesian wasn't even mentioned. Frequentist statistics simply was statistics for me, until grad school.
Statistical rethinking by McElreath is an exceptional resource for Bayesian modeling. It's very accessible and gives you some good examples to work through.
Some very basic Bayesian models can go a long way towards making informed decisions for a/b tests
Specifically for A/B or A/B/N testing, you can use a beta-bernoulli bandits, which give you confidence about which experience is best and will converge to an optimal experience faster than your standard hypothesis test. Challenges are that you have to frequently recompute which experience is best and thus, dynamically reallocate your traffic. They also only works on a single metric, so if your overall evaluation criterion isn’t just something like “clickthrough rate”, this type of testing becomes more difficult (if anyone else knows how multiple competing metrics are optimized with bandits, feel free to chime in).
Beta-Bernoulli multi-armed bandits (BB-MAB) are definitely a good way to get started on a Bayesian version of A/B testing where you've the additional benefit that your population is dynamically allocated to the most performant option (actually this dynamic allocation makes it similar to interim analysis [1] rather than vanilla A/B testing).
There are some caveats though - and I mention these from the experience of running such solutions on a large scale in production. First, BB-MAB can't adapt to context by design. They only look at click/no-click behavior across the population. So, if your population has two distinct segments - youth and elderly - who behave very differently wrt purchases, the BB-MAB won't pick a different winning advt. per group; its blind to these groups.
The solution is to use something like a contextual MAB - which assimilates user features (or whatever you might throw at it) into the MAB. There are simple ways to adapt simple MABs to the contextual setup [2] (in my experience, these can also be effective) but, of course, the literature in this area is wide and deep.
A second caveat is that if the ratio of the size of the pool of advts. to the number of impressions is high, the BB-MAB won't converge or converge to a good optima; the search space is simply too large relative to the data. In cases like this it becomes important to begin with the right Beta priors, instead of the standard recipe of starting with a Beta that looks like a uniform distribution.
And somehere down the line he will introduce the Beta/Binomial method for A/B testing.
In my (again humble) understanding the benefit of doing this in the Bayesian way is both that you can actually get an understandable answer, but also that the answer does not have to be the final result you can continue by adding a loss function for example.
I don't really see Frequentism and Bayes to be in conflict. You use them to answer different questions. I wouldn't test a point null hypothesis with a Bayesian model.
I think most Bayesians would claim that point null hypothesis tests are immoral or something like that, rather than propose to do them with a Bayesian model (thus avoiding measure zero problems)
I never really got Bayesian statistics to be honest.
- When sample size grows, frequentist and bayesian (if the prior is not too restrictive) point estimates seem to converge to each other anyway
- The distribution of your point estimate (frequentist) vs. the estimated distribution (bayesian) also don't seem to differ too much either
- When the sample size is small the Bayesian prior dominates
- Interestingly, when I see Bayesians simulate random data (to introduce the concepts on this data) they usually assume a true parameter value. E.g. when sampling from Y = a + b * X + e, they'll assume fixed, true values of a and b and not random variables - which is a frequentist assumption! So far I've never seen e.g. b being sampled from Normal(mu=2, sigma=1) instead of just setting b=2.
- The frequentist assumption of a true population value which we try to estimate just makes sense to me. For example there is a true mean income over the working population. It's not a random variable but a fixed value which can be computed if we just asked every single working person for their income and then compute the mean over all values.
I tried getting into Bayesian stats but honestly it just seems overkill for most cases. For a simple regression computing b_hat = inv(XX')Y is just faster and easier than numerically sampling traces. Bayesian forces you to think about the data generating process - I appreciate that, but you need to the same when it comes to frequentist stats, it's just a little less obvious.
> When sample size grows, frequentist and bayesian [...] estimates seem to converge to each other anyway
Yes. And so? Bayesians would argue (and I quote) that "the interesting limit in statistics is when the number of samples tends to one. The limit when the number of samples tends to infinity is completely useless."
> I tried getting into Bayesian stats but honestly it just seems overkill for most cases.
There are 3 black balls and 7 white balls in an opaque bag. How likely is it to pick a black ball? Bayesian statistics gives a straightforward answer (you just assume an uninformative prior and perform a computation). But frequentist statistics starts to argue about an infinite number of replicas of your own universe and other nonsensical constructions. Not sure that the Bayesian approach is overkill in that case...
> Yes. And so? Bayesians would argue (and I quote) that "the interesting limit in statistics is when the number of samples tends to one. The limit when the number of samples tends to infinity is completely useless."
The "and so?" is answered right after that. The prior dominates, which is a bad thing.
As the amount of data tends to 0 (idk why the quote is using 1), if course your belief tends to whatever your belief was before you saw any data. What else could it possibly tend to? Of course it's very sad that we don't have any data, but that's no fault of Bayesian.
> As the amount of data tends to 0 (idk why the quote is using 1)
The smallest amount of samples you can use is 1, isn't it? If you have 0 samples then you do nothing because you have no data. Is there a way to have half a sample?
> if course your belief tends to whatever your belief was before you saw any data
Your beliefs should tend to that, sure, but if you're trying to produce an actual number for sharing then your beliefs shouldn't be a huge factor, and an uninformative prior being a huge factor is also bad.
For numbers that leave my head/notebook, I'd rather keep the new evidence by itself and say it's weak.
Does Bayesian have a concept for absence of belief? I don't feel like believing anything is equally likely is equivalent to absence of belief. But maybe it is?
There is a concept of minimum knowledge (maximum entropy). There is a concept of invariance (like translation invariance where you have no reason to prefer one position to another because the origin could be anywhere - or scale invariance where the value of a magnitude could be high or low if you don't know anything about the unit of measurement).
I'm not sure if by "absence of belief" you mean "ignorance" or something else.
I think about something like known ignorance. I know that I don't know anything about this thus I refuse to have any belief about what it might be as a I know any belief would be unwarranted.
You need at least something to be ignorant about but for a given "this" you can specify what you do know and calculate a probability distribution representing just that knowledge avoiding any unwarranted belief.
If you have a die and you don't know anything else about it you should assume that the probability for each side is 1/6.
If you also know that the expected value is 4 (instead of 3.5 for a fair die) there is a way to calculate the probability distribution that reflects that constraint - and nothing else.
Now, if you don't even want to think about anything Bayesians can do that too.
It's just weird that the end result depends on something assumed. It reminds of LLMs that can't really express absence of knowledge so they make stuff up kind of assuming they know something.
Frequentist approach gives you something solid and independent of assumptions. Probability of observing this particular dataset accidentally if there was no change between two contexts.
On the positive side, the frequentist approach doesn’t need assumptions about the pre-data probability of the thing of interest.
On the negative side, the frequentist approach doesn’t produce a post-data probability for the thing of interest either.
It provides the probability of something else - as you mention - which can also be interesting but it’s not what people really would like to know (as the generalized misinterpretation of the meaning of frequentist results makes clear).
> It's just weird that the end result depends on something assumed.
It would be weirder if the result didn’t depend on the things assumed.
I don’t know what kind of questions are you thinking of but outside of mathematics they are rarely fully specified.
If the answer changes enough depending on the additional assumptions to seem weird that is a sign that the question was not completely clear.
Of course Bayesians can also say that there is not enough information to provide an answer when that’s the case, just like they can make additional assumptions explicit to provide one.
Note that there is no way to answer the question “what’s the probability of X conditional on the data observed” without taking into account “what was the probability of X before that observation”.
The thing with non-Bayesian analysis is that they don’t answer at all the question “what’s the probability of X conditional on the data observed”.
You said that the prior dominating is a bad thing. The prior is your beliefs about a parameter prior to observing data (as I suspect you know!). Maybe I'm not getting what you're saying.
I was assuming these are general purpose statistics, which means you might want to share them with someone. It's bad for those to get tainted by your personal priors. If it's a purely personal calculation then sure that's fine.
Let's say you have a personal belief that something is going to happen with probability x. Would you actually want to tell others that the probability is y, because that's what the data says, without letting people know that for other reasons that are not reflected in the data, you truly believe it is x?
That's fine and I would agree - you could share the summary statistic used, or the likelihood ratio between the null and some alternative models.
But you shouldn't share a frequentist parameter estimate or confidence interval if you have prior information that would influence it non-negligibly, at least not without sharing that prior information also.
Should you have any beliefs in the absence of data? And if you have some prior data but no additional data now why carve out past as separate thing and call it prior? Why not just call everything you have - data?
That's precisely how Bayesian inference works! But rather than having to repeat all analysis of prior data sets, we summarize that analysis in the form of a posterior, which becomes the prior for the next analysis.
I think the attraction of Bayesianism is kind of philosophical / aesethetical, it is is principled and sound and beautiful approach. It's kinda nice that it kinda extends and translates occams razors into numbers.
Yes frequentist statistics work very well in practice, but it's a bit adhoc and suffers from various problems like say if you estimate velocity and estimate kinetic energy, you get values that are incompatible which is kinda ugly and non-intuitive and makes you want to dig deeper into how such a thing happened.
Bayesianism has the answers.
Also sometimes it really does matter like in medicine, where some conditions have a very low prior probability.
I mean that's what's so good about them. The prior is a bit arbitrary and up for discussion. And Bayesianism is honest about it, it highlights that fact rather than trying to play fast and loose and sweep the issue under the rug.
The Bayesian formula is dreadfully useful in machine learning, in the modeling of generative problems. However because integration (in calculus) is computationally intractable, we usually have to use approximations instead of true bayesian stats.
> Interestingly, when I see Bayesians simulate random data (to introduce the concepts on this data) they usually assume a true parameter value. E.g. when sampling from Y = a + b * X + e, they'll assume fixed, true values of a and b and not random variables - which is a frequentist assumption! So far I've never seen e.g. b being sampled from Normal(mu=2, sigma=1) instead of just setting b=2.
The Bayesian philosophy of "random parameters" does not mean that Bayesian methods cannot be assessed for frequentist properties or compared against frequentist procedures.
Bayesian statistics, the way Andrew Gelman practices it, comes naturally when you are interested in generative models of data. You can still use maximum likelihood estimates, but these become fragile when you have hierarchical / multilevel models.
Multilevel models are fantastic to address a problem that is often ignored by frequentist approaches, the need for shrinkage and information sharing. This pops up all the time in modern statistics. For example, if you test 1000 hypotheses, calculating p-values and adjusting these with some multiplicity correction scheme is not sufficient.
You should borrow information across random variables with a multilevel model to avoid estimating exaggerated effects in tests whose outcome is deemed to be significant. Andrew Gelman's post is concerned with this topic.
Another point is that Gelman et al. use weakly informative hyperpriors. These are not really subjective. If anything, they usually regularize solutions by pushing effects towards zero. Plus, on multilevel models, priors are only needed on hyperparameters.
I use mixed level models for longitudinal analysis pretty regularly. There the point has been to account for correlated dependent observations (e.g. repeated variables within a participant.
However it seems that you are suggesting another use. If I have 10 cognitive measures each measured once in my subjectd, the default has been to do a multiple comparison correction, either FDR or FWER on 10 tests. We know that the 10 tests are not truly independent, so Bonferroni is probably too conservative.
It seems here you suggest running this with test being a random effect. I've seen this approach with item level data in a task, but I didn't really think to do it when the tests are not from the same battery, construct. And more to the point, this fixed effect model would be of no particular interest, while random effect CIs are difficult to estimate. So I am left a bit confused.
> The distribution of your point estimate (frequentist) vs. the estimated distribution (bayesian)
Ideally one should use the whole posterior distribution of your model parameters which is not the case for point estimates.
>So far I've never seen
Because people are lazy.
Bayesian works great if you have great knowledge in your field and you can fine tune everything. Frequentist stats just works and easily interpretable but easy to make mistakes esp. when starting out.
> Ideally one should use the whole posterior distribution of your model parameters which is not the case for point estimates.
This is a historical issue because of some hard-headed frequentist founders, but in modern days the frequentist concept of confidence distribution is gaining acceptance, which is the proper frequentist equivalent of the posterior, so this distinction between Bayesian and Frequentist is disappearing.
Rather than giving specific point estimates or interval estimates, calculating a frequentist confidence distribution allows you to compute confidence intervals for all possible confidence levels, just like the posterior does. See this excellent review paper for more info on this: https://statweb.rutgers.edu/mxie/RCPapers/insr.12000.pdf
The key insights is that a confidence distribution is an estimator for the parameter of interest, instead of an inherent distribution of the parameter.
The confidence distribution is generally derived from normalizing a likelihood function, and the likelihood function is arguably the proper underlying concept that provides a link to both Bayesian and frequentist inference, per https://en.wikipedia.org/wiki/Likelihood_principle
> the frequentist concept of confidence distribution is gaining acceptance, which is the proper frequentist equivalent of the posterior, so this distinction between Bayesian and Frequentist is disappearing
The major distinction remains: Frequentist confidence intervals are something quite different from Bayesian credible intervals. I don't think that having a distribution that can be used to calculate any desired confidence interval - like the posterior distribution can be used to calculate different credible intervals - changes much.
It's always struck me as rather strange that since the motive for creating any kind of model is to calculate predictions, and that the most general kind of calculation is algorithmic, people use anything but algorithmic probability as the gold standard against which other approaches are compared. The "problems" with algorithmic probability (uncomputability, UTM "choice" etc.) seem to be "the dog ate my homework" excuses. No scientific model is required to prove itself to be the best of all possible models relative to a given set of observations in order to be considered the best current model relative to those observations. No "UTM" chosen on the basis of the observations to be modeled is reasonably considered anything but post-hoc theorizing.
I had trouble grasping difference between frequentist vs bayesian. After googling frantically I found this article which turned out to be very informative about how bayesian analysis of a case might look like vs frequentist one:
I didn't understand much from the part where they use Markov chains to calculate probability of something that is hard to know but the rest illustrates differences really well.
There is a reproduceability crisis in published literature. Most of these study results rely on Frequentist statistics. Many of these studies are just not repeatable, or worse, clearly wrong. Use of Bayesian statistics might have averted this situation.
I’m really not a fan of Bayesian philosophy. The concept of a prior is kind of preposterous. It’s used consistently to make arguments that basically go “if you assume a prior it proves a probability of zero/1”. The concept of belief is slippery, all that matters is measured probability from data. Anything else is just speculation. In this case, using prior trial data to inflate sample size is entirely counter to the scientific method. You’re basically just reinforcing whatever conclusions previous trials came to. You’re potentially setting yourself up for a string of false positive trial results. If your first 20 trials have a hidden error in experiment design, future studies will continue to exaggerate the effect to an even greater degree as time goes on. I really wish this type of thinking never started in the first place.
The choice is not between using or not using priors, but between admitting it or denying it. A calculation that "does not use priors" is usually mathematically equivalent to one that does, for some specific prior.
The central limit theorem lets us calculate confidence intervals around population statistics regardless of the underlying distribution. If you're talking about working with small datasets, or speculating about the probability of events that have never occurred (n = 0), then just admit that math does not apply and that you're speculating.
The problem with frequentist confidence intervals is that they don't really tell us anything about the probability that the statistic of interest for our population is contained in the interval - regardless of the size of our dataset.
Not everyone wants to admit - or even understands - that the % confidence is about how frequently such intervals will contain the true value when the procedure is applied to other populations / datasets.
Could you explain what the difference is between the two following segments in your comment:
"[frequentist confidence intervals] don't really tell us anything about the probability that the statistic of interest for our population is contained in the interval"
and
"the % confidence is about how frequently such intervals will contain the true value when the procedure is applied to other populations / datasets"
My understanding is that a 95% confidence interval implies that there is a 95% chance the true statistic lies within the interval. What are you saying it means?
To some degree I'm interested in the philosophy behind probability but there are limits to my concern with it. If you're saying that ontologically this is not truly a 95% confidence interval, but practically speaking, it does mean that, then I'm more interested in the practical interpretation because it's more relevant to applications of statistics.
> My understanding is that a 95% confidence interval implies that there is a 95% chance the true statistic lies within the interval. What are you saying it means?
If you calculate one thousand confidence intervals in 950 the true statistic will lie within the interval. I'm not sure if that means that there is a 95% chance the true statistic lies within a given interval.
Let me use a extreme example to highlight the issue. Imagine that I use the following procedure to produce a 95% confidence interval for whatever quantity you're interested in:
with 95% probability return the interval [ -1e999 1e999 ]
with 5% probability return the interval [ -1e999 -1e999+1 ]
If you calculate one thousand confidence intervals in 950 the true statistic will lie within the interval. I wouldn't say in any given case that there is a 95% chance the true statistic lies within that particular interval. (There was a 95% chance though.)
This article (open access PDF available) discusses how different methods of producing confidence intervals perform in a toy problem:
One could argue that those misleading flaws can be avoided by being careful with what confidence intervals do you use and how - but the point is that coverage probabilities are about a set of counterfactual situations and not about the actual situation at hand.
I explicitely said I'm mostly interested in practical applications, and your example has e999 in it. Thank you for the PDF. I'll take a look at it, but it's really not helping the Bayesian case to start with an example like that. Is the assumption that the interval has a 95% chance of containing the true statistic going to lead to a bridge collapsing or an incorrect financial projection or not?
139 comments
[ 4.0 ms ] story [ 226 ms ] threadOne of the strengths of Bayesian methods is that you can use a less informative prior to model your skepticism of prior work.
[1]: https://www.sciencedirect.com/topics/medicine-and-dentistry/....
I wouldn't be surprised if many people using frequentist methods have no idea what assumptions need to be true for the methods to be valid. In a Bayesian framework you can't avoid the assumptions
It is ultimately the same math, if you are comparing apples to apples. Just different ways of looking at a problem. Sometimes one is better suited than the other.
It’s like as if physicists were arguing over whether Cartesian or Polar coordinates were better. It’s the same damn physics, just expressed differently. In some problems one approach is easier to work with than the other, and can even make seemingly intractable problems solvable. But that doesn’t mean the other approach was “wrong.”
The controversial part is the methodology for statistical analysis built on top of it (e.g., you need a prior but where did that come from?)
I will do it:
1. Computationally easier
2. often analytical theory available for most use cases so interpretability is high
3. more literature available so you can get unstuck faster if you mess up
4. no accusations of subjective bias in your prior (the con is clear, no ability to leverage subjective expertise)
5. In the asymptotic regime, MLE and bayesian MAP often converge anyways
6. king of hypothesis testing
For most people, it doesn't matter. It matters when you are doing treatment for small sample sizes or other situations that would cause low power.
This advantage can't be understated. Researchers, at least in psychology, almost never seem to care about the magnitude of an effect. It's simply enough to show that some effect happens in some direction. For this (generally acceptable) purpose, frequentist stats are great.
It's not entirely unreasonable, as often extra variables suppress the effect size. E.g. the effect is substantial in people with some genetics, insignificant otherwise. The fact that there is an effect at all makes it interesting as a starting point for further exploration e.g. to determine the mechanism or find the extra factors needed to make the effect significant.
That's not an advantage, that's one of the primary reasons psychology is among the worst subjects hit by the replication crisis.
So not a cheap shot at all IMO.
For example, suppose you are researching the effect of an emotional/negative picture on how a participant makes decision in an economic game. Here, you may think “a big effect size implies this study will have a large practical relevance.” However, following typical statistical designs, an effect size may be largely determined by just how many trials the experiment used for each participant. This is also not the only factor that’s relevant. For instance, it’s debatable how the emotion induced in the task compares to the intensity of real life emotional situations. With these factors in mind, the effect size really doesn’t tell you much that can be applied outside the laboratory.
Maybe in some non-experimental sub-fields (e.g. personality psychology) effect sizes are more meaningful…
Bayesian statistics is sound but I suspect it's often just used to justify biases. It is technically valid to use a prior and de facto never update it, because you know I'll get around to updating my prior next week, or... eventually, cough cough let's be honest, never
It's the frequentist one that gets your biases implicitly, on the form of corrections and hypothesis formulation, so that people don't notice them.
One issue is that Bayes estimates are almost always produced, even if no information is coming from empirical data, and all the information is coming from a prior. So it's possible to produce results heavily influenced by the prior with Bayesian estimation that with frequentist methods would fail completely because of lack of identification of the model, sending a strong signal that something is wrong. This can all be sussed out with Bayesian methods but people often don't do it.
Another more subtle issue is people aren't quite aware of how a prior can deviate from "maximal conservativism". Sometimes, for example, depending on the model, a very flat prior is actually not conservative, and is overweighting tails.
There's other examples too. Basically, yes, Bayesianism forces you to be explicit with your biases, but people are really bad at interpreting the actual impact of those biases in a formal Bayesian framework, or at least, aren't any better at it than with frequentist methods that are available.
If you approach statistical inference from the perspective of accuracy (as the linked paper seems to do) Bayesianism is better to the extent the priors are accurate. This is true a lot of the time empirically, but it does lead to a kind of tautology, in that you're doing the analysis because you don't really know what "truth" is. So if you're right in your priors, Bayesianism is accurate, but then you didn't really need new data as much in the first place; if you're wrong, it's more biased. Basically in the bias-variance tradeoff, Bayesianism makes a bet on reduced variance assuming that the resulting bias will be small enough.
Philosophically, though, there's a completely different argument, which is one of competitive fairness. You might say this doesn't matter, but consider consequential decisions, like hiring or admissions decisions: if someone was making a prediction about you, would you want them to use a strong prior, or something that's maximally conservative and fair?
This philosophy leads to frequentism basically.
My preference is to be maximally conservative in a Bayesian framework, which leads to reference priors, which are often flat in many canonical situations, which is basically frequentism. In other situations you might have a different kind of prior.
To me the linked paper is pretty interesting and makes a good point. On the other hand, I'd rather not make any assumptions about a new result based on past studies on other effects. I'd rather just collect lots of diverse real data and meta-analyze it. There's no substitute for data -- and that includes priors.
I broadly agree with you, but I'm wondering if you would reconsider your qualification as "less complicated" if you consider beginner learners. E.g. someone who knows basic descriptive statistics and probability theory, and is making first contact with inferential statistics. Specifically, assume a learner who knows what an integral is, but is far from proficient with it (UGRAD student, not a GRAD student).
I was reading this paper[1] recently, which highlights two difficulties of teaching Bayesian stats: 1) the mathematical complexity of understanding conditional probability distributions, and 2) the lack of well defined, broadly accepted conventions for what priors to use in specific data analysis scenarios.
I think a computational approach to prob theory could mitigate 1), but 2) remains a problem—the freedom to choose priors, is also a burden...
[1] https://www.stat.purdue.edu/~dsmoore/articles/BayesPedagogy....
Maybe someone here might have suggestions?
The closest thing that comes to mind is "Bayes factors," which has some traction (usage), but apparently they have lots of problems and limitations too, cf. https://www.youtube.com/watch?v=MqeWpR6S4XA
The canonical approach is to build a generative model with a parameter (or multiple for ~anova) that codes for the difference between groups and do inference on that parameter of interest. Most of the recipes taught in statistics classes can be modelled as a regression of some kind (this counts for frequentist stats too, see https://lindeloev.github.io/tests-as-linear/ ). Some advocate to do that inference with bayes factors. Others, like discussed elsewhere in this thread, advocate combining the resulting posterior with a cost/value function, but either way the lesson is that there is less focus on "t-test-vs-anova" because they're the same thing anyways.
I had previously started the BDA course, which is another famous Bayesian course, see https://avehtari.github.io/BDA_course_Aalto/ but I didn't finish it due to travel.
No more excuses in 2024... time to level-up the Bayesian modelling skill ;)
Of course the winners don't rant about anything. But any time we probe the consequences of Frequentist statistics they turn out to be horrific for science, our health, and our planet.
It’s like how you wouldn’t hear people in the US promote imperial measurements for home baking. When you dominate, you don’t have to advocate.
My college offered stats in two tracks: There was a one-semester course for science majors, which was mostly plugging numbers into formulas. The course was utterly baffling for most of the students who took it.
There was also a two-semester course for math majors. It was mainly about proofs, but also had time to go into more depth. You have to know the assumptions underlying a formula, if you're expected to prove it. ;-) But it was only taken by the math majors.
One thing we didn't have when I took stats was computers. I graduated from high school in 1982, and the colleges in my state were just beginning to get computers. I wonder if stats could be approached differently if it could start with nothing but data -- lots of it -- and graphing tools. This could even happen pre-college.
First course was almost purely frequentist
Second was half and half with several weeks approaching the frequentist bayesian approach from both sides
Our third stats course was mostly bayesian though self directed as to which point of view was “most appropriate” for the formulated problem sets. I remember roughly half the cohort used a frequentist perspective on a fairly obvious (but not explicit) bayesian assignment and were severely punished by the course coordinator (fairly for a late year subject tbh)
You can squish it into Bayes by considering it a uniform prior on the real number line that you never update, but you’re not really doing things “in the spirit” of Bayes, then.
IMO, null hypothesis testing is way overused. We should be quantifying the comparison of the null versus alternative hypotheses. People already compare them. Might as well bring some math into it.
If your hypothesis is that A is greater than B, then you're test boils down to "The probability that A is greater than B", which is arrived at through parameter estimation via Bayes' Theorem.
You can consider the distribution of estimates for a parameter to represent a space of possible hypothesis for the true value of that parameter and their absolutely or relative likelihood based on the information available.
The also has the pleasant consequence that hypothesis testing using Bayesian methods nearly always is closer to the actual question someone wants to answer rather than replying with confusing statements such as "the probability of rejecting the null hypothesis", which contains an implicit double negative.
but even in this situation you would need a threshold for decision making, i.e. a p-value.
Bob bob bob'bob bob bob bob bob, bob bob bob bob bob "Bob".
Bayesian decision making typically involves combining a cost function with your posterior distribution.
It's worth pointing out that this is typically not possible in most frequentist frameworks since they work in double negative style assertions and arbitrary threshold rather than modeling the problem itself.
While you can map Bayesian approaches to frequentists tests, this is usually a misunderstanding of Bayesian methods coming from a purely frequentist background. Bayesian analysis is fundamentally more flexible since it just the application of the sum and product rules (you don't even need Bayes' theorem since it can be trivially derived from these) as well as corresponding cost/value functions.
In the most common situations this typically means the expected value well exceeds the expected cost and you're comfortable with the risk if you're wrong.
A prior could be any previous knowledge that you have on the subject, or anything about the world.
Let me give you a silly example. Imagine I do an experiment, on if person A can see the future, and predict coin flips.
Now, imagine that I do this experiment, and lo and behold it passes the 5% threshold P value!
Remember what a P value is. It is simply the percent chance that the results that you got were by chance.
So, if I do that experiment, and it actually gets at the "P value" of 5%, which means that there is a 5% chance of it happening by chance, do you know believe that Person A can see the future?
I wouldn't. I would need a much lower P value to start believing that.
Thats all a prior is. It is using the fact that "Person A can see the future!" is an extraordinary claim. And for that, we require extraordinary evidence. Much better evidence than a mere "5% that it happened by chance". P
P-values are fundamentally Frequentist because this framework see that observed data as random, whereas Bayesian statistics believe our observations are the only thing that is actually known, and all the other parameters are what is random in our experiments. That is: the data is the only part of an experiment that is real, everything else that we can't directly observe is something we must hypothesize about.
At least in my undergrad education, I took two dedicated statistics classes and many more domain science classes that used statistics, all of which were so steeped in the frequentist paradigm that frequentist vs Bayesian wasn't even mentioned. Frequentist statistics simply was statistics for me, until grad school.
http://www.stat.columbia.edu/~gelman/research/published/pval...
I would love to embrace (or at least try to) such a new approach, but it feels like without a PhD in stats it's hard to get it started.
Some very basic Bayesian models can go a long way towards making informed decisions for a/b tests
There are some caveats though - and I mention these from the experience of running such solutions on a large scale in production. First, BB-MAB can't adapt to context by design. They only look at click/no-click behavior across the population. So, if your population has two distinct segments - youth and elderly - who behave very differently wrt purchases, the BB-MAB won't pick a different winning advt. per group; its blind to these groups.
The solution is to use something like a contextual MAB - which assimilates user features (or whatever you might throw at it) into the MAB. There are simple ways to adapt simple MABs to the contextual setup [2] (in my experience, these can also be effective) but, of course, the literature in this area is wide and deep.
A second caveat is that if the ratio of the size of the pool of advts. to the number of impressions is high, the BB-MAB won't converge or converge to a good optima; the search space is simply too large relative to the data. In cases like this it becomes important to begin with the right Beta priors, instead of the standard recipe of starting with a Beta that looks like a uniform distribution.
[1] https://en.wikipedia.org/wiki/Interim_analysis
[2] https://arxiv.org/abs/1811.04383
https://www.youtube.com/watch?v=QllfKQH-yQ4
But I wanted to say that when I looked A/B testing few years ago I started off with this book.
https://github.com/CamDavidsonPilon/Probabilistic-Programmin...
And somehere down the line he will introduce the Beta/Binomial method for A/B testing.
In my (again humble) understanding the benefit of doing this in the Bayesian way is both that you can actually get an understandable answer, but also that the answer does not have to be the final result you can continue by adding a loss function for example.
For Bayesian ML see https://probml.github.io/pml-book/book1.html and https://www.bishopbook.com/
For Bayesian statistics see http://www.stat.columbia.edu/~gelman/book/
- When sample size grows, frequentist and bayesian (if the prior is not too restrictive) point estimates seem to converge to each other anyway
- The distribution of your point estimate (frequentist) vs. the estimated distribution (bayesian) also don't seem to differ too much either
- When the sample size is small the Bayesian prior dominates
- Interestingly, when I see Bayesians simulate random data (to introduce the concepts on this data) they usually assume a true parameter value. E.g. when sampling from Y = a + b * X + e, they'll assume fixed, true values of a and b and not random variables - which is a frequentist assumption! So far I've never seen e.g. b being sampled from Normal(mu=2, sigma=1) instead of just setting b=2.
- The frequentist assumption of a true population value which we try to estimate just makes sense to me. For example there is a true mean income over the working population. It's not a random variable but a fixed value which can be computed if we just asked every single working person for their income and then compute the mean over all values.
I tried getting into Bayesian stats but honestly it just seems overkill for most cases. For a simple regression computing b_hat = inv(XX')Y is just faster and easier than numerically sampling traces. Bayesian forces you to think about the data generating process - I appreciate that, but you need to the same when it comes to frequentist stats, it's just a little less obvious.
Yes. And so? Bayesians would argue (and I quote) that "the interesting limit in statistics is when the number of samples tends to one. The limit when the number of samples tends to infinity is completely useless."
> I tried getting into Bayesian stats but honestly it just seems overkill for most cases.
There are 3 black balls and 7 white balls in an opaque bag. How likely is it to pick a black ball? Bayesian statistics gives a straightforward answer (you just assume an uninformative prior and perform a computation). But frequentist statistics starts to argue about an infinite number of replicas of your own universe and other nonsensical constructions. Not sure that the Bayesian approach is overkill in that case...
The "and so?" is answered right after that. The prior dominates, which is a bad thing.
The smallest amount of samples you can use is 1, isn't it? If you have 0 samples then you do nothing because you have no data. Is there a way to have half a sample?
> if course your belief tends to whatever your belief was before you saw any data
Your beliefs should tend to that, sure, but if you're trying to produce an actual number for sharing then your beliefs shouldn't be a huge factor, and an uninformative prior being a huge factor is also bad.
For numbers that leave my head/notebook, I'd rather keep the new evidence by itself and say it's weak.
I'm not sure if by "absence of belief" you mean "ignorance" or something else.
If you have a die and you don't know anything else about it you should assume that the probability for each side is 1/6.
If you also know that the expected value is 4 (instead of 3.5 for a fair die) there is a way to calculate the probability distribution that reflects that constraint - and nothing else.
Now, if you don't even want to think about anything Bayesians can do that too.
On the negative side, the frequentist approach doesn’t produce a post-data probability for the thing of interest either.
It provides the probability of something else - as you mention - which can also be interesting but it’s not what people really would like to know (as the generalized misinterpretation of the meaning of frequentist results makes clear).
It would be weirder if the result didn’t depend on the things assumed.
I don’t know what kind of questions are you thinking of but outside of mathematics they are rarely fully specified.
If the answer changes enough depending on the additional assumptions to seem weird that is a sign that the question was not completely clear.
Of course Bayesians can also say that there is not enough information to provide an answer when that’s the case, just like they can make additional assumptions explicit to provide one.
I know it's not like that. But it's still weird that at the end of Bayesian analysis the best you can deliver is if-by-whiskey style deliberation.
I know it's still valuable. Just weird.
The thing with non-Bayesian analysis is that they don’t answer at all the question “what’s the probability of X conditional on the data observed”.
Let's say you have a personal belief that something is going to happen with probability x. Would you actually want to tell others that the probability is y, because that's what the data says, without letting people know that for other reasons that are not reflected in the data, you truly believe it is x?
Your informed opinion incorporates this dataset, but you shouldn't imply it's "based on" this dataset.
But you shouldn't share a frequentist parameter estimate or confidence interval if you have prior information that would influence it non-negligibly, at least not without sharing that prior information also.
Yes frequentist statistics work very well in practice, but it's a bit adhoc and suffers from various problems like say if you estimate velocity and estimate kinetic energy, you get values that are incompatible which is kinda ugly and non-intuitive and makes you want to dig deeper into how such a thing happened.
Bayesianism has the answers.
Also sometimes it really does matter like in medicine, where some conditions have a very low prior probability.
That's how many people feel about Bayesian methods when trying to pick an initial prior.
The Bayesian philosophy of "random parameters" does not mean that Bayesian methods cannot be assessed for frequentist properties or compared against frequentist procedures.
Multilevel models are fantastic to address a problem that is often ignored by frequentist approaches, the need for shrinkage and information sharing. This pops up all the time in modern statistics. For example, if you test 1000 hypotheses, calculating p-values and adjusting these with some multiplicity correction scheme is not sufficient.
You should borrow information across random variables with a multilevel model to avoid estimating exaggerated effects in tests whose outcome is deemed to be significant. Andrew Gelman's post is concerned with this topic.
Another point is that Gelman et al. use weakly informative hyperpriors. These are not really subjective. If anything, they usually regularize solutions by pushing effects towards zero. Plus, on multilevel models, priors are only needed on hyperparameters.
However it seems that you are suggesting another use. If I have 10 cognitive measures each measured once in my subjectd, the default has been to do a multiple comparison correction, either FDR or FWER on 10 tests. We know that the 10 tests are not truly independent, so Bonferroni is probably too conservative.
It seems here you suggest running this with test being a random effect. I've seen this approach with item level data in a task, but I didn't really think to do it when the tests are not from the same battery, construct. And more to the point, this fixed effect model would be of no particular interest, while random effect CIs are difficult to estimate. So I am left a bit confused.
Ideally one should use the whole posterior distribution of your model parameters which is not the case for point estimates.
>So far I've never seen
Because people are lazy.
Bayesian works great if you have great knowledge in your field and you can fine tune everything. Frequentist stats just works and easily interpretable but easy to make mistakes esp. when starting out.
This is a historical issue because of some hard-headed frequentist founders, but in modern days the frequentist concept of confidence distribution is gaining acceptance, which is the proper frequentist equivalent of the posterior, so this distinction between Bayesian and Frequentist is disappearing.
Rather than giving specific point estimates or interval estimates, calculating a frequentist confidence distribution allows you to compute confidence intervals for all possible confidence levels, just like the posterior does. See this excellent review paper for more info on this: https://statweb.rutgers.edu/mxie/RCPapers/insr.12000.pdf
The key insights is that a confidence distribution is an estimator for the parameter of interest, instead of an inherent distribution of the parameter.
The major distinction remains: Frequentist confidence intervals are something quite different from Bayesian credible intervals. I don't think that having a distribution that can be used to calculate any desired confidence interval - like the posterior distribution can be used to calculate different credible intervals - changes much.
https://www.redjournal.org/article/S0360-3016(21)03256-9/ful...
I didn't understand much from the part where they use Markov chains to calculate probability of something that is hard to know but the rest illustrates differences really well.
Not everyone wants to admit - or even understands - that the % confidence is about how frequently such intervals will contain the true value when the procedure is applied to other populations / datasets.
"[frequentist confidence intervals] don't really tell us anything about the probability that the statistic of interest for our population is contained in the interval"
and
"the % confidence is about how frequently such intervals will contain the true value when the procedure is applied to other populations / datasets"
My understanding is that a 95% confidence interval implies that there is a 95% chance the true statistic lies within the interval. What are you saying it means?
To some degree I'm interested in the philosophy behind probability but there are limits to my concern with it. If you're saying that ontologically this is not truly a 95% confidence interval, but practically speaking, it does mean that, then I'm more interested in the practical interpretation because it's more relevant to applications of statistics.
If you calculate one thousand confidence intervals in 950 the true statistic will lie within the interval. I'm not sure if that means that there is a 95% chance the true statistic lies within a given interval.
Let me use a extreme example to highlight the issue. Imagine that I use the following procedure to produce a 95% confidence interval for whatever quantity you're interested in:
with 95% probability return the interval [ -1e999 1e999 ]
with 5% probability return the interval [ -1e999 -1e999+1 ]
If you calculate one thousand confidence intervals in 950 the true statistic will lie within the interval. I wouldn't say in any given case that there is a 95% chance the true statistic lies within that particular interval. (There was a 95% chance though.)
This article (open access PDF available) discusses how different methods of producing confidence intervals perform in a toy problem:
The fallacy of placing confidence in confidence intervals https://link.springer.com/article/10.3758/s13423-015-0947-8
One could argue that those misleading flaws can be avoided by being careful with what confidence intervals do you use and how - but the point is that coverage probabilities are about a set of counterfactual situations and not about the actual situation at hand.