The accepted answer seems wrong to me - he says that the frequentist is reasoning based on the Sun as the repeatable experiment when in fact it's the dice-roll, I would have thought ...
1/36 is the probability that the detector is lying, and so when it says "yes" that means it's a 1/36 probability that the sun hasn't gone nova, not the probability that it could.
$50 is only really any use if the sun hasn't gone nova (cause otherwise we're all dead), so therefore even if the odds are bad it's still a good bet.
EDIT: Also, a penny just dropped after reading another of the answers. The odds against the sun having gone nova are incredibly mind-bogglingly high such that the the 1/36 probability is negligible. So even if the probability that the machine is lying is small at 1/36 it's still far far far more likely than the sun having gone nova - which is the "prior" that the Bayesian is taking into account.
Another rub might also be that the frequentist's significance (P-value) is not sufficient to take into account the astronomical scale of the Nova probability but IANAS so that's just speculation.
The "1/36 chance" and the "negligible" aren't the corresponding/analogous probabilities. 1/36 is P(detector "yes" | not nova). The "negligible" is P(nova).
What we want to calculate is P(nova | detector "yes"). Using Bayes' rule it's
This is also quite intuitive. Unless going nova is more likely than malfunctioning, you should bet on malfunctioning. This is very much like David Hume's quote:
"no testimony is sufficient to establish a miracle, unless the testimony be of such a kind, that its falsehood would be more miraculous, than the fact, which it endeavors to establish." (test: would you say the same amount of "holy crap! WTF!!" if it turned out that the testimony was false, compared to if it turned out that the miraculous thing actually happened).
So: no detector "yes" is sufficient to establish the sun going nova, unless the detector saying "yes" be such, that it lying (1/36) would be more miraculous, than the sun going nova.
It's fascinating how precisely this can be formulated and how clear it is with probability theory.
Intuitively the main problem is that the chance of the machine lying is enormously higher than the chance of the sun going nova. The result of combining them says essentially nothing about the latter anymore.
I always thought that it was more about the difficulty of collecting winnings on the bet in the "sun actually gone nova" case. That holds not matter what the odds.
"Frequentists vs. Bayesians"? The problem with this comic is that it paints statisticians as belonging to one of two opposing tribes, when really frequentist and Baysian statistics are two different methods in the toolbox of a competent statistician – which of them is the appropriate one depends on the circumstances and the problem at hand.
I'm not sure where this idea of an alleged divide comes from, but I think it might have to do with a certain small but vocal group on the Internet [1].
No, the philosophical debate exists and has a history. Also Bayesian is quite a buzzword that people use in paper titles just because it sounds like some seal of approval, even if it just means that they used Bayes' rule somewhere in the derivation of their formula.
Anyway, using Bayes' formula is not the same as philosophical Bayesianism. Bayes' formula is simply a theorem of mathematics that follows from the axioms of probability theory. It's not controversial.
The controversy comes from the applicability of Bayesian philosophy. In theory and ideally, it seems nice, but there are some problems with it.
One well known problem is how to specifying your priors. A bigger issue that is often forgotten is that you can't calculate the conditional probability P(explanation | observation), without computing
Now, this means that you have to consider all possible explanations. But systematically scanning through all possible hypotheses is really difficult. Usually it takes rare moments of random genious ideas to come up with a hypothesis that works well. You could say this is a "global optimization" problem in hypothesis space.
Now, if you restrict the search for a particular class of hypotheses (a limited-flexibility model), then you can use computational methods, like Monte Carlo methods, to approximate this sum (integral) over a set of hypotheses. But you can never be sure if your modeling restrictions were valid or maybe the best explanation is outside "the box".
When I hear Internet people (bloggers, commenters) saying they use Bayesianism in their everyday life it's almost always about pulling numbers out of their asses and then feeling great about how smart and rational they are. Everyday life is just too messy, there are many possibilities that are hard to capture in a model.
Addendum: I also really hate how Lesswrongians hijacked and overmystify the term "Bayesian" (and even "rationalist") as if it were some panacea/silver bullet that magically turns you into some superhero. It's like a "one weird trick" BS just for smarter people.
Imagine that you have a coin, and you are told that it is not fair (asymmetrical mass distribution). But you are not told in which direction it is biased. If you flip it, what are the chances that it comes up heads?
A Bayesian would say the p(heads) = 1/2.
A frequentist would say that the only thing we can say about the p(heads) is that p(heads) ≠ 1/2.
I'm not at all qualified to talk about this. Without further ado:
Bayesianism has the concept of an A_p distribution, which is roughly "the probability that the probability is p".
My A_p distribution for this coin would be (almost) 50% on 0 and 50% on 1, because I don't think it's possible to bias a coin except by making both faces the same. If I didn't think that, it might be proportional to something like (p-1/2)^2, because more extreme biases seem more likely.
In both cases, my probability of seeing a heads is 1/2, because that's the mean of the A_p distribution. But after flipping the coin once, my A_p distribution updates, and now I can give a probability of seeing the next toss come up heads.
I was afraid somebody was going to bring up this paper. As the paper shows, they in fact do exist. But only if you spin rather than flip in the normal way.
There's no need to introduce any special new concepts. It's simply a hierarchical graphical model. You have one variable which stands for p (distributed according to some prior, like a Beta distribution perhaps), and you have another (perhaps multiple), Bernoulli distributed variable whose parameter comes from the first variable and represents the actual coin value that you get.
If you do multiple coin flips, you have more of these Bernoulli variables but they all share a common parameter p.
> You have one variable which stands for p (distributed according to some prior, like a Beta distribution perhaps)
This prior sounds like the A_p distribution.
The point is it's a distribution, not a number. You can't just say "I think the probability of seeing heads is p". But if you say "I think the probability of seeing heads is distributed like this, which has a mean of p", then that captures everything you need to know about the coin to update your expectations when you see the results.
(In case it was unclear, A_p isn't a specific shape of distribution, or class of shapes of distributions, like beta or gaussian. It's a distribution over hypothesis-space, where the hypotheses are probabilities, and can have any valid distribution shape.)
I was suggesting that this is less "mysterious" than it seems to sound, once you use a hierarchical model, where one random variable's outcome is used as the parameter for another variable.
Imagine one random variable p at the top center of a page, and then several variables under it, each connected by an arrow coming from p.
The random variables down there are the coin flips (each results in 0 or 1), and they are identically and independently distributed with a Bernoulli(p) distribution. And p itself is a random variable which has a distribution (expressing how likely each p is).
I'm just clarifying that all this fits into normal probability theory, there is no need to introduce any sort of "meta-probability theory".
So the Bayesian is reasoning based on what he doesn't know i.e. that the coin could land either side because his guess as to which side is heavier would be a 50/50 guess - not the actual outcome.
Frequentist bets on the outcome regardless of what he knows i.e. that the coin will be more likely to land on one side than the other and it doesn't matter which side that is.
Isn't this just two different perspectives: Slicing vs Limits. Bayesian is concerned with outcome of a single trial; frequentist is concerned with the sum of all outcomes.
Bayesian technique says that "probability" is a way of representing our collective knowledge and opinions about some matter. To say that there's a 1/2 chance of heads is to say that to the best of our knowledge and understanding we have no way to expect one outcome over the other. It lives in the present.
Frequentist technique says that "probability" is a way of describing repetition in experimentation. To say there's a 1/2 chance of heads is to say that should we flip it 100 times the best guess as to the number of heads we might obtain is (1/2)(100). It lives in a counterfactual future.
This is a foundation of the debate. The two schools are actually trying to accomplish different things. It turns out that much of the time these two endeavors land on identical solutions. It also turns out that in human experience we often flip between the two intentions without notice.
It's not possible to say one is better than the other unless you pick one semantics or the other to argue from (after which, obviously the other is poor). Instead, it's viable to look at what you're really trying to accomplish—often a task where errors and computation have expense and a tradeoff exists between accuracy, speed, information use, communicability, etc etc—and understand how each approach helps or harms your end.
Good statisticians, usually no matter how much they punch for one side or the other, will make these tradeoffs and use the technology which achieves the best end.
But we're all also at least a little bit philosophers and epistemologists so it's impossible to not want to take a few swings for one side or the other. Especially with historical figures like Jaynes leading the way.
Bayesian logic is arguably less useful for other people.
ex: If I read someone train of Bayesian logic from 1940 it's almost useless to me.
It can also be harder to combine information from multiple sources. AKA Study A is used as a prior in B and C's studies, I need to only count A's impact once.
However, it's arguably much better for making decisions.
I'm not sure I understand your 1940s point, but generally Bayesians would like to communicate (edit) likelihoods instead of posteriors so that they can be combined like you ask.
32 comments
[ 3.3 ms ] story [ 73.6 ms ] threadhttp://stats.stackexchange.com/questions/43339/whats-wrong-w...
Is that the best they can come up with?
For all we know the probabiliy of the sun going Nova is zero.
And 1/36 is not impossible. Hence the detector is lying.
Now, we can do the joke going the other way by thinking of another processes with conditional probabilities "expertly guessed"
$50 is only really any use if the sun hasn't gone nova (cause otherwise we're all dead), so therefore even if the odds are bad it's still a good bet.
EDIT: Also, a penny just dropped after reading another of the answers. The odds against the sun having gone nova are incredibly mind-bogglingly high such that the the 1/36 probability is negligible. So even if the probability that the machine is lying is small at 1/36 it's still far far far more likely than the sun having gone nova - which is the "prior" that the Bayesian is taking into account.
Another rub might also be that the frequentist's significance (P-value) is not sufficient to take into account the astronomical scale of the Nova probability but IANAS so that's just speculation.
- The sun hasn't gone nova and the detector is lying (with 1/36 chance)
- The sun has gone nova and the detector is not lying (negligible)
Now, given the result, one can calculate the probability of which of the cases
And the betting aspect of it makes sense as well
What we want to calculate is P(nova | detector "yes"). Using Bayes' rule it's
P(nova | detector "yes") = P(detector "yes" | nova) P(nova) / (P(detector "yes" | nova) P(nova) + P(detector "yes" | not nova) P(not nova)) = 35/36 * P_nova / (35/36 * P_nova + 1/36 * (1-P_nova))
P(nova | detector "yes") > 0.5 if P_nova > 1/36.
This is also quite intuitive. Unless going nova is more likely than malfunctioning, you should bet on malfunctioning. This is very much like David Hume's quote:
"no testimony is sufficient to establish a miracle, unless the testimony be of such a kind, that its falsehood would be more miraculous, than the fact, which it endeavors to establish." (test: would you say the same amount of "holy crap! WTF!!" if it turned out that the testimony was false, compared to if it turned out that the miraculous thing actually happened).
So: no detector "yes" is sufficient to establish the sun going nova, unless the detector saying "yes" be such, that it lying (1/36) would be more miraculous, than the sun going nova.
It's fascinating how precisely this can be formulated and how clear it is with probability theory.
What's wrong is that it's a useless machine.
I'm not sure where this idea of an alleged divide comes from, but I think it might have to do with a certain small but vocal group on the Internet [1].
[1] cough Eliezer Yudkowsky cough
Anyway, using Bayes' formula is not the same as philosophical Bayesianism. Bayes' formula is simply a theorem of mathematics that follows from the axioms of probability theory. It's not controversial.
The controversy comes from the applicability of Bayesian philosophy. In theory and ideally, it seems nice, but there are some problems with it.
One well known problem is how to specifying your priors. A bigger issue that is often forgotten is that you can't calculate the conditional probability P(explanation | observation), without computing
P(observation) = SUM_i [ P(observation | explanation_i) P(explanation_i) ]
for the denominator of the Bayes formula.
Now, this means that you have to consider all possible explanations. But systematically scanning through all possible hypotheses is really difficult. Usually it takes rare moments of random genious ideas to come up with a hypothesis that works well. You could say this is a "global optimization" problem in hypothesis space.
Now, if you restrict the search for a particular class of hypotheses (a limited-flexibility model), then you can use computational methods, like Monte Carlo methods, to approximate this sum (integral) over a set of hypotheses. But you can never be sure if your modeling restrictions were valid or maybe the best explanation is outside "the box".
When I hear Internet people (bloggers, commenters) saying they use Bayesianism in their everyday life it's almost always about pulling numbers out of their asses and then feeling great about how smart and rational they are. Everyday life is just too messy, there are many possibilities that are hard to capture in a model.
Addendum: I also really hate how Lesswrongians hijacked and overmystify the term "Bayesian" (and even "rationalist") as if it were some panacea/silver bullet that magically turns you into some superhero. It's like a "one weird trick" BS just for smarter people.
Imagine that you have a coin, and you are told that it is not fair (asymmetrical mass distribution). But you are not told in which direction it is biased. If you flip it, what are the chances that it comes up heads?
A Bayesian would say the p(heads) = 1/2.
A frequentist would say that the only thing we can say about the p(heads) is that p(heads) ≠ 1/2.
If the chance of biasing it T>H is 50% and biasing the other way is also 50% then the average is 1/2 (in a Schroedinger's cat kind of way)
At the same time the frequentist is right. p(heads) for the specific coin you're given is not 1/2
Bayesianism has the concept of an A_p distribution, which is roughly "the probability that the probability is p".
My A_p distribution for this coin would be (almost) 50% on 0 and 50% on 1, because I don't think it's possible to bias a coin except by making both faces the same. If I didn't think that, it might be proportional to something like (p-1/2)^2, because more extreme biases seem more likely.
In both cases, my probability of seeing a heads is 1/2, because that's the mean of the A_p distribution. But after flipping the coin once, my A_p distribution updates, and now I can give a probability of seeing the next toss come up heads.
I haven't read the whole of it, but relevant chapter from Jaynes: http://www-biba.inrialpes.fr/Jaynes/cc18i.pdf
(With skill, it's possible to bias a coin toss, but that's different.)
If you do multiple coin flips, you have more of these Bernoulli variables but they all share a common parameter p.
One such example model is the Beta-Bernoulli process: http://www.math.uah.edu/stat/bernoulli/BetaBernoulli.html
This prior sounds like the A_p distribution.
The point is it's a distribution, not a number. You can't just say "I think the probability of seeing heads is p". But if you say "I think the probability of seeing heads is distributed like this, which has a mean of p", then that captures everything you need to know about the coin to update your expectations when you see the results.
(In case it was unclear, A_p isn't a specific shape of distribution, or class of shapes of distributions, like beta or gaussian. It's a distribution over hypothesis-space, where the hypotheses are probabilities, and can have any valid distribution shape.)
Imagine one random variable p at the top center of a page, and then several variables under it, each connected by an arrow coming from p.
The random variables down there are the coin flips (each results in 0 or 1), and they are identically and independently distributed with a Bernoulli(p) distribution. And p itself is a random variable which has a distribution (expressing how likely each p is).
I'm just clarifying that all this fits into normal probability theory, there is no need to introduce any sort of "meta-probability theory".
Frequentist bets on the outcome regardless of what he knows i.e. that the coin will be more likely to land on one side than the other and it doesn't matter which side that is.
Isn't this just two different perspectives: Slicing vs Limits. Bayesian is concerned with outcome of a single trial; frequentist is concerned with the sum of all outcomes.
So in this example, what does each say when the coin were flipped 10 times, and it came up heads 6 times?
Frequentist technique says that "probability" is a way of describing repetition in experimentation. To say there's a 1/2 chance of heads is to say that should we flip it 100 times the best guess as to the number of heads we might obtain is (1/2)(100). It lives in a counterfactual future.
This is a foundation of the debate. The two schools are actually trying to accomplish different things. It turns out that much of the time these two endeavors land on identical solutions. It also turns out that in human experience we often flip between the two intentions without notice.
It's not possible to say one is better than the other unless you pick one semantics or the other to argue from (after which, obviously the other is poor). Instead, it's viable to look at what you're really trying to accomplish—often a task where errors and computation have expense and a tradeoff exists between accuracy, speed, information use, communicability, etc etc—and understand how each approach helps or harms your end.
Good statisticians, usually no matter how much they punch for one side or the other, will make these tradeoffs and use the technology which achieves the best end.
But we're all also at least a little bit philosophers and epistemologists so it's impossible to not want to take a few swings for one side or the other. Especially with historical figures like Jaynes leading the way.
ex: If I read someone train of Bayesian logic from 1940 it's almost useless to me.
It can also be harder to combine information from multiple sources. AKA Study A is used as a prior in B and C's studies, I need to only count A's impact once.
However, it's arguably much better for making decisions.
Edit: autocorrect ¯\_(ツ)_/¯
A Bayesian is one who, vaguely expecting a horse, and catching a glimpse of a donkey, strongly believes he has seen a mule.
http://www.statisticalengineering.com/frequentists_and_bayes...