I'm not sure why the author picks Myth #5 as the big pro-frequentist point. The "online learning" problem he's describing is a simple multi-armed bandit problem. The magical frequentist algorithm that he's touting is UCB-1. It's a great algorithm, it has finite-time optimality guarantees like he mentions. However, it still has a tuning parameter that can matter quite a bit in practice. Also, the Bayesian approach to MABs is Thompson sampling, which is also finite-time optimal. I guess I don't get the big deal on this point.
From a machine learning perspective, frequentist methods are great because they scale. All of these fancy nonparametric Bayes methods coming out these days take weeks to run and include so many approximations that you've got no idea whether what you're actually inferring is truly similar to the original model.
The biggest problem with frequentist methods is that they are conceptually odd in their approach. Bayesians take the view that you should fix the data and integrate over the possible parameters. Frequentists take the view that you should fix the parameters and integrate over the data. That is kind of weird to me.
So you either start out conceptually pure as a Bayesian and compromise to the point of meaninglessness or you abandon your morals at the start as a frequentist and things go smoothly from there.
Edit: As noted below, it's actually Exp3 rather than UCB-1. I glossed over the part about potentially adversarial bandits.
Actually it's not UCB-1. UCB-1 solves the stochastic bandit problem, where you assume the arms have fixed but unknown expected reward. He's talking about the adversarial setting, where you make no assumptions about the distribution of the rewards of the arms. They can even be set by an adversary that is trying to make you perform badly. Remarkably you can derive results here, though the "catch" is you are measuring performance against choosing a fixed arm for every play. (It's actually a bit more complex than that, but that's a reasonable simplification for direct comparison to the stochastic bandit.) The standard algorithm in the adversarial setting is Exp3: http://cseweb.ucsd.edu/~yfreund/papers/bandits.pdf
Otherwise I agree for the most part with what you've said.
Edit: In response to the parent edit, I should note that I know of no Bayesian algorithm for the adversarial setting. Thompson sampling is assuming a stochastic problem. I think a Bayesian algorithm would be possible, but the details elude me.
Same here. I was thinking after my edit that I don't know of such an algorithm either, but I don't think in principle that there is anything stopping someone from coming up with one.
So... who wants a NIPS paper? Noel and I are happy to just be 2nd authors, since you know... it was our idea and everything. ;)
There are probably a few ways to prove the result now, but one (nicely general) way to prove that online learning result would be the method of multiplicative weights: https://www.cs.princeton.edu/~arora/pubs/MWsurvey.pdf
>Frequentists take the view that you should fix the parameters and integrate over the data. That is kind of weird to me.
Integrating over the data makes a lot of sense during methods development. If you're the inventor of some algorithm and you want to convince people that they should apply it to their data set, then you'd want to show that it works well on a range of possible data sets.
It also makes some sense during policy development. Over your lifetime, you're going to see a lot of data sets, and having some policy about which statements about those data sets to believe can be useful. This policy doesn't have to be P<0.05. It could be six-sigma (like in physics) or something based on the false discovery rate (like in genomics).
I don't think frequentism is always the right approach, but I do see a place for it.
It's been a while since I was into this, but I think the problem of the bounds discussed in Myth 5 is that while the theory is fine, in practice the bounds are big, and Bayesian methods converge faster. The discussion also doesn't seem to discuss incremental decision making where posterior(t-1) -> prior(t).
I note that the discussion avoids the crazy stuff with frequentist stopping rules which is much more elegantly handled using Bayesian & decision theoretic methods.
I agree that there are quite a few myths around frequentist methods as described in the article.
Do people really argue against frequentist statistics? I was under the impression that in bayesian statistics you basically are just weighting your distribution by a prior, usually because there is not enough data yet to use robust frequentist methods. Put another way, I was under the impression that bayesian statistics are a way of solving the cold start problem[1]. Effectively you are using frequentist statistics in bayesian inference anyways. Please correct me if I am wrong.
I find this discussion (Bayesian vs Frequentist) somewhat baffling - it's as if physicists spent time arguing whether Lagrangian, Hamiltonian or Newtonian mechanics was "correct". In reality all three are equivalent, but different problems might be more naturally expressed (and easily solved) in one or the other form.
Similarly Bayesian and Frequentist descriptions are mathematically equivalent, the only question is which is the more natural description for a given problem.
Having multiple models like this is incredibly useful - some things that are intractable in one might be trivial in another - but trying to anoint one model as "the truth" seems perverse.
My impression was that frequentists and bayesians do actually disagree in some cases.
The example I remember is, suppose you have a coin with a uniformly random bias, except that the bias will not be zero. What is the probability that you will see heads if you flip it once?
- Bayesian: 50%
- Frequentist: Anything except 50%
Although you'd probably find they agreed on all the "what do I expect to see" stuff, so I guess that kind of is like interpretations of QM.
That's almost a straw-man argument. When you're making statements about pure probabilities, everyone is a bayesian. Most toy examples make "frequentists" look like morons, because essentially all of the judgement and discretion is removed from the problem, so you'd have to be an idiot not to apply Bayes's rule.
The difference shows up when you actually have an interesting data set to analyze. Bayesian statistics can disagree with frequentist stats in small samples because they're (often) using different normalization strategies; and they can disagree in large samples where the CLT fails. There may be other settings where they diverge too that I'm not aware of. But neither of those scenarios is one where insisting "I'm a Bayesian, so the answer is blah" or "I'm a frequentist, so... blah blah" is likely to be a good strategy. Those are the settings where it's hard.
If an example can illustrate a difference between two formal methodologies, the example should be as simple as possible, right? It doesn't paint one as superior to the other, but simply highlights the different methods by which they assign probabilities to events. If you make the example more complicated, the difference becomes less clear.
Right, but my point is that "as simple as possible" in this case is still "very complicated." It's sort of like trying to use "Hello world" to explain the difference between static and dynamic typing.
A practical way to summarize the difference is that Frequentists refuse to assign the word "probability" to anything other than the output of a formal generative model, whereas Bayesians use the word more broadly and are willing to ask "What is the probability of this hypothesis being true?". The latter is often called "subjective probability" in texts that attempt to discuss the division.
In response to the Bayesians' question, Frequentists say "You've specified no sampling process for hypotheses so I can't answer your question. If you want to represent your subjective belief system over hypotheses as a probability distribution you are welcome to do so by Cox's theorem. But the resulting number should not be understood as probability in the sense of fractional sampling, and we'd really be happier if you'd use a different word for it."
Frequentists and Bayesians agree precisely when the Bayesian's prior is the output distribution of a generative process for the inputs to the problem under study (in this case, biases for coins). If the Bayesian's prior is "just chosen based on judgement" then the Frequentist theory says "That's no longer probability, but it may still be a useful framework for quantifying subjective beliefs and the support for such beliefs from observational data."
So, in your coin example:
Bayesian: anything you want, based on your prior for coin biases. If your prior is symmetric, then 50% as you say.
Frequentist: the question is meaningless without a formally specified generating process for biased coins, but if you tell me the generating process then I can give you a ranking (likelihood) function over all possible biases.
if you tell me the generating process then I can give you a ranking (likelihood) function over all possible biases.
But if we already know the generating process, all the hard work is already done. The case of real interest is where we don't know the generating process, but we have a bunch of data, and we're trying to figure out what the generating process is from the data.
> But if we already know the generating process, all the hard work is already done.
Yes, I agree completely, and I believe that so would most frequentists! The point, even among the most strident critics, is rarely one of whether Bayesian approaches are useful; it is that Frequentists regard the proper domain of what they call "probability" to be exclusively related to generative processes and relative sampling ratios. From a purely technical point of view I see nothing wrong with that, even if it is a somewhat strict position (having trained primarily in mathematics I'm comfortable allowing people their strict definitions as long as they are recognized as such).
As a natural consequence of this divergence, Bayesian modeling allows you to explicitly punt the problem of generative modeling for the tricky bits, and it puts a nice big warning label on it saying "If you botch your prior, you're going to have a bad time." The biased coin is a perfect example. Knowing the bias, the generative process for a sequence of flips is trivial. But how are you going to create a generative process that seriously engages how and where the biases emerge? It is far easier to just say "empirically, biases seem to be distributed like so." At this point the Frequentist says "Well, that assumption is not derived from a formal model and is only loosely falsifiable, but if I accept it as a substitute for a generative model then you and I will reach the same conclusions about posterior probabilities."
I know of no frequentist who would disagree that getting good generative models for complex phenomena is often extraordinarily difficult. Where they disagree with Bayesians is whether doing something other than that should, strictly, be called "probability" or, whether it is more appropriate to call it something like "semi-empirical subjective-belief modeling".
how are you going to create a generative process that seriously engages how and where the biases emerge?
E. T. Jaynes would have said that you do this using your knowledge of the physics of coins and coin flipping. One of the examples he uses in his book Probability Theory: The Logic of Science is a robotic coin-flipper that can control the process so as to always make the coin land on the same side, i.e., the "bias" is in the flipping process, not in the coin itself. If you don't know anything about the flipping process or the relevant physics, then you have no way of constructing any hypotheses about what sort of generative process might be involved.
It is far easier to just say "empirically, biases seem to be distributed like so."
This corresponds to the case where you don't know anything about the underlying physics; in Bayesian terms, you are assuming a maximum entropy prior with a constraint--the constraint being the distribution observed in the flips so far. But if you do know something about the underlying physics--for example, if you know the coin is being flipped by a robotic flipper with such-and-such design--you might be able to come up with a much better prior using that knowledge. I'm not sure whether that sort of thing is included in the frequentist's concept of a "generative model".
Statisticians do not spend time arguing this issue; some people like some tools and other people like others, but at this point people are pretty accommodating.
But they're not mathematically equivalent. That's like saying that neural nets and random forrests are mathematically equivalent.
The first is discussing whether people treat Bayesian stats as a religion. The second includes a quote from someone in the frequentist religion (you can tell because they make a jibe about the "Bayesian religion"). So there are people on both sides, thankfully dwindling in number, who really do get their pants in a twist about the formalism one chooses for doing stats. Note that the author of the OP is not one of them, as he lays out in the first sentence of his post.
> Bayesian and Frequentist descriptions are mathematically equivalent
That's a very strong statement. Is the class of problems where they both approaches agree well understood (for some definition of agreement)? I was under the impression it was not. Perhaps the complete class theorem? Are the assumptions for that reasonable?
As I understand it, Cox's theorem suggests that freq/bayesian standpoints actually have functional consequences under certain assumptions, although whether those are the correct assumptions has been up for debate. Perhaps, too, there's a frequentist formulation, but this goes beyond my expertise.
I first became aware of it when I was fresh out of school. I was talking to a more senior PhD researcher and we were having a conversation about other people in our department and he said something to the effect of, "Do you know Bob? I always thought he seemed like a good guy, but then the other day I discovered he was a frequentist. Too bad." I thought for sure I had mis-heard him at first.
IMO he nails the key distinction here: "One definition says that a frequentist is a Bayesian trying to do well, or at least not too badly, against any possible prior distribution."
And there's some nice humor also: "The 250-year debate between Bayesians and frequentists is unusual among philosophical arguments in actually having important practical consequences."
> claim that frequentist methods need to make strong modeling assumptions.
> Assumption that horse race winners are not completely random and that there is a strategy
This is kind of the assumption that people tend to blame frequentist statistics for doing (well, any statistician I guess). There are nearly always assumptions made about what the random variables are or aren't. If the horse race was completely random, your guarantees fall apart as it's a simple dice roll with no strategy beyond chance. Yet you've used your assumption to dump money on the table, and now you've probably lost it.
I'm not a statistician, but I'm afraid I don't get his Myth 5:
"For some reason it’s assumed that frequentist methods need to make strong assumptions (such as Gaussianity), whereas Bayesian methods are somehow immune to this."
What I'd thought I'd heard before, from the Bayesian camp, was that both methods required strong assumptions, but the assumptions had to be explicit in the Bayesian model.
My attempt to summarise the difference in language familiar to computer scientists, is that you can look at the frequentist vs Bayesian debate as being about when a worst-case analysis is preferable to average-case analysis for unknown parameters of a statistical model.
There's something you don't know (the parameters). Are you looking to make statements which bound how bad things could be under the worst-case setting of those parameters? Or do you have some idea upfront about how likely different parameter settings are, and want to make statements about them in the "average" case?
Rather like with worst-case vs average-case analysis of algorithms, which is more appropriate depends what you're trying to do, and sometimes both are interesting.
28 comments
[ 3.0 ms ] story [ 35.5 ms ] threadFrom a machine learning perspective, frequentist methods are great because they scale. All of these fancy nonparametric Bayes methods coming out these days take weeks to run and include so many approximations that you've got no idea whether what you're actually inferring is truly similar to the original model.
The biggest problem with frequentist methods is that they are conceptually odd in their approach. Bayesians take the view that you should fix the data and integrate over the possible parameters. Frequentists take the view that you should fix the parameters and integrate over the data. That is kind of weird to me.
So you either start out conceptually pure as a Bayesian and compromise to the point of meaninglessness or you abandon your morals at the start as a frequentist and things go smoothly from there.
Edit: As noted below, it's actually Exp3 rather than UCB-1. I glossed over the part about potentially adversarial bandits.
Otherwise I agree for the most part with what you've said.
Edit: In response to the parent edit, I should note that I know of no Bayesian algorithm for the adversarial setting. Thompson sampling is assuming a stochastic problem. I think a Bayesian algorithm would be possible, but the details elude me.
So... who wants a NIPS paper? Noel and I are happy to just be 2nd authors, since you know... it was our idea and everything. ;)
There are probably a few ways to prove the result now, but one (nicely general) way to prove that online learning result would be the method of multiplicative weights: https://www.cs.princeton.edu/~arora/pubs/MWsurvey.pdf
Integrating over the data makes a lot of sense during methods development. If you're the inventor of some algorithm and you want to convince people that they should apply it to their data set, then you'd want to show that it works well on a range of possible data sets.
It also makes some sense during policy development. Over your lifetime, you're going to see a lot of data sets, and having some policy about which statements about those data sets to believe can be useful. This policy doesn't have to be P<0.05. It could be six-sigma (like in physics) or something based on the false discovery rate (like in genomics).
I don't think frequentism is always the right approach, but I do see a place for it.
I note that the discussion avoids the crazy stuff with frequentist stopping rules which is much more elegantly handled using Bayesian & decision theoretic methods.
I agree that there are quite a few myths around frequentist methods as described in the article.
[1] http://en.wikipedia.org/wiki/Cold_start
Similarly Bayesian and Frequentist descriptions are mathematically equivalent, the only question is which is the more natural description for a given problem.
Having multiple models like this is incredibly useful - some things that are intractable in one might be trivial in another - but trying to anoint one model as "the truth" seems perverse.
The example I remember is, suppose you have a coin with a uniformly random bias, except that the bias will not be zero. What is the probability that you will see heads if you flip it once?
- Bayesian: 50%
- Frequentist: Anything except 50%
Although you'd probably find they agreed on all the "what do I expect to see" stuff, so I guess that kind of is like interpretations of QM.
The difference shows up when you actually have an interesting data set to analyze. Bayesian statistics can disagree with frequentist stats in small samples because they're (often) using different normalization strategies; and they can disagree in large samples where the CLT fails. There may be other settings where they diverge too that I'm not aware of. But neither of those scenarios is one where insisting "I'm a Bayesian, so the answer is blah" or "I'm a frequentist, so... blah blah" is likely to be a good strategy. Those are the settings where it's hard.
In response to the Bayesians' question, Frequentists say "You've specified no sampling process for hypotheses so I can't answer your question. If you want to represent your subjective belief system over hypotheses as a probability distribution you are welcome to do so by Cox's theorem. But the resulting number should not be understood as probability in the sense of fractional sampling, and we'd really be happier if you'd use a different word for it."
Frequentists and Bayesians agree precisely when the Bayesian's prior is the output distribution of a generative process for the inputs to the problem under study (in this case, biases for coins). If the Bayesian's prior is "just chosen based on judgement" then the Frequentist theory says "That's no longer probability, but it may still be a useful framework for quantifying subjective beliefs and the support for such beliefs from observational data."
So, in your coin example:
Bayesian: anything you want, based on your prior for coin biases. If your prior is symmetric, then 50% as you say.
Frequentist: the question is meaningless without a formally specified generating process for biased coins, but if you tell me the generating process then I can give you a ranking (likelihood) function over all possible biases.
But if we already know the generating process, all the hard work is already done. The case of real interest is where we don't know the generating process, but we have a bunch of data, and we're trying to figure out what the generating process is from the data.
Yes, I agree completely, and I believe that so would most frequentists! The point, even among the most strident critics, is rarely one of whether Bayesian approaches are useful; it is that Frequentists regard the proper domain of what they call "probability" to be exclusively related to generative processes and relative sampling ratios. From a purely technical point of view I see nothing wrong with that, even if it is a somewhat strict position (having trained primarily in mathematics I'm comfortable allowing people their strict definitions as long as they are recognized as such).
As a natural consequence of this divergence, Bayesian modeling allows you to explicitly punt the problem of generative modeling for the tricky bits, and it puts a nice big warning label on it saying "If you botch your prior, you're going to have a bad time." The biased coin is a perfect example. Knowing the bias, the generative process for a sequence of flips is trivial. But how are you going to create a generative process that seriously engages how and where the biases emerge? It is far easier to just say "empirically, biases seem to be distributed like so." At this point the Frequentist says "Well, that assumption is not derived from a formal model and is only loosely falsifiable, but if I accept it as a substitute for a generative model then you and I will reach the same conclusions about posterior probabilities."
I know of no frequentist who would disagree that getting good generative models for complex phenomena is often extraordinarily difficult. Where they disagree with Bayesians is whether doing something other than that should, strictly, be called "probability" or, whether it is more appropriate to call it something like "semi-empirical subjective-belief modeling".
E. T. Jaynes would have said that you do this using your knowledge of the physics of coins and coin flipping. One of the examples he uses in his book Probability Theory: The Logic of Science is a robotic coin-flipper that can control the process so as to always make the coin land on the same side, i.e., the "bias" is in the flipping process, not in the coin itself. If you don't know anything about the flipping process or the relevant physics, then you have no way of constructing any hypotheses about what sort of generative process might be involved.
It is far easier to just say "empirically, biases seem to be distributed like so."
This corresponds to the case where you don't know anything about the underlying physics; in Bayesian terms, you are assuming a maximum entropy prior with a constraint--the constraint being the distribution observed in the flips so far. But if you do know something about the underlying physics--for example, if you know the coin is being flipped by a robotic flipper with such-and-such design--you might be able to come up with a much better prior using that knowledge. I'm not sure whether that sort of thing is included in the frequentist's concept of a "generative model".
But they're not mathematically equivalent. That's like saying that neural nets and random forrests are mathematically equivalent.
http://normaldeviate.wordpress.com/2013/09/01/is-bayesian-in...
http://normaldeviate.wordpress.com/2013/05/05/aaronson-colt-...
The first is discussing whether people treat Bayesian stats as a religion. The second includes a quote from someone in the frequentist religion (you can tell because they make a jibe about the "Bayesian religion"). So there are people on both sides, thankfully dwindling in number, who really do get their pants in a twist about the formalism one chooses for doing stats. Note that the author of the OP is not one of them, as he lays out in the first sentence of his post.
That's a very strong statement. Is the class of problems where they both approaches agree well understood (for some definition of agreement)? I was under the impression it was not. Perhaps the complete class theorem? Are the assumptions for that reasonable?
As I understand it, Cox's theorem suggests that freq/bayesian standpoints actually have functional consequences under certain assumptions, although whether those are the correct assumptions has been up for debate. Perhaps, too, there's a frequentist formulation, but this goes beyond my expertise.
http://ksvanhorn.com/bayes/Papers/rcox.pdf
(talk): http://videolectures.net/mlss09uk_jordan_bfway/
(slides): http://mlg.eng.cam.ac.uk/mlss09/mlss_slides/Jordan_1.pdf
IMO he nails the key distinction here: "One definition says that a frequentist is a Bayesian trying to do well, or at least not too badly, against any possible prior distribution."
And there's some nice humor also: "The 250-year debate between Bayesians and frequentists is unusual among philosophical arguments in actually having important practical consequences."
[1]: http://statweb.stanford.edu/~ckirby/brad/papers/2005BayesFre...
> Assumption that horse race winners are not completely random and that there is a strategy
This is kind of the assumption that people tend to blame frequentist statistics for doing (well, any statistician I guess). There are nearly always assumptions made about what the random variables are or aren't. If the horse race was completely random, your guarantees fall apart as it's a simple dice roll with no strategy beyond chance. Yet you've used your assumption to dump money on the table, and now you've probably lost it.
"For some reason it’s assumed that frequentist methods need to make strong assumptions (such as Gaussianity), whereas Bayesian methods are somehow immune to this."
What I'd thought I'd heard before, from the Bayesian camp, was that both methods required strong assumptions, but the assumptions had to be explicit in the Bayesian model.
I don't care about the method, I care about a correct prediction.
If one group can't do that, well, tough for them.
But in the end it's the structure of the solution and the "guesses" that had to be done that contribute to the success more than the method.
There's something you don't know (the parameters). Are you looking to make statements which bound how bad things could be under the worst-case setting of those parameters? Or do you have some idea upfront about how likely different parameter settings are, and want to make statements about them in the "average" case?
Rather like with worst-case vs average-case analysis of algorithms, which is more appropriate depends what you're trying to do, and sometimes both are interesting.