This code has no memory other than the prior probability distribution. It seems like if you had previously flipped a coin a thousand times to generate it, your prior beliefs should be more strongly held than if you had just made up some numbers. Shouldn't the number of previous trials be accounted for somehow?
In case it's not clear (that wording doesn't really click for me, since there are no exponents involved), the way this information is encoded is in the shape of the distribution itself. If there have been very few observations, it will be very wide, but after many it will narrow to a tiny spike.
In this context, at least, the prior distribution encodes everything - there's no meaning to the idea that you're more or less confident in the prior, because the prior already represents your uncertainty about the outcomes. If you were 50/50 on this prior versus another one, then your actual prior would be the average of the two.
There is a subfield of statistics that deals with imprecise probabilities, but that's a whole other can of worms and doesn't really relate to this problem. That said, it's fascinating stuff, and very useful in some contexts (if you're uncertain about your priors, it can be useful to do sensitivity analyses to figure out exactly how the end result depends on your prior).
What I meant was that for h_1%, you're going to wind up with probabilities in the area of like, 1.2e-34, very quickly, if the coin is a fair one. The update process is adjusting via multiplication, and for a really bad hypothesis, that's going to bring its probability dramatically close to zero without that many trials. Even though the absolute difference between 1e-4 and 1-e34 hypothesis feels smallish when you look on a linear scale from 0-1, a 1-e34 is a lot 'stickier'.
Your explanation has the benefit of being a better explanation; my goal was just to explain where the inertia was hiding.
A better way of thinking about this problem instead of adding N observations in a batch is adding them one at a time. The Beta(a,b) distribution does this - Uniform is Beta(1,1). IIRC adding observations you're just ticking up a and b one at a time depending on whether the outcome was a head or a tails. This applies to exchangeable observations (if you flipped some other coin previously it doesn't work so simply in this model).
A question for people who know more about this than I do:
Why was the uniform distribution on [0, 1] chosen initially? Choosing a different distribution would give a different result. (And it doesn't make much sense to say, "Always choose the uniform distribution!" because the choice of variable affects the meaning of the distribution -- if instead we wonder about the value of p^2 and choose a uniform distribution for it on [0, 1], won't we get a completely different result?)
We're interested in the probability of a coin flip yielding heads before we flip any coins. Uniform is just a really common prior to choose in this situation for a few reasons:
-It's a special case of the beta distribution, which is the conjugate prior for binomial problems. This means that the distribution of the probability of getting heads given the coin flips is in the same family as the prior itself (ie: beta priors with binomial likelihoods yield beta posteriors).
-The uniform (for this problem at least) is an "objective prior", which expresses that we don't have much information about whether the flip is biased. The example you give (modeling p^2 instead of p) is a great example of when the uniform would be a bad choice. The reason the uniform doesn't work in this case is because for binomial data (coin flips), a uniform prior is not invariant to reparametrization.
If choosing priors was so simple as always going with the uniform, there'd be little reason to go with Bayes! The choice of prior sometimes makes a radical difference in the posterior (especially with small samples), and there's many things to consider when you choose priors (computational convenience, uninformative versus informative priors, hierarchical modeling, etc).
If you were holding a physical coin in your hand, you would have to be crazy to select a uniform distribution unless it was shaped like a sphere. This is sort of a really minor pet peeve of mine when people use coin-flipping as an example for these things. If it's even vaguely coin-like, even the most ridiculous distortion (maybe it's made of uranium on one side and aluminum on the other) probably couldn't bring the true probability past 70% or something.
For argumentative purposes (if trying to reach a consensus with someone who claims to be 95% sure that the coin is 'fair' and accepts the '45-55% heads is fair' definition), you would concentrate most of your probability mass there, and find a mutually agreeable shape. The 'it's not shaped like a sphere' thing suggests that some people would take into account visual information about the object to inform their priors, and be prone to rejecting the fitness of the uniform.
That may conflict with your sensibilities if you expect very strange coins, but if the priors aren't too crazily bad, if you have enough time for a few more coin flips, and the coin isn't particularly brittle, what's an order of magnitude between friends?
Edit: And for formal mathy purposes, ask someone else :)
If it's a normal looking coin being flipped in the air and caught before presentation, the only reasonable prior is that almost all of the probability mass should be that it's completely fair. Maybe I'd leave 1% to spread amongst the rest just in case the adversary has some seriously devious tricks up their sleeve.
Because physics - it's not possible to bias a rigid body so that it rotates with non-constant angular speed when flipped, as long as air resistance can be neglected, and that means that a fair flip gives 50/50 odds as to what side you catch it on. (Edit: clarity)
If other stuff is going on, like you're letting it bounce or something, then it depends on the particulars. It's rather easy to load a die, for instance.
One of the beautiful aspects of Bayesian methods is that they are subjective. Due to that I think you should really be asking "What sort of priors are reasonable here?"
From that perspective the uniform prior I use is reasonable (though non-committal and fairly uninformative), a triangular distribution with its peak at 50% (http://en.wikipedia.org/wiki/Triangular_distribution), the prior I have after updating tails and then heads, a normal distribution, or using the beta distribution with an equal number of both parameters.
Honestly you may decide there's a fair chance the coin is heads biased so you choose a prior that has most of the mass of the probability above 50%. As long as you have a reason, it's reasonable. You don't have to worry too much that you've chosen the optimal prior since more data makes equals of priors in the long run.
The only terrible thing you can do, is pick a prior that absolutely excludes certain hypotheses with 0% likelihood. No amount of data can overcome that.
> you would have to be crazy to select a uniform distribution unless it was shaped like a sphere.
The distribution for a coin is uniform over the event space [heads,tails]; not over the angle of rotation it lands on. You can have continuous or discrete event spaces.
Sure. My point was that you can't physically construct a flat cylinder a couple centimeters across that you can flip fairly and achieve a 98% bias. And if you don't meet those physical requirements, it's not a coin. If you had a sphere shaped object and the head condition was a particular point being above the equator, maybe you could include some distribution of internal weighting that could achieve that level of bias. But really the sphere thing was just tongue-in-cheek.
If you have something shaped like a gömböc[1] you can have a "coin" that lands on heads ~100% of the time, except for the whole not-actually-being-a-coin part.
We would!
Choosing priors is difficult and different people with different experience and views will have different (but still reasonable) priors. You may take personal (just use all your experience) or objective (try to come up with some scheme for assigning priors to types of situations) view on this problem.
Work was done in both directions. It's more of a philosophical problem than math problem, although in many cases you want something specific: "I have no idea what the outcome might be but I think it's either extremely on the right or left so give me priori which converges fast to one or the other). Or in a coin example: "it's almost surely more or less fair, so I don't want to change my views after several unlucky flips".
With just the few updates I've given you're probably right that it would affect things significantly. However, the more data you have the less the prior matters. This is known as swamping the priors.
In this case a uniform prior isn't incorrect, but you could definitely say its suboptimal and that I could make my examples much more accurate by choosing a prior that most represents my initial beliefs (like the one head one tail histogram for example.
Once you figure out simple examples you slowly start thinking this way about the world. It's beautiful. Take people for example:
Someone with open mind has a priori with at least slight probability assigned to unlikely (for them!) hypothesis while on the other hand very religious people for example have 0 in their priori when it comes to possibility of their religion being made up so they are forced to ignore evidence to the contrary (because bayesian updating breaks for them due to division by zero and mind's way to signal this exception is denial).
In general someone with a lot of weight on given hypothesis is "stubborn" or just very convinced and someone with uniform or close distribution just doesn't know anything about given problem.
Someone unable to build heavily weighted distributions is a conspiracy theorist, someone reluctant to - a sceptic and someone too much eager to a fanatic. Someone with very bad priors is un/badly educated (in given domain) or biased or maybe just stupid, someone with good priors is an expert.
It's possible to combine expert with sceptic attitude or expert with fanatic or all too often stupid with fanatic (very bad and very heavily weighted priors with possible 0's on some options).
Once you start thinking this way you start expressing yourself differently, you start adding those probability qualifiers to your sentences: "I am very sure it's the way to go", "My intuition tells me this but I am not really sure", "I am very convinced and it's not worth discussing" (yes, it can be rational and good attitude) or "I would do X but I need more evidence to be reasonably sure".
It's all there in people's mind, language and interactions once you start thinking this way it's whole new world of perspective and understanding.
Completely agreed. Your comment is a really good advertisement for normal people to learn Bayesian probability theory. Not because it's useful in science or engineering, but because it's useful in life.
25 comments
[ 2.3 ms ] story [ 71.3 ms ] threadhttp://yudkowsky.net/rational/bayes/
http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=...
http://www.inference.phy.cam.ac.uk/itila/
In this context, at least, the prior distribution encodes everything - there's no meaning to the idea that you're more or less confident in the prior, because the prior already represents your uncertainty about the outcomes. If you were 50/50 on this prior versus another one, then your actual prior would be the average of the two.
There is a subfield of statistics that deals with imprecise probabilities, but that's a whole other can of worms and doesn't really relate to this problem. That said, it's fascinating stuff, and very useful in some contexts (if you're uncertain about your priors, it can be useful to do sensitivity analyses to figure out exactly how the end result depends on your prior).
Your explanation has the benefit of being a better explanation; my goal was just to explain where the inertia was hiding.
Why was the uniform distribution on [0, 1] chosen initially? Choosing a different distribution would give a different result. (And it doesn't make much sense to say, "Always choose the uniform distribution!" because the choice of variable affects the meaning of the distribution -- if instead we wonder about the value of p^2 and choose a uniform distribution for it on [0, 1], won't we get a completely different result?)
-It's a special case of the beta distribution, which is the conjugate prior for binomial problems. This means that the distribution of the probability of getting heads given the coin flips is in the same family as the prior itself (ie: beta priors with binomial likelihoods yield beta posteriors).
-The uniform (for this problem at least) is an "objective prior", which expresses that we don't have much information about whether the flip is biased. The example you give (modeling p^2 instead of p) is a great example of when the uniform would be a bad choice. The reason the uniform doesn't work in this case is because for binomial data (coin flips), a uniform prior is not invariant to reparametrization.
If choosing priors was so simple as always going with the uniform, there'd be little reason to go with Bayes! The choice of prior sometimes makes a radical difference in the posterior (especially with small samples), and there's many things to consider when you choose priors (computational convenience, uninformative versus informative priors, hierarchical modeling, etc).
http://en.wikipedia.org/wiki/Jeffreys_prior http://en.wikipedia.org/wiki/Beta_distribution http://en.wikipedia.org/wiki/Conjugate_prior
That may conflict with your sensibilities if you expect very strange coins, but if the priors aren't too crazily bad, if you have enough time for a few more coin flips, and the coin isn't particularly brittle, what's an order of magnitude between friends?
Edit: And for formal mathy purposes, ask someone else :)
Because physics - it's not possible to bias a rigid body so that it rotates with non-constant angular speed when flipped, as long as air resistance can be neglected, and that means that a fair flip gives 50/50 odds as to what side you catch it on. (Edit: clarity)
If other stuff is going on, like you're letting it bounce or something, then it depends on the particulars. It's rather easy to load a die, for instance.
From that perspective the uniform prior I use is reasonable (though non-committal and fairly uninformative), a triangular distribution with its peak at 50% (http://en.wikipedia.org/wiki/Triangular_distribution), the prior I have after updating tails and then heads, a normal distribution, or using the beta distribution with an equal number of both parameters.
Honestly you may decide there's a fair chance the coin is heads biased so you choose a prior that has most of the mass of the probability above 50%. As long as you have a reason, it's reasonable. You don't have to worry too much that you've chosen the optimal prior since more data makes equals of priors in the long run.
The only terrible thing you can do, is pick a prior that absolutely excludes certain hypotheses with 0% likelihood. No amount of data can overcome that.
The distribution for a coin is uniform over the event space [heads,tails]; not over the angle of rotation it lands on. You can have continuous or discrete event spaces.
If you have something shaped like a gömböc[1] you can have a "coin" that lands on heads ~100% of the time, except for the whole not-actually-being-a-coin part.
1: https://en.wikipedia.org/wiki/G%C3%B6mb%C3%B6c
With just the few updates I've given you're probably right that it would affect things significantly. However, the more data you have the less the prior matters. This is known as swamping the priors.
In this case a uniform prior isn't incorrect, but you could definitely say its suboptimal and that I could make my examples much more accurate by choosing a prior that most represents my initial beliefs (like the one head one tail histogram for example.
Someone with open mind has a priori with at least slight probability assigned to unlikely (for them!) hypothesis while on the other hand very religious people for example have 0 in their priori when it comes to possibility of their religion being made up so they are forced to ignore evidence to the contrary (because bayesian updating breaks for them due to division by zero and mind's way to signal this exception is denial). In general someone with a lot of weight on given hypothesis is "stubborn" or just very convinced and someone with uniform or close distribution just doesn't know anything about given problem.
Someone unable to build heavily weighted distributions is a conspiracy theorist, someone reluctant to - a sceptic and someone too much eager to a fanatic. Someone with very bad priors is un/badly educated (in given domain) or biased or maybe just stupid, someone with good priors is an expert. It's possible to combine expert with sceptic attitude or expert with fanatic or all too often stupid with fanatic (very bad and very heavily weighted priors with possible 0's on some options).
Once you start thinking this way you start expressing yourself differently, you start adding those probability qualifiers to your sentences: "I am very sure it's the way to go", "My intuition tells me this but I am not really sure", "I am very convinced and it's not worth discussing" (yes, it can be rational and good attitude) or "I would do X but I need more evidence to be reasonably sure".
It's all there in people's mind, language and interactions once you start thinking this way it's whole new world of perspective and understanding.