Using numbers for probabilities is not a panacea either, when those numbers are not the result of a calculation. One person might say "99% of the time" to mean "almost always", and another person might say "9 out of 10 times" to mean exactly the same thing.
The problem is that it's hard to estimate the probability of something that you know happens "almost always" without doing the counting, which most people don't do. And hence it's difficult to gain an intuition for what 99% means exactly.
Furthermore, there is also the possibility that two people share the same intuition about a given word/phrase, but then disagree about which number describes that intuition. This could account for part of the results of this paper.
This is somewhere that common usage and mathematical usage diverge strongly. To anyone versed in measure theory talking about "almost always" wrt probability will mean that there is zero probability of the event not happening (even thought there may be other outcomes in the space).
A good example of this is ratings on Amazon. You can give up to five stars, but most people just give 1 or 5 depending on if they were satisfied or dissatisfied. Additionally, people confuse precision for certainty and will perceive an estimate of 89.1% as more accurate than 90%.
Aren't all adjectives relative and therefore arguable?
It's amazing how often they're used in debates or political speech (ex. good, bad, strong, important, dangerous, etc.) when they clearly mean different things to different people, but maybe that's the idea.
I do think it's the idea, a lot of the time. On the other hand I suspect a lot of people do it without being aware.
There's only so much accuracy you can squeeze into a sentence - some people don't notice, some take advantage, some try to teach others to keep an ear out for this exact problem.
"Don't hedge your prose with little timidities. Good writing is lean and confident."
This is a separate issue, and one I would disagree with. In fact I feel like this is a problematic attitude in our culture. Too many people overconfidently asserting their subjective viewpoints. Real life is full of nuance, gray area, second-guessing, and different perspectives; simple bold assertions usually don't hold water.
David Foster Wallace, considered one of the greatest writers of recent times, filled his writings with qualifications, side notes, devil's advocacy, and at times insecure waffling. To me this is a more accurate reflection of the mind and reality and should not be suppressed for a macho bravado façade.
Yes, but the actual opinions run quite lean and are unqualified except for the “usually,” which the reader understands is placed to head off anecdotal contrary examples. The last one ends with a blunt insult. It’s certainly not DFW’s style of writing.
You are certainly entitled to your interpretation, but I don't think the writer put in "usually" just to head off anecdotal counterexamples. The principle of charity leads me to assume the writer genuinely believes that simple bold assertions are occasionally fine.
Though I do strongly agree with the original comment, so maybe that is the real reason why I am defending it.
That’s fair! I disagree with the reasons presented in the comment, but I agree that carefully qualified, elaborate writing, as well as florid descriptive writing, are all acceptable, and frequently desirable. I just thought the irony in the comment’s style was funny, and maintain that it is stylistically terse overall.
True, I fully see your point of people imposing their subjective views over-confidently. It is a real problem. :-(
But I don't think Zinsser there means to get rid of all 'timidities'. The subtext my mind's eye read there was: don't overdo it. And he goes on to expand: "Every little qualifier whittles away some fraction of the reader's trust. Readers want a writer who believes in himself and in what he's saying. Don't diminish that trust."
For instance, I know smart people who often apologize for no reason, do excessive hedging, or start a request with an apology. It gets a bit annoying when I have to read or hear that everyday.
It depends on why you're writing. If you're using the word "likely" you probably are dealing with some sort of scientific data. Writing for a technical audience needs to be precise, and words need to be only used when necessary and well defined.
>When you use a word to describe the likelihood of a probabilistic outcome, you have a lot of wiggle room to make yourself look good after the fact. If a predicted event happens, one might declare: “I told you it would probably happen.” If it doesn’t happen, the fallback might be: “I only said it would probably happen.”
The same is true with numbers. If a predicted event happens, one might declare: "I told you there 80 percent that will happen". If it dosen't happen, the fallback might be: "I told you there are 20 percent it will not happen."
Most folks took 70% to mean that she would certainly win and were bitterly disappointed the morning after.
On the other hand no sane person would (willingly) play Russian roulette with a 70% or even 5 of 6 chance of "winning".
In everyday life, one conflates probability with severity of outcome. This is normal and is part of how people assess and act on risk in a qualitative way.
Pascals' Wager: multiply the odds by the cost or benefit of each outcome. The higher the potential cost, the less good your odds look even if the percentage is the same.
And the negative side: eternal damnation in hell (which he valued at -infinity) times a very very small chance, still means it's worth putting in whatever work to be righteous.
Pascal's Wager assumes the only God that can possibly exist is the Judeo-Christian one, and that the Bible is correct in its description of what that God wants, and of what awaits those who fail Him.
But logically, if any one God can exist, then so might any God or set of Gods from any pantheon, including sets never even conceived of by human beings. Since the evidence of any one God over another is the same (zero, it's entirely a matter of faith,) the benefit of belief in any one God over another is also zero, since the set of all possible Gods is infinite.
Yeah there's a real difference between a very small but still finite chance, and an infinitesimally small chance. If you're trying to multiply negative infinity by reciprocal infinity, you have to be specific about how big each infinity is. :)
>Hillary Clinton had a >70% of winning the US presidential election
Probabilities without confidence intervals[1] are by-and-large meaningless (She has a 90% chance of of winning with a confidence interval of +11% -100%). No amount of d3.js on 538's blog will change this.
From a Bayesian perspective, or from a betting one, it doesn't make sense to put probabilities in a confidence interval. You might be uncertain about the world, but you can be certain about how much uncertainty you have, since it's a property of your own mind.
How do we quantify the difference between coin tosses, which we are very certain is 50% likely to end up heads, from political elections, where we only have a few examples to go off of?
In terms of predicting what's going to happen next, it really isn't that different if your model's any good. Looking backwards, if the 2016 election had been determined by two coin tosses both coming up heads, would we be surprised? And yet 538's polling model indicated that the probability was even better than two coins coming up heads.
They probably should have just written that right there: "71%, better odds than two coins coming up heads."
But: in order to understand if the model is any good, you need to try to build a good model, describe your model, and describe what you know about the underlying processes. With words, visuals, and math. Which is pretty much what 538 does better than most.
The difference is how the probability changes in response to new information. Learning more about the coin won't change the probability of heads from 50%, but doing more exhaustive polls would have likely improved our prediction about the election.
However I don't think there's as much of a difference as you think. If we learnt more about how the coin was going to be flipped then that would certainly improve our estimate of the outcome. If we found out the exact way it would be flipped we could calculate the outcome exactly.
So we can't really compare the two situations quantitatively, since there's no way to match up like-for-like the information we could receive. But we can say, for each possible piece of information we could receive, how much we expect it to change our probability.
This is worth emphasizing. If you have a model with a parameter that represents the probability of something happening, and then you have uncertainty in that parameter, your real probability needs to integrate over that uncertainty.
But the final probability is still just a single number.
But I suspect the GP's reasoning was getting interference from the valid point that poll numbers, which are not probabilities, are meaningless without confidence intervals or some other measure of uncertainty.
That uncertainty will then feed, via some model, into a probability estimate of a candidate winning. And that's just a single number.
Hmm. Doesn't it depend on whether re-rolls are allowed as to how confident we can be?
For example, I can imagine a well engineered coin that has a 0.5 probability of heads with a 0.005 confidence (ie, we suspect that the true weighting between heads and tails is likely between 0.495 and 0.505 19 times out of 20) and I can contrast it with a hastily made coin that can still have a 0.5 expected probability with a 0.1 confidence (ie, we expect the true probability to be between 0.4 to 0.6, 19 times out of 20).
Coin A and Coin B have dramatically different impacts on our decision making. For example, selling insurance against 5 identical flips in a row is a much more expensive proposition for Coin B than Coin A.
I feel like I really should know this given my data science background, but sometimes the basics slip away.
Right. Both coins have a probability of 0.5 on the next flip, but when you do multiple flips the hastily made coin has a higher chance of a result with lots of heads or lots of tails.
We could do the same thing for the 2016 election, but we would have to specify exactly what we meant by a "repeat". Do we just let Hillary run for the 2020 election and see what happens? Or do we put back every atom to the exact position it had in 2014, so that the only divergences between the two elections are caused by quantum randomness? Or something in between, like looking at all elections where a demagogue outsider runs against an established insider?
Really it doesn't matter what definition of "repeat" we choose. Since the repeat won't actually happen, we can't be called to bet on it, so knowing the probability in that case isn't too useful. Whereas a coin actually can be flipped multiple times.
It might be of some general interest to note that there exist such objects - distributions over distributions, or metaprobabilities. They have practical use - for example, if you are playing a game with uncertainty, you might observe something in the next step that changes your belief, or distribution over world state, b = P(s). If you have some expected distribution of next observations then you can talk about the probability of having some belief in the next state. Roughly P(b'|b) = P(P(s')|P(s)). You can collapse this into simple probabilities if you just care about your new observations, P(obs'|b) = \sum_s P(obs'|s)P(s|b).
I could imagine some frameworks where confidence intervals in this way would be useful - ex. I have 3 fairly different world models that I think are equally likely, each has a P(election), what's the P(election|world) and get confidence intervals across world models rather than just summing over them to P(election). But I agree that for most common approaches that simple probabilities are most useful and clear.
Something similar certainly is useful in the case of flipping a coin. Suppose we have two coins, one of which we know is fair and one of which we know is biased but we don't know which way. Then our knowledge about the bias of the first coin would be described by a distribution with a sharp peak around 0.5, but the distribution for the second coin would have two lumps either side of 0.5. Then the probability of heads on the next flip would be 0.5 for either coin. But if someone asked us to bet on the probability of heads on the next two flips, then the second coin would have a higher probability than the first. The different distributions for the two coins don't make a difference for one flip, but do for multiple flips.
But there's no meaningful way to repeat an election, so I don't think similar distributions or confidence intervals are useful in that case.
EDIT: We can imagine rerolling an election, but since we can't actually do it we don't have to bet on it so the information wouldn't be very useful to us.
> From a Bayesian perspective, or from a betting one, it doesn't make sense to put probabilities in a confidence interval
"Bayesian perspective" covers a lot of territory, and your assertion depends on the situation and the modeling objective. If a probability is a model parameter (for instance, frequency of heads for a particular coin), then summarizing the posterior distribution on that parameter with a confidence interval can be a sensible thing to do.
Or do you mean as opposed to a credibility interval?
> Probabilities without confidence intervals are by-and-large meaningless.
That's just not true. If I believe that my team has a 20% chance to win and you offer me a bet with anything better than 5-to-1 odds I should take the bet. If you offer me anything worse than 5-to-1, then I should not take the bet. There's no fuzz factor necessary; no confidence interval that I need to use to make the decision.
Perhaps you're getting at the idea of calibration? That it's difficult for a person to know what a 20% chance feels like? But there are still a lot of situations where it's not up to human judgement.
I think this is getting to the root of the problem — you're taking a perfectly valid frequentist view of probability, that is, viewing it as a series of discrete experiments. But the probabilities that were assigned to Clinton's victory were derived from Bayesian probability theory that estimates the likelihood that event occurs based on empirically determined prior probabilities that contribute to that event. Those prior distributions have uncertainty which leads to uncertainty in the resulting prediction.
I apologize if I've explained this poorly, this is just my layman's understanding of the matter.
It's a property of the thing being measured, not of the measuring system.
A Bayesian calculating that probability must get a single number too, without error margins. The only difference is that the Bayesian will have to weight his probabilities by how much of the interval is at the "win" and the "no win" scenarios.
It is different if you are measuring how many votes each candidate will have. For that both methods must get intervals and a confidence level.
I've been confused by this in the past too, so let me clarify for you.
Confidence intervals are for predictions of a given value. They are not for probabilities. It doesn't make sense to say, "I think she has a 70% chance of winning, plus or minus 5%." On the other hand, it does make sense to say, I think she will get 48% of the vote, with an interval of +-1%.
The analysis after the elections was interesting-- think pieces asking things like "How could the stats have been so wrong?!". A fair dice has a roughly 83% chance of landing on a number between 1 and 5, but if you roll a 6 you don't ask yourself the same question.
People are mixing up % of vote with likelihood of victory.
If she was polling at 70% that would be an almost 100% chance of winning because a 20% swing is unlikely. However a 70% chance of victory is much much closer.
Exactly right, and you can see the polling percentage on the linked page; scroll down to "How the forecast has changed" and click the almost-looks-like-a-button "popular vote"; it was bouncing around 46-48% Clinton and 40-44% Trump (and 5-8% Johnson).
> In everyday life, one conflates probability with severity of outcome.
My everyday example for that is the weather forecast and the question 'will it rain' often answered with a precipitation probability?
- Probability: How likely is it that I will be hit by at least one rain drop
- Severity: How many rain drops will hit me
It sounds a little abstract, but whenever I see some everyday weather forecast I wonder what they are trying to tell me. At least 0% seems unambiguous to me :-)
The "Probability of Precipitation" (PoP) describes the chance of precipitation occurring at any point you select in the area.
How do forecasters arrive at this value?
Mathematically, PoP is defined as follows:
PoP = C x A where "C" = the confidence that precipitation will occur somewhere in the forecast area, and where "A" = the percent of the area that will receive measurable precipitation, if it occurs at all.
So... in the case of the forecast above, if the forecaster knows precipitation is sure to occur (confidence is 100%), he/she is expressing how much of the area will receive measurable rain. (PoP = "C" x "A" or "1" times ".4" which equals .4 or 40%)
But, most of the time, the forecaster is expressing a combination of degree of confidence and areal coverage. If the forecaster is only 50% sure that precipitation will occur, and expects that, if it does occur, it will produce measurable rain over about 80 percent of the area, the PoP (chance of rain) is 40%. ( PoP = .5 x .8 which equals .4 or 40%. )
If you ask Alexa, "will it rain today?" she'll respond, "it probably won't rain today" for any probability below 50%. I always thought this was an interesting and illustrative example. If there's a 40% chance of rain, I wouldn't in casual conversation say that, "it probably won't rain today."
At least where I am in the summer 40% or higher is just about a guarantee that there will be rain... That seems like something that might need to change based on location and season
She shoulda just answered that it never rains in southern california. And followed up with what kind of croissant-bakery you'd like to stuff your mouth from.
> no sane person would (willingly) play Russian roulette with a 70% or even 5 of 6 chance of "winning".
That’s because there’s no upside to Russian roulette. Put some money on it, and people would be more likely to take the wager. In its standard form (as I understand it), the only benefit to winning is the rush from having played.
Yeah, certainly "A LOT" of people were actually pleasantly surprised.
From my point of view in Philadelphia, however, virtually everyone was disappointed modulo a few crazies or people who had been hiding under a rock or some closeted college republicans.
This just goes to show how bifurcated America has become, people on opposite sides view the other side as incomprehensibly batshit-crazy and/or evil.
Yeah, it was amazing to see the storm of hate that poured down on Nate Silver as a result. Thousands of people calling him a fraud (and worse), saying they would never trust him again, etc. Even if he had said there was a 99% chance, that still leaves 1%!
What's worse is before the election, 538 was getting massive heat for over-estimating the likelihood of Trump winning. People see what they want to see and rewrite the history accordingly.
When people say things like "serious possibility" they are not merely intending to alter the perception of the likelihood but also attempting to impress upon the listener the gravity of the event.
In cold war era America would people have felt any different knowing that a Russian invasion was only 20% likely rather than 30%? For something so /serious/ any non-zero probability is something that should be prepared for.
Another fun one: Fermi's definition of "remote possibility" as a chance of no more than 1 in 10. (Scary when talking about the odds of initiating a chain reaction, making a small atomic weapon, or even igniting the atmosphere in the first atomic test.)
Right, but I'm suggesting that the odds of lighting the atmosphere on fire were calculated to be much lower than 1 in 10, but there just wasn't a 'simple English' category to distinguish them. So I wouldn't get scared until I saw something that suggested it was anywhere near 1 in 10.
which defines the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL". Most subsequent RFCs begin with a reference to it.
Hopefully whoever wrote that never works in a data center, builds cars, sells insurance, gambles, works on crypto, does scientific computing, or is ever responsible for someone's life.
If you do any of those things, you use actual numbers with your words - which, if you actually care about the subject at hand, you should have used numbers in the first place.
Implicit in your statement is the claim that you cannot acknowledge the existence of a percentile without believing any given feature of your domain can easily satisfy that percentile. That's a non sequitur. The conclusion that follows from that one is also very strange, because it would mean we shouldn't bother quantifying things (or having a name for them) if they can only be expected to happen extremely rarely.
If you're talking the realm of five 9s, etc, that's in reference to service availability at some given point in time throughout the course of a year. If you discuss the probability there will be an outage once during a year, the answer is somewhere in the middle, around "more likely than not".
You could use this same probability around a pacemaker. The device is virtually certain (99%) to function at a given point throughout the year, but the probability that the device will not fail over the course of the year is not 99%. If the pacemaker had a 99% chance of not failing once during the course of a year, it would be virtually certain it would not fail during that year.
This makes me thankful for Amazon's document-writing culture. Ambiguous words like very/could/should/few/large are avoided or at least qualified with a range. This helps us avoid ambiguity, regardless of the amount of context a reader has.
In my experience, putting numbers to your gut feeling probabilities causes people to not take them seriously. Example: "I think there is 30% chances that I will get a promotion next month". If I say 30% people ask how I came up with that number. If I say "not very likely" then it gets accepted as an educated guess. Probably because we're used that those numbers must come from some dataset. Or should I say "I think there is a chance of 40% that things are like that because we're used that those numbers must come from some dataset".
30% is specific, as if it's the result of a calculation in a spreadsheet. But with something like getting a promotion, how would you calculate that so specifically? You can't. That's why people don't take it seriously.
"Not very likely" is a rough estimate, like "probably not", or "less than 30% likely". It's a lot more believable that with the information you know, you're able to predict with that certainty.
"Lesson 1: Use probabilities instead of words to avoid misinterpretation."
Probabilities are meaningless unless it’s a repeatable experiment otherwise its a ludic fallacy eg "There's a 70% chance of Hillary winning". This is an un-provable statement. Either she wins and prediction was right, or she loses and it counts as part of the 30%. This is Nate Silver's get-out-of-jail-free card so even when he's wrong he comes off as being right.
i.e. this statement makes sense in a casino and nowhere else.
"Lesson 2: Use structured approaches to set probabilities."
Probabilities are meaningless by themselves. Path dependency matters a great deal to actual humans but not to business professors. A strategy that works well for the ensemble wont necessarily work well for the individual.
e.g. if you save for retirement for the average life expectancy then 50% of the people would be screwed and 50% of the people would have saved too much.
i.e. The cost of being right/wrong is what matters and not the probability.
> Probabilities are meaningless unless it’s a repeatable experiment otherwise its a ludic fallacy
Um... this goes against the entirety of the Bayesian approach to statistics. I think you'd find a lot of very intelligent people who disagree strongly with this statement.
The Bayesian approach takes probabilities as subjective confidences. You can describe confidences as "well calibrated" if, when you look at their historical guesses, if their 70% assessments are correct 70% of the time.
". I think you'd find a lot of very intelligent people who disagree strongly with this statement. "
I never claimed to be intelligent.
" You can describe confidences as "well calibrated" if, when you look at their historical guesses, if their 70% assessments are correct 70% of the time."
- Again, if you're trying to figure out if a coin is split 50/50 then yes, but without being able to repeat the same experiment you're fooling yourself and the whole aspect of bayesian thinking I think goes out the window. eg me being right about unrelated topics doesn't mean I'm right/wrong about specific topics.
It is true that you cannot evaluate a single probabilistic prediction as being right or wrong (so you cannot say that Nate Silver was either wrong or right in some example - he gave a probability, and that probability was either accurate or inaccurate, but in isolation that cannot be evaluated).
However, if a consistent process is used to generate a sequence of probabilistic predictions, the accuracy of the predictions generated by that process certainly can be evaluated.
Absolutely - and my issue is conflating experiments that ARE repeatable (eg a probabalistic game of chance) with experiments that arent (eg election results of two particular candidates). So its one of those heads I win tails you lose bets when it comes to Nate
It works even if the process is "we ask an oracle for a prediction".
The more that Nate Silver produces new prior probability estimates for events, the better an idea we can get of the accuracy of his process. This measure is calibration, as the original article mentions.
If our oracle tells us that an event has a 70% chance of happening, then regardless of whether the event happens or not we can't say much about the oracle's calibration. But if, out of 100 independent events the oracle has predicted to each have a 70% chance of happening, 69 of those events actually happened, then that tells us quite a bit.
Depending on context, I just might ignore it as the padding it is. Same with the negative "I'm not entirely sure X is true" which is either becoming more common, or which I notice more and more. Hiding goal posts in a swamp is worse than moving them.
This article, and some of the comments here, reminded me of another article (and GDC talk) by Civilization game designer Sid Meier, about his experience with players' perception of probability in games.
Sid's talk grapples with the issue "If the game says you have 3-to-1 odds to win a battle, how often do players actually expect to win?"
> When designing the combat system in Civilization: Revolution, Sid Meier found himself up against some interesting design problems. His players didn't understand math. In Civ Rev, the strength of units were displayed up front to players before battle to show the odds of victory. For example, an attacking unit might be rated at 1.5 with the defending unit at 0.5. This is a 3-to-1 situation.
> Unfortunately, the testers expected to win this battle every time despite there being a 25% chance of losing each time. Sid tweaked the math to make the player win more in this situation. Next, the reverse case was tested. The player had 1-to-3 odds. If they won, the math was functioning properly. They had a slim chance to win and they did.
> Sid identified a few cases of interest. When the player was presented with 3-to-1 or 4-to-1 odds, they expect to win. With 2-to-1 odds, the player will accept losing some of the time, but expect to win at 20-to-10, which is just a larger expression of 2-to-1. When the numbers get larger, the perceived advantage grows.
(It's true. When playing a Civ game, it's not fun to have a 10-to-1 strength advantage and lose the battle anyway.)
In later games, Sid removed probability from the game in favor of an outcome that's predictable. Instead, each unit will be damaged in proportion to the ratio of the units' strengths (or something like that).
Maybe the key here is that strength ratio != odds, under most intuitive definitions of strength. If someone is twice as strong as you, they're arguably much more than twice as likely to defeat you. If someone has twice the number of units as you, they're also more than twice as likely to defeat you.
I work in security and I am asked all the time about the likelihood of horrible things which may happen.
I cannot, ever, give a percentage ("70% chance we will get hacked if this and that"). I either say that I do it know and nobody will, or use "probably" or "unlikely". These are wide, hand waving categories.
I do that not because I do not want to be wrong but just because I have only a vague idea, classified into "maybe" and "maybe not"
I PERSONALLY decided if it seems like 20-25% probability, this is 'likely' because it means I can accomplish it if I can try 4-5 times. If it is 1%, I'd have to attempt 100 times to make a thing happen.
115 comments
[ 94.9 ms ] story [ 3434 ms ] threadThe problem is that it's hard to estimate the probability of something that you know happens "almost always" without doing the counting, which most people don't do. And hence it's difficult to gain an intuition for what 99% means exactly.
Furthermore, there is also the possibility that two people share the same intuition about a given word/phrase, but then disagree about which number describes that intuition. This could account for part of the results of this paper.
1. Beyond a reasonable doubt
2. Perponderance of the evidence (“more likely than not”, which in theory is >50%)
3. Clear and convincing (which is somewhere in between the 2 above)
- Very common: affects more than 1 in 10 people – ie the risk is 10% or higher
- Common: affects between 1 in 100 and 1 in 10 people – ie risk is 1% to 10%
- Uncommon: affects between 1 in 1,000 and 1 in 100 people – ie risk is 0.1% to 1%
- Rare: affects between 1 in 10,000 and 1 in 1,000 people – ie risk is 0.01% to 0.1%
-Very rare: affects less than 1 in 10,000 people – ie risk is less than 0.01% (This includes isolated reports)
It's amazing how often they're used in debates or political speech (ex. good, bad, strong, important, dangerous, etc.) when they clearly mean different things to different people, but maybe that's the idea.
There's only so much accuracy you can squeeze into a sentence - some people don't notice, some take advantage, some try to teach others to keep an ear out for this exact problem.
- "Get rid of the adjective-by-habit."
- "Don't hedge your prose with little timidities. Good writing is lean and confident."
One big tinidit in the preface wouldn't be so annoying to read.
This is a separate issue, and one I would disagree with. In fact I feel like this is a problematic attitude in our culture. Too many people overconfidently asserting their subjective viewpoints. Real life is full of nuance, gray area, second-guessing, and different perspectives; simple bold assertions usually don't hold water.
David Foster Wallace, considered one of the greatest writers of recent times, filled his writings with qualifications, side notes, devil's advocacy, and at times insecure waffling. To me this is a more accurate reflection of the mind and reality and should not be suppressed for a macho bravado façade.
"I would disagree with" and "I feel" make it clear this is a personal opinion.
Also note the presence of "usually" in "simple bold assertions usually don't hold water."
Though I do strongly agree with the original comment, so maybe that is the real reason why I am defending it.
But I don't think Zinsser there means to get rid of all 'timidities'. The subtext my mind's eye read there was: don't overdo it. And he goes on to expand: "Every little qualifier whittles away some fraction of the reader's trust. Readers want a writer who believes in himself and in what he's saying. Don't diminish that trust."
For instance, I know smart people who often apologize for no reason, do excessive hedging, or start a request with an apology. It gets a bit annoying when I have to read or hear that everyday.
Or, this is an entirely subjective thing based on your own cultural presuppositions, and you are actually simply rude/prideful.
The coin flips both ways - there is no 'norm' here other than cultural/ethical/philosophical bias.
It's possible to state a lack of certainty with confidence and clarity.
I "dislike Shakespeare's style", I guess that makes it a tautology if you were to say he was great just because you like his style?
The same is true with numbers. If a predicted event happens, one might declare: "I told you there 80 percent that will happen". If it dosen't happen, the fallback might be: "I told you there are 20 percent it will not happen."
Hillary Clinton had a >70% of winning the US presidential election according to the most responsible analyses (see 538: https://projects.fivethirtyeight.com/2016-election-forecast/).
Most folks took 70% to mean that she would certainly win and were bitterly disappointed the morning after.
On the other hand no sane person would (willingly) play Russian roulette with a 70% or even 5 of 6 chance of "winning".
In everyday life, one conflates probability with severity of outcome. This is normal and is part of how people assess and act on risk in a qualitative way.
Now I understand Robert Downey Jrs maths skills in Kiss Kiss Bang Bang.
Plus wasn't it pascal who invented expected value?
And yes he was the inventor.
But logically, if any one God can exist, then so might any God or set of Gods from any pantheon, including sets never even conceived of by human beings. Since the evidence of any one God over another is the same (zero, it's entirely a matter of faith,) the benefit of belief in any one God over another is also zero, since the set of all possible Gods is infinite.
Probability * (new status - old status) = Expected Payoff
Probabilities without confidence intervals[1] are by-and-large meaningless (She has a 90% chance of of winning with a confidence interval of +11% -100%). No amount of d3.js on 538's blog will change this.
https://en.wikipedia.org/wiki/Confidence_interval
They probably should have just written that right there: "71%, better odds than two coins coming up heads."
But: in order to understand if the model is any good, you need to try to build a good model, describe your model, and describe what you know about the underlying processes. With words, visuals, and math. Which is pretty much what 538 does better than most.
The difference is how the probability changes in response to new information. Learning more about the coin won't change the probability of heads from 50%, but doing more exhaustive polls would have likely improved our prediction about the election.
However I don't think there's as much of a difference as you think. If we learnt more about how the coin was going to be flipped then that would certainly improve our estimate of the outcome. If we found out the exact way it would be flipped we could calculate the outcome exactly.
So we can't really compare the two situations quantitatively, since there's no way to match up like-for-like the information we could receive. But we can say, for each possible piece of information we could receive, how much we expect it to change our probability.
But the final probability is still just a single number.
But I suspect the GP's reasoning was getting interference from the valid point that poll numbers, which are not probabilities, are meaningless without confidence intervals or some other measure of uncertainty.
That uncertainty will then feed, via some model, into a probability estimate of a candidate winning. And that's just a single number.
For example, I can imagine a well engineered coin that has a 0.5 probability of heads with a 0.005 confidence (ie, we suspect that the true weighting between heads and tails is likely between 0.495 and 0.505 19 times out of 20) and I can contrast it with a hastily made coin that can still have a 0.5 expected probability with a 0.1 confidence (ie, we expect the true probability to be between 0.4 to 0.6, 19 times out of 20).
Coin A and Coin B have dramatically different impacts on our decision making. For example, selling insurance against 5 identical flips in a row is a much more expensive proposition for Coin B than Coin A.
I feel like I really should know this given my data science background, but sometimes the basics slip away.
We could do the same thing for the 2016 election, but we would have to specify exactly what we meant by a "repeat". Do we just let Hillary run for the 2020 election and see what happens? Or do we put back every atom to the exact position it had in 2014, so that the only divergences between the two elections are caused by quantum randomness? Or something in between, like looking at all elections where a demagogue outsider runs against an established insider?
Really it doesn't matter what definition of "repeat" we choose. Since the repeat won't actually happen, we can't be called to bet on it, so knowing the probability in that case isn't too useful. Whereas a coin actually can be flipped multiple times.
I could imagine some frameworks where confidence intervals in this way would be useful - ex. I have 3 fairly different world models that I think are equally likely, each has a P(election), what's the P(election|world) and get confidence intervals across world models rather than just summing over them to P(election). But I agree that for most common approaches that simple probabilities are most useful and clear.
But there's no meaningful way to repeat an election, so I don't think similar distributions or confidence intervals are useful in that case.
EDIT: We can imagine rerolling an election, but since we can't actually do it we don't have to bet on it so the information wouldn't be very useful to us.
"Bayesian perspective" covers a lot of territory, and your assertion depends on the situation and the modeling objective. If a probability is a model parameter (for instance, frequency of heads for a particular coin), then summarizing the posterior distribution on that parameter with a confidence interval can be a sensible thing to do.
Or do you mean as opposed to a credibility interval?
That's just not true. If I believe that my team has a 20% chance to win and you offer me a bet with anything better than 5-to-1 odds I should take the bet. If you offer me anything worse than 5-to-1, then I should not take the bet. There's no fuzz factor necessary; no confidence interval that I need to use to make the decision.
Perhaps you're getting at the idea of calibration? That it's difficult for a person to know what a 20% chance feels like? But there are still a lot of situations where it's not up to human judgement.
I apologize if I've explained this poorly, this is just my layman's understanding of the matter.
A Bayesian calculating that probability must get a single number too, without error margins. The only difference is that the Bayesian will have to weight his probabilities by how much of the interval is at the "win" and the "no win" scenarios.
It is different if you are measuring how many votes each candidate will have. For that both methods must get intervals and a confidence level.
You would leap at a bet with 10000-1 odds, and never go for 2-1.
Let's say you can make these bets repeatedly. Would you always bet on 5.01-1 odds and never on 4.99-1 odds?
Is it that odd to think that your team has about a 20% chance to win?
Confidence intervals are for predictions of a given value. They are not for probabilities. It doesn't make sense to say, "I think she has a 70% chance of winning, plus or minus 5%." On the other hand, it does make sense to say, I think she will get 48% of the vote, with an interval of +-1%.
- Bruno de Finetti, 1977
If she was polling at 70% that would be an almost 100% chance of winning because a 20% swing is unlikely. However a 70% chance of victory is much much closer.
In other terms, "Expected Value" can help in this case where you factor in the magnitude along with the frequency of occurrence.
My everyday example for that is the weather forecast and the question 'will it rain' often answered with a precipitation probability?
- Probability: How likely is it that I will be hit by at least one rain drop
- Severity: How many rain drops will hit me
It sounds a little abstract, but whenever I see some everyday weather forecast I wonder what they are trying to tell me. At least 0% seems unambiguous to me :-)
The "Probability of Precipitation" (PoP) describes the chance of precipitation occurring at any point you select in the area. How do forecasters arrive at this value? Mathematically, PoP is defined as follows: PoP = C x A where "C" = the confidence that precipitation will occur somewhere in the forecast area, and where "A" = the percent of the area that will receive measurable precipitation, if it occurs at all. So... in the case of the forecast above, if the forecaster knows precipitation is sure to occur (confidence is 100%), he/she is expressing how much of the area will receive measurable rain. (PoP = "C" x "A" or "1" times ".4" which equals .4 or 40%) But, most of the time, the forecaster is expressing a combination of degree of confidence and areal coverage. If the forecaster is only 50% sure that precipitation will occur, and expects that, if it does occur, it will produce measurable rain over about 80 percent of the area, the PoP (chance of rain) is 40%. ( PoP = .5 x .8 which equals .4 or 40%. )
http://www.bom.gov.au/places/vic/melbourne/forecast/detailed...
That’s because there’s no upside to Russian roulette. Put some money on it, and people would be more likely to take the wager. In its standard form (as I understand it), the only benefit to winning is the rush from having played.
Surely, given the outcome, more of those folks were pleasantly surprised at the outcome than "bitterly disappointed".
From my point of view in Philadelphia, however, virtually everyone was disappointed modulo a few crazies or people who had been hiding under a rock or some closeted college republicans.
This just goes to show how bifurcated America has become, people on opposite sides view the other side as incomprehensibly batshit-crazy and/or evil.
Yeah, it was amazing to see the storm of hate that poured down on Nate Silver as a result. Thousands of people calling him a fraud (and worse), saying they would never trust him again, etc. Even if he had said there was a 99% chance, that still leaves 1%!
In cold war era America would people have felt any different knowing that a Russian invasion was only 20% likely rather than 30%? For something so /serious/ any non-zero probability is something that should be prepared for.
https://books.google.com/books?id=2G2TlJOhGI8C&pg=PA280&lpg=...
which defines the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL". Most subsequent RFCs begin with a reference to it.
Take the question "are you coming?". "Maybe" can mean "yes, except in case of nuclear war" to "not unless you kidnap me", depending on the person.
virtually certain: 99-100%
extremely likely: 95-100%
very likely: 90-100%
likely: 66-100%
about as likely as not: 33-66%
more likely than not: >50-100%
more unlikely than likely: 0-<50%
unlikely: 0-33%
very unlikely: 0-10%
extremely unlikely: 0-5%
exceptionally unlikely: 0-1%
[0]: https://ipcc.ch/pdf/assessment-report/ar5/syr/AR5_SYR_FINAL_...
Hopefully whoever wrote that never works in a data center, builds cars, sells insurance, gambles, works on crypto, does scientific computing, or is ever responsible for someone's life.
You could use this same probability around a pacemaker. The device is virtually certain (99%) to function at a given point throughout the year, but the probability that the device will not fail over the course of the year is not 99%. If the pacemaker had a 99% chance of not failing once during the course of a year, it would be virtually certain it would not fail during that year.
If you thinking about crop failure, floods, macroeconomics, population trends, etc, 99% is fine for certainty.
Do you agree?
(Yes, I am mixing things up here)
https://www.gwern.net/docs/statistics/bayes/2008-kesselman.p...
See also https://en.wikipedia.org/wiki/Words_of_estimative_probabilit...
What I find more annoying in terms of reporting:
The fact that news reporting doesn't distinguish likelihood vs. probability (and vice versa).
"Not very likely" is a rough estimate, like "probably not", or "less than 30% likely". It's a lot more believable that with the information you know, you're able to predict with that certainty.
Probabilities are meaningless unless it’s a repeatable experiment otherwise its a ludic fallacy eg "There's a 70% chance of Hillary winning". This is an un-provable statement. Either she wins and prediction was right, or she loses and it counts as part of the 30%. This is Nate Silver's get-out-of-jail-free card so even when he's wrong he comes off as being right.
i.e. this statement makes sense in a casino and nowhere else.
"Lesson 2: Use structured approaches to set probabilities."
Probabilities are meaningless by themselves. Path dependency matters a great deal to actual humans but not to business professors. A strategy that works well for the ensemble wont necessarily work well for the individual.
e.g. if you save for retirement for the average life expectancy then 50% of the people would be screwed and 50% of the people would have saved too much.
i.e. The cost of being right/wrong is what matters and not the probability.
Um... this goes against the entirety of the Bayesian approach to statistics. I think you'd find a lot of very intelligent people who disagree strongly with this statement.
The Bayesian approach takes probabilities as subjective confidences. You can describe confidences as "well calibrated" if, when you look at their historical guesses, if their 70% assessments are correct 70% of the time.
I never claimed to be intelligent.
" You can describe confidences as "well calibrated" if, when you look at their historical guesses, if their 70% assessments are correct 70% of the time."
- Again, if you're trying to figure out if a coin is split 50/50 then yes, but without being able to repeat the same experiment you're fooling yourself and the whole aspect of bayesian thinking I think goes out the window. eg me being right about unrelated topics doesn't mean I'm right/wrong about specific topics.
However, if a consistent process is used to generate a sequence of probabilistic predictions, the accuracy of the predictions generated by that process certainly can be evaluated.
The more that Nate Silver produces new prior probability estimates for events, the better an idea we can get of the accuracy of his process. This measure is calibration, as the original article mentions.
If our oracle tells us that an event has a 70% chance of happening, then regardless of whether the event happens or not we can't say much about the oracle's calibration. But if, out of 100 independent events the oracle has predicted to each have a 70% chance of happening, 69 of those events actually happened, then that tells us quite a bit.
Sid's talk grapples with the issue "If the game says you have 3-to-1 odds to win a battle, how often do players actually expect to win?"
> When designing the combat system in Civilization: Revolution, Sid Meier found himself up against some interesting design problems. His players didn't understand math. In Civ Rev, the strength of units were displayed up front to players before battle to show the odds of victory. For example, an attacking unit might be rated at 1.5 with the defending unit at 0.5. This is a 3-to-1 situation.
> Unfortunately, the testers expected to win this battle every time despite there being a 25% chance of losing each time. Sid tweaked the math to make the player win more in this situation. Next, the reverse case was tested. The player had 1-to-3 odds. If they won, the math was functioning properly. They had a slim chance to win and they did.
> Sid identified a few cases of interest. When the player was presented with 3-to-1 or 4-to-1 odds, they expect to win. With 2-to-1 odds, the player will accept losing some of the time, but expect to win at 20-to-10, which is just a larger expression of 2-to-1. When the numbers get larger, the perceived advantage grows.
http://www.shacknews.com/article/62807/sid-meier-and-rob-par...
Here is a link to the actual GDC talk by Sid with the content about probability: https://youtu.be/bY7aRJE-oOY?t=18m22s
(It's true. When playing a Civ game, it's not fun to have a 10-to-1 strength advantage and lose the battle anyway.)
In later games, Sid removed probability from the game in favor of an outcome that's predictable. Instead, each unit will be damaged in proportion to the ratio of the units' strengths (or something like that).
I cannot, ever, give a percentage ("70% chance we will get hacked if this and that"). I either say that I do it know and nobody will, or use "probably" or "unlikely". These are wide, hand waving categories.
I do that not because I do not want to be wrong but just because I have only a vague idea, classified into "maybe" and "maybe not"