I’ve been doing something similar for investing in stocks and it has worked well. I’m exploring building an NLP-powered version of this (based on crawling news events). If anyone is interested in chatting or collaborating, let me know. :)
Anecdote time: an NLP specialist I once worked with told me about some research he did into determining the "goodness" or "badness" of press releases and SEC filings. Buy on good news, sell on bad news, hopefully fast enough to beat others to it.
After a great deal of crunching and study he said they did come up with a model that could do it reasonably well. Then realised it could be replaced with a simple rule:
Are there a bunch of numbers at the top? Good news.
Are the numbers buried way down? Bad news.
Another simple rule, by the way, is timing vs executive compensation events. If there's an announcement just before a big block of options or RSUs are vesting, then it's good likely to be good news. If it's being posted far away from vesting dates, then it's likely to be bad news.
This is the most dangerously stupid thing I've ever heard. Tetlock is a known academic charlatan pushing his absolutely useless "superforecasting" nonsense which @nntaleb keeps debunking on Twitter. If Tetlock and friends are so good about forecasting the future, why didn't they predict and warn us about COVID-19 BEFORE @nntaleb and friends (including myself) did?
I think you’re partially right. However, it also depends on what you’re comparing to and what the baseline is. There are lots of projects that try to predict the future and using a framework to do so in a systematic way that can improve over time with feedback is really valuable. That part resonates with me.
I do agree that keeping this high level and abstract (scenario planning + probabilistic estimates) isn’t necessarily doing them any favours. Their book on the topic is better, and I do appreciate their argument that it’s not about being a perfect predictor of the future, but simply better than what we have today.
You cannot fundamentally predict the future, fuhgedaboutit, partly for the same reasons we cannot use Turing machines to solve the halting problem for Turing machines.
What you can do, however, is control your exposure to outcomes of unpredictable events. You don't know when a pandemic will hit, but you know it is is matter of time, just like getting hacked. So prepare and design accordingly. Simple. No need for this superforecasting nonsense which doesn't even work.
They would actually argue the same thing. Certain events are impossible to predict.
Other events are not.
That’s part of the beauty of the problem. You need to be able to pick the right events to predict and ensure the event is well-defined enough to actually test. Your own examples of why this wouldn’t work are exactly what they discuss as the problem... Eventually a lot of forecasts are correct; what governments, companies, and leaders need are forecasts that are time bound, as that’s how they plan.
Show me the money: did they warn about COVID-19 way BEFORE it be came a problem, or did they not? If not, why should we take them seriously? It is easy to prognosticate AFTER the fact.
not seeing one crisis doesn't mean they aren't better than chance. Also do you know that they didn't make some accurate prediction of XX% of a pandemic that was born out in this situation? No one is claiming to have a literal crystal ball.
This argument relies on the nirvana fallacy. "You are not perfectly clairvoyant, therefore prediction is worthless".
I recall, but cannot locate, that questions related to the outbreak of trans-species respiratory illnesses from China have been canvassed in the past. It does appear that the US intelligence community expected a pandemic to emerge: https://www.mercurynews.com/2020/03/16/analysis-yes-people-d...
IDK how many more times I have to repeat this before it gets through your head that one warned about the pandemic blowing up before it happened, therefore only one can be taken seriously. You don't need evidence to run away when someone shouts that a bear is coming your way.
"There will be a pandemic" is a prediction with 100% accuracy. Any rare event, as Taleb likes to hang his entire career on, will eventually happen.
But such a prediction is basically worthless without a horizon. Information about when a pandemic happens affects what actions have to be made to deal with it. Pandemics that definitely happen tomorrow require far more expensive and disruptive actions than ones happening sometime in the next decade with gradually increasing probability.
Tetlock's work is about people who actually have to define a date and for whom the deciding measurements are objective. You can't just say "a pandemic!", you have to say "a pandemic is declared by WHO on or before January 1st, 2019".
In fact, Tetlock's data undermines Taleb's entire thesis that dramatic events are systematically under-predicted. He showed that dramatic scenarios are over-predicted by experts. They have higher emotional salience, and history that is taught focuses on dramatic, outlier events because of their disproportionate impact. Due to availability bias and hindsight bias, experts typically predict that dramatic events will happen more frequently than they actually do. Such predictions are regularly roflstomped by hilariously simple forecasting methods like "yesterday's weather" or fitting a line on a handful of data points.
As I said earlier: Taleb has anecdotes and insults. Tetlock has data.
When did anyone predict COVID? You can just throw around assertions that someone predicted something like COVID without providing evidence and expect people to make judgements on it.
You could be right, but I've people-watched on Twitter long enough to know that "person who insults those he disagrees with while name-dropping @nntaleb" is an entire PERSONALITY, one that I take approximately 0% seriously.
Taleb has deep insights about some things, is laughably ignorant on other things, and doesn't seem to know which is which. His followers definitely don't.
Thanks for the context. I was trying to figure what was noteworthy about an article saying we should think ahead which I kind of assumed we did anyway. Though I think "most dangerously stupid thing I've ever heard" is maybe an exaggeration. The US govt effectively banning coronavirus testing back in Feb kind of sticks out in my mind as outstanding on that front (https://www.nytimes.com/2020/03/10/us/coronavirus-testing-de...)
How was I "attacking" others, when they were allowed to put words in my mouth? Go ahead and ban me: @paulgraham and @sama were right about the intolerance of free speech in Silicon Valley.
Hey there, I’m a longtime Taleb follower (Antifragile was life-changing for me), and I watched him and you and others in your crew commentate on the pandemic and found it highly beneficial for understanding the scenario. I’d consider it a great shame if there were no place for you here.
But this attack on dang misses the mark badly.
pg and sama have always been fully supportive of the way he runs HN (pg hand-picked him to take it over, sama was YC CEO for most of the time since, and they've both repeatedly endorsed him and his HN work publicly [1], including recently in pg's case [2]).
dang’s approach to moderation makes it more conducive to free speech, as, unlike on Twitter, it allows difficult/complex topics to be discussed without being derailed by personal insults.
We don’t need to attack people here; we have the downvote arrow and flag link for when people are being assholes, otherwise we can just focus on presenting persuasive arguments.
I don't want to ban you! I'd rather try to persuade you to make your points without sharp elbows. Others also do bad things (for example, I can see how https://news.ycombinator.com/item?id=24772044 was a provocation) - but that it doesn't make it ok to escalate; we just get a downward spiral that way. Rather, we each need to spend the energy and time needed not to escalate. That creates capacity to then respond neutrally with good points. This strengthens the commons rather than damaging it—and as a bonus, your comments become more persuasive.
"Free speech" means different things to different people and depending on what one wants, there are different tradeoffs. On HN, we're trying to optimize for interesting speech, i.e. curious conversation: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... That means we have to moderate people away from being nasty to each other. I don't think that's "intolerance of free speech", because HN's guidelines leave lots of room for arguing most views.
pg had a much lower threshold for banning people than we do, FWIW.
If you're not persuaded, https://news.ycombinator.com/item?id=24620683 is an explanation I wrote recently which makes the same points I'm trying to make here and gives more examples. It also links to other comments on the same. The point is that smart people often come here with a combative model of how to do intellectual discussion that actually works well in smaller, more cohesive communities. The reason we don't allow it here is not because we think it's bad in itself, but because it doesn't translate well into HN's context—here it's just a recipe for tedious flamewar.
> Well-calibrated forecasters, for instance, can estimate the likelihood that a skirmish with the Chinese navy in the South China Sea will result in at least two American deaths by December 31. But what policymakers really want to know is the extent to which China will threaten U.S. interests in the coming years and decades.
This has never happened (a skirmish, let alone deaths) so how can we think that the forecasters are well-calibrated to predict the likelihood of this event? Because they predicted other, different, stuff?
In fact, yes. Prediction is a skill, skills can be developed and different predictors will have varying levels of skillfulness.
The biggest contributor IIRC is Fermi-ization: the skill of decomposing big hard predictions into many smaller predictions.
For example: what are the components of the question? They might include:
- The likelihood of deaths due to combat action. That can in turn include predictions of the relative effects of ship armour, weapons effectiveness, preparedness for combat conditions and so on.
- The number of people involved. If a skirmish involves a hundred ships on each side, 2 deaths is a very low estimate. How many ships are in the area? How many could reach it by December 31st? How many would be likely to be sent?
- The causes of conflict. What are the triggering events, and how likely are they to occur?
I'm sure smarter and better-qualified folks than me can argue all the details, but that's the point. You don't accept the question as a coin-flip with no priors. You go prior-hunting.
The other big component is using Bayesian inference. For that you need a prior. What's the prior for this? You could look at incidents the US has had in similar freedom of navigation actions, what China has done with other countries, or other similar situations. For example, I can only think of a few similar incidents, and none that escalated to fatalities at sea. So your prior would probably be pretty small, say 1%. After that you would adjust according to new data. For example, China seems willing to do more risky stuff lately, with India being a perfect example. You might raise your estimate based on that. So maybe that raises your estimate to 2%. But it probably shouldn't completely alter your estimate.
Ok, that makes sense, but my question isn't "how would I go about coming up with an estimate?" it's "how could I claim my estimate is accurate, given that the thing I'm estimating has never happened before?". Even if we extrapolate the verified accuracy of predicting other stuff over to this event, we surely can't know we're right?
The article seems to be suggesting that this can be accurately estimated, but I feel like this would be a stronger claim if they cited something that was predicted and actually did happen...
In "Superforecasting", it's a bit clearer. What they end up doing is asking you to predict an EVENT by a SPECIFIC TIME. In other words, what is the probability that "x" will happen by "DATE Y".
In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not.
There is some art to this -- the "event" in question needs to be very well-defined... I think the example they're using is a bit odd and they have much better examples in their book. The example of a counterfactual event (i.e., 'if X happens then estimate the likelihood of Y happening by DATE Z') adds a great deal of confusion here because you're predicting conditional probabilities. Still possible to verify but to your point, much more difficult.
At this point, you're asking forecasters to build a probabilistic graph and estimate causality, which is becoming more of a science (e.g., Judea Pearle's work) but still lots to do in that space. Anyway, I'm digressing.
Adding slightly to it: they use Brier scores to say how "good" someone is at prediction. Because these predictions require a probability of confidence, you can punish those who were confidently wrong or reward those who were confidently right.
So if I predict "aliens will land on or before December 1st" with 100% and no aliens appear, then my Brier score is 1 (the worst). If I say 0% confidence, then my score is 0 (the best).
There's strong empirical evidence that prediction ability cross generalizes. That is, if a person is good at predicting unrelated topics A, B, and C, they are also like to be good at topic D.
But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once?
Unless the prediction is "this won't happen" and then it continuing to not happen makes the predictor look accurate... (slightly joking here)
The article seems to claim that a specific event, which has never happened, can be estimated accurately. Is that really possible? How can the accuracy be assessed?
Also worth noting: it's about being better than an existing baseline. Effectiveness is measured in relation to a baseline accuracy, which can already be quite poor.
Suppose you make predictions about 100 events, and only 25 of these events actually take place. Your baseline might be in relation to these 25 events, and your 'superforecaster' needs to be better than an existing model, which could be really bad at predicting these odd/weird outcomes. Now you're better than everything else! :)
> But presumably you don't know how good they are at predicting topic D until topic D actually occurs at least once?
No, that's not right. The general idea is that you have forecasters who make predictions on multiple topics A, B, C, and D that have all not happened at the time of the prediction. Then, by looking at the outcomes of A, B, and C you can judge the ability of the forecaster to make prediction on topics what have not yet occurred at the time of prediction. You can then become confident on their ability to predict topic D before it occurs.
The argument you're making is the classic skeptical argument about the philosophical problem of induction: just because some procedure (e.g., an astronomical model of the solar system) has made successful predictions in the past, we cannot rule out the possibility that the sun will not rise tomorrow.
You can see where the problem with your argument lies by noticing that it depends heavily on the definition of what a "topic" is. If we have made successful predictions in the past on (say) the tides, and I make new predictions in the future, one can always point to some feature of the new predictions that could arguably make it a different topic, e.g., maybe El Nino is this year, or maybe the tides were for a slightly different region, etc. New predictions always differ from previous predictions in some way, but this does not mean the predictions are useless.
The resolution is that "topic" is not a precise idea. Rather, if a forecaster is successful on topics A, B, and C that are all in a natural reference class, and D is also in that reference class, then this is evidence — not incontrovertible evidence, but strong compelling evidence —that they are likely to be accurate on D too.
> The general idea is that you have forecasters who make predictions on multiple topics A, B, C, and D that have all not happened at the time of the prediction. Then, by looking at the outcomes of A, B, and C you can judge the ability of the forecaster to make prediction on topics what have not yet occurred at the time of prediction. You can then become confident on their ability to predict topic D before it occurs.
That's not my point, though - perhaps I didn't word it clearly.
I'm not saying this event can't or won't happen, I'm questioning the claim (maybe I'm misinterpreting the article?) that we can accurately forecast the likelihood of it happening, even though it's never happened.
Odd to see an article about forecasting without mentioning DeMesquita and Smith's predictioneer models and stakeholder salience models. Also without Charlie Munger's comments on incentives, which have had a better track record than pretty much anyone.
There was much point scoring done after the 2016 U.S. election about polling and predictions being wrong, particularly against Fivethirtyeight, but where I think critics got it wrong was, when someone says there's a %95 chance of an event going in one way, it's not valuable as a prediction, but it certainly is as an accurate assessment of the impact of the results. In 2016, if you told people that the %95 implied probability of winning translated to what felt like paying out a 19/1 loss, a lot of people would have agreed with you.
You can reliably bet on the effect of the upset being inversely proportional to the predicted probabilities. When I see election predictions that are 60:40 I think the 2nd order impacts of the result will be slightly lopsided, but within the realms of expectations. In this sense, %54 chance of winning close races may be better for stability than perceived polarized %95 ones because of how people perceive the results of a loss.
A travesty that the U.S. pays over a trillion dollars a year while hosting soldiers in 70 countries...
You would think with that much planning we would have paid attention to experts decades ago warning that we needed more charged fiber (N95) in storage in case of a pandemic at a cost of ~0.0001 trillion ...
50 comments
[ 0.33 ms ] story [ 114 ms ] threadAfter a great deal of crunching and study he said they did come up with a model that could do it reasonably well. Then realised it could be replaced with a simple rule:
Are there a bunch of numbers at the top? Good news.
Are the numbers buried way down? Bad news.
Another simple rule, by the way, is timing vs executive compensation events. If there's an announcement just before a big block of options or RSUs are vesting, then it's good likely to be good news. If it's being posted far away from vesting dates, then it's likely to be bad news.
P.S. I have been downvoted for saying this.
I do agree that keeping this high level and abstract (scenario planning + probabilistic estimates) isn’t necessarily doing them any favours. Their book on the topic is better, and I do appreciate their argument that it’s not about being a perfect predictor of the future, but simply better than what we have today.
What you can do, however, is control your exposure to outcomes of unpredictable events. You don't know when a pandemic will hit, but you know it is is matter of time, just like getting hacked. So prepare and design accordingly. Simple. No need for this superforecasting nonsense which doesn't even work.
Other events are not.
That’s part of the beauty of the problem. You need to be able to pick the right events to predict and ensure the event is well-defined enough to actually test. Your own examples of why this wouldn’t work are exactly what they discuss as the problem... Eventually a lot of forecasts are correct; what governments, companies, and leaders need are forecasts that are time bound, as that’s how they plan.
I recall, but cannot locate, that questions related to the outbreak of trans-species respiratory illnesses from China have been canvassed in the past. It does appear that the US intelligence community expected a pandemic to emerge: https://www.mercurynews.com/2020/03/16/analysis-yes-people-d...
Tetlock has data piled on data.
But such a prediction is basically worthless without a horizon. Information about when a pandemic happens affects what actions have to be made to deal with it. Pandemics that definitely happen tomorrow require far more expensive and disruptive actions than ones happening sometime in the next decade with gradually increasing probability.
Tetlock's work is about people who actually have to define a date and for whom the deciding measurements are objective. You can't just say "a pandemic!", you have to say "a pandemic is declared by WHO on or before January 1st, 2019".
In fact, Tetlock's data undermines Taleb's entire thesis that dramatic events are systematically under-predicted. He showed that dramatic scenarios are over-predicted by experts. They have higher emotional salience, and history that is taught focuses on dramatic, outlier events because of their disproportionate impact. Due to availability bias and hindsight bias, experts typically predict that dramatic events will happen more frequently than they actually do. Such predictions are regularly roflstomped by hilariously simple forecasting methods like "yesterday's weather" or fitting a line on a handful of data points.
As I said earlier: Taleb has anecdotes and insults. Tetlock has data.
https://www.nationalgeographic.com/science/2020/04/experts-w...
I'll bite, what?
I know he's rich and successful, so he has deeper insights than most.
But it doesn't mean he teaches them, as far I can see he just uses them. For instance he's very good at doublespeak.
You may not owe authors whom you feel are charlatans better, but you owe this community better if you're posting to it.
Edit: you've unfortunately been breaking the site guidelines a lot lately. Comments like https://news.ycombinator.com/item?id=24772993 and https://news.ycombinator.com/item?id=24772982 are not ok. Would you please review https://news.ycombinator.com/newsguidelines.html and stick to the rules? I don't want to ban you, but we can't have people throwing their weight around and attacking others like this.
But this attack on dang misses the mark badly.
pg and sama have always been fully supportive of the way he runs HN (pg hand-picked him to take it over, sama was YC CEO for most of the time since, and they've both repeatedly endorsed him and his HN work publicly [1], including recently in pg's case [2]).
dang’s approach to moderation makes it more conducive to free speech, as, unlike on Twitter, it allows difficult/complex topics to be discussed without being derailed by personal insults.
We don’t need to attack people here; we have the downvote arrow and flag link for when people are being assholes, otherwise we can just focus on presenting persuasive arguments.
[1] https://blog.ycombinator.com/two-hn-announcements/
[2] https://twitter.com/paulg/status/1282055086433284103
"Free speech" means different things to different people and depending on what one wants, there are different tradeoffs. On HN, we're trying to optimize for interesting speech, i.e. curious conversation: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor... That means we have to moderate people away from being nasty to each other. I don't think that's "intolerance of free speech", because HN's guidelines leave lots of room for arguing most views.
pg had a much lower threshold for banning people than we do, FWIW.
If you're not persuaded, https://news.ycombinator.com/item?id=24620683 is an explanation I wrote recently which makes the same points I'm trying to make here and gives more examples. It also links to other comments on the same. The point is that smart people often come here with a combative model of how to do intellectual discussion that actually works well in smaller, more cohesive communities. The reason we don't allow it here is not because we think it's bad in itself, but because it doesn't translate well into HN's context—here it's just a recipe for tedious flamewar.
This has never happened (a skirmish, let alone deaths) so how can we think that the forecasters are well-calibrated to predict the likelihood of this event? Because they predicted other, different, stuff?
In fact, yes. Prediction is a skill, skills can be developed and different predictors will have varying levels of skillfulness.
The biggest contributor IIRC is Fermi-ization: the skill of decomposing big hard predictions into many smaller predictions.
For example: what are the components of the question? They might include:
- The likelihood of deaths due to combat action. That can in turn include predictions of the relative effects of ship armour, weapons effectiveness, preparedness for combat conditions and so on.
- The number of people involved. If a skirmish involves a hundred ships on each side, 2 deaths is a very low estimate. How many ships are in the area? How many could reach it by December 31st? How many would be likely to be sent?
- The causes of conflict. What are the triggering events, and how likely are they to occur?
I'm sure smarter and better-qualified folks than me can argue all the details, but that's the point. You don't accept the question as a coin-flip with no priors. You go prior-hunting.
The article seems to be suggesting that this can be accurately estimated, but I feel like this would be a stronger claim if they cited something that was predicted and actually did happen...
Perhaps I misinterpreted the paragraph?
In this case, by the time DATE Y takes place, you have a clearly verifiable binary outcome, which you can use to judge whether the forecasters were correct or not.
There is some art to this -- the "event" in question needs to be very well-defined... I think the example they're using is a bit odd and they have much better examples in their book. The example of a counterfactual event (i.e., 'if X happens then estimate the likelihood of Y happening by DATE Z') adds a great deal of confusion here because you're predicting conditional probabilities. Still possible to verify but to your point, much more difficult.
At this point, you're asking forecasters to build a probabilistic graph and estimate causality, which is becoming more of a science (e.g., Judea Pearle's work) but still lots to do in that space. Anyway, I'm digressing.
So if I predict "aliens will land on or before December 1st" with 100% and no aliens appear, then my Brier score is 1 (the worst). If I say 0% confidence, then my score is 0 (the best).
There are dozens and dozens of ways of scoring prediction ability, as it happens: https://www.cawcr.gov.au/projects/verification/verif_web_pag...
Unless the prediction is "this won't happen" and then it continuing to not happen makes the predictor look accurate... (slightly joking here)
The article seems to claim that a specific event, which has never happened, can be estimated accurately. Is that really possible? How can the accuracy be assessed?
Suppose you make predictions about 100 events, and only 25 of these events actually take place. Your baseline might be in relation to these 25 events, and your 'superforecaster' needs to be better than an existing model, which could be really bad at predicting these odd/weird outcomes. Now you're better than everything else! :)
No, that's not right. The general idea is that you have forecasters who make predictions on multiple topics A, B, C, and D that have all not happened at the time of the prediction. Then, by looking at the outcomes of A, B, and C you can judge the ability of the forecaster to make prediction on topics what have not yet occurred at the time of prediction. You can then become confident on their ability to predict topic D before it occurs.
The argument you're making is the classic skeptical argument about the philosophical problem of induction: just because some procedure (e.g., an astronomical model of the solar system) has made successful predictions in the past, we cannot rule out the possibility that the sun will not rise tomorrow.
You can see where the problem with your argument lies by noticing that it depends heavily on the definition of what a "topic" is. If we have made successful predictions in the past on (say) the tides, and I make new predictions in the future, one can always point to some feature of the new predictions that could arguably make it a different topic, e.g., maybe El Nino is this year, or maybe the tides were for a slightly different region, etc. New predictions always differ from previous predictions in some way, but this does not mean the predictions are useless.
The resolution is that "topic" is not a precise idea. Rather, if a forecaster is successful on topics A, B, and C that are all in a natural reference class, and D is also in that reference class, then this is evidence — not incontrovertible evidence, but strong compelling evidence —that they are likely to be accurate on D too.
Thanks, that really clarified the idea for me :)
I'm not saying this event can't or won't happen, I'm questioning the claim (maybe I'm misinterpreting the article?) that we can accurately forecast the likelihood of it happening, even though it's never happened.
There was much point scoring done after the 2016 U.S. election about polling and predictions being wrong, particularly against Fivethirtyeight, but where I think critics got it wrong was, when someone says there's a %95 chance of an event going in one way, it's not valuable as a prediction, but it certainly is as an accurate assessment of the impact of the results. In 2016, if you told people that the %95 implied probability of winning translated to what felt like paying out a 19/1 loss, a lot of people would have agreed with you.
You can reliably bet on the effect of the upset being inversely proportional to the predicted probabilities. When I see election predictions that are 60:40 I think the 2nd order impacts of the result will be slightly lopsided, but within the realms of expectations. In this sense, %54 chance of winning close races may be better for stability than perceived polarized %95 ones because of how people perceive the results of a loss.
> You can reliably bet on the effect of the upset being inversely proportional to the predicted probabilities.
Broadly, yes, if you think of it in an information-theoretic sense. Rare signals carry more information than common ones.
You would think with that much planning we would have paid attention to experts decades ago warning that we needed more charged fiber (N95) in storage in case of a pandemic at a cost of ~0.0001 trillion ...