I made a small amount too, with a much smaller amount of effort :-). Very promising initial study on forecasting scientific replications with a prediction market.
I feel like I vaguely recall a study (probably from being posted on HN) that showed that most people can successfully predict which psychological/behavioral studies will replicate. That's good support for a market.
FWIW, I think it already has, actually, multiple times.
My guess there are few factors in place -- people recognized motivated reasoning -- there are many studies the authors clearly want to be true, e.g. "People I disagree with are evil ugly dum-dums" is a recurring one.
There's also the studies that go against our experience, or what seems obviously right "Mormons are more likely to be alcoholics". These have a tendency to grab headlines (and likely grant money), and while occasionally are true, generally seem not to be.
> most people can successfully predict which psychological/behavioral studies will replicate
Here's the underlying claim:
> Results showed that these laypeople predicted replication success with above-chance accuracy (i.e., 59%). In addition, when participants were informed about the strength of evidence from the original studies, this boosted their prediction performance to 67%
My problem is with what does "above chance" really mean.
> For instance, several systematic high-powered replication projects have demonstrated successful replication rates ranging from 36% (Open Science Collaboration, 2015) to 50% (Klein et al., 2018), 62% (Camerer et al., 2018), and 85% (Klein et al., 2014)
It sounds like we don't actually know what our ground truth replication rate is. I think what we're actually above-chance predicting is not the replication rate but just the baseline uncertainty challenge of scientific funding ("I have $X and I need to spend it on $SCIENCE - here go do some $SCIENCE things because my experts tell me you're good at $SCIENCE") + the low-level corruption it brings which blinds us to reality.
> If laypeople can indeed make accurate predictions about replicability, these predictions may supplement theoretical considerations concerning the selection of candidate studies for replication projects. Given limited resources, laypeople’s predictions concerning replicability could be used to define the subset of studies for which one can expect to learn the most from the data. In other words, researchers could use laypeople’s predictions as input to assess information gain in a quantitative decision-making framework for replication (Hardwicke, Tessler, Peloquin, & Frank, 2018; MacKay, 1992). This framework follows the intuition that—for original studies with surprising effects (i.e., low plausibility) or small sample sizes (i.e., little evidence)—replications can bring about considerable informational gain.
I'm betting against ^this as being the replicatable result (that this produces better scientific outcomes). That's just saying "humans have confirmation bias". This is a useful tool to understand our how we should remove confirmation bias from our experiment selection/setup NOT as a way to guide the selection/setup itself blindly. Imagine if you said "we should let the majority of people weigh in on surprising mathematical/biotech/chemistry/biology/drug discovery results". That's an INSANE statement for anyone in the business of scientific discovery. To me what it signals is that the people doing "social science" isn't building good theoretical models of how humans work and therefore the baseline quality of research is insanely low. This makes sense when you consider that the financial incentives and difficulty of the problem interplay with each other. There's some very good people who are doing the work of wading through the garbage, but they can't keep up with the volume of bullshit that's getting generated.
So if you're willing to place a bet on that specific hypothesis, let me know.
I worry a bit that it would just become self-reinforcing, although I'm largely agreement with you. Some former NIH directors suggested grant funding should basically be a lottery outcome; maybe grants should be funded in proportion to market predicted risk or something? You'd probably want to be funding some high risk studies. Or maybe you want to fund those more? Seems like there's some issues there, in that if the market is really strongly predicting some outcome, maybe the study is unnecessary?
In any event, I've often thought that grant decisions should somehow be opened up more somehow. Obviously it could get out of hand, but the idiosyncratic decisions of grant review panels seems to lead to a lot of navel gazing-type processes.
I once sketched out a cyber-dystopia plot like this (or was it a business plan?) - with all the hubbub around independent fact-checkers and fake news. I figure whoever has the most accurate information about what's true will have the best odds at predicting the future, so how about a platform for attaching news articles to a prediction, and paying out whether the prediction comes true. Essentially longbets.org except much shorter timescales, and with news articles attached to predictions.
Eventually you could even follow journalists based on how often their reporting is used to make an accurate prediction - would probably turn into a patrons-only newsletter sort of thing, the better you're able to predict what happens next. I guess the stock market is already doing this for corporate outcomes, but it might be interesting to be able to bet on the court cases, elections, global warming damages etc. (To be clear, I understand people already make bets on this, the dystopia is having the betting odds of some outcome attached to each piece of news)
10 comments
[ 4.5 ms ] story [ 31.3 ms ] threadIt could fund replication studies.
Also, companies who rely on scientific output can trade in the market to hedge against the science their business depends on being false.
At scale, this might even be a better way to fund science than blue-sky research grants or patents and licensing.
edit: https://news.ycombinator.com/item?id=24449795
Laypeople can predict which social-science studies will replicate successfully
My guess there are few factors in place -- people recognized motivated reasoning -- there are many studies the authors clearly want to be true, e.g. "People I disagree with are evil ugly dum-dums" is a recurring one.
There's also the studies that go against our experience, or what seems obviously right "Mormons are more likely to be alcoholics". These have a tendency to grab headlines (and likely grant money), and while occasionally are true, generally seem not to be.
> most people can successfully predict which psychological/behavioral studies will replicate
Here's the underlying claim:
> Results showed that these laypeople predicted replication success with above-chance accuracy (i.e., 59%). In addition, when participants were informed about the strength of evidence from the original studies, this boosted their prediction performance to 67%
My problem is with what does "above chance" really mean.
> For instance, several systematic high-powered replication projects have demonstrated successful replication rates ranging from 36% (Open Science Collaboration, 2015) to 50% (Klein et al., 2018), 62% (Camerer et al., 2018), and 85% (Klein et al., 2014)
It sounds like we don't actually know what our ground truth replication rate is. I think what we're actually above-chance predicting is not the replication rate but just the baseline uncertainty challenge of scientific funding ("I have $X and I need to spend it on $SCIENCE - here go do some $SCIENCE things because my experts tell me you're good at $SCIENCE") + the low-level corruption it brings which blinds us to reality.
> If laypeople can indeed make accurate predictions about replicability, these predictions may supplement theoretical considerations concerning the selection of candidate studies for replication projects. Given limited resources, laypeople’s predictions concerning replicability could be used to define the subset of studies for which one can expect to learn the most from the data. In other words, researchers could use laypeople’s predictions as input to assess information gain in a quantitative decision-making framework for replication (Hardwicke, Tessler, Peloquin, & Frank, 2018; MacKay, 1992). This framework follows the intuition that—for original studies with surprising effects (i.e., low plausibility) or small sample sizes (i.e., little evidence)—replications can bring about considerable informational gain.
I'm betting against ^this as being the replicatable result (that this produces better scientific outcomes). That's just saying "humans have confirmation bias". This is a useful tool to understand our how we should remove confirmation bias from our experiment selection/setup NOT as a way to guide the selection/setup itself blindly. Imagine if you said "we should let the majority of people weigh in on surprising mathematical/biotech/chemistry/biology/drug discovery results". That's an INSANE statement for anyone in the business of scientific discovery. To me what it signals is that the people doing "social science" isn't building good theoretical models of how humans work and therefore the baseline quality of research is insanely low. This makes sense when you consider that the financial incentives and difficulty of the problem interplay with each other. There's some very good people who are doing the work of wading through the garbage, but they can't keep up with the volume of bullshit that's getting generated.
So if you're willing to place a bet on that specific hypothesis, let me know.
In any event, I've often thought that grant decisions should somehow be opened up more somehow. Obviously it could get out of hand, but the idiosyncratic decisions of grant review panels seems to lead to a lot of navel gazing-type processes.
Eventually you could even follow journalists based on how often their reporting is used to make an accurate prediction - would probably turn into a patrons-only newsletter sort of thing, the better you're able to predict what happens next. I guess the stock market is already doing this for corporate outcomes, but it might be interesting to be able to bet on the court cases, elections, global warming damages etc. (To be clear, I understand people already make bets on this, the dystopia is having the betting odds of some outcome attached to each piece of news)
Kind of disappointing to read halfway through, though still a good article