and asked participants to sketch out a prediction for sea ice through 2025
The good thing is that 2025 is only 7 years away, so we will likely see the actual numbers to compare with. I hope the researchers have saved their data until then.
Actually the predictions of the participants is orthogonal to the purpose of the study. The study was meant to determine how political labels affect cooperation and understanding. Climate change was only one of many issues that could have been used for this.
I do however take issue with headlines that try to draw conclusions from a study other than “this study happened and this was the result.”
Remembering exactly this I went and found the paper and looked and the sample sizes aren't that small, I couldn't see any missing comparisons, and the relevant P values are less than 0.01 so not obviously super P-hacked.
On the other hand it is a priming study that would probably not have been published if the result went the other way, and it's a newsworthy result. All indicators of non-replication.
I got 88% accuracy on the replication quiz and if this was on the quiz I would have marked it as unsure. I don't think it's obviously bogus, but also I'm not totally convinced.
Allow me to hijack this thread. There is something very similar to the issue around repeatability of results in Psych fields and the lack of experimental evidence in theoretical particle physics which has been the focus of previous articles on HN. Both fields are dealing with large elephants in their respective fields in which some people know what the issue is, but rather than having a widespread reckoning, people keep publishing papers as if everything is okay. If anything, this demonstrates a problem that peer-reviewed science is not able to fix.
I have my own thoughts, it's due to the siloed nature of science today, given publish of perish forces us to keep our head down and keep publishing whilst ignoring the big picture problems with the hope someone with more funding in a more prestiguous position will do something. It's more problematic in Psych though, because people and their quality of life are at stake which is not a situation in theoretical physics.
Can we stop posting articles like this to HN when the underlying paper isn’t publicly available?
Especially these “doing X causes [major behavioral change]” papers where the sample size is super important, and they also tend to have very low reproducibility rates?
2400 people is not a lot, especially for a claim like this.
Also there was a similarly sized study a few weeks ago, and basically all participants were students or faculty at 2 different universities.
If you are going to make a claim like this, you need to have a very large and very significant measurable effect, and the smaller your sample size the larger the effect needs to be to have any confidence of the correctness of the study.
Again, to be on HN the paper itself, and the source data, should be freely available. By allowing them to be on HN as they are we are essentially allowing unchecked press releases that always over state the validity of claims.
>2400 people is not a lot, especially for a claim like this.
For a population size of 350,000,000 people the required sample size for a 95% confidence level within a 2% margin of error is 2,401 people. Assuming these 2,401 people were selected properly then that is just about enough, even for a claim like this.
To be „chosen properly” means that the sample must have similar age, racial, educational, gender structure as the 350M society it is supposed to represent. Recruiting 2401 college students as the only sample automatically makes any study unscientific bullshit.
Also, if you ask me, 95% confidence with 2% error margin is pretty low bar.
There's this cross section of people who physically wretch and convulse at hearing that word, and can be whipped into an animalistic rage with the mention of it.
When I hear it used in a certain way, by certain media outlets, it's obvious that they're gunning for a pavlovian response.
It's incredibly manipulative, and the real problem I have with it, is that an astounding number of people eat it up, eager to froth at the mouth and start speaking in tongues, wailing and gnashing teeth.
It's funny, but maybe three quarters of all people are looking for a reason to just unload on someone, at any given moment. They really want the lid lifted, so they can vent some rage onto another person. People may differ on how they rationalize the way they find targets, but one common trait unites them, in that more than half of everyone is looking for a fight.
18 comments
[ 0.21 ms ] story [ 53.2 ms ] threadThe good thing is that 2025 is only 7 years away, so we will likely see the actual numbers to compare with. I hope the researchers have saved their data until then.
I do however take issue with headlines that try to draw conclusions from a study other than “this study happened and this was the result.”
Here's a fun quiz where people can guess the results: https://80000hours.org/psychology-replication-quiz/
One of the trends was that priming studies usually failed. I would assume that the quality of study form OP is similar.
On the other hand it is a priming study that would probably not have been published if the result went the other way, and it's a newsworthy result. All indicators of non-replication.
I got 88% accuracy on the replication quiz and if this was on the quiz I would have marked it as unsure. I don't think it's obviously bogus, but also I'm not totally convinced.
Wasn’t the original study which claimed to “discover” priming recently debunked as not replicable?
> This page is not set up to run this program.
I have my own thoughts, it's due to the siloed nature of science today, given publish of perish forces us to keep our head down and keep publishing whilst ignoring the big picture problems with the hope someone with more funding in a more prestiguous position will do something. It's more problematic in Psych though, because people and their quality of life are at stake which is not a situation in theoretical physics.
Especially these “doing X causes [major behavioral change]” papers where the sample size is super important, and they also tend to have very low reproducibility rates?
> they recruited 2,400 people to join four custom-made social networks
Also there was a similarly sized study a few weeks ago, and basically all participants were students or faculty at 2 different universities.
If you are going to make a claim like this, you need to have a very large and very significant measurable effect, and the smaller your sample size the larger the effect needs to be to have any confidence of the correctness of the study.
Again, to be on HN the paper itself, and the source data, should be freely available. By allowing them to be on HN as they are we are essentially allowing unchecked press releases that always over state the validity of claims.
For a population size of 350,000,000 people the required sample size for a 95% confidence level within a 2% margin of error is 2,401 people. Assuming these 2,401 people were selected properly then that is just about enough, even for a claim like this.
Also, if you ask me, 95% confidence with 2% error margin is pretty low bar.
welcome to psychology studies
Search with the DOI on SciHub, you'll get a PDF of the paper.
When I hear it used in a certain way, by certain media outlets, it's obvious that they're gunning for a pavlovian response.
It's incredibly manipulative, and the real problem I have with it, is that an astounding number of people eat it up, eager to froth at the mouth and start speaking in tongues, wailing and gnashing teeth.
It's funny, but maybe three quarters of all people are looking for a reason to just unload on someone, at any given moment. They really want the lid lifted, so they can vent some rage onto another person. People may differ on how they rationalize the way they find targets, but one common trait unites them, in that more than half of everyone is looking for a fight.