I'm not much of a psychologist, but I can see at least one paper by Khaneman, who won the economics Nobel for that sort of stuff. So not sure if you're being sarcastic or not?
it's almost as if they are avoiding all the experiments that have been replicated so often the result barely bears mention in the papers using them in favor of random crap from the field.
"It was surprising that even with these efforts we were only able to obtain support for the original findings for half of the studies. These results do not definitively mean that the original findings were wrong, but they do suggest that they are not as robust as might have been assumed." - Michelangelo Vianello, one of the project leads.
Yes, yes they do. It's ok to say the original findings were wrong. It's ok to re-assert the null hypothesis. The unwillingness to do so undermines the process and is disconcerting to say the least.
Failure to replicate doesn't prove the original findings were wrong. Other things that can cause replication failure are:
- Poor explanation of experimental procedure, so the replication doesn't match the original experiment.
- Awareness of the effect may invalidate it. For example, it'd be hard to replicate the Stanford Prison Experiment because every psychology undergrad has heard the story.
- Change in social conditions or prevailing attitudes. Consumer behavior (the subject of some of these studies) has changed significantly over the decades. Homophobia (another study) varies dramatically over time and place.
Not exactly: it is ok to say that there were no original findings. This is a tiny semantic detail but important: if the author does not support his findings, it is OK to re-assert the null hypothesis but not to say that "the findings were wrong".
What I mean is: "no experiment" does not mean the same as "wrong experiment".
Well, let's agree that 'wrong' doesn't mean 'bad.' I use the term in the Feynman sense, if your guess doesn't agree with experiment, it's wrong. The assertion that they found a statistically significant effect was wrong (or to Trevor's point in a sibling comment, is no longer true.)
The thing with psychology experiments is that the fundamental thing being tested is not immutable.
This makes it different than many other science experiments where concrete parts of reality are tested.
Psychology really only tests the minds of humans in a global society at a point in time. If future minds change enough the results may no longer be valid.
Your statement in itself is a hypothesis to be proved. There are psychology experiments designed specifically to ascertain what parts of the mind are hard-wired and which are depended on nurture/culture.
I am not a Psychologist but I can give one example: Piaget hypothesized that the mind developed by first accumulating low-level sensory/motor data, and then progressively building higher levels of abstraction on that foundation. If that is true, then that process itself is not related to society or culture and should be observable so long as the biology underpinning cognitive development doesn't radically change.
Psychology is different than other "hard" sciences.
I only say "results _may_ no longer be valid" and I'd say this study proves that.
If you go deep enough into the mind to the point where social environment no longer influences the behavior you may be getting closer to biology than psychology but point taken.
Fun game: read through the summary of each experiment (psyarxiv.com/9654g) and try to guess whether it replicated or not before they tell you. I got 20/28.
Some of them were obvious (#3 - no, using a bad font does not make people answer questions more carefully; #22 - yes, people attribute more responsibility to adults than to babies). But there were some big surprises for me:
#6 - Participants read a hypothetical water conservation dilemma, then rate how much they trust the people in the scenario to conserve water. Sometimes the people are called "consumers" and sometimes "individuals". Does it make a difference? Apparently yes. When they're called "consumers", you expect them to conserve significantly less water than when they're "individuals".
#24 - if you ask "How satisfied are you with your relationship?" and then "How satisfied are you with your life-as-a-whole?", are the results more highly correlated than if you ask those questions in the opposite order? I thought for sure they would be - relationship happiness is one of several contributing factors to your overall happiness, so if it's fresh on your mind, you'll weigh it more heavily when you answer the life-as-a-whole question. Apparently not.
I'm about halfway through doing just that and I think a pattern has emerged: does there exists a plausible, concrete (you could put it into words) mechanism for cause and effect? That is, can a person tell you why the finding exists. For instance, a semantic association between capitalism and greed does not provide a plausible mechanism for why mentioning capitalism should make one greedy. The gay one failed that test, but otherwise that seems to be what's going on. A lot of these papers aren't testing a scientific theory. They're testing vague, half-baked ideas and associations.
I'd love for someone to go through and do this with education research. It's an absolute joke, and I'd argue it's worse than psychology research. And it doesn't help teachers are making students refuse to use old literature (or maybe I'm just pissed I can't quote older studies in my 'literature review' for my 'research paper'...)
Poor choice of headline for the original article given 'replicated' indicates success. This was an intensive attempt to replicate but 14 of the 28 did not replicate.
That is a 50 percent failure rate inspite of using over 186 researchers in 60 labs worldwide. The widely reported 2012 study could not replicate nearly 100 of 100 studies. And in this case if this kind of comprehensive effort can't replicate half the studies it raises serious questions about the original studies and adds more fuel to the reproducibility crisis.
Given studies are routinely used to justify and implement policies [1], and taken seriously as points of authority it's not just academic.
There needs to be much larger discussion on the use of studies to direct policy that impact real lives, and the level of oversight, additional replication and other credibility building measures that need to be considered.
35 comments
[ 2.5 ms ] story [ 84.4 ms ] threadYou don't have to feel so bad about not reading that Malcolm Gladwell book, now.
- The 28 findings was repeated in more than 60 laboratories all over the world resulting in a median sample size of 7,157
- Overall, 14 of the 28 findings (50%) replicated successfully
- The replication team obtained the original materials to ensure faithful replications of the original findings
- All 28 studies underwent formal peer review at the journal prior to conducting the studies
- All of the studies were preregistered at OSF
- Researchers can predict whether findings will replicate or not in surveys and prediction markets with pretty good accuracy. (prediction market)
Full paper (100 pages): https://psyarxiv.com/9654g/
1 Direction & SES (Huang et al., 2014) p<.001
2 Structure and goal-pursuit (Kay et al., 2014) p=0.97
3 Incidental disfluency (Alter et al., 2007) p=0.67
4 Moral Foundations (Graham et al., 2009) p<.001
5 Affect and Risk (Rottenstreich & Hsee, 2001) p=0.77
6 Priming consumerism (Bauer et al., 2012) p=0.15
7 Correspondence Bias (Miyamoto & Kitayama, 2002) p<.001
8 Disgust & Homophobia (Inbar et al., 2009) p=0.56
9 Incidental Anchors (Critcher & Gilovich, 2008) p=0.25
10 Social Value Orientation (Van Lange et al., 1997) p<.001
11 Trolley Dilemma 1 (Hauser et al., 2007) p<.001
12 SMS & Well-Being (Anderson et al., 2012) p=0.58
13 False Consensus 1 (Ross et al., 1977) p=0.23
14 False Consensus 2 (Ross et al., 1977) p<.001
15 Position & Power (Giessner & Schubert, 2007) p=0.31
16 Framing (Tversky & Kahneman, 1981) p=0.43
17 Trolley Dilemma 2 (Hauser et al., 2007) p=0.42
18 Tempting Fate (Risen & Gilovich, 2008) p=0.01
19 Actions are Choices (Savani et al. 2010) p<.001
20 Intuitive Reasoning (Norenzayan et al. 2002) p<.001
21 Less is Better (Hsee, 1998) p<.001
22 Moral Typecasting (Gray & Wegner, 2009) p<.001
23 Moral Cleansing (Zhong & Liljenquist, 2006) p=0.08
24 Assimilation and Contrast (Schwarz et al., 1991) p=0.39
25 Choosing or Rejecting (Shafir, 1993) p=0.10
26 Priming warmth (Zaval et al., 2014) p=0.01
27 Intentional Side Effects (Knobe, 2003) p<.001
28 Direction and Similarity (Tversky & Gati, 1978) p=0.99
https://en.wikipedia.org/wiki/Pit_of_despair
Scientifically, it might be of value to replicate the experiments, but I really can't imagine anybody attempting them.
Yes, yes they do. It's ok to say the original findings were wrong. It's ok to re-assert the null hypothesis. The unwillingness to do so undermines the process and is disconcerting to say the least.
- Poor explanation of experimental procedure, so the replication doesn't match the original experiment.
- Awareness of the effect may invalidate it. For example, it'd be hard to replicate the Stanford Prison Experiment because every psychology undergrad has heard the story.
- Change in social conditions or prevailing attitudes. Consumer behavior (the subject of some of these studies) has changed significantly over the decades. Homophobia (another study) varies dramatically over time and place.
(or just continue here on HN of course)
What I mean is: "no experiment" does not mean the same as "wrong experiment".
Psychology is a big field. I just find it hard to accept that these papers are representiative of much of anything.
This makes it different than many other science experiments where concrete parts of reality are tested.
Psychology really only tests the minds of humans in a global society at a point in time. If future minds change enough the results may no longer be valid.
I am not a Psychologist but I can give one example: Piaget hypothesized that the mind developed by first accumulating low-level sensory/motor data, and then progressively building higher levels of abstraction on that foundation. If that is true, then that process itself is not related to society or culture and should be observable so long as the biology underpinning cognitive development doesn't radically change.
Psychology is different than other "hard" sciences.
I only say "results _may_ no longer be valid" and I'd say this study proves that.
If you go deep enough into the mind to the point where social environment no longer influences the behavior you may be getting closer to biology than psychology but point taken.
Maybe, or maybe immutability is axiomatic in those other areas.
Some of them were obvious (#3 - no, using a bad font does not make people answer questions more carefully; #22 - yes, people attribute more responsibility to adults than to babies). But there were some big surprises for me:
#6 - Participants read a hypothetical water conservation dilemma, then rate how much they trust the people in the scenario to conserve water. Sometimes the people are called "consumers" and sometimes "individuals". Does it make a difference? Apparently yes. When they're called "consumers", you expect them to conserve significantly less water than when they're "individuals".
#24 - if you ask "How satisfied are you with your relationship?" and then "How satisfied are you with your life-as-a-whole?", are the results more highly correlated than if you ask those questions in the opposite order? I thought for sure they would be - relationship happiness is one of several contributing factors to your overall happiness, so if it's fresh on your mind, you'll weigh it more heavily when you answer the life-as-a-whole question. Apparently not.
Replication Failures
- Disgust sensitivity predicts homophobia (no it doesn’t)
- Assimilation and contrast - the order of asking specific vs. general questions about the same domain matters (no it doesn’t)
- Priming Consumerism - being told you’re acting as a consumer reduces your trust in others (no it doesn’t)
- Moral Cleansing - people exposed to unethical things have an urge to cleanse themselves physically (no they don't)
- Priming Warmth - on warm days people have higher belief in global warming (no they don’t)
- Structure & Goal Pursuit - being exposed to structure improves goal pursuit (no it doesn't)
- Incidental Disfluency - making tasks incidentally difficult (hard to read fonts) - improves analytic performance (no it doesn’t)
That is a 50 percent failure rate inspite of using over 186 researchers in 60 labs worldwide. The widely reported 2012 study could not replicate nearly 100 of 100 studies. And in this case if this kind of comprehensive effort can't replicate half the studies it raises serious questions about the original studies and adds more fuel to the reproducibility crisis.
Given studies are routinely used to justify and implement policies [1], and taken seriously as points of authority it's not just academic.
There needs to be much larger discussion on the use of studies to direct policy that impact real lives, and the level of oversight, additional replication and other credibility building measures that need to be considered.
[1] https://www.sciencedirect.com/science/article/pii/S001632871...