The most important sentence of the article, which appears at the end:
"Although 11 new papers have recently been published that do not show the downstream effects of power posing on behaviors, Cuddy is still fighting for power posing."
This is an astonishing number of replications to be published on one topic in psychology. 2-3x the next highest number I heave heard of.
The placement of that sentence, plus the flattering photos, leads me to suspect a publicist was able to place a thumb on the scale.
would note that near the end of the article its noted that Cuddy has conceded that the specific cortisol effects touted have not been borne out.
she is "still fighting" for power posing but only as a useful technique which sometimes works for some people in some cases, as even the replicated studies seem to show improvements in subjective measures of "confidence", whatever that means.
>This moment of fizziness for the discipline was already underway when Cuddy arrived at Princeton’s graduate program in 2000, transferring there to follow her adviser, Susan Fiske, with whom she first worked at the University of Massachusetts at Amherst.
You mischaracterize. Fiske is not criticizing "people who pointed out statistical errors" and begins the entire essay by explicitly stating as such: "Our field has always encourages -- required, really -- peer critiques"!
She is criticizing "unfiltered trash-talk [on social media]... [and] online vigilantes [who] are attacking individuals, their research programs, and their careers".
It's the same tension the article is referring to, between civility and public discourse. On one hand, you'd really like an open dialog about the quality of evidence presented. On the other hand, there's a real price to be paid when people are too worried about public humiliation to risk doing science on certain questions or working with certain people.
As the quote from Jay Van Bavel put it, "It's become like politics — we’ve created two camps of people who shouldn’t be in two camps in the first place"
This point resonated with me even though I'm on your side of those politics. What effect would it have on the OSS community if any even slightly bad code you write could end up in a hit piece published by The Atlantic, and if fact you witnessed people on Facebook saying your colleague, who maybe did write some kinda bad code that one time but is on the whole competent, is a charlatan who doesn't deserve the title of "programmer"?
Is that not the case anyway? I admit I have no experience contributing but that's from hearing things like how the linux development team runs and people like Sarah Sharp quitting over the abuse. It's given me the impression that if I don't create something that is up to a teams standard from commit 0 then I will be ridiculed in a public forum
Abuse from your own insular community re: a particular project is one thing; public abuse published in some of the most popular print media on the planet resulting in lost jobs/lost job opporunities is quite another.
People use their open source contributions as part of their resume when seeking jobs, correct? That sort of abuse would reach outside the group if you were showing off your contributions as a necessary part of your career.
I don't see as wide of a gap in effect on career between open source contributions for engineers and published papers for scientists as you seem to be implying
I certainly have a fair degree of sympathy for Cuddy's losses, but should she have been given the accolades (TED talk, tenure track at Harvard, etc.) in the first place?
Her work was interesting no doubt, and in our scientific age we have always celebrated the novelty and originality of the scientific geniuses that have advanced our understanding.
But the real value of science has always been novelty plus certainty. The scientists who we esteem should be known as much for their industrious management of the empirical process (recruiting replicators, statistical auditors, etc.) as they are for they are for their creative insight.
As I indicated up-thread, I'm mostly on your side of these politics.
geofft's comment hits the nail on the head when it comes to what effect this did vs. should've had on her tenure case, I think. A study that fails to replicate is no indication of malice or even incompetence. Some say she should've run self-replications, but how many years self-replication studies are you willing to run on $20k/yr (in fact, how many post-college years have you made 20k/yr for a full-time job)?
At some point you get the quality of science you can pay for, and throwing the grad student under the bus by denying them tenure a decade later is as shitty and spineless as Equifax throwing an IT guy under the bus for what was pretty clearly an organizational failure. IMO.
Re: TED talks, TED is not a scientific venue, and most of the things I hear in TED talks are a hell of a lot less scientific than even Cuddy's most flawed studies. Also, I don't think those sorts of things should weigh much in hiring and tenure cases either way (aside from e.g. a usually marginal "service/outreach" category).
If Harvard hired Cuddy because she gave a popular TED talk, then that was an extremely poor hiring decision by Harvard. But that's a separate issue from the one we're discussing.
> As I indicated up-thread, I'm mostly on your side of these politics.
I know, and I think we are mostly in agreement. I generally don't like seeing individuals dragged through the mud. It's also somewhat unfair to judge her actions pre-replication crisis by our standards post-replication crisis.
I also agree that a single study failing to replicate is not an instance of malice or incompetence, in fact I think all scientists should expect it to happen. But that also means that no career should be built on the back of a single study. It seems that she is still sticking to her guns even after 11 failed replications. That does cause me some concern.
> but how many years self-replication studies are you willing to run on $20k/yr? At some point you get the quality of science you can pay for...
Then maybe one should more hesitant about promoting those results?
> If Harvard hired Cuddy because she gave a popular TED talk, then that was an extremely poor hiring decision by Harvard. But that's a separate issue from the one we're discussing.
I think it makes me more sympathetic to her case. If Harvard had not hired her, or TED not promoted her, she would not have been subject to nearly the same scrutiny or ire as she was. Ironically, she might have had a better shot at a good long-term career. But who, at that point, would have turned down either of these prizes? I know I wouldn't have.
> Then maybe one should more hesitant about promoting those results?
We definitely agree on this. I think all the TED science stuff is silliness. Cuddy's mistake was both being a serious scientist and being a popular TED speaker. Popular TED speakers make BS science claims all the time, but they aren't scientists so it doesn't matter.
But I wouldn't advocate for firing someone because they disagree and think TED is a good place to popularize science (even if they happen to popularize science that later turns out to be bad science).
> I think it makes me more sympathetic to her case. If Harvard had not hired her, or TED not promoted her, she would not have been subject to nearly the same scrutiny or ire as she was. Ironically, she might have had a better shot at a good long-term career. But who, at that point, would have turned down either of these prizes? I know I wouldn't have.
There is a concept in management called the Peter principle, which is that "managers rise to the level of their incompetence". I've always worried about it in myself - I work hard to advance my career, but is it possible I'm sabotaging my own long term success? What if I get the high-profile new responsibilities I'm striving towards... and then fail spectacularly?
Before deciding that Linux (the most successful open source projecting history?) is poorly run. you have to weigh people quitting over the abuse against all the other contributors and maintainers building something that relies in part on that abuse.
> It's given me the impression that if I don't create something that is up to a teams standard from commit 0
Linus has said that he reserves his abuse for senior people who should know better. It's his antidote to a complacent
overclass that mistreats the kernel development process.
I never stated that it was poorly run, acting like this might get better results. I was replying to the poster who postulated what could happen if a hit piece was ran in a public setting against someone for their OSS contributions and how it seemed like reality wasn't already too far off from that to me
> You mischaracterize. Fiske is not criticizing "people who pointed out statistical errors" and begins the entire essay by explicitly stating as such: "Our field has always encourages -- required, really -- peer critiques"!
> She is criticizing "unfiltered trash-talk [on social media]... [and] online vigilantes [who] are attacking individuals, their research programs, and their careers".
Let me quote Gelman regarding the subject of her criticism:
> The other thing that’s sad here is how Fiske seems to have felt the need to compromise her own principles here. She deplores “unfiltered trash talk,” “unmoderated attacks” and “adversarial viciousness” and insists on the importance of “editorial oversight and peer review.” According to Fiske, criticisms should be “most often in private with a chance to improve (peer review), or at least in moderated exchanges (curated comments and rebuttals).” And she writes of “scientific standards, ethical norms, and mutual respect.”
> But Fiske expresses these views in an unvetted attack in an unmoderated forum with no peer review or opportunity for comments or rebuttals, meanwhile referring to her unnamed adversaries as “methological terrorists.” Sounds like unfiltered trash talk to me. But, then again, I haven’t seen Fiske on the basketball court so I really have no idea what she sounds like when she’s really trash talkin’.
> I bring this up not in the spirit of gotcha, but rather to emphasize what a difficult position Fiske is in. She’s seeing her professional world collapsing—not at a personal level, I assume she’ll keep her title as the Eugene Higgins Professor of Psychology and Professor of Public Affairs at Princeton University for as long as she wants—but her work and the work of her friends and colleagues is being questioned in a way that no one could’ve imagined ten years ago. It’s scary, and it’s gotta be a lot easier for her to blame some unnamed “terrorists” than to confront the gaps in her own understanding of research methods.
> To put it another way, Fiske and her friends and students followed a certain path which has given them fame, fortune, and acclaim. Question the path, and you question the legitimacy of all that came from it. And that can’t be pleasant.
Gelman has forged his own path to "fame, fortune, and acclaim". It's not clear to me that this path has actually improved (this particular branch of) science. In fact, by the sound of it, that path may've been at least as destructive as he claims Fiske et al's was. Which isn't an indict of Gelman, as I hardly intuit anything but positive intentions on his part.
I'm less concerned with Fiske vs. Gelman and much more concerned with Jay Van Bavel's observation that either way, science loses.
Confining scientific critique to scientific venues seems like a reasonable bulwark against this sort of thing. If it had been Reviewer #3 in a program committee or editorial board, or a well-received letter to the editor, instead of a blog post picked up by thousands of internet warriors with no stats training, perhaps the result could've been refined or thrown out without expelling someone from research. If the feud hadn't been picked up by pop science press, then maybe a bad piece of science could've been questioned without poisoning the well for further inquiry.
"Confining scientific critique to scientific venues"
Fiske controls the venue and is (mis-)using the "scientific venue" to denounce her critics. She is the one refusing to publish the very critiques you describe. The "scientific venue" has been corrupted. The well has not been "poisoned" for further inquiry, rather the necessity of replication and sensitivity to statistical errors has increased. Science was being done sloppily, and Fiske was both a beneficiary of that and in a role where she could have prevented it.
That's a pretty strong accusation. Do you have any evidence?
Cuddy herself, in the linked article, even asks: "Why not come to a conference or hold a seminar?"
I don't think you're wrong about the lack of quality reviewing, but I also think that blogs and Facebook and articles in pop sci news venues are pretty shitty places to address the quality of editorial review.
And your post demonstrates why: honest mistakes -- or even complete non-mistakes stemming from gaps in established best practices! -- turn into out-right accusations of full on corruption. The comments on NYT demonstrate the same problem from the other side: lots of gender politics when Gelman wasn't motivated by gender, implicitly or otherwise, but rather by the mass appeal of Cuddy's work.
Needless to say, I also think you're massively over-stating the maliciousness of people like Cuddy and Fiske.
> Science was being done sloppily, and Fiske was both a beneficiary of that and in a role where she could have prevented it.
Simmons seems to agree with me that you're really over-reaching here: "The original study wasn’t particularly egregious," he said. "It was published in 2010 before anyone was thinking about this."
There's a huge difference between doing science the way you were trained (but sub-optimally), and outright "corruption".
There's also a HUGE difference between noticing a problem and constructively pushing for more rigor within a sub-community, and going after particular people within that community so harshly you destroy their careers.
The former is great for science. The latter might possibly by great for improving rigor (I'm pessimistic and think it all comes down to incentives, which Gelman et al don't really do much to change). But equivocating bad methodological/statistical training/practice with unethical and even criminal ("corrupt") behavior? That sort of dishonest fire-and-brimstone talk is just as damaging as a lack of rigor.
1. failing to have read Meehl's critique doesn't amount to "corruption",
2. that calling people "corrupt" for having failed to read Meehl is not likely to be the most effective or the most civil path toward changing things, and
3. To the extent that discussion of scientific methodology on public forums causes or contributes to (2), that's a bad thing and perhaps suggests a fundamental cost of doing science in this manner.
I think you're using a different sense of the word "corrupt" from the one in the post you're replying to ("The "scientific venue" has been corrupted."). I know it's easy to lose nuance in internet discussions, but please read the posts above more carefully/charitably. Nothing Cuddy did was corrupt in any sense, only poor science.
I re-read the comment, and disagree. The preceding sentence is: Fiske controls the venue and is (mis-)using the "scientific venue" to denounce her critics
I'm having trouble finding a charitable way to read the claim that someone is misusing a scientific venue.
And much more to the point, I doubt even more that the subject of those sentences (or the e.g. tenure/promotion committee of that subject) could be reasonably expected to apply such an abundance of charity.
Think "corrupted file", not "corrupt official". The key difference is that there is a primary purpose for their existence (a file stores data, a scientific process produces good and reliable findings), and something can be corrupted (no longer suitable for its purpose) without malice involved. To call an official corrupted, you need to allege malice, to say that a file is corrupted, you only need to show that it's not valid for its purpose. It's not a good choice of a word anyway, you could make exactly the same valid point without any of the ambiguity.
How's this- Fiske publishes a letter critical of Gelman, with no peer review of said letter, in her journal. She then claims her letter is "science" because it is in a scientific journal, while his self-published piece is "methodological terrorism."
In a sense, both pieces are self-published, and she is using the mantle of Science to falsely claim that her self-publishing is superior because of the authority of her position as editor. This is corruption. This is misusing the scientific venue.
Please link to the place where she actually makes this claim.
> in her journal
Not hers, and not a journal.
> while his self-published piece is "methodological terrorism."
She actually outlines the practices that ought to be singled out as unprofessional and unacceptable in the second to last paragraph, and none of those are mere self-publication.
Have you considered the possibility that both sides are wrong? That this is crap science, and that the conduct of certain groups of people rallying against crap science is overzealous to the point of cruelty?
Finally, it's also worth noting you've COMPLETELY proven Fiske's point about social media: sure, there are some valid critiques. But there are also people being absolutely vicious to people actually trying to science because of very trivial and obvious misunderstandings (e.g., because they think that APS Observer is a journal). I think to some extent that's a reasonable price for free discourse when things stay online and especially when they're anonymous, but you're calling a real person with a real job on the line corrupt because you made an incorrect assumption (that would've taken you 10 seconds of googling to correct).
"Cuddy herself, in the linked article, even asks: "Why not come to a conference or hold a seminar?""
I am not criticizing Cuddy. Fiske is editor of PNAS, former president of APS, and is putting her criticisms in APS Observer. The difference between a journal and an official magazine of a scientific organization is irrelevant in this concept, in that the supposed superiority of an edited journal is what is in question here when the editor is not good, and is not considering the context of the discussion.
I am pointing out the utter hypocrisy in her argument that you shouldn't point out statistical errors on blogs, while she is pointing this out in a publication she controlled. She wants people to be able to protect their undeserved status, it is clear.
The entire idea that science needs to be locked up in limited access journals (or magazines) is not a good one. The real issue is that the quality of PNAS has been pretty low. Fiske's response to that is to try to discourage open criticism. Open data is one of the reasons that people were able to find these errors. Resistance to publishing failed replications is not proper behavior. We have now seen PNAS publish a few letters describing errors, and hopefully the standards will improve. However, it hardly seems like a person whose response is to lean toward hiding the discourse is the right person to do it?
Here's an interview with the brave researcher who asked her about this at a conference. Her quote "There is no mechanism in place that holds researchers accountable for practices that lead to a corrupted literature but cannot clearly be classified as fraud" is probably a less aggressive version of what I am saying.
"That's a pretty strong accusation. Do you have any evidence?"
Fiske is the journal editor. Fiske is allowing articles with weak stats to get published. Fiske is publishing her defense in that journal. Fiske is not publishing critical articles in that journal. That's corruption based on her position of authority.
Cuddy just had sloppy stats and her results failed to replicate. Notice how you deceptively switch to Cuddy from Fiske in your response, because you have no point about Fiske.
Furthermore, Fiske was not the president of APS or an editor of the Magazine when that article was published. And even if she were, what's the problem? I see letters from the editor in most magazines I read on a very regular basis.
Finally, the claim that Fiske was construing her opinion on conduct as a scientific fact is just... well.
I'll refrain from calling you corrupt or deceptive because it was probably just a misunderstanding :-)
> Notice how you deceptively switch to Cuddy from Fiske
I actually have no idea what you're trying to say or what this refers to. Can you be more explicit?
> Confining scientific critique to scientific venues seems like a reasonable bulwark against this sort of thing.
I don't think turning a call for greater scientific rigor into a political fight is desirable, but neither do I think that confining this struggle to scientific venues is desirable (or practical, for that matter).
"Confining scientific critique to scientific venues"
When you take your BS "studies" and sell it to the audience at TED and then the wider internet, go on corporate speaking tours, sign book deals to promote your "science", and try to affect public policy, you are open to criticism from anyone who sees through your shoddy methodology and wants to correct the record.
It occurs to me that the replication crisis largely came into being because scientists are rewarded for publishing papers that demonstrate some effect and not for publishing papers that fail to demonstrate anything, and so the right fix would have been to fix the broken incentives. But apparently (if this article is to be believed) no such change in incentives has happened - the only thing is that there are now also incentives for demonstrating that some other study wasn't replicable. Steadily doing good science, where good science is expected not to produce results much of the time, continues to have no support. Is it any wonder that the state of science seems a little better (at least we can admit the replication crisis now) but people like Cuddy feel like they have no future in the field, that having the misfortune of the one-in-twenty chance of getting a really interesting false positive at p = 0.05 means you either have to double down on the study being right after all or you have to quit the field? Are schools actually looking at past p-hacking while continuing to demand a huge publication record for hiring? This doesn't seem like we've solved any fundamental problems.
(Also, as a side note, I'd fully expect that CS is at least as vulnerable to the same thing. McSherry et al.'s COST work is a good example of something that seems far worse than p-hacking: instead of having a clearly defined hypothesis and publishing the 1-in-20 papers that falsely state it's true, as in other fields, there's nothing clearly defined in scalable systems and 20 out of 20 people who build something excessively complicated can get a paper out of it.)
Literally the papers are "I built a thing, and it gets faster when you do this thing to it." There's no condition on whether the base case is realistic.
To be clear, I don't think the authors of the papers he's taking down are bad people, or maliciously trying to get publications for uninteresting work, or too stupid to implement things efficiently on a single core, or anything. I think they're genuinely interested in the work they're doing, but responding to the pressures of the environment that allows them to keep doing that work.
> having the misfortune of the one-in-twenty chance of getting a really interesting false positive at p = 0.05 means you either have to double down on the study
Or you could get a large sample size to bring the p down to 0.01 or whatever. You know, do science.
Well, the smaller your p, the higher your chance of false negatives, too; maybe there was an actual effect there that was worth more study, and you dismissed it.
The problem is that a study that's significant at p = 0.05 but not p = 0.01 is worth something to science. It indicates something worth doing several more studies on, preferably by different teams from different institutes with non-overlapping confounding factors. But the practice of science gives you only two options once you find such a study, as a young scientist: promote it immediately and become visible for having discovered something, or aggressively ignore it and go find something else that you think is likely to be immediately promising. "Doing science," as you put it, is the worst option for you because it ensures you spend a lot more time on something for no reward -- and the rational self-interest here is perfectly rational, because if you don't get funding or tenure, you're not going to do any science any more at all. (As seems to have happened with Dr. Cuddy.)
To be clear, I have no well-formed opinion on whether Dr. Cuddy is a good scientist, having only read this article. I assume she is, or at least that she's honest in her desires. But I think talented, honest people are nonetheless pressured by incentives, and fixing incentives is better for society (assuming we as a larger society wish our scientists to continue to do science) than assuming that people will be self-sacrificial and altruistic in pursuit of a tiny chance at accomplishing the greater good.
Here's the structural problem I see: there are acres, literally, acres of computer science types in SF and certain other enclaves, slaving away on ads when they would rather be curing cancer. But they don't know the first thing about cancer.
Meanwhile, there are enclaves where biologists are slaving away capturing data and not even realizing it's data. "Bookkeeping data" they might call it.
There should be mixers between the biologists and the computer scientists. Especially non-academic physicians and computer scientists. Look at Kaiser-Permanente, Sutter Health, or Defense Health Agency. Huge orgs, very little research. Data falling off the shelves and they don't know what to do with it.
Mozilla Science runs the global science sprint each year to bring scientists of all kinds together with developers. I bet there's lots of projects that could still use help. https://github.com/mozilla/global-sprint/issues
You’re not wrong. Especially not about the ads. What we need, unfortunately, are (at least) 2 very difficult things to achieve legislatively:
1. A legal framework for health-related products & services that meaningfully protects individual privacy, and yet also preserves the conditions for the kind of velocity and rate of change that brings technological progress to the masses reliably and quickly.
2. Massive academic funding for more “pure” research in hardware & software design, engineering, the philosophy of what’s possible, the phenomenon and ethics of what we choose to build, and so on. TLDR Pay more people more money to do important research in the public domain, which I can’t help but think would attract many of the CS types you mentioned.
> a legal framework that protects privacy yet brings technological progress to the masses quickly
That exists, but there's upfront educational costs that need to be handled first.
> Massive academic funding
Look at Europe: their acedemics work much more closely with industry. Get a someone who can code in a room with a physician, and I garuntee they can find a problem to work on. Show a successful prototype, and we're way below "minimum viable product" here. Show a graph with real data and money will start looking for you.
Computer scientists, mathematicians, physicists, and engineers need to go to biology conferences. And biologists should be going to math, ML, and software conferences.
>"The problem is that a study that's significant at p = 0.05 but not p = 0.01 is worth something to science. It indicates something worth doing several more studies on"
I still don't understand this one. The null model is (almost) always false. In this situation a small p-value is only a matter of sample size and measurement precision. How can it possibly indicate you should work more on something?
One thing I think may make sense is to get a bunch of p-values and then choose the lowest 10 or whatever to focus on for further work. There is no reason to include an arbitrary cut-off here though.
>"Well, the smaller your p, the higher your chance of false negatives, too; maybe there was an actual effect there that was worth more study, and you dismissed it."
Also noticing this now... No, this is completely untrue. The p-value is not an error rate.
It's a guideline, not a rule. The more you treat it like a rule, the worse the situation is.
There will never be an objective measure for conclusion 'correctness'.
It will always be the case that you actually have to think about the data and statistics critically and decide whether the conclusions are appropriate. In certain well-defined situations, you can start to use harder metrics, but that's rarely where the controversy is in the first place.
So if you want to, you know, do science, I urge you to use that big beautiful brain of yours. As a bonus, you don't even need to commit mass rat genocide to do it.
> that having the misfortune of the one-in-twenty chance of getting a really interesting false positive at p = 0.05 means you either have to double down on the study being right after all or you have to quit the field?
There were (are?) enough degrees of freedom in the field in every study to always meet this threshold.
Steadily doing good science, where good science is expected not to produce results much of the time, continues to have no support.
We say this, but I haven't heard any particularly good ideas on how to build such a system that isn't also easy to game. How do you ensure the failed studies were actually worth doing? Is peer review of study design feasible?
Here's a system that's gameable, but no more gameable than other things our society seems to be fine with: have peers evaluate you for the quality of your work as opposed to your outputs. Retain people whose peers say "They do good work," even if they don't have a flashy success or novel discovery to show.
This is roughly how private industry works. I'm of course asked in job interviews about my successes, but the primary thing that determines whether I get a job, how much I get paid, whether I get other people to work with, etc. isn't what my past successes are, it's my ability to convince my peers that I am a steady and good worker. If I maintained some tech stack for two years, my employer is happy to let me continue doing that, and other employers are happy to hire me to maintain similar tech stacks, without me having been like the one person in the world to figure something out. (On rare occasion, I am, in the sense that I'll fix something upstream that's been challenging, and employers do value that! But there's no, e.g., "fix upstream bugs or perish" culture.)
As in private industry, it's gameable because it allows people doing poor-quality work to vouch for other people doing poor-quality work. But it's hard to game this long-term, because people interested in doing good-quality work will tend to find others who do, and in aggregate -- from a large number of such people working together over a period of time, not from any one individual -- these people will produce more measurable results.
Give funding and recognition/status to large organizations (groups of at least 100 researchers) that produce good work, without naming names except perhaps in truly exceptional cases. Refuse meddling from outside incentives that break the organizational abstraction barrier (prizes, fellowships, grant proposals that with individuals named on them, fellowships or grant proposals that stay with individuals across organizations, etc.). Create alternative structures that recognize organizations that are producing results with fame and fortune.
Note that by "organizations" I don't mean "universities" or "institutions". Departments should probably be treated separately, as different departments within an institution can easily drift in terms of quality. Meanwhile, it should be possible recognize collaborations of researchers across researchers as an entity: if the collaboration wants to support a researcher moving between institutions and wants to keep supporting them, that doesn't lead to the same flaws, I think.
> We say this, but I haven't heard any particularly good ideas on how to build such a system that isn't also easy to game. How do you ensure the failed studies were actually worth doing? Is peer review of study design feasible?
Oh, so, 'Capital S' Science[0] has moved on from publish-or-perish to funding-or-famine. Most researchers are in 'famine' and their spouse is financing their lifestyle of crazy hours and stress (love is strange, for sure). Some are in 'funding' and they generally get all the cash from grant agencies. Where I am, each square foot of your research is ~$420/month for rent, including your office. Then you start buying lab equipment and do research. The incentives are heavily skewed towards making cash, no matter the method.
Also, if you haven't been following the 'X-ray Carrots' fiasco, that is a BIG deal in science and Science. VERY short summery: Andrew Wansink at Cornell made up a bunch of data and results, he published them, he got caught. One of these papers was about calling carrots as 'x-ray carrots' makes kids eat more of them, hence the name of the fiasco. Here's a good way into the mess [1]. Anyway, some papers were retracted, there was an investigation, yadda yadda. Now, in 'normal' science, he would have been banished from the field for this nonsense. But in 'finding-or-famine', he got off essentially scot-free. Nothing happened, really. He's still the head of the department (AFAIK), he still brings in cash, the FDA still is using his findings. This is a huge blow to science and a huge boon to Science.
The existential crisis is now out in the open: you can cheat with impunity, publish whatever, if you get caught nothing will happen, just bring in the dollars.
Now is not a good time to go to grad school. I think they're just going to have to throw out ALL US based bio-science (at least) from about 2003 to whenever heads start rolling. It's all crap, maybe, who knows.
[0] Science and science are different, though look similar. Big Science is the grant machine and the big money side of science. Splashy, barely replicated, large clean buildings, pensions, perfectly fit for corporate needs. Think Big Tobacco, BPA plastics stuff, power poses like Cuddy's, that kinda stuff. It's like the difference between the Military Industrial Complex and actually kicking down doors in Mosul. 'Little s' science is what you are probably thinking of when you think of science generally.
I think this is just as misinformed as someone who thinks there's no problem.
In my experience, about a third or half of the 'real' scientists I've met are firmly in "Big S" science. It's really not that different. People that complain about this a lot are often jealous, in my experience. The failures are certainly high-profile, though.
As for reproducibility, it's all case by case. We've (real? scientists) been having the same discussions about data quality forever. It just seems to have found a fresh audience. There's a reason a curmudgeonly PI might insist in Western Blots and - gasp! - Northerns from time to time. Most fields were built off of some real science. Molecular biology certainly was, and to think that just stopped happening in the early 2000s is the height of ignorance.
There is cause for alarm. I give pretty much all 'genomic' work a very wide berth unless the lab has a real genetics background, and aside for a few labs, anything with 'microbiome' in it is certifiable garbage. And clearly some entire fields are on shaky ground, namely psychology. But lets not circlejerk our pure 'little s' science ego's too vigorously.
> ... the same discussions about data quality forever.
You misunderstand, perhaps. I mean that the data itself is fraudulent, let alone the interpretations or the controls on the experiment. Many PIs have been making up the data, essentially. Pulled from thin air with a little rand() thrown on top. Now the flood gates to this behavior are open and 'accepted', at least by the people that cut the checks.
I'm sure there's already been a discussion on how much of the effect of power poses is due to confirmation bias or placebo effect, as opposed to some other physiological or psychosomatic reason. Especially since in the original study the subjects weren't actually told what they were doing, but in Dr Cuddy's talks and books they explicitly are.
It's just curious to me that she and Dr Fiske wouldn't catch this crucial difference immediately. Effectively it turns power poses into little more than 'the power of positive thinking'.
That said, personal attacks and especially those focusing on her brain function really need to stop. Psychologists treating each other like shit is not just ironic, it's a disgrace. If they can't even behave properly with each other, how can we expect them to treat patients properly?
One note; in general, I think we tend to diminish finding when we find out that the placebo effect is partly behind them. Wrongly, in my opinion. Better reactions might be, "Wow, we take this action and our bodies/brains take care of the rest of the effect through some placebo magic? That's amazing!!"
But doesn't pointing out a placebo effect also have power? I know the placebo effect still exists even if you're aware of it being placebo, but does it lessen? And what if it's accompanied by shaming?
Meaning, say the power pose effect doesn't exist. But early on, it's exciting and new, and there's a placebo effect (which is fine!), but people also think it's real.
But then studies get replicated, and there's well-known doubt on whether the effect is real in a non-placebo manner. There might still be a placebo effect, but does it shrink?
And then there's a lot of controversy and the study is dragged through the mud, and power poses are mocked. Might the placebo effect be lessened even further (or obliterated) from this?
Could this be true even if it was originally valid and not just placebo?
I think you're moving into territory in human psychology that nobody really understands.
For the time being, the answer is probably "if it works for you, then keep doing it." If power posing before an important meeting is how you psyche yourself up to be unafraid and effective in the meeting, then why not keep doing it, even if it doesn't seem to work for everyone?
Knowing the replication problems, I won't suggest power posing as a general thing. Maybe I will tell someone to try a bunch of things including power posing, and tell them to keep the things that they think are working.
I thought that Gelman wasn't presented in the best of light, especially his awkward quotes about not wanting to engage in "interpersonal conflict". But he called the piece "excellent and fair to all sides" and was pretty gracious in response to Cuddy's criticisms and how the NYT portrayed him. I honestly wasn't a huge fan of the article but re-reading it after a bit of sleep, and knowing that Gelman thought it was strong, I appreciated the issues it brought up. Certainly has to be one of the longest stories about p-hacking/p-curves in mainstream media.
I sympathize with Cuddy and have no doubt that among the legit attention that Gelman and other critics have directed toward her, she's faced hatemail sparked by dumb sexism, which makes it easier to tar the legitimate criticism as being sexist.
I do think Cuddy needs to face legitimate criticism. She may not have been a malicious fraudster, but ignoring criticism after a long time is not much different than actively being deceptive. I also can believe that she faces hostile criticism that isn't equally distributed among all the other researchers who used weak, but previously acceptable standards. And I don't think sexism can be dismissed outright, if people find it safer to talk shit to a woman compared to a man.
What I wish the story had more of was how scientific criticism is done outside of Gelman's blog. Gelman is the main scientist/research I follow on the topic. Apparently some see his snarky critique as toxic, and Gelman says he is intending to reevaluate the way he writes about things so that his point doesn't get lost in cheap jokes. How do other blogs do it? I'm somewhat sympathetic to Gelman because I perceive that there is significant institutional resistance (and few incentives) to re-examine and correct a peer review study, and being harsh/edgy/rude and beating the drum over time might be the only way to get scientists to care? But that doesn't mean his critique can't be refined for the benefit of his mission.
Stakes matter. A million dollar poker game might be structurally identical to a game played for fun, but its effects on the people involved are very different. We wouldn't expect the players in these two games to think the same way.
Talking about the stakes is in direct conflict with the scientific ideal of focusing on the science without talking about its effects on the scientists involved.
For example, from Gelman's response: "I just wanted to emphasize that there’s no reason her career, or even her famous Ted talk, has to rely on a particular intriguing idea (on there being a large and predictable effect of a certain pose) that happened not to work out." Well, except that maybe she wouldn't have gotten a Ted talk without it?
He's saying, hey, this paper shouldn't have been as important as it was. And according to the scientific ideal, we shouldn't get too attached to any particular results. But in practice, putting bad results aside sometimes requires a significant change of identity. Forgetting about that, even for a moment, is always going to come across as tin-eared.
The scientific ideal is more compatible with low stakes than high stakes. But talking about the game as if it's low-stakes (or ignoring the stakes altogether) when actually it's high-stakes is always going to make discussions focused on the science alone come across as unsympathetic.
I think this is an institution level failure to adapt to what science is like in the modern age.
New results are significantly more difficult and expensive to achieve in modern science, as the easier stuff has already been conquered in centuries and decades past.
The harder problems that science now works on means that distinguishing between actual results and incorrect conclusions is also much more difficult.
The growth of science as a percentage of the population and economy has also finally slowed down a few decades ago, after centuries of rocket-like growth. It is now a maturing enterprise, instead of a high-growth startup.
This means increased competition for resources and mindshare from those wanting to do science. Not surprisingly, all proxy measures and heuristics for evaluating scientific success are thus more likely to be abused.
I don't think we have fully adapted to these fundamental changes.
Disclaimer: I'm not a historian or researcher, but I've heard aspects of the above from a variety of sources, and it makes a lot of sense to me.
Personally if I was going to stake my professional reputation on a set of results, I'd expand from an exploratory study with a 20% chance of being wrong to a more rigorous study with, say, a 0.1% chance of being wrong.
Of course, that's much easier for people in STEM than it is for social scientists.
> Well, except that maybe she wouldn't have gotten a Ted talk without it?
A recent BBC event aimed at solving problems faced by women in tech flew Amy Cuddy in to give design and professional advice to the participants. The BBC story repeatedly celebrated Cuddy for giving the "second most watched TED talk ever". It also talked up the "Wonder Woman power pose" as a way to change your life.
Notably not present at the event were Cuddy's co-authors Carney and Yap, who have been vastly more receptive to the replication question, and less quick to promote uncertain results in non-academic channels.
Scientific falsehoods are almost always more impressive than the truth - they aren't constrained to realistic effects. At a certain point it starts to look like we're actively rewarding people for doing bad research and then rewarding them further for refusing to correct course.
A senior poker player showing another in need of help how to play the game so that they have the best chance of coming out ahead is precisely the most sympathetic thing he could do.
Andrew Gelman is a rare breed -- he's brilliant, hard working, engages with the public, writes both technical and (semi-)pop articles, and isn't an asshole. And we works in a field -- mathematics -- where there is very little confusion as to whether a result is right or wrong.
This makes lesser people working in dirtier subjects very uncomfortable: when he hits them they have nothing of substance to hit back with except generic insults.
As someone outside the scientific-academic community, I find his posts very accessible for learning news/issues about studies. I greatly appreciate his passion in trying to stop junk science from reaching the media. Maybe his rhetoric is too brash and coarse for the respectful standards of scientific discourse, but it's as harsh as it needs to be for scolding media outlets for running with junk science:
I just read his blog for the first time and thought he’s a huge asshole. It seems to me the primary effect of science-shaming on the blog is to rally the pitchforks, something he can’t possibly be ignorant of. But he apparently keeps on doing it. No doubt it’s good for the hits, but doesn’t contribute to a dynamic that leads to good science. Oh wells.
I don't know. A lot of the people Gelman complains about are, in my opinion, building their career on fraudulent work. They are undermining science. If there was ever any doubt that they were deliberately deceiving people, that doubt disappeared upon seeing their response to simple and legitimate criticism.
A little needling on Gelman's blog is not too harsh, all things considered.
Could you be more specific about which details give you this impression, and what you mean by it? I haven't interacted with Gelman personally, but having read both the NYT article and his response to it (as well as reading his blog and regularly his papers occasionally) I see him as someone who cares about "the truth" more than social niceties, but not as someone who gets any joy out of causing harm to others (which would be my definition of a "huge asshole").
Sexism is more than "safer to talk shit to a woman compared to a man" as you say in your article.
Sexism comes out in target choice, and the strength of which ones chases their prey, and the vehemence of attacks.
I do strongly believe that Cuddy was overly punished for being a woman, and being prominent in the field. Everyone loves when the "high and might" come "crashing down" -> the underdog stories and the desire for retribution are sought after more powerfully than (a) justified and (b) end worthiness of the goal.
I don't think anyone can honestly say that this has improved the field and made science better. It just means now that there is money and recognition to be made being attack dogs. That's hardly better.
As for Gelman, he wasn't portrayed in a great light, but I think it is fair. He's willing to encourage incivility and outright sexism and hostility as long as he doesnt have to talk to someone he disagrees with. That's the baseline definition of a coward there. And yes, I would discuss that with him in person and call him such a thing to his face.
What I mean is that I haven't heard of guys receiving death, rape etc threats accompanied by doxxing (ie 'this is his address, let's go get him') at the levels I've heard about with women[0].
What does this have to do with gamergate? If you've never seen guy/gal on guy death threats/doxxing, then I'd say you're just being will fully ignorant.
A few examples come immediately to mind: Adam Orth, Tim Hunt, donglegate, and shirtgate.
It's interesting that (except for Adam Orth) the men involved, who were factually the victims of massive harassment, were portrayed as the villains by most of the media for minor infractions of progressive political values.
Edit: Just found another, an electoral college member who received death threats for voting for Trump (as he was required to).
You haven't heard of it because the reporting and coverage is biased, not because it doesn't happen.
That's similar to the situation with sexual harassment, sexual assault, and domestic violence.
It's not that men aren't often the victims of these things (studies report that they are, at nearly the same rate as women[1, 2]). It's that men rarely report these things, and when they do, receive little sympathy, coverage, or justice.
Thanks. Amazing stats in the Slate article. And the Adam Orth example in particular has strong parallels with high-profile examples of attacks on women.
Online discussion certainly does inspire bad behavior. Hacker News is the only place I know of where people who disagree can still have civil discussions.
I'm in an adjacent academic field and know some of the principals in the story.
Based on private conversations with mutual friends, the reason Cuddy received so much heat is that she continues to profit off the mistaken findings and refuses to disavow them, despite overwhelming evidence.
She embodies everything wrong with the popular discussion of science.
The only analogue I can think of is Brian Wansink, whose scandal is much newer; who did not go nearly so far to profit from his own results; and who has received plenty of heat so far.
"Although 11 new papers have recently been published that do not show the downstream effects of power posing on behaviors, Cuddy is still fighting for power posing."
I'd disagree with your position - After looking at the larger body of the article, your finding is not replicated.
> She mentioned, at a psychology conference where she was presenting her work, that a study had recently been conducted on power posing. “They found no hormonal effects,” she said before taking a breath. “That study is done very well, and I trust those results.” Although 11 new papers have recently been published that do not show the downstream effects of power posing on behaviors, Cuddy is still fighting for power posing. The research, she says, still shows its effect on feelings of power: At the conference, she presented a comprehensive meta-analysis, a version of which, she says, she will soon publish, with a strong P-curve supporting that, and she also presented a P-curve suggesting that power posing had a robust effect on self-evaluations, emotions and moods.
> I also can believe that she faces hostile criticism that isn't equally distributed among all the other researchers who used weak, but previously acceptable standards.
I agree that this is true, but at a certain point it seems like a justified reaction to unequal importance.
Other Gelman targets like the "himmicanes" paper generated a bit of media attention without any real consequence. Power poses, meanwhile, have become a major pop-science hit. They're still being widely read and promoted, even by tech conferences and news orgs like the BBC. And Cuddy in particular has chosen an aggressive non-scientific promotion campaign while her co-authors have made more honest attempts to take on the replication question - if Carvey was getting the same amount of flak I'd be more concerned that something unfair was happening.
If power poses don't work, it seems like a much worse indictment of academic social psych than most other failures to replicate - the results were widely touted and are still being raised to public acceptance even as they fall apart. The challenges to stereotype threat, priming, and the IAT are even scarier - those results have become well-established and even enshrined in law.
Throughout the replication crisis, my greatest frustration has been the inconsistent narrative coming from researchers. They're simultaneously demanding formal academic critiques instead of blogs, ignoring those critiques when they do exist, and then replying with the same sort of informal aggression they decried in the first place.
I think you're exactly right that Gelman's aggressive approach is motivated by the field's unwillingness to acknowledge anything milder. And seeing responses like Fiske, Cuddy, and Wansink have largely convinced me that there's a major double-standard at work between the level of seriousness applied when attacking critics and the level applied when presenting results as academic truth.
> the results were widely touted and are still being raised to public acceptance even as they fall apart. The challenges to stereotype threat, priming, and the IAT are even scarier
They're in the same boat. It is maddening how much stereotype threat gets brought up to defend certain ideas, or how it has worked its way into corporate diversity policy, and it appears to be built on a foundation of sand [1]. The worst part is, if you question it publicly, you stand a change of being treated like James Damore.
The article directly contradicts a lot of what you've written and addresses all your points.
It such a shame for you to start viciously attacking Cuddy again for positions she clearly doesn't hold, when this article is directly about your misconceptions.
I doubt its a double standard. This sounds like more of the usual, flame wars on forums and responses from blogs upend normal human behavior. Even in the article, one of the people who attacked Cuddy realize that they were the people who had told Cuddy to respond the way she did. He himself comments that he forgot about it and then wrote his article against her. he also adds "oh, that email exchange (before writing the article) I had was too polite." But he still respects her as a top notch psychologist.
Matter of fact I'd guess its comments like yours which are fueling the underlying issue. The usual issues with communicating emotion on the internet.
There should not be schisms on science, yet there are - camps on facts. This is a problem with far worse implications than replicability.
But he called the piece "excellent and fair to all sides" and was pretty gracious in response to Cuddy's criticisms and how the NYT portrayed him. I honestly wasn't a huge fan of the article but re-reading it after a bit of sleep, and knowing that Gelman thought it was strong, I appreciated the issues it brought up.
I love (genuinely, non-ironically) that you reconsidered the merits of the article after reading Gelman's positive assessment of it. A possibly underappreciated benefit of Gelman's unabashed approach to criticism is that when he writes "I thought Dominus’s article was excellent and fair to all sides", it's reasonable to assume that he actually believes this rather than just pretending to "make nice" with scripted double-talk. His willingness to criticize things that he sees as wrong makes it possible (for me, and apparently you) to trust his stated opinion of things he sees as right.
And I don't think sexism can be dismissed outright, if people find it safer to talk shit to a woman compared to a man.
You prefaced it with an "if", but do you think it is common that "people find it safer to talk shit to a woman compared to a man"? I'd guess that at this point most male academics would be significantly more reluctant to criticise in print the work of a female colleague than that of a fellow male. I see Gelman as trying to adhere (possibly recklessly) to a gender-blind approach to criticism, and find this refreshing.
A lot of the issues aren't from academics in journals, as they are from facebook groups and other places where the internet hyper-cycle of polarization and mob formation occurs.
> "Although 11 new papers have recently been published that do not show the downstream effects of power posing on behaviors, Cuddy is still fighting for power posing."
Cuddy deserved neither the spoils of her research - fame and fortune, nor the indignity of having her work toppled in a public forum. She certainly did her self no favors by digging in, but at the same time, the despoilers would use her as the poster child for bad science itself.
This is a story in which a few people look to be "right", but no one looks particularly flattering, which is also why its a good read. Your comment adds very little.
If you're going to make a claim like this, could you please also give us some information? Otherwise you're just starting a flamewar and this is not a site for those.
Cheers for scientific revolutions being gentler than political ones. Cuddy lost her reputation rather than her head. But losing a reputation hurts more, or at least longer. More civility could have helped, but there's no painless way to lose status.
The scandal is that Cuddy's work met the standard of the bulk of published social science until lately. She studied the same phlogiston with the same tools as her colleagues. But when the revolution that comes for them, gently, is an epistemic advance, their pain is our gain. Join the revolution, comrad Cuddy!
The tragedy of the ordeal is that she tried and was lined up against the wall regardless, mostly due to lack of civility and poor communication:
> But the email that Simmons and Simonsohn had sent was, in fact, ambiguous: They had explicitly told her to drop the P-curve and yet left the impression that the paper was otherwise sound. At my request, Simmons looked back at his original email. I watched as he read it over. “Oh, yeah,” he said quietly. He had a pained look on his face. “We did say to drop the graph, didn’t we?” He read it over again, then sat back. “I didn’t remember that. This may be a big misunderstanding about — that email is too polite.” Cuddy and Carney had taken their advice literally.
I don't see it in that comment. Where? I only see "Cuddy lost her reputation rather than her head.", which is not at all similar. (I don't see it in NYT either.)
It was the New York Time's metaphor, not mine. But your substance does hold. Cuddy may have lessened her punishment, though, had she denounced the bourgeoisie with more fervor.
This is really the key point of the article (or maybe one of two, the other being Cuddy's unwillingness to accept change in favor of continuing to survive on her unsupported results); and the overall message is that we all do dumb things, directed toward our own benefit, under pressure. (E.g. Carney saying there's no ninth planet, when in fact they only did not show there is a ninth planet). Gelman is actually the only primary player in the piece who mostly escapes this; he does some nerdy things but in no case says something other than what he means, or actively deceives (as Simmons appears to have done here) or self-deceives (Cuddy).
Perhaps the meta lesson is we massively overproduce academics, making the stakes and competition insane.
In the cutthroat world of academic soft sciences you're not getting tenure unless you make a breakthru and you're allowed zero mistakes. Its up or out. She made her one mistake, and she's out. And there's 10, 50, 5000 people equally qualified without opportunity to cheer on her demise. One of hundreds or thousands of equally qualified researchers now gets a chance at tenure, good for them bad for her.
On the other hand in the commercial software development world, we usually don't blacklist software devs from the field after their first bug because 1) we're low on supplies of software devs plus or minus propaganda, at least we don't have the excess of her field 2) the experience of having a software bug makes a better dev.
You can't expect non-pathological behavior from a pathological situation. Until the supply/demand hiring issues in academia are fixed, stories about academia are going to read like an insane elementary school playground version of Lord of the Flies. You can't produce 10 to 100 times as much product (new grads) as the market can adsorb without behavior for the few good positions being viciously cutthroat and extremely dramatic.
I think it's pretty easy for this discussion to become about personalities and not methodologies. Gelman is pretty brash and speaks his mind, so he could be correct on methodology and have that get lost in the noise.
So I'll give my opinion as a datascientist: null hypothesis significance testing is broken, the whole thing needs to get chucked. It's not fixable, and it's not only p-hacking that's the problem. Go read Frank Harrell, I like his writings on the topic more than Gelman's.
tl;dr Promising young researcher skated on stats and talked up faulty finding, career seriously damaged by lack of replicability; reformers who are too earnest about a good cause (replication, statistical rigor) can be total assholes to well-meaning people who just make mistakes; conflict ensues.
It's OK to be oppositional or confrontational to people based on what they say and do, but self-styled reformists sometimes become personally vindictive in pursuit of their own interests. Picking a fight with a public figure is an easy way to gain publicity, and if your own reputation rests on winning that fight then it can be tempting to carry on into harassment. Gamergate was an extreme example of this.
A good working definition of a good researcher, to me, is something like "strong opinions, loosely held".
I don't think Cuddy, or anyone else, deserves to be dragged through the mud because their idea turns out not to hold up. That happens all the time, and it's not a bad thing at all.
You start to deserve that dragging more when you become too attached to your pet theory and cease to be able to drop it in the face of overwhelming evidence to the contrary.
You really deserve it when you use what political capital you've acquired to stomp on your critics despite the fact that they're actually right and you're actually wrong.
What's interesting here is that a lot of the Sayre's Law-type political machinations are taking place on a much broader stage than normal. If someone was to go after my work with this kind of exposure, you bet I'd be defensive too. It's entirely possible to have a fair and neutral conversation about research practices without shanking someone's reputation because it seems like the field needs a sacrificial lamb this week. Nobody wins, and certainly no good research is advanced, when the knives come out like this with no evidence of actual malfeasance. (And, frankly, even when there is evidence of malfeasance. like with the STAP controversy, which lead to at least one actual death.)
has anyone gone down the list of most popular TED Talks and run similar replications of the claims made in other presentations? itd be interesting if there were some ratio that scored most-popular TED Talk vs. "B.S." content. i wonder if there'd be a link between popularity and amount of B.S.?
NYT tries to frame it as if the rules of science were unfairly changed. Not so - people had just been overly lax for a while. And sorry, NYT, but scientific knowledge is not a matter of opinions and goodwill. We can not simply decide her theories are true after all, because we empathize with her so much.
I published a couple of papers in grad school, and what the article describes made me imagine the peer review process busted open for the world to see. Reading reviews sucks: good reviewers provide smart reasons that you're wrong, which you have to beat with more research saying that you're right. The editor makes the final decision, not always in your favor.
In their "open peer review", the psychologists seemed to have all the class of a bunch of redditors. An uncomfortable process is made worse! Instead of motivating people to publish lame results for tenure, this new situation seems to motivate people to act like Internet trolls or tabloid journalists.
A more critical research community should help advance the field, but these researchers set a bad example.
The work of these social psychologists absolutely deserves the scrutiny they are receiving. The field is rife with "surprising scientific effects" that do not replicate and turn out to be completely wrong, yet they linger in the popular culture and are brought up over and over again to defend ideas and policies that are not based in reality.
Look at the reach of people like Malcolm Gladwell and stuff like his books and podcast, and NPR's Hidden Brain, which subsist on recounting these "surprising studies" and speculating about their profound meaning for society, to gigantic audiences. And they're wrong! Made up ideological BS with the gravitas of 'serious science' behind them.
Power posing, stereotype threat, the bystander effect, implicit bias. So many popular hypotheses have snuck into the popular culture and they are wrong. They're based on fudging numbers and statistics and throwing out results that don't fit the hypothesis. They deserve to be combated in the public sphere. They don't deserve to be shielded to protect the careers of the hucksters.
I used to love TED talks but over the last few years I have developed an aversion. It seems a lot of popular TED people have traded scientific integrity for showmanship (and money). Makes me wonder how Feynman would be doing today. He was a charismatic talker. He could be a superstar.
149 comments
[ 4.6 ms ] story [ 222 ms ] thread"Although 11 new papers have recently been published that do not show the downstream effects of power posing on behaviors, Cuddy is still fighting for power posing."
This is an astonishing number of replications to be published on one topic in psychology. 2-3x the next highest number I heave heard of.
The placement of that sentence, plus the flattering photos, leads me to suspect a publicist was able to place a thumb on the scale.
she is "still fighting" for power posing but only as a useful technique which sometimes works for some people in some cases, as even the replicated studies seem to show improvements in subjective measures of "confidence", whatever that means.
This is the same Susan Fiske who called people who pointed out statistical errors in psychology papers "methodological terrorists". http://andrewgelman.com/2016/09/21/what-has-happened-down-he... The apple does not fall far from the tree.
She is criticizing "unfiltered trash-talk [on social media]... [and] online vigilantes [who] are attacking individuals, their research programs, and their careers".
It's the same tension the article is referring to, between civility and public discourse. On one hand, you'd really like an open dialog about the quality of evidence presented. On the other hand, there's a real price to be paid when people are too worried about public humiliation to risk doing science on certain questions or working with certain people.
As the quote from Jay Van Bavel put it, "It's become like politics — we’ve created two camps of people who shouldn’t be in two camps in the first place"
This point resonated with me even though I'm on your side of those politics. What effect would it have on the OSS community if any even slightly bad code you write could end up in a hit piece published by The Atlantic, and if fact you witnessed people on Facebook saying your colleague, who maybe did write some kinda bad code that one time but is on the whole competent, is a charlatan who doesn't deserve the title of "programmer"?
I don't see as wide of a gap in effect on career between open source contributions for engineers and published papers for scientists as you seem to be implying
Her work was interesting no doubt, and in our scientific age we have always celebrated the novelty and originality of the scientific geniuses that have advanced our understanding.
But the real value of science has always been novelty plus certainty. The scientists who we esteem should be known as much for their industrious management of the empirical process (recruiting replicators, statistical auditors, etc.) as they are for they are for their creative insight.
geofft's comment hits the nail on the head when it comes to what effect this did vs. should've had on her tenure case, I think. A study that fails to replicate is no indication of malice or even incompetence. Some say she should've run self-replications, but how many years self-replication studies are you willing to run on $20k/yr (in fact, how many post-college years have you made 20k/yr for a full-time job)?
At some point you get the quality of science you can pay for, and throwing the grad student under the bus by denying them tenure a decade later is as shitty and spineless as Equifax throwing an IT guy under the bus for what was pretty clearly an organizational failure. IMO.
Re: TED talks, TED is not a scientific venue, and most of the things I hear in TED talks are a hell of a lot less scientific than even Cuddy's most flawed studies. Also, I don't think those sorts of things should weigh much in hiring and tenure cases either way (aside from e.g. a usually marginal "service/outreach" category).
If Harvard hired Cuddy because she gave a popular TED talk, then that was an extremely poor hiring decision by Harvard. But that's a separate issue from the one we're discussing.
I know, and I think we are mostly in agreement. I generally don't like seeing individuals dragged through the mud. It's also somewhat unfair to judge her actions pre-replication crisis by our standards post-replication crisis.
I also agree that a single study failing to replicate is not an instance of malice or incompetence, in fact I think all scientists should expect it to happen. But that also means that no career should be built on the back of a single study. It seems that she is still sticking to her guns even after 11 failed replications. That does cause me some concern.
> but how many years self-replication studies are you willing to run on $20k/yr? At some point you get the quality of science you can pay for...
Then maybe one should more hesitant about promoting those results?
> If Harvard hired Cuddy because she gave a popular TED talk, then that was an extremely poor hiring decision by Harvard. But that's a separate issue from the one we're discussing.
I think it makes me more sympathetic to her case. If Harvard had not hired her, or TED not promoted her, she would not have been subject to nearly the same scrutiny or ire as she was. Ironically, she might have had a better shot at a good long-term career. But who, at that point, would have turned down either of these prizes? I know I wouldn't have.
We definitely agree on this. I think all the TED science stuff is silliness. Cuddy's mistake was both being a serious scientist and being a popular TED speaker. Popular TED speakers make BS science claims all the time, but they aren't scientists so it doesn't matter.
But I wouldn't advocate for firing someone because they disagree and think TED is a good place to popularize science (even if they happen to popularize science that later turns out to be bad science).
There is a concept in management called the Peter principle, which is that "managers rise to the level of their incompetence". I've always worried about it in myself - I work hard to advance my career, but is it possible I'm sabotaging my own long term success? What if I get the high-profile new responsibilities I'm striving towards... and then fail spectacularly?
[0]: https://en.wikipedia.org/wiki/Peter_principle
> It's given me the impression that if I don't create something that is up to a teams standard from commit 0
Linus has said that he reserves his abuse for senior people who should know better. It's his antidote to a complacent overclass that mistreats the kernel development process.
Let me quote Gelman regarding the subject of her criticism:
> The other thing that’s sad here is how Fiske seems to have felt the need to compromise her own principles here. She deplores “unfiltered trash talk,” “unmoderated attacks” and “adversarial viciousness” and insists on the importance of “editorial oversight and peer review.” According to Fiske, criticisms should be “most often in private with a chance to improve (peer review), or at least in moderated exchanges (curated comments and rebuttals).” And she writes of “scientific standards, ethical norms, and mutual respect.”
> But Fiske expresses these views in an unvetted attack in an unmoderated forum with no peer review or opportunity for comments or rebuttals, meanwhile referring to her unnamed adversaries as “methological terrorists.” Sounds like unfiltered trash talk to me. But, then again, I haven’t seen Fiske on the basketball court so I really have no idea what she sounds like when she’s really trash talkin’.
> I bring this up not in the spirit of gotcha, but rather to emphasize what a difficult position Fiske is in. She’s seeing her professional world collapsing—not at a personal level, I assume she’ll keep her title as the Eugene Higgins Professor of Psychology and Professor of Public Affairs at Princeton University for as long as she wants—but her work and the work of her friends and colleagues is being questioned in a way that no one could’ve imagined ten years ago. It’s scary, and it’s gotta be a lot easier for her to blame some unnamed “terrorists” than to confront the gaps in her own understanding of research methods.
> To put it another way, Fiske and her friends and students followed a certain path which has given them fame, fortune, and acclaim. Question the path, and you question the legitimacy of all that came from it. And that can’t be pleasant.
I'm less concerned with Fiske vs. Gelman and much more concerned with Jay Van Bavel's observation that either way, science loses.
Confining scientific critique to scientific venues seems like a reasonable bulwark against this sort of thing. If it had been Reviewer #3 in a program committee or editorial board, or a well-received letter to the editor, instead of a blog post picked up by thousands of internet warriors with no stats training, perhaps the result could've been refined or thrown out without expelling someone from research. If the feud hadn't been picked up by pop science press, then maybe a bad piece of science could've been questioned without poisoning the well for further inquiry.
Fiske controls the venue and is (mis-)using the "scientific venue" to denounce her critics. She is the one refusing to publish the very critiques you describe. The "scientific venue" has been corrupted. The well has not been "poisoned" for further inquiry, rather the necessity of replication and sensitivity to statistical errors has increased. Science was being done sloppily, and Fiske was both a beneficiary of that and in a role where she could have prevented it.
That's a pretty strong accusation. Do you have any evidence?
Cuddy herself, in the linked article, even asks: "Why not come to a conference or hold a seminar?"
I don't think you're wrong about the lack of quality reviewing, but I also think that blogs and Facebook and articles in pop sci news venues are pretty shitty places to address the quality of editorial review.
And your post demonstrates why: honest mistakes -- or even complete non-mistakes stemming from gaps in established best practices! -- turn into out-right accusations of full on corruption. The comments on NYT demonstrate the same problem from the other side: lots of gender politics when Gelman wasn't motivated by gender, implicitly or otherwise, but rather by the mass appeal of Cuddy's work.
Needless to say, I also think you're massively over-stating the maliciousness of people like Cuddy and Fiske.
> Science was being done sloppily, and Fiske was both a beneficiary of that and in a role where she could have prevented it.
Simmons seems to agree with me that you're really over-reaching here: "The original study wasn’t particularly egregious," he said. "It was published in 2010 before anyone was thinking about this."
There's a huge difference between doing science the way you were trained (but sub-optimally), and outright "corruption".
There's also a HUGE difference between noticing a problem and constructively pushing for more rigor within a sub-community, and going after particular people within that community so harshly you destroy their careers.
The former is great for science. The latter might possibly by great for improving rigor (I'm pessimistic and think it all comes down to incentives, which Gelman et al don't really do much to change). But equivocating bad methodological/statistical training/practice with unethical and even criminal ("corrupt") behavior? That sort of dishonest fire-and-brimstone talk is just as damaging as a lack of rigor.
1. failing to have read Meehl's critique doesn't amount to "corruption",
2. that calling people "corrupt" for having failed to read Meehl is not likely to be the most effective or the most civil path toward changing things, and
3. To the extent that discussion of scientific methodology on public forums causes or contributes to (2), that's a bad thing and perhaps suggests a fundamental cost of doing science in this manner.
I'm having trouble finding a charitable way to read the claim that someone is misusing a scientific venue.
And much more to the point, I doubt even more that the subject of those sentences (or the e.g. tenure/promotion committee of that subject) could be reasonably expected to apply such an abundance of charity.
In a sense, both pieces are self-published, and she is using the mantle of Science to falsely claim that her self-publishing is superior because of the authority of her position as editor. This is corruption. This is misusing the scientific venue.
Please link to the place where she actually makes this claim.
> in her journal
Not hers, and not a journal.
> while his self-published piece is "methodological terrorism."
She actually outlines the practices that ought to be singled out as unprofessional and unacceptable in the second to last paragraph, and none of those are mere self-publication.
Have you considered the possibility that both sides are wrong? That this is crap science, and that the conduct of certain groups of people rallying against crap science is overzealous to the point of cruelty?
Finally, it's also worth noting you've COMPLETELY proven Fiske's point about social media: sure, there are some valid critiques. But there are also people being absolutely vicious to people actually trying to science because of very trivial and obvious misunderstandings (e.g., because they think that APS Observer is a journal). I think to some extent that's a reasonable price for free discourse when things stay online and especially when they're anonymous, but you're calling a real person with a real job on the line corrupt because you made an incorrect assumption (that would've taken you 10 seconds of googling to correct).
"Cuddy herself, in the linked article, even asks: "Why not come to a conference or hold a seminar?""
I am not criticizing Cuddy. Fiske is editor of PNAS, former president of APS, and is putting her criticisms in APS Observer. The difference between a journal and an official magazine of a scientific organization is irrelevant in this concept, in that the supposed superiority of an edited journal is what is in question here when the editor is not good, and is not considering the context of the discussion.
I am pointing out the utter hypocrisy in her argument that you shouldn't point out statistical errors on blogs, while she is pointing this out in a publication she controlled. She wants people to be able to protect their undeserved status, it is clear.
The entire idea that science needs to be locked up in limited access journals (or magazines) is not a good one. The real issue is that the quality of PNAS has been pretty low. Fiske's response to that is to try to discourage open criticism. Open data is one of the reasons that people were able to find these errors. Resistance to publishing failed replications is not proper behavior. We have now seen PNAS publish a few letters describing errors, and hopefully the standards will improve. However, it hardly seems like a person whose response is to lean toward hiding the discourse is the right person to do it?
Fiske is the journal editor. Fiske is allowing articles with weak stats to get published. Fiske is publishing her defense in that journal. Fiske is not publishing critical articles in that journal. That's corruption based on her position of authority.
Cuddy just had sloppy stats and her results failed to replicate. Notice how you deceptively switch to Cuddy from Fiske in your response, because you have no point about Fiske.
APS Observer is a magazine, not a journal.
Furthermore, Fiske was not the president of APS or an editor of the Magazine when that article was published. And even if she were, what's the problem? I see letters from the editor in most magazines I read on a very regular basis.
Finally, the claim that Fiske was construing her opinion on conduct as a scientific fact is just... well.
I'll refrain from calling you corrupt or deceptive because it was probably just a misunderstanding :-)
> Notice how you deceptively switch to Cuddy from Fiske
I actually have no idea what you're trying to say or what this refers to. Can you be more explicit?
I don't think turning a call for greater scientific rigor into a political fight is desirable, but neither do I think that confining this struggle to scientific venues is desirable (or practical, for that matter).
EDIT: wording fix
When you take your BS "studies" and sell it to the audience at TED and then the wider internet, go on corporate speaking tours, sign book deals to promote your "science", and try to affect public policy, you are open to criticism from anyone who sees through your shoddy methodology and wants to correct the record.
(Also, as a side note, I'd fully expect that CS is at least as vulnerable to the same thing. McSherry et al.'s COST work is a good example of something that seems far worse than p-hacking: instead of having a clearly defined hypothesis and publishing the 1-in-20 papers that falsely state it's true, as in other fields, there's nothing clearly defined in scalable systems and 20 out of 20 people who build something excessively complicated can get a paper out of it.)
http://www.frankmcsherry.org/assets/COST.pdf
the paper you referenced?
Literally the papers are "I built a thing, and it gets faster when you do this thing to it." There's no condition on whether the base case is realistic.
To be clear, I don't think the authors of the papers he's taking down are bad people, or maliciously trying to get publications for uninteresting work, or too stupid to implement things efficiently on a single core, or anything. I think they're genuinely interested in the work they're doing, but responding to the pressures of the environment that allows them to keep doing that work.
Or you could get a large sample size to bring the p down to 0.01 or whatever. You know, do science.
It's entirely reasonable, IMO, to do more "exploratory" studies at p = 0.05, and then do a follow-up on promising studies at p = 0.01
The problem is that a study that's significant at p = 0.05 but not p = 0.01 is worth something to science. It indicates something worth doing several more studies on, preferably by different teams from different institutes with non-overlapping confounding factors. But the practice of science gives you only two options once you find such a study, as a young scientist: promote it immediately and become visible for having discovered something, or aggressively ignore it and go find something else that you think is likely to be immediately promising. "Doing science," as you put it, is the worst option for you because it ensures you spend a lot more time on something for no reward -- and the rational self-interest here is perfectly rational, because if you don't get funding or tenure, you're not going to do any science any more at all. (As seems to have happened with Dr. Cuddy.)
To be clear, I have no well-formed opinion on whether Dr. Cuddy is a good scientist, having only read this article. I assume she is, or at least that she's honest in her desires. But I think talented, honest people are nonetheless pressured by incentives, and fixing incentives is better for society (assuming we as a larger society wish our scientists to continue to do science) than assuming that people will be self-sacrificial and altruistic in pursuit of a tiny chance at accomplishing the greater good.
Meanwhile, there are enclaves where biologists are slaving away capturing data and not even realizing it's data. "Bookkeeping data" they might call it.
There should be mixers between the biologists and the computer scientists. Especially non-academic physicians and computer scientists. Look at Kaiser-Permanente, Sutter Health, or Defense Health Agency. Huge orgs, very little research. Data falling off the shelves and they don't know what to do with it.
1. A legal framework for health-related products & services that meaningfully protects individual privacy, and yet also preserves the conditions for the kind of velocity and rate of change that brings technological progress to the masses reliably and quickly.
2. Massive academic funding for more “pure” research in hardware & software design, engineering, the philosophy of what’s possible, the phenomenon and ethics of what we choose to build, and so on. TLDR Pay more people more money to do important research in the public domain, which I can’t help but think would attract many of the CS types you mentioned.
That exists, but there's upfront educational costs that need to be handled first.
> Massive academic funding
Look at Europe: their acedemics work much more closely with industry. Get a someone who can code in a room with a physician, and I garuntee they can find a problem to work on. Show a successful prototype, and we're way below "minimum viable product" here. Show a graph with real data and money will start looking for you.
Computer scientists, mathematicians, physicists, and engineers need to go to biology conferences. And biologists should be going to math, ML, and software conferences.
I still don't understand this one. The null model is (almost) always false. In this situation a small p-value is only a matter of sample size and measurement precision. How can it possibly indicate you should work more on something?
One thing I think may make sense is to get a bunch of p-values and then choose the lowest 10 or whatever to focus on for further work. There is no reason to include an arbitrary cut-off here though.
Also noticing this now... No, this is completely untrue. The p-value is not an error rate.
There will never be an objective measure for conclusion 'correctness'.
It will always be the case that you actually have to think about the data and statistics critically and decide whether the conclusions are appropriate. In certain well-defined situations, you can start to use harder metrics, but that's rarely where the controversy is in the first place.
So if you want to, you know, do science, I urge you to use that big beautiful brain of yours. As a bonus, you don't even need to commit mass rat genocide to do it.
There were (are?) enough degrees of freedom in the field in every study to always meet this threshold.
We say this, but I haven't heard any particularly good ideas on how to build such a system that isn't also easy to game. How do you ensure the failed studies were actually worth doing? Is peer review of study design feasible?
This is roughly how private industry works. I'm of course asked in job interviews about my successes, but the primary thing that determines whether I get a job, how much I get paid, whether I get other people to work with, etc. isn't what my past successes are, it's my ability to convince my peers that I am a steady and good worker. If I maintained some tech stack for two years, my employer is happy to let me continue doing that, and other employers are happy to hire me to maintain similar tech stacks, without me having been like the one person in the world to figure something out. (On rare occasion, I am, in the sense that I'll fix something upstream that's been challenging, and employers do value that! But there's no, e.g., "fix upstream bugs or perish" culture.)
As in private industry, it's gameable because it allows people doing poor-quality work to vouch for other people doing poor-quality work. But it's hard to game this long-term, because people interested in doing good-quality work will tend to find others who do, and in aggregate -- from a large number of such people working together over a period of time, not from any one individual -- these people will produce more measurable results.
Note that by "organizations" I don't mean "universities" or "institutions". Departments should probably be treated separately, as different departments within an institution can easily drift in terms of quality. Meanwhile, it should be possible recognize collaborations of researchers across researchers as an entity: if the collaboration wants to support a researcher moving between institutions and wants to keep supporting them, that doesn't lead to the same flaws, I think.
Are you aware of Registered Reports? https://cos.io/rr/
Also, if you haven't been following the 'X-ray Carrots' fiasco, that is a BIG deal in science and Science. VERY short summery: Andrew Wansink at Cornell made up a bunch of data and results, he published them, he got caught. One of these papers was about calling carrots as 'x-ray carrots' makes kids eat more of them, hence the name of the fiasco. Here's a good way into the mess [1]. Anyway, some papers were retracted, there was an investigation, yadda yadda. Now, in 'normal' science, he would have been banished from the field for this nonsense. But in 'finding-or-famine', he got off essentially scot-free. Nothing happened, really. He's still the head of the department (AFAIK), he still brings in cash, the FDA still is using his findings. This is a huge blow to science and a huge boon to Science.
The existential crisis is now out in the open: you can cheat with impunity, publish whatever, if you get caught nothing will happen, just bring in the dollars.
Now is not a good time to go to grad school. I think they're just going to have to throw out ALL US based bio-science (at least) from about 2003 to whenever heads start rolling. It's all crap, maybe, who knows.
[0] Science and science are different, though look similar. Big Science is the grant machine and the big money side of science. Splashy, barely replicated, large clean buildings, pensions, perfectly fit for corporate needs. Think Big Tobacco, BPA plastics stuff, power poses like Cuddy's, that kinda stuff. It's like the difference between the Military Industrial Complex and actually kicking down doors in Mosul. 'Little s' science is what you are probably thinking of when you think of science generally.
[1]http://andrewgelman.com/2017/04/06/dear-cornell-university-p...
I think you missed a decimal point or two here -- $42,000/month for a 100 square foot office can't be right :).
In my experience, about a third or half of the 'real' scientists I've met are firmly in "Big S" science. It's really not that different. People that complain about this a lot are often jealous, in my experience. The failures are certainly high-profile, though.
As for reproducibility, it's all case by case. We've (real? scientists) been having the same discussions about data quality forever. It just seems to have found a fresh audience. There's a reason a curmudgeonly PI might insist in Western Blots and - gasp! - Northerns from time to time. Most fields were built off of some real science. Molecular biology certainly was, and to think that just stopped happening in the early 2000s is the height of ignorance.
There is cause for alarm. I give pretty much all 'genomic' work a very wide berth unless the lab has a real genetics background, and aside for a few labs, anything with 'microbiome' in it is certifiable garbage. And clearly some entire fields are on shaky ground, namely psychology. But lets not circlejerk our pure 'little s' science ego's too vigorously.
You misunderstand, perhaps. I mean that the data itself is fraudulent, let alone the interpretations or the controls on the experiment. Many PIs have been making up the data, essentially. Pulled from thin air with a little rand() thrown on top. Now the flood gates to this behavior are open and 'accepted', at least by the people that cut the checks.
It's just curious to me that she and Dr Fiske wouldn't catch this crucial difference immediately. Effectively it turns power poses into little more than 'the power of positive thinking'.
That said, personal attacks and especially those focusing on her brain function really need to stop. Psychologists treating each other like shit is not just ironic, it's a disgrace. If they can't even behave properly with each other, how can we expect them to treat patients properly?
Meaning, say the power pose effect doesn't exist. But early on, it's exciting and new, and there's a placebo effect (which is fine!), but people also think it's real.
But then studies get replicated, and there's well-known doubt on whether the effect is real in a non-placebo manner. There might still be a placebo effect, but does it shrink?
And then there's a lot of controversy and the study is dragged through the mud, and power poses are mocked. Might the placebo effect be lessened even further (or obliterated) from this?
Could this be true even if it was originally valid and not just placebo?
For the time being, the answer is probably "if it works for you, then keep doing it." If power posing before an important meeting is how you psyche yourself up to be unafraid and effective in the meeting, then why not keep doing it, even if it doesn't seem to work for everyone?
Knowing the replication problems, I won't suggest power posing as a general thing. Maybe I will tell someone to try a bunch of things including power posing, and tell them to keep the things that they think are working.
http://andrewgelman.com/2017/10/18/beyond-power-pose-using-r...
I thought that Gelman wasn't presented in the best of light, especially his awkward quotes about not wanting to engage in "interpersonal conflict". But he called the piece "excellent and fair to all sides" and was pretty gracious in response to Cuddy's criticisms and how the NYT portrayed him. I honestly wasn't a huge fan of the article but re-reading it after a bit of sleep, and knowing that Gelman thought it was strong, I appreciated the issues it brought up. Certainly has to be one of the longest stories about p-hacking/p-curves in mainstream media.
I sympathize with Cuddy and have no doubt that among the legit attention that Gelman and other critics have directed toward her, she's faced hatemail sparked by dumb sexism, which makes it easier to tar the legitimate criticism as being sexist.
I do think Cuddy needs to face legitimate criticism. She may not have been a malicious fraudster, but ignoring criticism after a long time is not much different than actively being deceptive. I also can believe that she faces hostile criticism that isn't equally distributed among all the other researchers who used weak, but previously acceptable standards. And I don't think sexism can be dismissed outright, if people find it safer to talk shit to a woman compared to a man.
What I wish the story had more of was how scientific criticism is done outside of Gelman's blog. Gelman is the main scientist/research I follow on the topic. Apparently some see his snarky critique as toxic, and Gelman says he is intending to reevaluate the way he writes about things so that his point doesn't get lost in cheap jokes. How do other blogs do it? I'm somewhat sympathetic to Gelman because I perceive that there is significant institutional resistance (and few incentives) to re-examine and correct a peer review study, and being harsh/edgy/rude and beating the drum over time might be the only way to get scientists to care? But that doesn't mean his critique can't be refined for the benefit of his mission.
Talking about the stakes is in direct conflict with the scientific ideal of focusing on the science without talking about its effects on the scientists involved.
For example, from Gelman's response: "I just wanted to emphasize that there’s no reason her career, or even her famous Ted talk, has to rely on a particular intriguing idea (on there being a large and predictable effect of a certain pose) that happened not to work out." Well, except that maybe she wouldn't have gotten a Ted talk without it?
He's saying, hey, this paper shouldn't have been as important as it was. And according to the scientific ideal, we shouldn't get too attached to any particular results. But in practice, putting bad results aside sometimes requires a significant change of identity. Forgetting about that, even for a moment, is always going to come across as tin-eared.
The scientific ideal is more compatible with low stakes than high stakes. But talking about the game as if it's low-stakes (or ignoring the stakes altogether) when actually it's high-stakes is always going to make discussions focused on the science alone come across as unsympathetic.
Science is about finding approximations of truth that are least influenced by the local context.
The whims of a particular scientist's ego (especially if you are that scientist!) make for a rather nasty contextual influence.
Of course, this is a difficult ask, but constantly striving towards this ideal is absolutely essential.
But does this make sense? Can a scientist survive without doing this? Can a scientist get very far without results right now?
Is this a problem of the scientist, or perhaps the social/economic expectation?
New results are significantly more difficult and expensive to achieve in modern science, as the easier stuff has already been conquered in centuries and decades past.
The harder problems that science now works on means that distinguishing between actual results and incorrect conclusions is also much more difficult.
The growth of science as a percentage of the population and economy has also finally slowed down a few decades ago, after centuries of rocket-like growth. It is now a maturing enterprise, instead of a high-growth startup.
This means increased competition for resources and mindshare from those wanting to do science. Not surprisingly, all proxy measures and heuristics for evaluating scientific success are thus more likely to be abused.
I don't think we have fully adapted to these fundamental changes.
Disclaimer: I'm not a historian or researcher, but I've heard aspects of the above from a variety of sources, and it makes a lot of sense to me.
Of course, that's much easier for people in STEM than it is for social scientists.
But you also can't prevent people from learning about how the world works: the rewards and penalties that will affect their careers.
Even if you could, teaching students a myth about what a career in science is like would basically be malpractice.
A recent BBC event aimed at solving problems faced by women in tech flew Amy Cuddy in to give design and professional advice to the participants. The BBC story repeatedly celebrated Cuddy for giving the "second most watched TED talk ever". It also talked up the "Wonder Woman power pose" as a way to change your life.
Notably not present at the event were Cuddy's co-authors Carney and Yap, who have been vastly more receptive to the replication question, and less quick to promote uncertain results in non-academic channels.
Scientific falsehoods are almost always more impressive than the truth - they aren't constrained to realistic effects. At a certain point it starts to look like we're actively rewarding people for doing bad research and then rewarding them further for refusing to correct course.
http://www.bbc.com/news/live/world-41476954
This makes lesser people working in dirtier subjects very uncomfortable: when he hits them they have nothing of substance to hit back with except generic insults.
http://andrewgelman.com/2016/05/05/npr-bites/
http://andrewgelman.com/2016/11/03/its-not-about-reality-its...
I just read his blog for the first time and thought he’s a huge asshole. It seems to me the primary effect of science-shaming on the blog is to rally the pitchforks, something he can’t possibly be ignorant of. But he apparently keeps on doing it. No doubt it’s good for the hits, but doesn’t contribute to a dynamic that leads to good science. Oh wells.
A little needling on Gelman's blog is not too harsh, all things considered.
Sexism comes out in target choice, and the strength of which ones chases their prey, and the vehemence of attacks.
I do strongly believe that Cuddy was overly punished for being a woman, and being prominent in the field. Everyone loves when the "high and might" come "crashing down" -> the underdog stories and the desire for retribution are sought after more powerfully than (a) justified and (b) end worthiness of the goal.
I don't think anyone can honestly say that this has improved the field and made science better. It just means now that there is money and recognition to be made being attack dogs. That's hardly better.
As for Gelman, he wasn't portrayed in a great light, but I think it is fair. He's willing to encourage incivility and outright sexism and hostility as long as he doesnt have to talk to someone he disagrees with. That's the baseline definition of a coward there. And yes, I would discuss that with him in person and call him such a thing to his face.
Theres a pattern of successful women receiving extreme levels of hate (Kathy Sierra et al). I'm not aware of this happening to guys.
Can you be more specific? Or do you mean this in a general sense? Because isn't this obvious ignorance if so?
What I mean is that I haven't heard of guys receiving death, rape etc threats accompanied by doxxing (ie 'this is his address, let's go get him') at the levels I've heard about with women[0].
[0] - https://en.wikipedia.org/wiki/Gamergate_controversy
But what I'm doing here is drawing parallels between large groups of people self-organizing against a prominent, successful person.
It's interesting that (except for Adam Orth) the men involved, who were factually the victims of massive harassment, were portrayed as the villains by most of the media for minor infractions of progressive political values.
Edit: Just found another, an electoral college member who received death threats for voting for Trump (as he was required to).
http://thehill.com/blogs/in-the-know/in-the-know/308226-elec...
That's similar to the situation with sexual harassment, sexual assault, and domestic violence.
It's not that men aren't often the victims of these things (studies report that they are, at nearly the same rate as women[1, 2]). It's that men rarely report these things, and when they do, receive little sympathy, coverage, or justice.
1: http://www.slate.com/articles/double_x/doublex/2014/04/male_...
2: http://domesticviolencestatistics.org/men-the-overlooked-vic...
Given the level of heat these discussions arouse, I understand the comments re modding in the item earlier today: https://news.ycombinator.com/item?id=15507821
Based on private conversations with mutual friends, the reason Cuddy received so much heat is that she continues to profit off the mistaken findings and refuses to disavow them, despite overwhelming evidence.
She embodies everything wrong with the popular discussion of science.
The only analogue I can think of is Brian Wansink, whose scandal is much newer; who did not go nearly so far to profit from his own results; and who has received plenty of heat so far.
"Although 11 new papers have recently been published that do not show the downstream effects of power posing on behaviors, Cuddy is still fighting for power posing."
> She mentioned, at a psychology conference where she was presenting her work, that a study had recently been conducted on power posing. “They found no hormonal effects,” she said before taking a breath. “That study is done very well, and I trust those results.” Although 11 new papers have recently been published that do not show the downstream effects of power posing on behaviors, Cuddy is still fighting for power posing. The research, she says, still shows its effect on feelings of power: At the conference, she presented a comprehensive meta-analysis, a version of which, she says, she will soon publish, with a strong P-curve supporting that, and she also presented a P-curve suggesting that power posing had a robust effect on self-evaluations, emotions and moods.
Gelman regularly eviscerates the research of Brian Wansink on his blog, in the exact same way that he did to Cuddy.
I agree that this is true, but at a certain point it seems like a justified reaction to unequal importance.
Other Gelman targets like the "himmicanes" paper generated a bit of media attention without any real consequence. Power poses, meanwhile, have become a major pop-science hit. They're still being widely read and promoted, even by tech conferences and news orgs like the BBC. And Cuddy in particular has chosen an aggressive non-scientific promotion campaign while her co-authors have made more honest attempts to take on the replication question - if Carvey was getting the same amount of flak I'd be more concerned that something unfair was happening.
If power poses don't work, it seems like a much worse indictment of academic social psych than most other failures to replicate - the results were widely touted and are still being raised to public acceptance even as they fall apart. The challenges to stereotype threat, priming, and the IAT are even scarier - those results have become well-established and even enshrined in law.
Throughout the replication crisis, my greatest frustration has been the inconsistent narrative coming from researchers. They're simultaneously demanding formal academic critiques instead of blogs, ignoring those critiques when they do exist, and then replying with the same sort of informal aggression they decried in the first place.
I think you're exactly right that Gelman's aggressive approach is motivated by the field's unwillingness to acknowledge anything milder. And seeing responses like Fiske, Cuddy, and Wansink have largely convinced me that there's a major double-standard at work between the level of seriousness applied when attacking critics and the level applied when presenting results as academic truth.
They're in the same boat. It is maddening how much stereotype threat gets brought up to defend certain ideas, or how it has worked its way into corporate diversity policy, and it appears to be built on a foundation of sand [1]. The worst part is, if you question it publicly, you stand a change of being treated like James Damore.
[1] - https://www.psychologytoday.com/blog/rabble-rouser/201512/is...
It such a shame for you to start viciously attacking Cuddy again for positions she clearly doesn't hold, when this article is directly about your misconceptions.
Matter of fact I'd guess its comments like yours which are fueling the underlying issue. The usual issues with communicating emotion on the internet.
There should not be schisms on science, yet there are - camps on facts. This is a problem with far worse implications than replicability.
I love (genuinely, non-ironically) that you reconsidered the merits of the article after reading Gelman's positive assessment of it. A possibly underappreciated benefit of Gelman's unabashed approach to criticism is that when he writes "I thought Dominus’s article was excellent and fair to all sides", it's reasonable to assume that he actually believes this rather than just pretending to "make nice" with scripted double-talk. His willingness to criticize things that he sees as wrong makes it possible (for me, and apparently you) to trust his stated opinion of things he sees as right.
And I don't think sexism can be dismissed outright, if people find it safer to talk shit to a woman compared to a man.
You prefaced it with an "if", but do you think it is common that "people find it safer to talk shit to a woman compared to a man"? I'd guess that at this point most male academics would be significantly more reluctant to criticise in print the work of a female colleague than that of a fellow male. I see Gelman as trying to adhere (possibly recklessly) to a gender-blind approach to criticism, and find this refreshing.
I don't know whether they find it safer, but they certainly do it more, and are conditioned to do so from childhood.
The co-author of this study, Dana Carney, distanced herself from this study [1]. She doesn't believe all the claims the study made were true.
[1] http://www.npr.org/2016/10/01/496093672/power-poses-co-autho...
Sounds like straight up bad science to me
This is a story in which a few people look to be "right", but no one looks particularly flattering, which is also why its a good read. Your comment adds very little.
https://news.ycombinator.com/newsguidelines.html
The scandal is that Cuddy's work met the standard of the bulk of published social science until lately. She studied the same phlogiston with the same tools as her colleagues. But when the revolution that comes for them, gently, is an epistemic advance, their pain is our gain. Join the revolution, comrad Cuddy!
> But the email that Simmons and Simonsohn had sent was, in fact, ambiguous: They had explicitly told her to drop the P-curve and yet left the impression that the paper was otherwise sound. At my request, Simmons looked back at his original email. I watched as he read it over. “Oh, yeah,” he said quietly. He had a pained look on his face. “We did say to drop the graph, didn’t we?” He read it over again, then sat back. “I didn’t remember that. This may be a big misunderstanding about — that email is too polite.” Cuddy and Carney had taken their advice literally.
(The NYT begins the metaphor with "revolution", parent extends it with "violent revolution", and I switched out one form of violence with another).
In the cutthroat world of academic soft sciences you're not getting tenure unless you make a breakthru and you're allowed zero mistakes. Its up or out. She made her one mistake, and she's out. And there's 10, 50, 5000 people equally qualified without opportunity to cheer on her demise. One of hundreds or thousands of equally qualified researchers now gets a chance at tenure, good for them bad for her.
On the other hand in the commercial software development world, we usually don't blacklist software devs from the field after their first bug because 1) we're low on supplies of software devs plus or minus propaganda, at least we don't have the excess of her field 2) the experience of having a software bug makes a better dev.
You can't expect non-pathological behavior from a pathological situation. Until the supply/demand hiring issues in academia are fixed, stories about academia are going to read like an insane elementary school playground version of Lord of the Flies. You can't produce 10 to 100 times as much product (new grads) as the market can adsorb without behavior for the few good positions being viciously cutthroat and extremely dramatic.
So I'll give my opinion as a datascientist: null hypothesis significance testing is broken, the whole thing needs to get chucked. It's not fixable, and it's not only p-hacking that's the problem. Go read Frank Harrell, I like his writings on the topic more than Gelman's.
It's OK to be oppositional or confrontational to people based on what they say and do, but self-styled reformists sometimes become personally vindictive in pursuit of their own interests. Picking a fight with a public figure is an easy way to gain publicity, and if your own reputation rests on winning that fight then it can be tempting to carry on into harassment. Gamergate was an extreme example of this.
I don't think Cuddy, or anyone else, deserves to be dragged through the mud because their idea turns out not to hold up. That happens all the time, and it's not a bad thing at all.
You start to deserve that dragging more when you become too attached to your pet theory and cease to be able to drop it in the face of overwhelming evidence to the contrary.
You really deserve it when you use what political capital you've acquired to stomp on your critics despite the fact that they're actually right and you're actually wrong.
What's interesting here is that a lot of the Sayre's Law-type political machinations are taking place on a much broader stage than normal. If someone was to go after my work with this kind of exposure, you bet I'd be defensive too. It's entirely possible to have a fair and neutral conversation about research practices without shanking someone's reputation because it seems like the field needs a sacrificial lamb this week. Nobody wins, and certainly no good research is advanced, when the knives come out like this with no evidence of actual malfeasance. (And, frankly, even when there is evidence of malfeasance. like with the STAP controversy, which lead to at least one actual death.)
In their "open peer review", the psychologists seemed to have all the class of a bunch of redditors. An uncomfortable process is made worse! Instead of motivating people to publish lame results for tenure, this new situation seems to motivate people to act like Internet trolls or tabloid journalists.
A more critical research community should help advance the field, but these researchers set a bad example.
Look at the reach of people like Malcolm Gladwell and stuff like his books and podcast, and NPR's Hidden Brain, which subsist on recounting these "surprising studies" and speculating about their profound meaning for society, to gigantic audiences. And they're wrong! Made up ideological BS with the gravitas of 'serious science' behind them.
Power posing, stereotype threat, the bystander effect, implicit bias. So many popular hypotheses have snuck into the popular culture and they are wrong. They're based on fudging numbers and statistics and throwing out results that don't fit the hypothesis. They deserve to be combated in the public sphere. They don't deserve to be shielded to protect the careers of the hucksters.