Kind of fitting but I wonder if last few lines will turn out to fit too: "The President say ... isn't it a shame what the river has done to this poor cracker's land."
Not sure who this guy is but he seems to have completely missed the point about what Fiske was saying. Sure the internet has heralded a new paradigm but that doesn't mean that we have to just accept the negatives. And the negatives with the internet are just as severe as the positives.
Go spend time on any politics forum for example and it is downright scary. Conspiracies run rampant and are unchecked, experts are personally vilified and viciously attacked and facts come second to feelings and opinions. And all evidence to date suggests that this is not helping society but actually making it more polarised and less cohesive.
And exactly the same thing has happened in other fields e.g. climate change. Legitimate criticism is always useful and welcomed but it needs to be constructive.
I think Fiske met that constructive criticism with negative hyperbole and vague counter arguments that were not on point. People protect their vested interests and status quo by praying on people's decency and politeness, but ignore the fundamental facts in the criticism brought against them, and that leads to a proportional negative reaction from the other side. This situation reminds me of the Eastern European political systems where failure to recognise and deal with basic problems like fairness, corruption, nepotism, etc leads to a progressive degradation of trust that in turn leads to more violent/drastic changes, with total disregard for decency. So, if you want to be treated decently, than you'd better act decently as well, and not only superficially.
That's interesting, but maybe not so surprising, because the techniques that people like Fiske use stem (at least in parts) from the communist tradition of undermining and taking over institutions (e.g. Trotsky's entryism).
(e.g. Trotsky's entryism) - known in modern times as diversity advocates. Let me in your group. Then change your group culture to suit me instead of me blending in.
Trotsky's entryism was pretty specific, I don't think particularly related to your beefs. His argument was that forming small far-left workers' parties didn't make sense if there already existed large mass workers' parties, and instead communists should work within those, even if they were more moderate than you'd like, rather than forming new parties, which would be ideologically pure but not contain the actual masses of the working class who would have to form the basis of any workers' government. In most Western European countries, the majority of the working class supported various kinds of Labour and Social Democratic parties, so that's what he advocated joining.
I think the idea isn't to listen to everyone on the Internet but to take seriously the conversations scientists have on their blogs and social media accounts.
Your comment seems to me to go off into a completely different direction, one far away from saying anything concrete about actual research and going towards a very fuzzy general direction. This is completely different from the article, which has a lot of very concrete points about a concrete subject. I see a big mismatch between the article you criticize and how you do it. "Strawman" comes to mind too, most of your comment is about things that you yourself introduce into the debate.
Sadly, experts are being vilified because the perception of their accuracy has been damaged and expectations about the certainty of results have been poorly managed.
Blind respect got us where we are. Now we have to find a different way.
I also find this quote remarkable, and not in a good way; but I recall that Kahneman's studies do replicate pretty well. (This is not to say that all their claimed implications are true - this is the part where "no choice but to accept" rubs me the wrong way.)
The full no choice but to accept quote was specifically about priming studies, IIRC, and I guess what irked Gelman (a statistician) the most was that Kahneman got statistics wrong by overestimating the strength of evidence.
Reading Susan Fiske's Wikipedia article makes this even more depressing. But please resist the urge to write off all of psychology, or even all of social psychology. Psychophysics and psychometrics both replicate very, very well, and they're not the only areas of psychology to do so. This is true even if you file biological psychology and neuroscience somewhere other than psychology. And social psychology makes sociology look really rigorous.
https://en.m.wikipedia.org/wiki/Susan_Fiske
A recent quantitative analysis identifies her [Susan Fiske] as the 22nd most eminent researcher in the modern era of psychology (12th among living researchers, 2nd among women).
^ Diener, E., Oishi, S., & Park, J. (in press). An incomplete list of eminent psychologists in the modern era. Archives of Scientific Psychology
Fiske's Wikipedia article reads like PR material. It should be edited to be more somber, and reflect the extant criticisms of her research performance. I suspect that that would lead to an edit war though.
Psychophysics and psychometrics
both replicate very, very well,
It might be good if researchers in those sub-fields lead the drive of improving psychology's research methodology.
From Wikipedia: "Her four most well-known contributions to the field of psychology are the stereotype content model, ambivalent sexism theory, the continuum model of impression formation, and the power-as-control theory."
"Recently, Fiske has been involved in the field of social cognitive neuroscience. This emerging field examines how neural systems are involved in social processes, such as person perception. Fiske's own work has examined neural systems involved in stereotyping, intergroup hostility, and impression formation."
I have a long-term bet with a friend on the truth behind Wiseman & Schlitz (I think I bet 1 dollar on "beyond known science" to his 50 on "methodological error", but maybe the odds were different, and the amounts were definitely more. Maybe we will try to reproduce it at home, because otherwise who knows if it will be resolved at all in our lifetime.)
TL; DR: uncurated social media is used to critisize scientific peer-reviewed publications in psychology. This hurts scientists (peer reputation / society reputation?).
IMO this is not specific to psychology: climate change, biology, computer science, physics, etc. research is exposed to the same phenomena. The hotter the topic, the bigger the flames get.
However, if publication is clear about methodology, data collection and conclusions those flames do not hurt -- the published work stands on its own and anyone can go back to it as a sanity check on whether accusations have merit. If I disagree on methodology I will not think worse about the author. However, if I suspect he manipulated the data I will; and real quick.
Being clear on what was done is, IMO, a practically foolproof way of protecting your reputation. Researchers unnecessarily making publications readable to only a tiny minority of field experts (sometimes hoping to minimize criticism -- "I just need this published; and 3 more for a postdoc") or being vague about how they collected data shoot themselves in the foot. And this, IMO, is a major source of self inflicted wounds that are most frequent in the fuzzier fields like psychology.
I would claim it is: My only reason to make it was to prevent people from going away with that TL;DR. I don't want to make one myself, because why not just click on the link and read the article? The comment is not a summary of that article at all, that is all.
This is a complex subject and details matter very much. I don't think it is healthy to have people get only a caricature of the whole thing. I also recommend reading at least some of the many comments, and even to follow some of the links provided there. For example, I found this one in the comments and I've just reached the comment section: http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
> TL;DR. I don't want to make one myself, because why not just click on the link and read the article?
There's an essentially infinite amount of information out there on the internet, and people have a finite amount of free time; thus, many of us try to glean the thesis of an article before committing an hour to reading it -- particularly slow readers and non-native English speakers.
Especially in articles like this one, with a vague headline, no pull quotes, section titles that are too symbolic to actually describe the sections they head, and a break from the convention of using the opening paragraphs to provide a summary of what's to come, a TLDR comment is greatly appreciated to give a potential reader at least some clue what it's about.
TLDR: You oppose TLDR comments because you feel they keep people from reading the full article, but in cases like this, the lack of one is doing so.
A very broad TLDR would be that the article takes an article with the parent commentator's TLDR, and refutes it while providing insights into why the the original author might have written it in the first place.
That is a TL; DR of an article to which this article is a rebuttal to. Your TL; DR thus omits literally the entire content of what the author has written.
Please read the article through before you make a TL; DR. It's well worth the read in any case.
This is a digression, but maybe one day someone will satisfactorily explain to me why the lovely and understandable term "summary" was replaced by the ugly and unpronounceable term "TL;DR"
Susan Fiske's name came up the other day, I wonder if the context for this essay in the article is the Statcheck bot that found errros in published papers. https://news.ycombinator.com/item?id=12643978
Apparently (I gleaned from the comments) Susan was very vocally opposed to that project and said it was an attack.
She's painting a bleak picture of widespread social attacks and commentary causing the problems in academics that I don't think matches reality, and she's also suggesting the public shouldn't comment on publicly funded research.
I don't know of a single academic that has left the field due to negative commentary coming from outside the peer review system, and I know a lot of people that have left academics and a lot of people still in academics.
Some ambitious academic peers are vicious, and people are leaving due to infighting, trouble getting funding, academic stealing, difficulty getting tenure, and general departmental and university politics. The single biggest threat to education is our funding model and the exploding costs stemming from exploding amounts of administration.
The paper review process not only isn't one of the major problems, I personally believe it's actually working better now than it ever has, and that the quality and standards are higher than they've ever been.
The comments on Statcheck were wildly in favor of having automated bots fact-checking publications. I contend that it's best use is before publication, and not after, but otherwise I agree - it's awesome.
Oh, free speech and criticism is okay as long as it's moderated and appropriate... Well played.
The work of psychology researchers influences our LAWS. They influence how I can live my life. It's a very personal thing. I will criticise them however I want and as much as I want until this branch of 'science' has cleaned up its conduct.
If your work cannot withstand criticism, it's worthless. Get out of science, Susan T. Fiske, you give it a bad name. Or rather, what you were doing was probably never science in the first place. Your attitude fills me with disgust.
This is not a Russel's Paradox situation that can be patched up. The crisis in social 'sciences' is of such a magnitude that the only reasonable course of action is "start over" in many cases.
> The work of psychology researchers influences our LAWS. They influence how I can live my life. It's a very personal thing.
While I agree with the above sentiment:
> If your work cannot withstand criticism, it's worthless. Get out of science, Susan T. Fiske, you give it a bad name.
I think the tone is very much incorrect. While researchers should be responsible for bad science, you may it sound like it was necessarily done with malicious intent - I think this attitude is unproductive for everyone involved.
While I am sympathetic to people concerned with the amount of scientific trash talk online, I have to agree with the basic thesis:
Evidence-based policy is important. The alternative is gut-feel based populist wingnuttery.
To stand against Trumpism, evidence-based policy needs to be above reproach. The correct way to be above reproach is to be flawless, not to silence informed criticism.
Yeah, and this extends even to the purest studies of nature, like mathematics, as we know from Gödel's incompleteness theorems. It's just that e.g. whether or not the Rieman Hypothesis is true doesn't influence whether the state cracks down on a certain demographic.
LOL. Reminded me of a friend who skydove/skydived. He read a magazine about the sport, which had sprinkled throughout concise descriptions of skydiving accidents.
They had one thing in common - cause of death:impact.
Gelman is being a bit modest (for him) because he only mentions in passing probably the biggest recent example of the "social media takedown of psychology research" phenomenon: he was the first person to publicly raise questions about the LaCour and Green study on changing people's support for gay marriage, and he did it in his washingtonpost.com blog five days after the study was published.
Is Fiskes article related to the replication crisis?
Remember that whole cultural appropriation thing last year? Or the safe space movements on campus? There are plenty of movements that abuse political correctness to bully anyone around. On the opposite side there are plenty of 4chan types who love to shamelessly bully anyone who dare come out with a feministic message.
I imagine academics in related fields are ripe for being targeted by Internet bullies and trolls.
Isn't it this kind of abuse that Fiske is referring to?
I don't think she's talking about personal abuse. Some people just get upset when mistakes are pointed out in public. There is a crisis in social psychology: its status as a science is crumbling. Naturally, feelings are raw; many careers are precarious.
Indeed. It seems likely that there are prominent people in these fields, with decades of research and publishing behind them, almost all of it based on bad statistical methods. Naturally, there is a certain amount of panic.
This is literally nothing to do with any of the things that you mention. Why do you think it is? Indeed, it has been discussed many times in different contexts on Gelman's blog and here on HN too, so you can just read this and the previous discussions to see that it's specifically about research and statistics methodology in psychology. The blog post itself contains plenty of links itself that you can read to find out what it's about.
" In short, Fiske doesn’t like when people use social media to publish negative comments on published research. She’s implicitly following what I’ve sometimes called the research incumbency rule: that, once an article is published in some approved venue, it should be taken as truth. "
after citing an article of Susan Fiske. The sentences in quotes are blatant lies, obvious if the cited article is read. She says, in the cited article:
" In contrast, the self-appointed destructive critic's role now includes public shaming and blaming, often implying dishonesty on the part of the target and other innuendo based on unchecked assumptions. Targets often seem to be chosen for scientifically irrelevant reasons: their contrary opinions, professional prominence or career stage vulnerability. "
First, I don't understand why rapid feedback is bad. Communication adapts to new mediums all the time, and academia goes with it. The arXiv pre-print system is a great example of a field adapting well to a new technology.
Secondly, addressing the article's main point, I'm being told that social media comments about their studies are so traumatic and detrimental that they leave their careers. That's a problem with the researcher's ability to handle digital hate mail. Doctors, lawyers, dentists, and veterinarians all get massive amounts of digital hate mail.
Finally, after the failure to replicate studies, "self-appointed data police" and "methodological terrorism"[0] sound like exactly what the field needs.
[0] I have to say, only an incredibly insular community could come up with such an insensitive name. Checking your methods is not terrorism. Someone doing preliminary reviews on research data is not terrorism. To compare data critics to terrorists both insults the victims of actual terrorism, and dilutes the definition of the term so far as to be almost meaningless. They may as well call them "Methodological Nazis."
Junior Open Data Vexationer(2008-2010)
Associate Shameless Little Replication Bully (2010-2012)
Lead Methodological Harasser (2012-2015)
Senior Methodological Terrorist (2015-2016)
Chief Research Parasite (2016-Current)
All those terms have really been used to label people trying to get psych and medical research up to pretty minimal scientific standards. It is really telling about the kind of immature high school mentality that pervades those fields.
You are falling for a classic argumentative attack. Everyone agrees "abuse" (e.g. beating your wife) is bad, so Fisk just redefines "abuse" to mean "earnest, blunt, non-sugar-coated feedback" but you keep on using the connotations of the real definition.
Like wyager said, she takes a term that usually applies to real, undeniable harm then applies it without evidence of equivalence to something controversial. She uses the same word, though. This is so reducing (controversial thing here) is equivalent to reducing "abuse." This is to setup the passive-aggressive (aka SJW) defense where anyone arguing against the controversial claim must also be arguing for "abuse." They can then side-track the discussion by accusing those that disagree of not caring about stopping "abuse" or being an abuse "apologist." You can insert any negative claim related to social justice into that construct and the setup still works on many readers regardless of logical claims of their opponent.
It's a rhetorical device that relies on readers being fooled into the equivalence of what practice the sophist wants to go away and some form of harm. There's no equivalence, though. It's just rhetoric to obscure their actual argument that's probably indefensible or just weaker to rational people. The author of the counterpoint shows that nicely by illustrating what the actual critiques were with evidence, showing the author that was griping had a personal stake in that evidence not coming out, and was just using sophistry to preserve preferred status quo and her career.
Do you understand the technique now? I can provide more links to disinformation tactics in general or the ones preferred by manufactured-harm subset commonly called SJW's if you need.
I was generalizing it into one word that's a superset of all the others. Probably should've put it in parentheses or something. My bad. Here's her words that she uses to describe any critiques people have been doing:
That's just first two paragraphs. All of these are verbal equivalence to "abuse" or "harm" that gives the perception that whatever was going on is something bad with no rhyme or reason and needs to stop. Because who could possibly argue with her if they were supporting "smear tactics," "trash-talk," and some kind of "terrorism?"
In reality, a number of scientists applied the principles of fact-checking and replication to a lot of work, including hers. The work failed these. They reported failures to use scientific method. Her side is resisting. Instead of addressing methodological failures, she instead calls it all trash-talk or terrorism while saying she can't or won't give examples of either the [methodological] terrorist attacks or "victims" of that abuse. Everyone should instead just keep doing things the broken way that made her career and only question things in the channels that proliferated these problems and that people like her control. Arguing against that is supporting "trash-talk," "attacks," "smear tactics," and "terrorism" coming from a "dangerous minority." Typical, SJW sophistry.
wyager actually introduced the word. You then said that Fiske used the term. I don't see the term anywhere in Gelman's article.
In fact, Gelman uses the term in a comment on the linked page:
>Frederic:
>I’m as bothered by anyone by trolls etc., and I’d’ve had no problem if Fiske had written an article about trolls, abuse of communication channels, etc. (ideally with some examples). But this has nothing to do with replication. These are two unrelated topics! What Fiske seems to be doing is conflating the replication movement, which she doesn’t like, with all sorts of “terroristic” behaviors which none of us like. I’d prefer for Fiske to write two articles, then I could say I agree with her article about bad behavior and I disagree with her argument about scientific criticism.
Gelman has lots of valid points. I just didn't see any real discussion on abuse...anywhere...and was wondering what was being discussed in regards to using and redefining the word.
I don't understand what methodological terrorism is, but I do see ad hominem arguments use. The words really should be defined somewhere for use to read, if they are going to be published. Also, examples of exactly what she is talking about would be really useful.
[We are in the thread about people just making shit up and publishing it, right?]
"I just didn't see any real discussion on abuse...anywhere...and was wondering what was being discussed in regards to using and redefining the word."
"[We are in the thread about people just making shit up and publishing it, right?]"
I already corrected that and you're still focused on the use of the word instead of the tactics we're saying Fiske (and others) used. The word was a tiny point in that which I already owned up to as a mistake or missing clarification.
What's your position on the characterization by Fiske of anyone that disagrees with her or doesn't use channels people like her control to post criticisms as (all the negative connotations like "smear tactics" or "terrorists" here)? And do you agree that scientists should only be allowed to do what established names in academia say (status quo) and be automatically labeled as supporting the same, negative things for any form of dissent? Or publish dissent however they choose so long as there's evidence like in the counterpoint?
I think people should be able to publish whatever they'd like. Honestly, I don't fully follow Fiske's argument. There are too many undefined phrases and an explicit unwillingness to use any examples.
I'd believe many engage in ad hominem arguments and abuse. I'd also believe that many engage in arguments of authority. I'd even believe that possibly most people who call themselves scientists make those arguments from time to time.
The thing about ad hominem and arguments from authority is that they are extremely easy to defeat. One simply needs to point out what is happening and continue going.
But it's true, Fisk didn't redefine this particular word, just some other ones (like "terrorism"). My mistake, I shouldn't have assumed the guy above me was correct.
What about "earnest, blunt, non-sugar-coated feedback" that is useless, uninformed, and counterproductive to researchers? I was under the impression that that was the concern here.
Just because someone says something non-sugar-coated doesn't mean it has any truth or value to it. In my experience, a lot of the people who publicly congratulate themselves on never pulling any punches and telling it like it is and keeping it real, etc., aren't motivated because they feel like they have valuable feedback to offer (they usually don't), but rather it's just a way to act out and maintain a tough-guy public persona that they want everyone to believe in.
I'm 100% in favor of public peer review, I just question the value of stereotypical Linus Torvalds-style "I'm just telling it like it is, you idiot" feedback that some people fetishize in these parts. The best course of action here is by definition whatever leads to more and better research, with whatever the appropriate amount of public and private peer review is that leads to that outcome.
> Linus Torvalds-style "I'm just telling it like it is, you idiot" feedback
Torvalds never starts out that way, and if you look at any given episode where he's shown to have done this, the historical commentary will show Torvalds trying to politely educate the other party, and the other party just not getting it (often wilfully).
As for fetishising it, I've never seen anyone here on HN laud Torvalds for doing that; quite the opposite, it's only ever brought up to denigrate him. Ironically, de Raadt's behaviour around OpenBSD does get fetishised by some around here. I've never really understood why de Raadt gets a reasonable amount of respect for that behaviour whilst Torvalds is a pariah for it here on HN.
This original editorial by Fiske, especially the "methodological terrorism" soundbite, reminds me of the NEJM data sharing editorial where they described people who use datasets that others shared as "research parasites".
Both editorials are written by people in positions of academic power (Fiske is a PNAS editor - PNAS editors have a unique and often criticized ability to unilaterally review and approve publication of articles without a peer review panel; NEJM editors wield great power in medical sciences).
Both are originally directed toward a narrower audience of their journal, but taken out of the confines of their academic cloister, start to sound ridiculous in a world where the public starts to point out where their funding is coming from, and poke holes in their reasoning.
> start to sound ridiculous in a world where the public starts to point out where their funding is coming from
I have to say I disagree (did I misunderstand you?): they sound absolutely ridiculous even if your only goal is to do good science and you don't care one jot for the public or funding. That was always a key problem with research parasites and Fiske's methodological terrorism: apart from everything else, it's just bad science.
"sounds ridiculous in a world where X" does not mean "sounds ridiculous only in a world where X". It doesn't imply that the thing wouldn't be ridiculous if X wasn't the case.
"A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it."
-- Max Planck
A pithy quote, but it does not have to be. There is no reason or law that prevents us from acknowledging our failures in the past and updating our beliefs to the corrected findings of the present. Death is just the minimum bar, the floor, for progress, not the ceiling.
> There is no reason or law that prevents us from acknowledging our failures
I'd argue that there is. That law is Goodhart's law. [0]
As long as there's a benefit to having people think you're right—in terms of credit, money, power, fame, or even just the satisfaction of believing that you've solved a problem—actually being right will tend to take a backseat to appearing right.
I was a big fan of "Thinking fast, thinking slow", like pretty much everyone, until the replication crisis came along. Now I see a lot of the work in this field as a sort of scientifically informed stereotyping that might someday end up on par with phrenology, but it's still fun to read and consider as if it were true. It's really awful how pseudo-science can get so far though.
Good scientists should welcome this challenge with open arms, sunlight all of their data and code in a repository, and show how proper statistically rigorous research is conducted. That's an elevator right to the top of the field.
If I wrote code that messed up, I'd patch it. If someone else points out a bug? Even better, now I don't have to find it myself. But I would definitely not feel attacked or nor would I blame the QA engineer. It's my mistake, and so I must own it. These researchers need to do the same.
I believe full publication of code and data should be required for peer-reviewed research. If I can't look at your raw data or your algorithm, it is simply not possible to determine whether the study is correct. The social sciences must adapt to more statistically rigorous methods.
After all, the only two choices are adaptation or extinction.
This is not always possible. For example, if you are working with proprietary subscription-only datasets where the researcher has no redistribution rights.
Presumably however other researchers could subscribe to the same source... third-party proprietary datasets, if non-partisan wrt research results, are still better than first-party proprietary datasets, right?
If the community demands the ability to review code and data then the usage of closed data sets will simply decrease until they comply. We should not base our standards of research on the ease of complying. We should do what we can to produce the most accurate results.
The problem is not one of convenience, the problem is that you are depriving yourself of huge amounts of valuable (closed) data, and this may cause large gaps in what is researchable.
My point is that there are large fields where we may not get access to open source data in the foreseeable future; in the meantime, we are depriving ourselves of potentially useful or valuable knowledge. Imagine if you could reduce the number of traffic deaths by analyzing a closed data set from Lyft or Uber; is this worth doing, or should we just tell the PhD candidate not to bother?
Are you a researcher and following those principles? Because it is very easy to demand other people pay the price for what you believe are the right principles.
The reality of the way research funding and academia is set up (at least in Europe) is that the price you pay is likely your career. You can go through all the work and effort to follow those principles, and all that will happen is that the politicians will give the research funding and tenure positions to competitors that published more and told more exciting stories.
Academia and research will necessarily reflect the incentives put in place by the people with the money (i.e., politicians). And right now they are giving money for exciting stories and publication counts. That is where the change must start.
Then it should be required whenever and to the extent it is possible. And where it is not possible, we should be considerably more skeptical of claims made, because trusting a claim based on a proprietary dataset requires a hefty dollop of faith.
The problem is that a lot of studies don't suffer from this problem, and their data/code isn't open either. What we're looking at is a default-closed mindset, which needs to be inverted ASAP if these people want us to trust them in any significant capacity.
Then the provider of the subscription must be able to vouch for it or that data should not be used. The provider should also be .... sensitive to the need for that data to be used well, for any conclusion drawn from it to be reproducible.
> Good scientists should welcome this challenge with open arms
They should welcome online trolling, abuse, and personal attacks? That is what Fiske criticizes in her article, not serious criticism and debate, which she explicitly welcomes and encourages.
The Motte is that we should condemn personal attacks, trolling, and abuse. This is completely uncontroversial and nobody sane would disagree.
The Bailey is that criticism made outside of the peer review process is somehow terrorism, which is so ludicrous that ones reaction should involve spirited laughter.
That was a great read. Illustrated the problems really well. I thought the antibiotic thing was hilarious. Still need more condensed posts to drop on anyone skeptical about the SJW effect happening where lines are drawn in prejudicial ways with words that are used as weapons. Many people might not go through such a long piece.
Have you seen what passes for "online abuse" right now? The fact that Fisk earnestly uses the phrase "methodological terrorism" suggests that she is not to be trusted with negative connotations.
Everyone agrees "abuse" (connotation: wife beating) is bad, so people like Fisk just redefine "abuse" to mean "being blunt online", and people who aren't inoculated against semantic overloading techniques fall for it.
They should welcome online trolling,
abuse, and personal attacks?
Nobody should welcome it, but -- as it comes with the territory -- they need to ignore it.
She works on what is historically and cross-culturally arguably the most controversial subject, namely human sexuality. Attacks are to be expected, and the mature reaction to this is to lead by example: ignore the attacks and be as meticulously scientific as possible in the hope of inspiring others to be similarly mature. All the more so if you work in a field (psychology) that has a long history of unsupported claims and lack of methodological rigour.
The reason science works, the reason we can fly to Mars, have a supercomputer in everybody's pocket today, can treat many diseases, is because sufficiently many adhere to a rigorous scientific methodology and every scientist is extremely critical of colleagues who don't. It is incumbent
upon all scientists to hold themselves and others up to the highest standards.
Computer science grew out of worries about the lack of rigor in the foundations of mathematics,triggered by the (re-)discovery of non-Euclidian geometry. Mathematics, the most rigorous of all scientific fields, is currently beginning to move towards formally verified proofs. To be even more reliable.
I disagree; that's an excuse for abusive, unproductive behavior. In many environments, people manage to criticize ideas respectfully and productively, even on social forums like HN. Abuse is counter-productive and completely unnecessary.
> Claims of scientific misconduct (in a general sense) ought to be taken seriously by scientists.
I disagree. Not all claims should be taken seriously; most should be ignored. Claims accompanied by abuse seriously and rightly damage their own credibility; if I see abuse anywhere, I just stop reading and move on.
There are not nearly enough resources to address every random person's claim about every issue in the world. People don't read every book and website, address every conspiracy theory, or give time to every crackpot or amateur who wants to have a say. Google doesn't listen to every users' ideas about their software; the military doesn't listen to everyone's strategic recommendations; physicists don't listen to every person's theory of thermodynamics; the pilots of your airplane don't want your input on how to do their jobs. I have no interest in random people's ideas about IT; they have no idea what they are talking about.
You need to demonstrate that you are worth their time by establishing credibility. It seems silly that because people have access to a platform that amplifies their voices, they think professionals will want to hear from them.
When reasonable criticism by highly qualified statisticians is dismissed as terrorism, perhaps it is possible that abusive behavior by critics is not the issue at hand.
I think you are missing the point of both the blog post and Fiske's own article. What she labels as "personal attacks" was, in fact, legitimate scientific criticism, and she tries to dismiss it based on it not being published according to the right norms (according to her). She's trying to bring accusations of trolling and abuse, and you are taking her word for it as if that is the right description of what has been happening so far. She doesn't "explicitly welcome" serious criticism, because criticism published in scientific journals is rather seriously flawed and ineffective (too slow, too strong an incentive to be biased in favour of already published papers: see, for example, the discussions and failed replications that arose from Bem's ESP paper, that was really terrible). This is consistent with her comment about "methodological terrorism", which is as wrong-headed as can be, and I think clear evidence that it's not about "online abuse" in the usual sense.
If you read Gelman's posts (going back quite a while), for example, you'll see that the criticism that Fiske dismisses as trolling and abuse is nothing of the sort, and being published on a blog does not make it less valid.
In the end, it is part of the culture of science and part of being a good scientist that one should be willing to accept criticism, and maybe refutations, of one's own work, so trying to get above it by claiming "personal attacks" is seriously bad form.
It's practically impossible to anonymize data without throwing away the information contained in it, doubly so if you don't know at the time what questions critics will want to see answered on that data.
That's not quite true. E.g. Google's email spam models are trained without access to the raw text. In general the field of differential privacy is developing rather quickly.
One typical strategy for spam detection is to convert text to a "bag of words" representation[0]. If you take this bag of words representation and hash all the values, then rather than training words like ED, you are training on word number 19213123. The number of these hashed values is smaller than the number of words, just like a hash table, and this generally doesn't harm the accuracy of the machine learning. When you receive feedback on the classification (from people reporting spam or people marking things as not spam), you just turn the reported email into a bag of hashed words and feed that change into your model.
Because the order of the entries in the bag of words is arbitrary, and the words have been hashed, it is impossible to go back from a bag of words representation to the original email. I don't know if this is what google does, but it is pretty normal to do so.
This is actually a really tricky topic. Things that sound like they should give very good security, often don't in practice. The "hashed bag of words" technique that you describe here sounds an awful lot like some recent attempts at letting legacy systems search on encrypted data [1,2].
We took at look at this recently [3], and it turns out that mapping the word numbers back to the original words is actually a lot more doable than you'd think.
Yeah, I definitely don't think that this would give you mathematically provable security, especially if you are including n-grams which would allow you to chain together sentences combined with a language model. How much that matters in practice depends on the dimensionality reduction. In the context of google I doubt if this is even much of a specific goal, since no matter what they train on, they actually have access to the source text if they want it.
You need to think of medical privacy the way you think of browser fingerprinting; it's not necessarily any single piece of information that lets you be tracked, but rather the combination of a bunch of them -- even when "anonymized" -- adds up to enough bits to pin a consistent identity on the people involved.
And given what the adtech space has been able to figure out pretty easily in terms of tracking even people who take serious steps to avoid it, you should not think that there is any reasonable amount of "anonymization" that can make otherwise-useful medical details safe to release.
Netflix knows your approximate age, gender, and where you live based on your movie likes alone. I suspect the comment you replied to was making the point that Big Data is making privacy impossible. See here for an example: https://datafloq.com/read/re-identifying-anonymous-people-wi...
Don't you remember that crowd-sourced stock predication website that publish the anonymized data and let crowd build model without knowing which specific stock/portfolio is used?
Also, in many cases in the physical sciences, the data sets (both from experiments and simulations) may simply be too big for long term storage, let alone sharing. CERN is pushing the envelope at around 1 TB new shared data per day, but projects run by groups as small as five people can easily generate tens of terabytes in a day. E.g. with turbulence simulations or tomographic PIV or even just multiple high-speed cameras.
Conservatively, it's about $7.00 per terabyte to store and retrieve data on Amazon Glacier per month. Working with petabyte size streams and storage is a pain, but it can be done.
There's always a clever solution to anonymize any data. Post a description of the data and the problems with anonymizing it, and someone out there will always find a clever way to handle it.
You don't have to solve everything yourself. That's the whole point of a peer driven scientific community.
Inferences in large data sets are more powerful than you think. Individual people have been identified from multiple "anonymized" scrubbed datasets, and its only getting easier as the datasets get bigger and the analytics tech gets better.
You can only conclude it is clearly possible from that if you assume the USA federal government is infallible. I don't think that's the case.
The way I read it, the "Safe Harbor" part of that standard allows keeping around part of the zip code if that designates over 20,000 people.
For such a group of just over 20,000, add in birth year and sex (both allowed), and you're down to smallest groups of around 200 people. For the topic at hand (publish the data set so that critics can draw their own conclusions), often general health, education level and race, maybe even line of work (blue collar/white collar/agriculture) must be added, so that critics can check that your sample is representative.
I bet that gets you down to a single person in quite a few of those groups.
Yes, they end with "The covered entity does not have actual knowledge that the information could be used alone or in combination with other information to identify an individual who is a subject of the information.", but the earlier list doesn't make me confident that the USA federal government can make that judgment.
We learn from our mistakes, but only if we recognize that they are mistakes. Debugging is a collaborative process. If you approve some code and I find a bug in it, I’m not an adversary, I’m a collaborator. If you try to paint me as an “adversary” in order to avoid having to correct the bug, that’s your problem.
Fiske's Cognitive Miser theory: "The theory suggests that humans, valuing their mental processing resources, find different ways to save time and effort when negotiating the social world." (https://en.m.wikipedia.org/wiki/Cognitive_miser) Such an indictment that this "science" can debate these topics with a straight face.
As Feynman said: "We've learned from experience that the truth will come out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature's phenomena will agree or they'll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven't tried to be very careful in this kind of work. And it's this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in cargo cult science."
It's good that these issues finally get a bit more attention, however I fear that we're still only seeing the tip of the iceberg.
There are so many fields of science that don't care about replication at all, work with far too low numbers of subjects and don't bother about issues like publication bias. I don't think it's a stretch to say that the majority of stuff that's done at university is cargo cult science.
And don't get illuded about your own field: Pretty much all of this is also true for computer science when it comes to quantitative research. How often have I seen studies like "we have tested this with 10 users which we divided in two groups".
That's interesting. What kind of computer science research is done with control/experimental groups of users? I haven't paid any attention to academic computer science in, well, decades, but I kind of assumed it was all "we got this algorithm to do this thing," like math, with not much of a human element.
> 2011: Daryl Bem publishes his article, “Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect,” in a top journal in psychology. Not too many people thought Bem had discovered ESP but there was a general impression that his work was basically solid, and thus this was presented as a concern for pscyhology research.
> In retrospect, Bem’s paper had huge, obvious multiple comparisons problems—the editor and his four reviewers just didn’t know what to look for—but back in 2011 we weren’t so good at noticing this sort of thing.
I was a postdoc in a Psychology department when this was going on, and "obvious multiple comparisons problems" isn't a good characterization. Any competent psychology researcher in 2011 (a) understood multiple comparisons and looked for them as a matter of course (b) knew there was something wrong with Bem's paper (see the editorial disclaimer).
That is some pretty advanced statistics, not just "correct for multiple comparisons".
What was ongoing then, and continues now, is that psychology and social science in general is coming around to the realization that the tools of the past 50 years are flawed, and to correct them, they need to become better statisticians. But it isn't a matter of "take stats 101 noobs", these are people who have been doing statistical analysis routinely for years. I think there is anxiety that to really do things right you need to _primarily be_ a statistician.
So there is some defensiveness in social sciences about this, certainly not helped by the fact that every jackass on the internet whose taken an undergrad math class thinks they know better.
In the end I quit my psych research job to be a software engineer since all the stats hurt my head and I needed something less quantitative to do.
I too am coming from psychology looking to make the transition into a career in tech, and would be very interested to hear more about your experience making the transition*. But I would like to offer my experience having just got out of school.
I agree that there is absolutely a need for a transition to more advanced statistical methods in the field. In cognitive psychology at least, you are starting to see growing interest in adopting Bayesian techniques and moving away from null hypothesis significance testing. But unless you come across it on your own, the difference between Bayesian and Frequentist statistics is unlikely to be referenced until the graduate level. I believe Bayesian methods may alleviate some issues. For instance, one of the studies we were starting up towards the end of my time at the lab had an interesting property in that using Bayesian methods the study expected to do what could be thought of as corroborating or supporting the null hypothesis. If Bayesian methods start seeing wider adoption, I have to wonder how careful people will be thinking about their choice of priors, but it's a step in the right direction.
With regards to the statistics that are being taught. Quite a few of the K300 (Statistics for Psychology) courses are now being taught using R, but my own K300 course emphasized learning to do it by hand and didn't allow calculators. An interesting point brought up on a podcast I was listening to [1], was that we still teach statistical methods in the order they were developed/discovered and not the order that makes the most sense. I could see how teaching ANOVA as a special case of linear models might be less hand wavy. Interesting podcast, the professor they're interviewing is advocating a technique called structural equation modeling which I would love to find time to read up on.
However the research the lab I was with does specializes in building mathematical models of category learning, and has a relatively strong quantitative focus, so I can't say how this extends to other subfields or other universities. I no longer have the paper, but I saw a survey of psychology departments awhile ago that I believe found the number of methods courses being required in graduate programs was declining and fewer universities having researchers that specialize specifically in methodology. The paper made an interesting point that when you're specialized in methodology, you may play an important role increasing the quality of everyone else's work. However if your work specifically targets researchers, you're unlikely to see the same kind of funding or high profile journal publications seen by people in more applied areas. We can't all be like Tversky, but hopefully we'll start seeing some of that return now especially after it kind of declined around whenever psychophysics decreased in prominence.
I guess part of where the apprehension of increasing the complexity of statistical methods may be coming from is (1) that people might be worried about decreasing the accessibility of their work or (2) if you don't have a strong understanding of the math, you run the risk of pushing complexity somewhere you aren't as equipped to deal with it. With regards to (1), I know that one of our frequent collaborators had developed a quantum dynamics model of decision making that was showing impressive results characterizing the data, but I do not envy the amount of effort I'm sure he has to put into explaining the math in his papers. (2) might be addressed through more interdisciplinary collaboration, but I think you need both the support for development of methodology and adoption.
If you don't mind me asking, what area were you working in? And how did you go about transitioning to becoming a software engineer? I started out programming doing simulations like Conway's Game of Life and the like in a course that taught programming for cognitive scientists, and along the way kinda fell in love with it. When I decided to do an honors ...
Excerpt from post on Robin Hanson's blog covers this well[0]:
"In the rest of society, however, we often both try to hire people who seem to show off the highest related abilities, and we let those most prestigious people have a lot of discretion in how the job is structured. For example, we let the most prestigious doctors tell us how medicine should be run, the most prestigious lawyers tells us how law should be run, the most prestigious finance professionals tell us how the financial system should work, and the most prestigious academics tell us how to run schools and research.
This can go very wrong! Imagine that we wanted research progress, and that we let the most prestigious researchers pick research topics and methods. To show off their abilities, they may pick topics and methods that most reduce the noise in estimating abilities. For example, they may pick mathematical methods, and topics that are well suited to such methods. And many of them may crowd around the same few topics, like runners at a race. These choices would succeed in helping the most able researchers to show that they are in fact the most able. But the actual research that results might not be very useful at producing research progress.
Of course if we don’t really care about research progress, or students learning, or medical effectiveness, etc., if what we mainly care about is just affiliating with the most impressive folks, well then all this isn’t much of a problem. But if we do care about these things, then unthinkingly presuming that the most prestigious people are the best to tell us how to do things, that can go very very wrong."
The winds may have changed, but landscape erodes very slowly…
186 comments
[ 4.6 ms ] story [ 313 ms ] threadhttps://youtu.be/MGs2iLoDUYE
If you only know about him as the guy who does Disney/Pixar music you're really missing out.
Go spend time on any politics forum for example and it is downright scary. Conspiracies run rampant and are unchecked, experts are personally vilified and viciously attacked and facts come second to feelings and opinions. And all evidence to date suggests that this is not helping society but actually making it more polarised and less cohesive.
And exactly the same thing has happened in other fields e.g. climate change. Legitimate criticism is always useful and welcomed but it needs to be constructive.
I think Fiske is arguing exactly for this point. Feelings are more important than data.
> this is not helping society but actually making it more polarised and less cohesive.
Do you think stuff like "ambivalent sexism theory" is making society more cohesive?
He is not some random commentator.
Sadly, experts are being vilified because the perception of their accuracy has been damaged and expectations about the certainty of results have been poorly managed.
Blind respect got us where we are. Now we have to find a different way.
https://en.m.wikipedia.org/wiki/Susan_Fiske A recent quantitative analysis identifies her [Susan Fiske] as the 22nd most eminent researcher in the modern era of psychology (12th among living researchers, 2nd among women).
^ Diener, E., Oishi, S., & Park, J. (in press). An incomplete list of eminent psychologists in the modern era. Archives of Scientific Psychology
"Recently, Fiske has been involved in the field of social cognitive neuroscience. This emerging field examines how neural systems are involved in social processes, such as person perception. Fiske's own work has examined neural systems involved in stereotyping, intergroup hostility, and impression formation."
Neural systems.
I have a long-term bet with a friend on the truth behind Wiseman & Schlitz (I think I bet 1 dollar on "beyond known science" to his 50 on "methodological error", but maybe the odds were different, and the amounts were definitely more. Maybe we will try to reproduce it at home, because otherwise who knows if it will be resolved at all in our lifetime.)
IMO this is not specific to psychology: climate change, biology, computer science, physics, etc. research is exposed to the same phenomena. The hotter the topic, the bigger the flames get.
However, if publication is clear about methodology, data collection and conclusions those flames do not hurt -- the published work stands on its own and anyone can go back to it as a sanity check on whether accusations have merit. If I disagree on methodology I will not think worse about the author. However, if I suspect he manipulated the data I will; and real quick.
Being clear on what was done is, IMO, a practically foolproof way of protecting your reputation. Researchers unnecessarily making publications readable to only a tiny minority of field experts (sometimes hoping to minimize criticism -- "I just need this published; and 3 more for a postdoc") or being vague about how they collected data shoot themselves in the foot. And this, IMO, is a major source of self inflicted wounds that are most frequent in the fuzzier fields like psychology.
Instead of looking for an TL;DR, this time I suggest actually reading it. I would say it's well worth it.
I'm intrigued. I haven't read the whole article, but if I did and say something like you did I would explain myself.
If it's very misleading, even wrong why don't you expose your reasons?
I would claim it is: My only reason to make it was to prevent people from going away with that TL;DR. I don't want to make one myself, because why not just click on the link and read the article? The comment is not a summary of that article at all, that is all.
This is a complex subject and details matter very much. I don't think it is healthy to have people get only a caricature of the whole thing. I also recommend reading at least some of the many comments, and even to follow some of the links provided there. For example, I found this one in the comments and I've just reached the comment section: http://slatestarcodex.com/2014/04/28/the-control-group-is-ou...
There's an essentially infinite amount of information out there on the internet, and people have a finite amount of free time; thus, many of us try to glean the thesis of an article before committing an hour to reading it -- particularly slow readers and non-native English speakers.
Especially in articles like this one, with a vague headline, no pull quotes, section titles that are too symbolic to actually describe the sections they head, and a break from the convention of using the opening paragraphs to provide a summary of what's to come, a TLDR comment is greatly appreciated to give a potential reader at least some clue what it's about.
TLDR: You oppose TLDR comments because you feel they keep people from reading the full article, but in cases like this, the lack of one is doing so.
A very broad TLDR would be that the article takes an article with the parent commentator's TLDR, and refutes it while providing insights into why the the original author might have written it in the first place.
So go ahead and actually read the article!
Please read the article through before you make a TL; DR. It's well worth the read in any case.
Apparently (I gleaned from the comments) Susan was very vocally opposed to that project and said it was an attack.
She's painting a bleak picture of widespread social attacks and commentary causing the problems in academics that I don't think matches reality, and she's also suggesting the public shouldn't comment on publicly funded research.
I don't know of a single academic that has left the field due to negative commentary coming from outside the peer review system, and I know a lot of people that have left academics and a lot of people still in academics.
Some ambitious academic peers are vicious, and people are leaving due to infighting, trouble getting funding, academic stealing, difficulty getting tenure, and general departmental and university politics. The single biggest threat to education is our funding model and the exploding costs stemming from exploding amounts of administration.
The paper review process not only isn't one of the major problems, I personally believe it's actually working better now than it ever has, and that the quality and standards are higher than they've ever been.
The comments on Statcheck were wildly in favor of having automated bots fact-checking publications. I contend that it's best use is before publication, and not after, but otherwise I agree - it's awesome.
The work of psychology researchers influences our LAWS. They influence how I can live my life. It's a very personal thing. I will criticise them however I want and as much as I want until this branch of 'science' has cleaned up its conduct.
If your work cannot withstand criticism, it's worthless. Get out of science, Susan T. Fiske, you give it a bad name. Or rather, what you were doing was probably never science in the first place. Your attitude fills me with disgust.
This is not a Russel's Paradox situation that can be patched up. The crisis in social 'sciences' is of such a magnitude that the only reasonable course of action is "start over" in many cases.
While I agree with the above sentiment:
> If your work cannot withstand criticism, it's worthless. Get out of science, Susan T. Fiske, you give it a bad name.
I think the tone is very much incorrect. While researchers should be responsible for bad science, you may it sound like it was necessarily done with malicious intent - I think this attitude is unproductive for everyone involved.
Evidence-based policy is important. The alternative is gut-feel based populist wingnuttery.
To stand against Trumpism, evidence-based policy needs to be above reproach. The correct way to be above reproach is to be flawless, not to silence informed criticism.
The point being that not everything we know as true shows up in a study.
They had one thing in common - cause of death:impact.
Remember that whole cultural appropriation thing last year? Or the safe space movements on campus? There are plenty of movements that abuse political correctness to bully anyone around. On the opposite side there are plenty of 4chan types who love to shamelessly bully anyone who dare come out with a feministic message.
I imagine academics in related fields are ripe for being targeted by Internet bullies and trolls.
Isn't it this kind of abuse that Fiske is referring to?
If my code was in a similar state, I would consider my career "precarious" or "at risk" too.
" In short, Fiske doesn’t like when people use social media to publish negative comments on published research. She’s implicitly following what I’ve sometimes called the research incumbency rule: that, once an article is published in some approved venue, it should be taken as truth. "
after citing an article of Susan Fiske. The sentences in quotes are blatant lies, obvious if the cited article is read. She says, in the cited article:
" In contrast, the self-appointed destructive critic's role now includes public shaming and blaming, often implying dishonesty on the part of the target and other innuendo based on unchecked assumptions. Targets often seem to be chosen for scientifically irrelevant reasons: their contrary opinions, professional prominence or career stage vulnerability. "
Shame on you Mr. Gelman.
Secondly, addressing the article's main point, I'm being told that social media comments about their studies are so traumatic and detrimental that they leave their careers. That's a problem with the researcher's ability to handle digital hate mail. Doctors, lawyers, dentists, and veterinarians all get massive amounts of digital hate mail.
Finally, after the failure to replicate studies, "self-appointed data police" and "methodological terrorism"[0] sound like exactly what the field needs.
[0] I have to say, only an incredibly insular community could come up with such an insensitive name. Checking your methods is not terrorism. Someone doing preliminary reviews on research data is not terrorism. To compare data critics to terrorists both insults the victims of actual terrorism, and dilutes the definition of the term so far as to be almost meaningless. They may as well call them "Methodological Nazis."
However, I do like the ring of it. Instead of Data Scientist, I think my new title will be Senior Methodological Terrorist.
She doesn't say rapid feedback is bad. She says online personal attacks and abuse are bad.
It's a rhetorical device that relies on readers being fooled into the equivalence of what practice the sophist wants to go away and some form of harm. There's no equivalence, though. It's just rhetoric to obscure their actual argument that's probably indefensible or just weaker to rational people. The author of the counterpoint shows that nicely by illustrating what the actual critiques were with evidence, showing the author that was griping had a personal stake in that evidence not coming out, and was just using sophistry to preserve preferred status quo and her career.
Do you understand the technique now? I can provide more links to disinformation tactics in general or the ones preferred by manufactured-harm subset commonly called SJW's if you need.
"unfiltered trash-talk" "unmoderated attacks" "sheer adversarial viciousness" "methodological terrorism" "ad hominem smear tactics" "dangerous minority trend"
That's just first two paragraphs. All of these are verbal equivalence to "abuse" or "harm" that gives the perception that whatever was going on is something bad with no rhyme or reason and needs to stop. Because who could possibly argue with her if they were supporting "smear tactics," "trash-talk," and some kind of "terrorism?"
In reality, a number of scientists applied the principles of fact-checking and replication to a lot of work, including hers. The work failed these. They reported failures to use scientific method. Her side is resisting. Instead of addressing methodological failures, she instead calls it all trash-talk or terrorism while saying she can't or won't give examples of either the [methodological] terrorist attacks or "victims" of that abuse. Everyone should instead just keep doing things the broken way that made her career and only question things in the channels that proliferated these problems and that people like her control. Arguing against that is supporting "trash-talk," "attacks," "smear tactics," and "terrorism" coming from a "dangerous minority." Typical, SJW sophistry.
In fact, Gelman uses the term in a comment on the linked page:
>Frederic:
>I’m as bothered by anyone by trolls etc., and I’d’ve had no problem if Fiske had written an article about trolls, abuse of communication channels, etc. (ideally with some examples). But this has nothing to do with replication. These are two unrelated topics! What Fiske seems to be doing is conflating the replication movement, which she doesn’t like, with all sorts of “terroristic” behaviors which none of us like. I’d prefer for Fiske to write two articles, then I could say I agree with her article about bad behavior and I disagree with her argument about scientific criticism.
Gelman has lots of valid points. I just didn't see any real discussion on abuse...anywhere...and was wondering what was being discussed in regards to using and redefining the word.
I don't understand what methodological terrorism is, but I do see ad hominem arguments use. The words really should be defined somewhere for use to read, if they are going to be published. Also, examples of exactly what she is talking about would be really useful.
[We are in the thread about people just making shit up and publishing it, right?]
"[We are in the thread about people just making shit up and publishing it, right?]"
I already corrected that and you're still focused on the use of the word instead of the tactics we're saying Fiske (and others) used. The word was a tiny point in that which I already owned up to as a mistake or missing clarification.
What's your position on the characterization by Fiske of anyone that disagrees with her or doesn't use channels people like her control to post criticisms as (all the negative connotations like "smear tactics" or "terrorists" here)? And do you agree that scientists should only be allowed to do what established names in academia say (status quo) and be automatically labeled as supporting the same, negative things for any form of dissent? Or publish dissent however they choose so long as there's evidence like in the counterpoint?
I'd believe many engage in ad hominem arguments and abuse. I'd also believe that many engage in arguments of authority. I'd even believe that possibly most people who call themselves scientists make those arguments from time to time.
The thing about ad hominem and arguments from authority is that they are extremely easy to defeat. One simply needs to point out what is happening and continue going.
No I didn't, the person above me did.
But it's true, Fisk didn't redefine this particular word, just some other ones (like "terrorism"). My mistake, I shouldn't have assumed the guy above me was correct.
Just because someone says something non-sugar-coated doesn't mean it has any truth or value to it. In my experience, a lot of the people who publicly congratulate themselves on never pulling any punches and telling it like it is and keeping it real, etc., aren't motivated because they feel like they have valuable feedback to offer (they usually don't), but rather it's just a way to act out and maintain a tough-guy public persona that they want everyone to believe in.
I'm 100% in favor of public peer review, I just question the value of stereotypical Linus Torvalds-style "I'm just telling it like it is, you idiot" feedback that some people fetishize in these parts. The best course of action here is by definition whatever leads to more and better research, with whatever the appropriate amount of public and private peer review is that leads to that outcome.
Torvalds never starts out that way, and if you look at any given episode where he's shown to have done this, the historical commentary will show Torvalds trying to politely educate the other party, and the other party just not getting it (often wilfully).
As for fetishising it, I've never seen anyone here on HN laud Torvalds for doing that; quite the opposite, it's only ever brought up to denigrate him. Ironically, de Raadt's behaviour around OpenBSD does get fetishised by some around here. I've never really understood why de Raadt gets a reasonable amount of respect for that behaviour whilst Torvalds is a pariah for it here on HN.
Both editorials are written by people in positions of academic power (Fiske is a PNAS editor - PNAS editors have a unique and often criticized ability to unilaterally review and approve publication of articles without a peer review panel; NEJM editors wield great power in medical sciences).
Both are originally directed toward a narrower audience of their journal, but taken out of the confines of their academic cloister, start to sound ridiculous in a world where the public starts to point out where their funding is coming from, and poke holes in their reasoning.
I have to say I disagree (did I misunderstand you?): they sound absolutely ridiculous even if your only goal is to do good science and you don't care one jot for the public or funding. That was always a key problem with research parasites and Fiske's methodological terrorism: apart from everything else, it's just bad science.
I'd argue that there is. That law is Goodhart's law. [0]
As long as there's a benefit to having people think you're right—in terms of credit, money, power, fame, or even just the satisfaction of believing that you've solved a problem—actually being right will tend to take a backseat to appearing right.
[0] https://en.wikipedia.org/wiki/Goodhart%27s_law
If I wrote code that messed up, I'd patch it. If someone else points out a bug? Even better, now I don't have to find it myself. But I would definitely not feel attacked or nor would I blame the QA engineer. It's my mistake, and so I must own it. These researchers need to do the same.
I believe full publication of code and data should be required for peer-reviewed research. If I can't look at your raw data or your algorithm, it is simply not possible to determine whether the study is correct. The social sciences must adapt to more statistically rigorous methods.
After all, the only two choices are adaptation or extinction.
This is not always possible. For example, if you are working with proprietary subscription-only datasets where the researcher has no redistribution rights.
If you take your suggested easy route then this day may never come.
The reality of the way research funding and academia is set up (at least in Europe) is that the price you pay is likely your career. You can go through all the work and effort to follow those principles, and all that will happen is that the politicians will give the research funding and tenure positions to competitors that published more and told more exciting stories.
Academia and research will necessarily reflect the incentives put in place by the people with the money (i.e., politicians). And right now they are giving money for exciting stories and publication counts. That is where the change must start.
Someone could set up a Wikileaks type site for scientific and medical datasets and then what the law says becomes irrelevant.
They should welcome online trolling, abuse, and personal attacks? That is what Fiske criticizes in her article, not serious criticism and debate, which she explicitly welcomes and encourages.
http://slatestarcodex.com/2014/07/07/social-justice-and-word...
The Motte is that we should condemn personal attacks, trolling, and abuse. This is completely uncontroversial and nobody sane would disagree.
The Bailey is that criticism made outside of the peer review process is somehow terrorism, which is so ludicrous that ones reaction should involve spirited laughter.
Everyone agrees "abuse" (connotation: wife beating) is bad, so people like Fisk just redefine "abuse" to mean "being blunt online", and people who aren't inoculated against semantic overloading techniques fall for it.
She works on what is historically and cross-culturally arguably the most controversial subject, namely human sexuality. Attacks are to be expected, and the mature reaction to this is to lead by example: ignore the attacks and be as meticulously scientific as possible in the hope of inspiring others to be similarly mature. All the more so if you work in a field (psychology) that has a long history of unsupported claims and lack of methodological rigour.
The reason science works, the reason we can fly to Mars, have a supercomputer in everybody's pocket today, can treat many diseases, is because sufficiently many adhere to a rigorous scientific methodology and every scientist is extremely critical of colleagues who don't. It is incumbent upon all scientists to hold themselves and others up to the highest standards.
Computer science grew out of worries about the lack of rigor in the foundations of mathematics,triggered by the (re-)discovery of non-Euclidian geometry. Mathematics, the most rigorous of all scientific fields, is currently beginning to move towards formally verified proofs. To be even more reliable.
I disagree; that's an excuse for abusive, unproductive behavior. In many environments, people manage to criticize ideas respectfully and productively, even on social forums like HN. Abuse is counter-productive and completely unnecessary.
It's not relevant that some manage to discuss sexuality respectfully and productively. Most can't, as of October 2016.
Claims of scientific misconduct (in a general sense) ought to be taken seriously by scientists. There is nothing else to say.
What alternative do you suggest? Somebody said mean things on Twitter, hence psychologists like Fiske should continue as before?
I disagree. Not all claims should be taken seriously; most should be ignored. Claims accompanied by abuse seriously and rightly damage their own credibility; if I see abuse anywhere, I just stop reading and move on.
There are not nearly enough resources to address every random person's claim about every issue in the world. People don't read every book and website, address every conspiracy theory, or give time to every crackpot or amateur who wants to have a say. Google doesn't listen to every users' ideas about their software; the military doesn't listen to everyone's strategic recommendations; physicists don't listen to every person's theory of thermodynamics; the pilots of your airplane don't want your input on how to do their jobs. I have no interest in random people's ideas about IT; they have no idea what they are talking about.
You need to demonstrate that you are worth their time by establishing credibility. It seems silly that because people have access to a platform that amplifies their voices, they think professionals will want to hear from them.
If you read Gelman's posts (going back quite a while), for example, you'll see that the criticism that Fiske dismisses as trolling and abuse is nothing of the sort, and being published on a blog does not make it less valid.
In the end, it is part of the culture of science and part of being a good scientist that one should be willing to accept criticism, and maybe refutations, of one's own work, so trying to get above it by claiming "personal attacks" is seriously bad form.
P.S. There was another example of this kind of an "attack" when somebody wrote a bot for finding statistics errors: https://news.ycombinator.com/item?id=12643978
Good idea where possible, but "all their data" will be problematic in many social science studies, and also in medicine, because of privacy laws.
Because the order of the entries in the bag of words is arbitrary, and the words have been hashed, it is impossible to go back from a bag of words representation to the original email. I don't know if this is what google does, but it is pretty normal to do so.
0: https://en.wikipedia.org/wiki/Bag-of-words_model
We took at look at this recently [3], and it turns out that mapping the word numbers back to the original words is actually a lot more doable than you'd think.
[1] ShadowCrypt: Encrypted Web Applications for Everyone http://dl.acm.org/citation.cfm?doid=2660267.2660326
[2] Mimesis Aegis: A Mimicry Privacy Shield–A System’s Approach to Data Privacy on Public Cloud https://www.usenix.org/conference/usenixsecurity14/technical...
[3] The Shadow Nemesis: Inference Attacks on Efficiently Deployable, Efficiently Searchable Encryption https://www.sigsac.org/ccs/CCS2016/agenda/
And given what the adtech space has been able to figure out pretty easily in terms of tracking even people who take serious steps to avoid it, you should not think that there is any reasonable amount of "anonymization" that can make otherwise-useful medical details safe to release.
https://aws.amazon.com/glacier/pricing/
No it can't. Some data can be anonymized, some can't.
You don't have to solve everything yourself. That's the whole point of a peer driven scientific community.
http://www.hhs.gov/hipaa/for-professionals/privacy/special-t...
The way I read it, the "Safe Harbor" part of that standard allows keeping around part of the zip code if that designates over 20,000 people.
For such a group of just over 20,000, add in birth year and sex (both allowed), and you're down to smallest groups of around 200 people. For the topic at hand (publish the data set so that critics can draw their own conclusions), often general health, education level and race, maybe even line of work (blue collar/white collar/agriculture) must be added, so that critics can check that your sample is representative.
I bet that gets you down to a single person in quite a few of those groups.
Yes, they end with "The covered entity does not have actual knowledge that the information could be used alone or in combination with other information to identify an individual who is a subject of the information.", but the earlier list doesn't make me confident that the USA federal government can make that judgment.
We learn from our mistakes, but only if we recognize that they are mistakes. Debugging is a collaborative process. If you approve some code and I find a bug in it, I’m not an adversary, I’m a collaborator. If you try to paint me as an “adversary” in order to avoid having to correct the bug, that’s your problem.
As Feynman said: "We've learned from experience that the truth will come out. Other experimenters will repeat your experiment and find out whether you were wrong or right. Nature's phenomena will agree or they'll disagree with your theory. And, although you may gain some temporary fame and excitement, you will not gain a good reputation as a scientist if you haven't tried to be very careful in this kind of work. And it's this type of integrity, this kind of care not to fool yourself, that is missing to a large extent in much of the research in cargo cult science."
And don't get illuded about your own field: Pretty much all of this is also true for computer science when it comes to quantitative research. How often have I seen studies like "we have tested this with 10 users which we divided in two groups".
> In retrospect, Bem’s paper had huge, obvious multiple comparisons problems—the editor and his four reviewers just didn’t know what to look for—but back in 2011 we weren’t so good at noticing this sort of thing.
I was a postdoc in a Psychology department when this was going on, and "obvious multiple comparisons problems" isn't a good characterization. Any competent psychology researcher in 2011 (a) understood multiple comparisons and looked for them as a matter of course (b) knew there was something wrong with Bem's paper (see the editorial disclaimer).
Here is the main takedown of it: https://dl.dropboxusercontent.com/u/1018886/Bem6.pdf
That is some pretty advanced statistics, not just "correct for multiple comparisons".
What was ongoing then, and continues now, is that psychology and social science in general is coming around to the realization that the tools of the past 50 years are flawed, and to correct them, they need to become better statisticians. But it isn't a matter of "take stats 101 noobs", these are people who have been doing statistical analysis routinely for years. I think there is anxiety that to really do things right you need to _primarily be_ a statistician.
So there is some defensiveness in social sciences about this, certainly not helped by the fact that every jackass on the internet whose taken an undergrad math class thinks they know better.
In the end I quit my psych research job to be a software engineer since all the stats hurt my head and I needed something less quantitative to do.
I agree that there is absolutely a need for a transition to more advanced statistical methods in the field. In cognitive psychology at least, you are starting to see growing interest in adopting Bayesian techniques and moving away from null hypothesis significance testing. But unless you come across it on your own, the difference between Bayesian and Frequentist statistics is unlikely to be referenced until the graduate level. I believe Bayesian methods may alleviate some issues. For instance, one of the studies we were starting up towards the end of my time at the lab had an interesting property in that using Bayesian methods the study expected to do what could be thought of as corroborating or supporting the null hypothesis. If Bayesian methods start seeing wider adoption, I have to wonder how careful people will be thinking about their choice of priors, but it's a step in the right direction.
With regards to the statistics that are being taught. Quite a few of the K300 (Statistics for Psychology) courses are now being taught using R, but my own K300 course emphasized learning to do it by hand and didn't allow calculators. An interesting point brought up on a podcast I was listening to [1], was that we still teach statistical methods in the order they were developed/discovered and not the order that makes the most sense. I could see how teaching ANOVA as a special case of linear models might be less hand wavy. Interesting podcast, the professor they're interviewing is advocating a technique called structural equation modeling which I would love to find time to read up on.
However the research the lab I was with does specializes in building mathematical models of category learning, and has a relatively strong quantitative focus, so I can't say how this extends to other subfields or other universities. I no longer have the paper, but I saw a survey of psychology departments awhile ago that I believe found the number of methods courses being required in graduate programs was declining and fewer universities having researchers that specialize specifically in methodology. The paper made an interesting point that when you're specialized in methodology, you may play an important role increasing the quality of everyone else's work. However if your work specifically targets researchers, you're unlikely to see the same kind of funding or high profile journal publications seen by people in more applied areas. We can't all be like Tversky, but hopefully we'll start seeing some of that return now especially after it kind of declined around whenever psychophysics decreased in prominence.
I guess part of where the apprehension of increasing the complexity of statistical methods may be coming from is (1) that people might be worried about decreasing the accessibility of their work or (2) if you don't have a strong understanding of the math, you run the risk of pushing complexity somewhere you aren't as equipped to deal with it. With regards to (1), I know that one of our frequent collaborators had developed a quantum dynamics model of decision making that was showing impressive results characterizing the data, but I do not envy the amount of effort I'm sure he has to put into explaining the math in his papers. (2) might be addressed through more interdisciplinary collaboration, but I think you need both the support for development of methodology and adoption.
If you don't mind me asking, what area were you working in? And how did you go about transitioning to becoming a software engineer? I started out programming doing simulations like Conway's Game of Life and the like in a course that taught programming for cognitive scientists, and along the way kinda fell in love with it. When I decided to do an honors ...
"In the rest of society, however, we often both try to hire people who seem to show off the highest related abilities, and we let those most prestigious people have a lot of discretion in how the job is structured. For example, we let the most prestigious doctors tell us how medicine should be run, the most prestigious lawyers tells us how law should be run, the most prestigious finance professionals tell us how the financial system should work, and the most prestigious academics tell us how to run schools and research.
This can go very wrong! Imagine that we wanted research progress, and that we let the most prestigious researchers pick research topics and methods. To show off their abilities, they may pick topics and methods that most reduce the noise in estimating abilities. For example, they may pick mathematical methods, and topics that are well suited to such methods. And many of them may crowd around the same few topics, like runners at a race. These choices would succeed in helping the most able researchers to show that they are in fact the most able. But the actual research that results might not be very useful at producing research progress.
Of course if we don’t really care about research progress, or students learning, or medical effectiveness, etc., if what we mainly care about is just affiliating with the most impressive folks, well then all this isn’t much of a problem. But if we do care about these things, then unthinkingly presuming that the most prestigious people are the best to tell us how to do things, that can go very very wrong."
The winds may have changed, but landscape erodes very slowly…
[0] http://www.overcomingbias.com/2016/06/beware-prestige-based-...