Past performance is a reasonable indicator in a lot of situations. I've had this discussion in the context of conference submittals. While there are lots of reasons to be inclusive of new presenters, the reality is that I also want to be aware as a conference committee member of applicants who have been very popular in the past.
If you want a review to be truly double blind, you have to start censoring the paper -- things like "in previous work [1-10] we showed..." makes it obvious who the authors are even if you remove the names from the top of the paper.
You do not have to censor: you simply refer to yourselves in the third person to make your shared identity plausibly deniable. A work that is worthy of being of publication should largely be able to stand on its own anyway. This may have the effect of reducing minimum publishable unit (MPU) CV spam papers as a bonus.
I'm talking about editors censoring papers. Authors could avoid this problem by writing differently, but as a reader I'd prefer they don't; knowing that a body of work is closely related is very useful, especially when it comes to deciding which references to follow up.
The double blind process only requires that this be done during the review drafts. The final camera-ready version for publication can use the first person without compromising the integrity of the review process. I don't think the editors would remove self-referential text from a final draft after the paper had been accepted.
Usually if you count who has more citations you get the author of the paper (or the advisor, or the leader of the team). Sometimes it's a lifehack to get more citations, but most of the times is just natural because someone of the team was working in tool X and other member of the team in tool Y and now you are adjusting all the details to make X and Y work together and get a new result.
There are some computer science conferences that do this. If you cite your prior work, you are supposed to blank out the names in the citation and then attach an anonomyzed version of the prior work so that the reviewers can reference it if necessary. If the paper is accepted, you de-anonomyze the citation in the camera ready version.
"This work is based on the ideas of [1-10] which showed..."
(Also, it is a major red flag to cite 10 papers from one group and no related work from anyone else. Either the topic is completely irrelevant, or you didn't do a cursory literature search - at least skim the references from papers you cite!)
That's usually a consequence of too much nepotism, or some super expensive piece of equipment.
Double blind helps the former, though not the latter.
In a part of my field that is capital intensive, some well funded newcomers have recently invested a lot, and "broke in", while some incumbent testbeds went away; the "expensive equipment" problem is usually temporary if the field is expanding.
It's just that experiments in astroparticle physics are expensive and have lots of people on them. Like, how is the IceCube Collaboration supposed to write an anonymous paper? Even the most cursory description of the detector would give it away...
Well, maybe there will be many IceCube's later? (Probably not, but I'm routinely shocked when I learn about new giant testbeds in my field...)
Alternatively, when IceCube 2 comes out, the old IceCube crowd might be focusing on other stuff, and not paying attention to the IceCube 2 politics. That makes them great peer reviewers (no horse in the race, but knowledgeable).
The proposed IceCube Gen2 is mostly a superset and only slightly disjoint with IceCube (for example, I am not an IceCube collaborator but I am on Gen2...). But the point is that for experiments that are larger than a few PI's, anonymity of authors is basically impossible (since all papers have everyone on them).
Jobseekers with Anglo-Saxon, easy to pronounce and common names are the most likely to get to the interview stage compared to candidates with unfamiliar names, according to research by the Australian National University published in the Oxford Bulletin of Economics and Statistics.
I'd love to see this tested. Identical name; unique first name identical only; last name...it would be funny to see the results if this research is any indication.
The data in a paper can be objectively measured, the methodology used to conduct a study - like sample size, cohort composition, cohort groups chosen for comparison - could go both ways, the conclusions drawn are often subjective. If two of the three component parts of a paper are arguably subjective, not surprising that Nobel Prize winners get a pass on the quality of their papers.
The data is objective but the quality of the research is subjective. For example if my sample size was 150 for some study that's subjective whether it was sufficient of size
There are objective measures of statistical power, given an effect size a priori ahead of time you can estimate the power of a particular procedure. The trouble is that what a "reasonable" effect size may be is subjective and requires prior knowledge; post-hoc power calculations are widely regarded as misleading and conveying little additional information beyond a p-value.
Peer review doesn't verify the correctness of data. At best they'll flag flagrantly fabricated data that doesn't pass the sniff test, but attempted replication of the results is not what peer reviewers are doing.
Honestly, this is the least of academia's problems.
I've seen it from the inside. Probably read 80-100 widely-cited papers during my PhD (before dropping out), and maybe half a dozen of them were written by people who had any interest in discovering truth and pushing mankind forward.
Seriously cannot overstate both the willful ignorance of established scientists, and the extent to which this is enforced onto the next generation.
It seems to me that some subjects are so specialized that the group that makes up one's 'peers' is fairly small. How do they have real anonymized reviews when it becomes easy to recognize the writings of the author? The more papers that someone writes, the easier it would be for their peers to recognize the writing style and other quirks that would give the author away.
It might happen unconsciously. But in my experience it's not that easy for a reviewer to consciously use writing cues for anything useful.
Sometimes it seems very obvious that a given paper is by specific authors (either because of style, or because of how familiar it is with their previous work) but I've had many experiences where I later learned that my supposition was completely wrong. Similarly, when you encounter a paper that doesn't have any obvious cues (which is the overwhelming majority of them) then it's pretty much impossible to tell whether it's an author you admire or someone you've never heard of -- and this is a good thing.
Some conferences don't use blind submissions, and yes: I have felt an awful lot of influence there. "Surely [famous Turing-award winning authors] don't need me to double-check their proof."
This happens all the time in the field I'm in (phages, phage therapy) and the same names appear over and over. Peer review are anonymous but you can easily tell who's reviewed your paper or grant.
I'm not sure how to fix this problem though. In our phage newsletter we try to avoid using names and universities to focus on the paper/topic/finding itself, but I keep finding myself looking at author names and affiliations before diving into any paper.
I know it's "wrong" and I recognize myself doing it, but I still do it all the time.
I have a friend who is somewhat famous in his little research niche. Even when submitting papers for blind review he will do things like deliberately spell some words British and others American to make it harder for peers to figure out who the paper is from. The peer group is indeed small and rivalries happen quickly. Sometimes you know what other labs are doing very similar research and it's a race to publish first.
> "he will do things like deliberately spell some words British and others American to make it harder for peers to figure out who the paper is from."
This is a good initiative, but a catch is that if he's the only person deliberately spelling some words British and others American, the spelling choice becomes a unique identifier.
Though, it could work as long as no one within the group knows who is using the varied spelling.
This reminds me of a bias I saw on Quora, where some popular contributor gets tons of upvotes by fans regardless of whether their answer is any good or not (often you have to dig deep into the comments to find out that it's not a good answer). (I should qualify that I don't know if it's still that way; I started using it when it was first launched and then quit 6-7 years ago once it started turning into a fanboi and you-upvote-me-I-upvote-you club.)
Well, that's because Quora has distribution mechanics like the home feed and email digests that can amplify content by popular contributors. It's not like Reddit or HN or StackExchange where answers/comments compete directly against each other on the question page. I don't think the situation is analogous.
It's probably inherent to any site that allows you to follow individuals. For Twitter that's not much of a problem because most people know better than to rely on Twitter for life advice or product reviews. Letterboxd, on the other hand, manifests exactly the problem you're talking about, and the whole point of the site is to assign purported ratings to film and TV shows.
Reddit and IMDb are mostly immune. Which is not to say that they're immune to all problems (astroturfing, brigading, personal biases, etc.).
They have been _trying_ to make it more user oriented for a while but even in the new design, users aren't really that important or recognized any more than they always have.
The main change is you can post posts to your profile rather than a particular subreddit. But that's not a big feature. Back in the day people used to just create a subreddit of their username.
It's not different from Reddit or H.N. in the sense that most people that vote never bothered to read the entire post and the same thing is true with peer review. This isn't simply the case in soft-science as the Bognanov affair showed.
They skim, see if it looks okayish and give it their approval.
Peer-reviewers also don't follow particular names around; they see the name above the submission and it influences them, as on H.N. no doubt.
But it's far worse, even without a recognized names, most votes are cast without proper reading, and I'm fairly certain also by the least intelligent subsection given how often submissions are upvoted on H.N. and Reddit that are pure clickbait and demolished in the comments by people that actually read it. People that vote by and large only read the title or the first sentence and make up their mind from there.
That’s what we want in society right? Perks for doing hard work and then being recognized for it. That’s what “street cred” is. Maintaining a reputation is hard work and the fast pass and acknowledgment from the broader society is your reward.
As in this person has more than proved himself, let’s not vet him as much.
This is literally how one of the biggest scandals in academic publishing occurred[1], so... no? Maybe it's a nice way to treat experts when asking them for their opinion, they are experts after all, but we absolutely don't want to give celebrities more leeway than unknowns in research papers where new claims are being made. You better have done the work properly, again, and again, and again. No free passes, no blind trust.
In this case it looks like his good reputation was falsified from the start? They were unable to prove misconduct in his dissertation only because the data had been destroyed. So consistently missing his misconduct would be more to blame here than riding on a reputation.
But that ignores the years prior where he had enough reputation to be scrutinized less, which lets him publish more, which made him more famous, which got him scrutinized less, allowing him to keep publishing, boosting his reputation, [etc].
Right up to the very end, this man was able to publish like a celebrity because no on questioned it because he was famous. So let's not bake that into the process: scrutinize all papers equally. Experts don't get a free pass, if they have new claims to make, those claims are just as "I don't believe you yet" as anyone else's.
No. You get gatekeeping and laziness and stale group think and unstated quid pro quo and discrimination. It's worse science.
A famous name getting the same paper published easier is a failure of peer review. The whole point of science is that ideas and evidence stands on its own merit. Not celebrity or seniority or power or any other axis that doesn't matter.
Fairness and efficiency are often at odds. As a reviewer, I was sometimes asked to review papers from unknown authors at unknown institutions. My level of effort was likely proportional to how well I knew the quality of the institution from which the paper came. There is a great deal of time involved in understanding a paper well enough to do a full review.
It's true, unfortunately. When I was a TA and graded homework I'd learn which students did well over time and generally would put less effort into checking their work because generally it was more likely to be correct. They were also more likely to self-correct, and to present work well, etc. Reviewing papers isn't really too different.
I agree. I'm in an applied computational field, that attracts a lot of computational and mathematical researchers that don't have knowledge of the applied system. I've learned from experience that often researchers new to the field make a whole lot of poor assumptions and modeling decisions, even if the mathematical part is fine. For better or worse, I'm going to more heavily scrutinize the details of a paper from a lab I haven't heard than I am one that I believe has a strong track record. That doesn't mean I'll give a pass for the latter, I'll still read it with a critical eye. But certain types of trust build up over time.
I would expect a much higher than sixfold increase in paper acceptances by winning the Nobel Prize. Why wouldn’t there be a massive lift? Even if you can’t personally see why this result is important you know that one of the biggest contributors to the field thinks it’s interesting. That’s a pretty credible signal in a noisy world.
Yet most non-economists would have predicted exactly this. If it's astonishing to you, well, perhaps you've had the misfortune to learn to think like an economist (ie. poorly & very muddy about normative vs actual).
It would be possible to respond to your comment in earnest but ad hominem like this is more of a psychological cushion than an invitation to genuine discussion.
In what world is a daft attempt to psychoanalyze your interlocutor not ad hominem?
My comment was snarky but not remotely playing the man. I meant it quite literally as a criticism of the thinking at issue. Economistic thought is riddled with confusion, and being surprised that scientists don't behave normatively (ie. not as those engaged in peer review are "supposed to") exhibits perfectly one variant of said confusion.
I'm not 'astonished' that non-blinded peer reviewers are influenced by social status. No-one I know would be surprised at all, let alone 'astonished'.
> I'm not 'astonished' that non-blinded peer reviewers are influenced by social status.
I am astonished, in the same way I would be astonished to find out that papers that smell like fish are more likely to be accepted for publication at the majority of peer reviewed venues, and that the reason is that reviewers for those publications let their house cats stack rank the submissions.
Much of academia seems to have become an outwardly transparent no hold barred status battleground. So this just seems to exactly align with my expected outcome given that assumption.
I'm not saying there aren't people in academia who aren't driven by doing good research but I certainly don't see that as the driving force in the US academia.
It is nearly impossible to enforce double blind peer review because of the existence of online pre-prints (hosted on arXiv, SSRN, or just the authors' websites). A referee can copy and paste from the text into DuckDuckGo and find the authors.
In this case an easy 50% solution seems better than nothing. Some people would just comply with the guideline, some would not think of your workaround and it would make peer reviews a little more equitable even if many broke the rule.
I think you are overestimating how much reviewers care which grad student wrote which paper, or who their advisor is.
Reviewers are generally overworked (for other reasons), and the main goal is to be constructive and somehow not sound like an idiot (in front of the other reviewers, who might be subject experts and definitely know who you are!) in any of the pile of reviews you need to write after reading a pile of papers.
Also, reviewers declare conflicts ahead of paper assignments, which makes accidents less frequent.
Besides the sibling comment, in a lot of fields, people just know which group a paper is coming from even without doing any sort of searching or looking at preprints.
Yeah it's often pretty obvious that a paper extends / is incremental on a previous body of work (maybe the same code, maybe same dataset) such that it could only have come from one lab.
Yeah, but good CS conferences have diverse reviewer pools, including ones from industry or different subdisciplines. Such reviewers often haven't heard of the lab, or paid attention to which grants are currently funded, the movement of students into postdoc positions at other labs, etc, etc.
I've seen many arguments against double-blind which were very unconvincing, and they were all from the people that already had a big name, the ones that decide how journals operate.
In any case, this is the tip of the iceberg. The entire structure surrounding academic publication is absolutely ill-designed and few people in that world are even remotely interesting in safeguarding veracity.
The overwhelming majority of peer reviewed scientific results is infotainment for other scientists with no party having any material stake in the veracity of it, which is almost no one bothers to replicate because accuracy is irrelevant so long it be interesting to read.
The moment a company has bet a sizable stake on it's accuracy, then they suddenly check, and double check and have another party independently verify it because they do not want to loose money, obviously. But most science is nothing like that.
Just to add more to the siblings. Research is inherently social. In a small field you can even guess the reviewers. Sometimes there is the "oh yeah that professor talks like that. It's definitely her writing me this review" moment, and reviews are supposed to be anonymous at least in most of the venues. Unfortunately the social dynamics and trust in some cases extend too far -- many sloppy research get published due to trust or favoritism. Double-blind makes this a bit better, but beyond this, at a certain point you will be changing what research is about from a social enterprise to something much more result oriented.
Even with double blind reviews, it’s often very clear who wrote the paper.
Authors regularly cite work by themselves or their team, so a statement like “In a previous study[1] we established a relationship between X and Y” renders double blind pointless.
That’s odd. I don’t do a lot of peer review, but when I do the authors names are masked. Sometimes I can figure it out based on the citations, but I haven’t reviewed any papers where the author’s name is listed.
That being said an author isn’t published randomly in Nature, so I expect subsequent papers from an author to be better, on average, than non-Nature published authors.
Discussed is not the same thing as reviewed though. I agree it is not odd. It is a good thing to know about though and be mindful of. It might be advantageous in closed circumstances, i.e. you can skim a review of a more trusted developer.
We yeah. At least in science, it’s a small world and reviewers often know the authors personally.
And your reputation follows you. If it’s a big-name lab who has an amazing track record, you’re going to review the paper in the context of their entire body of work.
Academia was bought and paid for long ago and the money was used to build an incredibly broken and overly political bureaucratic engine of scholarly and scientific work that doesn't get anywhere near as much peer-review scrutiny as it should and commands far more respect in politics and legal proceedings than we should allow.
Universities and experts are the best we can do sometimes so we have to rely on it, but it doesn't mean it's truth or absolute and people like to use it as if it is to sell ideas like global warming instead of educating people on climate change.
The 2 academics I know aren’t bought and paid for. Either could make far more money in the private sector than they make now. They work in their fields out of love of research.
This was my impression of most of my professors years ago too, especially as we saw so many quit for high paying engineering jobs at companies year to year — the ones who stayed aren’t in it for the money
I suspect it’s the same at the top: most senior administrators I bet would make more in senior Fortune 500 jobs
HN isn't a journal and so academic sources of evidence really are overkill. However, I'll throw you a bone. Go to your favorite major journal and start looking at the "conflicts of interest" and "grants" section.
Once you take money from someone (except for NIST, in my experience) you're basically beholden to try your hardest to get the results they're looking for. Some scientists are moral enough to still return bad results. A lot of scientists aren't. There's a lot of garbage out there, and the worse the journal, the more garbage it gets. Famously, Phillip Morris studies "passed" the scrutiny of several major journals. It's amazing what greasing a few palms will get you.
"HN isn't a journal and so academic sources of evidence really are overkill"
I disagee, you should backup your accusations or statements (unless widly accepted) with some reputable source. This person claimed that science was bought and paid for. Did they mean all of it? Or most? That's an insane accusation that requires evidence.
"Go to your favorite major journal and start looking at the "conflicts of interest" and "grants" section.
Once you take money from someone (except for NIST, in my experience) you're basically beholden to try your hardest to get the results they're looking for"
This doesn't mean people falsely data. You're simply providing motive.
Everyone wants money, you are using greed to then claim mass fraud in science
Is it really that insane of an accusation? This is how almost everything in the world works.
This isn’t only about greed either. People want their research published for reasons other than greed. For example, they want to move up in their career or achieve recognition.
After looking at a lot of medical studies related to COVID during the last couple years, I have seen first hand how biased and inaccurate many of them are. Some of these studies are even mentioned in major news outlet despite their obvious flaws when you actually begin to scrutinize them. Think big pharma providing research grants for studies that conclude their products are effective.
The OP never said that people falsify data as a result of receiving grants from interested parties. They often don’t have to. They can simply design the experiment in a way that doesn’t account for specific variables or behaviors then use the resulting data to reach a specific conclusion.
I remember seeing an article related to AI research on HN a little while ago that somewhat explained this problem. The grant money all goes to people researching deep neural networks which creates a reinforcing feedback loop. Since all the money goes to one branch of research, it creates very few opportunities to research competing ideas. I believe it was this one:
Most declare no conflicts of interest. One author of one paper seems to have started a company based on similar technology: potentially a bias, but also potentially putting one's money where their mouth is. One other author lists some consulting work for a few companies.
As for grants, I doubt people are bending their results to appease the NSF or NIH. There's certainly groupthink in what gets funded. We're still throwing money down the ABeta-for-Alzhemier's hole, for example. That eventually shapes what topics get published, but maybe not the specific results. The recent Abeta articles are pretty negative, for example.
Agree. It's stagnant, fearful and bureaucratic.
"Trust the science" never changed anyones mind.
However, I think your climate change example is a bit strange. If anything, it's big oil who's been trying to sell the idea that we don't have to stop using fossile fuels. Hiding evidence, and spreading confusion by paying lobbyists and scientists. Global warming was proved beyond reasonable doubt decades ago.
Climate change is one of the most politicized fields at the moment.
Any questioning it, even slightly means being banned from grants and academia.
Its also interesting that most climate models are NOT open source. Most recorded data from satellites is also NOT open source. So everyone works with a pre cleaned data set.
Its also worth pointing out that data sets like HadCRUT have never been audited by any respected scientist/group of scientists. This data was collected in stations not meant for long term measurements and they have a lot of errors. Just download it yourself and see.
(Climate scientists are not really data experts, since they go from clean datasets in school to "clean" data sets in real life)
Calculating global temperature is also one of those things that is done in quite an obscure way, extrapolating too much IMO.
You seem to be much more knowledgeable about this than myself. But I was under the impression that while large climate models are used to infer the extent and impact of a changing climate, they are not actually an important component in determining climate change caused by carbon emissions to be a real phenomenon.
No other hypotheses about geophysics are required to show realistic and fault free simulations of the climate of the planet over several centuries to be generally accepted. Why should we apply such extreme prejudice to the hypothesis of climate change caused by the greenhouse effect? The basic mechanisms is quite simple and well understood, there is a variety of kinds of measurements supporting the claim that the temperature of the planet is increasing (meteorological temperature measurements, glaciers disappearing etc.). Full understanding of all the geophysical processes and feedback loops involved is not necessary, and very likely impossible.
I also have a hard time understanding the motives for such an enormous scam.
Who would stand to gain from this except a relatively small number of researchers and the renewable energy industry? On the other hand, it's well documented that the fossile fuel industry has tried to sabotage climate science for the purpose of limiting political action on the issue.
> Most recorded data from satellites is also NOT open source.
This is false. NASA and ESA science data is free.
There is often an embargo period for very novel sensors, and always a delay of hours-to-weeks to allow processing to catch up, but it's free.
If it's the source code of the analysis pipeline you mean -- even though you said data - that's a harder lift, because the processing is complex. But even that is changing (https://science.nasa.gov/open-science-overview).
Even in the absence of the open science initiative above, today you can always get the raw data ("Level 1 radiances") or sometimes even uncalibrated straight-off-the-sensor data ("Level 0"), if you want to process it. (https://www.earthdata.nasa.gov/engage/open-data-services-and... -- "All EOS instruments must have Level 1 Standard Data Products (SDPs)")
And if you want to look in to how the processing works, there are detailed documents ("ATBD's") that explain how the pipeline works, for each data product. Also free.
> Climate scientists are not really data experts, ....
Dreadfully wrong. Do you work in this area at all?
Yes. In fact, people are so likely to make assumptions about future performance based on past performance that we invented a word for that social construct: "reputation."
Yes, because it's easier to do that than assess people on their merits. This is especially true when you are reviewing a technical paper (often for free) when you have competing demands on your time and mental energy (research, teaching). It takes hours and hours to properly review a paper, and that is still trusting that the author is acting in good faith - to reproduce their work might in some cases require a large amount of time and money. These are the same reasons that letters of recommendation (written by people you've heard of) are so useful in academia. Lots and lots of bias problems, but the bias isn't the point, as I think is sometimes implied.
A while back I subscribed for The Economist on a whim. I'm not really into economics, but I felt they provided a nice alternative view of what was going on elsewhere in the world compared to what the local news here in Norway reported.
One thing I quickly noticed was that the articles were never signed by the authors directly. I found this striking, but also refreshing. It caused me to pay more attention to what was said. And, as I later found out, that seems to be the main intention[1]:
The main reason for anonymity, however, is a belief that what is written is more important than who writes it. In the words of Geoffrey Crowther, our editor from 1938 to 1956, anonymity keeps the editor "not the master but the servant of something far greater than himself…it gives to the paper an astonishing momentum of thought and principle."
Norway really has a huge issue of sensationalism in local papers, and it does not help that even the government funded NRK is deep into sensationalism as well.
Opening NRK right now gives "Now the government must see the madness in this", "Electricity prices can make going to the cinema more expensive", and "He is so tired" as the top articles for today.
I follow so much International/American news thanks to publications like Reuters, AP, and Qz. But no matter how much time or money I use, I can not follow my own country's news. I definitely am not alone in this, and it terrifies me how unworried most of Norway seems to be about this fact as well.
> In 2008, in response to a series of Call-for-Paper e-mails, SCIgen was used to generate a false scientific paper titled Towards the Simulation of E-Commerce, using "Herbert Schlangemann" as the author. The article was accepted at the 2008 International Conference on Computer Science and Software Engineering (CSSE 2008), co-sponsored by the IEEE, to be held in Wuhan, China, and the author was invited to be a session chair on grounds of his fictional Curriculum Vitae.
When I was an undergrad in college, I helped design a study on the role of gender perception in expertise.
We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials.
We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's sports beforehand (women's basketball), or no sports. And finally we randomized whether the person giving the subjects instructions would be a young woman (one of my classmates) or a young man (me).
Our small study showed what you'd expect- students- men and women students rated male writers more highly than female writers, and initials were right in the middle, though they tended to be more like responses for men.
The tester's gender made no difference that we found.
The sports thing made a measurable difference, but it didn't reverse the skew.
My 18/19 year old self was somewhat skeptical we'd find a difference, but I was totally wrong. It taught me a lot about bias and perception, which this study also shows.
Not needed because "women are wonderful" is to some degree mostly orthogonal to "men are more professional/deserve more/"... wonderful good attributes are not needed or even to some degree counter to "high performance" (but depends a lot where we look at, sure!)
Also pay-gap has likely also to do with the current men-bias also? Wonder if that would be the same if we had 90% woman throughout in these decising positions..
In the end all mine assumptions, what I actually want to say is: I don't think those two contradict or relate at all.. what GP already said well with "biases swing in all sorts of directions depending on the context"
> There are two distinct numbers regarding the pay gap: non-adjusted versus adjusted pay gap. The latter typically takes into account differences in hours worked, occupations chosen, education and job experience.[1] In the United States, for example, the non-adjusted average woman's annual salary is 79% of the average man's salary, compared to 95% for the adjusted average salary.[2][3][4][5]
The remaining 5% could be from the "women are wonderful" positive attributes not being the narrow selection of ones that are highly sought after in well-renumerated jobs, or (spitballing here) psychosocial (Expectations of a pay gap driving negotiation behavior or something).
I'm not a psychologist, I have nothing but that BA in the field, but if I had to make a guess, my guess is that "wonderful" is not the same as "competent".
Even at the time, there were studies that showed traits associated with women were rated more positively by men and women than traits associated with men.
The way you study this is you'd give a list of words:
And then you'd have a group of subjects rate them as more associated with men or women on a scale.
Ideally you'd get a big sample and replicate this study.
Then either in the same study at a different time, or another study, you'd take those same words and you'd ask your subjects to rate them as positive or negative.
That's where you see effects like the one mentioned in the wikipedia article.
That kind of thing's been done a bunch of times.
But studies on gender and competency have been done too, and at the time they showed the same pattern as we found.
I say "At the time" because this is >20 years ago.
>my guess is that "wonderful" is not the same as "competent".
Right. There are two different sets of benefits that help or hurt someone in different ways. Being competent when standing trial can work against you while being wonderful will reduce the risk of a conviction and reduce the sentence if you are convicted.
We have identified the "competent" bias and are taking steps to correct it, but we need to do the same with the "wonderful" bias in other systems. For starters we need to recognize how strong that bias is in certain fields. For one example, there are specific crimes that people would bet are extremely gendered in nature, and the crime statistics show they would be right, but interviewing the population at large and querying victims, including those who never went to the police or who were turned away by the police (or even worse, who couldn't legally be victims because of how biased even the laws are), we see the gender component goes away. The rate of men victimized by women and women victimized by men are at near a 50/50 ration (I think 49.8 to 50.2).
Even the extent of studies measuring the impact of the wonderful effect is lacking compared to studies measuring the competent effect (which itself is likely a bias of the wonderful effect).
Maybe someone in your field would know, is there any difference in the actual reliability/reproducibility difference in research done by men and women? Maybe you are simply measuring a real phenomenon but in a round about way?
This was one of the few factors we kept the same- same text.
I think if it had been a larger study (not a one credit class run by 4 undergrads but an actual study with funding) we might have had multiple texts and randomized them, but we had one text- I don't even remember what the topic was.
That reminds me of a bias study where the researchers were investigating the effects of stereotypes.
They told one group of Asian women that “women are worse at math than men” and another group of Asian women that “Asians are better at math than non-Asians”. Both are common stereotypes.
They then measured how well the two groups did on subsequent math exercises.
Interestingly, they found that the latter group (positive stereotype) performed better than those in the former group (negative stereotype).
As I recall, other research found similar results with Black males and golf scores when told “white people are better at golf” as opposed to “black people are better athletes”.
Not only do stereotypes influence our perceptions of “the other”, but they influence the performance of the other.
The experiment you discuss is called "stereotype threat" and it belongs into the huge set of psychological results that cannot be replicated. [1] It seems that there is a bias against publishing studies with null results, which skews the overall picture both among the researchers and in the media.
That’s called “stereotype threat”, and it’s been caught up in the replication crisis. In short, the effect is hard to reproduce and tends to be small. It’s been known for some time now, here’s an article from 2015: https://www.psychologytoday.com/us/blog/rabble-rouser/201512...
I wish people would stop bringing up studies that don’t reproduce, they’re no better than anecdotes.
Setting this aside for a deeper read, but it appears at first glance that the concerns revolve around statistical technique rather than methodological soundness?
> Our small study showed what you'd expect- students- men and women students rated male writers more highly than female writers, and initials were right in the middle, though they tended to be more like responses for men.
Most things I read I have no idea of the sex of the author. Do people even look at author names before they start reading (online, say), and even then they can be nom de plumes, or non-gendered names (sometimes surprisingly).
We announced the name of the author. The script went something like "We're going to ask you to read this piece, written by J. Smith.".
During debriefing, we would tell the subjects what we were actually looking for (perceptions based on gender) and most reported not remembering/caring about the gender of the author, but nonetheless the results were clear that there was a gender bias, regardless of whether they reported remembering or caring about the gender of the author.
This is a really long time ago but if I remember, we ready a study about how sports effected perception. In the other study they did the male sport vs female sport thing, but not the name part, and so we just were combing two previous studies.
the sports thing seems like a weird addition. did the text have some relevance to sport? if not it seems like the kind of thing that would skew results just by reminding the participants they’re in a weird study
Would be interesting to hear from OP, but I assumed it was a reminder to check ones stereotypes -- eg 'remember women can be good at traditionally male pursuits' (would work for some sports only).
The simpler version of the study had been done before, and the class I was a study design class. Our class assignment was to design, execute, and analyze a study.
We were simply combining two existing studies into a new study.
It bears mentioning- this was ~1996/1997 (I don't remember), I was a freshman, it was a single credit supplement, and we never published the results.
If it is known that most discoveries are made by males it is reasonable to give larger weight to new male-done research, if the only thing that is known about the researchers is their gender and if the reader is not enough of an expert on the subject matter to treat it 100% on its own merit.
I don't see this "bias" as inefficient or counter-productive. It is just an artifact of the way you designed your flawed experiment.
To find any real bias you would have to assure the subjects that the researchers were equally accomplished.
An experiment like OP's would 100% show bias. There weren't multiple researchers, each reader read the exact same paper, but with a different gendered (or non-gendered) name as the author.
> To find any real bias you would have to assure the subjects that the researchers were equally accomplished.
What do you mean? I think you didn't understand the experiment. Subjects were shown the same text with, at random, a male name, female name, or initials. These names were not real researchers, nor familiar to the subjects.
> If it is known that most discoveries are made by males it is reasonable to give larger weight to new male-done research
I'm sure you can see the two different fallacies in this short sentence:
- There have been more male researchers than female, therefore more "discoveries" were made by men than women. No shit. This does not mean male researchers are better than female, obviously.
- There are more men being researchers therefore men's research should be given more weight therefore there are more men researchers therefore...
It's actually you who doesn't understand probabilities. If a woman is twice as likely to make a discovery but women are only 5% of researchers, then it's both true that:
1. most discoveries are made by males
2. it is reasonable to give larger weight to new female-done research
But you pulled those number out of thin air. The fact of the matter is that if I pick one research paper at random out of all the papers written by men and I pick another research paper at random out of all the papers written by women the former will be of higher average quality.
Even if one sex was 10x as likely as the other to produce good research, it is still sexist bias to judge research on the basis of the author's sex. Fairness and meritocracy mean everyone gets a chance; no one should be stopped from succeeding because they belong to a low-performing group.
I think it's safe to say that this study can be treated as an anecdote because we don't know the effect size or exact methodology and it was never peer-reviewed. And I'd agree with your point if we were talking about the Nobel prize study, which evaluates people as individuals. But arguing that 'real bias' is bias that doesn't come from empirical evidence doesn't make a whole lot of sense- all bias comes from empirical evidence of varying quality.
Sorry, but there are several things problematic about that study, if your description is accurate.
Just one of the issues is the specific perception regarding that particular sport (basketball) and relative perceptions between sex differentiated similar sports (men’s and women’s basketball are not the same sport, albeit similar). A sport should have been chosen where there are as few differentiating perceptions as possible, which is nearly impossible, because men are inherently more competent at sport due to physiological realities. It bakes in a bias towards men, which is what you likely actually confirmed with the study you described.
Introducing sports alone essentially corrupted that research and likely biased the responses for relative innate competency of males in sports.
For example, ignoring its own challenges, what could have been done, is show video of a relatively broadly positively evaluated sport (not basketball), an activity that is also relatively broadly and positively evaluated female clustered competencies, e.g., effective child rearing, communicating effectively, conflict resolution, creative outputs, or even beauty pageants, etc., along with the inverse, i.e., poor performance of men in the same sport and poor performance of women in whatever is chosen, e.g., screaming and yelling at poorly behaved children.
It is in the past and by no means are or were you the only one who has long engaged in this type of poorly executed “research” that has merely been confirming researcher biases and often, thereby immensely damaging society, but maybe my illustration of a few of the several issues with what you described, will cause some changes in thinking.
There is a real hidden epidemic of not only single order thinking, but what really should be called negative order thinking in research, where research is not only not finding the truth, but rather even doing damage through confidence in false findings.
I say these things with an expensive relevant background, in the trenches of bad and … frankly … destructive research, if you will.
Very common for anyone who has ever spent time in academia. When I was still in grad school I noticed a trend in some major mathematical and CS journals where a lot of time it would be the same N people writing articles. This was driven home when my advisor told me "without such and such's name this paper has no chance of making it past review".
Non-sense. However, for all the people that say "trust the science" this is the science you're working with. It's politics all the way down just like everything else. For example, if Knuth decided to post some utter garbage chances are it would accepted based on his name alone.
You trust science because it has the highest probability of being accurate. What's the alternative? Most people aren't capable of analysis scietific research to see if it's correct.
What makes something political? If new research shows that X product is dangerous and republicans, who that company donates to, come out and disagree with the research they just made it political. Now, someone like you, can just say "This research is political"
You've created a way to dismiss scientific research simply by disagreeing with it
You are completely misinterpreting the meaning of GP's comment. First, they are not talking about government politics, but rather interpersonal politics. Second, they are defending science, and saying that it shouldn't be dismissed for political reasons, rather than the opposite.
Science is a process and it is never complete. You are confusing a process with the results of the process, which are by definition imperfect. What makes Science good is that it's results are fundamentally imperfect and tainted, and that it respects that.
When you go around saying 'science is perfect' you disrespect its core principles.
There's a lot of middle ground between blindly trusting all 'science' and blatantly disregarding everything that rubs you the wrong way.
It's possible to both have a generally high amount of faith in 'the science' while still remaining critical of its flaws, or rather, the flaws of the system in which its conducted.
I don't think this is the case. The point of science as one commenter pointed out is a lack of trust (or faith). Hence the need for pages and pages of sources, concerted efforts on reproducibility, etc. In certain fields reproducibility is extremely important and a single scientific article should be taken with extreme skepticism until it's reproduced several times. I think this is the thing that's often skipped. A single article being published in a journal is not a sign of changing times. If there's no attempts to reproduce it we have an idea of something, but no idea if it works (really). Often times this lack of reproducibility is associated with so-called corporate "science" (Phillip Morris, from another example I posted).
I hesitate to use the word faith at all, even when describing the scientific process itself. Having faith in the process leads to (for lack of a better phrase) a failure to trust, but verify. IMO, it is one thing to approach scientific literature with a trusting demeanor and still walk through the proper processes to verify the results, and another thing entirely to literally take the author's word. Often times especially when big names are attached we aren't trusting-and-verifying we are simply taking their word. That's a problem that weakens the credibility of science.
I don't want to come off as someone who disregards everything he doesn't like. However, I do approach science with great pessimism and have since graduate school. Not because I hate it, or something is wrong with the scientific method, but because approaching it pessimistically allowed me to write better work by forcing me to actually perform the method instead of p-hacking my way to a result.
Trusting the outcomes produced by science is inherently faith for most people who aren't an expert in that particular field of research, and since no one is an expert in every field everyone has to take certain scientific outcomes on faith.
I want to disagree with a small part of this. Faith is the belief in something that you can't prove. That is, even if you became an expert in a field, read all the papers, etc you couldn't prove the existence of god, hence faith.
I mean it was an example I chose because the name recognition in HN. I would doubt Knuth would publish garbage. Then again, how would you know? If you've never been in academia it's understandable to think that way. But if you have, you realize the immense pressure, self-doubt, and scrutiny you'd have even making a bold, well researched, attempt to prove someone like knuth wrong even if it was obvious he was. I recall several instances in mathematics where this happened and it bordered on starting a war between mathematicians.
Rather, Knuth could publish some mediocre paper that would be instantly accepted by all the top journals, widely read, discussed and cited instead of more salient works on similar topics.
That's the problem really, excellence is rare while mediocrity is aplenty and academia is a numbers game. The mediocres with prodigious output will get the cites, tenure, TAs, lab assistants and budgets to do research and perhaps stumble onto some other mediocre result, reinforcing the cycle.
Or he would never publish something "garbage" because people are convinced by his reputation that it will not be garbage.
Emperor's new clothes, while the rest need to be whiter than white and all that. These things happen because people are blinded by titles in a complex world with complex people.
The point is that a famous name gives an edge in peer review to a (potentially mediocre) paper written primarily by somebody else with the famous name appended.
Even great mathematicians try to put out garbage unaware of their senility in their final years. I've seen it personally, and the reviewer made contact with a friend of the author, who volunteered to be a pre-filter to avoid future embarrassment
> For example, if Knuth decided to post some utter garbage chances are it would accepted based on his name alone.
You mean like Elon Musk? When he came up with that giant box to extract kids from a cave with passages so narrow rescuers couldn't wear their air tanks on them, they had to push them ahead of themselves, reddit and HN and twitter were full of people defending him. There were even people defending him when he accused the rescue leader of being a pedo - simply because he was british and living in thailand.
Or how about the CEO of Brave, who apparently is an expert in public health, viruses, vaccines, epidemiology, etc? Endlessly defended here on HN any time his name comes up.
"Trust the science" is a silly slogan and should be dismissed.
But you should also remember that "science" doesn't guarantee that _every_ paper is good. No one can. "90% of everything is crap" is a law of the universe as solid as the second principle of thermodynamics.
Science is a _process_ that is able to weed out the crap more efficiently than anything else we tried. Any big name can put out crap, true, but as soon as they do, there is an eager army of people who jumps at the opportunity of "proving X wrong". If the original paper was crap, that would be a pretty easy thing to do.
Note that I said "more efficiently", not "efficiently" in general. There will always be an army of people "trusting the science" (the authority, really) and so it might take time for the crap to be weeded out. But it will eventually, because science is like evolution: if nothing can be built on top of the crap, it will be progressively abandoned, because it can't reproduce (pun intended).
> But you should also remember that "science" doesn't guarantee that _every_ paper is good.
That's correct and I agree. Actual Science doesn't require trust. As you aptly pointed out it's excellent at weeding out crap most of the time. The issue is the academic politics involved in it make it harder to determine if the process is weakened. The problem of course comes when a major result was not reviewed appropriately and it becomes the standard until someone is brave enough to write another paper failing to reproduce it. That's the scary thing - even if someone is brave enough to write a paper saying its non-reproducible they still have to get through the review committee to have their voice heard. Sure, they could post to Arxiv or a blog if all else fails...but none of that will ever get picked up where it is needed most.
> There will always be an army of people "trusting the science" (the authority, really)
This is really the crux of it. It's appeal to authority. Even higher order thinking people (that is, those outside pop-sci nonsense) fall victim to this because it's human nature. The authority is also how mediocre work gets past reviewers by attaching a name to it.
Of course it does. Professors with high caliber research run tight ship and their labs publish consistently high quality work.
It does not mean that reviewers lower their bar when assigned a paper from a famous professor, they just know a priori that the work will not be completely scam.
318 comments
[ 3.0 ms ] story [ 291 ms ] threadBesides, invited speakers are a thing for a reason.
(Also, it is a major red flag to cite 10 papers from one group and no related work from anyone else. Either the topic is completely irrelevant, or you didn't do a cursory literature search - at least skim the references from papers you cite!)
Double blind helps the former, though not the latter.
In a part of my field that is capital intensive, some well funded newcomers have recently invested a lot, and "broke in", while some incumbent testbeds went away; the "expensive equipment" problem is usually temporary if the field is expanding.
Alternatively, when IceCube 2 comes out, the old IceCube crowd might be focusing on other stuff, and not paying attention to the IceCube 2 politics. That makes them great peer reviewers (no horse in the race, but knowledgeable).
Lifehack discovered: legally change your name to that of a Nobel prizewinner if pursuing academia.
Jobseekers with Anglo-Saxon, easy to pronounce and common names are the most likely to get to the interview stage compared to candidates with unfamiliar names, according to research by the Australian National University published in the Oxford Bulletin of Economics and Statistics.
https://www.independent.co.uk/news/business/news/unusual-nam...
https://www.behindthename.com/names/usage/anglo-saxon
"No no no, within which historical context does it even make sense to ask this question? No? You see?"
“Jobseekers with Anglo-Saxon” means “candidates who speak Anglo-Saxon.”
And I’m like “I thought that was just a medieval studies thing, but I guess the older languages really do help …”
People will choose what they listen, see, or read heavily based on who the performer is.
A performer who has consistently given good content will obviously have a bigger pull than a nobody still trying to get their first break.
The identical paper being rejected more often based on the celebrity of the author isn't that.
I've seen it from the inside. Probably read 80-100 widely-cited papers during my PhD (before dropping out), and maybe half a dozen of them were written by people who had any interest in discovering truth and pushing mankind forward.
Seriously cannot overstate both the willful ignorance of established scientists, and the extent to which this is enforced onto the next generation.
Sometimes it seems very obvious that a given paper is by specific authors (either because of style, or because of how familiar it is with their previous work) but I've had many experiences where I later learned that my supposition was completely wrong. Similarly, when you encounter a paper that doesn't have any obvious cues (which is the overwhelming majority of them) then it's pretty much impossible to tell whether it's an author you admire or someone you've never heard of -- and this is a good thing.
Some conferences don't use blind submissions, and yes: I have felt an awful lot of influence there. "Surely [famous Turing-award winning authors] don't need me to double-check their proof."
I'm not sure how to fix this problem though. In our phage newsletter we try to avoid using names and universities to focus on the paper/topic/finding itself, but I keep finding myself looking at author names and affiliations before diving into any paper.
I know it's "wrong" and I recognize myself doing it, but I still do it all the time.
This is a good initiative, but a catch is that if he's the only person deliberately spelling some words British and others American, the spelling choice becomes a unique identifier.
Though, it could work as long as no one within the group knows who is using the varied spelling.
People should copy the writing style of famous auctor's then to expose the system and keep people honest because then they know there are copycats.
https://www.wired.com/2012/07/mklopez-digg-power-user-interv...
Reddit and IMDb are mostly immune. Which is not to say that they're immune to all problems (astroturfing, brigading, personal biases, etc.).
The main change is you can post posts to your profile rather than a particular subreddit. But that's not a big feature. Back in the day people used to just create a subreddit of their username.
They skim, see if it looks okayish and give it their approval.
But it's far worse, even without a recognized names, most votes are cast without proper reading, and I'm fairly certain also by the least intelligent subsection given how often submissions are upvoted on H.N. and Reddit that are pure clickbait and demolished in the comments by people that actually read it. People that vote by and large only read the title or the first sentence and make up their mind from there.
Agreed. I was referring to Quora.
Each group trended towards catapulting a single band / musician, but it always just depended on which one in the group got the momentum first.
I wish I could find the study again but it's hard to google for.
Pretty eye opening.
https://www.science.org/doi/10.1126/science.1121066
As in this person has more than proved himself, let’s not vet him as much.
[1] https://en.wikipedia.org/wiki/Diederik_Stapel
Right up to the very end, this man was able to publish like a celebrity because no on questioned it because he was famous. So let's not bake that into the process: scrutinize all papers equally. Experts don't get a free pass, if they have new claims to make, those claims are just as "I don't believe you yet" as anyone else's.
A famous name getting the same paper published easier is a failure of peer review. The whole point of science is that ideas and evidence stands on its own merit. Not celebrity or seniority or power or any other axis that doesn't matter.
You don’t want to go on reputation.
My comment was snarky but not remotely playing the man. I meant it quite literally as a criticism of the thinking at issue. Economistic thought is riddled with confusion, and being surprised that scientists don't behave normatively (ie. not as those engaged in peer review are "supposed to") exhibits perfectly one variant of said confusion.
I'm not 'astonished' that non-blinded peer reviewers are influenced by social status. No-one I know would be surprised at all, let alone 'astonished'.
I am astonished, in the same way I would be astonished to find out that papers that smell like fish are more likely to be accepted for publication at the majority of peer reviewed venues, and that the reason is that reviewers for those publications let their house cats stack rank the submissions.
I'm not saying there aren't people in academia who aren't driven by doing good research but I certainly don't see that as the driving force in the US academia.
I didn't realize that HN was an academic journal where all statements should come with proof.
Do you have proof that I'm wrong?
This does explain some of the recent and embarrassing [lack of] retractions of Nature papers though.
Reviewers are generally overworked (for other reasons), and the main goal is to be constructive and somehow not sound like an idiot (in front of the other reviewers, who might be subject experts and definitely know who you are!) in any of the pile of reviews you need to write after reading a pile of papers.
Also, reviewers declare conflicts ahead of paper assignments, which makes accidents less frequent.
In any case, this is the tip of the iceberg. The entire structure surrounding academic publication is absolutely ill-designed and few people in that world are even remotely interesting in safeguarding veracity.
The overwhelming majority of peer reviewed scientific results is infotainment for other scientists with no party having any material stake in the veracity of it, which is almost no one bothers to replicate because accuracy is irrelevant so long it be interesting to read.
The moment a company has bet a sizable stake on it's accuracy, then they suddenly check, and double check and have another party independently verify it because they do not want to loose money, obviously. But most science is nothing like that.
Authors regularly cite work by themselves or their team, so a statement like “In a previous study[1] we established a relationship between X and Y” renders double blind pointless.
That being said an author isn’t published randomly in Nature, so I expect subsequent papers from an author to be better, on average, than non-Nature published authors.
And your reputation follows you. If it’s a big-name lab who has an amazing track record, you’re going to review the paper in the context of their entire body of work.
Great feedback, she thought, because She was xxxxx.
Academia was bought and paid for long ago and the money was used to build an incredibly broken and overly political bureaucratic engine of scholarly and scientific work that doesn't get anywhere near as much peer-review scrutiny as it should and commands far more respect in politics and legal proceedings than we should allow.
Universities and experts are the best we can do sometimes so we have to rely on it, but it doesn't mean it's truth or absolute and people like to use it as if it is to sell ideas like global warming instead of educating people on climate change.
This was my impression of most of my professors years ago too, especially as we saw so many quit for high paying engineering jobs at companies year to year — the ones who stayed aren’t in it for the money
I suspect it’s the same at the top: most senior administrators I bet would make more in senior Fortune 500 jobs
Once you take money from someone (except for NIST, in my experience) you're basically beholden to try your hardest to get the results they're looking for. Some scientists are moral enough to still return bad results. A lot of scientists aren't. There's a lot of garbage out there, and the worse the journal, the more garbage it gets. Famously, Phillip Morris studies "passed" the scrutiny of several major journals. It's amazing what greasing a few palms will get you.
I disagee, you should backup your accusations or statements (unless widly accepted) with some reputable source. This person claimed that science was bought and paid for. Did they mean all of it? Or most? That's an insane accusation that requires evidence.
"Go to your favorite major journal and start looking at the "conflicts of interest" and "grants" section. Once you take money from someone (except for NIST, in my experience) you're basically beholden to try your hardest to get the results they're looking for"
This doesn't mean people falsely data. You're simply providing motive.
Everyone wants money, you are using greed to then claim mass fraud in science
This isn’t only about greed either. People want their research published for reasons other than greed. For example, they want to move up in their career or achieve recognition.
After looking at a lot of medical studies related to COVID during the last couple years, I have seen first hand how biased and inaccurate many of them are. Some of these studies are even mentioned in major news outlet despite their obvious flaws when you actually begin to scrutinize them. Think big pharma providing research grants for studies that conclude their products are effective.
The OP never said that people falsify data as a result of receiving grants from interested parties. They often don’t have to. They can simply design the experiment in a way that doesn’t account for specific variables or behaviors then use the resulting data to reach a specific conclusion.
I remember seeing an article related to AI research on HN a little while ago that somewhat explained this problem. The grant money all goes to people researching deep neural networks which creates a reinforcing feedback loop. Since all the money goes to one branch of research, it creates very few opportunities to research competing ideas. I believe it was this one:
https://nautil.us/deep-learning-is-hitting-a-wall-14467/
Most declare no conflicts of interest. One author of one paper seems to have started a company based on similar technology: potentially a bias, but also potentially putting one's money where their mouth is. One other author lists some consulting work for a few companies.
As for grants, I doubt people are bending their results to appease the NSF or NIH. There's certainly groupthink in what gets funded. We're still throwing money down the ABeta-for-Alzhemier's hole, for example. That eventually shapes what topics get published, but maybe not the specific results. The recent Abeta articles are pretty negative, for example.
However, I think your climate change example is a bit strange. If anything, it's big oil who's been trying to sell the idea that we don't have to stop using fossile fuels. Hiding evidence, and spreading confusion by paying lobbyists and scientists. Global warming was proved beyond reasonable doubt decades ago.
Any questioning it, even slightly means being banned from grants and academia.
Its also interesting that most climate models are NOT open source. Most recorded data from satellites is also NOT open source. So everyone works with a pre cleaned data set.
Its also worth pointing out that data sets like HadCRUT have never been audited by any respected scientist/group of scientists. This data was collected in stations not meant for long term measurements and they have a lot of errors. Just download it yourself and see. (Climate scientists are not really data experts, since they go from clean datasets in school to "clean" data sets in real life)
Calculating global temperature is also one of those things that is done in quite an obscure way, extrapolating too much IMO.
No other hypotheses about geophysics are required to show realistic and fault free simulations of the climate of the planet over several centuries to be generally accepted. Why should we apply such extreme prejudice to the hypothesis of climate change caused by the greenhouse effect? The basic mechanisms is quite simple and well understood, there is a variety of kinds of measurements supporting the claim that the temperature of the planet is increasing (meteorological temperature measurements, glaciers disappearing etc.). Full understanding of all the geophysical processes and feedback loops involved is not necessary, and very likely impossible.
I also have a hard time understanding the motives for such an enormous scam. Who would stand to gain from this except a relatively small number of researchers and the renewable energy industry? On the other hand, it's well documented that the fossile fuel industry has tried to sabotage climate science for the purpose of limiting political action on the issue.
This is false. NASA and ESA science data is free.
There is often an embargo period for very novel sensors, and always a delay of hours-to-weeks to allow processing to catch up, but it's free.
If it's the source code of the analysis pipeline you mean -- even though you said data - that's a harder lift, because the processing is complex. But even that is changing (https://science.nasa.gov/open-science-overview).
Even in the absence of the open science initiative above, today you can always get the raw data ("Level 1 radiances") or sometimes even uncalibrated straight-off-the-sensor data ("Level 0"), if you want to process it. (https://www.earthdata.nasa.gov/engage/open-data-services-and... -- "All EOS instruments must have Level 1 Standard Data Products (SDPs)")
And if you want to look in to how the processing works, there are detailed documents ("ATBD's") that explain how the pipeline works, for each data product. Also free.
> Climate scientists are not really data experts, ....
Dreadfully wrong. Do you work in this area at all?
(he had only one peer anonymous reviewed paper and it had an error)
". Albert Einstein only had one anonymous peer review in his career — and the paper was rejected2. This happened in 1936."
How is just one paper submitted anonymously being rejected an indication of a trend?
As in, that would be the null hypothesis. It would be astonishing if most academics overcame it.
EDIT: apparently, the size of the effect (factor of six more likely to be accepted if it came from a Nobel Prizewinner) is larger than anticipated.
One thing I quickly noticed was that the articles were never signed by the authors directly. I found this striking, but also refreshing. It caused me to pay more attention to what was said. And, as I later found out, that seems to be the main intention[1]:
The main reason for anonymity, however, is a belief that what is written is more important than who writes it. In the words of Geoffrey Crowther, our editor from 1938 to 1956, anonymity keeps the editor "not the master but the servant of something far greater than himself…it gives to the paper an astonishing momentum of thought and principle."
[1]: https://www.economist.com/the-economist-explains/2013/09/04/...
Opening NRK right now gives "Now the government must see the madness in this", "Electricity prices can make going to the cinema more expensive", and "He is so tired" as the top articles for today.
I follow so much International/American news thanks to publications like Reuters, AP, and Qz. But no matter how much time or money I use, I can not follow my own country's news. I definitely am not alone in this, and it terrifies me how unworried most of Norway seems to be about this fact as well.
Its a ticking time bomb.
> In 2008, in response to a series of Call-for-Paper e-mails, SCIgen was used to generate a false scientific paper titled Towards the Simulation of E-Commerce, using "Herbert Schlangemann" as the author. The article was accepted at the 2008 International Conference on Computer Science and Software Engineering (CSSE 2008), co-sponsored by the IEEE, to be held in Wuhan, China, and the author was invited to be a session chair on grounds of his fictional Curriculum Vitae.
We had a piece of text that the subjects (undergrads) would read and rate the expertise of. We had the same text but we randomized whether the name would be a commonly male name, a commonly female name, or initials.
We also randomized if they'd watch a clip of men's sports beforehand (men's basketball), women's sports beforehand (women's basketball), or no sports. And finally we randomized whether the person giving the subjects instructions would be a young woman (one of my classmates) or a young man (me).
Our small study showed what you'd expect- students- men and women students rated male writers more highly than female writers, and initials were right in the middle, though they tended to be more like responses for men.
The tester's gender made no difference that we found.
The sports thing made a measurable difference, but it didn't reverse the skew.
My 18/19 year old self was somewhat skeptical we'd find a difference, but I was totally wrong. It taught me a lot about bias and perception, which this study also shows.
I imagine these biases swing in all sorts of directions depending on the context. Some are intuitive, many are not.
Also pay-gap has likely also to do with the current men-bias also? Wonder if that would be the same if we had 90% woman throughout in these decising positions..
In the end all mine assumptions, what I actually want to say is: I don't think those two contradict or relate at all.. what GP already said well with "biases swing in all sorts of directions depending on the context"
From your wikipedia link
> There are two distinct numbers regarding the pay gap: non-adjusted versus adjusted pay gap. The latter typically takes into account differences in hours worked, occupations chosen, education and job experience.[1] In the United States, for example, the non-adjusted average woman's annual salary is 79% of the average man's salary, compared to 95% for the adjusted average salary.[2][3][4][5]
The remaining 5% could be from the "women are wonderful" positive attributes not being the narrow selection of ones that are highly sought after in well-renumerated jobs, or (spitballing here) psychosocial (Expectations of a pay gap driving negotiation behavior or something).
Even at the time, there were studies that showed traits associated with women were rated more positively by men and women than traits associated with men.
The way you study this is you'd give a list of words:
gentle caring assertive stubborn aggressive loving
(ideally you'd randomize the word order too)
And then you'd have a group of subjects rate them as more associated with men or women on a scale.
Ideally you'd get a big sample and replicate this study.
Then either in the same study at a different time, or another study, you'd take those same words and you'd ask your subjects to rate them as positive or negative.
That's where you see effects like the one mentioned in the wikipedia article.
That kind of thing's been done a bunch of times.
But studies on gender and competency have been done too, and at the time they showed the same pattern as we found.
I say "At the time" because this is >20 years ago.
Right. There are two different sets of benefits that help or hurt someone in different ways. Being competent when standing trial can work against you while being wonderful will reduce the risk of a conviction and reduce the sentence if you are convicted.
We have identified the "competent" bias and are taking steps to correct it, but we need to do the same with the "wonderful" bias in other systems. For starters we need to recognize how strong that bias is in certain fields. For one example, there are specific crimes that people would bet are extremely gendered in nature, and the crime statistics show they would be right, but interviewing the population at large and querying victims, including those who never went to the police or who were turned away by the police (or even worse, who couldn't legally be victims because of how biased even the laws are), we see the gender component goes away. The rate of men victimized by women and women victimized by men are at near a 50/50 ration (I think 49.8 to 50.2).
Even the extent of studies measuring the impact of the wonderful effect is lacking compared to studies measuring the competent effect (which itself is likely a bias of the wonderful effect).
If there is a bias, is it good, bad, or neutral? Are there biases in other directions?
Bias isn’t inherently good or bad - they are normative judgements of distance from rationality.
Some biases are probably evolutionary advantageous to the individual but harmful to the society or others.
Sticking with the group is great, except when it isn’t, for example.
Or, a “gut check” is great, but you can’t always make decisions based on your gut:
https://en.m.wikipedia.org/wiki/Affect_heuristic
I think if it had been a larger study (not a one credit class run by 4 undergrads but an actual study with funding) we might have had multiple texts and randomized them, but we had one text- I don't even remember what the topic was.
They told one group of Asian women that “women are worse at math than men” and another group of Asian women that “Asians are better at math than non-Asians”. Both are common stereotypes.
They then measured how well the two groups did on subsequent math exercises.
Interestingly, they found that the latter group (positive stereotype) performed better than those in the former group (negative stereotype).
As I recall, other research found similar results with Black males and golf scores when told “white people are better at golf” as opposed to “black people are better athletes”.
Not only do stereotypes influence our perceptions of “the other”, but they influence the performance of the other.
”You call me an addict and refuse prescription? Yeah sure whatever doc, I’ll just buy that fentanyl off the street since I’m already an addict.”
We need to end prohibition.
[1] https://www.tandfonline.com/doi/full/10.1080/23743603.2018.1...
I wish people would stop bringing up studies that don’t reproduce, they’re no better than anecdotes.
Are they still a reputable source, today?
Setting this aside for a deeper read, but it appears at first glance that the concerns revolve around statistical technique rather than methodological soundness?
How much more? That's really important.
Most things I read I have no idea of the sex of the author. Do people even look at author names before they start reading (online, say), and even then they can be nom de plumes, or non-gendered names (sometimes surprisingly).
During debriefing, we would tell the subjects what we were actually looking for (perceptions based on gender) and most reported not remembering/caring about the gender of the author, but nonetheless the results were clear that there was a gender bias, regardless of whether they reported remembering or caring about the gender of the author.
At the time, I felt like "Sexism is dead" and the university I was in was 2/3rds women to 1/3rd men, so we'd never find a bias.
As a young man (18/19) I learned to question my assumptions. I thought "It's the 90s, sexism is dead.", but it just wasn't true.
The simpler version of the study had been done before, and the class I was a study design class. Our class assignment was to design, execute, and analyze a study.
We were simply combining two existing studies into a new study.
It bears mentioning- this was ~1996/1997 (I don't remember), I was a freshman, it was a single credit supplement, and we never published the results.
I don't see this "bias" as inefficient or counter-productive. It is just an artifact of the way you designed your flawed experiment.
To find any real bias you would have to assure the subjects that the researchers were equally accomplished.
What do you mean? I think you didn't understand the experiment. Subjects were shown the same text with, at random, a male name, female name, or initials. These names were not real researchers, nor familiar to the subjects.
> If it is known that most discoveries are made by males it is reasonable to give larger weight to new male-done research
I'm sure you can see the two different fallacies in this short sentence:
- There have been more male researchers than female, therefore more "discoveries" were made by men than women. No shit. This does not mean male researchers are better than female, obviously.
- There are more men being researchers therefore men's research should be given more weight therefore there are more men researchers therefore...
1. most discoveries are made by males
2. it is reasonable to give larger weight to new female-done research
contradicting you point.
No it isn’t. Dont you see this is just circular reasoning?
I think it's safe to say that this study can be treated as an anecdote because we don't know the effect size or exact methodology and it was never peer-reviewed. And I'd agree with your point if we were talking about the Nobel prize study, which evaluates people as individuals. But arguing that 'real bias' is bias that doesn't come from empirical evidence doesn't make a whole lot of sense- all bias comes from empirical evidence of varying quality.
Just one of the issues is the specific perception regarding that particular sport (basketball) and relative perceptions between sex differentiated similar sports (men’s and women’s basketball are not the same sport, albeit similar). A sport should have been chosen where there are as few differentiating perceptions as possible, which is nearly impossible, because men are inherently more competent at sport due to physiological realities. It bakes in a bias towards men, which is what you likely actually confirmed with the study you described.
Introducing sports alone essentially corrupted that research and likely biased the responses for relative innate competency of males in sports.
For example, ignoring its own challenges, what could have been done, is show video of a relatively broadly positively evaluated sport (not basketball), an activity that is also relatively broadly and positively evaluated female clustered competencies, e.g., effective child rearing, communicating effectively, conflict resolution, creative outputs, or even beauty pageants, etc., along with the inverse, i.e., poor performance of men in the same sport and poor performance of women in whatever is chosen, e.g., screaming and yelling at poorly behaved children.
It is in the past and by no means are or were you the only one who has long engaged in this type of poorly executed “research” that has merely been confirming researcher biases and often, thereby immensely damaging society, but maybe my illustration of a few of the several issues with what you described, will cause some changes in thinking.
There is a real hidden epidemic of not only single order thinking, but what really should be called negative order thinking in research, where research is not only not finding the truth, but rather even doing damage through confidence in false findings.
I say these things with an expensive relevant background, in the trenches of bad and … frankly … destructive research, if you will.
Non-sense. However, for all the people that say "trust the science" this is the science you're working with. It's politics all the way down just like everything else. For example, if Knuth decided to post some utter garbage chances are it would accepted based on his name alone.
What makes something political? If new research shows that X product is dangerous and republicans, who that company donates to, come out and disagree with the research they just made it political. Now, someone like you, can just say "This research is political"
You've created a way to dismiss scientific research simply by disagreeing with it
Science is a process and it is never complete. You are confusing a process with the results of the process, which are by definition imperfect. What makes Science good is that it's results are fundamentally imperfect and tainted, and that it respects that.
When you go around saying 'science is perfect' you disrespect its core principles.
It's possible to both have a generally high amount of faith in 'the science' while still remaining critical of its flaws, or rather, the flaws of the system in which its conducted.
If there's a vaccine and the majority of people in that field say it's safe but you want me to be critical what do I do?
I hesitate to use the word faith at all, even when describing the scientific process itself. Having faith in the process leads to (for lack of a better phrase) a failure to trust, but verify. IMO, it is one thing to approach scientific literature with a trusting demeanor and still walk through the proper processes to verify the results, and another thing entirely to literally take the author's word. Often times especially when big names are attached we aren't trusting-and-verifying we are simply taking their word. That's a problem that weakens the credibility of science.
I don't want to come off as someone who disregards everything he doesn't like. However, I do approach science with great pessimism and have since graduate school. Not because I hate it, or something is wrong with the scientific method, but because approaching it pessimistically allowed me to write better work by forcing me to actually perform the method instead of p-hacking my way to a result.
Rather, Knuth could publish some mediocre paper that would be instantly accepted by all the top journals, widely read, discussed and cited instead of more salient works on similar topics.
That's the problem really, excellence is rare while mediocrity is aplenty and academia is a numbers game. The mediocres with prodigious output will get the cites, tenure, TAs, lab assistants and budgets to do research and perhaps stumble onto some other mediocre result, reinforcing the cycle.
Emperor's new clothes, while the rest need to be whiter than white and all that. These things happen because people are blinded by titles in a complex world with complex people.
You mean like Elon Musk? When he came up with that giant box to extract kids from a cave with passages so narrow rescuers couldn't wear their air tanks on them, they had to push them ahead of themselves, reddit and HN and twitter were full of people defending him. There were even people defending him when he accused the rescue leader of being a pedo - simply because he was british and living in thailand.
Or how about the CEO of Brave, who apparently is an expert in public health, viruses, vaccines, epidemiology, etc? Endlessly defended here on HN any time his name comes up.
Smart people like Bill, Elon and the CEO of Brave are polymaths and can cross domains quite easily.
Musk is just a guy getting rich via insider trading. Gates is most likely using his foundation to push his interests.
But you should also remember that "science" doesn't guarantee that _every_ paper is good. No one can. "90% of everything is crap" is a law of the universe as solid as the second principle of thermodynamics.
Science is a _process_ that is able to weed out the crap more efficiently than anything else we tried. Any big name can put out crap, true, but as soon as they do, there is an eager army of people who jumps at the opportunity of "proving X wrong". If the original paper was crap, that would be a pretty easy thing to do.
Note that I said "more efficiently", not "efficiently" in general. There will always be an army of people "trusting the science" (the authority, really) and so it might take time for the crap to be weeded out. But it will eventually, because science is like evolution: if nothing can be built on top of the crap, it will be progressively abandoned, because it can't reproduce (pun intended).
That's correct and I agree. Actual Science doesn't require trust. As you aptly pointed out it's excellent at weeding out crap most of the time. The issue is the academic politics involved in it make it harder to determine if the process is weakened. The problem of course comes when a major result was not reviewed appropriately and it becomes the standard until someone is brave enough to write another paper failing to reproduce it. That's the scary thing - even if someone is brave enough to write a paper saying its non-reproducible they still have to get through the review committee to have their voice heard. Sure, they could post to Arxiv or a blog if all else fails...but none of that will ever get picked up where it is needed most.
> There will always be an army of people "trusting the science" (the authority, really)
This is really the crux of it. It's appeal to authority. Even higher order thinking people (that is, those outside pop-sci nonsense) fall victim to this because it's human nature. The authority is also how mediocre work gets past reviewers by attaching a name to it.
It does not mean that reviewers lower their bar when assigned a paper from a famous professor, they just know a priori that the work will not be completely scam.