Yes. When I was doing peer review work in my field (Mass Com), it took about a year to get the landscape sorted. That includes being able to ID people behind submissions, based on writing style.
Unfortunately, I saw submissions get down voted not because of the content, but because of some long standing intellectual feud (I found this out after I became a sought-after "tie breaker." The journal editors figured out I was indeed casting votes based solely on the submission in hand).
Not only that, but you inevitably self-site your earlier work in your later work. In most cases, the most cited author in a paper is the author of the paper itself. So just look at the last page of the paper, and you can probably tell who wrote it.
That previous discussion is of a different article by a different author, but, yes, this issue deserves a lot of discussion. The article submitted to open this thread is by two authors who follow the research literature on this topic closely and who review much of the prior literature in their commentary article.
A different article, yes, but I think the currently submitted one is a reaction to the linked one. I was going to link to the previous discussion, too, as I think this discussion and that one are necessarily related.
Which makes me think maybe HN should have some mechanism to say "This article is related to this other one", and point to the HN submission for it.
Which makes me think maybe HN should have some mechanism to say "This article is related to this other one", and point to the HN submission for it.
I guess our user comments here in this subthread are that mechanism. By the way, this whole thread is an implementation of the new HN algorithm that revives threads that are missed on first submission and gain upvotes later. I notice that is implemented by resetting the submission date on the submission. (I was the original submitter, so I remember when I first submitted this thread.)
At the root of this proposal is the fantasy that scientists would somehow be selfless adherents to the alter of the scientific method. They give up all honor, pride and recognition so that the "scientific method" be protected from the often selfish desires of human beings.
In the real world, having names on papers is a really good thing. In real academia, I've heard professors say "well that work was done by so-and-so, and they're bad researchers so I don't trust their results".
There's this popular misconception that just because something is published in a peer-reviewed journal that it must be true. Nobody in research actually believes that. All it means is the authors were able to convince the reviewers and the peer reviewers to publish it.
What if papers were cryptographically signed so that they can be claimed later and there were a period of, say, one year of anonymity after publication? This period would allow things like peer review and initial recognition of the paper to occur with less social bias.
Just as important would be institutional anonymity. A paper from Nowheresville should be considered equally to a paper from Harvard.
Finally I think papers should be required in their headings to disclose sources of funding for the work therein, although this would have to be done later so as not to indirectly compromise the anonymity period.
What problem are you trying solve? As bad as problems as the academic community has (it has lots!!), names on papers is not one of them.
Science is the organic collection of all people and resources and knowledge engaged in figuring out how our world works. Having blind papers would mean I would just have to spend an extra 10 minutes reading each paper to figure out what group wrote it.
The fact that Prof. Ivy League gets her results recognized more easily then Prof. Nowhereville is, actually, an ok system. Because Prof. Ivy is more likely to be pursuing interesting problems to the rest of the scientific community, she's more likely to be trusted by other scientists to be a good scientist (i.e. not careless), and the system supports people who have "made it" through the academic gauntlet.
If Prof. Nowhereville actually finds something interesting, pursues publication, passes peer review, and it gets published then great. His results are probably worth the attention of the scientific community. But the fact that the barriers are higher is ok, because he hasn't proven himself to be as trustworthy.
I think this is ultimately a manifestation of "scientific deontology ethics", a fantasy that we should approach each paper with identical, non-prejudicial priors and evaluate it on its own merits. (It is the scientific analog to the reductio ad absurdum of maximum non-prejudice in human affairs, where one never considers any background about a person before dealing with them.) This position only seem tenable to those who haven't yet had to wade through the morass of terrible and misleading result in the literature.
In other words, this sort of idea is not a consequentialist attempt to solve a problem, it's a deotological attempt to stop unfairness.
Every new scientist having to learn the heuristics to judge a high-quality paper from its metadata seems like a local maxima in fighting spam, and something that makes the profession harder to get into.
The community would be forced to develop tools and documentation to judge new papers based on actual content and aggregated opinions, which seems like a better way to filter poor-quality human-written content (we're not talking about email spam here). Plus, by making the process explicit and visible, it'd be easier for an outsider to engage with new scientific literature too.
Perhaps that's an approach we could use in software development. It's mostly how I use HN or Reddit -- I read plenty of comments, and only ever look at the name if the comment provokes interest in the writer's background.
> The community would be forced to develop tools and documentation to judge new papers based on actual content and aggregated opinions
I think you've got it backwards. First make the good tools, then stop using the bad ones. You don't force people to develop good Bayesian spam filters by banning whitelists. Rather, the Bayesian spam filters get invented, then people are happy to stop using whitelists.
Likewise, scientists would love nothing more than to have the tools that enable a transparent frictionless utopia free to barriers to entry. They'd get access to a few new ideas caught in the spam filter! But we don't get there by anonymizing papers. That would just lead (almost instantly!) to good scientists whispering in the ears of their colleagues about which papers they just published, or to take a look at that new paper by so-and-so, etc. (And then the only people who could access the metadata would be those who are already at good institutions!)
I don't think anonymity is achievable if the paper's to be any good. Papers build on previous work, and are reviewed by people working in a relevant field. In most fields, most of the time, you have a good idea who wrote a given paper based on what they're doing, whom they cite, and what software/methods they use.
Both have pros and cons. One con with having names on a paper is that it can give it attention purely because of the authors name at the expense of something else.
However I agree that scientist wouldn't do it (which kind of supports the idea of nameless papers IMO)
Wouldn't work. Gotta justify getting your grants by being able to point to previously published work.
Plus, everyone can tell who did what-- if not exactly who, then one of a small (5-10) group of people who are likely in the same geographical location.
No, but it would be nice if we had a place where we could discuss scientific papers anonymously. Of course, to keep things concentrated, it would be nice if there was exactly one such place.
I understand though that simple anonymous comments don't cut it. It can devolve into youtube comments quickly -- personal attacks to the authors, low effort comments, spamming, "trolling", etc.
Those sites need a private reputation system: you can vote comments but the commentator is anonymous; it's hidden reputation should help sorting the comments.
While I'm not sure about the complete anonymity suggested by Hanel, I do wish that journals would move to double-blind reviews. It would not be perfect, as often research know enough to guess who the authors might be, but it would still reduce bias.
One of the journals in my field requires you to strip out all identifiers before submission. This is remarkably difficult. Consider the things that might give you away:
- GitHub, FigShare, institutional repositories etc. for code and data
- Information on the location of the study
- Statistical packages used (hope you didn't author the one you're using)
Of course, anonymity will be shattered the moment anyone talks about their work at a conference. Furthermore, several systematic reviews and meta-analysis studies I've worked on benefitted greatly from being able to email the authors directly with questions.
I'd also be concerned that people will mentally assign things to a "Big Name" in the field anyway. "Ah, another paper on HAART. Must be out of so-and-so's lab..."
Sure- you can absolutely figure out who wrote the paper given enough work. Yet, I really do like the idea of first the paper being reviewed on it's own merit/ideas/findings, and then potentially people seeing who was involved as a secondary thing at some later point after judgements had already been made.
This was more a note that, when the paper is accepted, those identifying things go back in. The proposal to have papers be anonymous would have to leave them back out, stripping the paper of important context, and making open science a bigger pain than it already is. Which is kinda contrary to the goal.
I have submitted several papers for double-blind peer review. (I work in the systems side of computer science.) My rule is that I will not weaken a paper through the double-blind process - that is, even if saying this paper builds on prior work may give enough information to out me or my co-authors, I still do it. I still say what system we used, and say it builds on previous work, but I pretend that that work was done by someone else.
In practice, someone who really wants to break the double-blind veil probably can. But I don't think that's a problem, as very few people are consciously biased, and will actively seek out such identifying information. By removing names and affiliations, and pretending all of our work was done by someone else, I think that's enough to help people avoid unconscious bias.
"(AUTHOR et al.) performed a small scale lab study. (ANONYMOUS SUBMISSION) performed that exact same study, on hundreds of people, and expand on the work of authors..."
Well, there's "stripping out identifiers" and then there's "FBI-proof OPSEC" -- no one is going to run your word corpus through stylometric analysis, removing obvious links should be enough.
Makes me wonder if you could implement the exporting of a "review" version in LaTeX, similar to a debug mode -- set a flag, and any content with an {ID} wrapper around it gets left out or is substituted for something anonymous.
I don't think it's "FBI-proof OPSEC" to note that there are many things in a journal article that will give away the identity of the author.
While you can say "Remove obvious links", to many experts $Topic in $Location is enough to allow a pretty good guess of whose lab its out of. Or the name of a long running study. And I'm not going to maintain a whole separate Github account to anonymize code distribution.
It sounds like a good idea for the topics that are highly skewed by the political agenda. Like global warming.
Imagine a situation when some climatologist makes a discovery that global warming is a fluke. In the current political climate no reputable journal will accept his article on this topic. Moreover, no scientist will even SUBMIT one because of the possible damages to his reputation.
On the condition of anonymity, it may stand a chance.
How would one decide who gets tenure, promotion and raises if all papers are anonymous? Once universal basic income is implemented, maybe. Meanwhile, no.
Better idea: Submit it anonymously with your public PKI hash. Then wait X years (5?) and optionally announce yourself by signing the hash to prove you are the author. That way, enough time is given to have the content unbiasedly considered, yet you still get to take credit a bit later.
The landscape of incentives within scientific publishing is complicated. You have to map out what drives each of the parties: journals, authors, reviewers, readers.
Journals want high-quality submissions from well-known authors in order to maintain the brand. They also want ongoing subscription fees.
Authors want to be published in good journals, but they also want raw publication count to be high. So the primary incentive may not be intellectual honesty.
Reviewers often are in competition with the authors of the papers they're reviewing, hence the term peer-review. They have an incentive to prevent publication of competing work.
Readers want all knowledge to be free, so they have an incentive to not pay for journals. They also want to know which works are important and which are crock, so name-recognition comes into play.
I've probably left something out, but you get the idea. Coming up with new ideas is fun, but until you align the incentives, nothing meaningful will happen. And you've got hundreds of years of inertia and thousands of careers in the balance.
"Reviewers often are in competition with the authors of the papers they're reviewing, hence the term peer-review. They have an incentive to prevent publication of competing work."
This seems backwards to me. There isn't a fixed amount of interest in a field that is captured by peers. If others in your industry are getting attention then it draws attention to your industry as a whole. I'd think reviewers would be biased towards having competing work published not against.
In some fields, there is extreme pressure to be "first to publish" specific results, and sometimes multiple labs are working on the exact same problem. In this case, if the "peer" is with a competing lab, they are faced with the unethical decision of rejecting the paper in favor of their lab. Or even worse, of stealing the results and trying to publish first.
The correct term is not anonymized, but de-identified.
This wouldn't work in almost any field because the community can already tell with high accuracy who the peer reviewers are, and it's usually easy to tell who wrote a paper by the language style and the conclusions.
Peer review can apply the scientific method to the paper well enough. This may take time. That is just fine.
People having their names on papers helps with motivation to publish in most cases, and it's good to understand who contributed what when trying to understand how an idea has formed and who might be with collaborating or talking to.
Some topics may be difficult politically. Those are special cases that can and should warrant doing whatever it takes to insure publication and the safety of the scientist.
There are also ethics. Anon publishing may well increase questionable research. This should fall into the special case bucket too, as it overlaps with politics.
I think the main issue here is the problem of bias in the scientific community. Anonymity is just one proposed solution. There could be others.
We should remember that there are three major forms of incentive in the scientific community :
1. Prestige (requires name)
2. Salary and grants (requires credentials, and hence names)
3. The joy of analysis and discovery (doesn't require names)
(I might have missed some forms of incentives, but these seem to be the major ones to me.)
Anonymity will take the top two incentives away. The second one is a very practical one. There's an existing competition for salary and grants, a pretty brutal one in some areas where money is scarce. We need to solve this problem first before the concept of anonymity can be realistically entertained.
I have heard other proposals for reducing bias that seem more tangible. One of them is democratization (partially or fully) of peer review in a centralized publication system (like arxiv).
Simple solution is to have blind and double blind reviews. People write papers not only for the greater good but also to improve their careers and profiles. Because thats the only way to get more funding to do more work.
By mandating blind reviews, the bias can be removed easily.
There are many who give good/bad reviews just by looking at the authors names, not everyone of course.
I am not sure why blind reviews are not mandatory by now
50 comments
[ 3.1 ms ] story [ 95.1 ms ] threadUnfortunately, I saw submissions get down voted not because of the content, but because of some long standing intellectual feud (I found this out after I became a sought-after "tie breaker." The journal editors figured out I was indeed casting votes based solely on the submission in hand).
Which makes me think maybe HN should have some mechanism to say "This article is related to this other one", and point to the HN submission for it.
I guess our user comments here in this subthread are that mechanism. By the way, this whole thread is an implementation of the new HN algorithm that revives threads that are missed on first submission and gain upvotes later. I notice that is implemented by resetting the submission date on the submission. (I was the original submitter, so I remember when I first submitted this thread.)
In the real world, having names on papers is a really good thing. In real academia, I've heard professors say "well that work was done by so-and-so, and they're bad researchers so I don't trust their results".
There's this popular misconception that just because something is published in a peer-reviewed journal that it must be true. Nobody in research actually believes that. All it means is the authors were able to convince the reviewers and the peer reviewers to publish it.
Just as important would be institutional anonymity. A paper from Nowheresville should be considered equally to a paper from Harvard.
Finally I think papers should be required in their headings to disclose sources of funding for the work therein, although this would have to be done later so as not to indirectly compromise the anonymity period.
Science is the organic collection of all people and resources and knowledge engaged in figuring out how our world works. Having blind papers would mean I would just have to spend an extra 10 minutes reading each paper to figure out what group wrote it.
The fact that Prof. Ivy League gets her results recognized more easily then Prof. Nowhereville is, actually, an ok system. Because Prof. Ivy is more likely to be pursuing interesting problems to the rest of the scientific community, she's more likely to be trusted by other scientists to be a good scientist (i.e. not careless), and the system supports people who have "made it" through the academic gauntlet.
If Prof. Nowhereville actually finds something interesting, pursues publication, passes peer review, and it gets published then great. His results are probably worth the attention of the scientific community. But the fact that the barriers are higher is ok, because he hasn't proven himself to be as trustworthy.
I think this is ultimately a manifestation of "scientific deontology ethics", a fantasy that we should approach each paper with identical, non-prejudicial priors and evaluate it on its own merits. (It is the scientific analog to the reductio ad absurdum of maximum non-prejudice in human affairs, where one never considers any background about a person before dealing with them.) This position only seem tenable to those who haven't yet had to wade through the morass of terrible and misleading result in the literature.
In other words, this sort of idea is not a consequentialist attempt to solve a problem, it's a deotological attempt to stop unfairness.
The community would be forced to develop tools and documentation to judge new papers based on actual content and aggregated opinions, which seems like a better way to filter poor-quality human-written content (we're not talking about email spam here). Plus, by making the process explicit and visible, it'd be easier for an outsider to engage with new scientific literature too.
Perhaps that's an approach we could use in software development. It's mostly how I use HN or Reddit -- I read plenty of comments, and only ever look at the name if the comment provokes interest in the writer's background.
I think you've got it backwards. First make the good tools, then stop using the bad ones. You don't force people to develop good Bayesian spam filters by banning whitelists. Rather, the Bayesian spam filters get invented, then people are happy to stop using whitelists.
Likewise, scientists would love nothing more than to have the tools that enable a transparent frictionless utopia free to barriers to entry. They'd get access to a few new ideas caught in the spam filter! But we don't get there by anonymizing papers. That would just lead (almost instantly!) to good scientists whispering in the ears of their colleagues about which papers they just published, or to take a look at that new paper by so-and-so, etc. (And then the only people who could access the metadata would be those who are already at good institutions!)
Firing people to either develop tools or have a terrible situation on their hands doesn't necessarily mean they would develop those tools.
If you push someone into water, it forces them to either swim or drown, but it doesn't necessarily imply that they swim.
Both have pros and cons. One con with having names on a paper is that it can give it attention purely because of the authors name at the expense of something else.
However I agree that scientist wouldn't do it (which kind of supports the idea of nameless papers IMO)
Anonymous papers will be a pile of garbage.
Plus, everyone can tell who did what-- if not exactly who, then one of a small (5-10) group of people who are likely in the same geographical location.
I understand though that simple anonymous comments don't cut it. It can devolve into youtube comments quickly -- personal attacks to the authors, low effort comments, spamming, "trolling", etc.
Those sites need a private reputation system: you can vote comments but the commentator is anonymous; it's hidden reputation should help sorting the comments.
Have you looked at PubPeer? There have been some very interesting posts there.
https://pubpeer.com/
- GitHub, FigShare, institutional repositories etc. for code and data
- Information on the location of the study
- Statistical packages used (hope you didn't author the one you're using)
Of course, anonymity will be shattered the moment anyone talks about their work at a conference. Furthermore, several systematic reviews and meta-analysis studies I've worked on benefitted greatly from being able to email the authors directly with questions.
I'd also be concerned that people will mentally assign things to a "Big Name" in the field anyway. "Ah, another paper on HAART. Must be out of so-and-so's lab..."
This was more a note that, when the paper is accepted, those identifying things go back in. The proposal to have papers be anonymous would have to leave them back out, stripping the paper of important context, and making open science a bigger pain than it already is. Which is kinda contrary to the goal.
In practice, someone who really wants to break the double-blind veil probably can. But I don't think that's a problem, as very few people are consciously biased, and will actively seek out such identifying information. By removing names and affiliations, and pretending all of our work was done by someone else, I think that's enough to help people avoid unconscious bias.
"(AUTHOR et al.) performed a small scale lab study. (ANONYMOUS SUBMISSION) performed that exact same study, on hundreds of people, and expand on the work of authors..."
Makes me wonder if you could implement the exporting of a "review" version in LaTeX, similar to a debug mode -- set a flag, and any content with an {ID} wrapper around it gets left out or is substituted for something anonymous.
While you can say "Remove obvious links", to many experts $Topic in $Location is enough to allow a pretty good guess of whose lab its out of. Or the name of a long running study. And I'm not going to maintain a whole separate Github account to anonymize code distribution.
Imagine a situation when some climatologist makes a discovery that global warming is a fluke. In the current political climate no reputable journal will accept his article on this topic. Moreover, no scientist will even SUBMIT one because of the possible damages to his reputation.
On the condition of anonymity, it may stand a chance.
Journals want high-quality submissions from well-known authors in order to maintain the brand. They also want ongoing subscription fees.
Authors want to be published in good journals, but they also want raw publication count to be high. So the primary incentive may not be intellectual honesty.
Reviewers often are in competition with the authors of the papers they're reviewing, hence the term peer-review. They have an incentive to prevent publication of competing work.
Readers want all knowledge to be free, so they have an incentive to not pay for journals. They also want to know which works are important and which are crock, so name-recognition comes into play.
I've probably left something out, but you get the idea. Coming up with new ideas is fun, but until you align the incentives, nothing meaningful will happen. And you've got hundreds of years of inertia and thousands of careers in the balance.
This seems backwards to me. There isn't a fixed amount of interest in a field that is captured by peers. If others in your industry are getting attention then it draws attention to your industry as a whole. I'd think reviewers would be biased towards having competing work published not against.
and
"We'll publish this paper anonymously, clearing our industry of any wrongdoing - and it'll look like good science!"
This wouldn't work in almost any field because the community can already tell with high accuracy who the peer reviewers are, and it's usually easy to tell who wrote a paper by the language style and the conclusions.
Peer review can apply the scientific method to the paper well enough. This may take time. That is just fine.
People having their names on papers helps with motivation to publish in most cases, and it's good to understand who contributed what when trying to understand how an idea has formed and who might be with collaborating or talking to.
Some topics may be difficult politically. Those are special cases that can and should warrant doing whatever it takes to insure publication and the safety of the scientist.
There are also ethics. Anon publishing may well increase questionable research. This should fall into the special case bucket too, as it overlaps with politics.
We should remember that there are three major forms of incentive in the scientific community :
1. Prestige (requires name) 2. Salary and grants (requires credentials, and hence names) 3. The joy of analysis and discovery (doesn't require names)
(I might have missed some forms of incentives, but these seem to be the major ones to me.)
Anonymity will take the top two incentives away. The second one is a very practical one. There's an existing competition for salary and grants, a pretty brutal one in some areas where money is scarce. We need to solve this problem first before the concept of anonymity can be realistically entertained.
I have heard other proposals for reducing bias that seem more tangible. One of them is democratization (partially or fully) of peer review in a centralized publication system (like arxiv).
By mandating blind reviews, the bias can be removed easily.
There are many who give good/bad reviews just by looking at the authors names, not everyone of course.
I am not sure why blind reviews are not mandatory by now