I'm not a fan of returning to this form of communication.
In a recent review, which was single-blind, I had a paper from a group I know well and the principle author was a heavy-weight in my field, and certainly will review proposals I write in the future. The paper had good content but it was clearly written by a graduate student of his and it really needed work. I made numerous suggestions for revision that I would have felt very hesitant to do if the reviews were fully open (e.g. if my name had been attached). I would have been inclined to simply accept this paper with minor revisions so that the PI did not have any impression I was being unreasonable.
As another example, I had a recent double-open review for a conference where I had to attach my name to a review that was not able to accept the paper. I felt (and still do) that this submission was absolutely not acceptable given the standards of this proceedings. Now, this group has seen my name and associated it with a 'reject', and I am worried that should we meet in the future, this could cloud our interactions.
For both, I've thought about this a great deal and I stand completely by my reviews, my criticisms, and I did try to come across as professional as possible. But I would have preferred to be anonymous, and I see no reason my name being attached would have helped the review process.
>I would have been inclined to simply accept this paper with minor revisions so that the PI did not have any impression I was being unreasonable.
The impression I get is that this article is about the fact that despite your connections with the author and team, various pressures of the review process (and probably timelines) prevented you from saying to the principle author in private, "Hey buddy, your work looks promising, but your graduate student really needs an editor."
I find that review of the form "this is good, but needs cleanup" is the easiest to give or to receive in the type of reviews I participate in. If you can't even attach your name to that, I'm inclined to believe there are problems with the system that anonymity alone doesn't solve.
It's not like you can't get a 95% idea of who is reviewing a paper now, at least in CS. Look up the list of technical committee members (since CS is conference centric), find the two with enough expertise in your area to be the primary or secondary reviewer, boom you have found two of your reviewers.
Typically works when the papers are blind too, people have very different paper writing styles.
Good point. For the first paper, I bet that group would have put me in a top-5 list, and they certainly would have put my group (or someone from it) in the first place.
But that slightly anonymous element helps, because typically you aren't sure who reviewed your paper.
Peer review, as it is known today, only took off at a large scale with the invention of the photocopier.
There's a story where Einstein withdrew a paper of his after he found out the editor had sent it for peer review without his permission. He was offended - it just wasn't the norm back then.
That's a great article you linked to. Pretty eye-opening, to me at least. I feel a little different about peer review now.
> decisions about what to publish and what to reject were usually made by journal editors, often acting largely on their own. These decisions were often made rapidly, with papers appearing days or weeks after submission, after a cursory review by the editor. Rejection rates at most journals were low, with only obviously inappropriate or unsound material being rejected
Maybe this isn't possible today because of extreme specialization, but this sounds like a system with a lot of positives (relative to what we have now). ArXiv-like.
Yes, when I looked into it I found there is actually zero evidence that institutionalized peer review is helpful at all. In fact, it looked like it's primary role was to impede science by enforcing whatever prevailing bias exists at the moment. For example:
>"In addition to being poor at detecting gross defects and almost useless for detecting fraud it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused."
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/
>Yes, when I looked into it I found there is actually zero evidence that institutionalized peer review is helpful at all. In fact, it looked like it's primary role was to impede science by enforcing whatever prevailing bias exists at the moment.
While I've experienced what you've described, I think throwing it all out is merely going to the other extreme.
I think peer review should remain, but may need reform. The larger problem is the attitude of scientists today - some of the problems you yourself mentioned. Removing peer review will not fix those. And "fixing" peer review to handle those problems is merely treating a symptom. Ultimately, the larger problem is the mentality that has spread amongst academics - and the problems will remain no matter how you tweak peer review.
>In addition to being poor at detecting gross defects and almost useless for detecting fraud
This gets mentioned a lot, so I'll repeat what many others have said: Peer review was never meant to detect fraud, and should not be expected to.
Back during the Schon scandal, people kept asking how peer review had failed the system to let him get so far before he was caught. And the answer was always the same: Peer review is not the appropriate method to catch such people. It was never meant for it.
I agree that peer review should not be used to detect fraud. I would even say there is no reason to have any method devoted to detecting fraud. Instead just use independent replication to filter out fraudulent, sloppy, and unreliable results.
I've met many critiques of the peer-review system (including myself), but have never heard a compelling suggestion with what to replace it and why the replacement would be better. Without peer review, editors would have to assign papers by gut feeling or perceived reputation of the author, or the scientific community would be flooded with massive amounts of papers that are way below acceptable quality or even entirely false, not the speak of the dangers of pseudo-science and fake scientists or even paid shills.
However, I believe open access journals can and should experiment with alternative forms of peer review. For example, forcing every submitter to anonymously score 5 papers before his/her own submission is accepted, a karma system, a good meta-moderation system, and a good document revision system taken together could work for online publications. There is a lot of potential for innovation in online publishing.
The problems are in the end similar to those on online forums and social media. It's hard to keep trolls and people who try to game the system away and equally hard to prevent people from scoring solely according to their personal tastes and with kind of 'political' (as in politics of science) motivations, especially when the process is anonymous. But ordinary peer review has the same problem.
I believe that there are simple modifications that could be done now to the existing system. Namely, why are reviews typically single blind, where the author doesn't know the referees, but not double blind? Certainly, the reviewers could figure out who wrote the paper based on the context, but it seems inexcusable that we don't even attempt to remove this bias.
>"have never heard a compelling suggestion with what to replace it and why the replacement would be better"
You can't just replace/remove peer review and expect things to improve. It is part of the whole "assembly line" science environment that was created when the US government became the primary source of funding in the 1940s or so.
If we want functioning science, we need to replace at least two things:
1) The search for "significant" p-values with comparing precise predictions to data.
2) Peer review with independent replication.
Both are pretty much just reverting to how things worked earlier.
>The search for "significant" p-values with comparing precise predictions to data.
Problems:
1. Not sure we have a decent alternative.
2. While p-values are heavily used in some fields (social sciences, medicine, etc) they are not in others (e.g. physics). Yet those other disciplines still have similar problems.
What do you mean about not having a decent alternative? The perfect alternative is given in the second part of what you quoted.
Also, I see that physics is now starting to rely on statistical significance (eg LIGO), which I guarantee will be to their detriment. They are just behind those other fields in adopting it.
>or the scientific community would be flooded with massive amounts of papers that are way below acceptable quality or even entirely false, not the speak of the dangers of pseudo-science and fake scientists or even paid shills.
All problems that exist even with peer review.
>For example, forcing every submitter to anonymously score 5 papers before his/her own submission is accepted
This leads to low quality reviews.
>a karma system, a good meta-moderation system,
On the whole Internet, I have yet to see a system of karma/moderation that lasts. They are always gamed in the end.
And I'm not even sure what you mean here. What will high karma provide to the researcher?
19 comments
[ 3.1 ms ] story [ 52.2 ms ] threadIn a recent review, which was single-blind, I had a paper from a group I know well and the principle author was a heavy-weight in my field, and certainly will review proposals I write in the future. The paper had good content but it was clearly written by a graduate student of his and it really needed work. I made numerous suggestions for revision that I would have felt very hesitant to do if the reviews were fully open (e.g. if my name had been attached). I would have been inclined to simply accept this paper with minor revisions so that the PI did not have any impression I was being unreasonable.
As another example, I had a recent double-open review for a conference where I had to attach my name to a review that was not able to accept the paper. I felt (and still do) that this submission was absolutely not acceptable given the standards of this proceedings. Now, this group has seen my name and associated it with a 'reject', and I am worried that should we meet in the future, this could cloud our interactions.
For both, I've thought about this a great deal and I stand completely by my reviews, my criticisms, and I did try to come across as professional as possible. But I would have preferred to be anonymous, and I see no reason my name being attached would have helped the review process.
The impression I get is that this article is about the fact that despite your connections with the author and team, various pressures of the review process (and probably timelines) prevented you from saying to the principle author in private, "Hey buddy, your work looks promising, but your graduate student really needs an editor."
I find that review of the form "this is good, but needs cleanup" is the easiest to give or to receive in the type of reviews I participate in. If you can't even attach your name to that, I'm inclined to believe there are problems with the system that anonymity alone doesn't solve.
But those days were the days of the amateur. And a return to amateurism may well be the best way forward in a number of intellectual pursuits.
Typically works when the papers are blind too, people have very different paper writing styles.
But that slightly anonymous element helps, because typically you aren't sure who reviewed your paper.
There's a story where Einstein withdrew a paper of his after he found out the editor had sent it for peer review without his permission. He was offended - it just wasn't the norm back then.
http://michaelnielsen.org/blog/three-myths-about-scientific-...
> decisions about what to publish and what to reject were usually made by journal editors, often acting largely on their own. These decisions were often made rapidly, with papers appearing days or weeks after submission, after a cursory review by the editor. Rejection rates at most journals were low, with only obviously inappropriate or unsound material being rejected
Maybe this isn't possible today because of extreme specialization, but this sounds like a system with a lot of positives (relative to what we have now). ArXiv-like.
>"In addition to being poor at detecting gross defects and almost useless for detecting fraud it is slow, expensive, profligate of academic time, highly subjective, something of a lottery, prone to bias, and easily abused." https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1420798/
While I've experienced what you've described, I think throwing it all out is merely going to the other extreme.
I think peer review should remain, but may need reform. The larger problem is the attitude of scientists today - some of the problems you yourself mentioned. Removing peer review will not fix those. And "fixing" peer review to handle those problems is merely treating a symptom. Ultimately, the larger problem is the mentality that has spread amongst academics - and the problems will remain no matter how you tweak peer review.
>In addition to being poor at detecting gross defects and almost useless for detecting fraud
This gets mentioned a lot, so I'll repeat what many others have said: Peer review was never meant to detect fraud, and should not be expected to.
Back during the Schon scandal, people kept asking how peer review had failed the system to let him get so far before he was caught. And the answer was always the same: Peer review is not the appropriate method to catch such people. It was never meant for it.
https://en.wikipedia.org/wiki/Sch%C3%B6n_scandal
However, I believe open access journals can and should experiment with alternative forms of peer review. For example, forcing every submitter to anonymously score 5 papers before his/her own submission is accepted, a karma system, a good meta-moderation system, and a good document revision system taken together could work for online publications. There is a lot of potential for innovation in online publishing.
The problems are in the end similar to those on online forums and social media. It's hard to keep trolls and people who try to game the system away and equally hard to prevent people from scoring solely according to their personal tastes and with kind of 'political' (as in politics of science) motivations, especially when the process is anonymous. But ordinary peer review has the same problem.
You can't just replace/remove peer review and expect things to improve. It is part of the whole "assembly line" science environment that was created when the US government became the primary source of funding in the 1940s or so.
If we want functioning science, we need to replace at least two things:
1) The search for "significant" p-values with comparing precise predictions to data.
2) Peer review with independent replication.
Both are pretty much just reverting to how things worked earlier.
Problems:
1. Not sure we have a decent alternative.
2. While p-values are heavily used in some fields (social sciences, medicine, etc) they are not in others (e.g. physics). Yet those other disciplines still have similar problems.
Also, I see that physics is now starting to rely on statistical significance (eg LIGO), which I guarantee will be to their detriment. They are just behind those other fields in adopting it.
All problems that exist even with peer review.
>For example, forcing every submitter to anonymously score 5 papers before his/her own submission is accepted
This leads to low quality reviews.
>a karma system, a good meta-moderation system,
On the whole Internet, I have yet to see a system of karma/moderation that lasts. They are always gamed in the end.
And I'm not even sure what you mean here. What will high karma provide to the researcher?