Certainly not a new problem (this paper is from 1982, so old that my University subscription doesn't cover it). This might be a better link to start such a discussion: http://jrs.sagepub.com/content/99/4/178.long
In my field, it's common (and even encouraged) for authors to post their work to their own website. When authors give up their copyright to the IEEE, they explicitly retain the right to redistribute it on their own personal web sites.. From the text of the CVPR IEEE copyright agreement:
Author Online Use
6. Personal Servers. Authors and/or their employers shall have the right to post the accepted version of IEEE-copyrighted articles on their own
personal servers or the servers of their institutions or employers without permission from IEEE, provided that the posted version includes a
prominently displayed IEEE copyright notice and, when published, a full citation to the original IEEE publication, including a link to the article
abstract in IEEE Xplore. Authors shall not post the final, published versions of their papers.
(the last sentence refers to the "camera-ready" version, which ideally only contains eg. typo fixes relative to the accepted version)
At least one aspect of the peer review process should be better now than in 1982, when the paper was published: resubmission detection. Detecting plagiarism seems easy these days, just google for "detect plagiarism". But are reviewers actually using these tools? It would be interesting to repeat the resubmission experiment and see the results.
Many publishers allow Google to crawl the full text of the protected resources. This is why Google Scholar works, for example. Copy+pasting the abstract into Google Scholar is surprisingly effective at finding plagiarism.
Every reviewer that I've met does plagiarism detection on their own. My advisor has even found instances where the author copy+pasted articles from the likes of the Wikipedia article on the subject or 'about.com'; such papers get desk-rejected without review, and in some cases, the author is forbidden from submitting to IEEE conferences for the rest of their career. That's how important plagiarism is.
Just copy+pasting the abstract into Google is often enough to find it. At least for computer science conferences, the conference organizers run the papers through additional plagiarism screening tools before publishing, but it's mostly the reviewers' job.
"Every reviewer that I've met does plagiarism detection on their own."
I don't think it's a bad idea, but it never occurred to me to do this. If you're familiar with the field in which you are reviewing papers, you'll know if something is not new, in which case it is rejected, quite aside from the possibility of the author having lifted some actual text.
Sure, but plagiarism is more than just trying to publish an old idea. One paper was of something novel, but in their background section, they just copied a paragraph or two from an older article. That's the no-no.
You're right, my comment was not thoroughly considered. Maybe I should start doing some of this plagiarism detection. For some reason the idea depresses me, but you make a good case for it.
Correction again. Apparently the IEEE takes action if I EXPLICITLY say which paper does copy which other paper. But here the same author is copying - including pictures - another paper again:
It's just searching for certain phrases in the text. In this case the first one I tried "This process is defined as binding process. In The immune system" was already enough!
I just recently received a review of my paper, and it came with a pdf-report of which text fragments match a database of previous publications and the internet, made by some software called iThenticate. So clearly the journal now has a policy to automatically run this test for every submitted manuscript.
I haven't seen this before, so maybe that is getting more common these years now.
In academia, articles already have a form of post-publication review: citation count. If you don't do good work, you won't get any citations, though the converse isn't always true.
Professors usually ask their grad students to read the most important and influential papers in the field. When those grad students then turn around and start composing papers of their own, they'll refer to the ones they read, creating a kind of circular feedback cycle. Thus, if you want to find a field's best work, it often suffices to look for highly cited papers first.
It's like Reddit's karma voting system, just a lot slower. :)
Sure, just like how a lot of terrible and uninteresting articles on Reddit have a high karma count. By and large, both systems do a fairly good job at sorting out the interesting from the uninteresting, though there are always mistakes
True, but citations are qualitative as well as quantitative, so at least researchers should know that Controversial Paper X mostly gets citations as an example of "unusual findings, possibly due to methodological shortcomings" if they are deciding where it's worth summarising in the literature review section of their paper. And arguably (especially in social sciences) papers which provoke interesting rebuttals them are deservedly more influential than papers which are esoteric, trivial or dull.
If grad students are reading the most important and influential papers in the field then they're certainly not going to read the just published paper that tells everyone how to exploit magitech energy for great win - or whatever - because it's not important or influential yet.
All that process should do is reinforce the status-qua.
Your edit is itself very interesting :) Did the Cornell affiliation of one of the authors cause you to change your opinion in a positive or a negative way, and why?
Only joking. :) The joke being something along the lines of "look - I can peer review papers too - that wasn't so hard".
Buy the way, I'd love to see this research taken one step further, in a humoristic direction: a full statistical analysis comparing the predictive power of academic affiliation in determining acceptance for publication, to that of the contents of the paper itself. :P
The other popularly suggested solution is sort-of the opposite: use double-blind reviewing, where the reviewer doesn't know who the authors are, at least formally.
This is the standard in most disciplines. I wish the authors of this article had done the same test with journals that do blind reviewing. I'd bet they still got a lot of rejections of previously accepted articles for bizarre reasons, since in general acceptance depends a lot on the particular reviewers you draw.
Two counter-arguments: First, in many disciplines, it is common to circulate pre-prints of papers which are being refereed. In this case, anonymity for the authors is impossible, practically speaking.
Second, the OP describes a snob effect, but there is also the reverse: Some referees will be more patient in dealing with badly organized or poorly written papers from a junior researcher. Not necessarily lower standards, but spending more time figuring out the paper, making suggestions for improvement.
And it can be obvious who the authors are in a field you are following closely; which, ideally, you should be if you are reviewing papers in that field. That's why I said "formally".
Two of my biggest gripes with the peer-review process would be relatively easy to address:
1. Time
2. One-way communication
1. You sometimes have to waste a lot of time to get articles published. The way the current process works, it is very easy for referees who are either competitors, incompetent, or simply assholes to effectively filibuster the review process. Even when things go smoothly the process is often agonizingly slow.
2. The peer-review process operates under the obsolete limitation of snail-mail correspondence. There is no two-way communication between the referees. Referees often return either unprofessional or obviously wrong (to everyone but the editor!) comments. If the referees could talk to each other this would be sorted out quickly. It doesn't help that editors are typically very reluctant to reconsider rejection verdicts even when they are based on clear cases of referee error.
How would I address this? Use the #$%@!ing internet people! Journals should abandon this one-way snail-mail process and build a private message board for each paper being reviewed. Referees are invited to post their comments anonymously on the forum. Other referees and, ideally, the authors could then reply to those comments. After a reasonable time period has elapsed, perhaps a couple of weeks, the editor should take stock of what valid concerns remain and reach a verdict.
How so? The one time I was a referee I got the paper by downloading a PDF and sent in my comments by entering them into a form. That technology is newer than snail mail.
There are lots of things you might consider, that would be impossible with snail mail. For example, the reviewers might be able to send regular emails with questions to the authors, or the authors might be able to constantly update the paper in response to initial reviewer comments, there could be an anonymous "chat room" of some kind, etc. Not sure if any of these are good ideas, but I think it is a fair point that most journals simply use the internet as a "faster snail mail".
That said, I review for conferences that are taking advantage of the internet, with author response to reviewer comments, discussions among the reviewers, etc.
Regular emails with questions would certainly be possible with snail mail. But the fundamental problem with all of those suggestions is that only reviewers who really wanted to take a lot of time and care with reviewing would take advantage of them, and since there number of reviewers like that is vanishingly small, it wouldn't be worth the effort to establish the necessary systems.
I totally agree-- the problem with reviews isn't technological. Reviewers fundamentally need better incentives to do a good job, which I'm not sure how to do.
This is an early paper by Stephen Ceci, and illustrates the approach Ceci takes of challenging the assumptions that other psychologists assume to be true. This is why I love reading books and articles by Ceci. His Web-based CV
lists his extensive publications (you hope that most of those got better peer review after his 1982 paper than psychology papers used to). Google Scholar
helpfully lists some papers by Ceci for which there are free full-text versions that you or I can download for our learning and reading pleasure.
From Jelte Wicherts (another psychologist skeptical about peer review in psychology) writing in Frontiers of Computational Neuroscience (an open-access journal) comes a much more recent set of general suggestions
Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020
on how to make the peer-review process in scientific publishing more reliable. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence.
"With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses."
48 comments
[ 2.9 ms ] story [ 63.9 ms ] threadhttps://www.google.com/search?q=Richard+Smith+peer+review+fi...
Here's the .pdf: http://www2.psych.ubc.ca/~schaller/349and449Readings/Smith20...
In my field, it's common (and even encouraged) for authors to post their work to their own website. When authors give up their copyright to the IEEE, they explicitly retain the right to redistribute it on their own personal web sites.. From the text of the CVPR IEEE copyright agreement:
(the last sentence refers to the "camera-ready" version, which ideally only contains eg. typo fixes relative to the accepted version)Worth a read though, for the author's recollection of his (unsuccessful) attempts to reject a Karl Popper paper he thought was substandard.
Every reviewer that I've met does plagiarism detection on their own. My advisor has even found instances where the author copy+pasted articles from the likes of the Wikipedia article on the subject or 'about.com'; such papers get desk-rejected without review, and in some cases, the author is forbidden from submitting to IEEE conferences for the rest of their career. That's how important plagiarism is.
Just copy+pasting the abstract into Google is often enough to find it. At least for computer science conferences, the conference organizers run the papers through additional plagiarism screening tools before publishing, but it's mostly the reviewers' job.
I don't think it's a bad idea, but it never occurred to me to do this. If you're familiar with the field in which you are reviewing papers, you'll know if something is not new, in which case it is rejected, quite aside from the possibility of the author having lifted some actual text.
Search on Google scholar: http://scholar.google.com/scholar?hl=en&as_sdt=1,5&a...
Do I really have to write IEEE for all violations? http://ieeexplore.ieee.org/search/searchresult.jsp?searchWit...
It's just searching for certain phrases in the text. In this case the first one I tried "This process is defined as binding process. In The immune system" was already enough!
And it's sad.
I haven't seen this before, so maybe that is getting more common these years now.
http://futureofscipub.wordpress.com/open-post-publication-pe... http://www.frontiersin.org/Computational_Neuroscience/10.338... http://retractionwatch.wordpress.com/2013/03/22/brian-deers-...
Professors usually ask their grad students to read the most important and influential papers in the field. When those grad students then turn around and start composing papers of their own, they'll refer to the ones they read, creating a kind of circular feedback cycle. Thus, if you want to find a field's best work, it often suffices to look for highly cited papers first.
It's like Reddit's karma voting system, just a lot slower. :)
A lot of terrible, and even retracted papers have a high citation count, though.
All that process should do is reinforce the status-qua.
EDIT: Oops, I didn't see Cornell there, I take it back.
Buy the way, I'd love to see this research taken one step further, in a humoristic direction: a full statistical analysis comparing the predictive power of academic affiliation in determining acceptance for publication, to that of the contents of the paper itself. :P
At least The European Geosciences Union publishes 15 academic journals that do exactly that. (They're also Open Access, and CC-licensed.)
http://www.egu.eu/publications/open-access-journals/
Some on them, like Atmospheric Chemistry and Physics, are also pretty highly regarded.
Any citation for this? I'm curious, because in physics, although this is often suggested, it has not been generally adopted.
I guess I should've said "this is standard in most disciplines ... that I know about". I'm really surprised that it isn't done in physics!
In computer vision journals, reviewing is only half-duplex blind -- the reviewers can see who you are, but not vice versa.
Second, the OP describes a snob effect, but there is also the reverse: Some referees will be more patient in dealing with badly organized or poorly written papers from a junior researcher. Not necessarily lower standards, but spending more time figuring out the paper, making suggestions for improvement.
1. Time
2. One-way communication
1. You sometimes have to waste a lot of time to get articles published. The way the current process works, it is very easy for referees who are either competitors, incompetent, or simply assholes to effectively filibuster the review process. Even when things go smoothly the process is often agonizingly slow.
2. The peer-review process operates under the obsolete limitation of snail-mail correspondence. There is no two-way communication between the referees. Referees often return either unprofessional or obviously wrong (to everyone but the editor!) comments. If the referees could talk to each other this would be sorted out quickly. It doesn't help that editors are typically very reluctant to reconsider rejection verdicts even when they are based on clear cases of referee error.
How would I address this? Use the #$%@!ing internet people! Journals should abandon this one-way snail-mail process and build a private message board for each paper being reviewed. Referees are invited to post their comments anonymously on the forum. Other referees and, ideally, the authors could then reply to those comments. After a reasonable time period has elapsed, perhaps a couple of weeks, the editor should take stock of what valid concerns remain and reach a verdict.
That said, I review for conferences that are taking advantage of the internet, with author response to reviewer comments, discussions among the reviewers, etc.
http://far.human.cornell.edu/FAR/uploads/webcv/sjc9_webcv.pd...
lists his extensive publications (you hope that most of those got better peer review after his 1982 paper than psychology papers used to). Google Scholar
http://scholar.google.com/scholar?hl=en&q=Stephen+Ceci&#...
helpfully lists some papers by Ceci for which there are free full-text versions that you or I can download for our learning and reading pleasure.
From Jelte Wicherts (another psychologist skeptical about peer review in psychology) writing in Frontiers of Computational Neuroscience (an open-access journal) comes a much more recent set of general suggestions
Jelte M. Wicherts, Rogier A. Kievit, Marjan Bakker and Denny Borsboom. Letting the daylight in: reviewing the reviewers and other ways to maximize transparency in science. Front. Comput. Neurosci., 03 April 2012 doi: 10.3389/fncom.2012.00020
http://www.frontiersin.org/Computational_Neuroscience/10.338...
on how to make the peer-review process in scientific publishing more reliable. Wicherts does a lot of research on this issue to try to reduce the number of dubious publications in his main discipline, the psychology of human intelligence.
"With the emergence of online publishing, opportunities to maximize transparency of scientific research have grown considerably. However, these possibilities are still only marginally used. We argue for the implementation of (1) peer-reviewed peer review, (2) transparent editorial hierarchies, and (3) online data publication. First, peer-reviewed peer review entails a community-wide review system in which reviews are published online and rated by peers. This ensures accountability of reviewers, thereby increasing academic quality of reviews. Second, reviewers who write many highly regarded reviews may move to higher editorial positions. Third, online publication of data ensures the possibility of independent verification of inferential claims in published papers. This counters statistical errors and overly positive reporting of statistical results. We illustrate the benefits of these strategies by discussing an example in which the classical publication system has gone awry, namely controversial IQ research. We argue that this case would have likely been avoided using more transparent publication practices. We argue that the proposed system leads to better reviews, meritocratic editorial hierarchies, and a higher degree of replicability of statistical analyses."