I think ML, at least when I was involved in it, had a much more sane way of approaching publication than other fields (I came from a biology background).
Open science addresses a lot of the pain points that are mentioned in the article. I think we can all agree that having certain magazines that include what can be considered the best works is a good thing; people for better or worse need these instruments to seamlessly judge the base quality of an article. Regardless of my personal criticisms and potential declining quality of a particular magazine it is obviously very helpful to be able to know "hey, at least I can expect some degree of quality given that it was published in this or another magazine" specially as a student.
But this does not mean we ought to gate-keep research. Not only that, but having your publication out can open up a lot of feedback that can be incorporated and addressed. And this is where I go back to my original statement, in ML, it was very common to publish your work to arxiv for people to read through it and I think that greatly improved the speed upon which the field developed. Access is very important.
So I'm all for abolishing "pre-publication peer review".
This vastly underestimates the esoteric nature, experiential knowledge and complexity required to review most research published in the scientific community. The common public cannot be expected to provide non-populist capable review of pharmaceutical level research
But they can do so for deep learning and mathematics? Because in those fields, as I said, pre-publication peer review is not as common.
Of course we need peer-review and neither me nor the paper in the OP say otherwise. We just can't be gate-keeping research because of the review process; the review process needs to be re-thought and that includes the when and why.
Double-blind peer review sounds nice in theory, but I don't think it would hold up in practice, especially in specialized domains.
In those domains, everyone knows everyone else. Even if you hid the name, looking at the writing style, the subject, the choice of analysis software, the type of study. etc would largely give away who the authors are. For example I bet you could figure out that Richard Stallman wrote a particular paper, even if I removed his name from it. Or on HN, how many times have you read a comment and gone, "that sounds like..." even without fully noticing who posted it.
Worse still, many papers are basically unintelligible to anyone outside of an "inner circle".
Every time this is raised, inevitably people will pipe up and say "Oh, you're just not a <specialist in field>, that's why you can't read the paper!"
For example, a new particle discovery paper was being debated in a forum and several people raised the issue that the paper was impenetrable, and was followed with the usual retorts of "you're not a particle physicist!". There was also a response from a particle physicist who works at CERN. He basically said that he himself could not understand the paper, despite working on a similar experiment on a different instrument. His office was down the hall from the office where the paper was written, but each team has their own terminology, jargon, and instrument-specific calibrations. Essentially, they can review their own papers, but nobody else can meaningfully validate it, or even understand it in a useful way!
The problem with how modern science is performed is that it rewards the act of publishing a paper, not the utility, validity, or comprehensibility of the paper.
In fact, writing papers to superficially look more 'serious' is rewarded, so papers that start off quite readable are often edited to be more obtuse and full of unnecessary symbols, terminology, or equations.
I noticed this in Computer Science, where earlier papers in the field from the 60s and 70s were very readable, but then slowly started getting filled with Greek symbols and obscure terminology for no reason. There is no long history of Ancient Greek Computer Science that needs to be preserved for continuity with papers published hundreds of years ago! This is pure intellectual wankery, and has resulted in people like me unable to follow some modern CS papers on topics that I'm already familiar with.
PS: Imagine the quality of "open source libraries" if everyone got accolades for publishing their code to GitHub, NuGet, or NPM, but no reward for anything else. No in-place updates. No direct usage. No pull requests. No feedback other than a one-time rubber stamp. No issue tracker. Nothing. Just... publish your code once and walk away, you get a cookie. Doesn't even matter if it compiles. Doesn't matter if it's pseudo-code you made up yourself. Just get on there, and you get something. But you get nothing for anything else.
> Worse still, many papers are basically unintelligible to anyone outside of an "inner circle".
Right now we have papers simply filled with nonsense (https://link.springer.com/article/10.1007/s12517-021-08021-2) and papers on midi-chlorians (https://www.livescience.com/59927-midi-chlorians-paper-accep...). Even without the needlessly over-complicated writing, some papers may really only be able to be meaningfully reviewed by a small number of experts, but that won't be a huge number. Really I'd be happy if we managed to catch the papers any child in grade school could spot as phony.
Maybe we shouldn't be asking if peer-review is a good idea based on it's performance so far until we give it a serious effort.
These perverse incentives (publish or perish; using big words to appear more serious) are far bigger problems than peer review being single-blind or not.
Yesterday I started looking for a token embedding method, and my search turned up word2vec and GLoVe. After skimming both papers I am leaning toward word2vec solely because it was easier for me to understand the big idea, even though GLoVe may perform better.
Personally I would rather see code than a LaTeX-typeset equation. With typeset equations, the author has plausible deniability that you didn't interpret their notation just right, or oops that was supposed to be a boldface capital X, which implies a different data type. But if you provide code, anyone can run it and independently verify the results.
Maybe the fact that no one can independently verify your equations/pseudocode is a feature, not a bug.
So, I could be charitably described as mid-career, and I had this discussion a few times a couple decades ago -- in which I took the position you have. It's not an unreasonable position.
I was a PC member for what is now called NeurIPS, and I remember one particular disagreement with a conference co-chair who advocated blinded reviews.
I have since come to believe that the above objections to blinding are mistaken, and that things should generally be moved to double-blind where at all possible.
What happened is, I participated in blinded review processes (for proposals and for conferences) and they turned out to be different. Just that level of de-coupling -- where you aren't 100% sure that the author/proposer is who you suspect -- turns out to matter a great deal psychologically. It changes the framing. And more important, the results seem better.
Know of a few people that have gotten dismissive comments to review better the work from X paper of Y author... when the person submitting is that same one.
The field(s) I'm in use double blind and it's broken for a host of other reasons. It's really hard to build off a string of work for example. Students (eg the ones writing papers) are often bad at the anonymization requirements. On the other side, the reviewers can often form an opinion on likely authors (whether correct or not). And then up one level (at the editorial / program committee level) that anonymization goes away since they need to check for reviewer conflicts, etc.
Also in fields small enough that everyone knows each other, you know exactly what the rest is working on because they collaborate often and there aren’t that many reviewers
Perhaps, surprisingly, it might go both ways: when I know who is on the PC, I think I can tell which person wrote which review, because I both know their work and met them personally. In fact, those people on the PC is why I am submitting to a venue in the first place, because I want reviewers who will easily understand my work.
I've been in academic research for 22 years. Based on my experience, I'll go out on a limb and say that even single-blind reviews are to be discarded.
This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Programme Committees are relieved to find a negative review, however unfair or off-kilter it is, because more rejected papers will decrease the acceptance ratio of the conference, hence make it appear more competitive.
Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review.
The one experience I had with peer review before leaving academia was actually pretty nice. One of the reviewers was a bit of a hardass, but they made good points and I think made the paper much better.
As a consumer of papers now, I find peer review pretty useful. I can’t read everything that comes out. Even if I just read abstracts, I need some sort of filter. Publication in a top tier journal raises the probability that egregious errors haven’t been made and that the paper is worthy of my time.
There would probably be a need for a mechanism to ensure objectivity. E.g., if Reviewer A rejects Reviewer B, there may be a higher chance Reviewer B will reject Review A's future work, just out of spite.
I have no serious involvement in academia. What is the proportion of work that gets reviewed by someone with an axe to grind? I don’t doubt that it happens regularly but I have no conception of the magnitude of the problem.
In experimental fields, when submitting to Nature / Science / Cell, it's likely that your work ends up getting reviewed by someone who sees you as a competitor (this mentality drives me mad).
Hence, they will sometimes try to make sure you don't publish or at least get downgraded to sister journals, such as Nature Communications.
Another problem are editorial decisions. Great work is often desk rejected because it's judged not to be significant or interersting enough (usually the author comes from a less prominent university).
For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years...
>For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years...
This is interesting. Has there been a write up on this saga anywhere?
In my experience, about 30-40% of the reviews I've seen are a mix between "axe to grind" and "I have no idea what I'm talking about". Some are extremely blatant, such as reviewers "suggesting" you should cite a certain paper that isn't really related to your work. Others are very strict with the "this is just practical knowledge so it doesn't advance the science", and then there's the set of "I didn't understand the paper so I'm going to criticize it based on my misconceptions of it and reject it".
I'm so glad to be out of that world, to be honest.
My favorite anecdote is from Adam Grant, who once got a snarky reply from a reviewer that "the author would do well to familiarize themselves with the work of Adam Grant."
A lot of authors are also very sensitive to negative feedback. How many times did I hear "the reviewer doesn't like us" when the feedback was legitimate.
> Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review.
In some fields like AI knowing the exact authors is also quite common. Because the vast majority of researchers are employed by just a handful of big labs, the reviewers (and sometimes organizers) with experience in the subfield are employed by exactly the same lab(s). So they already know each other and each other's research anyway. Add to that that citations and prior work usually give away the set of authors too.
The opposite issue also - conflicts of interest can automatically disqualify reviewers, and if you are conflicted with all of microsoft research, or all of google research, etc., then the number of qualified reviewers shrinks dramatically.
In my experience, yes, essentially all self-reporting. They might do something like automatically conflict your email address domain (*.mit.edu), which is fun when random unrelated people sign up with gmail addresses.
Not only this, but people publish their papers months ahead on arxiv. Big labs have big publicity and thousands will read their papers. They can often do more impactful work too because the massive amount of compute that they have. There's a good chance that if the reviewers are qualified to review the material they have already read the paper. (or a public version can be found)
One thing I really wish is that we ensured that code works properly. I don't expect a full training but at least verify checkpoints.
> What's the difference between posting stuff on arxiv or uploading a pdf to your blog?
There isn't. My point is that the papers aren't blind because they've been read before. Publishing onto your blog would have the same result. I'll note that this isn't common practice in other research communities.
> Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
I don't see how this could even happen. Let's even say you are a no name lab and get accepted, you're in a list of hundreds or even a thousand other papers. There's publicity, yes, but it isn't massive.
Let me put it this way. If you are a grad student at MIT or Stanford you will get hundreds of twitter followers relatively quickly. These followers are going to see you posting links to your papers. You have a larger network effect just by being at a larger and more prestigious school. That is the main advantage of prestigious schools and I don't think there's anything necessarily wrong with that.
But we do have to admit that there are factors that are highly important here beyond merit based. Being at MIT or Stanford does not mean your paper has more merit than a no name school or lab. That no name lab will not have the same reach, which results in fewer citations (our main metric of relevancy), and fewer researchers will be able to expand upon the work.
The major problem here is that we're pretending that this doesn't happen. I don't know how to solve this issue or even know if it can be solved. But it is really dumb to go around saying everything is merit based and not acknowledge the other factors at play here. It is pretending that there's a stronger signal in the noise than there is.
Agree. I used to think double blind was an obvious requirement for a good peer review system. Your “theatre” take is aptly harsh I’d say.
Also hard agree on publishing reviews. It seems to be the only viable modification of peer review that we have.
My latest experience with peer review was just bad. No matter the conclusion they drew, there was just a clear lack of quality and understanding in their thoughts and effort. And this is not infrequent.
More generally, such a modification has knock on effects that probably warrant some thought on what we do with our publication practices. Published reviews will probably require more work from reviewers who are already time poor. Better reviews, hopefully, but fewer of them and fewer papers?
Counting a good review as a citable publication might be worth considering, as essentially mini idiosyncratic literature reviews. Perhaps in combination with Registered Reports[1], where the proposal is published and reviewed before the study/work is done, in which case the reviewer is closer to symbiotic co-publisher.
Either way, it seems that for both authors and reviewers, who are effectively the same people and both rely on peer-review as a guarantee of the value of their work, that the actual work of peer review needs to be taken more seriously and not conceived of as some sort of feudal aristocratic gentlemen’s duty.
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper.
Never heard this argument, but having been under many double blind peer reviews in the last 10 years, I can sign this.
Interestingly, in my last paper for PLOS One, one of two reviewers chose to lift anonymity and specifically his comments were on-focus, supportive and substantial.
Of course, in practice, this has the problem that it is already hard to get reviewers, because nobody has time anymore. Removing the anonymity of reviewers leads to the judge becoming open to judgement themselves, and many would not like that. Furthermore, there might be severe repercussions if the reviewed is more powerful than the reviewer, especially in authoritarian societies. But maybe one should just exclude these societies from peer review, and let them do their own thing.
I think peer review has to become a market place. Let everyone choose for themselves which paper they would like to review, and there is both positive and negative credit for both reviewers and authors. The pool of reviewers of a paper shares 25% of the credit that the authors of the paper get. This way you are incentivised to review even outlier papers, because if they are successful you get a substantial amount of credit for them. How the credit is distributed among the reviewers should also incorporate a time factor, so that reviewers rushing to a paper that is already successful don't get nearly as much credit as a lone early reviewer.
That's why I use the term "credit". It needs to have real-world value of some sort. I don't think just paying people is the solution, that would just mean the richer you are, the more reviews you get, and the better your reviews are. You can argue they should be paid via a neutral source of money, but there is no such thing.
If the problem of gaming the system could be gotten around, something very similar to internet comments could work - anyone can post a public review of anyone's work, in which case reputation would become the currency. Successfully criticizing someone with higher reputation than yourself would gain more status than praising them, but also come with more risk. It would also make it a lot less meaningful to get those 'not new ideas lol' comments that others on this thread have complained of.
I quite like the fact that whatever someone says on the internet now, the first comment is usually the best-argued counterpoint. It seems like a dynamic that could be brought to science quite well. I may not have the time to parse an entire paper in a field unfamiliar to me, but if I come across it and the top comment is 'there's a math error in table 6 that throws paragraph 4 into doubt', I can certainly verify that for myself. That sort of 'peer' review would make science much more accessible to the interested layperson.
I can imagine some unintended consequences from this. If I'm a hard-going reviewer, people might choose to avoid me, and I'll be compensated less. Conversely, if I'm easy-going, people might find ways to get their papers to me.
I suppose there can be some other layer of reviewer meta-review to account for this, much like the role 'acceptance rate' has come to have for journals and conferences. But, following Goodhart's law, even that has come to be gamed now that a meaning - a proxy for journal prestige or quality - has been placed on it.
All of the process creates incredible gatekeeping effects. Science would be much more robust if published github-style: post your results, engage with the community, fix issues, respond to critics, and eventually wrap it up, but always have the priority date of your original work recorded so that there's no dispute as to who invented it first.
> This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
That misses the whole point of anonymous reviews, or even anonymous voting.
You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved.
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper.
Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
If you pick a journal to publish your paper, which means you explicitly want the paper's editors to go through each and every single line of text you wrote to poke holes, but in the end once those holes are picked you complain that the journal you picked is not the right one for your flawless paper and that they all suck and their problem is that they don't understand your genius, what does this say about you and your work?
When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Your comment reads a whole lot like "the fox and the grapes".
First, journals and conferences are different. I would expect more valuable feedback from journals than from conference reviews.
Second, journals are overloaded, the journal reviewers unpaid, and frankly, they are not doing a good job anymore in reviewing work. Feedback that amounts to "maybe submit first to a conference" is not that valuable. Effectively, you have to have the idea, then execute it perfectly, then do a major marketing effort to popularise your work, and why exactly would you at the end of this still submit to a journal? Given that the journal either wants the rights to your work, or charges you for the pleasure of publishing with them. At this point, a journal is just another marketing step, and for marketing you pay, I guess.
> You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved.
This door is already open, and impossible to close [1]. For any reviewer closely familiar with the area the paper is in it is trivial to guess which group the paper came out of 9 times out of 10. Style, approach, citations, and just knowing what the other people in your area are working on all contribute. To achieve any sort of blindness you have to go to reviewers who don't know that area as well... which, I suppose, does decrease the potential for corruption--at the cost of the reviewers not understanding the paper as well.
> Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
Even a well written paper often requires some effort to understand, due to conceptual novelties, mathematical complexities, etc.
Unfortunately there's little incentive for reviewers to put in much mental effort. Some of them will. Many won't.
> When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Because the "problems" are often bullshit. You're assuming the reviewers are acting in good faith and finding legitimate problems with the paper, but they're often not--usually not out of malice, just laziness. There's no incentive to put any effort into a review above the minimum to convince the editor that your review is plausible. This doesn't take much since the editor is probably only going to glance briefly at the reviews. If you can finish your review in an hour instead of a day you can spend the rest of the day writing your own papers or doing other work that actually furthers your career.
[1] In all but the largest, most active areas, where there are so many people working that nobody can keep up.
It would be interesting to see if there is a distinction between review and acceptance statistics depending on timeframe.
For example, in fields where the peer-review is done primarily by academics are there better reviews during academic breaks vs. say, exam season? Does this translate to more accepted work because the reviewer can spend more time digesting the content?
>Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
Yes, but there is unfortunately a wide variability in the knowledge of reviewers. It's not uncommon to have reviewers directly contradict each other about the shortcomings of the paper. E.g.,
Reviewer 1: "Section A needs to be condensed, this level of background is unnecessary. Readers will find this as common knowledge."
Reviewer 2: "Section A should be expanded as all readers may not be knowledgeable in this area."
Particularly for grad students, this can make the process feel arbitrary and the feedback of limited value. There also can be an inordinate amount of stylistic review comments (e.g., "however" should be changed to "nonetheless"). These often come across as the reviewer feels the need to critique something because they don't have sustentative critiques. Good critiques strengthen a paper, bad feedback seems to do nothing but check the process boxes.
I actually agree with the OP that one potential solution is to encourage reviewers to decline reviewing papers that they don't have the requisite knowledge to provide good feedback. Unfortunately, this decision is often only offered based on reading the title and abstract.
In some publications (mostly CS conferences), there are associate chairs or 'meta-reviewers' whose sole job is to manage the peer review for a paper. They will read the reviews and if there isn't consensus, they'll ask the reviewers to read the other reviews and have a discussion. It is usually 3 reviewers and 1 meta-reviewer per paper, so there are often 2-1 splits.
The meta-reviewer doesn't just send the reviews to the authors with no commentary. They write a summary of the reviews with a recommendation about what to do next. If there are contradictions and no consensus is reached, it is their job to decide which concerns the authors need to respond to in order to get it published. They aren't bound by the reviews, they can disregard one if they feel like it is unfair, although sometimes the rules say they must get another reviewer to write a 4th review.
CS publication culture is a large part of the problem. The average CS paper is small (in terms of person-years put into the project), so people submit many papers. Most papers are submitted to conferences first, which usually accept or reject after a single round of reviews. When the typical outcome is rejection and resubmission to another conference, the negative aspects of peer review become prominent.
Over the years, I've drifted to the biological side of bioinformatics methods. The papers I write are similar to CS papers, but the projects are bigger, and I no longer submit papers to CS conferences. My typical experience with peer review is accept after 1-3 rounds of revisions. Hence I get to see more of the positive side, where the reviewers act as editors and their suggestions improve the paper.
Of course, it's possible to get a CS-like experience with journals by being ambitious. If you try your luck by submitting the paper to prestigious journals it's unlikely to get in, your typical outcome is a desk rejection or rejection after a single round of reviews. Then you get to see more of the negative side of peer review. But I'm just not interested in playing that game.
This is a very thoughtful take on the issue. My personal experience corroborates with this - CS conferences have more of these culture issues than mathematics journals, where the reviews tend to more thoughtful, and at the same time, considerate.
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper.
This problem goes even deeper, because it not only applies to papers but also to research grant applications. I even had once got a (positive) recommendation for one of my applications, but I could see from the statement that the reviewer did not really understand what was important.
Not meant in a snarky way, but that almost feels to me as if the process worked. If they application committee did not see the value in the proposed work, it indicates they probably don't have much need for it. If they don't have much need, they shouldn't fund it.
That's not to say the proposal isn't worthwhile, just that it didn't find the right audience. A contractor may have a whiz-bang new method of snow removal, but it doesn't mean people in Phoenix will want it (or even understand it).
Your parent is not arguing against peer reviews, but against blind and double blind reviews.
They are actually proposing that peer reviews are credited:
> Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
And yes, I do think reviewers should be credited. Because it's work, for one, that needed to be done so the paper could be published, and because it would probably indeed incentivize reviewers to do the work correctly.
However, it could incentivize them to reject more papers, in fear of accepting a bad paper and be publicly advertised as a reviewer who reviewed and accepted it.
While knowing who the authors are can cause bias, hiding behind anonymization (as a reviewer) can as well. I think publishing the reviews (and who reviews) would help a lot (some fields do this). Science is really a dialog. The authors are putting forth part of the dialog and the reviewers are reflecting on it and giving their opinions. Hopefully publishing names next to reviews would minimize crappy reviews (often by the most senior folks). However the other problem is there are not enough qualified reviewers to go around and I could see many opting out of putting their name next to a review. But, we're likely better off.
There also often isn't a clear cut line between "good" and "bad". Almost all papers have flaws and putting those out in the open for others to improve upon, or at least acknowledge, would help move humanities knowledge forward.
> I have a partial solution: researchers “publish” papers to arXiv or similar, then “submit” them to the journal, which conducts peer review. The “journal” is a list of links to papers that it has accepted or verified.
This made me curious about how arxiv operates. It seems that you require endorsement to become a registered author, and the submissions are moderated[1][2].
This already seems sufficient to keep out spam and clearly junk science. What is the value add of official(?) journals?
No? Maybe I misinterpreted your comment. I was answering your question
> Where is that quote from?
by providing a link to the source of the quote.
Ironically, maybe the person to whom you first replied was actually intending to respond to the comment you just linked instead of making a top-level comment.
Submissions are moderated, but depending on the field, anything reasonable-looking may be accepted. For example, there is more than one "proof" of the Riemann Hypothesis in the arXiv. But the moderation system certainly keeps out spam.
Edit: Here's a proof of the Riemann hypothesis submitted just last week: https://arxiv.org/pdf/2209.01890.pdf You can have a little fun exploring "proofs" of famous theorems on the arXiv over the years.
"Official" (aka well-known/have a high "Impact Factor") journals give you a stamp of approval that help you get recognised in your career, and in some fields are where your peers go to stay up to date on new potentially relevant research.
(Disclosure: I volunteer for https://plaudit.pub, a non-profit that aims to separate that from the publication process.)
The standard isn't peer review, it is unusual. https://xkcd.com/882/ applies to science and will get you publishes that green jelly beans cause cancer with no need to mention all experiments that got expected results.
Which is why reproducibility should be the most important topic.
There's lots of science that can't be reproduced before publishing or where reproducing doesn't help:
* Theoretical work
* Computer simulations (re-run the simulation? Well that's not going to detect most issues. Re-create the program from scratch? expensive and hard to make a re-write meaningfully different systematically)
* Cosmological observations (can't re-run that supernova!)
* Medical case studies (can't just find a new patient)
* Large scale studies (build a second LHC? Do results based on the Framingham Study have to kick off another 30 years of data collection before they publish)
In fact it's only small and self contained experiments where this is really practical.
>Computer simulations (re-run the simulation? Well that's not going to detect most issues
It would have detected the fraud going on at ICL with their covid model that was used by the governments of the world to set the course of pandemic response. I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here. Running the same simulation with the same inputs produced wildly different results between runs. That would have been an easily detected red flag but because no one was able to do so until long after the pandemic response was set in motion and changes would have been politically uncomfortable for the powers that be, we ended with a terrible response that many still cannot bring themselves to admit was both poor and instigated by a terrible model that should have been easily detected as being garbage.
> Among the two popular models, we found that the ICL model is more transparent and reproducible compared to the IHME model. The former sometimes over-predicted future deaths while the latter clearly under-predicted post-peak deaths. Both models predicted the timing of peaks reasonably well using data until one week prior. The ICL model produced a much wider band of uncertainty for New York state, possibly because the pattern did not conform well with their internal training data used from European countries.
Nature (on 08 June 2020) reported "a computational neuroscientist has reported that he has independently rerun the simulation and reproduced its results. And other scientists have told Nature that they had already privately verified that the code is reproducible." - https://www.nature.com/articles/d41586-020-01685-y citing the CODECHECK certificate at https://zenodo.org/record/3865491 from May 29, 2020.
You should also be asking why public health is so under-funded world-wide that many countries looked to the ICL model - which in turn was a repurposed influenza model - rather than use local expertise or be able to use better validated models.
> Running the same simulation with the same inputs produced wildly different results between runs.
As I recall, the replication issues were the normal issues related to parallelization and random number generation, and not things which greatly affected the results. There were a number of HN threads on the topic back in 2020. Eg, https://news.ycombinator.com/item?id=23212268 , which linked to a couple of those issues and their resolution.
> I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here.
If it’s not reproducible then assertions such as “the science is settled” and “trust the science” are completely inane and certainly should not be used as policy cudgels.
I like the basic principle of peer review, but not how it's done. I understand why it was done that way for so long, but we have the technology to do better now.
I read an interview with a TV show runner who said you can come up with any crazy plot twist thing, and within 10 minutes of the first episode airing some guy on the internet has already figured the whole thing out. I think this phenomenon could be put to good use.
Just publish your papers publicly with a comments section. If there's problems with it people will tear it apart. Source their work and let the world help you improve your work.
The way Journal of Open Source (JOSS) - https://joss.theoj.org - does the reviewing process is good. Everything is out in the open, reviewers and authors names, reviews, responses, discussions etc. The whole process is out there for everyone to see.
having worked in academic publishing, i've concluded that peer review would perform best if it were embedded in the very act of reading (so post-publication, non-formalized kind of peer review) instead of having it be a formalized process riddled with issues and cracks as it is currently.
it would enable faster and more seamless communication, academics wouldn't be burdened with extra volunteer work, and publishers would still be able to curate works as the authors suggest (+ there are other ways to do it).
sadly my impression is that it's simply too entrenched in the publishing process -- and publisher's raison d'etre to a large extent -- for the publishers to relinquish control of it
As someone who is inside the academic publication loop, I am in favor of double-blind, non-public peer reviews because I don't want to invite 4Chan into my daily work.
I believe that double-blind, non-public reviews do a great job at protecting those in a more precarious situation: grad students are not judged by their lack of publication record, there is no permanent public record of all those times they got rejected, and reviewers can be both more honest and more certain that a rejection won't lead to a 4Chan/Twitter mob coming for them.
According to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents". I'd be weary of implementing any changed that would make academia even more hostile to women and minorities.
I just don't see how double-blind peer review can work. To properly peer review a manuscript you often have to read the papers cited (because often details of the methods of the current manuscript are described in previously published work). These citations are almost always to the authors' own work. So it is trivial for any peer reviewer to figure out who wrote the manuscript they are reviewing, even if their names are blinded.
I hear this a lot and I just don't buy it. The whole idea of science is that you can describe methods in text, send that text around the world, and have other people use those same methods to do their own independent studies. So even if you know that lab X really likes using a particular method, it could be someone from lab Y who read the papers from lab X and wanted to use them as well.
Sure, people like citing themselves. Editors of double-blind journals already are supposed to desk reject work that says something like "in our prior study (Smith 2020)..." They should also desk reject obnoxiously self-referential citations --- like "Great advances have been made in method X (Smith 2016, 2017a, 2017b, 2018, 2019a, 2019b, 2020, 2021)"
I'm sure different fields have different dynamics, and in some what you describe here might be viable and accurate.
But my own impression and experience is that your statement is naive. The key thing I think you miss is that researchers/labs/groups develop fairly specific interests, expertise and "research programs". So taking any paper and looking at the specific area or phenomenon being probed, the techniques/methods used, and the questions or interests driving the research and its analysis, and you'll get a pretty unique "fingerprint" of the researchers that anyone in the same field should be able to identify with pretty good accuracy.
The naivety of your statement I think is this idea that experiments or acts of research are kind of plucked from the air by any who are interested and then the method or technique required is simply downloaded and executed like software.
Researchers aren't generic like computing hardware tries to be. Far from it. A researcher and their research or ability to do research is much more like a tree ... grown organically through experience and pretty fixed and inflexible once it's mature.
None of this is to be really critical of your statement. In fact, I think it's very useful (provided I'm onto something and not merely bitter) to state the ideal practice of science and contrast it with reality.
That we can't do double-blind review is almost certainly a bad thing. And it also tells us something about what kind of "science" (here the global collective enterprise) we have. I think I'm correct about the "fixation" of researchers. But I also think that the culture of science we have encourages this process. I've seen it myself: researchers trying a minor change in field and technique (very minor) and hitting massive resistance from the funding agencies. The conclusion we all drew from seeing it was that the techniques you can get funding for are mostly fixed by your first 10 papers or so. This could be changed and research could get closer to what you describe and I think that would be a good thing. But, as I stated in another comment, to get there requires change beyond the peer review process.
Something else it tells us is that our science operates in a relatively closed and inward looking cultural ecosystem, which indicates to me some degree of corruption is almost guaranteed. If we don't have good diversity and competitive and corrective interactions across space/culture, then, as the "coffins in the ground" adage tells us, we have to rely on diversity across time/generations. And indeed researchers know this just from experience. Someone winning a Nobel prize is like carving their thought in stone ... it takes time for us to move on.
An interesting study in this regard might science during the USSR. I've read glancingly that mathematics research progressed differently between the West and USSR and that, for this interested, cross-pollination was fruitful. An analysis of this could prove insightful. I wonder whether something similar is going on in China at the moment.
> I am in favor of double-blind, non-public peer reviews because I don't want to invite 4Chan into my daily work.
At least for mathematics papers, the community who works in a particular area, and is thus actually able to do a review the paper, is pretty small. Since, on the other hand, the style that you use for mathematical proofs is like a fingerprint, double-blind peer reviews are next to impossible.
Having been on both sides of the peer review process in my field (Theoretical Physics), I say we should completely abolish it. Just publish the papers on the preprint archives. Some will be bad, some will be good, and you will know the impact of a paper after 10 years, nothing different from how it is now. Currently the very big majority of the effort in the reviews (on both sides of the process) is wasted time.
95 comments
[ 2.8 ms ] story [ 173 ms ] threadOpen science addresses a lot of the pain points that are mentioned in the article. I think we can all agree that having certain magazines that include what can be considered the best works is a good thing; people for better or worse need these instruments to seamlessly judge the base quality of an article. Regardless of my personal criticisms and potential declining quality of a particular magazine it is obviously very helpful to be able to know "hey, at least I can expect some degree of quality given that it was published in this or another magazine" specially as a student.
But this does not mean we ought to gate-keep research. Not only that, but having your publication out can open up a lot of feedback that can be incorporated and addressed. And this is where I go back to my original statement, in ML, it was very common to publish your work to arxiv for people to read through it and I think that greatly improved the speed upon which the field developed. Access is very important.
So I'm all for abolishing "pre-publication peer review".
Of course we need peer-review and neither me nor the paper in the OP say otherwise. We just can't be gate-keeping research because of the review process; the review process needs to be re-thought and that includes the when and why.
In those domains, everyone knows everyone else. Even if you hid the name, looking at the writing style, the subject, the choice of analysis software, the type of study. etc would largely give away who the authors are. For example I bet you could figure out that Richard Stallman wrote a particular paper, even if I removed his name from it. Or on HN, how many times have you read a comment and gone, "that sounds like..." even without fully noticing who posted it.
Every time this is raised, inevitably people will pipe up and say "Oh, you're just not a <specialist in field>, that's why you can't read the paper!"
For example, a new particle discovery paper was being debated in a forum and several people raised the issue that the paper was impenetrable, and was followed with the usual retorts of "you're not a particle physicist!". There was also a response from a particle physicist who works at CERN. He basically said that he himself could not understand the paper, despite working on a similar experiment on a different instrument. His office was down the hall from the office where the paper was written, but each team has their own terminology, jargon, and instrument-specific calibrations. Essentially, they can review their own papers, but nobody else can meaningfully validate it, or even understand it in a useful way!
The problem with how modern science is performed is that it rewards the act of publishing a paper, not the utility, validity, or comprehensibility of the paper.
In fact, writing papers to superficially look more 'serious' is rewarded, so papers that start off quite readable are often edited to be more obtuse and full of unnecessary symbols, terminology, or equations.
I noticed this in Computer Science, where earlier papers in the field from the 60s and 70s were very readable, but then slowly started getting filled with Greek symbols and obscure terminology for no reason. There is no long history of Ancient Greek Computer Science that needs to be preserved for continuity with papers published hundreds of years ago! This is pure intellectual wankery, and has resulted in people like me unable to follow some modern CS papers on topics that I'm already familiar with.
PS: Imagine the quality of "open source libraries" if everyone got accolades for publishing their code to GitHub, NuGet, or NPM, but no reward for anything else. No in-place updates. No direct usage. No pull requests. No feedback other than a one-time rubber stamp. No issue tracker. Nothing. Just... publish your code once and walk away, you get a cookie. Doesn't even matter if it compiles. Doesn't matter if it's pseudo-code you made up yourself. Just get on there, and you get something. But you get nothing for anything else.
Picture that.
That's modern science.
Right now we have papers simply filled with nonsense (https://link.springer.com/article/10.1007/s12517-021-08021-2) and papers on midi-chlorians (https://www.livescience.com/59927-midi-chlorians-paper-accep...). Even without the needlessly over-complicated writing, some papers may really only be able to be meaningfully reviewed by a small number of experts, but that won't be a huge number. Really I'd be happy if we managed to catch the papers any child in grade school could spot as phony.
Maybe we shouldn't be asking if peer-review is a good idea based on it's performance so far until we give it a serious effort.
Yesterday I started looking for a token embedding method, and my search turned up word2vec and GLoVe. After skimming both papers I am leaning toward word2vec solely because it was easier for me to understand the big idea, even though GLoVe may perform better.
Personally I would rather see code than a LaTeX-typeset equation. With typeset equations, the author has plausible deniability that you didn't interpret their notation just right, or oops that was supposed to be a boldface capital X, which implies a different data type. But if you provide code, anyone can run it and independently verify the results.
Maybe the fact that no one can independently verify your equations/pseudocode is a feature, not a bug.
I was a PC member for what is now called NeurIPS, and I remember one particular disagreement with a conference co-chair who advocated blinded reviews.
I have since come to believe that the above objections to blinding are mistaken, and that things should generally be moved to double-blind where at all possible.
What happened is, I participated in blinded review processes (for proposals and for conferences) and they turned out to be different. Just that level of de-coupling -- where you aren't 100% sure that the author/proposer is who you suspect -- turns out to matter a great deal psychologically. It changes the framing. And more important, the results seem better.
You can't do much.
This is leading to quite irresponsible reviews. Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper. Programme Committees are relieved to find a negative review, however unfair or off-kilter it is, because more rejected papers will decrease the acceptance ratio of the conference, hence make it appear more competitive.
Double-blind reviews are just peer-review theater. It is quite simple to guess which group the paper is from. It is difficult to guess the exact set of authors, but reviewers who are out to settle a score or to discard dismissively just need to know a ballpark of where the paper is from in order to stonewall with an irascible review.
As a consumer of papers now, I find peer review pretty useful. I can’t read everything that comes out. Even if I just read abstracts, I need some sort of filter. Publication in a top tier journal raises the probability that egregious errors haven’t been made and that the paper is worthy of my time.
Hence, they will sometimes try to make sure you don't publish or at least get downgraded to sister journals, such as Nature Communications.
Another problem are editorial decisions. Great work is often desk rejected because it's judged not to be significant or interersting enough (usually the author comes from a less prominent university).
For example, the original CRISPR article was desk rejected by most journals and the fact that the author's affiliation was a no-name research centre in Alicante (Spain) surely played a role. This delayed CRISPR getting widely known by a few years...
This is interesting. Has there been a write up on this saga anywhere?
I'm so glad to be out of that world, to be honest.
In some fields like AI knowing the exact authors is also quite common. Because the vast majority of researchers are employed by just a handful of big labs, the reviewers (and sometimes organizers) with experience in the subfield are employed by exactly the same lab(s). So they already know each other and each other's research anyway. Add to that that citations and prior work usually give away the set of authors too.
One thing I really wish is that we ensured that code works properly. I don't expect a full training but at least verify checkpoints.
What's the difference between posting stuff on arxiv or uploading a pdf to your blog?
> Big labs have big publicity and thousands will read their papers.
Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
There isn't. My point is that the papers aren't blind because they've been read before. Publishing onto your blog would have the same result. I'll note that this isn't common practice in other research communities.
> Peer review is supposed to circumvent the problem of a marketing budget being a factor in whether your paper is deemed relevant or not.
I don't see how this could even happen. Let's even say you are a no name lab and get accepted, you're in a list of hundreds or even a thousand other papers. There's publicity, yes, but it isn't massive.
Let me put it this way. If you are a grad student at MIT or Stanford you will get hundreds of twitter followers relatively quickly. These followers are going to see you posting links to your papers. You have a larger network effect just by being at a larger and more prestigious school. That is the main advantage of prestigious schools and I don't think there's anything necessarily wrong with that.
But we do have to admit that there are factors that are highly important here beyond merit based. Being at MIT or Stanford does not mean your paper has more merit than a no name school or lab. That no name lab will not have the same reach, which results in fewer citations (our main metric of relevancy), and fewer researchers will be able to expand upon the work.
The major problem here is that we're pretending that this doesn't happen. I don't know how to solve this issue or even know if it can be solved. But it is really dumb to go around saying everything is merit based and not acknowledge the other factors at play here. It is pretending that there's a stronger signal in the noise than there is.
Also hard agree on publishing reviews. It seems to be the only viable modification of peer review that we have.
My latest experience with peer review was just bad. No matter the conclusion they drew, there was just a clear lack of quality and understanding in their thoughts and effort. And this is not infrequent.
More generally, such a modification has knock on effects that probably warrant some thought on what we do with our publication practices. Published reviews will probably require more work from reviewers who are already time poor. Better reviews, hopefully, but fewer of them and fewer papers?
Counting a good review as a citable publication might be worth considering, as essentially mini idiosyncratic literature reviews. Perhaps in combination with Registered Reports[1], where the proposal is published and reviewed before the study/work is done, in which case the reviewer is closer to symbiotic co-publisher.
Either way, it seems that for both authors and reviewers, who are effectively the same people and both rely on peer-review as a guarantee of the value of their work, that the actual work of peer review needs to be taken more seriously and not conceived of as some sort of feudal aristocratic gentlemen’s duty.
~~~~~~~~
[1] https://www.cos.io/initiatives/registered-reports
Never heard this argument, but having been under many double blind peer reviews in the last 10 years, I can sign this.
Interestingly, in my last paper for PLOS One, one of two reviewers chose to lift anonymity and specifically his comments were on-focus, supportive and substantial.
The open discussions not only benefit the peer review process but also act as context and learning material for others.
Of course, in practice, this has the problem that it is already hard to get reviewers, because nobody has time anymore. Removing the anonymity of reviewers leads to the judge becoming open to judgement themselves, and many would not like that. Furthermore, there might be severe repercussions if the reviewed is more powerful than the reviewer, especially in authoritarian societies. But maybe one should just exclude these societies from peer review, and let them do their own thing.
I think peer review has to become a market place. Let everyone choose for themselves which paper they would like to review, and there is both positive and negative credit for both reviewers and authors. The pool of reviewers of a paper shares 25% of the credit that the authors of the paper get. This way you are incentivised to review even outlier papers, because if they are successful you get a substantial amount of credit for them. How the credit is distributed among the reviewers should also incorporate a time factor, so that reviewers rushing to a paper that is already successful don't get nearly as much credit as a lone early reviewer.
I quite like the fact that whatever someone says on the internet now, the first comment is usually the best-argued counterpoint. It seems like a dynamic that could be brought to science quite well. I may not have the time to parse an entire paper in a field unfamiliar to me, but if I come across it and the top comment is 'there's a math error in table 6 that throws paragraph 4 into doubt', I can certainly verify that for myself. That sort of 'peer' review would make science much more accessible to the interested layperson.
I suppose there can be some other layer of reviewer meta-review to account for this, much like the role 'acceptance rate' has come to have for journals and conferences. But, following Goodhart's law, even that has come to be gamed now that a meaning - a proxy for journal prestige or quality - has been placed on it.
That misses the whole point of anonymous reviews, or even anonymous voting.
You want your work to stand by itself. Otherwise you're blasting open a very corrupt door where appeals to authority, petty politics, careerism, and funding play a role in how a paper is approved.
> Instead, in CS, there is a tendency to hide behind an abrasive negative review when the reality is that the reviewer does not understand the paper.
Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
If you pick a journal to publish your paper, which means you explicitly want the paper's editors to go through each and every single line of text you wrote to poke holes, but in the end once those holes are picked you complain that the journal you picked is not the right one for your flawless paper and that they all suck and their problem is that they don't understand your genius, what does this say about you and your work?
When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Your comment reads a whole lot like "the fox and the grapes".
https://en.wikipedia.org/wiki/The_Fox_and_the_Grapes
Second, journals are overloaded, the journal reviewers unpaid, and frankly, they are not doing a good job anymore in reviewing work. Feedback that amounts to "maybe submit first to a conference" is not that valuable. Effectively, you have to have the idea, then execute it perfectly, then do a major marketing effort to popularise your work, and why exactly would you at the end of this still submit to a journal? Given that the journal either wants the rights to your work, or charges you for the pleasure of publishing with them. At this point, a journal is just another marketing step, and for marketing you pay, I guess.
This door is already open, and impossible to close [1]. For any reviewer closely familiar with the area the paper is in it is trivial to guess which group the paper came out of 9 times out of 10. Style, approach, citations, and just knowing what the other people in your area are working on all contribute. To achieve any sort of blindness you have to go to reviewers who don't know that area as well... which, I suppose, does decrease the potential for corruption--at the cost of the reviewers not understanding the paper as well.
> Isn't the whole point of a paper to present a topic in a clear and understandable way, so that your peers are able to cut through the bullshit?
Even a well written paper often requires some effort to understand, due to conceptual novelties, mathematical complexities, etc.
Unfortunately there's little incentive for reviewers to put in much mental effort. Some of them will. Many won't.
> When you pick a journal you pick the subset of your peers to review your work. If your peers point out problems then why not listen to them?
Because the "problems" are often bullshit. You're assuming the reviewers are acting in good faith and finding legitimate problems with the paper, but they're often not--usually not out of malice, just laziness. There's no incentive to put any effort into a review above the minimum to convince the editor that your review is plausible. This doesn't take much since the editor is probably only going to glance briefly at the reviews. If you can finish your review in an hour instead of a day you can spend the rest of the day writing your own papers or doing other work that actually furthers your career.
[1] In all but the largest, most active areas, where there are so many people working that nobody can keep up.
For example, in fields where the peer-review is done primarily by academics are there better reviews during academic breaks vs. say, exam season? Does this translate to more accepted work because the reviewer can spend more time digesting the content?
Yes, but there is unfortunately a wide variability in the knowledge of reviewers. It's not uncommon to have reviewers directly contradict each other about the shortcomings of the paper. E.g.,
Reviewer 1: "Section A needs to be condensed, this level of background is unnecessary. Readers will find this as common knowledge."
Reviewer 2: "Section A should be expanded as all readers may not be knowledgeable in this area."
Particularly for grad students, this can make the process feel arbitrary and the feedback of limited value. There also can be an inordinate amount of stylistic review comments (e.g., "however" should be changed to "nonetheless"). These often come across as the reviewer feels the need to critique something because they don't have sustentative critiques. Good critiques strengthen a paper, bad feedback seems to do nothing but check the process boxes.
I actually agree with the OP that one potential solution is to encourage reviewers to decline reviewing papers that they don't have the requisite knowledge to provide good feedback. Unfortunately, this decision is often only offered based on reading the title and abstract.
The meta-reviewer doesn't just send the reviews to the authors with no commentary. They write a summary of the reviews with a recommendation about what to do next. If there are contradictions and no consensus is reached, it is their job to decide which concerns the authors need to respond to in order to get it published. They aren't bound by the reviews, they can disregard one if they feel like it is unfair, although sometimes the rules say they must get another reviewer to write a 4th review.
Over the years, I've drifted to the biological side of bioinformatics methods. The papers I write are similar to CS papers, but the projects are bigger, and I no longer submit papers to CS conferences. My typical experience with peer review is accept after 1-3 rounds of revisions. Hence I get to see more of the positive side, where the reviewers act as editors and their suggestions improve the paper.
Of course, it's possible to get a CS-like experience with journals by being ambitious. If you try your luck by submitting the paper to prestigious journals it's unlikely to get in, your typical outcome is a desk rejection or rejection after a single round of reviews. Then you get to see more of the negative side of peer review. But I'm just not interested in playing that game.
This problem goes even deeper, because it not only applies to papers but also to research grant applications. I even had once got a (positive) recommendation for one of my applications, but I could see from the statement that the reviewer did not really understand what was important.
That's not to say the proposal isn't worthwhile, just that it didn't find the right audience. A contractor may have a whiz-bang new method of snow removal, but it doesn't mean people in Phoenix will want it (or even understand it).
.) an unsolvable problem inherent to all peer reviews?
.) a problem that comes from specific practices, and could be solved by doing peer reviews differently?
I'm just a layman - but if reviews are not done by peers (= experts working in the same field)... who else could properly review a paper?
They are actually proposing that peer reviews are credited:
> Instead, authored and credited reviews might lead to more responsible reviews, or reviewers respectfully declining when they might not know the topic.
And yes, I do think reviewers should be credited. Because it's work, for one, that needed to be done so the paper could be published, and because it would probably indeed incentivize reviewers to do the work correctly.
However, it could incentivize them to reject more papers, in fear of accepting a bad paper and be publicly advertised as a reviewer who reviewed and accepted it.
There also often isn't a clear cut line between "good" and "bad". Almost all papers have flaws and putting those out in the open for others to improve upon, or at least acknowledge, would help move humanities knowledge forward.
This made me curious about how arxiv operates. It seems that you require endorsement to become a registered author, and the submissions are moderated[1][2].
This already seems sufficient to keep out spam and clearly junk science. What is the value add of official(?) journals?
[1] https://arxiv.org/about
[2] https://arxiv.org/help/submit
> Where is that quote from?
by providing a link to the source of the quote.
Ironically, maybe the person to whom you first replied was actually intending to respond to the comment you just linked instead of making a top-level comment.
Edit: Here's a proof of the Riemann hypothesis submitted just last week: https://arxiv.org/pdf/2209.01890.pdf You can have a little fun exploring "proofs" of famous theorems on the arXiv over the years.
(Disclosure: I volunteer for https://plaudit.pub, a non-profit that aims to separate that from the publication process.)
Which is why reproducibility should be the most important topic.
* Theoretical work * Computer simulations (re-run the simulation? Well that's not going to detect most issues. Re-create the program from scratch? expensive and hard to make a re-write meaningfully different systematically) * Cosmological observations (can't re-run that supernova!) * Medical case studies (can't just find a new patient) * Large scale studies (build a second LHC? Do results based on the Framingham Study have to kick off another 30 years of data collection before they publish)
In fact it's only small and self contained experiments where this is really practical.
It would have detected the fraud going on at ICL with their covid model that was used by the governments of the world to set the course of pandemic response. I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here. Running the same simulation with the same inputs produced wildly different results between runs. That would have been an easily detected red flag but because no one was able to do so until long after the pandemic response was set in motion and changes would have been politically uncomfortable for the powers that be, we ended with a terrible response that many still cannot bring themselves to admit was both poor and instigated by a terrible model that should have been easily detected as being garbage.
> Among the two popular models, we found that the ICL model is more transparent and reproducible compared to the IHME model. The former sometimes over-predicted future deaths while the latter clearly under-predicted post-peak deaths. Both models predicted the timing of peaks reasonably well using data until one week prior. The ICL model produced a much wider band of uncertainty for New York state, possibly because the pattern did not conform well with their internal training data used from European countries.
Nature (on 08 June 2020) reported "a computational neuroscientist has reported that he has independently rerun the simulation and reproduced its results. And other scientists have told Nature that they had already privately verified that the code is reproducible." - https://www.nature.com/articles/d41586-020-01685-y citing the CODECHECK certificate at https://zenodo.org/record/3865491 from May 29, 2020.
You should also be asking why public health is so under-funded world-wide that many countries looked to the ICL model - which in turn was a repurposed influenza model - rather than use local expertise or be able to use better validated models.
> Running the same simulation with the same inputs produced wildly different results between runs.
As I recall, the replication issues were the normal issues related to parallelization and random number generation, and not things which greatly affected the results. There were a number of HN threads on the topic back in 2020. Eg, https://news.ycombinator.com/item?id=23212268 , which linked to a couple of those issues and their resolution.
> I can't think of better example of a situation where requiring replication would have altered the path of humanity as much as it would have here.
I can. The Reinhart-Rogoff spreadsheet error which was widely used as the excuse for "austerity" cuts, rather than the real reason of wanting to lower taxes on the rich. https://truthout.org/articles/excel-spreadsheet-error-lesson...
I wonder where that tax money could have been used. Perhaps into public health measures?
You will be amazed what you didn't spot/assumed and learn a lot.
I read an interview with a TV show runner who said you can come up with any crazy plot twist thing, and within 10 minutes of the first episode airing some guy on the internet has already figured the whole thing out. I think this phenomenon could be put to good use.
Just publish your papers publicly with a comments section. If there's problems with it people will tear it apart. Source their work and let the world help you improve your work.
it would enable faster and more seamless communication, academics wouldn't be burdened with extra volunteer work, and publishers would still be able to curate works as the authors suggest (+ there are other ways to do it).
sadly my impression is that it's simply too entrenched in the publishing process -- and publisher's raison d'etre to a large extent -- for the publishers to relinquish control of it
I believe that double-blind, non-public reviews do a great job at protecting those in a more precarious situation: grad students are not judged by their lack of publication record, there is no permanent public record of all those times they got rejected, and reviewers can be both more honest and more certain that a rejection won't lead to a 4Chan/Twitter mob coming for them.
According to the ACL's 2019 survey [1], "female respondents were less likely to support public review than male respondents". I'd be weary of implementing any changed that would make academia even more hostile to women and minorities.
[1] http://acl2019pcblog.fileli.unipi.it/wp-content/uploads/2019...
Sure, people like citing themselves. Editors of double-blind journals already are supposed to desk reject work that says something like "in our prior study (Smith 2020)..." They should also desk reject obnoxiously self-referential citations --- like "Great advances have been made in method X (Smith 2016, 2017a, 2017b, 2018, 2019a, 2019b, 2020, 2021)"
But my own impression and experience is that your statement is naive. The key thing I think you miss is that researchers/labs/groups develop fairly specific interests, expertise and "research programs". So taking any paper and looking at the specific area or phenomenon being probed, the techniques/methods used, and the questions or interests driving the research and its analysis, and you'll get a pretty unique "fingerprint" of the researchers that anyone in the same field should be able to identify with pretty good accuracy.
The naivety of your statement I think is this idea that experiments or acts of research are kind of plucked from the air by any who are interested and then the method or technique required is simply downloaded and executed like software.
Researchers aren't generic like computing hardware tries to be. Far from it. A researcher and their research or ability to do research is much more like a tree ... grown organically through experience and pretty fixed and inflexible once it's mature.
None of this is to be really critical of your statement. In fact, I think it's very useful (provided I'm onto something and not merely bitter) to state the ideal practice of science and contrast it with reality.
That we can't do double-blind review is almost certainly a bad thing. And it also tells us something about what kind of "science" (here the global collective enterprise) we have. I think I'm correct about the "fixation" of researchers. But I also think that the culture of science we have encourages this process. I've seen it myself: researchers trying a minor change in field and technique (very minor) and hitting massive resistance from the funding agencies. The conclusion we all drew from seeing it was that the techniques you can get funding for are mostly fixed by your first 10 papers or so. This could be changed and research could get closer to what you describe and I think that would be a good thing. But, as I stated in another comment, to get there requires change beyond the peer review process.
Something else it tells us is that our science operates in a relatively closed and inward looking cultural ecosystem, which indicates to me some degree of corruption is almost guaranteed. If we don't have good diversity and competitive and corrective interactions across space/culture, then, as the "coffins in the ground" adage tells us, we have to rely on diversity across time/generations. And indeed researchers know this just from experience. Someone winning a Nobel prize is like carving their thought in stone ... it takes time for us to move on.
An interesting study in this regard might science during the USSR. I've read glancingly that mathematics research progressed differently between the West and USSR and that, for this interested, cross-pollination was fruitful. An analysis of this could prove insightful. I wonder whether something similar is going on in China at the moment.
At least for mathematics papers, the community who works in a particular area, and is thus actually able to do a review the paper, is pretty small. Since, on the other hand, the style that you use for mathematical proofs is like a fingerprint, double-blind peer reviews are next to impossible.