This article seems like it would be very interesting to me but after dismissing three intrusive popups in my mobile browser I stopped trying to read it.
Does anyone know of a mobile browser friendly way to read this kind of site without being constantly bombarded? It's exhausting. It seems like the sort of problem that would have been solved, but my attempts to discover the solution have failed.
The 'Kill Sticky' plugin for Firefox works pretty well for me for most stuff. Most popups darken the screen, though, and it doesn't handle that. It's very useful to me nonetheless. If you don't use Firefox, you can go their site and copy/paste it to your bookmarks.
There's just a cookie prompt and then a newsletter sign-up a little later? I don't think there's any constant bombardment, if there is for you I'm interested to know more.
The constant bombardment was more in reference to the state of the web generally. Popups and overlays that aren't a result of an interaction I've chosen to make are exhausting for me (and people with ADHD generally, and many others). I don't go back to sites that have them, but would be happy to read (and often pay for) the content if that wasn't the context.
I was expecting the author to define "real peer review," but didn't see that. The best approximation is probably gleaned from the conclusion:
- integration of preprint servers and alt metrics
- tweaking incentives to review
- making comments on papers public
- use of software to detect fraud
- directing resources specifically to improving peer review
The bigger problem is that the author doesn't seem to actually zero in on the problem peer review is supposed to solve today. The author notes that peer review really got going in the 1970s as a way to filter content flowing to overwhelmed editors. But the emergence of the internet largely nullifies that problem. Wide distribution of scientific information no longer requires scientific publishers.
The real problem is the ways in which science funding, journals, and peer review have become intertwined, with publishers playing the role of bankers in this economy. This problem is cultural, not technical. It's a historical relic and it increasingly does not serve science well.
So, what is the actual problem that journal-supervised peer review is supposed to solve in the age of the internet?
Nowadays, journals and peer review solve the problem of, people need to make hiring decisions and funding decisions. But these decisionmakers don't have enough technical expertise or time to evaluate all the papers from all the applicants.
The decisionmakers do have enough time to learn which are the most prestigious journals in the field. So, they can pick the people with the most papers submitted to prestigious journals, or at least use that to filter applicants down to a short list for closer examination.
>Wide distribution of scientific information no longer requires scientific publishers.
Wide and voluminous distribution of bad information requires filters to extract the good, peer review has some form of filtering functionality, although I wouldn't say it is great I think it would probably be better than the filter that a Facebook or Twitter of Science would provide (or just Facebook or Twitter if you don't like the 'of science' locution)
I think having a public forum for publishing and reviewing is valuable, and then having journals or something similar, that produces curated selection of published science based on that public review.
> It's a historical relic and it increasingly does not serve science well.
It's also pretty devastating evidence that the world is not going to improve in any sort of an organized way if the experts that we would expect to lead the effort for a rational world can't clean their own house. It's hard to trust academic systems to design ways to improve society when the academic system is built around an irrational base in journals.
An academic system that exhibits the same shitty array of characteristics as every other corrupt status quo institution doesn't give me a lot of hope for everything else.
Politicians get voted in, so they answer to the newspapers but in case that’s not funny enough, we tried just having an all powerful person in charge of things in the past but it turns out that works even even worse.
I dont know why would one expected academics and scientists to "lead the effort for a rational world" or be especially good at organizing or leading. It makes sense to expect them to be good at math, physics, sociology and what not.
I would not even say those things would count as “real peer review.” Peer review is supposed to involve replication. Unfortunately that almost never happens these days.
What gives you that impression? Replication is a too-often neglected aspect of the scientific process, but I've never heard anyone ascribe it to the peer review aspect.
"But the emergence of the internet largely nullifies that problem. Wide distribution of scientific information no longer requires scientific publishers."
I'm not at all convinced that this is true - people have been saying it for a long time, and it's not manifested itself in a particularly compelling fashion yet.
Arxiv, and it's various follow-ons - I think I was one of the first submissions to medRxiv, are excellent as far as they go, but if we take the COVID-19 pandemic as a stress test, they still have a long way to go.
Beyond that, their curation is...lackluster at best, even if we're not talking about gatekeeping, filtering, etc. but just "What if I want to do something other than key word search?"
I find unexpectedly interesting articles all the time reading journals. I have yet to do so for a preprint server unless someone sends it my way, or it ends up on my Twitter feed.
I'm in mathematics, and this is very much true in my field. Papers can be, and nearly always are, widely distributed on the Internet (arXiv) before formal publishing. They are widely read, cited, and built upon.
The publishing industry survives because researchers need to put "Published in Journal X" on their CVs. The peer review process also can lead to incremental improvements, and will occasionally catch major errors, but at this point everything other than the stamp of approval is secondary.
Journals still play the role of quality filters/QA, which is very important. While your typical Internet-only journal is going to be like Alibaba, a top journal like Nature is more like Harrods - you're not going to waste money by browsing through trash there.
The problem is that looking at journal quality is the best quick way to evaluate how good a paper is, when you aren't an expert on the topic. And a lot of employment and funding decisions are made by people who aren't experts.
Nowadays a journal provides essentially no distribution, but there's no good alternative to journals as a "stamp of quality".
> And a lot of employment and funding decisions are made by people who aren't experts.
This should be considered a flaw (bug in tech terms) not a feature. why would you take decisions about funding something or not if you are not an expert. At least you are not expert in the same field but have knowledge and it is easier for you to communicate and discuss the proposals. But getting someone who never did a real scientific research and the last time he wrote a scientific essay was when he was in college to determine which research should be funded is wrong.
> This should be considered a flaw (bug in tech terms) not a feature. why would you take decisions about funding something or not if you are not an expert.
Because funding requires taking from something else -- either raising more revenue or reducing funding for any of the infinite possibilities that it could be spent on. And nobody is an expert in everything.
Where a particular area of expertise does cover the choices to direct funding, it generally does. With a lot of inefficiency and corruption and incompetence thrown in of course, but politicians and bureaucrats sure didn't decide how every dollar on the JWST would be spent.
I am an academic mathematician, who (like most) is an expert in a sub-subfield in my discipline. Outside my specialty I certainly have some knowledge, but I usually can't distinguish a major breakthrough from incremental progress.
Meanwhile, I've been asked to evaluate hundreds of other mathematicians, on hiring committees and grant panels. Most of the applications are in specialities bordering mine, but not squarely in my area of expertise. Usually I can get a sense of what they've accomplished, but in most cases I can't reliably evaluate its depth or importance.
Unfortunately, external markers of quality are needed.
Having just gotten extensive edits back on two papers of mine, including fixing something that I had overlooked but am glad they didn't, the role of journals is something that I think people underestimate in some fields.
> And a lot of employment and funding decisions are made by people who aren't experts.
Where is this the case? Federally-funded research generally involves experts. When Congress ascribes budgets to research agencies, those don't come from a vacuum. There are civil servants whose jobs revolves around providing expertise to budgetary oversight. They usually have titles like "program manager."
"And reviews also vary widely in their conclusions – how they rate papers and whether they recommend editors to accept or reject the paper."
This is assumed to be a bug, and not a feature. A lot of this comes from different perspectives, weighting, etc. Widening the number of reviewers isn't going to reduce the variance on this. You see it in grant review panels as well, which are much more "discuss and engage with your fellow reviewers".
"We live in a very different world today than we did when journals were circulated through print: One with the Internet, where data and research don’t need to be circulated in limited and expensive publications. Research can be shared almost immediately with massive networks of scientists through blogs, preprints [1] , data storage platforms and online forums, at little to no cost."
One of the assumptions here is that this access is uniform. It isn't. During COVID-19, some senior scientists essentially went with "Publication via press release", when more junior scientists can't do that. The popularity of figures on my field's Twitter hashtag is astonishingly non-uniform, cuts out whole swathes of people (govt. scientists), and is only lightly correlated IMO with the impact of their work.
I will say that, when I was a junior scientist, getting some of my work in good journals carried it much further than internet distribution would have.
"Finally, other types of centralized platforms could be hugely important too. Journals currently have exclusivity agreements (meaning researchers can only submit to one journal at a time) but they also have different formatting requirements, which take a long time to fulfill. By one survey estimate, articles are submitted an average of two times before they are published, and formatting takes up a total of 14 hours per article."
The number of journals that only require formatting post-acceptance is growing, and is awesome.
When it comes down to it, I think peer review's biggest strength is actually something the author takes as a negative. It's not "This paper has only been reviewed by two people", it's that every paper in this journal has been reviewed by at least two people, plus the public. For a lot of papers, I'm actually skeptical that that number is lower than if we went with "Well, the community will review it in the fullness of time".
> Before the adoption of peer review, journals focused on quickly disseminating letters and communications between scientists, with little to no editing or external reviewing
It seems to me that a return to this standard is inevitable:
* Scientific papers are in enormously difficult to peer review properly because fields have become so specialized
* The incentive structure is highly unbalanced, and the truth is, the people you’d like to peer review your papers are already too busy
* Peer review is very expensive (in terms of time) for a conscientious reviewer
* Many of these papers are low quality, serving to buttress the reputation of the author more than moving the field forward
Since it’s an open secret that so many journals are worthless, it would probably be better just to return to the publish fast model instead of trying to fix a sick endeavor.
38 comments
[ 4.8 ms ] story [ 93.3 ms ] threadDoes anyone know of a mobile browser friendly way to read this kind of site without being constantly bombarded? It's exhausting. It seems like the sort of problem that would have been solved, but my attempts to discover the solution have failed.
I only had de reject all cookies prompt.
With firefox you can aldo use the reade mode, that extracts the main taxt and displays it plainly.
1. cookies 2. email newsletter 3. cookies again
The constant bombardment was more in reference to the state of the web generally. Popups and overlays that aren't a result of an interaction I've chosen to make are exhausting for me (and people with ADHD generally, and many others). I don't go back to sites that have them, but would be happy to read (and often pay for) the content if that wasn't the context.
- integration of preprint servers and alt metrics
- tweaking incentives to review
- making comments on papers public
- use of software to detect fraud
- directing resources specifically to improving peer review
The bigger problem is that the author doesn't seem to actually zero in on the problem peer review is supposed to solve today. The author notes that peer review really got going in the 1970s as a way to filter content flowing to overwhelmed editors. But the emergence of the internet largely nullifies that problem. Wide distribution of scientific information no longer requires scientific publishers.
The real problem is the ways in which science funding, journals, and peer review have become intertwined, with publishers playing the role of bankers in this economy. This problem is cultural, not technical. It's a historical relic and it increasingly does not serve science well.
So, what is the actual problem that journal-supervised peer review is supposed to solve in the age of the internet?
The decisionmakers do have enough time to learn which are the most prestigious journals in the field. So, they can pick the people with the most papers submitted to prestigious journals, or at least use that to filter applicants down to a short list for closer examination.
Wide and voluminous distribution of bad information requires filters to extract the good, peer review has some form of filtering functionality, although I wouldn't say it is great I think it would probably be better than the filter that a Facebook or Twitter of Science would provide (or just Facebook or Twitter if you don't like the 'of science' locution)
It's also pretty devastating evidence that the world is not going to improve in any sort of an organized way if the experts that we would expect to lead the effort for a rational world can't clean their own house. It's hard to trust academic systems to design ways to improve society when the academic system is built around an irrational base in journals.
An academic system that exhibits the same shitty array of characteristics as every other corrupt status quo institution doesn't give me a lot of hope for everything else.
Are you going to delay publication of results from the LHC or JWST until someone has built another one?
The traditional unit of replication is a published paper, not a submitted paper.
A peer review takes positions on these things:
- does the paper correctly describe the state of the art, with appropriate references?
- does the paper claim and explain a novel contribution?
- is the methodology sound?
- do the claimed results justify the claimed contribution?
- does the paper clearly communicate to its audience?
That's it.
Replication will be attempted later if the audience cares about the results.
I'm not at all convinced that this is true - people have been saying it for a long time, and it's not manifested itself in a particularly compelling fashion yet.
Beyond that, their curation is...lackluster at best, even if we're not talking about gatekeeping, filtering, etc. but just "What if I want to do something other than key word search?"
I find unexpectedly interesting articles all the time reading journals. I have yet to do so for a preprint server unless someone sends it my way, or it ends up on my Twitter feed.
Interesting. In math, I don't think anyone "reads journals", but lots of people check the arXiv frequently to see what has been posted.
The publishing industry survives because researchers need to put "Published in Journal X" on their CVs. The peer review process also can lead to incremental improvements, and will occasionally catch major errors, but at this point everything other than the stamp of approval is secondary.
Instead of deciding what gets published, peer review should help decide what gets noticed.
Nowadays a journal provides essentially no distribution, but there's no good alternative to journals as a "stamp of quality".
This should be considered a flaw (bug in tech terms) not a feature. why would you take decisions about funding something or not if you are not an expert. At least you are not expert in the same field but have knowledge and it is easier for you to communicate and discuss the proposals. But getting someone who never did a real scientific research and the last time he wrote a scientific essay was when he was in college to determine which research should be funded is wrong.
Because funding requires taking from something else -- either raising more revenue or reducing funding for any of the infinite possibilities that it could be spent on. And nobody is an expert in everything.
Where a particular area of expertise does cover the choices to direct funding, it generally does. With a lot of inefficiency and corruption and incompetence thrown in of course, but politicians and bureaucrats sure didn't decide how every dollar on the JWST would be spent.
Meanwhile, I've been asked to evaluate hundreds of other mathematicians, on hiring committees and grant panels. Most of the applications are in specialities bordering mine, but not squarely in my area of expertise. Usually I can get a sense of what they've accomplished, but in most cases I can't reliably evaluate its depth or importance.
Unfortunately, external markers of quality are needed.
Where is this the case? Federally-funded research generally involves experts. When Congress ascribes budgets to research agencies, those don't come from a vacuum. There are civil servants whose jobs revolves around providing expertise to budgetary oversight. They usually have titles like "program manager."
This is assumed to be a bug, and not a feature. A lot of this comes from different perspectives, weighting, etc. Widening the number of reviewers isn't going to reduce the variance on this. You see it in grant review panels as well, which are much more "discuss and engage with your fellow reviewers".
"We live in a very different world today than we did when journals were circulated through print: One with the Internet, where data and research don’t need to be circulated in limited and expensive publications. Research can be shared almost immediately with massive networks of scientists through blogs, preprints [1] , data storage platforms and online forums, at little to no cost."
One of the assumptions here is that this access is uniform. It isn't. During COVID-19, some senior scientists essentially went with "Publication via press release", when more junior scientists can't do that. The popularity of figures on my field's Twitter hashtag is astonishingly non-uniform, cuts out whole swathes of people (govt. scientists), and is only lightly correlated IMO with the impact of their work.
I will say that, when I was a junior scientist, getting some of my work in good journals carried it much further than internet distribution would have.
"Finally, other types of centralized platforms could be hugely important too. Journals currently have exclusivity agreements (meaning researchers can only submit to one journal at a time) but they also have different formatting requirements, which take a long time to fulfill. By one survey estimate, articles are submitted an average of two times before they are published, and formatting takes up a total of 14 hours per article."
The number of journals that only require formatting post-acceptance is growing, and is awesome.
When it comes down to it, I think peer review's biggest strength is actually something the author takes as a negative. It's not "This paper has only been reviewed by two people", it's that every paper in this journal has been reviewed by at least two people, plus the public. For a lot of papers, I'm actually skeptical that that number is lower than if we went with "Well, the community will review it in the fullness of time".
It seems to me that a return to this standard is inevitable:
* Scientific papers are in enormously difficult to peer review properly because fields have become so specialized
* The incentive structure is highly unbalanced, and the truth is, the people you’d like to peer review your papers are already too busy
* Peer review is very expensive (in terms of time) for a conscientious reviewer
* Many of these papers are low quality, serving to buttress the reputation of the author more than moving the field forward
Since it’s an open secret that so many journals are worthless, it would probably be better just to return to the publish fast model instead of trying to fix a sick endeavor.