I believe peer review to be generally harmful. It forces you to do things in a certain direction, in a certain way.
To quote Knuth:
"In fact what I would like to see is thousands of computer scientists let loose to do whatever they want. That's what really advances the field."
And today, the idea marketplace does not need to be mediated by conferences and papers.
There is however, one aspect in which removing them is tricky, and that is funding. Currently, publication is a metric used to grant funds or evaluate the result of research. If we removed that metric, it would just likely be replaced by other forms of gate-keeping that are as bad. And giving money to anyone is kind of problematic, of course; even just a permissive filter (checking for "reasonableness") would let in way too much candidates w.r.t. available money. I don't really know what could solve that particular conondrum.
> And today, the idea marketplace does not need to be mediated by conferences and papers.
You could say that posting to HN is a form of peer review (in the vain you mention it at least). Anyone is free to comment and tear the author's article apart.
But then the review themselves get reviewed, and those get reviewed, etc
In the end you get a rough global consensus. It's far (very far) from perfect, but maybe more representative of impact than the opinion 3 guys deeply embedded in a system that doesn't exchange with the outside world enough, and that are often competing with you.
It's really telling that git came into existence all because Linus Torvalds got frustrated with the existing state of version control tooling and went out on a 2 week code binge to create the thing he wanted. No amount of discussions or dialog or code reviews or retrospectives or any of that helped resolve the issues everyone in the industry was having with the state of version control. It came down to one guy just getting frustrated to the point of sitting down and writing the dang thing.
??? Peer review is there to disseminate good knowledge. Nothing stops you from attempting to do stuff with things you've invented. For some things though, it's not necessarily self evident that what you did was good, and expert review helps ensure that randos reading the literature can at least trust it to make basic sense, even if it may not always be correct.
Yes, but peer review is limited by the understanding, imagination, vision, and intelligence of the peers, who are not always peers in such aspects. Sometimes it takes creativity and individuality, as well as a little confidence, to come up with the right solution. But committees are not designed for that kind of solution. They're merely designed to analyze data and come up with a dry answer in conformity with "core principles", which is at odds with the kind of raw humanistic creativity you get when you just say "screw it" and go on a 2 week code binge to write the thing you needed all along. That's why git exists. It could never have come about through code reviews or other types of committees.
Sure, but nothing ever stops you from publishing stuff yourself (unless you've signed e.g. patient confidentiality agreements or are otherwise prohibited by law). It just bears a bit of a stigma if the proper venue is a peer reviewed journal.
So to answer the author's question; peer review makes no sense when your peers are incapable of judging a paper's validity either through a lack of domain knowledge on the subject or an overall poor grasp of scientific concepts in the community as a whole.
Another situation where peer review makes no sense is sometimes recognized by traditional peer reviewed journals. Physical Review Letters has suspended peer review on occasion for high-energy physics experiments with thousands of authors, acknowledging that it would be difficult to find a competent reviewer who is not already on the author list.
When did p < 0.05 become acceptable? I remember in astronomy/astrophysics where 5sigma (3x10^-7 probability) was always the minimum required to publish.
social sciences and natural sciences have pretty much always had vastly different standards for what is acceptable. Getting 5 sigmas in many sorts of psychological studies would be pretty much impossible.
I very much doubt there was ever a time when experimental psychologists or sociologists held themselves to that standard.
0.05 was first proposed by Fisher himself, which is 2 sigma if you look at a normal distribution (95% falls within 2 standard deviations[1]). Indeed, physics (generally particle physics, such as the Higgs Boson) goes for 5 or 6 sigma.
In biology, 0.05 is used because, well, it's hard otherwise. The cost, time, effort required for plants, animals, and even cells get exorbitant quickly, and are subject to many, many variations (animals generally have additional IACUC limits on procedures and numbers).
I could make arguments about how particles obey laws while biology is at least a few levels abstracted over that fundamental behavior, but xkcd has already done that. I personally feel that p<0.05 is absurd, and the entire biological field NEEDS to move to p<0.01. There's a movement toward that end, and it's about damn time.[3]
16 comments
[ 2.3 ms ] story [ 53.0 ms ] threadTo quote Knuth:
"In fact what I would like to see is thousands of computer scientists let loose to do whatever they want. That's what really advances the field."
And today, the idea marketplace does not need to be mediated by conferences and papers.
There is however, one aspect in which removing them is tricky, and that is funding. Currently, publication is a metric used to grant funds or evaluate the result of research. If we removed that metric, it would just likely be replaced by other forms of gate-keeping that are as bad. And giving money to anyone is kind of problematic, of course; even just a permissive filter (checking for "reasonableness") would let in way too much candidates w.r.t. available money. I don't really know what could solve that particular conondrum.
You could say that posting to HN is a form of peer review (in the vain you mention it at least). Anyone is free to comment and tear the author's article apart.
In the end you get a rough global consensus. It's far (very far) from perfect, but maybe more representative of impact than the opinion 3 guys deeply embedded in a system that doesn't exchange with the outside world enough, and that are often competing with you.
(This is not a typo :] )
I very much doubt there was ever a time when experimental psychologists or sociologists held themselves to that standard.
In biology, 0.05 is used because, well, it's hard otherwise. The cost, time, effort required for plants, animals, and even cells get exorbitant quickly, and are subject to many, many variations (animals generally have additional IACUC limits on procedures and numbers).
I could make arguments about how particles obey laws while biology is at least a few levels abstracted over that fundamental behavior, but xkcd has already done that. I personally feel that p<0.05 is absurd, and the entire biological field NEEDS to move to p<0.01. There's a movement toward that end, and it's about damn time.[3]
1: https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rul...
2: https://xkcd.com/435/ & https://xkcd.com/1052/ & https://xkcd.com/1520/ spring to mind
3: A lot of internet ink has been spilled on the topic, but http://www.nature.com/nmeth/journal/v12/n3/full/nmeth.3288.h... & http://www.pnas.org/content/110/48/19313.long & http://www.nature.com/news/scientific-method-statistical-err... are good places to start.