Since I've seen this paper mentioned a few times now on several different sites and heard varying replies, I'd be very interested in informed opinions about this. I've taken a look at it but don't consider my skill level in statistics high enough to assess the paper on my own.
I think just your first post is a snappy reply to the headline, but doesn't seem to address the content of the paper in any way, so I was interested in your specific thoughts about the content of the paper.
It's a perfectly valid, and indeed important question.
Self-application is an important form of scientific methodology. Think e.g. of programming languages development: one of the first things you do is bootstrap a compiler.
Programming language development is not standard software development.
PL research is a discipline with no really feasible empirical methods for robust and replicable evaluation of programming language ideas. Using self-application is a test of internal coherence -- if your new language is not good at writing a compiler, then maybe it isn't such a good idea.
Self-application is a well-known scientific methodology. Would you trust a physical theory that predicted the impossibility of physicists?
Generally, no. The article was published in PLoS Med, and concerns generally medical research studies. It has been more than 10 years and i think most people still don't disagree
with it. Here is the authors response to criticisms:
I generally agree with this. The publication game tends to focus on exciting new discoveries way more than reproduced ones. I think this is because universities push you to focus on "original ideas" when publishing theses instead of reproducing other ideas because the former is "harder" than the latter.
Having an open, accessible repository of scientific data sounds like a good thing, but I doubt it would help the problem of reproducibility.
The biggest problem of such a "github" would be that a scientific repository is only as good as its maintainer. In software development land we already see a lot of egos clashing over the code, but software has a clear "works/doesn't work" criteria. In many scientific areas such a criteria is hard to find and we'll have to rely on the personality of a repository maintainer.
The repositories will soon become a brand on its own. Think "nature.github.com", "physrev.github.com" and "lancet.github.com".
If you don't have a clear mechanism by which a startup could solve this problem, I suspect the result would be the same structures that already exist, but with better marketing and a Web 2.0 interface. If anything, this would make the problem worse, because it creates a company that now has a vested interest in keeping the status quo because it's part of the status quo, and that company will be manipulating politics and public opinion with its marketing.
A second problem with this sort of company is that it often doesn't really understand the industry it's "disrupting", so they end up not solving a lot of the problems which older solutions solved.
At a more fundamental level, I think that a company is always going to be motivated to make money rather than solve a problem. Even if intentions start off well, the people in the company will believe that the company is good and needed, so they'll make choices that give up a little of the original purpose of the company to make the company survive, because if the company doesn't survive then it can't do any good, right? And after a bunch of one-degree deviations from its original direction, the company is going a completely different direction than its original purpose and isn't even geared toward solving the problem any more.
Sure, "We're going to save the world" is great marketing, but at this point I simply don't ever believe that a startup is a solution to a major problem like this.
I think the solution to this is that science publication needs to be done like science itself. Just as scientists commit to doing experiments without knowing what the results of the experiments will be, publishers need to commit to publishing experiments without knowing what the results of the experiments will be.
Publishers need to be involved earlier in the process. Instead of submitting a completed research finding, scientists should apply to publish research before doing it, and if the topic of research is interesting to the publisher, both scientist and publisher should commit to publish in that journal, no matter what the results are.
Additionally, publishers should commit to publish only something like 1/3 research into new topics, with the other 2/3 being publishing attempt to reproduce previous interesting results. I don't mean a slavish repetition of existing experiments, but taking a critical eye toward testing the same hypotheses with different (hopefully better) methodology.
Science doesn't seem to have fixed itself since. At this point it is prudent to shift the public's perception from "research is true" to "research is valid". To the extent in which public policy is dictated by research, it should be based, not on journal-published research, but perhaps on meta-analysis or third party research-on-research.
I learned this in middle school during the Science Fair. My experiment didn't work. Up against the deadline and not wanting a 0, I simply made up the data. I got a nice big A. Probably exactly what happens in the real world too.
If you would have gotten a 0 then your science teachers failed you I'd say. A failed experiment is just as valid a result as a successful one if the goal is to gather data.
Just because the bar is low doesn't mean its height is in any way related to its discernment of quality. Say, for example, the bar was simply based on the institution that the author is employed by. That may be a low bar in that anyone from Harvard can get a paper published, but it also doesn't actually tell us anything about the paper's validity. Whereas I wouldn't be skeptical of a good Rush University paper, simply because its authors weren't prestigious enough for Nature to pay attention to.
I was talking about getting your paper peer reviewed, not placing it in a particular journal. If you want to get into PLOS One I don't think your university matters very much.
Inside the Fake Science Factory (2018) is an amazing DEFCON talk, by 3 people who stumbled on and investigated an unbelievably huge worldwide network (400,000 scientists involved) of fake-scientific conferences, journals, websites raking in money and enabling people to make claims that their products are scientifically proven. The presenters got hackers to track down the people behind one of the largest organizations, and analyze the huge numbers of conferences/papers. They present/publish nonsensical talks on graphs and bees+cancer to see what peer review there is... A must watch. Very funny, if depressing.
40 comments
[ 2.3 ms ] story [ 91.3 ms ] threadHow can I elaborate on answer that I don't have?
Self-application is an important form of scientific methodology. Think e.g. of programming languages development: one of the first things you do is bootstrap a compiler.
PL research is a discipline with no really feasible empirical methods for robust and replicable evaluation of programming language ideas. Using self-application is a test of internal coherence -- if your new language is not good at writing a compiler, then maybe it isn't such a good idea.
Self-application is a well-known scientific methodology. Would you trust a physical theory that predicted the impossibility of physicists?
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1297555/
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1896210/
An empirical estimate: https://academic.oup.com/biostatistics/article/15/1/1/244509
And a review with some context: https://www.biorxiv.org/content/10.1101/050575v1
https://www.nature.com/news/1-500-scientists-lift-the-lid-on...
The biggest problem of such a "github" would be that a scientific repository is only as good as its maintainer. In software development land we already see a lot of egos clashing over the code, but software has a clear "works/doesn't work" criteria. In many scientific areas such a criteria is hard to find and we'll have to rely on the personality of a repository maintainer.
The repositories will soon become a brand on its own. Think "nature.github.com", "physrev.github.com" and "lancet.github.com".
A second problem with this sort of company is that it often doesn't really understand the industry it's "disrupting", so they end up not solving a lot of the problems which older solutions solved.
At a more fundamental level, I think that a company is always going to be motivated to make money rather than solve a problem. Even if intentions start off well, the people in the company will believe that the company is good and needed, so they'll make choices that give up a little of the original purpose of the company to make the company survive, because if the company doesn't survive then it can't do any good, right? And after a bunch of one-degree deviations from its original direction, the company is going a completely different direction than its original purpose and isn't even geared toward solving the problem any more.
Sure, "We're going to save the world" is great marketing, but at this point I simply don't ever believe that a startup is a solution to a major problem like this.
Publishers need to be involved earlier in the process. Instead of submitting a completed research finding, scientists should apply to publish research before doing it, and if the topic of research is interesting to the publisher, both scientist and publisher should commit to publish in that journal, no matter what the results are.
Additionally, publishers should commit to publish only something like 1/3 research into new topics, with the other 2/3 being publishing attempt to reproduce previous interesting results. I don't mean a slavish repetition of existing experiments, but taking a critical eye toward testing the same hypotheses with different (hopefully better) methodology.
Publication is a sign of persistence, plausibility and a minimal quality check. Peer review improves articles, it does not guarantee correctness.
That's also why one should be extremely careful about things that are not peer reviewed. Not because the bar is so high, but because it's so low.
https://www.youtube.com/watch?v=ras_VYgA77Q