You have one point "it's a conservative think tank" - which does not mean anything - coupled with another empty statement.
Now I sure don't mind criticism of that entity, but I find it ironic that it is done with such an empty statement that's just a not very cleverly disguised type of ad hominem attack.
May I suggest you take some time actually reading the paper and then coming back with criticism of their actual points?
It is absolutely relevant to consider the political motivations of publishers of reports like this, when the publisher is a politically active organization as opposed to an independent research group.
While the report itself may not push an obvious partisan agenda, a critical reader should consider the conservative agenda of the NAS in relation to why they might publish such a report. In the conservative wing's long-standing opposition to science, it's not far-fetched to view this report as being published tactically -- even if all of its points are well-founded -- as something to point to when future efforts attempt to undermine science-based policy on issues like climate change, stem-cell research, abortion, etc.
Especially when the President of the NAS is a public "climate skeptic".
A position paper aims to make a political position appealing, a study about the sociology of science on the other hand would seek explanations for a position. Pointing out the purpose of something isn't a character attack.
That you're being voted down shows that HN has really changed.
(I blame all the overtly political articles that I believe have changed the tenor of the site, and politics has infected every article now. Thanks @dang!)
If you won't stop posting this off-topic meta noise that we've asked you many times before to cut out, we'll ban the account. Post according to the guidelines or not at all. Fulminate elsewhere.
From the intro: The new, alternative view, was that college and universities should be places where fresh ideas untrammeled by hidden connections to the established structures of power in American society should have the chance to develop themselves. In practice this meant a hearty welcome to neo-Marxism, radical feminism, historicism, post-colonialism, deconstructionism, post-modernism, liberation theology, and a host of other ideologies.
Also from the intro: The National Association of Scholars (NAS) has long been interested in the politicization of science.
not sure what you're getting at. are you saying meta analysis is more important than individual studies? but if a large % of your studies in the meta analysis can't be reproduced, what faith do you have in the meta analysis at all?
Take into account that meta analysis should not only consider the results of its participant studies, but also whether there is enough information to reproduce the study and whether it has accounted for possible confounding factors.
How often is a study even repeated in the course of normal scientific research? I think most studies are focused on expanding the research and therefore knowledge about the science being studied. Accepting previous studies as fact is dangerous and could easily lead to a house of cards scenario. I think this is especially true in the very specific, niche studies that are common in today's highly competitive graduate student and publish or die research landscape.
That is why, as a (global) society, we should spend our resources in making sure the data we have is accurate by funding research and repeated studies, and that data should be open to all to verify and build upon.
I think we're passed the point where studies can be reproduced effectively. I actually think the scientific model is falling apart. We've discovered the easy things, and those studies were easy to reproduce. We built a model of science around the low hanging fruit.
How are you going to reproduce a study about a cancer drug? Synthesizing the drug itself is a huge endeavor, never mind finding a group of participants with the exact cancer you need. How would you reproduce the Large Hadron Collider experiments? You'd need to rebuild one of the most technically challenging, most expensive scientific instruments ever made.
There's enough resources to conduct multiple wars and bailout the financial industry every 10 years, but not enough resources to educate our fellow citizens and fund research? (with all results open and accessible to all through this fancy thing called the internet)
Our problem isn't lack of resources. There's 7.5 billion people in the world, I can't believe we've ran out of control groups and experiments to do. Maybe it's a little more difficult with medical edge cases, but there's plenty of other things to research. It's just hard to do it politically, especially with people that are "anti-science", meaning against funding research that might show outcomes that aren't good for them economically, or against their religion.
> I actually think the scientific model is falling apart.
You must have a totally different definition of "the scientific model" than I do. I suspect (although I could be wrong) is that you have an oversimplified view of just what the scientific process is.
The scientific model/process is not a linear list of discrete steps moving from "Ask a question" to "Communicate results"; it is not a discrete and static set of criteria we apply to individual experiments or instrument output.
Berkeley University does a good job of providing visualizations on this topic:
Ok, I read your link. I still think my comments stands. My argument would be that the "Community Analysis and Feedback" section is practically impossible to maintain.
Back in early science days, this used to be a valuable section because the pool of "scientists" was huge and they often crossed disciplines. Now you have all sorts of specialties such that the pool of colleagues who are able to understand and assess your work might be, like, 8 people.
There was a Japanese mathematician who recently put out a huge proof that claimed to solve a major math problem (I am forgetting all of the important details here, so forgive me). The news wasn't so much that he created the proof, but that they couldn't find anyone to verify it!
Most scientists in my profession look for converging evidence from multiple lines. While the same study is rarely repeated, the same concept does get tested multiple times with different approaches.
There are a lot of "new" articles that are just a small tweak of a previous article with a lot of hype to make it appear as a groundbreaking result. So there is a lot of implicit informal reproduction.
You can't trust blindly the previous studies because they may be wrong, or have hidden assumptions. A well known case is the use of a control group. If you want to study a new drug, you must compare the results of the patients with a similar group that is taking the previous drug and/or a group that is taking no drug. You can't just trust that the cure rate of the previous study is 23.573% and compare it with your results. (And there are some trick to get more reliable results, like using a placebo instead of no drug, and doing a double blind study.)
Explicitly replicated, rarely. But other similar studies will either reinforce or call into question the original result. This has a similar effect as replication over time.
The issues mentioned in the introduction affect entire fields, not just individual studies:
> Improper use of statistics, arbitrary research techniques, lack of accountability, political groupthink, and a scientific culture biased toward producing positive results
Are you claiming that reproductively is unimportant when multiple studies cover the same area? Do you think so because you imagine errors in the studies would be uncorrelated? Not the case: look at the social sciences. Errors are very much coordinated.
There are many examples where multiple experiments moved from an original bad result and slowly converged to a correct value. People will throw out results that where overly far from what was expected, but the drift is in the correct direction.
And really science need not move quickly as long as it continues to converge on accurate answers that's plenty useful.
EX: Nutrition research often get's dumped on, but it's far from useless. There are 41 things we know you need or issues will show up: (http://www.nutrientsreview.com/glossary/essential-nutrients + energy + water) Getting the specifics right is complex and we are working on it. Thiamine deficiency from white rice is the kind of thing you need to get right or die, 'optimal health' is a harder target.
This is the exact reason I don't believe science should be used blindly to affect government policy. It flies in the face of democracy and is subject to falsehoods just like any other human endeavor.
Lysenkoism happens in the American and European scientific establishments as well. As an example I would suggest Ancel Keys and the cholesterol hypothesis of heart disease.
That's a bad example. It's a much more complex scenario, where the drugs that fight the expected cause of the disease does in fact help protecting against it. Confusing causality is very hard to solve.
If you're talking about statins, Ancel Keys was promoting shoddy science on high fat diets to influence government policy before statins were discovered.
Three is NO way to do that. There are ways to misuse scientific results. Premature generalisation, assumptions of safety. Cherry picking to support preconceptions.
Attributing strong support to tentative and/or unchallenged results. And more...
If we don't use science to make decisions, what are we supposed to use? Gut instinct?
Of course science is imperfect, but it tends towards truth over time, whereas dogma and populism have no way of correcting mistakes that isn't just as likely to introduce new ones.
Racial 'science' was used by the Nazi's to justify antisemitism, eugenics, and everything they did beyond that.
The scientific method is important to be used, but don't trust something just because another person believes it to be science. You need the hypothesis and the verification method. Others (all of us) have the obligation to challenge both the hypothesis as well as the verification methods.
Those views really were thought to be rational and scientific at the time they were promulgated, though, and Nazism was only their most extreme expression.
The problem with the Nazis is that they were genocidal maniacs, not that they p-hacked their scientific papers. Genocidal maniacs have spouted many creeds, some of which sound much nicer (like brotherly love & eternal life... or equality of mankind & ownership of the means of production) and they are still evil. Because mass murder is wrong. Full stop.
I think it's really dangerous to tie the wrongness of the supposedly scientific facts they spouted to the wrongness of their deeds. Dangerous because if it turns out they were right about some point, then you look like an idiot and have to change your tune. And dangerous because it warps subsequent science, to avoid such embarrassment.
They believed in breeding a master race. In response to this, the acceptable opinion in the West ever since seems to have been that this is impossible, that humans are unique among creatures on this planet in that they are immune to the pressures of selective breeding. We can double the size of our chickens (and breed tame foxes, and smart dogs) but humans are apparently immutable. This is creationist nonsense, and it becomes more and more obviously nonsense every year. We need not have painted ourselves into such a stupid corner.
So if the Nazis weren't mass-murders, and instead just had an ideology revolving around the inherent inferiority of anyone other than Aryans (naturally that would probably include bans on intermarriage, restrictions on social race-mixing, and the like), that would have been alright? I don't think I agree.
Frankly, genocide, or at least dehumanization and extreme indifference, seems like the logical endpoint of sincere belief in the concept of a "master race" which is inherently good and is hamstrung by perfidious inferior races. Once you've determined that a member of one race is worth less than another, it's natural to judge that the death of a member of this inferior race would be justifiable to save the race of a member of the superior race, and in this way the concept of genocide becomes, in your mind, a "defensive" action.
They would not have acquired much notoriety by believing these things in private. It's their actions which got them into the history books.
If I'm not mistaken, the Brahmins tick most of your boxes. Are they alright? Some would argue that they go in for a bit of dehumanization... but nobody is proposing a world war to stop them.
Actually I'm alright with people believing lots of crazy shit, in private. Because I think intellectual monoculture is pretty dangerous.
Well, I don't know, the Indian government seems to think caste discrimination is a real problem (c.f. https://en.wikipedia.org/wiki/Reservation_in_India). I certainly would object quite strenuously to a government that operated on caste principles and enforced caste segregation and diminished rights for some castes. World War II did not begin because of Nazis' views on Jews, so I don't know that that part of the question really makes sense.
I do find it a little strange how many people want to turn the concept of "diversity" on its head and tell me the real diversity problem is that we don't let enough people with racist views into power.
Sure, me too. But we're back to talking about policies and actions, like invading Poland or enforcing caste apartheid. These are bad things.
My objection is to conflating them with scientific matters, i.e. empirical claims, especially about human biology. I worry that doing so actually weakens our arguments against such evil. We shouldn't build into our case against them scientific claims on which we may ourselves be proven wrong. It's not necessary.
(And, much as I enjoy using words like diversity in their older meanings, in the hope of tripping people into thinking, here I resisted the temptation. Intellectual monoculture was however a feature of most of the evil regimes of recent times, too, and one of their weaknesses.)
I don't see much point in making the distinction. All of these policies you're saying are bad are completely logical ones to enact for someone who holds race-supremacist views.
By way of analogy, when someone tells you he believes global warming is impossible because it would go against the Bible, you don't have to wonder that much what he would do about carbon emissions.
The issue is that it's not easy for the populace to tell the difference between 'science' and science in a lot of cases. You see this on all parts of the political spectrums. Soviet Lysenkoism, left wing anti vax crap, and neo liberal austerity.
EDIT: Krugman had this to say about the R&R austerity paper "What the Reinhart-Rogoff affair shows is the extent to which austerity has been sold on false pretenses. For three years, the turn to austerity has been presented not as a choice but as a necessity. Economic research, austerity advocates insisted, showed that terrible things happen once debt exceeds 90 percent of G.D.P. But "economic research" showed no such thing; a couple of economists made that assertion, while many others disagreed. Policy makers abandoned the unemployed and turned to austerity because they wanted to, not because they had to."
Particularly telling: "The paper was published in an annual 'Papers and Proceedings' edition of The American Economic Review that was not subject to the same peer-review standards that other editions use before publication"
The same Krugman who pronounced the markets would never recover from Trump's election or that since we didn't pass a large enough stimulus we were doomed to low GDP growth rates forever?
Don't just listen to Krugman, read how monetary policy for the better part of a decade was based on a paper that wasn't peer reviewed, but was held up as irrefutable 'science'. Politicians latched on to it because it told them what they wanted to hear.
No he's wrong because he supported a policy which saw the US debt double in 8 years, and the deficit go from 52% of GDP to 77% of GDP, and resulted in the weakest recovery in US history. And all the people who agreed with him in their widely accept science, were also wrong.
I think that is simply another misinterpretation of what people mean when the word science is used.
What was said above was "one of the most horrific and important examples of widely accepted science". Important words here are "widely accepted science". So science didn't go wrong, the LABEL of science was attached to something that didn't meet the scientific method. That something was then sold to people in power, and the science LABEL was used to convince people of the correctness of that something.
>The data was bad, the processes were bad, there was no replication.
People are really quick to say this, but how many people have actually studied the 'racial science' literature of the 20s and 30s and figured out to what extent it meets the methodological standards of today? Yes, we know that they drew incorrect (and reprehensible) conclusions. It doesn't necessarily follow that their methods were any worse (epistemically speaking) than methods that currently pass muster.
Studies about people in general are of poor quality, due to difficult to define metrics. Even today, psychology and sociology studies aren't rigorous for the simple reason that it's hard to measure these things. Biological studies are hard too, it's hard to suss out the confounding variables and mechanisms of bodily functions.
Just the premise of the "racial science" studies would render them invalid, how could they even have controlled for the myriad variables that we have no idea how to control today?
Which studies are you referring to? If you don't have anything specific in mind, then you can't really judge the quality or compare it to the quality of comparable research today.
As I wrote above, studies about people, which the aforementioned Nazi stuff would fall into. I don't have a specific study in mind, but I do know that the further you stray from math/physics/chemistry, creating metrics and isolating variables is very, very difficult.
I don't need a specific study to know that. Any sociology/economic/psychology research should be taken with a grain of salt.
>Any sociology/economic/psychology research should be taken with a grain of salt.
So how is that not an example of "science going wrong"? Unless you are proposing that science is right by definition, or that governments should ignore most of the branches of science that are actually relevant to government.
There are plenty of branches of physics that are at least as speculative as sociology, economics and psychology (e.g. cosmology). The only difference is that these branches of physics have few political implications.
It's not going wrong, unless it's making conclusive statements that are beyond the strengths of the study. That's why I say grain of salt. I understand what you're saying, but my point is that it's not science or the scientific method that results in undesirable outcomes, such as the Nazi example, but just a lack of any other good option.
Governments (and people) should evaluate the merits of the information they receive. Properly conducted and repeated experiments with noted mechanisms of actions (e.g. vaccines) should have more weight than studies about trickle down wealth or whatever. Obviously, decisions have to be made, and some will be wrong unfortunately.
>Properly conducted and repeated experiments with noted mechanisms of actions (e.g. vaccines)
This is a good example of why it's not so simple as you're making out. Earlier, you put biology on your list of unreliable branches of science, and you claimed that you could know that the racial science of the 30s wasn't "real science" simply because it consisted of "studies about people". But vaccine science falls into both those categories!
In reality you're deciding whether or not any given branch of science is methodologically sound not by examining any actual studies in the relevant disciplines, but simply on the assumption that any study which has arrived at the "wrong" conclusion must have had a bad methodology.
Why would that be helpful? The problem is obvious because of the complexity of the variables and problems measuring them. There might be some problem studies cited in the link below:
The point is that some research is weaker than others, and the reasons why haven't changed since WW2, so if it's a problem now, then it was a problem then.
Well OK, sorry maybe I mixed up your point of view.
Soft science has many questions to which we'd love to have solid answers. So (perhaps unfortunately) it's not going away, and I think we need to get better at sorting out what it tells us.
This does not seem to be a line of argument that flatters technocratic rule, especially considering that governance can hardly ignore the issues that are the purviews of economics, sociology, psychology, and other "soft" disciplines.
The citation of "Godwin's law" in this way is so tiresome. Firstly, the law just says that Nazis will come up -- not that bringing them up necessarily invalidates one's argument. Godwin himself has endorsed likening some groups to Nazis.
More importantly, the example of the Nazis is useful in an ethical argument, because they're an almost universally acknowledged example of evil. Much like we aim to test our programs with extreme inputs, we can use the "extreme input" of Nazism to test our ethical arguments. If I posit some universal ethical standard, and then find that it doesn't seem to work if we take the Nazis, we've proven by counterexample that my standard is not universal after all.
To be fair, I took a step back from saying Hitler, and I said Nazis, I was one step removed from invoking Godwin's law in its formal definition.
More seriously, it wasn't just this group that believed in this stuff, it was the many many institutions in the western world, as emodendroket points out. Certain departments at Yale, most notably, supported eugenics.
Even if science were perfect it couldn't tell us about moral axioms or value judgments.
Additionally, using science and other expert judgments can easily tend toward simply choosing experts and studies that align with whatever policy you want to push for unrelated reasons, or outright seeking to manipulate research for political ends.
> Additionally, using science and other expert judgments can easily tend toward simply choosing experts and studies that align with whatever policy you want to push for unrelated reasons, or outright seeking to manipulate research for political ends.
Of course it can, but that's equally true for any other way of making decisions. So it's not a downside of science in particular.
It's a downside of any way of decision-making that starts to get imbued with a sense of being beyond question or argument. Many invocations of "science" and "rationality" seem to treat those things more as oracles that "tell us" whatever it is they happen to want us to think than as processes or tools for investigation.
Current scientific consensus is that it is caloric surplus - confounded by a bunch of poorly researched factors. For weight gain source does not seem to matter too much. This does not say anything about dietary specifics or how they affect intake and metabolism.
Technically nutrition science does not provide strong support for such statements as this carb vs fat one based on interventional studies.
Epidemiology alone does not tell you "therefore this change will be beneficial"...
> How long ago was it that science was controverting centuries of populist knowledge that carbs are what made you fat, not eating fat?
I've looked into that and all I can say is the history is disputed - some people say there really was a nutritionist consensus, others that this was a popular misunderstanding based on overzealous popularisation of a small number of papers (similar to the myth that the consensus among climate scientists in the '70s predicted an ice age) and that working scientists never thought this.
Either way it's a failing of the process, but the appropriate response is quite different; in the first case all we can say is mistakes do happen, in the second we need to get better at ensuring that the real scientific consensus gets translated into appropriate policy actions.
If a decision has to be made then science as a guide may be the best option but much of the time it would be better for government not to try and make policy at all in complex areas. Using "healthy eating" as an example, governments might try to influence what people eat by taxing or even outlawing "unhealthy" foods or subsidizing "healthy" foods. Instead of arguing over whether and how much the government should be imposing sin taxes on fats (and what kind of fats?) or sugars (and what kind of sugars), the better approach would be to do nothing and let people make their own choices based on the best information available to them at the time.
> Instead of arguing over whether and how much the government should be imposing sin taxes on fats (and what kind of fats?) or sugars (and what kind of sugars), the better approach would be to do nothing and let people make their own choices based on the best information available to them at the time.
Why? Do we believe that individuals are likely to be better informed than governments? (why?)
Uncertainty is a fact of life. Any decision-making process will lead to some mistakes. That's not a reason to give up on trying to make things better. If the government believes, based on the best evidence available at the time, that a particular fat tax policy (say) will, overall, be good for society, surely it's better to implement that policy than not.
The history of government attempts to get people to eat "healthily" is a pretty good illustration of the many ways that government policy is a bad tool for addressing these type of issues. Government policy tends to have a lot of inertia and is bad at adapting to changing information. This is particularly evident when it attempts to track evolving scientific understanding of complex topics like nutrition and diet and their relation to health. It is also prone to being overly influenced by special interests and this particular area is full of textbook examples of regulatory capture and public choice theory.
I certainly believe that some individuals will be better informed and better able to translate that information into the best course of action for themselves given their goals than governments, especially "governments" as represented by a patchwork of laws and regulations the impact a complex thing like individual health. This is true even at the time that new policy is enacted and tends to become more true over time due to the inertia of government policy that goes down the wrong path.
> I certainly believe that some individuals will be better informed and better able to translate that information into the best course of action for themselves given their goals than governments, especially "governments" as represented by a patchwork of laws and regulations the impact a complex thing like individual health.
Some individuals, sure. Most individuals, probably not.
I agree that government policy has inertia, and that overconfidence on uncertain issues can lead to imposing well-intentioned policy that ends up being harmful. But it's equally possible to make the opposite mistake, and hold off introducing policy changes for too long because of perceived uncertainty. E.g. car seatbelt laws and tobacco advertising laws should have been introduced earlier than they were. I don't think there's any general principle we can draw here; some government policymakers could do with more humility, others could do with more confidence, but all any one of them can do "from the inside" is try to figure out how strong the evidence actually is on a given issue, and use their own best judgement in determining whether there's a clear enough scientific/medical consensus to warrant policy changes, or not.
> If we don't use science to make decisions, what are we supposed to use? Gut instinct?
I thought the parent comment made it clear: the alternative we are supposed to use is "democracy". That is, the people who are ruled should determine the rules that are imposed on them.
Now, democracy does not preclude the use of science to make decisions. The voters may rely on science to inform their decisions (when the science is available). It does however preclude a rule by scientific experts/bureaucrats.
But I also think it's foolish to frame the decision as solely one of "science" vs. "gut instinct". The are many phenomena, especially when it comes to human society, that are too causally dense for us to sufficiently tease out the causal factors to make accurate predictions.
In those conditions, the goal should be to minimize the consequences of being wrong. We do that by decentralizing decisions to the greater extent possible.
How do you know democratizing decisions results in minimax optimal decisions?
I would expect the result to be strongly biased due to societal norms, much less direct manipulation.
Suppose a majority wants strong social programmes and basic income. The tiny problem is the economy is nowhere near big enough to afford it. Satisficing (!) executive branch puts some of the policies starving innovation and education budgets. Causing long term harm and stagnation.
Where is the minimax decision in here? Over what timeframe?
(P.s. I could've as easily picked liberal, fascist, military or conservative example. This one is relatively easy to understand.)
I should clarify that there were two parts to my answer: first was answering your question "on behalf of" the parent poster. It seemed their position was to "let people vote on it". I was somewhat defending that position, but more just clarifying that democracy wasn't necessarily in conflict with using science to make decisions.
The second part of my response was to your framing of "science vs. gut-instinct". My point is that sometimes ignorance is thrust upon us by the complexity of the circumstances. In that case, my point was that decentralized policy making (separate from any notion of voting) was essential to reducing the worst case.
In other words, I am not claiming that democratization "results in minimax optimal decisions". I am claiming that decentralized governance structures do "result in minimax optimal decisions". Does that change your question?
> In those conditions, the goal should be to minimize the consequences of being wrong. We do that by decentralizing decisions to the greater extent possible.
In some cases yes. In others this doesn't seem to be the case - e.g. decentralised housing/planning/zoning policy seems to have lead to worse outcomes than more centralised approaches.
I haven't surveyed zoning policies, so I don't know whether I agree with your conclusion, but I'd like to clarify a couple things.
First, we must ask ourselves what the goal of optimal zoning policy is. Is it to maximize the number of people who live there? Achieve an ideal rent/wage ratio? Maximize the efficiency of shared infrastructure?
Perhaps it is to maximize the desirability of living in an area. If that is the case, then if people are complaining about the house prices that may mean you're doing something right.
In any case, assuming we settle on a shared metric (which may involve a combination of these various factors), we need to distinguish between comparing the average case and the worst case performance of centralized vs. decentralized zoning policy.
I would say that my claim is that decentralized zoning policy will have a significantly better worst-case than centralized zoning policy, and that crafting a good zoning policy is enough of a gamble that I'd rather settle for a less-than-ideal decentralized policy than risk a horrendous centralized one.
Now, to tie back to the original debate of scientific expertise in particular: would adding a group of economists and social scientists to the central planning committee tilt the situation in favor of central planning? I very much doubt it.
"At the core […] is the idea that people should design for themselves their own houses, streets and communities. This idea […] comes simply from the observation that most of the wonderful places of the world were not made by architects but by the people."
— Christopher Alexander et al., A Pattern Language, front bookflap[1]
Because the costs of building new housing are local but the benefits are global, or at least more widely distributed than the costs. If one borough builds denser housing, that borough is the one that has to worry about building work, plumbing, schools and all the rest of it for those residents, while businesses from all its neighbours benefit from more customers and cheaper employees. For a starker example of the same effect think about taxation and public services: if we relied on local taxation being spent locally then rich areas would have low tax rates and high-quality public services (because even at low rates they can still raise a lot of money) while poor areas would have both higher taxes and worse services.
Localised government is rather like privatisation: it works well when the costs and benefits are on the same scale, but rather less well when there are externalities.
too causally dense for us to ... make accurate predictions.
And even if we had perfect predictions, we would still have to choose what to do.
For example, if we had a perfect economics, we could predict exactly how many people would be made unemployed compared to how many made how much richer by raising the minimum wage (or trade barriers, or taxes). How should we weight these?
I think it's easy to greatly overstate how useful science is to policy-making. There are narrow areas where they should (and do) call a specialist in. But the big debates are not over factual, empirical, matters.
The problem with this approach is that once you toss science there is not much left to inform policy decisions (religion? profit? personal views?).
One less drastic solution may be to require a full disclosure of the process for all experimental results (to allow reproduction) and place much higher value in funding decisions on reproducibility.
One particularly egregious approach reproducibility requirement can address is doing enough "independent" studies until you like the result, then report that one study only (e.g., toss a coin 10 times; repeat as a new study until you find 10 tails; report a new discovery). My 2c.
> doing enough "independent" studies until you like the result, then report that one study only
This happens not only in which studies get submitted to the public, but also which submitted studies get chosen for publication. Journals are proving to be a poor way to handle the amount of science humanity is doing.
Heuristics: use what has worked in the past and keeps working well enough today? You don't need to fully understand why it works. Aka "conservatism" in the broadest sense.
Exactly. Traditionalism isn't perfect, but it acts as a sort of long-term experimentation. That which has worked over long periods of time can be a good default to stick with. Environments do change, but the nature of humanity and society also has constants over time that are important to consider.
I agree it shouldn't be used blindly but that seems like the extreme. In fact the science side of things would say don't use anything blindly. Science isn't some infallible entity it is a process that is supposed to be self correcting over periods of time. That includes falsehoods from human bias and input. It doesn't always work and it is a slow process but in science's essence it is supposed to be minimizing those errors.
But again, getting back to agreeing, scientific findings should not be used blindly, they should be used to inform. And as reproduction of findings occurs, the the topic that is being informed gets re-enforced or course corrected. It is very similar to policy, you have to try things. Some times the policy does what you want, other times it doesn't, and the process should continue and be correcting.
Of course science can and should be used to inform government policy, but that's a bit removed from technocracy, which I think is what the OP means to describe.
Of course science flies in the face of democracy. Democracy gives all the power to individual subjective opinion. Hence the battle against emotional bias, fake news, and propaganda tactics which all work wonders in a democracy. This battle is fought with objective facts, which are only attainable by science.
Science is the only tool specifically designed to weed out falsehoods and reach an objective consensus based on how nature works.
How is this "subject to falsehoods just like any other human endeavor"?
There is no substitute for science.
We all seem to agree that democracy only works when the population is informed. Well, are we going to inform ourselves with the beliefs and tweets of those which we grant moral authority? Or science?
This quote from Einstein on the subject sums up my thoughts on this as well.
"Science, however, cannot create ends and, even less, instill them in human beings; science, at most, can supply the means by which to attain certain ends. But the ends themselves are conceived by personalities with lofty ethical ideals and—if these ends are not stillborn, but vital and vigorous—are adopted and carried forward by those many human beings who, half unconsciously, determine the slow evolution of society.
For these reasons, we should be on our guard not to overestimate science and scientific methods when it is a question of human problems; and we should not assume that experts are the only ones who have a right to express themselves on questions affecting the organization of society."
I mean, this is still a good thing, since it charts a map of decidedly grey areas, where information may always be ambiguous, and useful information needs to be sussed out carefully.
It's better than presumptively assuming that "Science" is an infallible always black-and-white.
Recently I have been beginning to question university education. It seems that today's education system has mismatched priorities which may have an impact on the reproducibility crisis. You see, to come up with a fantastic new alloy, you don't have to be a genius. Or even very good. You have to be persistent, and clever enough to realize when you've found something interesting. You will need to understand your field very well, but there are many, many fields. And each one is quite narrow, so while it is hard to understand your field well, many, many people can do this.
So my less dystopian future goes like this. Train lots and lots of physicists, chemists, engineers, programmers, materials scientists, biologists, mathematicians. And (shock horror) through higher taxes employ them. Employ them on promising ideas, and on impossible ideas. Employ them on the problems of the day, and on arcane research that will almost certainly never bear fruit. And sure, right at the top in the Apples and Googles of the world, you'll have the best and brightest pulling together all of the discoveries made without all these reproducibility issues.
Or we could continue down the dystopian path. The low tax path where there really are only meaningful jobs for the best and the brightest. Where we rely on Apple and Google to decide what ideas are worth pursuing. Where the skills and abilities of a great many are not used. And most of us fight for the crumbs from the oligarch's tables. And the oligarchs? For every Elon Musk, there will be 10 who just want the biggest yacht and the prettiest mistress.
Let me say that again. In a low tax environment, the skills and abilities of a great many will not be used. And we will all be poorer for that. Genuinely poorer, because great discoveries will not be made. And it is happening now.
And I'll go one step further. We are also wasting the talents of our best writers and artists. The BBC and other national broadcasters are squeezed. Squeezed so that the idea of something as original and amazing as Monty Python appearing now is laughable. And again, we are all the poorer for that. Not just the lucky ones who get to make amazing shows or works of art, but all of us who never get to revel in them.
The low tax, small government world is a step backwards. If you wish to sum it up succinctly, you may say it is a world where, in order to placate the very rich, we take away opportunities from many, and impoverish the world as a whole.
> Recently I have been beginning to question university education. Why are we bothering to educate anyone except the very, very best students in physics, maths and engineering? In fact, why aren't we identifying these students and sending them straight to Caltech? It seems intuitive that this would solve one half of the reproducibility crisis.
Haven't downvoted, but I'd guess it probably starts with "Why are we bothering to educate anyone except the very, very best students in physics, maths and engineering?"
Sure, we will undoubtedly get better society-wide results if we identify a subset of elite students (notably defined primarily in terms of STEM fields) and only "bother" educating them, leaving everyone else to Just Trust Them™ and underutilizing the long tail of talent.
And then there's the premise that society somehow isn't looking to utilize the existing population of elite students, which of course it is, no shortage of organizations looking for outliers.
About the only good point there is that our system for allocating the best/brightest to certain problems could use improvement.
That would never work. We have no accurate way to identify the "best" students so early in the educational pipeline. High school grades and standardized test scores are poor predictors of who will make major contributions to science.
More public funding for basic research (including reproduction studies) would probably deliver major long term benefits.
Intelligence is such a spectrum that if you select individuals on traits you rot through lack of diversity. A breakthrough will generally come from someone with feet in utterly different fields.
Concentrating on grades in STEM poisons the well of raw thinking that science needs to look at everything in unique ways.
The only way that we can move forward is that everyone moves forward.
I'm not sure I see how low taxes inherently cause the misallocation of talent or more broadly, the underspending on R&D.
I do suspect that profit motive isn't enough to drive effective R&D, though, I just don't know exactly how to articulate a model for why except at the basic level that that truly searching R&D has a higher risk profile than other opportunities for profit.
I support the idea of more tax-supported public funding for a variety of things. I'd also support reshaping the tax code to incentivize R&D. But I'm also not sure that's a going to give everybody their opportunity -- I'd imagine there'd still be lucky ones there too. And we'd still have a lot of work to do to address incentives that drive reproducibility issues.
> Train lots and lots of physicists, chemists, engineers, programmers, materials scientists, biologists, mathematicians. [...] Employ them on promising ideas, and on impossible ideas.
The scientifically minded are not the ones who will be deciding what's worthy of research. Politicians and bureaucrats will, and there are plenty of them who believe in all kinds of pseudo-scientific nonsense. How does your system prevent them from shoveling tax dollars into research on homeopathy and young-Earth creationism? And how does it prevent them from using funding to incentivize scientists to find the "correct" results?
To phrase my question more generally: I would like to know how your system would avoid making science merely a tool of political interests instead of a tool of moneyed interests.
Some of the comments miss an important distinction.. Science in the popular press, often refers to science versus completely non-science ways of forming an opinion or deciding policy. Meanwhile, science within a technical community, is subject to human error and manipulation, and relies on a reproducible result, as well as peer-review, to find answers to conflicting claims.
There are certainly non-Science ways of forming an opinion and deciding policy, in many cases totally legitimate. But once science is claimed, then of course it has to be subject to science rigors.
Yes, thank you for saying this. I feel like, often, people use science as some ethereal being. It is a process that has to be used. If that process isn't happening it isn't science. That doesn't mean everything has to be science, but things that are science should be held to a certain standard but should also be taken with the extra weight if that standard is met.
It's important to make the distinction between Pure Science and anything on the sliding scale from technical description to ELI5. There are levels to it, you and I know.
Oh absolutely there is a sliding scale on both the actual practice of science as well as the reporting. From the practice of it, I know as simple anecdote I have participated in conducting research in hard science (physics), softer hard science (biology), social science (behavioral and economic), as well as straight policy analysis with a "scientific approach" through data science. The findings from those are reported differently and have stronger or weaker words and conclusions. I think that is where the practice sliding scale comes in, is how the researcher decided to write up and report their findings and appropriately identifying their methodology and short-comings.
From the journalism side I think you end up with the problem the wide spread field of journalism has, you have to be read by your audience. That means something with a wider audience is probably going to be more on the ELI5 side which means both the reader and the journalist have to take any reported findings more lightly.
The sliding scale, I believe, comes in with the rigor something is reported. That is both in the practice and the journalistic side. More details can give a more granular understanding showing limitations but some times broad strokes are necessary to get some information out to a wider audience or more quickly.
Never mind the massive debate that occurs within the scientific community itself on some new or controversial claims. When the press gets its hands on the views of one dissenter and one proponent, it makes it seem like there are only sides A and B. This is not as simplistic as politics, where politicians form into immutable groups Left vs Right. We are dealing with numerous camps, and within each scientific camp, there are numerous arguments made for/against an issue.
Take climate change, no doubt a controversial issue. To say it is "controversial" in the political sense would mean Left and Right (in the USA alone) have vociferously different stances on the issue. However, if viewed as a scientific controversy, we are now talking about detailed methodological concerns, like methods of data collection, analysis, kinds of statistical bias, and subtle changes in arbitrary parameters. Any scientist can tweak this or that in their model to make it conform more easily to their preconceived notions about climate change. Unfortunately, we also have some oil industry shills out there who got trotted out as an equally weighted side B to the side A of the dozens of scientists who would generally disagree. Then, you also have anti-science proponents who use the legitimate self-criticism of scientists to attack science as a whole.
It's a sad state of affairs. Science should be reported in the press, but it should also be reported much better than it is. In the USA in particular, STEM education is lagging behind: the average person can't delineate the good science coverage from the bad, and we have ridiculous notions and conspiracies that fail to become filtered out (anti vax, climate change deniers, flat earthers, moon landing was faked, etc.)
For me, reproducibility is one problem in a broader ecosystem of scientific problems, including science education generally, as well as misuse of statistics, and a saddening drive for incremental results at the expense of more broad-based thinking which might lead to fundamental breakthroughs. Our education systems must be reformed to deal with these problems, that's the only way out that I see.
We did an article trying to help people be better critical thinkers when they hear about psychology research. I thought the author did a good job getting into how these studies go wrong and also the types of magical results you should be suspicious of.
https://medium.com/@jhreha/most-psychology-research-is-bs-73...
We know that weak classifiers can be bagged to produce a strong classifier al la Adaboost.
Each study is a weak classifier and would have a 'reproducibility crisis' if retested on new data. However after lots of studies of similar phenomena, a strong classifier emerges. In the field, we call this converging lines of evidence.
I'm not sure I really agree with this, because in science studies are biased towards aligning with the results of previous studies. This can be due to many factors: authors' expectations, journals like to publish positive over negative results, peer reviewers will look more critically (in an analytical sense) at work that disagrees with accepted research, etc....
It can be very hard for many reasons to stand up and say "this prior work is wrong".
The argument is not that all is perfect in science. It is not. The argument is that strong replication for each study is not required for progress. Back to my analogy to the algorithms like Adaboost. Why ever aggregate over a a bunch of weak classifiers over a single strong one? Well, please correct me if I am wrong (machine learning is not my field), but the primary advantage is computational cost. Sometimes, for the same overall classification performance, training hundreds or thousands of weak classifiers requires fewer computations than training the strong classifier.
To me it brings up an interesting point. If we view experiments as classifiers, then how would a machine learning expert set science policy and practice?
Also viewed this way, it makes me wonder about p-hacking, which increases sensitivity while increasing false positives. Since negative results are not generally reported, I wonder whether p-hacking diminishes replicability at the study level for efficiency at the aggregate level. This is an empirical question, as ethically it is of course a dubious practice under current understanding.
The ensemble advantage is completely the other way around - we might choose to use a large ensemble of classifiers because they can get slightly better results than the best single strong classifier we can make; and we might choose not to use an ensemble because of computational cost reasons, especially because you train a model once but infer forever (and likely on more limited hardware) and inference for a hundred classifiers takes a hundred times more computing power even if you skimped on training length. A good example is the Netflix Prize, where the best accuracy was achieved by a large ensemble, but the practical implementation afterwards chose a slightly less accurate non-ensemble approach for performance reasons.
What you describe sometimes happens in a tradeoff between very different ML models (e.g. an ensemble of decision trees versus a single deep neural network has the properties you describe), but within any single paradigm (an ensemble of neural networks versus one NN with more training time) it's the other way around.
Always willing to learn more. I was going off of https://en.wikipedia.org/wiki/AdaBoost
which mentions "Unlike neural networks and SVMs, the AdaBoost training process selects only those features known to improve the predictive power of the model, reducing dimensionality and potentially improving execution time as irrelevant features need not be computed."
edit: When I read about ensemble theory, you receive support for the increased computational cost.
Evaluating the prediction of an ensemble typically requires more computation than evaluating the prediction of a single model, so ensembles may be thought of as a way to compensate for poor learning algorithms by performing a lot of extra computation. Fast algorithms such as decision trees are commonly used in ensemble methods (for example Random Forest), although slower algorithms can benefit from ensemble techniques as well.
But I found this interesting:
Empirically, ensembles tend to yield better results when there is a significant diversity among the models.[4][5] Many ensemble methods, therefore, seek to promote diversity among the models they combine.[6][7] Although perhaps non-intuitive, more random algorithms (like random decision trees) can be used to produce a stronger ensemble than very deliberate algorithms (like entropy-reducing decision trees).[8] Using a variety of strong learning algorithms, however, has been shown to be more effective than using techniques that attempt to dumb-down the models in order to promote diversity.[9]
..which under my analogy would seem to suggest that exact replication is less productive than having a diversity of study designs.
Not quite. If you read the latter paragraph closely, you would note that an ensemble of stronger models performs better than a multitude of dumb models.
Then there is a problem that an ensemble of biased classifiers (mostly in the same ditection in this case - positivity bias) will magnify the bias.
This is a reason why metaanalyses have to get at the actual raw data to pool and analyse as well as correct for multiple biases. Even then the process is not perfect.
In response to several threads here: it is important to distinguish when scientists are self critical vs. when non-scientists are critical of the scientific method. For instance, there is a long history of scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor. That work is sometimes used by non-scientists who question the overall scientific method. However, such use is invalid as the scientific self-criticism
1. assumes the validity of the scientific method
2. relies on the scientific method as its critical lens
Whereas those who critique science as a whole:
1. assume that the scientific method does not work and does not arrive at "truth"
2. then use scientists being self critical to prove #1.
Such a "proof" does not work as there is its uses the assumption "the scientific method arrives at truth" to derive the contradiction "the scientific method does not arrive at truth". See for instance comment: https://news.ycombinator.com/item?id=16859200
In reality, work on reproducibility is about improving the practice of science overall. It does not in itself show that science is inherently untrustworthy. What it does show is that scientific discovery is difficult and it takes a lot of effort and new findings should be treated critically. What does critically mean in this context? It means with in the boundaries of science analyzing the theoretical basis, hypothesis, method, and experimental results for potential flaws. It does not mean to be skeptical as a default because science "doesn't work."
I agree with your overall point, but technically speaking it is logically valid to prove a hypothesis false by first assuming it and then deriving a contradiction, even when the contradiction is the negation of the original hypothesis (as it is in your example).
What you should have said is that some critics start with the premise "the scientific method does not arrive at truth", and then use other people's arguments that depend on the premise "the scientific method arrives at truth" to support their claim, which is indeed logically invalid.
>scientists criticizing how the scientific process is currently conducted for the purposes of improving the scientific endeavor.
I think what happening here is a bit more serious. They are showing a widespread crisis. It is not just some minor feedback to improve the process.
>It does not in itself show that science is inherently untrustworthy.
I think when statistics is involved, the results are inherently untrustworthy. This is not really surprising because there is a whole bunch of ways these studies that involve statistics could go wrong. And we are still finding new ways on how this could go wrong.
Then there are things like publication bias, that takes this to a whole new level. Things like that means that a biased body of journals can project any consensus that it favors just by selecting studies that fit its narrative. The inherent issues with statistics means that you can find studies that shows any possible outcome.
>I think when statistics is involved, the results are inherently untrustworthy.
Ummm...are you kidding? Statistically vetted results are inherently UNCERTAIN, but how could they possibly be inherently untrustworthy?
Even if a mechanistic effect is observed, its relationship to a particular cause or influence is only established statistically. In fact, the very observation is often performed u Dee the umbrella of statistical calibration of appropriate instruments.
As Pearson said, "Statistics is the grammar of science".
I cannot speak for the person you are responding to, but in my [agreement] of his critique of statistics, I am implicitly speaking of social statistics. I think there is a vast difference in e.g. a statistical modeling of the behavior of electrons and e.g. the statistical modeling of some sort of human behavior.
...but that just isn't true. Statistical analysis is (among other things) a way to quantify our uncertainty. If done approriately, the statistics simply communicate the role probability played in moving from the experimental premises to the results.
The level of uncertainty in most (perhaps all) experiments involving particles in a vacuum is far lower than experiments involving human behavior. Statistics doesn't create the uncertainty, it communicates the uncertainty.
I can use propositional logic to arrive at the conclusion that "Unicorns are real" yet I don't encounter many people throwing logic itself under the bus because of that.
People lie while speaking English every day yet I don't encounter many people claiming that English is inherently untrustworthy.
Statistics has been a whipping boy for a long time by people who don't know much about statistics or just fail to think clearly about what is being said.
I'm not entirely sure what you're trying to say, and your post formatting is not helping.
I would recommend reading the article. The executive summary summarizes things. The issue is not statistics in and of themselves, but how they are used in the overwhelming number of studies - particularly those in the social sciences or related with human physiology.
This is the whole point. It is the difficulty in doing so, or the difficulty in deciding if this was done appropriately or not is the only thing that it untrustworthy..
>People lie while speaking English every day yet I don't encounter many people claiming that English is inherently untrustworthy.
No one expect a statement to be true simply because it was uttered in English.
>"I think when statistics is involved, the results are inherently untrustworthy. This is not really surprising because there is a whole bunch of ways these studies that involve statistics could go wrong. And we are still finding new ways on how this could go wrong."
Another very real issue here is that malicious use of statistics can be used to show nearly anything in ways that can be extremely difficult to detect, even when the maliciousness is hidden in plain sight. And then going a step beyond that there's plain old number fudging which is almost impossible to prove since variance works as sufficient plausible deniability. And finally there is of course plain old ineptitude. Like you mention even when trying to do things completely by the book, statistics are incredibly difficult to get right.
Something that comes to mind here is the recent MIT study stating that Uber drivers earned $3.37/hour. That study was completely broken. [1] It's debatable whether the cause was maliciousness or ineptitude, but the point is that these problems arise, with a disturbing regularity, even when the most reputable of names are attached to them.
Feynmann said, "The first rule is that you must not fool yourself. And you are the easiest person to fool."
There can be malice, sure. But there can also be desire to believe. And, hey, here's a statistical analysis that shows what the investigator is already biased to believe anyway...
The site is down so I can't read the original report, but I've read reports on this topic in the past so I'm going to chime in with some "usual suspects" caveats:
1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter, but enumerating those has to be part of the theory you are testing, and so you can never be 100% sure that you have a complete list.
2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science.
3. The end-product of science is not truth, it is explanations of observations. Those observations can (indeed must) include non-reproducible ones. Sometimes the explanation of non-reproducible results is "experimental error" or "delusion" or "we just don't know." But non-reproducible events are nonetheless within the purview of science.
On the other hand...
4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That, combined with the suppression of negative results, guarantees that the results published in scientific journals that adhere to those standards will be unreliable. But that doesn't really have anything to do with reproducibility per se, it has to do with the fact that the journals use a weak criterion for defining positive results. That, combined with our human predilection to value positive results over negative ones and the understandable desire of scientists to advance their careers, all but guarantees that journals will contain many defensible but false results. But this is not because of a lack of reproducibility per se, it's because of poor policy choices.
When did this become a "standard"? Most papers I have seen in respectable journals and other sources would not strive for such a high Pvalue, it would be much lower like 0.001.
I'm guessing you meant to say that no papers are published with P<0.05. And of course they are, but that's irrelevant. What matters it that many journals, especially in "softer" sciences (i.e. not physics) use p<0.05 as their threshold, which guarantees that at least one paper in 20 will be "wrong". Then add self-suppression of negative results and the actual percentage of "wrong" papers becomes higher, potentially much higher if real positive results are rare.
Part of the mistake here is in interpreting the p value as being a measure of anything really meaningful. P<0.05 doesn't mean 1 in 20 are wrong. It does mean that in the narrow confines of the definition of the statistical model used, it is possible that 1 in 20 would be explainable by the null hypothesis.
It is not, in any way a measure of reality. It is a guess at how accurately you've modeled the system, how accurate your priors are, and so forth.
> It does mean that ... 1 in 20 would be explainable by the null hypothesis.
That seems like splitting a pretty fine hair to me, particularly since I haven't actually committed to any of the many possible interpretations of the word "wrong".
p>0.05 has been a mainstay in entire fields of study for quite some time. Most notably, the psychological and biological science journals have unofficially endorsed 0.05 as a cutoff for "significant" results.
While not everyone in a particular field has been a party to this myopic view of statistical significance, enough have been that the American Statistical Association released a statement on the topic not long ago. That may not seem like a big deal but, given the organization's historical reticence to officially weigh in on such issues, it really is.
Most people would say in the 20s and point at Fischer's publications, though you can have arguments for later and earlier.
Some of that is mentioned in the Wiki page, and you can see 0.05 all over the article [0], and the other related wiki pages[1]. You can find some key quotes there:
"The significance level for a study is chosen before data collection, and typically set to 5% or much lower, depending on the field of study"
"In 1925, Ronald Fisher advanced the idea of statistical hypothesis testing, which he called "tests of significance", in his publication Statistical Methods for Research Workers. Fisher suggested a probability of one in twenty (0.05) as a convenient cutoff level to reject the null hypothesis.In a 1933 paper, Jerzy Neyman and Egon Pearson called this cutoff the significance level, which they named α."
If you mostly see lower p-values, that most likely means that you usually read non-social science papers. I guarantee that most social science (and related) fields use 0.05 regardless of the respectability of the journal.
As an example of fields with lower p-values - 0.001 is common in medicine, and physics commonly has even lower values.
I think this depends on the field. In some areas of neuroscience for example, brain data can have extremely high variability and is very expensive to collect a lot of, so p<0.05 is not uncommon.
I'd add a last point that a scientific paper (or a particular experimental result) is not sufficient for 'science'. Citing a paper as either a truth or discovery is a disingenuous. The endeavour of science gets at a kind of consensus by having lots of papers, lots of experiments, lots of additional layers of experiments build on each other. And yes it's annoying to realize a given paper is misleading, has errors, or is even wrong - but that annoyance and workarounds are part of the scientific process. It's actually fairly rare for an experiment or paper to be strong enough to stand completely on its own as a singular, thorough, and completed discovery. If you've been part of the scientific field you know that it yet takes years after even a good publication for the knowledge to accrete to the greater scientific body.
Some of the best advice I got was to spend a lot more time than one might want to really come up with an experimental design that avoided a statistical outcome and instead probed at an either/or mechanism. It's not always possible, but when it is, it can be a powerful reply to fighting statistical battles (see the Michelson-Morely experiment for a classic example).
Further careers, families, reputations, humans are also part of that scientific process, which together with the above require and admit some deviation from perfect efficiency. Which we as a society should demand be okay.
Thank you for saying this! This is a common view I see outside academia, where people interpret a single paper to mean that something is definitively shown.
I think news organizations make this worse. Whenever a new paper comes out they'll champion it as fact if doing so leads to clicks.
Peer reviewers can only hope to validate that the science was done well, not that it is inarguably correct. It requires a body of work around a topic before it's fair to say at all that something is "for sure" proven.
We need a journal that exclusively publishes papers with results that were totally contrary to the hypothesis.
So long as the researchers aren't outright inept or fraudulent, what's there to be ashamed about? Those would be some of the more interesting papers to read, in my opinion.
Negative results are really hard to do right, and even harder to do interestingly. There are so many possible confounders in experiment design and execution that they generally aren't worth evaluating.
So we choose to spend our time reviewing and evaluating those experiments that do seem to have at least some chance of saying something interesting. Even then, though, depending on the discipline, if the researcher did everything perfectly, they're still saying that there's a 5% or some% chance that they just "got lucky" and there is no underlying effect at all.
I would actually prefer that each and every journal devote at least half of their space to negative results. I have seen some sublime experimental setups that produced negative results. I can't begin to imagine how instructive it would be to see more such experiments.
>For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science.
I am not sure this is correct. The rules that govern celestial bodies is same as the ones that govern objects on earth. So why are they not reproducible? It does not require to measure forces between celestial bodies to measure the value of G. Measuring the forces between two masses on earth is enough.
Yes, of course the rules are the same. Figuring this out was the event that launched the entire modern scientific endeavor. But nonetheless, the experimental data that went into this discovery was largely non-reproducible. The stars and planets go where they go and only very rarely does a given configuration repeat itself. In fact, no configuration ever really repeats itself in every detail.
So, for example, we can predict with ridiculous accuracy where the next total solar eclipse is going to happen. But once it happens, we can't roll back the clock and make it happen again.
> But nonetheless, the experimental data that went into this discovery was largely non-reproducible.
I am finding a hard time to grasp what you are telling here. If you deduce a set of laws A from an event X, you don't need X to repeat to check the validity of A. You can check it against another event Y, where you predict its characteristics using A. If the laws are valid, the prediction ll match the actual observation..
Yes, all that is true. What does that have to do with reproducibility? You're talking about doing a lot of different experiments to test a single theory. Reproducibility is about doing the same experiment over again to see if you get the same results as the first time you did it, i.e. reproduce the results.
That is exactly what I wanted to ask you originally when you cited the celestial mechanics example. It is not relevant in cases we are deriving a theory underlying the behavior, rather than axiomizing the very specific behavior itself...
Suppose you see an object A, moving straight line getting caught in the gravity of object B, ending up in a orbit around it.
Deriving an underlying theory would be deducing the force of gravity from it.
Axiomizing the very specific behavior would be making a rule that says. "If the object A (and object A only), while moving in a straight line, comes at so and so coordinates with respective to B, will end up in an orbit around it"
So these are two completely orthogonal issues. The question of what you do with the result of an experiment is completely independent of whether or not you can reproduce that result. The latter is what is under discussion here.
So you can, for example, observe that there was a solar eclipse at a particular time and a particular place. But you cannot repeat that observation. (By way of contrast, you can drop and object from a particular height and observe that it takes a particular time to fall. You can repeat that observation.)
What you do with those observations has (almost) nothing to do with whether or not you can repeat them.
> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true.
I have a pedantic correction, but a relevant one. If a journal makes a rule that results must have p < .05, and then scientists go off and do a bunch of well-managed science they will get results with a range of p-values. .05 will be a ceiling, not a floor, on the possible p-values. More papers have a p-values right around .05 than they would if results were being produced fairly, so that's actually evidence of bad practices.
Is it? You're capped at 0.05. You can't print "obvious" results. ("With a confidence of p=0.00002173, we find that people are more likely to be angry in the slapped-in-the-face-by-the-researcher group than the control group...") So there's a natural floor determined by what is worth doing research on.
Now, it turns out that the floor is far lower than researchers believe and the field would be better off by slower incremental progress built atop "obvious" results, but still, I assert that publishability is an adequate explanation for high p values and obscures the evidence for bad practices.
Not that I have any doubt that bad practices abound; I have an undergraduate degree in psychology and am well aware of how things are done. It's not even malicious; people don't understand statistics, don't understand significance, don't understand models, and are mostly just doing whatever they can to make themselves feel good about their own intuitions and biases.
> The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true.
That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the case. There is often a _reason_ for conducting the test and a plausible cause of action. There are two possible reasons for a significant result:
* The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance)
* There is a real effect (and the null hypothesis is actually wrong)
This becomes more problematic in reproducibility tests, though, since that biases my prior towards a correct null hypothesis and now you must be very careful about pre-registration and the numbers of folks worldwide that are trying to reproduce a given experiment.
It's different because you're ignoring my second bullet point. There's a very common misconception that p-value is the percent chance the data confirm that the effect is real. That's not at all how p-values work.
They are showing the _likelihood_ of generating the data were there no "effect." You make a "null hypothesis" — that is, you assume there is no effect — and you construct a distribution of the results you might see in such a case. If your results are relatively unlikely, then you say the null hypothesis is rejected at a p-value level, leading support to the fact that there is such an effect.
But there's another bullet in my comment. You could _also_ see a low p-value if the null hypothesis is wrong! That would _also_ lead to a low p-value, but for a completely different reasons — reasons that are typically elucidated in the entire rest of the paper. Thus, you should be most suspicious of papers that do not have a reasonable cause of action, as that should increase your prior that the null hypothesis is actually correct.
If I were to edit that post, I'd add a third bullet — that they cheated or did bad science. My objection is to you stating that the statistical tests alone guarantee at least a 1 in 20 failure rate with a p-value of .05. There are indeed many other reasons to be suspect, but the statistical tests don't guarantee this.
I think you're just misinterpreting what I mean by "failure" or "wrong result". I mean a positive result that is due to chance rather than to the null hypothesis actually being false. On that view, statistics alone guarantee a "failure" rate of at least 1 in 20. That's what choosing a threshold of 0.05 means.
The on top of that baseline rate of false results you also have self-selection and occasional outright cheating that drives the failure rate higher than 1 in 20. 1 in 20 is what you get when you use a 0.05 threshold and everything else in the system works flawlessly.
Ah, yes, I read you as saying that "1 out of every 20 statistical tests in every journal is guaranteed to be wrong. It's just stats, people."
If what you mean to say is that "1 out of every 20 statistical tests of a false finding will demonstrate a 'significant' p-value and may subsequently get published," then we're in agreement.
I meant to say what I said. Getting a significant p-value when in fact there is no effect is a wrong result, hence, at a minimum, 1 in 20 results (on average) in a journal that uses a p-value threshold of 0.05 will be wrong.
Then that is incorrect. The situation is simultaneously better and worse than your interpretation:
> while a low P value indicates that your data are unlikely
> assuming a true null, it can’t evaluate which of two
> competing cases is more likely:
> * The null is true but your sample was unusual.
> * The null is false.
You're conflating the two populations, and indeed we don't know which is which. But the P-value is only talking about the likelihood of observing your data given a true null. It only has any effect on that first population. Without knowing the size of the second population, you cannot say anything about the fraction of erroneous results in a journal based upon the statistical test alone.
At the same time, though, converting p-values to a bayesian posterior or conditional probability that says something about the likelihood of the _hypothesis_ (like you're trying to do) is hard and generally way more pessimistic than 1 in 20 — it's more like 1 in 3.
> it can’t evaluate which of two competing cases is more likely
Of course it can. In fact, it can tell you exactly how much more likely one case is versus the other. P=0.05 means that there is a 5% chance that the null is true and hence a 95% chance the the null is false.
> says something about the likelihood of the _hypothesis_ (like you're trying to do)
No. I am simply saying that there is a lower bound on the rate of false positives caused by the choice of the threshold p value. If you chose P<=0.05 then you will have an error rate of at least 5%. The actual error rate is almost certainly higher (that's what "lower bound" means) for a host of other reasons.
Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result. If the effect is small and I didn't gather enough data, a p-value of 0.05 could result in only a handful of experiments accurately reflecting reality. Let's say 30 make the cut.
Were I to run another 100 independently designed experiments that all tested big real effects, then a p-value of 0.05 could result in all 100 accurately reflecting reality.
Were I to run 100 independently designed experiments that all tested false effects, then at a p-value level of 0.05, I should see about 5 erroneously significant results.
Now I submit all these significant results to a journal and they all get published. Of the 135 results, only 5 are coming to the wrong conclusion.
Look at that, an error rate lower than your lower bound. We simply don't know the sizes of these populations. It could be better or worse than my example, and it could be better or worse than 5%. There simply is no lower bound.
Of course you're right that there is a problem here but there isn't a lower bound guaranteed by the stats tests. Quite a bit of the problem stems from misinterpretations of p-values. For someone who likes to condescendingly explain concepts, you might want to make sure you have the harder concept down before assuming others don't understand "ratios" and "lower bounds." At this point you're arguing against the sources I cited. You should check them out. :)
> Were I to run 100 independently designed experiments that all tested real effects, my choice of p-value does not determine the number that erroneously find no result.
That's actually not true. The problem is that you cannot define what is a "real effect" without begging the question. Let me illustrate with an example: Suppose I do what appears to be a legitimate experiment to test a well-accepted law of nature. Unbeknownst to me, my instrument is faulty (or has been tampered with) and is in fact returning essentially random results. If I choose p<=0.05 then I will get a positive result 1 time in 20, i.e. I will "erroneously find no result" 19 times out of 20. Except that I'm not really erroneously finding no result. I am correctly finding the results of a different experiment than the one I think I'm running.
And, of course, an instrument that returns random numbers is an extreme example. See:
Now, you do have a valid point in that I made a tacit assumption when I claimed that statistics put a lower bound on the error rate. That assumption is that most experiments that are done test hypotheses that ultimately turn out to be false. I believe this assumption is actually true. This is one of the reasons making scientific discoveries is hard. But I didn't actually state this assumption, so I'll cop to that.
> At this point you're arguing against the sources I cited.
That's not true either. The first source says that the error rate is much higher than the p-value, which agrees with me. (I'm actually a little skeptical that the error rate is as high as your source says -- "At least 23% (and typically close to 50%" for p=0.05, but that's certainly not refuting my position!) I don't have time to go through your second citation right now.
> The problem is that you cannot define what is a "real effect" without begging the question
That's precisely my point. We don't know the sizes of these populations, so you really can't say anything quantitative about how the p-value affects the proportion of bad results in any given journal. My example was intentionally contrived.
You just have to be really careful when you talk about p-values because _so many people_ have this misunderstanding and it's actively harmful to getting the correct interpretation. That's why we keep going back and forth here — I'm not disagreeing that the situation is bad, I just want folks to recognize what the stats say and what they don't.
I think we can make some pretty reasonable guesses.
In any case, I certainly agree with you about the big picture: it's confusing, it's important to get it right, and a lot of people don't understand it. I may even be one of those people. But at the moment I believe that our disagreement is really over something else, namely, whether it's reasonable to have a prior on the null hypothesis.
> P=0.05 means that there is a 5% chance that the null is true
No. It simply does not mean that. It would be very convenient if it did, but it does not and can not.
p=0.05 means that if the null hypothesis were true and you ran your experiment, then you would only have a 5% chance of getting the results that you did.
The distinction is not at all obvious, but you'll need to understand it before you can make sense of statistics.
The p-value is not the number you want; what you want is the probability that the null is false, but you can't have that unless you know the actual probability distribution. Which generally isn't something you could possibly know or figure out.
(You can make educated guesses about that distribution, by making a guess and then updating it based on observed results. That's Bayesian statistics, which is great because it does give you a full probability distribution and all you have to feed into it is... uh... a probability distribution.)
Yes, you're right. I was making the tacit assumption that the null is true most of the time. On this assumption, the two statements are more or less equivalent. I believe this assumption is correct. If it weren't, making real scientific discoveries would be a lot easier. But it is an assumption.
You're falling prey to the very common "p-value fallacy." P-values talk about the likelihood of the data, not the likelihood of the hypothesis. The two are unfortunately not linked.
Yes, I am aware of this. P-values tell you the probability that what looks like a positive result is not in fact a positive result but merely a fluke, i.e. the p-value is (a lower bound on) the probability of a false positive. So?
"p-hacking" is also a thing, it's a reasonable possibility that they tried 20 false but plausible null hypothesis and published the one that got a significant result.
If there's no pre-registration, my prior of a paper actually being true would be much lower than statistics would say simply because the incentives are aligned so that there's a motivation to exaggerate significance.
Are you aware of the scale of failure here? The article (referring to the executive summary here) gives some really incredible figures. In psychology 100 reproducibility studies were only able to produce statistically significant results from 36% of prominent papers - compared to the 97% of the original studies. In biotech, a firm tried to reproduce 53 "landmark" studies in hematology and oncology - only 6 could be replicated.
The reason this is being called a 'crisis' is because it's looking like the vast majority of science, particularly in the social and human physiological sciences, is junk.
Absolutely! I'm only addressing the common misconception in the grandparent that the p-value is the percent likelihood that the effect is real. That's not at all accurate.
There are definitely other reasons that lead us to such horrifying results — especially in fields where experimenters _do_ conduct experiments seemingly at random and without a reasonable pathway of action (I'm looking at you, social psychology and friends).
It's worse than just 5% because p hacking. Every analysis has many parameters, and you can choose the most significant out of thousands of possibilities.
On top of this, the more interesting results are the most surprising. And therefore the least likely to be true a priori. So the chance of an interesting study being true is even lower.
> 3. The end-product of science is not truth, it is explanations of observations. Those observations can (indeed must) include non-reproducible ones. Sometimes the explanation of non-reproducible results is "experimental error" or "delusion" or "we just don't know." But non-reproducible events are nonetheless within the purview of science.
Yes and no. I think you have to have predictions in there somewhere. That's why your celestial mechanics is science: whatever the movement is, you were able to say what it was going to be beforehand, even if it only happened once.
As for explanation, that is not necessarily so simple, either. Sure, a lot of things seem to make more sense with a theory, but not every theory makes sense. A lot of stuff in fundamental physics is predicted by the equations, but in what sense are the equations explanations? "Shut up and calculate" comes to mind.
Shut up and calculate means that "sure, that's a nice toy model, but show that it reproduces the standard model", and also "sure, nobody really understands why the model is that way, but it works, so shut up and use it".
And also, shut up and come up with something falsifiable (in our lifetime).
That depends on what you mean by "truth". Newtonian mechanics says that gravity is pulling you down towards the surface of the earth. Is that "true"? It has tremendous explanatory power, but I would argue that it is not actually true. The truth is (as far as we can tell at the moment) that the surface of the earth is pushing you up and accelerating you through curved space-time.
Ultimately, it comes down to semantics. Does a vacuum suck or does external pressure push in on the vacuum? My highschool chem teacher said, "there is no suck," but, depending on your perspective, either explanation is valid. Reality is subjective and language is imperfect.
> 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment.
It's worth noting that this is part of the goal of reproducing results. If two identical experiments produce the same result, you're only providing proof for the very narrow theory the experiment tests. Slightly different experiments allow you to demonstrate that the theory is applicable. Imagine if evolution only occurred in the Galapagos, or if gravity only worked in an apple orchard in Cambridge. The only reason we know that evolution and gravity aren't local oddities is that they have both been tested in a wide variety of locations, with a wide variety of experiments.
> 3. The end-product of science is not truth, it is explanations of observations.
The term 'science' is used in this way. Perhaps it should not be. Perhaps semantics drive the issue.
In the field in which I am familiar, we have 'expert witnesses' that will use reproducible scientific principles to analyze, for instance, blood scatter to determine the angle in which a shotgun was held when it discharged and blew off a particular person's head. The principles -- based on physics, chemistry, etc -- are sound and scientific.
The problem is, we will invariably have TWO very scientific expert witnesses who BOTH use reproducible and accepted methods. Yet, using their scientific tools, they will come to two utterly different conclusions.
It's left to a jury to decide which interpretation is correct.
While the tools used by the experts are scientific, their opinions are not. While their scientific tools are reproducible (for instance, the speed at which coagulating blood and brain tissue will slide down a stucco wall given all the variables such as temperature, humidity, friction coefficients, etc.) their ultimate reconstruction of what happened is not reproducible, and it is left to lay people to draw their conclusions.
There should be more of a distinction between Science and History. That is, science -- strictly defined -- is all about reproducibility. History -- by definition -- is not. Using scientific tools to reconstruct a past event does not result in a 'scientific' theory. Rather, it results in a historical theory arrived at with scientific tools.
Whether it is blood scatter from a murder (or suicide, depending on whom you ask), the formation of the Grand Canyon, or the development of finch beaks on Galapagos, the ultimate theories are inherently non-scientific as they can not be tested. The methods and tools used to derive the theories can be... but not the conclusion itself.
In short, perhaps the term 'science' is being used for things outside its domain. (Or, alternatively, if we wish to include such things inside the domain, we should broaden the strict definition of science. Like I said, there are some issues of semantics that may be driving some of the issues in the article... other issues, of course, pertain to sloppiness, errors, and the like.)
Power is also a concern that many researchers do not pay attention to (at least in psychology/neuroscience).
If the original study was under-powered, the estimated effect size in that study will be inflated and any replication attempt that uses this inflated effect size estimate will be severely underpowered.
Plus, two independently conducted studies that are both powered at 80% to detect a true effect will both be positive results only 64% of the time (assuming absolutely nothing fishy going on, e.g. p-hacking).
> 1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment.
This is why you include the error in your results. We don't care if your two experiments result in 100% the same results. We care about if your results line up within the error of your experiment.
> 2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science.
This is why we share data. That is in essence our observation. A celestial event might be caught by only one instrument, but several people can develop several different models. We wait and watch for similar events though, to check if models are consistent (within error).
> 3. The end-product of science is not truth, it is explanations of observations.
I just wanted to repeat this because it can never be stated enough.
> 4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true.
While I don't like p values, because of hacking, (especially 0.05) that's not how stats work. Flipping 10 heads doesn't guarantee 5 tails, not even 1. I wouldn't use as strong as a word as guarantee.
But this is also an argument FOR reproducing. If multiple experiments are consistent with one another (within error) than that strengthens the argument.
TLDR: More brains that look at a problem helps solve the problem.
I will add my own statements about the reproducability problem in science. One is because there is less funding for it. Reproducing an experiment isn't sexy. Another problem just stems from that data isn't always open. It is hard to review work if you don't know everything about the experiment. Data can even have simple mistakes that just weren't caught. But it is also embarrassing to share data.
I never said it did. But in the long run, flipping a fair coin will generate pretty close to 50% heads and 50% tails. That's what it means to be a fair coin. Likewise, in the long run, using a p threshold of 0.05 (which many journals do) will generate 5% false positive results (that's that the 0.05 means), i.e. on average 1 false positive result for every 20 experiments you do.
Your experiment shouldn't need to presume the coin is fair.
Flipping an unfair coin twice in a row and discarding the HH and TT doubles will generate close to 50% HT pairs and 50% TH pairs. This only presumes that subsequent flips are independent events.
It's still not going to be exactly 50%, because random events are random. You can get as small an error margin as you need by increasing the number of trials. And in some cases, this property of statistical analysis makes it easier to increase N than reduce the randomness and uncertainty in the experiment.
Because there is a very long causal chain between raw physical phenomena and your perceptions. Your equipment could be faulty. You could be suffering from hallucinations. You could be a brain in a vat.
Example: I want to test if evolution is true. I hypothesize that if evolution is not true, then God will give me some sort of sign, e.g. I pray to God to make a coin that I flip come up heads if evolution is false. I flip the coin and it comes up tails. I conclude that evolution is true. That conclusion would be wrong even if evolution is true.
You are moving into metaphysics and including stuff that is not covered by Null Hypothesis Significance Testing. But on the original point "Likewise, in the long run, using a p threshold of 0.05 (which many journals do) will generate 5% false positive results" is just plain wrong. P(A|B) != P(A)
It is if P(A) is 1, which is what you stipulated when you asked "How can u get a wrong positive result if you only test correct H1s?"
> "... using a p threshold of 0.05 ... will generate 5% false positive results" is just plain wrong
No, it's not wrong, it just makes a tacit assumption that most hypotheses that get experimentally tested are incorrect. Which is a reasonable assumption. If it were not true, making scientific progress would be a lot easier.
> On the meta-level, you’re studying some phenomenon and you get some positive findings. That doesn’t tell you much until you take some other researchers who are studying a phenomenon you know doesn’t exist – but which they themselves believe in – and see how many of them get positive findings. That number tells you how many studies will discover positive results whether the phenomenon is real or not. Unless studies of the real phenomenon do significantly better than studies of the placebo phenomenon, you haven’t found anything.
This is such an astute observation that reproduction studies can be used to find a placebo control group for bad scuence. Parapsychology was one such group, but now we can find many others. Brilliant.
What I'd like to know is if the "irreproducibility crisis" is really some combination of "sample size was too small" and "effect size was too small". When I went through a lot of these studies myself and saw the ones that don't reproduce, I saw this theme over and over. "P-hacking" is less of a concern to me when the effect is real and widespread.
It's so bad now that for any article/study, I look at the sample size first. If it's too small (especially < 100) or they don't say I just ignore it. And if you don't publish or give some estimate of your effect size I just think about it directionally but don't give it much weight mentally.
I'm not sure this captures the whole problem. Sure, huge sample sizes are great, but look at the current top comment by lisper - we have gained a boatload (that's a scientific term) of insight from single events. LIGO is currently rocking a sample size of 7 but I doubt you ignored it.
Of course there are exceptions and what I said doesn't capture all cases (like with massive fixed costs or unique kinds of studies), you're right. I find it's a pretty good rule of thumb, though.
Huge sample studies with high stat significance can have very low effect sizes, though. If an effect is shown on a small sample, it means it is a really big effect, and will likely be reproduced in a bigger study.
Lets take a page from Marx. Science is many things, in particular a relationship between capital and labor. The scientific method is a wonderful idea, but it is subordinate to the economic forces that underlie scientific activity. Look at the conflicts and contradictions between those doing science (labor) and those deciding the science to be done (capital), and that is the ultimate source of these crises.
The executive summary lists 40 points on how to improve the reproducibility of science. A bit over half of them are addressed to the sources of capital such as private organizations, universities, and governments. I think many of those points are good. However, I don't think the other points, the ones that recommend doing science in different ways, have a good punch. Even if you fix the problems there are today, so long as science is a rat race of trying to get grant money to stay afloat while burning out grad student after grad student, I think other pathological practices will creep in as a completely rational response on the parts of scientists to a hostile ecosystem. There's just a very big gap between how science should be done and what capital owners want from science.
I think this is pretty self evident, but the issue is that let's say you have some system where economic forces are removed. Essentially a researcher basic income in one scenario. This would suddenly massively incentivize people towards this direction since it's basically a career path that guarantees a stable livelihood, which is something that's extremely rare today. Well you need to ensure there's nobody just completely gaming the system and so inextricably you'll end up with some sort of qualifier for results. And now suddenly you've done nothing but kick the can since this new qualifier is what's going to be gamed.
Maybe the biggest problem is what you mention, but in another direction. For whatever reason there seems to be extremely little interest in the private direct funding of science. In times past the aristocracy would often fund scientific research on all sorts of topics. Today the practice seems to have all but disappeared, certainly if we measure relative ratios of the practice.
Small correction. A basic income is guaranteed no matter what; there’s no means testing or anything like that. You can’t game the system because there’s nothing to game: you get the money regardless. You seem to be describing the system we have now, where you’re paid for results.
I think what the parent is saying is that with a basic income only for researchers that creates a scenario where ‘Well you need to ensure there's nobody just completely gaming the system and so inextricably you'll end up with some sort of qualifier for results.’ That you don’t have ‘fake researchers’ gaming the ‘basic income only for researchers’ reward.
A universal basic income is guaranteed no matter what; but it's certainly possible to have basic income schemes that are not universal but limited to some (small) subpopulation, and the criteria for being in that subpopulation can be gamed.
I don't see why you would want to solve this with a researcher basic income (and overhead/bureaucracy of figuring out who is a researcher) and not the simpler solution of an actual universal basic income.
> and not the simpler solution of an actual universal basic income
Eliminating fossil fuel use is also simple. But there's another dimension to this problem, called "political viability", which I feel you're overlooking.
Fair point and even I managed to live off of and conduct research on 25k/yr in grad school. Though much of that cost was also offset by access to facilities, labs, equipment, and supplies. Would love to see more public access to these sorts of resources.
While we're talking political economy, I was thinking that academia tends towards a degenerate form resembling subsistence agriculture: defending a small patch of turf and making just enough meaning to support a career. Even if the raw economic incentive is removed (e.g "publish or perish") there is sufficient ego investment to preserve this pattern.
> There's just a very big gap between how science should be done and what capital owners want from science
It was "a team from Bayer Healthcare" who "tried to replicate the results of basic cancer studies," failed, "and kicked off a media storm questioning the legitimacy of cancer science—and science in general" [1]. The "capital owners" looking out for their own buck are performing more effectively, narrowly speaking, than academia.
Even if that's true, that only science that can have a profit motive slapped on it can be done effectively in the current system is a problem. If only such science were done, then we would have different crises of science to deal with because profit seeking is at odds with much science, in particular basic research.
It's also quite obviously the case that profit seeking research has incentives to be bad science in other ways.
> It's also quite obviously the case that profit seeking research has incentives to be bad science in other ways
I agree. My point is the Marxist framework is a bad one for modern science.
We have endowed academics and ones working for private institutions and getting grants from public bodies run by scientists. Privately-financed biotech companies founded by academics taking moon shots and getting acquired by marketing and distribution powerhouses. Even delineating who is "capital" and who is "labour" quickly becomes pedantic more than useful.
> My broader point is the Marxian framework is a bad one for modern science.
I will concede that it's reasonable to think it's an incomplete and limited framework, but I have to take issue with calling it bad. For example, I think this starting point is more productive than the premises of this report.
> Privately-financed biotech companies founded by academics taking moon shots and getting acquired by marketing and distribution powerhouses. Even delineating who is "capital" and who is "labour" quickly becomes pedantic more than useful.
In this example it's plain to me that VCs are capital and entrepreneurs are labor. That's usually the game in such moonshots. Here we can analyze what sort of science is being done through this relationship through a Marxian lens too. VC's want to make big bets with potentially astronomical upside. That means only certain kinds of science will ever be done in this way. Yea, this kind of science is probably more replicable on average, but what about something like Theranos? This sort of science funding has its own problems with inefficient and incomplete allocation of resources.
I'm not a hardcore Marxist; I don't think you can reduce the world to one grand theory based on hot takes inspired by Hegel, but I think it's a more powerful tool than most care to admit.
I do not see much need for delineation. In today's world, every higher-level employee can be analyzed just fine from both sides:
- she is exploited, i.e., forced to act in a profit-maximizing way (in particular, short-term profit). Each other behavior will be penalized (no career, no tenure, ...).
- she is exploiting, i.e., forces her workforce to act in way that they maximize her profit.
Your original thesis just got completely destroyed and now you are bring up unrelated speculative issues not explored in the article.
Just admit that the marxist analysis doesn't hold much water in this case. That doesn't mean it is never useful, just that you might need to think a bit before applying it.
It certainly seems true that the owners of capital are suffering from the reproducibility crisis as much as anyone is. What 'danharaj is arguing, I think, is that the owners of capital are not well-placed to solve it.
The very nature of capitalism is that people who produce monetizable results should be rewarded for that work, and that people who don't shouldn't. This is fundamentally incompatible with doing science in a reproducible way: paying for positive results and not negative ones gets you p-hacking as an emergent effect. Of course the capitalists don't want to pay for non-reproducible positive results, hence the firestorm. But they more strongly don't want to pay for negative results.
The capital owners in the science industries will probably end up doing something sort of like what the capital owners in the software engineering industries have done: begrudgingly accepted that a non-capitalist approach (open source development across multiple companies without proprietary advantage) produces better results for everyone, and figured out a way to kind of support it. Much like the capital owners in the software engineering industries, they still won't be very effective at it. Their nature will be to still try to extract proprietary advantage when possible, to cut costs per the tragedy of the comments, etc., and so this model will only work well when funded by companies rich enough that they can allow themselves not to show an immediate return on their sponsorship.
Also capital owners would preferably not pay at all and have you work for free. They cannot approach this ideal only because the workers are not constrained spatially (like serfs) not forced (like slaves) to work for them. Sometimes they ate forced due to economic factors to do labour at marginal gain or even loss...
Issues with reproducibility have been discovered independently in a number of different scientific disciplines going back to the 1990s at least. This was then covered in a New Yorker article by Jonah Lehrer, published in 2010 [1]. It included the phrase "decline effect", coined by a researcher studying ESP in the 1930s.
A great deal of the research described in that article is academic in nature. The article also alludes to the contributory influence of funding as one of the causes of this phenomenon:
"The bias was first identified by the statistician Theodore Sterling, in 1959, after he noticed that ninety-seven per cent of all published psychological studies with statistically significant data found the effect they were looking for." ... "In recent years, publication bias has mostly been seen as a problem for clinical trials, since pharmaceutical companies are less interested in publishing results that aren’t favorable. But it’s becoming increasingly clear that publication bias also produces major distortions in fields without large corporate incentives, such as psychology and ecology."
Academia too is subject to the laws of capitalism. It is not some siloed-off eden of labour working for itself. Anyone working in academia can attest to the same pressures you see in the free market, whether it be short termism or maddening bureaucracy.
You might enjoy the book, The Fellowship: Gilbert, Bacon, Harvey, Wren, Newton, and the Story of a Scientific Revolution.
The book is set around 1660, as the center of the scientific universe shifted from Italy to England. Interestingly, the formation of the Royal Society occurred in the midst of the English Civil war. Not only economic forces, but military forces and political forces had a dramatic impact on shaping what became the formalization of of the scientific method, and peer review process.
Consider that the very existence of Oxford (and other universities) hung on the arbitrary conceptions of military generals and the political implications of their decisions.
The reproducibility crisis is most severe in the social sciences. Hard sciences like physics are on much firmer ground, and conflating the two is clownish. Social science research is funded almost entirely by the government[1].
It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination. In fact, much of it is due to cultural marxist insistence on outcomes conforming to political correctness, as well as uncritical acceptance of politically conforming work. See the ongoing fight against the idea of a high genetic basis for adult IQ.
All of this is obvious enough: capitalists aren't interested in non-reproducable results unless it is obviating them from blame for a given externality. But that sort of research is a small fraction of overall research (and should absolutely be done by impartial third parties.)
Maybe none of this ideological bullshit is the real issue? Maybe the major problem is that hard vs soft sciences are subject to different experimental controls and confounding factors, not to mention standards of rigor. If you claim to discover a new particle your n will be in the billions, and a whole community will analyze your results with a fine comb. Meanwhile if you make a claim about implicit bias or the nutritional value of coffee it’s based on tiny samples with huge variability.
It’d be cool if your and the other poster’s ideological spat could try to not use science as the battleground, or at least eliminate more likely issues before going there. Social sciences are hardly even science from the perspective of a chemist or physicist, there’s no need to delve into petty ideology. Methodology is a much richer source of hurdles to overcome than old and sad debates.
> It’d be cool if your and the other poster’s ideological spat could try to not use science as the battleground, or at least eliminate more likely issues before going there.
I don't disagree. A big part of the problem is the social sciences claiming the mantel of science (often just statistics layered over tiny samples, as you point out), in order to imply some sort of impartiality and precision on par with the hard (i.e. real) sciences.
Unfortunately, and I say this sincerely, I believe the ideological nature of the social sciences since the 60s has been responsible for a huge amount of uncritical acceptance of bad (false) work as well as preventing a lot of unpopular good (true) work. I don't know how to deal with that fact other than stating so forthrightly, which, unfortunately from the perspective of science, carries necessary ideological freight with it.
>Social sciences are hardly even science from the perspective of a chemist or physicist
A chemist or a physicist still have hard time dealing with particle assemblies of say 100-100000 particles. For several last centuries they learnt to deal with [aggregate properties of] assemblies on the scale of 10e20+ particles. Only during last century they came to pretty good understanding of 1 particle.
In social sciences this is Facebook and Cambridge Analytica (ie. they are real social science today, not some professor in an office). This is the scale where aggregate properties appear, and they start to be able to drill down from it.
> It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination.
You understand literally nothing about Marxism. For example, "capitalist" isn't a moral designation, it is a description of an economic agent's role in an economic relationship. Capital owners have access to resources and they give permission to laborers to use those resources. That's it.
> cultural marxist
You can't be serious.
> See the ongoing fight against the idea of a high genetic basis for adult IQ.
This plays out more in the public sphere than in scientific communities. It's also the case that a few popular intellectuals try to sell psychometric science as saying more than it actually does about how social policies should be designed. This, again, happens moreso outside of scientific circles.
Pharmaceutical companies influence research according to [1] to gain support for their medicines.
Studying economic factors underlying social activity isnt by itself Marxist. This methodology is adopted in support of different political positions. In fact, many free-market economists also do this and get accused of 'economics imperialism' by social science faculty who want to study other factors.
Another point made in [1] is the blame isnt just on companies but people being influenced.
Please don't take HN threads on generic ideological tangents, which never lead anywhere new. I know the GP pointed there, but that's no reason to follow the pointer.
Fair enough but I was reacting to the clearly idealogical. To suggest Marx has something to add in this issue is both provocative and ridiculous. To entertain drivel this degrades HN.
As I tried to explain, it's a matter of degree. If a comment contains both bait and substance, it falls to the rest of us not to take the bait, since doing that guarantees a useless tangent or worse. The acceptable options are either (1) react to the substance, or (2) don't react.
I have met several "refugees" from particle physics, that left in part because of flimsy statistical methods. So let me ask: who is going to reproduce the experiment that found the Higgs boson?
There has to be similar prestige/career-building/notoriety/funding for spending time on reproducing the experiments of others. Without that shift there will clearly be a greater tendency to just try something new.
Also, when experiments depend on source code, etc. we need real engineering tools/principles applied. (Something like: “you can’t publish paper X if you aren’t including a public repository with build/run instructions”.) Unfortunately, there are all kinds of reasons why scripts/builds could fail just a few months or years later so they would have to be checked too.
I think it would be cool if document-generation caught on in the publication of papers, i.e. the paper itself is generated by running actual experiment scripts and producing charts, etc. from plain text source.
To be clear, the "NAS" (nas.org) that published this study is the National Association of Scholars [0], a political group, not the National Academy of Sciences (nasonline.org) [1], a nongovernmental organization that consists of scientists elected by their peers to provide independent scientific advice to the US government. There was in fact a study published recently in PNAS, the Proceedings of the National Academy of Sciences, on this topic [2].
Good catch thanks! Wikipedia says "The National Association of Scholars (NAS) is an American non-profit politically conservative advocacy group, with a particular interest in education"
Thanks for the link to the PNAS -- looks like a whole special issue, not just an article, in fact. Great. I'm gonna ignore the fake-NAS one, and read the real-PNAS one.
I wonder what National Association of Scholars' motivation is here exactly.
The purpose of reproducibility is to be able to properly analyze the quality and accuracy of a study. The purpose is to support critical thinking and objective facts.
However, this paper isn't a study. It's just an essay. And while, yes, the paper in question does highlight numerous issues in reasonable and rational ways, it also uses this as the breakout quote on page 36:
> Scientists readily accept results that confirm liberal political arguments, and frequently reject contrary results out of hand.
Now, there's a couple problems with this statement. The most obvious is that it implies blame on liberal politics. On the same page they detail that the percentage of self-identified liberal psychologists at a conference (~800 of 1,000) wildly outnumbers the number of self-identified conservative psychologists at the same conference (3 of 1,000). Well, this conference is the Society for Personality and Social Psychology, which is based in southern California. Would that bias attendance? However, even if we assume that distribution is correct, the quoted statement doesn't say anything about the conservative scientists. Would it also be correct that conservative scientists readily accept results that confirm their political arguments? We don't know. Maybe it would be more correct to say this:
> Scientists readily accept results that confirm pre-existing personal political arguments, and frequently reject contrary results out of hand.
Instead, this paper chooses to imply that scientists who are more liberal are somehow inherently more biased. If we assert that good scientists are those who don't allow political arguments to influence their acceptance of their results, then the natural conclusion to draw from that is that being liberal is incompatible with being a good scientist.
Don't get me wrong, this paper makes a lot of legitimately good points with good examples, but the whole section on groupthink comes across a little bit biased itself. However, it's existence means I can't shake the feeling that maybe I'm missing something.
So having show that there's a reproducibility problem, the paper concludes. Then very end of the paper, the afterword lists these points introduced seemingly out of the blue:
> Symptoms of Pathological Science:
> 1. The maximum effect that is observed is produced by a causative agent of barely detectable intensity, and the magnitude of the effect is substantially independent of the intensity of the cause.
> 2. The effect is of a magnitude that remains close to the limit of detectability; or, many measurements are necessary because of the very low statistical significance of the results.
> 3. Claims of great accuracy.
> 4. Fantastic theories contrary to experience.
> 5. Criticisms are met by ad hoc excuses thought up on the spur of the moment.
> 6. Ratio of supporters to critics rises up to somewhere near 50% and then falls gradually to oblivion.
These are from Irving Langmuir (https://en.wikipedia.org/wiki/Irving_Langmuir) and his criticisms on research conducted with unconscious biases or subjective elements. He meant it as a criticism of research like cold fusion.
Now, there's not any real one-for-one discussion of these items or how they relate to what came before. They're just presented basically as-is as the final portion of the essay with few notes that they correlate to what came before. You'll notice, however, they they all call into question the credibility of the scientist as well as the credibility of the scientific community, and not the quality of their research or data.
However, let's say you're politically conservative, and you're reading this essay. Maybe you don't believe in climate change. You get to the end of this paper, and it tells you that in spite of the claims of accuracy of the data, in spit...
To eliminate any questions about social context or bias or agenda? Totally not possible, social context is an inherent part of science, as in all human endeavors.
I think the purpose of reproducibility is to well, make science _possible_, it is the fundamental bedrock of science -- to determine what and how we can make reliable predictions about the world. It doesn't mean bias and effects of social context no longer exists though. Science exists anyway, in a social context -- as evidenced by the past couple hundred years!
In general, I don't think science exists to eliminate _any_ questions, science exists to ask questions. Eliminating questions may be a goal of ideology or religion, but not science. (and in the real world, they certainly influence each other. science is a goal to strive for, not a fait accompli. Scientific understandings of the world are always changing; science is an approach to understanding the world, not a set of settled conclusions).
Also, if we're looking for the social context that leads to the reproducibility crisis, I think it's a lot more likely to be about academic career incentives (you _gotta_ publish things, a lot of things, that all seem like important and socially useful results, none of which are negative results -- when the actual practice of science means negative results are an expected and useful thing, not indication you are a bad scientist), than the political ideologies/biases of researchers.
Maybe I'm wrong. There are probably ways to investigate it. This essay isn't one of them.
A good solution would be to simply reduce the p-value threshold.
The current "standard" p-value in many fields, of 5%, is arguably too high. consider throwing a double six in backgammon has probability of 3%. That means that, on some level, throwing 6-6 would be a valid scientific "proof" of ESP. (p-value<0.05)
This of course is even before p-hacking.
A high p-value is basically externalizing part of the research cost onto other scientists and in the process creates a lot of false-positive "noise".
264 comments
[ 2.7 ms ] story [ 190 ms ] threadhttps://www.npr.org/sections/money/2018/03/07/591213302/epis...
What does that even mean?
You have one point "it's a conservative think tank" - which does not mean anything - coupled with another empty statement.
Now I sure don't mind criticism of that entity, but I find it ironic that it is done with such an empty statement that's just a not very cleverly disguised type of ad hominem attack.
May I suggest you take some time actually reading the paper and then coming back with criticism of their actual points?
While the report itself may not push an obvious partisan agenda, a critical reader should consider the conservative agenda of the NAS in relation to why they might publish such a report. In the conservative wing's long-standing opposition to science, it's not far-fetched to view this report as being published tactically -- even if all of its points are well-founded -- as something to point to when future efforts attempt to undermine science-based policy on issues like climate change, stem-cell research, abortion, etc.
Especially when the President of the NAS is a public "climate skeptic".
https://www.chronicle.com/blogs/innovations/bottling-up-glob...
(I blame all the overtly political articles that I believe have changed the tenor of the site, and politics has infected every article now. Thanks @dang!)
https://news.ycombinator.com/newsguidelines.html
Also from the intro: The National Association of Scholars (NAS) has long been interested in the politicization of science.
That IMO is a far more dangerous stance. Any study can have hidden flaws, none should be trusted without some form of replication.
How are you going to reproduce a study about a cancer drug? Synthesizing the drug itself is a huge endeavor, never mind finding a group of participants with the exact cancer you need. How would you reproduce the Large Hadron Collider experiments? You'd need to rebuild one of the most technically challenging, most expensive scientific instruments ever made.
Our problem isn't lack of resources. There's 7.5 billion people in the world, I can't believe we've ran out of control groups and experiments to do. Maybe it's a little more difficult with medical edge cases, but there's plenty of other things to research. It's just hard to do it politically, especially with people that are "anti-science", meaning against funding research that might show outcomes that aren't good for them economically, or against their religion.
Berkeley University does a good job of providing visualizations on this topic:
http://undsci.berkeley.edu/article/0_0_0/howscienceworks_02
Back in early science days, this used to be a valuable section because the pool of "scientists" was huge and they often crossed disciplines. Now you have all sorts of specialties such that the pool of colleagues who are able to understand and assess your work might be, like, 8 people.
There was a Japanese mathematician who recently put out a huge proof that claimed to solve a major math problem (I am forgetting all of the important details here, so forgive me). The news wasn't so much that he created the proof, but that they couldn't find anyone to verify it!
You can't trust blindly the previous studies because they may be wrong, or have hidden assumptions. A well known case is the use of a control group. If you want to study a new drug, you must compare the results of the patients with a similar group that is taking the previous drug and/or a group that is taking no drug. You can't just trust that the cure rate of the previous study is 23.573% and compare it with your results. (And there are some trick to get more reliable results, like using a placebo instead of no drug, and doing a double blind study.)
> Improper use of statistics, arbitrary research techniques, lack of accountability, political groupthink, and a scientific culture biased toward producing positive results
And really science need not move quickly as long as it continues to converge on accurate answers that's plenty useful.
EX: Nutrition research often get's dumped on, but it's far from useless. There are 41 things we know you need or issues will show up: (http://www.nutrientsreview.com/glossary/essential-nutrients + energy + water) Getting the specifics right is complex and we are working on it. Thiamine deficiency from white rice is the kind of thing you need to get right or die, 'optimal health' is a harder target.
Three is NO way to do that. There are ways to misuse scientific results. Premature generalisation, assumptions of safety. Cherry picking to support preconceptions. Attributing strong support to tentative and/or unchallenged results. And more...
Of course science is imperfect, but it tends towards truth over time, whereas dogma and populism have no way of correcting mistakes that isn't just as likely to introduce new ones.
The scientific method is important to be used, but don't trust something just because another person believes it to be science. You need the hypothesis and the verification method. Others (all of us) have the obligation to challenge both the hypothesis as well as the verification methods.
I think it's really dangerous to tie the wrongness of the supposedly scientific facts they spouted to the wrongness of their deeds. Dangerous because if it turns out they were right about some point, then you look like an idiot and have to change your tune. And dangerous because it warps subsequent science, to avoid such embarrassment.
They believed in breeding a master race. In response to this, the acceptable opinion in the West ever since seems to have been that this is impossible, that humans are unique among creatures on this planet in that they are immune to the pressures of selective breeding. We can double the size of our chickens (and breed tame foxes, and smart dogs) but humans are apparently immutable. This is creationist nonsense, and it becomes more and more obviously nonsense every year. We need not have painted ourselves into such a stupid corner.
Frankly, genocide, or at least dehumanization and extreme indifference, seems like the logical endpoint of sincere belief in the concept of a "master race" which is inherently good and is hamstrung by perfidious inferior races. Once you've determined that a member of one race is worth less than another, it's natural to judge that the death of a member of this inferior race would be justifiable to save the race of a member of the superior race, and in this way the concept of genocide becomes, in your mind, a "defensive" action.
If I'm not mistaken, the Brahmins tick most of your boxes. Are they alright? Some would argue that they go in for a bit of dehumanization... but nobody is proposing a world war to stop them.
Actually I'm alright with people believing lots of crazy shit, in private. Because I think intellectual monoculture is pretty dangerous.
I do find it a little strange how many people want to turn the concept of "diversity" on its head and tell me the real diversity problem is that we don't let enough people with racist views into power.
My objection is to conflating them with scientific matters, i.e. empirical claims, especially about human biology. I worry that doing so actually weakens our arguments against such evil. We shouldn't build into our case against them scientific claims on which we may ourselves be proven wrong. It's not necessary.
(And, much as I enjoy using words like diversity in their older meanings, in the hope of tripping people into thinking, here I resisted the temptation. Intellectual monoculture was however a feature of most of the evil regimes of recent times, too, and one of their weaknesses.)
By way of analogy, when someone tells you he believes global warming is impossible because it would go against the Bible, you don't have to wonder that much what he would do about carbon emissions.
EDIT: Krugman had this to say about the R&R austerity paper "What the Reinhart-Rogoff affair shows is the extent to which austerity has been sold on false pretenses. For three years, the turn to austerity has been presented not as a choice but as a necessity. Economic research, austerity advocates insisted, showed that terrible things happen once debt exceeds 90 percent of G.D.P. But "economic research" showed no such thing; a couple of economists made that assertion, while many others disagreed. Policy makers abandoned the unemployed and turned to austerity because they wanted to, not because they had to."
https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt
Particularly telling: "The paper was published in an annual 'Papers and Proceedings' edition of The American Economic Review that was not subject to the same peer-review standards that other editions use before publication"
What was said above was "one of the most horrific and important examples of widely accepted science". Important words here are "widely accepted science". So science didn't go wrong, the LABEL of science was attached to something that didn't meet the scientific method. That something was then sold to people in power, and the science LABEL was used to convince people of the correctness of that something.
People are really quick to say this, but how many people have actually studied the 'racial science' literature of the 20s and 30s and figured out to what extent it meets the methodological standards of today? Yes, we know that they drew incorrect (and reprehensible) conclusions. It doesn't necessarily follow that their methods were any worse (epistemically speaking) than methods that currently pass muster.
Just the premise of the "racial science" studies would render them invalid, how could they even have controlled for the myriad variables that we have no idea how to control today?
I don't need a specific study to know that. Any sociology/economic/psychology research should be taken with a grain of salt.
So how is that not an example of "science going wrong"? Unless you are proposing that science is right by definition, or that governments should ignore most of the branches of science that are actually relevant to government.
There are plenty of branches of physics that are at least as speculative as sociology, economics and psychology (e.g. cosmology). The only difference is that these branches of physics have few political implications.
Governments (and people) should evaluate the merits of the information they receive. Properly conducted and repeated experiments with noted mechanisms of actions (e.g. vaccines) should have more weight than studies about trickle down wealth or whatever. Obviously, decisions have to be made, and some will be wrong unfortunately.
This is a good example of why it's not so simple as you're making out. Earlier, you put biology on your list of unreliable branches of science, and you claimed that you could know that the racial science of the 30s wasn't "real science" simply because it consisted of "studies about people". But vaccine science falls into both those categories!
In reality you're deciding whether or not any given branch of science is methodologically sound not by examining any actual studies in the relevant disciplines, but simply on the assumption that any study which has arrived at the "wrong" conclusion must have had a bad methodology.
Just stating that all of soft science is dodgy doesn't really distinguish us from them, does it? We do a lot of soft science!
https://en.wikipedia.org/wiki/Replication_crisis
The point is that some research is weaker than others, and the reasons why haven't changed since WW2, so if it's a problem now, then it was a problem then.
Soft science has many questions to which we'd love to have solid answers. So (perhaps unfortunately) it's not going away, and I think we need to get better at sorting out what it tells us.
More importantly, the example of the Nazis is useful in an ethical argument, because they're an almost universally acknowledged example of evil. Much like we aim to test our programs with extreme inputs, we can use the "extreme input" of Nazism to test our ethical arguments. If I posit some universal ethical standard, and then find that it doesn't seem to work if we take the Nazis, we've proven by counterexample that my standard is not universal after all.
More seriously, it wasn't just this group that believed in this stuff, it was the many many institutions in the western world, as emodendroket points out. Certain departments at Yale, most notably, supported eugenics.
https://embryo.asu.edu/pages/american-eugenics-society-1926-...
edited portion in case people think I meant all of Yale, bad generalization, my bad.
Additionally, using science and other expert judgments can easily tend toward simply choosing experts and studies that align with whatever policy you want to push for unrelated reasons, or outright seeking to manipulate research for political ends.
Of course it can, but that's equally true for any other way of making decisions. So it's not a downside of science in particular.
Talk about science creating a mind boggling ongoing health crisis by telling people to limit their fat intake and replace it with cereals and bread.
Current scientific consensus is that it is caloric surplus - confounded by a bunch of poorly researched factors. For weight gain source does not seem to matter too much. This does not say anything about dietary specifics or how they affect intake and metabolism.
Technically nutrition science does not provide strong support for such statements as this carb vs fat one based on interventional studies.
Epidemiology alone does not tell you "therefore this change will be beneficial"...
I've looked into that and all I can say is the history is disputed - some people say there really was a nutritionist consensus, others that this was a popular misunderstanding based on overzealous popularisation of a small number of papers (similar to the myth that the consensus among climate scientists in the '70s predicted an ice age) and that working scientists never thought this.
Either way it's a failing of the process, but the appropriate response is quite different; in the first case all we can say is mistakes do happen, in the second we need to get better at ensuring that the real scientific consensus gets translated into appropriate policy actions.
Why? Do we believe that individuals are likely to be better informed than governments? (why?)
Uncertainty is a fact of life. Any decision-making process will lead to some mistakes. That's not a reason to give up on trying to make things better. If the government believes, based on the best evidence available at the time, that a particular fat tax policy (say) will, overall, be good for society, surely it's better to implement that policy than not.
I certainly believe that some individuals will be better informed and better able to translate that information into the best course of action for themselves given their goals than governments, especially "governments" as represented by a patchwork of laws and regulations the impact a complex thing like individual health. This is true even at the time that new policy is enacted and tends to become more true over time due to the inertia of government policy that goes down the wrong path.
Some individuals, sure. Most individuals, probably not.
I agree that government policy has inertia, and that overconfidence on uncertain issues can lead to imposing well-intentioned policy that ends up being harmful. But it's equally possible to make the opposite mistake, and hold off introducing policy changes for too long because of perceived uncertainty. E.g. car seatbelt laws and tobacco advertising laws should have been introduced earlier than they were. I don't think there's any general principle we can draw here; some government policymakers could do with more humility, others could do with more confidence, but all any one of them can do "from the inside" is try to figure out how strong the evidence actually is on a given issue, and use their own best judgement in determining whether there's a clear enough scientific/medical consensus to warrant policy changes, or not.
I thought the parent comment made it clear: the alternative we are supposed to use is "democracy". That is, the people who are ruled should determine the rules that are imposed on them.
Now, democracy does not preclude the use of science to make decisions. The voters may rely on science to inform their decisions (when the science is available). It does however preclude a rule by scientific experts/bureaucrats.
But I also think it's foolish to frame the decision as solely one of "science" vs. "gut instinct". The are many phenomena, especially when it comes to human society, that are too causally dense for us to sufficiently tease out the causal factors to make accurate predictions.
In those conditions, the goal should be to minimize the consequences of being wrong. We do that by decentralizing decisions to the greater extent possible.
I would expect the result to be strongly biased due to societal norms, much less direct manipulation.
Suppose a majority wants strong social programmes and basic income. The tiny problem is the economy is nowhere near big enough to afford it. Satisficing (!) executive branch puts some of the policies starving innovation and education budgets. Causing long term harm and stagnation.
Where is the minimax decision in here? Over what timeframe?
(P.s. I could've as easily picked liberal, fascist, military or conservative example. This one is relatively easy to understand.)
The second part of my response was to your framing of "science vs. gut-instinct". My point is that sometimes ignorance is thrust upon us by the complexity of the circumstances. In that case, my point was that decentralized policy making (separate from any notion of voting) was essential to reducing the worst case.
In other words, I am not claiming that democratization "results in minimax optimal decisions". I am claiming that decentralized governance structures do "result in minimax optimal decisions". Does that change your question?
edit: fixed spelling
In some cases yes. In others this doesn't seem to be the case - e.g. decentralised housing/planning/zoning policy seems to have lead to worse outcomes than more centralised approaches.
I haven't surveyed zoning policies, so I don't know whether I agree with your conclusion, but I'd like to clarify a couple things.
First, we must ask ourselves what the goal of optimal zoning policy is. Is it to maximize the number of people who live there? Achieve an ideal rent/wage ratio? Maximize the efficiency of shared infrastructure?
Perhaps it is to maximize the desirability of living in an area. If that is the case, then if people are complaining about the house prices that may mean you're doing something right.
In any case, assuming we settle on a shared metric (which may involve a combination of these various factors), we need to distinguish between comparing the average case and the worst case performance of centralized vs. decentralized zoning policy.
I would say that my claim is that decentralized zoning policy will have a significantly better worst-case than centralized zoning policy, and that crafting a good zoning policy is enough of a gamble that I'd rather settle for a less-than-ideal decentralized policy than risk a horrendous centralized one.
Now, to tie back to the original debate of scientific expertise in particular: would adding a group of economists and social scientists to the central planning committee tilt the situation in favor of central planning? I very much doubt it.
"At the core […] is the idea that people should design for themselves their own houses, streets and communities. This idea […] comes simply from the observation that most of the wonderful places of the world were not made by architects but by the people."
— Christopher Alexander et al., A Pattern Language, front bookflap[1]
[1] https://en.wikipedia.org/wiki/A_Pattern_Language
Because the costs of building new housing are local but the benefits are global, or at least more widely distributed than the costs. If one borough builds denser housing, that borough is the one that has to worry about building work, plumbing, schools and all the rest of it for those residents, while businesses from all its neighbours benefit from more customers and cheaper employees. For a starker example of the same effect think about taxation and public services: if we relied on local taxation being spent locally then rich areas would have low tax rates and high-quality public services (because even at low rates they can still raise a lot of money) while poor areas would have both higher taxes and worse services.
Localised government is rather like privatisation: it works well when the costs and benefits are on the same scale, but rather less well when there are externalities.
And even if we had perfect predictions, we would still have to choose what to do.
For example, if we had a perfect economics, we could predict exactly how many people would be made unemployed compared to how many made how much richer by raising the minimum wage (or trade barriers, or taxes). How should we weight these?
I think it's easy to greatly overstate how useful science is to policy-making. There are narrow areas where they should (and do) call a specialist in. But the big debates are not over factual, empirical, matters.
One less drastic solution may be to require a full disclosure of the process for all experimental results (to allow reproduction) and place much higher value in funding decisions on reproducibility.
One particularly egregious approach reproducibility requirement can address is doing enough "independent" studies until you like the result, then report that one study only (e.g., toss a coin 10 times; repeat as a new study until you find 10 tails; report a new discovery). My 2c.
This happens not only in which studies get submitted to the public, but also which submitted studies get chosen for publication. Journals are proving to be a poor way to handle the amount of science humanity is doing.
Environment has changed enough - especially number of people - that attempting to use traditional methods en masse could be very risky.
(Much less in nutrition or medicine where we have made major gains on the average.)
But science tells us nothing about what should be. That’s the realm of morals, ethics, and its applied (mis)application: politics.
But again, getting back to agreeing, scientific findings should not be used blindly, they should be used to inform. And as reproduction of findings occurs, the the topic that is being informed gets re-enforced or course corrected. It is very similar to policy, you have to try things. Some times the policy does what you want, other times it doesn't, and the process should continue and be correcting.
Science is the only tool specifically designed to weed out falsehoods and reach an objective consensus based on how nature works.
How is this "subject to falsehoods just like any other human endeavor"?
There is no substitute for science.
We all seem to agree that democracy only works when the population is informed. Well, are we going to inform ourselves with the beliefs and tweets of those which we grant moral authority? Or science?
"Science, however, cannot create ends and, even less, instill them in human beings; science, at most, can supply the means by which to attain certain ends. But the ends themselves are conceived by personalities with lofty ethical ideals and—if these ends are not stillborn, but vital and vigorous—are adopted and carried forward by those many human beings who, half unconsciously, determine the slow evolution of society.
For these reasons, we should be on our guard not to overestimate science and scientific methods when it is a question of human problems; and we should not assume that experts are the only ones who have a right to express themselves on questions affecting the organization of society."
https://monthlyreview.org/2009/05/01/why-socialism/
It's better than presumptively assuming that "Science" is an infallible always black-and-white.
So my less dystopian future goes like this. Train lots and lots of physicists, chemists, engineers, programmers, materials scientists, biologists, mathematicians. And (shock horror) through higher taxes employ them. Employ them on promising ideas, and on impossible ideas. Employ them on the problems of the day, and on arcane research that will almost certainly never bear fruit. And sure, right at the top in the Apples and Googles of the world, you'll have the best and brightest pulling together all of the discoveries made without all these reproducibility issues.
Or we could continue down the dystopian path. The low tax path where there really are only meaningful jobs for the best and the brightest. Where we rely on Apple and Google to decide what ideas are worth pursuing. Where the skills and abilities of a great many are not used. And most of us fight for the crumbs from the oligarch's tables. And the oligarchs? For every Elon Musk, there will be 10 who just want the biggest yacht and the prettiest mistress.
Let me say that again. In a low tax environment, the skills and abilities of a great many will not be used. And we will all be poorer for that. Genuinely poorer, because great discoveries will not be made. And it is happening now.
And I'll go one step further. We are also wasting the talents of our best writers and artists. The BBC and other national broadcasters are squeezed. Squeezed so that the idea of something as original and amazing as Monty Python appearing now is laughable. And again, we are all the poorer for that. Not just the lucky ones who get to make amazing shows or works of art, but all of us who never get to revel in them.
The low tax, small government world is a step backwards. If you wish to sum it up succinctly, you may say it is a world where, in order to placate the very rich, we take away opportunities from many, and impoverish the world as a whole.
That doesn't seem intuitive at all.
>Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something.
Sure, we will undoubtedly get better society-wide results if we identify a subset of elite students (notably defined primarily in terms of STEM fields) and only "bother" educating them, leaving everyone else to Just Trust Them™ and underutilizing the long tail of talent.
And then there's the premise that society somehow isn't looking to utilize the existing population of elite students, which of course it is, no shortage of organizations looking for outliers.
About the only good point there is that our system for allocating the best/brightest to certain problems could use improvement.
More public funding for basic research (including reproduction studies) would probably deliver major long term benefits.
Concentrating on grades in STEM poisons the well of raw thinking that science needs to look at everything in unique ways.
The only way that we can move forward is that everyone moves forward.
I do suspect that profit motive isn't enough to drive effective R&D, though, I just don't know exactly how to articulate a model for why except at the basic level that that truly searching R&D has a higher risk profile than other opportunities for profit.
I support the idea of more tax-supported public funding for a variety of things. I'd also support reshaping the tax code to incentivize R&D. But I'm also not sure that's a going to give everybody their opportunity -- I'd imagine there'd still be lucky ones there too. And we'd still have a lot of work to do to address incentives that drive reproducibility issues.
The scientifically minded are not the ones who will be deciding what's worthy of research. Politicians and bureaucrats will, and there are plenty of them who believe in all kinds of pseudo-scientific nonsense. How does your system prevent them from shoveling tax dollars into research on homeopathy and young-Earth creationism? And how does it prevent them from using funding to incentivize scientists to find the "correct" results?
To phrase my question more generally: I would like to know how your system would avoid making science merely a tool of political interests instead of a tool of moneyed interests.
There are certainly non-Science ways of forming an opinion and deciding policy, in many cases totally legitimate. But once science is claimed, then of course it has to be subject to science rigors.
From the journalism side I think you end up with the problem the wide spread field of journalism has, you have to be read by your audience. That means something with a wider audience is probably going to be more on the ELI5 side which means both the reader and the journalist have to take any reported findings more lightly.
The sliding scale, I believe, comes in with the rigor something is reported. That is both in the practice and the journalistic side. More details can give a more granular understanding showing limitations but some times broad strokes are necessary to get some information out to a wider audience or more quickly.
Take climate change, no doubt a controversial issue. To say it is "controversial" in the political sense would mean Left and Right (in the USA alone) have vociferously different stances on the issue. However, if viewed as a scientific controversy, we are now talking about detailed methodological concerns, like methods of data collection, analysis, kinds of statistical bias, and subtle changes in arbitrary parameters. Any scientist can tweak this or that in their model to make it conform more easily to their preconceived notions about climate change. Unfortunately, we also have some oil industry shills out there who got trotted out as an equally weighted side B to the side A of the dozens of scientists who would generally disagree. Then, you also have anti-science proponents who use the legitimate self-criticism of scientists to attack science as a whole.
It's a sad state of affairs. Science should be reported in the press, but it should also be reported much better than it is. In the USA in particular, STEM education is lagging behind: the average person can't delineate the good science coverage from the bad, and we have ridiculous notions and conspiracies that fail to become filtered out (anti vax, climate change deniers, flat earthers, moon landing was faked, etc.)
For me, reproducibility is one problem in a broader ecosystem of scientific problems, including science education generally, as well as misuse of statistics, and a saddening drive for incremental results at the expense of more broad-based thinking which might lead to fundamental breakthroughs. Our education systems must be reformed to deal with these problems, that's the only way out that I see.
Each study is a weak classifier and would have a 'reproducibility crisis' if retested on new data. However after lots of studies of similar phenomena, a strong classifier emerges. In the field, we call this converging lines of evidence.
It can be very hard for many reasons to stand up and say "this prior work is wrong".
If the replication crisis had been as well-known back then, she might have fared better.
To me it brings up an interesting point. If we view experiments as classifiers, then how would a machine learning expert set science policy and practice?
Also viewed this way, it makes me wonder about p-hacking, which increases sensitivity while increasing false positives. Since negative results are not generally reported, I wonder whether p-hacking diminishes replicability at the study level for efficiency at the aggregate level. This is an empirical question, as ethically it is of course a dubious practice under current understanding.
What you describe sometimes happens in a tradeoff between very different ML models (e.g. an ensemble of decision trees versus a single deep neural network has the properties you describe), but within any single paradigm (an ensemble of neural networks versus one NN with more training time) it's the other way around.
edit: When I read about ensemble theory, you receive support for the increased computational cost.
Evaluating the prediction of an ensemble typically requires more computation than evaluating the prediction of a single model, so ensembles may be thought of as a way to compensate for poor learning algorithms by performing a lot of extra computation. Fast algorithms such as decision trees are commonly used in ensemble methods (for example Random Forest), although slower algorithms can benefit from ensemble techniques as well.
But I found this interesting:
Empirically, ensembles tend to yield better results when there is a significant diversity among the models.[4][5] Many ensemble methods, therefore, seek to promote diversity among the models they combine.[6][7] Although perhaps non-intuitive, more random algorithms (like random decision trees) can be used to produce a stronger ensemble than very deliberate algorithms (like entropy-reducing decision trees).[8] Using a variety of strong learning algorithms, however, has been shown to be more effective than using techniques that attempt to dumb-down the models in order to promote diversity.[9]
..which under my analogy would seem to suggest that exact replication is less productive than having a diversity of study designs.
Then there is a problem that an ensemble of biased classifiers (mostly in the same ditection in this case - positivity bias) will magnify the bias.
This is a reason why metaanalyses have to get at the actual raw data to pool and analyse as well as correct for multiple biases. Even then the process is not perfect.
1. assumes the validity of the scientific method
2. relies on the scientific method as its critical lens
Whereas those who critique science as a whole:
1. assume that the scientific method does not work and does not arrive at "truth"
2. then use scientists being self critical to prove #1.
Such a "proof" does not work as there is its uses the assumption "the scientific method arrives at truth" to derive the contradiction "the scientific method does not arrive at truth". See for instance comment: https://news.ycombinator.com/item?id=16859200
In reality, work on reproducibility is about improving the practice of science overall. It does not in itself show that science is inherently untrustworthy. What it does show is that scientific discovery is difficult and it takes a lot of effort and new findings should be treated critically. What does critically mean in this context? It means with in the boundaries of science analyzing the theoretical basis, hypothesis, method, and experimental results for potential flaws. It does not mean to be skeptical as a default because science "doesn't work."
What you should have said is that some critics start with the premise "the scientific method does not arrive at truth", and then use other people's arguments that depend on the premise "the scientific method arrives at truth" to support their claim, which is indeed logically invalid.
I think what happening here is a bit more serious. They are showing a widespread crisis. It is not just some minor feedback to improve the process.
>It does not in itself show that science is inherently untrustworthy.
I think when statistics is involved, the results are inherently untrustworthy. This is not really surprising because there is a whole bunch of ways these studies that involve statistics could go wrong. And we are still finding new ways on how this could go wrong.
Then there are things like publication bias, that takes this to a whole new level. Things like that means that a biased body of journals can project any consensus that it favors just by selecting studies that fit its narrative. The inherent issues with statistics means that you can find studies that shows any possible outcome.
Ummm...are you kidding? Statistically vetted results are inherently UNCERTAIN, but how could they possibly be inherently untrustworthy?
Even if a mechanistic effect is observed, its relationship to a particular cause or influence is only established statistically. In fact, the very observation is often performed u Dee the umbrella of statistical calibration of appropriate instruments.
I would recommend reading the article. The executive summary summarizes things. The issue is not statistics in and of themselves, but how they are used in the overwhelming number of studies - particularly those in the social sciences or related with human physiology.
This is the whole point. It is the difficulty in doing so, or the difficulty in deciding if this was done appropriately or not is the only thing that it untrustworthy..
>People lie while speaking English every day yet I don't encounter many people claiming that English is inherently untrustworthy.
No one expect a statement to be true simply because it was uttered in English.
Another very real issue here is that malicious use of statistics can be used to show nearly anything in ways that can be extremely difficult to detect, even when the maliciousness is hidden in plain sight. And then going a step beyond that there's plain old number fudging which is almost impossible to prove since variance works as sufficient plausible deniability. And finally there is of course plain old ineptitude. Like you mention even when trying to do things completely by the book, statistics are incredibly difficult to get right.
Something that comes to mind here is the recent MIT study stating that Uber drivers earned $3.37/hour. That study was completely broken. [1] It's debatable whether the cause was maliciousness or ineptitude, but the point is that these problems arise, with a disturbing regularity, even when the most reputable of names are attached to them.
[1] - https://qz.com/1222744/mits-uber-study-couldnt-possibly-have...
There can be malice, sure. But there can also be desire to believe. And, hey, here's a statistical analysis that shows what the investigator is already biased to believe anyway...
1. No result is 100% reproducible because you can never completely reproduce the conditions of any experiment. The best you can hope to do is to reproduce the conditions that matter, but enumerating those has to be part of the theory you are testing, and so you can never be 100% sure that you have a complete list.
2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science.
3. The end-product of science is not truth, it is explanations of observations. Those observations can (indeed must) include non-reproducible ones. Sometimes the explanation of non-reproducible results is "experimental error" or "delusion" or "we just don't know." But non-reproducible events are nonetheless within the purview of science.
On the other hand...
4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true. That, combined with the suppression of negative results, guarantees that the results published in scientific journals that adhere to those standards will be unreliable. But that doesn't really have anything to do with reproducibility per se, it has to do with the fact that the journals use a weak criterion for defining positive results. That, combined with our human predilection to value positive results over negative ones and the understandable desire of scientists to advance their careers, all but guarantees that journals will contain many defensible but false results. But this is not because of a lack of reproducibility per se, it's because of poor policy choices.
You mean that papers are published with P<0.05?
When did this become a "standard"? Most papers I have seen in respectable journals and other sources would not strive for such a high Pvalue, it would be much lower like 0.001.
I'm guessing you meant to say that no papers are published with P<0.05. And of course they are, but that's irrelevant. What matters it that many journals, especially in "softer" sciences (i.e. not physics) use p<0.05 as their threshold, which guarantees that at least one paper in 20 will be "wrong". Then add self-suppression of negative results and the actual percentage of "wrong" papers becomes higher, potentially much higher if real positive results are rare.
> When did this become a "standard"?
I have no idea. You'll have a ask a historian.
It is not, in any way a measure of reality. It is a guess at how accurately you've modeled the system, how accurate your priors are, and so forth.
> It does mean that ... 1 in 20 would be explainable by the null hypothesis.
That seems like splitting a pretty fine hair to me, particularly since I haven't actually committed to any of the many possible interpretations of the word "wrong".
While not everyone in a particular field has been a party to this myopic view of statistical significance, enough have been that the American Statistical Association released a statement on the topic not long ago. That may not seem like a big deal but, given the organization's historical reticence to officially weigh in on such issues, it really is.
ASA statement on p-Values: https://www.amstat.org/asa/files/pdfs/P-ValueStatement.pdf
Most people would say in the 20s and point at Fischer's publications, though you can have arguments for later and earlier. Some of that is mentioned in the Wiki page, and you can see 0.05 all over the article [0], and the other related wiki pages[1]. You can find some key quotes there:
"The significance level for a study is chosen before data collection, and typically set to 5% or much lower, depending on the field of study"
"In 1925, Ronald Fisher advanced the idea of statistical hypothesis testing, which he called "tests of significance", in his publication Statistical Methods for Research Workers. Fisher suggested a probability of one in twenty (0.05) as a convenient cutoff level to reject the null hypothesis.In a 1933 paper, Jerzy Neyman and Egon Pearson called this cutoff the significance level, which they named α."
If you mostly see lower p-values, that most likely means that you usually read non-social science papers. I guarantee that most social science (and related) fields use 0.05 regardless of the respectability of the journal.
As an example of fields with lower p-values - 0.001 is common in medicine, and physics commonly has even lower values.
0. https://en.wikipedia.org/wiki/P-value#History 1. https://en.wikipedia.org/wiki/Statistical_significance#Histo...
I think this depends on the field. In some areas of neuroscience for example, brain data can have extremely high variability and is very expensive to collect a lot of, so p<0.05 is not uncommon.
Some of the best advice I got was to spend a lot more time than one might want to really come up with an experimental design that avoided a statistical outcome and instead probed at an either/or mechanism. It's not always possible, but when it is, it can be a powerful reply to fighting statistical battles (see the Michelson-Morely experiment for a classic example).
Further careers, families, reputations, humans are also part of that scientific process, which together with the above require and admit some deviation from perfect efficiency. Which we as a society should demand be okay.
I think news organizations make this worse. Whenever a new paper comes out they'll champion it as fact if doing so leads to clicks.
Peer reviewers can only hope to validate that the science was done well, not that it is inarguably correct. It requires a body of work around a topic before it's fair to say at all that something is "for sure" proven.
University PR is far more guilty. Buzzfeed isn't scanning the current issue of "Physical Review Letters" for the latest scoop.
That is the idea behind Rutherford's famous motto, If your experiment needs statistics, you ought to have done a better experiment.
So long as the researchers aren't outright inept or fraudulent, what's there to be ashamed about? Those would be some of the more interesting papers to read, in my opinion.
No, that won't help. You'd end up with the Journal of Creation Science and things like that.
What we need is pre-registration: https://cos.io/prereg/
We also need to change the culture so that positive results are not disproportionately rewarded.
So we choose to spend our time reviewing and evaluating those experiments that do seem to have at least some chance of saying something interesting. Even then, though, depending on the discipline, if the researcher did everything perfectly, they're still saying that there's a 5% or some% chance that they just "got lucky" and there is no underlying effect at all.
http://www.jnr-eeb.org/index.php/jnr/index
I would actually prefer that each and every journal devote at least half of their space to negative results. I have seen some sublime experimental setups that produced negative results. I can't begin to imagine how instructive it would be to see more such experiments.
I am not sure this is correct. The rules that govern celestial bodies is same as the ones that govern objects on earth. So why are they not reproducible? It does not require to measure forces between celestial bodies to measure the value of G. Measuring the forces between two masses on earth is enough.
So, for example, we can predict with ridiculous accuracy where the next total solar eclipse is going to happen. But once it happens, we can't roll back the clock and make it happen again.
I am finding a hard time to grasp what you are telling here. If you deduce a set of laws A from an event X, you don't need X to repeat to check the validity of A. You can check it against another event Y, where you predict its characteristics using A. If the laws are valid, the prediction ll match the actual observation..
That is exactly what I wanted to ask you originally when you cited the celestial mechanics example. It is not relevant in cases we are deriving a theory underlying the behavior, rather than axiomizing the very specific behavior itself...
Deriving an underlying theory would be deducing the force of gravity from it.
Axiomizing the very specific behavior would be making a rule that says. "If the object A (and object A only), while moving in a straight line, comes at so and so coordinates with respective to B, will end up in an orbit around it"
So these are two completely orthogonal issues. The question of what you do with the result of an experiment is completely independent of whether or not you can reproduce that result. The latter is what is under discussion here.
So you can, for example, observe that there was a solar eclipse at a particular time and a particular place. But you cannot repeat that observation. (By way of contrast, you can drop and object from a particular height and observe that it takes a particular time to fall. You can repeat that observation.)
What you do with those observations has (almost) nothing to do with whether or not you can repeat them.
I have a pedantic correction, but a relevant one. If a journal makes a rule that results must have p < .05, and then scientists go off and do a bunch of well-managed science they will get results with a range of p-values. .05 will be a ceiling, not a floor, on the possible p-values. More papers have a p-values right around .05 than they would if results were being produced fairly, so that's actually evidence of bad practices.
Now, it turns out that the floor is far lower than researchers believe and the field would be better off by slower incremental progress built atop "obvious" results, but still, I assert that publishability is an adequate explanation for high p values and obscures the evidence for bad practices.
Not that I have any doubt that bad practices abound; I have an undergraduate degree in psychology and am well aware of how things are done. It's not even malicious; people don't understand statistics, don't understand significance, don't understand models, and are mostly just doing whatever they can to make themselves feel good about their own intuitions and biases.
That's wildly overly pessimistic. That would only be the case if scientists just went around, looking at the world, and came up with null-hypotheses willy nilly. That's generally not the case. There is often a _reason_ for conducting the test and a plausible cause of action. There are two possible reasons for a significant result:
* The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance)
* There is a real effect (and the null hypothesis is actually wrong)
This becomes more problematic in reproducibility tests, though, since that biases my prior towards a correct null hypothesis and now you must be very careful about pre-registration and the numbers of folks worldwide that are trying to reproduce a given experiment.
> That's wildly overly pessimistic
Why? You yourself said:
> The null hypothesis is correct but they got "unlucky" data (5% chance or p% chance)
How is that different from what I said? 5% is just another way of writing 1 in 20.
They are showing the _likelihood_ of generating the data were there no "effect." You make a "null hypothesis" — that is, you assume there is no effect — and you construct a distribution of the results you might see in such a case. If your results are relatively unlikely, then you say the null hypothesis is rejected at a p-value level, leading support to the fact that there is such an effect.
But there's another bullet in my comment. You could _also_ see a low p-value if the null hypothesis is wrong! That would _also_ lead to a low p-value, but for a completely different reasons — reasons that are typically elucidated in the entire rest of the paper. Thus, you should be most suspicious of papers that do not have a reasonable cause of action, as that should increase your prior that the null hypothesis is actually correct.
If I were to edit that post, I'd add a third bullet — that they cheated or did bad science. My objection is to you stating that the statistical tests alone guarantee at least a 1 in 20 failure rate with a p-value of .05. There are indeed many other reasons to be suspect, but the statistical tests don't guarantee this.
The on top of that baseline rate of false results you also have self-selection and occasional outright cheating that drives the failure rate higher than 1 in 20. 1 in 20 is what you get when you use a 0.05 threshold and everything else in the system works flawlessly.
If what you mean to say is that "1 out of every 20 statistical tests of a false finding will demonstrate a 'significant' p-value and may subsequently get published," then we're in agreement.
http://blog.minitab.com/blog/adventures-in-statistics-2/how-...
At the same time, though, converting p-values to a bayesian posterior or conditional probability that says something about the likelihood of the _hypothesis_ (like you're trying to do) is hard and generally way more pessimistic than 1 in 20 — it's more like 1 in 3.
http://www.dcscience.net/Sellke-Bayarri-Berger-calibration-o...
Of course it can. In fact, it can tell you exactly how much more likely one case is versus the other. P=0.05 means that there is a 5% chance that the null is true and hence a 95% chance the the null is false.
> says something about the likelihood of the _hypothesis_ (like you're trying to do)
No. I am simply saying that there is a lower bound on the rate of false positives caused by the choice of the threshold p value. If you chose P<=0.05 then you will have an error rate of at least 5%. The actual error rate is almost certainly higher (that's what "lower bound" means) for a host of other reasons.
Were I to run another 100 independently designed experiments that all tested big real effects, then a p-value of 0.05 could result in all 100 accurately reflecting reality.
Were I to run 100 independently designed experiments that all tested false effects, then at a p-value level of 0.05, I should see about 5 erroneously significant results.
Now I submit all these significant results to a journal and they all get published. Of the 135 results, only 5 are coming to the wrong conclusion.
Look at that, an error rate lower than your lower bound. We simply don't know the sizes of these populations. It could be better or worse than my example, and it could be better or worse than 5%. There simply is no lower bound.
Of course you're right that there is a problem here but there isn't a lower bound guaranteed by the stats tests. Quite a bit of the problem stems from misinterpretations of p-values. For someone who likes to condescendingly explain concepts, you might want to make sure you have the harder concept down before assuming others don't understand "ratios" and "lower bounds." At this point you're arguing against the sources I cited. You should check them out. :)
That's actually not true. The problem is that you cannot define what is a "real effect" without begging the question. Let me illustrate with an example: Suppose I do what appears to be a legitimate experiment to test a well-accepted law of nature. Unbeknownst to me, my instrument is faulty (or has been tampered with) and is in fact returning essentially random results. If I choose p<=0.05 then I will get a positive result 1 time in 20, i.e. I will "erroneously find no result" 19 times out of 20. Except that I'm not really erroneously finding no result. I am correctly finding the results of a different experiment than the one I think I'm running.
And, of course, an instrument that returns random numbers is an extreme example. See:
https://arstechnica.com/science/2018/04/new-measurements-set...
for a current real-world example.
Now, you do have a valid point in that I made a tacit assumption when I claimed that statistics put a lower bound on the error rate. That assumption is that most experiments that are done test hypotheses that ultimately turn out to be false. I believe this assumption is actually true. This is one of the reasons making scientific discoveries is hard. But I didn't actually state this assumption, so I'll cop to that.
> At this point you're arguing against the sources I cited.
That's not true either. The first source says that the error rate is much higher than the p-value, which agrees with me. (I'm actually a little skeptical that the error rate is as high as your source says -- "At least 23% (and typically close to 50%" for p=0.05, but that's certainly not refuting my position!) I don't have time to go through your second citation right now.
That's precisely my point. We don't know the sizes of these populations, so you really can't say anything quantitative about how the p-value affects the proportion of bad results in any given journal. My example was intentionally contrived.
You just have to be really careful when you talk about p-values because _so many people_ have this misunderstanding and it's actively harmful to getting the correct interpretation. That's why we keep going back and forth here — I'm not disagreeing that the situation is bad, I just want folks to recognize what the stats say and what they don't.
I think we can make some pretty reasonable guesses.
In any case, I certainly agree with you about the big picture: it's confusing, it's important to get it right, and a lot of people don't understand it. I may even be one of those people. But at the moment I believe that our disagreement is really over something else, namely, whether it's reasonable to have a prior on the null hypothesis.
No. It simply does not mean that. It would be very convenient if it did, but it does not and can not.
p=0.05 means that if the null hypothesis were true and you ran your experiment, then you would only have a 5% chance of getting the results that you did.
The distinction is not at all obvious, but you'll need to understand it before you can make sense of statistics.
The p-value is not the number you want; what you want is the probability that the null is false, but you can't have that unless you know the actual probability distribution. Which generally isn't something you could possibly know or figure out.
(You can make educated guesses about that distribution, by making a guess and then updating it based on observed results. That's Bayesian statistics, which is great because it does give you a full probability distribution and all you have to feed into it is... uh... a probability distribution.)
If there's no pre-registration, my prior of a paper actually being true would be much lower than statistics would say simply because the incentives are aligned so that there's a motivation to exaggerate significance.
The reason this is being called a 'crisis' is because it's looking like the vast majority of science, particularly in the social and human physiological sciences, is junk.
There are definitely other reasons that lead us to such horrifying results — especially in fields where experimenters _do_ conduct experiments seemingly at random and without a reasonable pathway of action (I'm looking at you, social psychology and friends).
On top of this, the more interesting results are the most surprising. And therefore the least likely to be true a priori. So the chance of an interesting study being true is even lower.
Yes and no. I think you have to have predictions in there somewhere. That's why your celestial mechanics is science: whatever the movement is, you were able to say what it was going to be beforehand, even if it only happened once.
As for explanation, that is not necessarily so simple, either. Sure, a lot of things seem to make more sense with a theory, but not every theory makes sense. A lot of stuff in fundamental physics is predicted by the equations, but in what sense are the equations explanations? "Shut up and calculate" comes to mind.
And also, shut up and come up with something falsifiable (in our lifetime).
That sounds a whole lot like truth to me, or at least, like any sane construal of truth.
And in fact, General Relativity is the best explanation science has given us at the moment, which seems to fit what I said.
Until this moment, I never suspected how insightful the following is:
https://wiki.tfes.org/Universal_Acceleration
It's worth noting that this is part of the goal of reproducing results. If two identical experiments produce the same result, you're only providing proof for the very narrow theory the experiment tests. Slightly different experiments allow you to demonstrate that the theory is applicable. Imagine if evolution only occurred in the Galapagos, or if gravity only worked in an apple orchard in Cambridge. The only reason we know that evolution and gravity aren't local oddities is that they have both been tested in a wide variety of locations, with a wide variety of experiments.
The term 'science' is used in this way. Perhaps it should not be. Perhaps semantics drive the issue.
In the field in which I am familiar, we have 'expert witnesses' that will use reproducible scientific principles to analyze, for instance, blood scatter to determine the angle in which a shotgun was held when it discharged and blew off a particular person's head. The principles -- based on physics, chemistry, etc -- are sound and scientific.
The problem is, we will invariably have TWO very scientific expert witnesses who BOTH use reproducible and accepted methods. Yet, using their scientific tools, they will come to two utterly different conclusions.
It's left to a jury to decide which interpretation is correct.
While the tools used by the experts are scientific, their opinions are not. While their scientific tools are reproducible (for instance, the speed at which coagulating blood and brain tissue will slide down a stucco wall given all the variables such as temperature, humidity, friction coefficients, etc.) their ultimate reconstruction of what happened is not reproducible, and it is left to lay people to draw their conclusions.
There should be more of a distinction between Science and History. That is, science -- strictly defined -- is all about reproducibility. History -- by definition -- is not. Using scientific tools to reconstruct a past event does not result in a 'scientific' theory. Rather, it results in a historical theory arrived at with scientific tools.
Whether it is blood scatter from a murder (or suicide, depending on whom you ask), the formation of the Grand Canyon, or the development of finch beaks on Galapagos, the ultimate theories are inherently non-scientific as they can not be tested. The methods and tools used to derive the theories can be... but not the conclusion itself.
In short, perhaps the term 'science' is being used for things outside its domain. (Or, alternatively, if we wish to include such things inside the domain, we should broaden the strict definition of science. Like I said, there are some issues of semantics that may be driving some of the issues in the article... other issues, of course, pertain to sloppiness, errors, and the like.)
If the original study was under-powered, the estimated effect size in that study will be inflated and any replication attempt that uses this inflated effect size estimate will be severely underpowered.
Plus, two independently conducted studies that are both powered at 80% to detect a true effect will both be positive results only 64% of the time (assuming absolutely nothing fishy going on, e.g. p-hacking).
This is why you include the error in your results. We don't care if your two experiments result in 100% the same results. We care about if your results line up within the error of your experiment.
> 2. Even a completely non-reproducible result can be scientifically significant. For example, celestial events are almost never reproducible. Our understanding of celestial mechanics nonetheless rests on solid science.
This is why we share data. That is in essence our observation. A celestial event might be caught by only one instrument, but several people can develop several different models. We wait and watch for similar events though, to check if models are consistent (within error).
> 3. The end-product of science is not truth, it is explanations of observations.
I just wanted to repeat this because it can never be stated enough.
> 4. The statistical tests currently in widespread use as a criterion for publication in peer-reviewed journals guarantee that at least one result in 20 will be due to chance and not because the hypothesis being tested is actually true.
While I don't like p values, because of hacking, (especially 0.05) that's not how stats work. Flipping 10 heads doesn't guarantee 5 tails, not even 1. I wouldn't use as strong as a word as guarantee.
But this is also an argument FOR reproducing. If multiple experiments are consistent with one another (within error) than that strengthens the argument.
TLDR: More brains that look at a problem helps solve the problem.
I will add my own statements about the reproducability problem in science. One is because there is less funding for it. Reproducing an experiment isn't sexy. Another problem just stems from that data isn't always open. It is hard to review work if you don't know everything about the experiment. Data can even have simple mistakes that just weren't caught. But it is also embarrassing to share data.
I never said it did. But in the long run, flipping a fair coin will generate pretty close to 50% heads and 50% tails. That's what it means to be a fair coin. Likewise, in the long run, using a p threshold of 0.05 (which many journals do) will generate 5% false positive results (that's that the 0.05 means), i.e. on average 1 false positive result for every 20 experiments you do.
Flipping an unfair coin twice in a row and discarding the HH and TT doubles will generate close to 50% HT pairs and 50% TH pairs. This only presumes that subsequent flips are independent events.
It's still not going to be exactly 50%, because random events are random. You can get as small an error margin as you need by increasing the number of trials. And in some cases, this property of statistical analysis makes it easier to increase N than reduce the randomness and uncertainty in the experiment.
Example: I want to test if evolution is true. I hypothesize that if evolution is not true, then God will give me some sort of sign, e.g. I pray to God to make a coin that I flip come up heads if evolution is false. I flip the coin and it comes up tails. I conclude that evolution is true. That conclusion would be wrong even if evolution is true.
It is if P(A) is 1, which is what you stipulated when you asked "How can u get a wrong positive result if you only test correct H1s?"
> "... using a p threshold of 0.05 ... will generate 5% false positive results" is just plain wrong
No, it's not wrong, it just makes a tacit assumption that most hypotheses that get experimentally tested are incorrect. Which is a reasonable assumption. If it were not true, making scientific progress would be a lot easier.
This is such an astute observation that reproduction studies can be used to find a placebo control group for bad scuence. Parapsychology was one such group, but now we can find many others. Brilliant.
It's so bad now that for any article/study, I look at the sample size first. If it's too small (especially < 100) or they don't say I just ignore it. And if you don't publish or give some estimate of your effect size I just think about it directionally but don't give it much weight mentally.
The executive summary lists 40 points on how to improve the reproducibility of science. A bit over half of them are addressed to the sources of capital such as private organizations, universities, and governments. I think many of those points are good. However, I don't think the other points, the ones that recommend doing science in different ways, have a good punch. Even if you fix the problems there are today, so long as science is a rat race of trying to get grant money to stay afloat while burning out grad student after grad student, I think other pathological practices will creep in as a completely rational response on the parts of scientists to a hostile ecosystem. There's just a very big gap between how science should be done and what capital owners want from science.
Maybe the biggest problem is what you mention, but in another direction. For whatever reason there seems to be extremely little interest in the private direct funding of science. In times past the aristocracy would often fund scientific research on all sorts of topics. Today the practice seems to have all but disappeared, certainly if we measure relative ratios of the practice.
Eliminating fossil fuel use is also simple. But there's another dimension to this problem, called "political viability", which I feel you're overlooking.
Too much income equality. You can't have Rockefeller University without Rockefeller.
It was "a team from Bayer Healthcare" who "tried to replicate the results of basic cancer studies," failed, "and kicked off a media storm questioning the legitimacy of cancer science—and science in general" [1]. The "capital owners" looking out for their own buck are performing more effectively, narrowly speaking, than academia.
[1] https://www.wired.com/2017/01/fighting-cancers-crisis-confid...
It's also quite obviously the case that profit seeking research has incentives to be bad science in other ways.
I agree. My point is the Marxist framework is a bad one for modern science.
We have endowed academics and ones working for private institutions and getting grants from public bodies run by scientists. Privately-financed biotech companies founded by academics taking moon shots and getting acquired by marketing and distribution powerhouses. Even delineating who is "capital" and who is "labour" quickly becomes pedantic more than useful.
I will concede that it's reasonable to think it's an incomplete and limited framework, but I have to take issue with calling it bad. For example, I think this starting point is more productive than the premises of this report.
> Privately-financed biotech companies founded by academics taking moon shots and getting acquired by marketing and distribution powerhouses. Even delineating who is "capital" and who is "labour" quickly becomes pedantic more than useful.
In this example it's plain to me that VCs are capital and entrepreneurs are labor. That's usually the game in such moonshots. Here we can analyze what sort of science is being done through this relationship through a Marxian lens too. VC's want to make big bets with potentially astronomical upside. That means only certain kinds of science will ever be done in this way. Yea, this kind of science is probably more replicable on average, but what about something like Theranos? This sort of science funding has its own problems with inefficient and incomplete allocation of resources.
I'm not a hardcore Marxist; I don't think you can reduce the world to one grand theory based on hot takes inspired by Hegel, but I think it's a more powerful tool than most care to admit.
- she is exploited, i.e., forced to act in a profit-maximizing way (in particular, short-term profit). Each other behavior will be penalized (no career, no tenure, ...).
- she is exploiting, i.e., forces her workforce to act in way that they maximize her profit.
Just admit that the marxist analysis doesn't hold much water in this case. That doesn't mean it is never useful, just that you might need to think a bit before applying it.
You're really set off by the word M__x, aren't you?
The very nature of capitalism is that people who produce monetizable results should be rewarded for that work, and that people who don't shouldn't. This is fundamentally incompatible with doing science in a reproducible way: paying for positive results and not negative ones gets you p-hacking as an emergent effect. Of course the capitalists don't want to pay for non-reproducible positive results, hence the firestorm. But they more strongly don't want to pay for negative results.
The capital owners in the science industries will probably end up doing something sort of like what the capital owners in the software engineering industries have done: begrudgingly accepted that a non-capitalist approach (open source development across multiple companies without proprietary advantage) produces better results for everyone, and figured out a way to kind of support it. Much like the capital owners in the software engineering industries, they still won't be very effective at it. Their nature will be to still try to extract proprietary advantage when possible, to cut costs per the tragedy of the comments, etc., and so this model will only work well when funded by companies rich enough that they can allow themselves not to show an immediate return on their sponsorship.
Issues with reproducibility have been discovered independently in a number of different scientific disciplines going back to the 1990s at least. This was then covered in a New Yorker article by Jonah Lehrer, published in 2010 [1]. It included the phrase "decline effect", coined by a researcher studying ESP in the 1930s.
A great deal of the research described in that article is academic in nature. The article also alludes to the contributory influence of funding as one of the causes of this phenomenon:
"The bias was first identified by the statistician Theodore Sterling, in 1959, after he noticed that ninety-seven per cent of all published psychological studies with statistically significant data found the effect they were looking for." ... "In recent years, publication bias has mostly been seen as a problem for clinical trials, since pharmaceutical companies are less interested in publishing results that aren’t favorable. But it’s becoming increasingly clear that publication bias also produces major distortions in fields without large corporate incentives, such as psychology and ecology."
[1]: https://www.newyorker.com/magazine/2010/12/13/the-truth-wear...
The book is set around 1660, as the center of the scientific universe shifted from Italy to England. Interestingly, the formation of the Royal Society occurred in the midst of the English Civil war. Not only economic forces, but military forces and political forces had a dramatic impact on shaping what became the formalization of of the scientific method, and peer review process.
Consider that the very existence of Oxford (and other universities) hung on the arbitrary conceptions of military generals and the political implications of their decisions.
It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination. In fact, much of it is due to cultural marxist insistence on outcomes conforming to political correctness, as well as uncritical acceptance of politically conforming work. See the ongoing fight against the idea of a high genetic basis for adult IQ.
All of this is obvious enough: capitalists aren't interested in non-reproducable results unless it is obviating them from blame for a given externality. But that sort of research is a small fraction of overall research (and should absolutely be done by impartial third parties.)
[1] https://en.wikipedia.org/wiki/Funding_of_science
It’d be cool if your and the other poster’s ideological spat could try to not use science as the battleground, or at least eliminate more likely issues before going there. Social sciences are hardly even science from the perspective of a chemist or physicist, there’s no need to delve into petty ideology. Methodology is a much richer source of hurdles to overcome than old and sad debates.
I don't disagree. A big part of the problem is the social sciences claiming the mantel of science (often just statistics layered over tiny samples, as you point out), in order to imply some sort of impartiality and precision on par with the hard (i.e. real) sciences.
Unfortunately, and I say this sincerely, I believe the ideological nature of the social sciences since the 60s has been responsible for a huge amount of uncritical acceptance of bad (false) work as well as preventing a lot of unpopular good (true) work. I don't know how to deal with that fact other than stating so forthrightly, which, unfortunately from the perspective of science, carries necessary ideological freight with it.
A chemist or a physicist still have hard time dealing with particle assemblies of say 100-100000 particles. For several last centuries they learnt to deal with [aggregate properties of] assemblies on the scale of 10e20+ particles. Only during last century they came to pretty good understanding of 1 particle.
In social sciences this is Facebook and Cambridge Analytica (ie. they are real social science today, not some professor in an office). This is the scale where aggregate properties appear, and they start to be able to drill down from it.
https://en.wikipedia.org/wiki/Bogdanov_affair
> It's a nice story you've got there, but the reason there is a replication crisis in the softer sciences is not due to the evil capitalists of the marxist imagination.
You understand literally nothing about Marxism. For example, "capitalist" isn't a moral designation, it is a description of an economic agent's role in an economic relationship. Capital owners have access to resources and they give permission to laborers to use those resources. That's it.
> cultural marxist
You can't be serious.
> See the ongoing fight against the idea of a high genetic basis for adult IQ.
This plays out more in the public sphere than in scientific communities. It's also the case that a few popular intellectuals try to sell psychometric science as saying more than it actually does about how social policies should be designed. This, again, happens moreso outside of scientific circles.
Studying economic factors underlying social activity isnt by itself Marxist. This methodology is adopted in support of different political positions. In fact, many free-market economists also do this and get accused of 'economics imperialism' by social science faculty who want to study other factors.
Another point made in [1] is the blame isnt just on companies but people being influenced.
[1]https://medium.com/@drjasonfung/the-corruption-of-evidence-b...
If you see everything through a such a strong lens, you see very little.
https://news.ycombinator.com/newsguidelines.html
Getting triggered happens to everyone but is not an act of intellectual curiosity. https://news.ycombinator.com/newsguidelines.html
That said, they should be well-informed on the subject. So it probably does contain a lot of good information.
Also, when experiments depend on source code, etc. we need real engineering tools/principles applied. (Something like: “you can’t publish paper X if you aren’t including a public repository with build/run instructions”.) Unfortunately, there are all kinds of reasons why scripts/builds could fail just a few months or years later so they would have to be checked too.
I think it would be cool if document-generation caught on in the publication of papers, i.e. the paper itself is generated by running actual experiment scripts and producing charts, etc. from plain text source.
[0] https://en.wikipedia.org/wiki/National_Association_of_Schola...
[1] https://en.wikipedia.org/wiki/National_Academy_of_Sciences
[2] http://www.pnas.org/content/early/2018/03/08/1802324115
Thanks for the link to the PNAS -- looks like a whole special issue, not just an article, in fact. Great. I'm gonna ignore the fake-NAS one, and read the real-PNAS one.
I wonder what National Association of Scholars' motivation is here exactly.
However, this paper isn't a study. It's just an essay. And while, yes, the paper in question does highlight numerous issues in reasonable and rational ways, it also uses this as the breakout quote on page 36:
> Scientists readily accept results that confirm liberal political arguments, and frequently reject contrary results out of hand.
Now, there's a couple problems with this statement. The most obvious is that it implies blame on liberal politics. On the same page they detail that the percentage of self-identified liberal psychologists at a conference (~800 of 1,000) wildly outnumbers the number of self-identified conservative psychologists at the same conference (3 of 1,000). Well, this conference is the Society for Personality and Social Psychology, which is based in southern California. Would that bias attendance? However, even if we assume that distribution is correct, the quoted statement doesn't say anything about the conservative scientists. Would it also be correct that conservative scientists readily accept results that confirm their political arguments? We don't know. Maybe it would be more correct to say this:
> Scientists readily accept results that confirm pre-existing personal political arguments, and frequently reject contrary results out of hand.
Instead, this paper chooses to imply that scientists who are more liberal are somehow inherently more biased. If we assert that good scientists are those who don't allow political arguments to influence their acceptance of their results, then the natural conclusion to draw from that is that being liberal is incompatible with being a good scientist.
Don't get me wrong, this paper makes a lot of legitimately good points with good examples, but the whole section on groupthink comes across a little bit biased itself. However, it's existence means I can't shake the feeling that maybe I'm missing something.
So having show that there's a reproducibility problem, the paper concludes. Then very end of the paper, the afterword lists these points introduced seemingly out of the blue:
> Symptoms of Pathological Science:
> 1. The maximum effect that is observed is produced by a causative agent of barely detectable intensity, and the magnitude of the effect is substantially independent of the intensity of the cause.
> 2. The effect is of a magnitude that remains close to the limit of detectability; or, many measurements are necessary because of the very low statistical significance of the results.
> 3. Claims of great accuracy.
> 4. Fantastic theories contrary to experience.
> 5. Criticisms are met by ad hoc excuses thought up on the spur of the moment.
> 6. Ratio of supporters to critics rises up to somewhere near 50% and then falls gradually to oblivion.
These are from Irving Langmuir (https://en.wikipedia.org/wiki/Irving_Langmuir) and his criticisms on research conducted with unconscious biases or subjective elements. He meant it as a criticism of research like cold fusion.
Now, there's not any real one-for-one discussion of these items or how they relate to what came before. They're just presented basically as-is as the final portion of the essay with few notes that they correlate to what came before. You'll notice, however, they they all call into question the credibility of the scientist as well as the credibility of the scientific community, and not the quality of their research or data.
However, let's say you're politically conservative, and you're reading this essay. Maybe you don't believe in climate change. You get to the end of this paper, and it tells you that in spite of the claims of accuracy of the data, in spit...
To eliminate any questions about social context or bias or agenda? Totally not possible, social context is an inherent part of science, as in all human endeavors.
I think the purpose of reproducibility is to well, make science _possible_, it is the fundamental bedrock of science -- to determine what and how we can make reliable predictions about the world. It doesn't mean bias and effects of social context no longer exists though. Science exists anyway, in a social context -- as evidenced by the past couple hundred years!
In general, I don't think science exists to eliminate _any_ questions, science exists to ask questions. Eliminating questions may be a goal of ideology or religion, but not science. (and in the real world, they certainly influence each other. science is a goal to strive for, not a fait accompli. Scientific understandings of the world are always changing; science is an approach to understanding the world, not a set of settled conclusions).
Maybe I'm wrong. There are probably ways to investigate it. This essay isn't one of them.
The current "standard" p-value in many fields, of 5%, is arguably too high. consider throwing a double six in backgammon has probability of 3%. That means that, on some level, throwing 6-6 would be a valid scientific "proof" of ESP. (p-value<0.05) This of course is even before p-hacking.
A high p-value is basically externalizing part of the research cost onto other scientists and in the process creates a lot of false-positive "noise".