HN Explorer

Suggest HN: Studies based on small sample sizes shouldn't be on HN

31 points by syllogism ↗ HN
Reading about studies based on small sample sizes has in total misinformed me. I've been convinced of many surprising things based on a trust that the research was well conducted, and that published results are on average true. The building replication crisis has shown that this faith was misplaced, particularly in certain fields.

For instance, it's now clear that I would have believed more true things and fewer false things if I had not heard about a single result from social psychology. I'm still rooting out beliefs that I came to based on bad research. (Unfortunately my mind doesn't maintain an index of beliefs by source. I wish I could request this feature.)

I'm rightly bitter about this, and I see the problem continuing. For instance, this submission is worse than worthless: https://news.ycombinator.com/item?id=13144483 . It's impossible to tell what experiment was done without buying the paper, and on average experiments of this type have been shown to be of negative value: http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-winds-have-changed/ .

We're all worse off for reading and talking about these articles. I'd like to see HN add a heavy link penalty on all reports of research based on small samples, with an exception for good replication studies. Off the top of my head, n<10,000 might be a good threshold.

26 comments

[ 2.8 ms ] story [ 74.9 ms ] thread
sz4kerto
Good idea, but a hard limit like 10000 is not that good. Statistical methods don't require specific sample sizes. Cherry picking, incorrectly chosen methods are the problems, not the sample sizes, usually. A hard limit would suggest that anything above this sample size is credible, while that's very far from the truth. The reproducibility crisis is very much unrelated to sample sizes.
syllogism
Sure, you can't design a study badly and make it up on volume. — but there's a point at which the sample is so small we really don't need to look much closer.

For instance, I don't want to spend time figuring out whether an n<100 study is any good. It isn't.

stenl
Surely the required sample size depends on the effect size. You won't need n>100 to learn that elephants are larger than ants. The problem is in experiment design and interpretation generally, not sample size specifically.

That said, all of us who know how to read a scientific paper (and have institutional access to the journals) should take 5 mins and point out flaws when we see them.

jdietrich
You don't need a large sample to establish that beheading is fatal. Sample size is only one factor in statistical significance; often it's a fairly unimportant factor.
aaron695
> For instance, I don't want to spend time figuring out whether an n<100 study is any good. It isn't.

So an AIDs cure that cured 80 out of 90 men wouldn't be worth reporting on HN?

Studies that have >10,000 people I'd think are close to non existent, these are hugely expensive and I think most are well known.

I think it's a job for the scientific community to work out, I can't see moderators on HN doing it.

syllogism
Personally I wouldn't click on a link that was reporting on a potential AIDS cure that was tested on 80 or 90 people.
davelnewton
I'd click on a link on a report that cured 80 of 90 AIDS patients.

I think you've misunderstood the nature of preliminary clinical trials.

golemotron
That's a great idea for a journal aggregator. This is a news site.
syllogism
It's a news site that has editorial policies about which stories are worth paying attention to and which ones aren't.
vivekd
I disagree with this idea. For starters, sample sizes alone don't give you reliable results, even with large sample sizes you might have other issues that affect the results such as poor design, it may not be reproducible or (as is often the case) it may be that no one has tried to reproduce it yet.

Scientific research is often imperfect, as a practical matter due to lack of resources and many other reasons. But over time, as scientists peer review and reproduce each other's research, we come closer to scientific consensus regarding the matters of study.

Scientific studies, by their very nature, are often designed to inquire into matters of uncertainty. And no one study can completely claim to banish uncertainty and provide concrete and certain answers. The path to certainty requires the toil of reviewing and reproducing the results.

So I think anyone who puts forward a single scientific study to bolster a claim or point of view is making a very weak and unconvincing argument to anyone who understands science. Rather, we like scientific studies, not because they give us concrete neat solutions but because they offer interesting insights that may or not be conclusive.

I don't think it should be the job of moderators to determine the weight or conclusivity of scientific studies. I think that should be done by the community.

I also think the community would lose something if were to get rid of articles on interesting research just because that research was inconclusive. Research can be interesting and open up interesting possibilities even when it doesn't give us concrete truths to hold to.

syllogism
It sounds to me like you reject the premise that an intelligent reader who reads studies like this will have their beliefs diverge from the truth over time. I read your comment as saying, "Sure you shouldn't be convinced by a single study. But if you read enough of these, you'll converge towards the truth". I used to believe that, but I don't now.

Let's say some study is an increment of evidence about some hypothesis H. We start off with a prior on H, and we want to know how to update it after reading about a study.

What I've taken away from the replication crisis is that my average update was an order of magnitude too large. The appropriate evidentiary weighting of a study in social psychology is so small that I can't usefully consider it as any evidence at all. I'm better off ignoring it.

A little example for you. Let's say you read some study about the act of physically smiling causing a positive change of mood, due to a feedback effect. Before reading this study, you would've given this claim p=0.1. Possible, and you can make sense of the explanation, but no particular reason to believe it. After the study, you give it p=0.4. By no means settled, and you don't even necessarily believe it. But you're giving it much more credibility than before. Later it turns out that the study was actually on very few people and was p-hacked. It therefore fails to replicate. The appropriate post-reading belief level turns out to be around p=0.101. You were tricked.

If you repeat this pattern, you end up carrying around a bunch of surprising beliefs that have a quite random relationship to the truth. Some of the studies will be correct, but on average you'll be more wrong.

Reading these articles might be "interesting", on an entertainment-based definition, but they're the opposite of information. If you read enough of these, you'll make yourself superstitious.

davelnewton
Information isn't the opposite of information.

You more or less sum up my reaction to your suggestion with this (rather telling) phrase:

"you'll make yourself superstitious"

(Emphasis mine.)

syllogism
I don't think I understand your comment.

What I'm saying is that if you read these studies, you'll become less informed over time. You will believe fewer true things and more false things. I think it's reasonable to call that the opposite of information

davelnewton
I will? No, no I won't. And no, you don't become less-informed by taking in information. You become ill-informed if you choose to believe incomplete, partial, or misleading information. But that's your choice. Some people read a fairy tale and believe it--not the fairy tale's fault.

To me it seems like you're absolving yourself of responsibility.

If I don't understand the nature of a study then I take it for what it is: a report about results of a study I don't understand. I choose whether or not to believe it based on partial data. It doesn't make me believe it. I may believe it, or parts of it, or none of it. I make that decision.

DanBC
> But over time, as scientists peer review and reproduce each other's research, we come closer to scientific consensus regarding the matters of study.

That's the problem though. These small sometimes poorly run studies are conducted, then published, then a press release is issued. The press release sometimes over hypes the paper. Then newspapers sometimes over hype or misinterpret the press release, and rarely if ever link to the actual paper.

And then that news paper link ends up on HN, where people talk about their interpretations of a misinterpreted press release that hyped up the original paper.

It sucks.

Kenji
If you take the results of soft 'sciences' seriously you've already lost. Recognize submissions like the one you linked as noise and move on. Common sense goes a long way.
aaron695
I think this comment actually nails it.

Most of the current issues in published science are around the 'soft sciences'

They are notoriously hard to disprove I think because they are so wishy washy.

Maths, Physics, Chemistry can get you disproven in a hard way. There's a lot more incentive to make sure your study is valid.

gus_massa
There is a lot of dubious Physics that is very popular here.

My antifavorite is the Em-drive:

Most popular article: "EmDrive study officially published" https://news.ycombinator.com/item?id=12995125 (595 points, 21 days ago, 432 comments)

(My second choice is Cold fusion / LENR.)

This has slightly less votes that the most popular stories about colossal experiments:

"Second Gravitational Wave Detected at LIGO" https://news.ycombinator.com/item?id=11910700 (621 points, 178 days ago, 175 comments)

"Higgs Boson Explained by Cartoon" https://news.ycombinator.com/item?id=4193590 (646 points, 1622 days ago, 128 comments)

The problem is that most people like nice stories, and a massless thruster that can make space travel much faster is interesting. And it's much easier to explain than why we need the Higgs's boson or gravitational waves.

The two later have no possible application in sight. They are interesting results, but it's not clear how they will change your life. (I'm optimistic, let's see what happens in 30 years.)

The Em-drive looks more promising. Cheap and fast space travel soon, for some definition of soon. It's easier to explain, in particular because the theoretical explanation is worse than the experimental setup. No one tries to explain something about gauge invariance of curvature tensors.

Kenji
As far as I know, the EmDrive is still considered controversial and it is not certain whether and how it works. It is interesting because it could cause a revolution to how we view impulse, so it might be a colossal thing just as much as it might be complete nonsense. So I think the comparison is disingenous.
gus_massa
It's difficult to explain in a short comment, but the Em-drive is as controversial as the yeti.

If it were true, then it breaks the current laws of physics. In particular the conservation of momentum and energy. There are some bogus explanations that try to make it compatible with the current laws of Physics, but they are nonsense.

Breaking the current laws of Physics is not bad if you have a good experiment. The problem is that they are measuring a very small force that is very close to the experimental error. And there are many other forces that are difficult to calculate and may explain the force they are measuring.

ebbv
Seems to me like you haven't learned the right lesson. Small sample sizes can still yield useful results. It depends on the nature of the study.

Instead of closing yourself (and the rest of us) based on a fairly arbitrary criteria, you're going to have to use critical thinking. That should be applied to everything you read not just studies.

brudgers
Generally, I think scientific studies are uninformative as news because:

1. Most people are not close enough to the science to form an informed technical opinion.

2. The scientific conclusions of most studies are impractical to implement.

John Oliver's analysis is both sophisticated and entertaining: https://www.youtube.com/watch?v=0Rnq1NpHdmw

gus_massa
In an ideal world I agree, but it's very difficult to implement. I'd ask to not upvote stories with a small sample size.

In many articles consider writing a small rant about the methodological problems. Lucky most of the times I just found a comment by tokenadult that is better than my draft and has link to support the claims. It's a pity that he is not writing them recently.

Try writing a civil comment explaining that the study is dubious due to the small sample size. Most press coverage skip this detail, so you have to look at the research article to copy it. You'll get at least 1 upvote.

Don't worry too much. Just eat some chocolate https://news.ycombinator.com/item?id=9714985

drbigglesworth
The focus on sample size is misplaced: small sample sizes frequently have sufficient power to be statistically significant.

The error you are making is in placing too much faith in a single study, and possibly in the journalists/bloggers who report on a single study. I have lost count of the number of times I've read a press release or news article that completely misrepresents the findings of a study (i.e., the error is in the reporting, not the article). It's also frequently forgotten that journal articles are written to communicate findings to peers, not the larger public. There are nuances that are understood within a community that are unknown when one is coming from a different background.

Until there is a body of work on a phenomenon, perhaps framed by a review article or two, it's best to regard single reports (regardless of perceived quality) as interesting curios but nothing more.