> Wish the study was not behind a paywall but...5 human subjects?
Its an experimental study, the number of subjects isn't really that important. I can't access the full text, either, but I assume they did an ABAC test pattern (control, treatment caffeine, control, treatment caffeine + bright light) or something similar with all 5 subjects simultaneously.
Generally speaking, you really only need many participants for field studies, e.g. situations where you cannot control most variables beside treatment itself. The assumption is that the Law of large numbers takes care of equal distribution of those confounding variables between treatment group and control group.
In the part that can be accessed (2nd link) they mention it's a within-subject setup so yeah "with all 5 subjects simultaneously" is correct.
within-subject basically means each subject gets all the treatments as opposed to between-subject which is the typical A/B-test setup.
"Its an experimental study, the number of subjects isn't really that important."
I'd be interested to understand why this is? My logical reaction would be that it's always important - as a crude example, surely doing an experiment on every single human on Earth would give you much more accurate results that on say 100 people, because you'd be sure to have covered all the innate variables that exist when experimenting with humans? (different metabolisms, etc)
I have a feeling this test is not so much about accuracy as it is tendency. I.e. they avoid entirely specific projections, instead they want to see if the effect can be observed even once and thus have groundwork for further investigation.
Depends on what kind of generality of statement you're trying to make. Here, the generality might be a cause and effect one in which case you're attempting to generalize over possible future treatments and you attack detractors who might yell "that was a fluke!" or "it wasn't the caffeine, but instead the presence of the doctor!". To do this, you design an experiment which carefully controls for all expected irrelevant interactions and then show a response which is significantly different from random variation.
You end up limited, as you note, to your population. 5 people won't defeat detractors who believe that this effect is limited within some, e.g., metabolic profile but it ought to give them serious food for thought as to how wide the affected metabolic profile actually is.
If these 5 volunteers were chosen at random, then the potential generality of effect can still be large as a detractor would be fighting, at best, with the notion that the 5 chosen were circumstantially susceptible to this effect (as compared to a study of convenience where one might believe that "college students" or "hospital volunteers" are especially susceptible).
So, in a certain sense, testing every human on earth improves the power of the statement you can make (not really its "accuracy" though maybe its "precision", in a sense), but in many other ways that may be too expensive for the kind of result the author seeks.
>> "Its an experimental study, the number of subjects isn't really that important."
I'd be interested to understand why this is?
I replied to cossatot below in more detail. The short version: In studies like this one, N isn't 5, but humans(e.g. the original N) x treatment repetitions x measurement points.
You are right that it would be dangerous to ignore it, that wasn't what I implied.
If the 5 subjects capture all the relevant differences among humans then sure. But, for example, what if they inadvertently selected only heavy coffee drinkers? That being said, I think the strength of their paper might be the molecular experiments they did on cultured cells. The media articles lead with the human experiments because that might be more relatable.
No, this is wrong. With small sample sizes you may get a statistically significant result, but it still might not be a real result and might not be reproducible. This is a major issue in science today and why a lot of studies can't be replicated.
Statistically significant means statistically significant and is independent of sample size. If your p-value is less than 0.01, then there's less than a 1% chance that the pattern you're seeing is due to random fluctuations of the variable itself that you cannot predict.
The problem is that the statistical model (in my field we do a lot of ANOVA and t-tests, along with the occasional chi-square) can only account for what you model. So there could be some kind of systematic error that influences your results in a fashion that is not modeled by the statistics. Having a large-N study makes it harder to have that systematic error (but not impossible - as an example: look at complaints about how much psychological and cognitive science research is only on WEIRD subjects - western, educated, industrial, rich, developed).
The other problem, of course, is that one time in a hundred, you'll get a p < 0.01 significant result by chance. Which is a lot in the long run. Worse, you can induce type two errors by running hundreds of trials (or testing hundreds of variables) and not accounting for that - just pick the one thing that had significant results on a single test. This approach is unscrupulous, but not unheard of in academic circles where you need to publish tons of work to get promoted.
> If your p-value is less than 0.01, then there's less than a 1% chance that the pattern you're seeing is due to random fluctuations of the variable itself that you cannot predict.
This is a dangerous misinterpretation of p values, which cannot provide that kind of information. A p value assumes the pattern is due to random fluctuations, and asks how common this kind of fluctuation is.
That's actually a more articulate, but redundant codicil to the argument I made in the rest of the post. Multiple tests will result in significance at some alpha, since you just have to test enough times to get a lucky test. There are techniques (outlined in your link), for addressing that, but the central point I think is still cogent.
If you have a test of significance that results in p < 0.01, there's a one percent chance that you're rejecting the null hypothesis due to normally-distributed variation in your data. The base rate fallacy is more about interpreting what that p = 0.01 means, and why systematic bias is important to worry about - if you're testing cancer drugs, you don't want to test them on people who don't have cancer.
> If you have a test of significance that results in p < 0.01, there's a one percent chance that you're rejecting the null hypothesis due to normally-distributed variation in your data.
No, this is absolutely not true. If p < 0.01, then if there is no systematic effect and only normally-distributed variation, you would see this effect 1% of the time. That is, the p is P(data | null is true), and not P(null is true | data). You cannot invert the conditional.
In the extreme case, when the null is true for every test, you will get significant results for 5% of them. Thus 100% of your statistically significant results are false positives, no matter how small their p values.
Given that we do not know what fraction of the time the null is true, we cannot know the chance that we're rejecting the null falsely. But it is invariably larger than p.
This misunderstanding is why scientists routinely overestimate the strength of their evidence and discount the possibility that their results may be flukes.
> No, this is wrong. With small sample sizes you may get a statistically significant result, but it still might not be a real result and might not be reproducible. This is a major issue in science today and why a lot of studies can't be replicated.
Reproducability indeed is a major problem, but looking at statistical significance alone isn't the cure (especially if applied a posterior).
We should rather look at effect sizes and robust study designs.
In fact, modern studies aiming for causality often calculate the population size needed for statistical significance beforehand. It's a standard formula in most textbooks. You only need the expected effect size and then can calculate the population needed to guarantee significance.
This isn't correct. The statistical power of n=5 humans is quite low.
It is, however, a good example of the "law of small numbers" of Tversky and Kahneman, a cognitive bias in which people believe that the law of large numbers applies to small numbers as well.
See Tversky and Kahneman 1971, or Kahneman's fantastic recent book Thinking, Fast and Slow which is an excellent guide to how our cognitive biases can wrongly influence our thinking.
> This isn't correct. The statistical power of n=5 humans is quite low.
A few points are important to consider.
First, I was only talking about experimental studies searching causal relationships. There are other possible designs, for example field studies (e.g. "school district A gets the new math curriculum, school district B the old one. Which one fares better?") or simple population observations ("people playing golf live longer than the average population."). Each design has advantages and disadvantages regarding generality of the statement one can make, and for each one different statistical considerations apply.
Second, the statistical power does not rely on a high population alone, as that (more or less) only affects the significance tests. Much more important is the effect size. If you can measure a large effect (as this study did), it's pretty hard not to reach significance anyway.
Third, from a statistical point of view, the population isn't 5, but much higher.
Let me explain:
There are certain kinds of treatments whose effect is reversable. Caffeine intake is an good example: Once you stop taking caffeine, the effect recedes. While designing the study, you can use that property. One common way is an ABAB design, where A is a phase with treatment and B is a phase without. You can chain as much AB pairs as time permits, and additionally you can measure multiple times per phase. Statistically, the population now is real_humans x number_of_phases x measure_points_per_phase.
There's the problem that with 5 events you can't know if it was a fluke and also the subpopulation that you are sampling from.
I think for instance how the drug Naltrexone seems to work very well for treating alcoholism in Asians and poorly in Blacks. If you don't take this into account whatever result you get is going to indicate that the drug is too effective or not effective enough.
I'm quite aware of P-values :) I neither mentioned posterior calculation of probabilites nor talked about correlation studies, so I'm not seeing your point in linking to Gelman's article?
If only caffeine kept me up during the day nowadays. I cut back to two mugs a day and am seriously dragging. Although, anecdotally, I have noticed drinking a substantial amount of water does keep the lights on. I wonder if it similarity does so in the evening?
Caffeine is like a purchase on a adrenaline system credit card, and your body's system is probably in huge debt - might take a while to regain composure.
In my case it takes a couple of weeks. I've had to go cold turkey once in my 20s and since then i keep an eye on my intake. In some cases (crunch time, onsite, etc) I binge and then it takes a while to readjust, but never less than 3-4 days. I'm slightly obese as well so that doesn't help either, but I've noticed that the solution when I feel slow is usually "more and more water".
Caffeine on net suppresses stress systems and improves many cellular level functions. Reams of epidemiological data strongly indicate the equivalent of four or five cups of coffee a day is good for you.
If caffeine makes you jittery or causes other signs of increased adrenaline that means you are probably in poor health. Note that it's best to take it with food or a bit of sugar and taking it while hungry or low on glycogen can cause a bit of stress.
Coffee is nothing but calming for me and I have zero withdrawal symptoms if I go a day or two without it.
"anecdotally, I have noticed drinking a substantial amount of water does keep the lights on"
I only drink water in the morning, 1/2 - 1 litre. This replaces the loss during the night and I found out that I don't need any coffee after that any more (provided I got enough sleep).
Similar things happen during the daytime: if I don't drink at work, I crawl home completely run down. With lots of water, I leave the office in good mood and after a few minutes, I am back to full capacity again.
All this raised many doubts in the effects of caffeine in the morning. Overrated, I think. Its the liquid, not the coffee that does most of it (most, not all).
I've had the same issue and here is what worked for me. You might want to work on cutting back to zero caffeine, and work on drinking plenty of water and doing general diet improvement. If you're eating and drinking lots of sugar that can give you uneven energy levels throughout the day. Some grains like rice or toast in the morning for energy. Eating meals and/or snacks every 2-3 hours, and getting away from your screen for a mental break in the afternoon, combined with a healthy diet and some exercise cured me of my late afternoon sleepiness problem.
This matches my experience. Kicking caffeine was a pain, and ditto for refined carbs. But off of those, my energy levels are very even throughout the day. I also have automated my house lights [1] so that they mimic a day-night cycle in brightness and color. Afternoon sleepiness is not a problem for me anymore.
I recently just quit caffeine (went down one drink a day for a week, then quit 7 days ago.) It's been a process getting my energy back.
I have noticed that food affects me MUCH more than it used to. If I don't pay attention to what I eat now (or when I'm not eating, for that matter), I will end up feeling like crap.
It's difficult because food isn't something I've focused on for years, thanks to caffeine masking my low energy.
This is really awesome to hear. For years and years I have thought I was probably just weird or anxious because if I have caffeine later than about noon I have serious trouble getting to sleep that night (bedtime around 10pm). It's nice to know that there could possibly be a reasonable physical cause for this and it's not just all in my head.
The majority of the human race lack the ability to digest milk (more specifically lactose sugar) once they are passed weaning.
It was a relatively recent mutation in some human populations (Northern Europe et al) that allows them to continue digesting lactose into adulthood.
They think this was because in Northern Climates where growing crops was more difficult milk and dairy products where a much larger source of calories than in other places as well as an excellent way of storing a valuable food product (Cheese however has relatively low levels of lactose at around ~1% compared to regular milk at ~5%).
Yeah, evolution is definitely a cycle, but it can also be a state (mankind's current state of evolution). Maybe I could have been more clear about that.
Not because of water, but because of other factors. I'm not trying to say that we have to go back to the savannas and live as hunter gatherers. I just think that the more we learn about ourselves, the most likely it looks like we are better off with putting in our bodies only the minimum that's necessary to sustain life.
I wonder if amphetamines and similar substances work in the same way. On speed for instance, people can stay awake and party for days on end without sleep; that would result in a tremendous push on the circadian clock, though I have an inkling it's not simple subtraction and addition.
Taking "speeds" will have a huge impact on your body. Most people that abuse of such substance will find themselves in some sort of chemical induced depression. Your body will stop giving you pleasure responses, nothing will seem worth doing or living.
How much coffee do you drink in a day? If you drink a lot of coffee you will develop tolerance, so you would need to drink more in order to see this effect.
I can do the same, and I'm an extremely irregular coffee drinker. A lot of people in a lot of cultures regularly have coffee after dinner to aid digestion. I think whether you stay up or not after coffee drinking is largely a matter of expectation: http://www.bris.ac.uk/news/2010/7051.html "Coffee consumption unrelated to alertness" [2010]
sample size: 379
People just think that they're better reporters of their own conscious experience than they are.
Some people aren't effected by stimulants the same way others are. For example, generally with ADD/ADHD stimulants help calm the person down where with other in general people it would wire them up. Personally, I have to take enough caffeine to have an irregular heartbeat before I notice any effect besides going to the bathroom more often.
69 comments
[ 1.0 ms ] story [ 138 ms ] threadI have a dentist on 9/19/2015. I don't want to fuck that up, LOL.
http://stm.sciencemag.org/content/7/305/305ra146
http://www.colorado.edu/news/releases/2015/09/16/caffeine-ni...
Its an experimental study, the number of subjects isn't really that important. I can't access the full text, either, but I assume they did an ABAC test pattern (control, treatment caffeine, control, treatment caffeine + bright light) or something similar with all 5 subjects simultaneously.
Generally speaking, you really only need many participants for field studies, e.g. situations where you cannot control most variables beside treatment itself. The assumption is that the Law of large numbers takes care of equal distribution of those confounding variables between treatment group and control group.
I'd be interested to understand why this is? My logical reaction would be that it's always important - as a crude example, surely doing an experiment on every single human on Earth would give you much more accurate results that on say 100 people, because you'd be sure to have covered all the innate variables that exist when experimenting with humans? (different metabolisms, etc)
You end up limited, as you note, to your population. 5 people won't defeat detractors who believe that this effect is limited within some, e.g., metabolic profile but it ought to give them serious food for thought as to how wide the affected metabolic profile actually is.
If these 5 volunteers were chosen at random, then the potential generality of effect can still be large as a detractor would be fighting, at best, with the notion that the 5 chosen were circumstantially susceptible to this effect (as compared to a study of convenience where one might believe that "college students" or "hospital volunteers" are especially susceptible).
So, in a certain sense, testing every human on earth improves the power of the statement you can make (not really its "accuracy" though maybe its "precision", in a sense), but in many other ways that may be too expensive for the kind of result the author seeks.
I'd be interested to understand why this is?
I replied to cossatot below in more detail. The short version: In studies like this one, N isn't 5, but humans(e.g. the original N) x treatment repetitions x measurement points.
You are right that it would be dangerous to ignore it, that wasn't what I implied.
The problem is that the statistical model (in my field we do a lot of ANOVA and t-tests, along with the occasional chi-square) can only account for what you model. So there could be some kind of systematic error that influences your results in a fashion that is not modeled by the statistics. Having a large-N study makes it harder to have that systematic error (but not impossible - as an example: look at complaints about how much psychological and cognitive science research is only on WEIRD subjects - western, educated, industrial, rich, developed).
The other problem, of course, is that one time in a hundred, you'll get a p < 0.01 significant result by chance. Which is a lot in the long run. Worse, you can induce type two errors by running hundreds of trials (or testing hundreds of variables) and not accounting for that - just pick the one thing that had significant results on a single test. This approach is unscrupulous, but not unheard of in academic circles where you need to publish tons of work to get promoted.
This is a dangerous misinterpretation of p values, which cannot provide that kind of information. A p value assumes the pattern is due to random fluctuations, and asks how common this kind of fluctuation is.
Typically the chance the result is a random fluctuation is much higher; for examples, see http://www.statisticsdonewrong.com/p-value.html
If you have a test of significance that results in p < 0.01, there's a one percent chance that you're rejecting the null hypothesis due to normally-distributed variation in your data. The base rate fallacy is more about interpreting what that p = 0.01 means, and why systematic bias is important to worry about - if you're testing cancer drugs, you don't want to test them on people who don't have cancer.
No, this is absolutely not true. If p < 0.01, then if there is no systematic effect and only normally-distributed variation, you would see this effect 1% of the time. That is, the p is P(data | null is true), and not P(null is true | data). You cannot invert the conditional.
In the extreme case, when the null is true for every test, you will get significant results for 5% of them. Thus 100% of your statistically significant results are false positives, no matter how small their p values.
Given that we do not know what fraction of the time the null is true, we cannot know the chance that we're rejecting the null falsely. But it is invariably larger than p.
This misunderstanding is why scientists routinely overestimate the strength of their evidence and discount the possibility that their results may be flukes.
(Source: I wrote the link provided earlier. Also, the discussion leading to table 1 in this paper is good http://journals.plos.org/plosmedicine/article?id=10.1371/jou...)
Reproducability indeed is a major problem, but looking at statistical significance alone isn't the cure (especially if applied a posterior).
We should rather look at effect sizes and robust study designs.
In fact, modern studies aiming for causality often calculate the population size needed for statistical significance beforehand. It's a standard formula in most textbooks. You only need the expected effect size and then can calculate the population needed to guarantee significance.
It is, however, a good example of the "law of small numbers" of Tversky and Kahneman, a cognitive bias in which people believe that the law of large numbers applies to small numbers as well.
See Tversky and Kahneman 1971, or Kahneman's fantastic recent book Thinking, Fast and Slow which is an excellent guide to how our cognitive biases can wrongly influence our thinking.
A few points are important to consider.
First, I was only talking about experimental studies searching causal relationships. There are other possible designs, for example field studies (e.g. "school district A gets the new math curriculum, school district B the old one. Which one fares better?") or simple population observations ("people playing golf live longer than the average population."). Each design has advantages and disadvantages regarding generality of the statement one can make, and for each one different statistical considerations apply.
Second, the statistical power does not rely on a high population alone, as that (more or less) only affects the significance tests. Much more important is the effect size. If you can measure a large effect (as this study did), it's pretty hard not to reach significance anyway.
Third, from a statistical point of view, the population isn't 5, but much higher.
Let me explain: There are certain kinds of treatments whose effect is reversable. Caffeine intake is an good example: Once you stop taking caffeine, the effect recedes. While designing the study, you can use that property. One common way is an ABAB design, where A is a phase with treatment and B is a phase without. You can chain as much AB pairs as time permits, and additionally you can measure multiple times per phase. Statistically, the population now is real_humans x number_of_phases x measure_points_per_phase.
I think for instance how the drug Naltrexone seems to work very well for treating alcoholism in Asians and poorly in Blacks. If you don't take this into account whatever result you get is going to indicate that the drug is too effective or not effective enough.
Not sure if it's accurate or not, but it sounds plausible to me.
But the lack of energy is real and persistent. My body and routine are both recovering.
If caffeine makes you jittery or causes other signs of increased adrenaline that means you are probably in poor health. Note that it's best to take it with food or a bit of sugar and taking it while hungry or low on glycogen can cause a bit of stress.
Coffee is nothing but calming for me and I have zero withdrawal symptoms if I go a day or two without it.
Or you're sensitive to caffeine.
(I'm hellbanned, this comment would probably not be hidden otherwise.)
I only drink water in the morning, 1/2 - 1 litre. This replaces the loss during the night and I found out that I don't need any coffee after that any more (provided I got enough sleep). Similar things happen during the daytime: if I don't drink at work, I crawl home completely run down. With lots of water, I leave the office in good mood and after a few minutes, I am back to full capacity again.
All this raised many doubts in the effects of caffeine in the morning. Overrated, I think. Its the liquid, not the coffee that does most of it (most, not all).
[1] https://github.com/wpietri/sunrise
I have noticed that food affects me MUCH more than it used to. If I don't pay attention to what I eat now (or when I'm not eating, for that matter), I will end up feeling like crap.
It's difficult because food isn't something I've focused on for years, thanks to caffeine masking my low energy.
>double espresso three hours before bedtime
>double espresso
>three hours before bedtime
Somebody get the CNN on the phone, this is groundbreaking work right here!!
The majority of the human race lack the ability to digest milk (more specifically lactose sugar) once they are passed weaning.
It was a relatively recent mutation in some human populations (Northern Europe et al) that allows them to continue digesting lactose into adulthood.
They think this was because in Northern Climates where growing crops was more difficult milk and dairy products where a much larger source of calories than in other places as well as an excellent way of storing a valuable food product (Cheese however has relatively low levels of lactose at around ~1% compared to regular milk at ~5%).
https://en.wikipedia.org/wiki/Lactase_persistence
"We propose that humans have evolved to withstand energy crises by decreasing their body size" http://www.nature.com/pr/journal/v64/n1/abs/pr2008135a.html
:humans evolved to modulate social behaviors like mating and parenting in response to specific environmental cues: http://link.springer.com/chapter/10.1007%2F978-1-4612-3760-0...
After just few seconds of search.. So apparently some scientists didn't get the memo.
Hopefully some day you will also evolve to be less nitpicky.
A sample size of 5 is way too low.
If 15% of the population does not have sleep effected by caffeine at all, there is a good chance that this study missed people like me completely.
sample size: 379
People just think that they're better reporters of their own conscious experience than they are.