Tragically, I bet this is all too common. Wansink's mistake was blogging about it.
This emblematic of a larger problem with how science is practiced: the obsessive focus on p-value thresholds leads to irrational practices like trawling data for interesting "findings."
But on a certain level Wansink was right: a data set is not completely worthless if it showed a null result. So we need to start thinking about how to communicate the value of data even when the null is not rejected.
One way to do this is to encourage widespread sharing of data sets regardless of the outcome of the experiment. Maybe for a given study the data did not show a definitive result - but does the data point to potential future paths of study? Maybe another researcher could get ideas for new experiments.
> a data set is not completely worthless if it showed a null result.
Why is a null result worthless? I don't understand. It goes against my common sense that paying more for food doesn't make you eat more. The fact that this is not the case is valuable knowledge to me. I understand that it's not as glamorous as "paying more makes you eat less!", but it's still valuable knowledge that should be published.
> a data set is not completely worthless if it showed a null result.
A null result is not worthless. A data set is not worthless if it showed a null result. Nobody is claiming a null result is worthless. Even the original article gives a good reason for publishing a null result.
Null results are "worthless" only in the sense that they're harder, if not impossible, to publish, and do less for your career.
As a former neuroscientist, I had a unicorn data set of intracranial EEG data, and we'd spent so much time collecting it, that we were determined to find something. I left grad school before we found anything of interest. I believe my former prof eventually published something on it, but I analyzed the shit out of it, even knowing I was fishing, because so much time would have been lost to not use it.
Fishing like this was one of the reasons I left. The pursuit of knowledge and the pursuit of your career don't align often enough.
To me, what's unusual is not that Wansink pushed to keep analyzing the data, but that he was called out for it. Everyone seemed to be doing it when I was in academia.
I feel the same way about current practice research, but find that industry standards are even more abysmal in terms of fishing. What field are you in now that you find an acceptable level of rigor?
Even the post-hoc analysis isn't invalid as a way of discovering potential avenues of future research. But you really have to run new experiments designed to test your new hypotheses.
> a data set is not completely worthless if it showed a null result
As the other commenters have pointed out, the fact that it showed a null result is not really relevant to future utility.
A better way of phrasing it might be "A data set is not completely worthless after it has been used to the test the hypotheses for which it was gathered."
As to how it is useful, I think a sound summary would be "a data set is _only_ useful for validating (or invalidating) those hypotheses for which it is gathered, but it may be useful for deriving new hypotheses for which new data can be gathered to validate."
Or more negatively as "a data set is not useful for validating or invalidating hypotheses invented after the data set is seen, but it may be useful for deriving those hypotheses and leading to further research."
To some very limited extent, if a mechanism of action is established (which is to say there is an external reason to infer a correlation) then cross-validation with an existing data set may be able to derive the parameters of that correlation.
I think what's needed is a strategy for making actual data analysis an easier and more visible component of the peer-review and publishing process. Submissions are typically sent to three people for review, and the decision to accept or reject is made based on the feedback or criticism given. If the reviewers are busy, overworked professors, they may not have the time to really perform a deep dive into the data and conduct an independent analysis. Furthermore, it's likely these submissions don't include enough data for a proper statistical review.
This is the 21st century, where most if not all reputable journals have an online presence and submission portal. It would be great if authors had to upload their anonymized data sets in a common format (.csv?) that can easily be imported into statistical analysis packages (ex: Minitab, read into R, etc). Journals provide analysis and test recommendations, reviewers run the tests, upload their independent analysis results as part of the review process. The idea here is to produce something like an auditable paper trail.
There has to be some sort of solution for this problem, especially in 2017. Hiding behind shitty (or worse, deceitful) data analysis shouldn't be possible.
Inside universities, it does not seem realistic that this will lower people's opinions on the value of social science statistical research, or in the related fields of psychology (esp. management/organisational psychology), or even all of the humanities.
That opinion is already "reject it all".
Even when it comes to medicine that opinion is strongly represented. A few people even ascribe the placebo effect to massive and widespread statistical error.
The Last Psychiatrist talked about this sometimes. The problem is not just that the original studies need to be rescinded, or discredited going forward. The problem is what do you do about everything that cited them, that took the knowledge in the faulty papers and used it as a foundation for more knowledge? The faulty ideas in the paper have permeated out to researchers and (sometimes) the broader culture generally, what do you do about that?
The article makes Wansink seem negligent and/or incompetent. Also, the embedded "CBS This Morning" video is cringeworthy. It appears that the recommendations in his book amount to "study normal weight people; do what they do." Isn't that fraught with survivorship bias? Do the recommendations in his book control for people that have the same habits but still end up overweight?
Anecdote: my household is a family of four. Myself and my wife, and our two teenage kids, daughter and son. Since we live together, the answers to the 10 questions on http://www.slimbydesign.com/get-scored/ are the same for the household. My son and I are normal weight. My wife and daughter are overweight. Conclusion: weight has very little to do with the design of your kitchen. Caveat: this conclusion has not been peer reviewed and is not scientifically sound.
Wansink strikes me as an extremely bright, creative researcher. His problem seems to be that he didn't collaborate with people who were strong methodologists.
If anything, he reminds me of a lot of brilliant researchers, who are too confident in their methodological prowess. (And this is the outcome that should have been avoided through collaboration).
It is, but it's not a completely invalid starting point. Even better is to study people who were once fat, but lost weight and keep it off. This is what the National Weight Control Registry does. It's similar to studies that have tried to pin down why marriages fail. Most studies focus on what went wrong in failed marriages, but some of the most helpful suggestions have come from what people who stay married do.
> My son and I are normal weight. My wife and daughter are overweight.
Do your wife and daughter eat the same portions as you and your son? It's a common complaint of women, who are on average shorter than men and therefore need fewer calories, that they can't eat as much as their significant others.
'Why did peer review not catch this? “Because peer review doesn’t do this,” Heathers told Ars. The point of peer review has always been for fellow scientists to judge whether a paper is of reasonable quality; reviewers aren't expected to perform an independent analysis of the data. ...In fact, without open data—something that’s historically been hit-or-miss—it would be impossible for peer reviewers to validate any numbers.'
It doesn't seem unreasonable to think that the peer review process should include a statistician or someone who can review the statistics.
"Wansink added another note to his blog post, acknowledging the problems and announcing that a statistician would be redoing the analyses."
With all of the issues lately around replication of studies it seems like having a statistician review the math should be step one of any peer-review process.
Interestingly, a few years ago Nature planned to include statisticians...
> as consultants on the rare papers in which there are serious issues about the statistical techniques used, at the editor's discretion and based on the referees' suggestions.
I think the crux is that editors already have enough trouble finding reviewers, and see getting a statistician as often unreasonable. It's too bad, because the nature of statistical issues is that people tend to be unaware of them until a statistician (used broadly) points them out.
> In fact, without open data ... it would be impossible for peer reviewers to validate any numbers
Even this is not sufficient -- the fact is that this is a methodological statistical error, not a mathematical statistical error.
The mistake is thinking that a dataset alone can yield data supporting a hypothesis derived from that dataset, rather than deriving a testable hypothesis from an existing dataset, and then gathering new data to validate.
I would say the only way that you could publish results derived from an existing data source is if you also published all the null results you got along the way when examining the data set; but this is not really a feasible thing to print in a publication; either the list would be very long and thus it would be clear that some sort of p-hacking was involved, or the list would be short and accidental or intentional omission would be suspected.
This dramatically reduces the amount of useful results that can be squeezed from a dataset, which is unfortunate, as many of them are hard to gather in the first place. It might be necessary to protect these datasets better -- to restrict access conditional on specifying the hypothesis being tested with the requirement that all results, including null results, be recorded, even if only in summary form if the results are not interesting enough to be accepted for publication or to be worth the effort of composing into a quality research paper.
So what if you split the dataset to say 70-30, tried to find hypotheses in the the 70% dataset, then validate them with the 30% dataset. This is the process I use often with machine learning, would it also apply here?
Cross-validation (and the bootstrap, a related tool) is a useful technique. It's most useful when deriving the parameters for the model, though, rather than for confirming the validity of the model. Basically, if you have a strong prior reason to believe that the model is basically valid, but you need to know the parameters that fit the data.
The more hypotheses that are tested using the 30% "test" dataset, the more that dataset inadvertently becomes a "training" dataset, by progressively invalidating models. For applications with real-world significance, you usually have to accept that the "test" dataset will expire after some number of experiments into the "training" dataset, and gather a new test dataset to prevent overfitting. Hastie, Tibshirani, and Friedman[1], which I heartily recommend, cover this extensively in chapter 7.
That's really good to know, I'm still just beginning to learn about data analytics, and the link you provided is already proving to be an interesting read, so thanks!
> Even this is not sufficient -- the fact is that this is a methodological statistical error, not a mathematical statistical error.
That said, Wansink's papers had plenty of mathematical statistical errors that were evident without access to the original data, as documented in the mentioned "Statistical Heartburn" preprint: https://peerj.com/preprints/2748.pdf
Very much so -- I had only read the first part of the article when I wrote this reply, having read the whole thing now (and the linked pizza paper) I'm more than a little horrified at what can, in the most optimistic case, be described as extreme carelessness.
I have a pre-existing bias against papers relating to nutrition science, and this looks worse even than I expect. I would love to see a treatment that looks at especially noteworthy research in the area that passes a rigorous methodological review, much less has been replicated with any reliability.
Feel like we really approach the whole dieting process incorrectly. People eat to fulfill a need. They are either hungry and eat to no longer feel hungry or they eat because it produces positive feelings (whether this is to negate negative feelings or just to enhance your already good mood).
Hunger is your body signalling your brain that you need nutrients. If you eat foods that cover all your nutrient requirements for a low number of calories and your body is taking up these nutrients properly, then you should have no problems. Thus it makes sense to inform people which foods they need to be eating as well as educate people on which disorders can cause poor nutrient uptake.
If you suffer from stress and you eat high calorie foods to feel better, changing your diet is likely going to exacerbate your stress, not make you feel better. I feel this is the case for many who are overweight.. stress management strategies should be the first priority as you won't be able to handle changing your diet for the long term until you get your stress under control.
This doesn't even begin to get into the effects different macronutrients have on you. It's a complicated subject that has reached religion-like levels of zealotry.
Yes, but if we started acknowledging that everyone's different and functions on their own level, people will stop giving statisticians money! Who wants to live in that world?
If the academic world let scientists publish their validation of the null hypothesis, then they wouldn't go out of their way to "deep dive" into data to look for other hypothesis to prove!
PSA I always post into weight threads: Every legitimate long term study of non surgical weight loss shows that it doesn't happen for the vast, vast majority of people.
1) ["In controlled settings, participants who remain in weight loss programs usually lose approximately 10% of their weight. However, one third to two thirds of the weight is regained within 1 year, and almost all is regained within 5 years. "](http://www.ncbi.nlm.nih.gov/pubmed/1580453)
2) Giant meta study of long term weight loss: ["Five years after completing structured weight-loss programs, the average individual maintained a weight loss of >3% of initial body weight."](http://ajcn.nutrition.org/content/74/5/579.full)
3) Less Scientific: [Weight Watcher's Failure - "about two out of a thousand Weight Watchers participants who reached goal weight stayed there for more than five years."](https://fatfu.wordpress.com/2008/01/24/weight-watchers/)
4) [The reason why it's impossible seems to be that although calories in < calories out works, the body of a fat person makes it extremely difficult psychologically to eat less.](http://www.nytimes.com/2012/01/01/magazine/tara-parker-pope-...) This is borne out by the above data.
Moreover, you won't find any reputable study on the web where the average person lost 10%+ of their body weight and kept it off for five years. Not even one.
> Moreover, you won't find any reputable study on the web where the average person lost 10%+ of their body weight and kept it off for five years. Not even one.
Was going to link to the National Weight Control Registry, thanks! I'll just add that all those studies in GP seem to prove is that A) weight loss programs (especially fad diets) don't work and B) it is a psychological issue. There are plenty of people on MFP, /r/loseit or just counting calories themselves that have successfully kept off weight for years. I'm one of them.
Many times I'll read a story of someone who lost weight and kept it off. And then they detail their pre weight-loss diet and I think, well of course you were overweight. You were inactive and had a terrible diet (sugary drinks, processed foods, etc). You started getting some exercise and learned a few things about nutrition and the weight fell off.
But then there are others who seem to do everything right and are over weight in spite of that.
For example. I was never overweight as a kid and relatively active. In college and for the start of my career, I stopped being active and my diet was awful (e.g. I thought a large Jamba Juice smoothie and a carrot cake was a healthy breakfast choice). My weight ballooned up to almost 190 lbs (20+ lbs overweight), my blood pressure went up, I started having rosacea.
I started running and fixed my diet. Quickly my weight dropped down to 150 and I've kept it in the 140-150 range for over a decade. The other health issues cleared up as well. But it wasn't hard work for me. Being thin is my natural state if you will, and I had to do everything wrong to stay overweight.
My wife meanwhile continues to struggle with her weight. She's successfully lost weight through extremely diligent calorie counting, but after a year or so she starts to put it back on. I have never counted calories. Our diets are similar (in kind, not quantity of course, she eats much less than me). She is active, but not quite as active as me. So similar diet and life styles, but my weight stays off and hers does not.
Hereditarily, no one in my family is over weight. There is obesity on both sides of her heredity.
And I see this playing out in our kids. My son has an athletic build and will probably never have weight issues. My daughter takes after her mom and it will take a life time of diligence for her to remain at a healthy weight.
It seems that some people are optimized for famine, and some for feast. :-(
Obviously there are a lot of factors involved in the growing obesity crises. But I feel for people who struggle with their weight despite doing all the right things, I really do.
The food tastes too damn good! I've only been overweight because of binging and poor eating. I've never eaten in a normal, healthy way, and gained weight.
Calories are such that if you screw up once per week (birthday party, company event, family dinner) that could mean you gain weight if you eat regularly the rest of the days.
> "It seems that some people are optimized for famine, and some for feast. :-("
You're actually onto something there! I don't know if you've read much about epigenetics, but if a person experiences a famine, it can "switch on" prepare for famine genes in their descendants. It's fascinating stuff.
> Our diets are similar (in kind, not quantity of course, she eats much less than me).
Maybe. I've heard this sort of story before, and I don't tend to believe it. It's hard enough to estimate how much you are eating yourself, and comparing against others is even more error-prone.
I'm facing a similar situation, but I'm loathe to start counting calories just to confirm my hypothesis. Being forgetful and apathetic about meals almost certainly contributes to why I've maintained a healthy weight. I worry that the rigor required for proper observation will change my behaviour.
Kelly Brownell, director of the Rudd Center for Food Policy and Obesity at Yale University, says that while the 10,000 people tracked in the registry are a useful resource, they also represent a tiny percentage of the tens of millions of people who have tried unsuccessfully to lose weight. “All it means is that there are rare individuals who do manage to keep it off,” Brownell says. “You find these people are incredibly vigilant about maintaining their weight. Years later they are paying attention to every calorie, spending an hour a day on exercise. They never don’t think about their weight.”
That just described reddit subs focused on weight loss.
> they also represent a tiny percentage of the tens of millions of people who have tried unsuccessfully to lose weight.
They also represent an unknown number of people who do keep off weight successfully, but never report in, for whatever reason. As just one example, I asked for an application and when it came I realized I did not qualify for the registry as I don't have a before photo.
> “You find these people are incredibly vigilant about maintaining their weight. Years later they are paying attention to every calorie, spending an hour a day on exercise. They never don’t think about their weight.”
Anti-dieters love to trot this out, but they have no evidence to back it up. There are plenty of people who log in MFP, check to see if what they are contemplating ordering fits their budget for the day, then go about the rest of their lives. It takes all of a couple of minutes. There are other people who just more carefully mind what they eat, listen to their body, have changed habits (eg, cutting out soda) and never even track calories. I'm one of them.
As for spending an hour a day on exercise, that's not unreasonable. Most people spend multiple times that amount of time on things like TV or web browsing. It's also not mandatory for weight loss.
Rather than just positing the question, share with us why you think it shouldn't have been downvoted. And explore reasons—even if you don't agree with them or think they're groundless—why someone might downvote it. Of course there are those out there who downvote for reasons we think are frivolous: they're not likely to respond to your comment anyway.
The guidelines ask us not to comment on being downvoted: I think in general this should extend to commenting on other's downvotes as well: it makes for boring reading. You mention this yourself—it's a broken record. If you are going to do so, put some effort in to make the comment worthwhile. It's also a good exercise in improving discourse, if that's something you're interested in.
I kind of agree with you but doing that would turn a a genuine question into a high school hand-in complete with a discussion of the results.
I will likely consider your opinion next time I'm tempted to point out that someone is downvoted for seemingly no good reason and with no explanation.
But I am not sure of the outcome - I would like others to defend me when I am accused or picked on for no good reason. Basically I'm doing to others what I hope they would do for me.
But I am not sure of the outcome - I would like others to defend me when I am accused or picked on for no good reason.
Right, and by providing a explanation than just posing the question, you're doing a better job of doing exactly that—showing why you think it shouldn't have been downvoted—while also contributing to the conversation. In my experience, the people who are responding to such questions are not the people who have downvoted—they're doing some version of what I've outlined above. If you have no idea why something might be downvoted, it likely would be good for you to stretch a bit and imagine how someone else might read or take the comment you're referring to. It'll likely make you a better comment writer and reader.
Thanks for following with a citation, but I am well aware of that research. The national weight control registry is a heavily self selected group of people who have already lost significant weight before joining - therefore weeding out most of the failure rate. And even then, only 20% of their audience lost over 10% of their initial body weight and kept it off for one year.
> you won't find any reputable study on the web where the average person lost 10%+ of their body weight and kept it off for five years. Not even one.
To refute this, it's sufficient to present a counterexample. Since many studies exist, and they are reputable, the only argument is whether the people in them are average.
You say they are not, because they lost weight. But that cannot be your whole argument, because if we assume average people don't lose weight we assume the premise you have taken up to prove.
So again: other than the fact that these people lost weight, what exactly is exceptional about them?
Hang on, has a counterexample been presented? Maybe we're reading the challenge differently, but I parsed "reputable study on the web where the average person lost 10%+ of their body weight and kept it off for five years" as meaning a study in which a bunch of already-overweight people were selected to try some specific intervention (eg, "follow this diet & exercise program") and as a result of the intervention the median outcome was to lose that much weight and keep it off for 5 years.
A study of only people who succeeded, selected for the study because they succeeded, tells you nothing about how effective their particular strategy is.
For instance, suppose there exists a Grapefuit Diet which has the following effect: 1% of the people who try it lose weight, 2% of the people who try it gain weight, and 97% of the people who try it see no effect. If you take 100 people, tell them to try that diet, that is the result you'll get - a result which tells you the strategy is no good. But if you look at the national weight loss registry you'll only find people who were in the tiny 1% for which that diet worked. In fact, even if there are a LOT of Grapefruit Diet successes in the registry, that just tells you how popular the Grapefruit Diet was, it doesn't tell you if the Grapefruit Diet works.
What we want is an intervention study where a bunch of fat people do some specific thing and that thing is actually effective at producing a clinically significant and stable amount of weight loss. Ideally we'd want them to lose enough weight such that they are no longer "overweight" and then maintain that state.
Does that study exist? Is there any study meeting that set of criteria? My impression is that it does not; there is in fact no non-surgical intervention known to "work". Which explains why people keep grasping at straws to find options that plausibly might work.
To contradict this a little bit (and this is not 100% relevant because it involves exactly one participant, so who knows whether it would ever work for anyone else) this remains one of the most interesting weight loss related things I've read on the internet: http://edwardjedmonds.com/wp-content/uploads/2013/12/Stewart...
The study participant went from 456 to 180lbs through the course of a year, and when they checked on him five years later, he weighed 196lbs. Obviously this is an extreme case, but it is one tiny and yet super interesting datapoint on the subject.
As someone trying to lose weight myself, calorie tracking has so far yielded about 5% weight loss over a period of several months but I'm realizing that if I stop counting calories there's a strong possibility of gaining that back. This just lends some credence to that idea.
Don't give up! 5% is nothing to sneeze at, and that pace is actually probably the best way to go about it as it's not radical changes to your lifestyle. It also gives your body, both inside and out, more time to adjust.
The tracking might be a thing for the rest of your life, but it's so easy these days with things like MFP. There's also hope in that once you maintain a weight for a year, your hormones (ghrelin and leptin) will adjust and you won't be as hungry.
Also, people love to trot out the "all diets fail" and claim no one has ever shown otherwise, when there are things like the National Weight Control Registry that put that pernicious myth to rest. As another poster said, most people don't realize that you don't just diet temporarily and stop. The word "diet" derives from the Greek "diaita" which means "way of life".
All diets fail for nearly everyone who tries them, the National Weight Control Registry notwithstanding.
The fact that a tiny percentage of the populace successfully loses weight does not make "try to do what those people claim they were doing" good advice, any more than the fact that a tiny percentage of people win the lottery makes "buy lottery tickets!" good financial advice.
The notion that "all diets fail" persists because thus far it sadly remains true - telling people to lose weight and keep it off by dieting is faith-based advice, wishful thinking. It's not science.
Weight loss is difficult because dieting isn't a train you get off once you're at your target weight. It's basically a way of life to maintain your target weight. I don't know too many people that are capable of rewiring themselves to that kind of degree for the long-term
Reliably changing someone's weight for the long term probably requires swapping environment and altering their peer group/friends. Kinda like getting off of drugs.
Honestly, the human body just isn't built to lose weight. We're built for starvation and packing on as much as we can, while we can. Humanity is just a victim of its own success with the sheer plentiful amount of food that's available now, at least in the first world.
Working as intended, WONTFIX, etc. Obesity at this stage of our civilization is not abnormal. It sucks, but it isn't abnormal.
To add to this: this is 90% behavioral in nature. We like food, eating food makes us feel good, therefore we eat food and snack constantly. Obviously, you'd do it - it's like getting a hit off your favorite drug. If it didn't give us that high, we wouldn't do it.
It bothers me to see three or four comments saying there's no point in trying to lose weight go unchallenged.
Widespread obesity is a recent development in our society. And it's not caused by the abundance of our wealth, as the poor are most affected. Rather it is caused by outsourcing cooking from the family to corporations, whose incentives are at odds with our health. They make fattening, craveable, nutritionally deficient food, and work hard to ensure that we and our kids are confused about what is healthy, while making sure that their products are always more available and convenient than planning and cooking meals.
Society has only turned this way in the last two generations, and it can be fixed. We are not doomed by biology.
It's interesting that the comments on any article having anything to do with food automatically become a "weight thread." The topic of this article is poor scientific methodology and the failure of the current peer review system to address it, not diet or weight loss. And (based on the studies summarized in the article) the researcher discussed in the article works on psychological and economic factors in eating behavior, not defining what people should eat or promoting diets.
This is something that irritates me to the point of irrationality.
Why so many people spend so much time and resources on $name diets, questionable research, news about how 'xyz' is good for you while 'abc' is bad, exercises to eliminate calories in 30 days, exercises to remove localized fat etc? All of this is shown to be evidently bullshit for decades for the minimally rational observer.
Meanwhile the knowledge of what works is blatantly obvious: the energy expenditure must be higher than the intake. Having a healthy diet and exercise regimen along that is highly desirable but not strictly necessary, stating this just because understanding the core concept is more important than sheepishly believing the latest fad. Also that this is a process that takes a lot of time and is not a one-off procedure but a process of learning life-changing habits.
I'd like to remark that this is not easy at all! In fact is really hard due to the inherent biological and societal tricks that play on our minds. But if people are already suffering psychologically and financially with this, why don't just try the basics?
I understand how this is incentivized by an industry that extracts money from desperate people trying very hard to feel accepted by what society indicates as an acceptable and desirable appearance. The irony is that the failure of their latest hope is what makes them unable to understand that the problem is much simpler (not easy), and only try again on a more desperate attempt with the latest extreme measure.
At least if people would be honest enough to blame themselves for their bad habits (lack of a healthy diet and exercise regimen) they could start the process of accepting what they are as a result of their choices, and finally notice that they have some agency on this.
I am not talking here about existent mental and biological disorders of course and would not downplay their role.
Sorry for the rant, the only way I can rationalize that this exists is that there are people that deny the existence of climate change.
> Meanwhile the knowledge of what works is blatantly obvious: the energy expenditure must be higher than the intake.
This is like saying about the poor, "What works is blatantly obvious: Poor people just need to increase income or reduce expenditures. Boom! Poverty solved."
The secret is that this isn't the problem. The actual problem that fat people must solve is not what to do, but how to do it, both in the short- and long-term.
> Having a healthy diet and exercise regimen along that is highly desirable but not strictly necessary...
1,000 calories of cupcake are 1,000 calories of chicken breast are, in theory, the same amount of energy. In practice, diet and exercise considerations are integral aspects of the actual problem that fat people must solve.
Example: For some, exercise is actually counter-productive and causes more hunger. For others, exercise is a crucial part of success.
Example: For some, calorie restriction can be done with no particular attention paid to the balance of carbs, proteins, and fats. For others, a low-carb diet means lower hunger/higher satiation for longer and is a must for success.
> The secret is that this isn't the problem. The actual problem that fat people must solve is not what to do, but how to do it, both in the short- and long-term.
You're right, it just gets to my nerves as this is a source of huge suffering to most people and there's an industry specialized to prey on this.
I just want to point out that this is not restricted to 'fat' people, but a great majority of people who feel a huge pressure to adhere to a physical appearance standard without the proper education of how to get there and maintain it.
Sometimes I find myself with an empty fridge and can't be bothered to go shopping, so I'll just eat less for a day. It's not hard to not do something. Just don't have too much food in your home, then you can't eat too much of it.
>For some, exercise is actually counter-productive and causes more hunger. For others, exercise is a crucial part of success.
You don't have to eat just because you are hungry. This is actually the one and only problem, eating impulsively, but even in general, doing anything impulsively.
> the energy expenditure must be higher than the intake
I think you might be overestimating the predictive power of this statement. The reason that these fad diets and questionable research end up being so popular is that people have a huge amount of anecdotal evidence that this does not apply as a necessary condition for weight loss.
I think it would be hard to argue that it isn't sufficient; assuming that "energy expenditure" and "intake" are well-defined (note that even the model of "calories" in food is not a chemical notion, as many think, it's a heuristic based on a known incorrect model of metabolism), but people see evidence all around them (once again, anecdotal, and in some cases based on incomplete evidence) that people who neither restrict input nor expand expenditure can maintain a healthy weight.
People want there to be a magic bullet that will let their metabolism shift into a state where the body would simply stop storing excess calories as fat. I'm fairly convinced that we haven't yet found it, but I think it is a laudable goal. And, because losing weight has positive health and social implications, this is a place where people are desperate enough for a solution that won't involve unpleasantness that they're willing to believe unreasonable things.
I agree with your reasoning, and that complements or corrects what I've written perfectly.
The last paragraph is extremely sad though: that it would require a new complicated drug, procedure or technology to overcome the forces present in the average person blocking it from having agency on this issue.
Maybe this problem points out to huge psychological (people that can't afford treatment or realize underlying mental problems that exarcebate the condition) or economical (a healthy lifestyle is more difficult to attain if you're constantly stressed to have minimal provisions or time) issues on society as a whole.
For what it's worth, this article is not about "$name diets." Rather, it is about a researcher's use of questionable methods. The researcher's field happens to be psychological factors affecting eating behavior, but his questionable methods can be/are used in other disciplines (as pointed out in the article). And based on the summaries in the article, the researcher seems to be in complete agreement with your rant: his studies are not about what to eat, but how to more easily achieve the intake vs. expenditure balance (for example, by using smaller plates to make it psychologically easier to limit portion size, or giving low-calorie foods a name that makes them more psychologically attractive). So, it's hard to see what in the article this rant is responding to.
I did not make myself clear (hence the irrational part of it): this rant was not a direct response to the article, instead revolving about how awful it is that we have an environment that pushes people to follow an agenda that is not suited to their best interests.
> Many scientists receive only cursory training in statistics,
This seems to be true (at least I have many younger colleagues who are woefully ill informed about even basic stats.) but it's very strange.
Most of what I learned of statistics, including tests for significance and regression, I learned in senior high school at the age of 17.
Learning not to search of correlations like this is pretty much equivalent to ensuring that you do not over fit your data set which is surely such conventional wisdom that no one getting a scientific education in the last sixty or seventy years should have missed it.
I'm aware that there is a lot of highly sophisticated statistical analysis that I don't know about but things like p value are not among them.
83 comments
[ 0.22 ms ] story [ 111 ms ] threadThis emblematic of a larger problem with how science is practiced: the obsessive focus on p-value thresholds leads to irrational practices like trawling data for interesting "findings."
But on a certain level Wansink was right: a data set is not completely worthless if it showed a null result. So we need to start thinking about how to communicate the value of data even when the null is not rejected.
One way to do this is to encourage widespread sharing of data sets regardless of the outcome of the experiment. Maybe for a given study the data did not show a definitive result - but does the data point to potential future paths of study? Maybe another researcher could get ideas for new experiments.
Why is a null result worthless? I don't understand. It goes against my common sense that paying more for food doesn't make you eat more. The fact that this is not the case is valuable knowledge to me. I understand that it's not as glamorous as "paying more makes you eat less!", but it's still valuable knowledge that should be published.
A null result is not worthless. A data set is not worthless if it showed a null result. Nobody is claiming a null result is worthless. Even the original article gives a good reason for publishing a null result.
As a former neuroscientist, I had a unicorn data set of intracranial EEG data, and we'd spent so much time collecting it, that we were determined to find something. I left grad school before we found anything of interest. I believe my former prof eventually published something on it, but I analyzed the shit out of it, even knowing I was fishing, because so much time would have been lost to not use it.
Fishing like this was one of the reasons I left. The pursuit of knowledge and the pursuit of your career don't align often enough.
To me, what's unusual is not that Wansink pushed to keep analyzing the data, but that he was called out for it. Everyone seemed to be doing it when I was in academia.
1. https://en.wikipedia.org/wiki/Bonferroni_correction
As the other commenters have pointed out, the fact that it showed a null result is not really relevant to future utility.
A better way of phrasing it might be "A data set is not completely worthless after it has been used to the test the hypotheses for which it was gathered."
As to how it is useful, I think a sound summary would be "a data set is _only_ useful for validating (or invalidating) those hypotheses for which it is gathered, but it may be useful for deriving new hypotheses for which new data can be gathered to validate."
Or more negatively as "a data set is not useful for validating or invalidating hypotheses invented after the data set is seen, but it may be useful for deriving those hypotheses and leading to further research."
To some very limited extent, if a mechanism of action is established (which is to say there is an external reason to infer a correlation) then cross-validation with an existing data set may be able to derive the parameters of that correlation.
This is the 21st century, where most if not all reputable journals have an online presence and submission portal. It would be great if authors had to upload their anonymized data sets in a common format (.csv?) that can easily be imported into statistical analysis packages (ex: Minitab, read into R, etc). Journals provide analysis and test recommendations, reviewers run the tests, upload their independent analysis results as part of the review process. The idea here is to produce something like an auditable paper trail.
There has to be some sort of solution for this problem, especially in 2017. Hiding behind shitty (or worse, deceitful) data analysis shouldn't be possible.
That opinion is already "reject it all".
Even when it comes to medicine that opinion is strongly represented. A few people even ascribe the placebo effect to massive and widespread statistical error.
http://thelastpsychiatrist.com/2009/10/the_problem_with_scie...
Anecdote: my household is a family of four. Myself and my wife, and our two teenage kids, daughter and son. Since we live together, the answers to the 10 questions on http://www.slimbydesign.com/get-scored/ are the same for the household. My son and I are normal weight. My wife and daughter are overweight. Conclusion: weight has very little to do with the design of your kitchen. Caveat: this conclusion has not been peer reviewed and is not scientifically sound.
If anything, he reminds me of a lot of brilliant researchers, who are too confident in their methodological prowess. (And this is the outcome that should have been avoided through collaboration).
It is, but it's not a completely invalid starting point. Even better is to study people who were once fat, but lost weight and keep it off. This is what the National Weight Control Registry does. It's similar to studies that have tried to pin down why marriages fail. Most studies focus on what went wrong in failed marriages, but some of the most helpful suggestions have come from what people who stay married do.
> My son and I are normal weight. My wife and daughter are overweight.
Do your wife and daughter eat the same portions as you and your son? It's a common complaint of women, who are on average shorter than men and therefore need fewer calories, that they can't eat as much as their significant others.
No. I expand on that a bit here - https://news.ycombinator.com/item?id=14196631
It doesn't seem unreasonable to think that the peer review process should include a statistician or someone who can review the statistics.
"Wansink added another note to his blog post, acknowledging the problems and announcing that a statistician would be redoing the analyses."
With all of the issues lately around replication of studies it seems like having a statistician review the math should be step one of any peer-review process.
> as consultants on the rare papers in which there are serious issues about the statistical techniques used, at the editor's discretion and based on the referees' suggestions.
I think the crux is that editors already have enough trouble finding reviewers, and see getting a statistician as often unreasonable. It's too bad, because the nature of statistical issues is that people tend to be unaware of them until a statistician (used broadly) points them out.
Even this is not sufficient -- the fact is that this is a methodological statistical error, not a mathematical statistical error.
The mistake is thinking that a dataset alone can yield data supporting a hypothesis derived from that dataset, rather than deriving a testable hypothesis from an existing dataset, and then gathering new data to validate.
I would say the only way that you could publish results derived from an existing data source is if you also published all the null results you got along the way when examining the data set; but this is not really a feasible thing to print in a publication; either the list would be very long and thus it would be clear that some sort of p-hacking was involved, or the list would be short and accidental or intentional omission would be suspected.
This dramatically reduces the amount of useful results that can be squeezed from a dataset, which is unfortunate, as many of them are hard to gather in the first place. It might be necessary to protect these datasets better -- to restrict access conditional on specifying the hypothesis being tested with the requirement that all results, including null results, be recorded, even if only in summary form if the results are not interesting enough to be accepted for publication or to be worth the effort of composing into a quality research paper.
The more hypotheses that are tested using the 30% "test" dataset, the more that dataset inadvertently becomes a "training" dataset, by progressively invalidating models. For applications with real-world significance, you usually have to accept that the "test" dataset will expire after some number of experiments into the "training" dataset, and gather a new test dataset to prevent overfitting. Hastie, Tibshirani, and Friedman[1], which I heartily recommend, cover this extensively in chapter 7.
[1] https://statweb.stanford.edu/~tibs/ElemStatLearn/printings/E...
That said, Wansink's papers had plenty of mathematical statistical errors that were evident without access to the original data, as documented in the mentioned "Statistical Heartburn" preprint: https://peerj.com/preprints/2748.pdf
I have a pre-existing bias against papers relating to nutrition science, and this looks worse even than I expect. I would love to see a treatment that looks at especially noteworthy research in the area that passes a rigorous methodological review, much less has been replicated with any reliability.
Hunger is your body signalling your brain that you need nutrients. If you eat foods that cover all your nutrient requirements for a low number of calories and your body is taking up these nutrients properly, then you should have no problems. Thus it makes sense to inform people which foods they need to be eating as well as educate people on which disorders can cause poor nutrient uptake.
If you suffer from stress and you eat high calorie foods to feel better, changing your diet is likely going to exacerbate your stress, not make you feel better. I feel this is the case for many who are overweight.. stress management strategies should be the first priority as you won't be able to handle changing your diet for the long term until you get your stress under control.
[/sarcasm]
That's most of the field of macroeconomics. It's all data analysis, not controlled experimentation.
But doing that is hard, so most people don't even try ...
yes, practically, that's how modern data science works.
1) ["In controlled settings, participants who remain in weight loss programs usually lose approximately 10% of their weight. However, one third to two thirds of the weight is regained within 1 year, and almost all is regained within 5 years. "](http://www.ncbi.nlm.nih.gov/pubmed/1580453)
2) Giant meta study of long term weight loss: ["Five years after completing structured weight-loss programs, the average individual maintained a weight loss of >3% of initial body weight."](http://ajcn.nutrition.org/content/74/5/579.full)
3) Less Scientific: [Weight Watcher's Failure - "about two out of a thousand Weight Watchers participants who reached goal weight stayed there for more than five years."](https://fatfu.wordpress.com/2008/01/24/weight-watchers/)
4) [The reason why it's impossible seems to be that although calories in < calories out works, the body of a fat person makes it extremely difficult psychologically to eat less.](http://www.nytimes.com/2012/01/01/magazine/tara-parker-pope-...) This is borne out by the above data.
5) [The only thing that does seem to work in the long term is gastric surgery.](http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1421028/)
Moreover, you won't find any reputable study on the web where the average person lost 10%+ of their body weight and kept it off for five years. Not even one.
Of course there is: http://www.nwcr.ws/Research/published%20research.htm.
Here's a good question: other than the fact that these people lost weight, what is identifiably unusual about them?
In case someone out there is serious about losing weight and not making excuses, here's how you calculate your actual TDEE: https://www.reddit.com/r/leangains/comments/2rv09z/this_is_h...
But then there are others who seem to do everything right and are over weight in spite of that.
For example. I was never overweight as a kid and relatively active. In college and for the start of my career, I stopped being active and my diet was awful (e.g. I thought a large Jamba Juice smoothie and a carrot cake was a healthy breakfast choice). My weight ballooned up to almost 190 lbs (20+ lbs overweight), my blood pressure went up, I started having rosacea.
I started running and fixed my diet. Quickly my weight dropped down to 150 and I've kept it in the 140-150 range for over a decade. The other health issues cleared up as well. But it wasn't hard work for me. Being thin is my natural state if you will, and I had to do everything wrong to stay overweight.
My wife meanwhile continues to struggle with her weight. She's successfully lost weight through extremely diligent calorie counting, but after a year or so she starts to put it back on. I have never counted calories. Our diets are similar (in kind, not quantity of course, she eats much less than me). She is active, but not quite as active as me. So similar diet and life styles, but my weight stays off and hers does not.
Hereditarily, no one in my family is over weight. There is obesity on both sides of her heredity.
And I see this playing out in our kids. My son has an athletic build and will probably never have weight issues. My daughter takes after her mom and it will take a life time of diligence for her to remain at a healthy weight.
It seems that some people are optimized for famine, and some for feast. :-(
Obviously there are a lot of factors involved in the growing obesity crises. But I feel for people who struggle with their weight despite doing all the right things, I really do.
Calories are such that if you screw up once per week (birthday party, company event, family dinner) that could mean you gain weight if you eat regularly the rest of the days.
You're actually onto something there! I don't know if you've read much about epigenetics, but if a person experiences a famine, it can "switch on" prepare for famine genes in their descendants. It's fascinating stuff.
http://www.radiolab.org/story/251885-you-are-what-your-grand...
https://www.newscientist.com/article/dn25884-famine-puts-nex...
Maybe. I've heard this sort of story before, and I don't tend to believe it. It's hard enough to estimate how much you are eating yourself, and comparing against others is even more error-prone.
I'm facing a similar situation, but I'm loathe to start counting calories just to confirm my hypothesis. Being forgetful and apathetic about meals almost certainly contributes to why I've maintained a healthy weight. I worry that the rigor required for proper observation will change my behaviour.
Kelly Brownell, director of the Rudd Center for Food Policy and Obesity at Yale University, says that while the 10,000 people tracked in the registry are a useful resource, they also represent a tiny percentage of the tens of millions of people who have tried unsuccessfully to lose weight. “All it means is that there are rare individuals who do manage to keep it off,” Brownell says. “You find these people are incredibly vigilant about maintaining their weight. Years later they are paying attention to every calorie, spending an hour a day on exercise. They never don’t think about their weight.”
That just described reddit subs focused on weight loss.
They also represent an unknown number of people who do keep off weight successfully, but never report in, for whatever reason. As just one example, I asked for an application and when it came I realized I did not qualify for the registry as I don't have a before photo.
> “You find these people are incredibly vigilant about maintaining their weight. Years later they are paying attention to every calorie, spending an hour a day on exercise. They never don’t think about their weight.”
Anti-dieters love to trot this out, but they have no evidence to back it up. There are plenty of people who log in MFP, check to see if what they are contemplating ordering fits their budget for the day, then go about the rest of their lives. It takes all of a couple of minutes. There are other people who just more carefully mind what they eat, listen to their body, have changed habits (eg, cutting out soda) and never even track calories. I'm one of them.
As for spending an hour a day on exercise, that's not unreasonable. Most people spend multiple times that amount of time on things like TV or web browsing. It's also not mandatory for weight loss.
Why was this downvoted?
The guidelines ask us not to comment on being downvoted: I think in general this should extend to commenting on other's downvotes as well: it makes for boring reading. You mention this yourself—it's a broken record. If you are going to do so, put some effort in to make the comment worthwhile. It's also a good exercise in improving discourse, if that's something you're interested in.
I will likely consider your opinion next time I'm tempted to point out that someone is downvoted for seemingly no good reason and with no explanation.
But I am not sure of the outcome - I would like others to defend me when I am accused or picked on for no good reason. Basically I'm doing to others what I hope they would do for me.
Right, and by providing a explanation than just posing the question, you're doing a better job of doing exactly that—showing why you think it shouldn't have been downvoted—while also contributing to the conversation. In my experience, the people who are responding to such questions are not the people who have downvoted—they're doing some version of what I've outlined above. If you have no idea why something might be downvoted, it likely would be good for you to stretch a bit and imagine how someone else might read or take the comment you're referring to. It'll likely make you a better comment writer and reader.
I'll try that next time I guess.
> you won't find any reputable study on the web where the average person lost 10%+ of their body weight and kept it off for five years. Not even one.
To refute this, it's sufficient to present a counterexample. Since many studies exist, and they are reputable, the only argument is whether the people in them are average.
You say they are not, because they lost weight. But that cannot be your whole argument, because if we assume average people don't lose weight we assume the premise you have taken up to prove.
So again: other than the fact that these people lost weight, what exactly is exceptional about them?
A study of only people who succeeded, selected for the study because they succeeded, tells you nothing about how effective their particular strategy is.
For instance, suppose there exists a Grapefuit Diet which has the following effect: 1% of the people who try it lose weight, 2% of the people who try it gain weight, and 97% of the people who try it see no effect. If you take 100 people, tell them to try that diet, that is the result you'll get - a result which tells you the strategy is no good. But if you look at the national weight loss registry you'll only find people who were in the tiny 1% for which that diet worked. In fact, even if there are a LOT of Grapefruit Diet successes in the registry, that just tells you how popular the Grapefruit Diet was, it doesn't tell you if the Grapefruit Diet works.
What we want is an intervention study where a bunch of fat people do some specific thing and that thing is actually effective at producing a clinically significant and stable amount of weight loss. Ideally we'd want them to lose enough weight such that they are no longer "overweight" and then maintain that state.
Does that study exist? Is there any study meeting that set of criteria? My impression is that it does not; there is in fact no non-surgical intervention known to "work". Which explains why people keep grasping at straws to find options that plausibly might work.
The study participant went from 456 to 180lbs through the course of a year, and when they checked on him five years later, he weighed 196lbs. Obviously this is an extreme case, but it is one tiny and yet super interesting datapoint on the subject.
The people who trot out "all diets fail" or "you'll regain!" would class that as his diet failing.
As someone trying to lose weight myself, calorie tracking has so far yielded about 5% weight loss over a period of several months but I'm realizing that if I stop counting calories there's a strong possibility of gaining that back. This just lends some credence to that idea.
The tracking might be a thing for the rest of your life, but it's so easy these days with things like MFP. There's also hope in that once you maintain a weight for a year, your hormones (ghrelin and leptin) will adjust and you won't be as hungry.
Also, people love to trot out the "all diets fail" and claim no one has ever shown otherwise, when there are things like the National Weight Control Registry that put that pernicious myth to rest. As another poster said, most people don't realize that you don't just diet temporarily and stop. The word "diet" derives from the Greek "diaita" which means "way of life".
The fact that a tiny percentage of the populace successfully loses weight does not make "try to do what those people claim they were doing" good advice, any more than the fact that a tiny percentage of people win the lottery makes "buy lottery tickets!" good financial advice.
The notion that "all diets fail" persists because thus far it sadly remains true - telling people to lose weight and keep it off by dieting is faith-based advice, wishful thinking. It's not science.
Working as intended, WONTFIX, etc. Obesity at this stage of our civilization is not abnormal. It sucks, but it isn't abnormal.
Widespread obesity is a recent development in our society. And it's not caused by the abundance of our wealth, as the poor are most affected. Rather it is caused by outsourcing cooking from the family to corporations, whose incentives are at odds with our health. They make fattening, craveable, nutritionally deficient food, and work hard to ensure that we and our kids are confused about what is healthy, while making sure that their products are always more available and convenient than planning and cooking meals.
Society has only turned this way in the last two generations, and it can be fixed. We are not doomed by biology.
Why so many people spend so much time and resources on $name diets, questionable research, news about how 'xyz' is good for you while 'abc' is bad, exercises to eliminate calories in 30 days, exercises to remove localized fat etc? All of this is shown to be evidently bullshit for decades for the minimally rational observer.
Meanwhile the knowledge of what works is blatantly obvious: the energy expenditure must be higher than the intake. Having a healthy diet and exercise regimen along that is highly desirable but not strictly necessary, stating this just because understanding the core concept is more important than sheepishly believing the latest fad. Also that this is a process that takes a lot of time and is not a one-off procedure but a process of learning life-changing habits.
I'd like to remark that this is not easy at all! In fact is really hard due to the inherent biological and societal tricks that play on our minds. But if people are already suffering psychologically and financially with this, why don't just try the basics?
I understand how this is incentivized by an industry that extracts money from desperate people trying very hard to feel accepted by what society indicates as an acceptable and desirable appearance. The irony is that the failure of their latest hope is what makes them unable to understand that the problem is much simpler (not easy), and only try again on a more desperate attempt with the latest extreme measure.
At least if people would be honest enough to blame themselves for their bad habits (lack of a healthy diet and exercise regimen) they could start the process of accepting what they are as a result of their choices, and finally notice that they have some agency on this.
I am not talking here about existent mental and biological disorders of course and would not downplay their role.
Sorry for the rant, the only way I can rationalize that this exists is that there are people that deny the existence of climate change.
This is like saying about the poor, "What works is blatantly obvious: Poor people just need to increase income or reduce expenditures. Boom! Poverty solved."
The secret is that this isn't the problem. The actual problem that fat people must solve is not what to do, but how to do it, both in the short- and long-term.
> Having a healthy diet and exercise regimen along that is highly desirable but not strictly necessary...
1,000 calories of cupcake are 1,000 calories of chicken breast are, in theory, the same amount of energy. In practice, diet and exercise considerations are integral aspects of the actual problem that fat people must solve.
Example: For some, exercise is actually counter-productive and causes more hunger. For others, exercise is a crucial part of success.
Example: For some, calorie restriction can be done with no particular attention paid to the balance of carbs, proteins, and fats. For others, a low-carb diet means lower hunger/higher satiation for longer and is a must for success.
You're right, it just gets to my nerves as this is a source of huge suffering to most people and there's an industry specialized to prey on this.
>For some, exercise is actually counter-productive and causes more hunger. For others, exercise is a crucial part of success.
You don't have to eat just because you are hungry. This is actually the one and only problem, eating impulsively, but even in general, doing anything impulsively.
Thank you for solving obesity!
I think you might be overestimating the predictive power of this statement. The reason that these fad diets and questionable research end up being so popular is that people have a huge amount of anecdotal evidence that this does not apply as a necessary condition for weight loss.
I think it would be hard to argue that it isn't sufficient; assuming that "energy expenditure" and "intake" are well-defined (note that even the model of "calories" in food is not a chemical notion, as many think, it's a heuristic based on a known incorrect model of metabolism), but people see evidence all around them (once again, anecdotal, and in some cases based on incomplete evidence) that people who neither restrict input nor expand expenditure can maintain a healthy weight.
People want there to be a magic bullet that will let their metabolism shift into a state where the body would simply stop storing excess calories as fat. I'm fairly convinced that we haven't yet found it, but I think it is a laudable goal. And, because losing weight has positive health and social implications, this is a place where people are desperate enough for a solution that won't involve unpleasantness that they're willing to believe unreasonable things.
The last paragraph is extremely sad though: that it would require a new complicated drug, procedure or technology to overcome the forces present in the average person blocking it from having agency on this issue.
Maybe this problem points out to huge psychological (people that can't afford treatment or realize underlying mental problems that exarcebate the condition) or economical (a healthy lifestyle is more difficult to attain if you're constantly stressed to have minimal provisions or time) issues on society as a whole.
And how easily it is able to.
This seems to be true (at least I have many younger colleagues who are woefully ill informed about even basic stats.) but it's very strange.
Most of what I learned of statistics, including tests for significance and regression, I learned in senior high school at the age of 17.
Learning not to search of correlations like this is pretty much equivalent to ensuring that you do not over fit your data set which is surely such conventional wisdom that no one getting a scientific education in the last sixty or seventy years should have missed it.
I'm aware that there is a lot of highly sophisticated statistical analysis that I don't know about but things like p value are not among them.