Built using those newfangled convolutional neural network things. Trained caffe on a set of about a million food related images. Can't believe how accurate the classifier is...
UI needs work. Would love feedback on how to make food logging more intuitive.
Your app kinda sucks now, but the detection is great. You could also do food identification as a service...
I'm trying to use fatsecret/myfitnesspal for counting calories. Something like this could make it much easier!
To me, a better workflow would be: take pictures of everything you eat and classify later (at night, maybe on the PC). Of course, this software would simplify the classification, the user would only fix any errors and adjust sizes.
This workflow would be well suited to your app, which operates on a server? (my understanding).
Congrats, I understand the relevance of AI for simple apps now ;)
Ive been thinking of that 'classify later' workflow. I think of it kind of like facebooks photo tagging, and it might even be better as web app that you get email reminders for to complete entries.
It would also work nicely for background geolocation services, since if the photograph was taken near a restaurant the service could extrapolate those things.
Anyways, I might make this project opensource and build a nice community around it. Depends on whether or not that model would best support my research goals.
Thats another huge design challenge Ive faced, specifically because it really complicates the data model and UI.
Implicitly, since the app presents a list of possible identified foods, and the user can easily check multiple boxes, thats straightforward, but then having the app pull each of those items AND ask the user to estimate portions for each .... gets a little messy.
So I put that complexity off for v2. That use case in particular would also be best off for a web-interface, I think.
What about dividing the whole plate weight by the foods detected? (in Brazil there are many restaurants where you pay by weight, thats why I'm suggesting..)
Cut my teeth on the netflix prize several years back. Took ML in my graduate coursework, and have dabbled with it on and off since. I love the whole theory of matrix factorization and its myriad uses.
Edit: oops, should have read further. I see that you pulled appropriate terms from WordNet and used ImageNet to gather training images from Flickr. Cool!
I feel like its easy to make a better app if you don't try to please all use cases - just try to make the food logging in less taps. Logging in those apps takes me around 5-10 taps plus typing the food name. Ify ou could do it in 2 or 3, it would be great.
I wonder if somebody would buy a "highly opinionated calorie tracker" that only has non-barcode foods stored in it and mocks you when you try to add bad food to its database. If you miss your macros for the day, it would then text everyone in your phone that you're a fat weenie with no willpower.
I don't understand this sentiment. My grocery store allows us to print barcodes that describe the weight and price of any produce or bulk order we purchase. This ranges from quinoa, oats, nuts, seeds, tea, along with the usual fruits and veggies.
Also, don't bulk packages of vegetables tend to have barcodes on them inherently?
But those barcodes wont show up in a public database. Also bulk foods are not fresh. Fresh food from local farmers does not usually have a barcode on it. This is not just snobbery.
Produce is probably the most common type of food that does not typically have a barcode. It has a product code on it that is entered in manually while the weight measurement is taken.
This is not compatible with my device which is an LeTV X500, running Android 5.0.2. Is there any reason that it might not work? I'm in the UK, if that might be the reason.
Im new to cordova (the toolkit I used to build the app), and I did notice that the number of supported devices is kind of low for my build. I may have included a plugin that narrows down the pool. I need to dig into this.
Fatsecret API for nutritional information and Foursquare for checkins, but installed and trained the neural network on my AWS instance for the prediction stuff.
I remember the latest Android camera app being able to extrapolate distances in the app (thus simulating photos with large apertures/shallow depth of field).
I wonder if you could use this visual distance algorithm to somehow judge portion sizes/plate sizes?
Good thinking. Assuming you don't have the portion size data, you may be able to run your data set through Mechanical Turk or hire an oDesk worker, and you could build a training set..
Isn't it strange that in [1] the beginning of the "Normal" BMI has more risk of death than the beginning of the "Obese"? I'd even venture to say that the "Overweight" range has, in average, lower Mortality Risk than the "Normal".
PS, another plus for the Metric System. The Imperial System form includes Feet+Inches, while the metric one only needs Centimeters for the height.
Its been contested in a few papers now that overweight people can outlive normal weight, according to these BMI subcategories. But I am not surprised that things can be fuzzy, since ultimately its a continuum.
The finding that people with an "Overweight" BMI have a lower rate of all-cause mortality is consistent and has been called the "obesity paradox." A very interesting study came out this month that addressed it.[0] Basically, they did a prospective study of 55,000 (mostly elderly women) where they measured their body fat percentage directly using DEXA scans. It turns out that when you control for body fat percentage, it's actually better for all cause mortality to have the highest BMI category and the lowest body fat percentage. Of course, the measure "BMI after controlling for body fat percentage" is kind of unintuitive, since BMI is supposed to be a crappy proxy for body fat percentage. But basically, as long as your body fat percentage is low, being bigger is healthier than being smaller.
Thanks for posting this! I wonder if part of this paradox is also explainable because people with high muscle mass will tip towards the high end of the scale.
BMI is a terrible metric - it's just a function of weight and height and a terrible indicator for health - most professional atheletes would have high BMIs due to a high muscle to fat ratio (muscle tissue being denser than fat tissue). It's not abnormal to be "overweight" - "obese" at 10% body fat depending on what sport you're doing. (long distance atheletes are normally quite lean, sprinters are bulkier)
It's really not a terrible indicator for health despite this argument getting trotted out every time the subject of BMI comes up on the internet. Obviously, as you pointed out, there's individuals for whom BMI is inaccurate but they represent a very small portion of the whole population. At a population level, it tracks very nicely with health outcomes.
If you're a powerlifter or a sprinter, don't change your habits just because your BMI is high. On the flip side, if you sit behind a desk all day, don't blame a high BMI on the metric being inaccurate.
The metric is very convenient and a useful rule of thumb which is therefore perhaps useful for some proportion of the general population: especially for those with limited medical knowledge and/or equipment - but it's really not a niche minority who is unaccounted for by the BMI.
Muscle density and bone density, amongst other factors, will vary by ethnicity enough that researchers are discussing the value of one singular set of windows for the BMI[0], and although I don't have evidence I would also say that it really doesn't take much exercise for a person to skew their BMI by a noticeable amount - I would venture the top 30% of all exercisers would see their BMI affected by their habits.
So while there is value in the BMI for being easy to collect its data and make recommendations based on its value, in this day and age you'd really expect a more granular and precise measurement for use in research and disseminating health information.
It shouldn't be such a big leap for smartwatches / fitbits to measure body fat %, for example.
> PS, another plus for the Metric System. The Imperial System form includes Feet+Inches, while the metric one only needs Centimeters for the height.
Don't be absurd. There's no reason one couldn't choose to input inches only, or decimal feet, or metres, decimetres and centimetres, or decimal kilometres.
Where did you get your training data set?
I'm trying to build a network to recognize faces from ID cards and match them to faces/selfies. Onenof the hard things has been to get a training data set.
If you click the 'no barcode' button to pull down a fatsecret nutrition record, there is a servings pulldown that is displayed. Its kinda hard to see. Working on the CSS
Ive been meaning to make a blog for some time now, since my nonprofit also has some preliminary research results with the genetic basis of obesity. Just havent had the time really. It will happen though. These neural networks are amazing and worth sharing.
Great work! I'd love to chat with you to help feed your data set outside of wordnet. I've been collecting my own data set where I breakdown the macros in images for anyone who sends me their meals.
I've heard that Google is working on that problem. They have way more resources than my one-man part time nonprofit operation, so I look forward to how awesome their thing will be when they finish it.
Will probably opensource this, since I want to grow up a nice research community around quantified self fanatics.
That is very cool subcosmos! I am also very interested in both cooking+nutrition and AI (see for example my web site cookingspace.com). My contact information is in my HN profile - shoot me an email if you want to talk about AI, food technology and business, etc.
Hey Subcomos, great work! We've been working on something pretty similar, an app called Logameal (only available in the UK). We have run into (are running into) a lot of those UI issues too and would be glad to help. Let me know if you'd like to chat.
Someone should combine this AI with something like Google glass (so you don't have to pull out your phone), a basic calorie/nutrition filter, and a shock bracelet[1].
That way when I look at a Krispy Kreme donut I get immediate feedback!
In some niche areas it's really useful. Some people I know recently raised 17M to build software for surgeons. Looking forward to seeing more uses like this.
This should be in use at those self-registry machines at food markets. I imagine that you place the food item on the scanner and it will automatically register it or ask for human input when something unrecognized is found.
I've been thinking of this exact thing for a few years now, good to see others are interested too. I think there is HUGE potential for an app that can accurately (within a reasonable error range) count macro-nutrients through pictures of food, but I think that is WAY more difficult than you would imagine. How can you train a classifier to tell how much butter is in a dish? How can you account for food that is literally underneath other food?
Maybe a combination of spectrometer, weight, and photos would be a future solution. I certainly don't have an idea of how it would be implemented, but I think using data from all 3 of those sources could be a good step.
Another angle is post-consumption. Ive always wondered if precisely monitoring body temperature might tell you something about consumption, since there is this 'thermic effect of food'. Would be hard to get right though ...
In the lab, we can use doubly-labelled water to precisely measure calorie intake and burn, but its so damned expensive. If only it were commercially viable.
You have a business opportunity here if you train that on fresh vegetables. Many people have trouble with these issues:
1. Knowing what the hell a (example) rhubarb is or looks like.
2. Knowing if it looks fresh or about to go bad.
Being able to aim a camera slowly across an isle with it getting highlighted would solve first problem. Spinning vegetable around in front of the camera would could solve second. So, try that and make it a paid app.
You know, its funny. being a former fat guy, Im not entirely familiar with all of my vegetables. I took this app to a farmers market recently and it was actually helping me identify a few things ...
Had never seen a beet before un-cut. Ashamed of my ignorance ;)
I live in a semi-rural area. Gives me a bit more experience. I still don't know what all these things are in the supermarket, though. So, I can't judge. ;)
That said, you should try training it on different kinds of produce that look very similar. One example I recall is Zucchini vs Cucumber as the stem at one side is the giveaway. Another is Nappa Cabbage vs Romaine Lettuce. Only two examples I could think of off top of head that would confuse people.
I would pay something like $5 per day for this if it was a real person who looks at my photos, categorizes them accurately, and enters them into MyFitnessPal.
I won't trust an AI until it has a massive database and a million users. Until then, it's just not going to work well enough. You also need to convince users why they should use your app, instead of MyFitnessPal (which already has a massive database, including barcodes.)
I have considered something like this as a business model, although I was thinking of nutrition coaching instead of mere logging itself, but thats actually a really great idea.
There's nothing stopping you from having a human aspect do the training and verification behind the scenes.
It's how companies such as Expensify were able to handle their receipt OCR gracefully for such a wide variety of invoices and receipts.
I'd err on the side of saying this is a more palpable sales approach to selling it as AI. Letting your customers know that while AI is running the show, things are being monitored closely and fixed by real people when misclassifications arise.
Ive been running the underlying nonprofit (infino.me) for about 3 years with this ambition to study gene-exercise-nutrition interactions, with very little interest in making any money at it. But this thought of having a paid service for people who don't want to go through the asspain of logging .... this does have me intrigued. Everyone tells me they want an effortless meal logging app ... and photo snaps might be the closest we can get short of having something like google glass persistently monitor intake.
I wouldn't pay $5 a day. I would pay $30 a month. Mechanical turk, with a note that the picture needs to contain some sort of scale (and a request to manually scale the item if you didn't include one and it's not obvious), would probably take you pretty far.
I agree with your suggestions for human assessment because of problem posed by food calories varying significantly due to different preparations.
Take for example soda, if I gave you a picture of soda in a glass could you tell it was diet or regular? You might scoff at such an edge case but it quickly becomes more common when looking into food perpetration techniques. This is why caloric estimation is a really difficult and causes restaurants to not list their calories as the calories of a meal do not equal the sum of it's parts.
All these solutions are common as people want a signal to tell them to stop eating, but these are insufficient as people will simply ignore it due to hunger cravings (as happens on diets). Any nutritionist service in addition to detailing calories would need to incentive the patient to recognize the need to lose weight or setup a helpline.
He may be talking about cooking. This could involve chemical reactions, or facilitate / slow the absorption of nutrients, therefore changing the effective caloric intake from various ingredients.
Thermodynamics should add up once you account for heating, cooling, evaporation, enzymes, waste, etc
> causes restaurants to not list their calories as the calories of a meal do not equal the sum of it's parts.
In my area, restaurants list the calories for each menu item, exactly to the extent required by law (chain restaurants are required to public calorie counts), plus some promotional "under 500 calories" or what-have-you for diet-targeting places.
It would be difficult to account properly for ingredients such as butter and oil which are hard to see in a finished dish and drastically affect the calories.
But people would probably still pay as long as it's a good faith attempt, since they can't necessarily do better themselves, so it would be just as accurate and save time.
And with geolocation, you could start to figure out where people are and then you'd know exactly the calories if they're at a chain restaurant.
Yup! The app as it stands does geolocation and factors this into account for nutrition lookups, but I could certainly do this automatically without requiring a check in.
139 comments
[ 3.3 ms ] story [ 236 ms ] threadUI needs work. Would love feedback on how to make food logging more intuitive.
Things like Main screen, App Icon and colour palette.
Wish you luck with the app, really cool idea!
I'm trying to use fatsecret/myfitnesspal for counting calories. Something like this could make it much easier!
To me, a better workflow would be: take pictures of everything you eat and classify later (at night, maybe on the PC). Of course, this software would simplify the classification, the user would only fix any errors and adjust sizes.
This workflow would be well suited to your app, which operates on a server? (my understanding).
Congrats, I understand the relevance of AI for simple apps now ;)
It would also work nicely for background geolocation services, since if the photograph was taken near a restaurant the service could extrapolate those things.
Anyways, I might make this project opensource and build a nice community around it. Depends on whether or not that model would best support my research goals.
How well does it work on plates with multiple foods "kinda mixed"?
Implicitly, since the app presents a list of possible identified foods, and the user can easily check multiple boxes, thats straightforward, but then having the app pull each of those items AND ask the user to estimate portions for each .... gets a little messy.
So I put that complexity off for v2. That use case in particular would also be best off for a web-interface, I think.
Cant compete with that awesomesauce!
We're working on a similar component for our commercial behavioral-economics-driven app suite, targeted toward patients with chronic diseases.
Do you use location as a way to filter the set of possible foods, as in Google's im2calories project/paper last year?
They do some pretty awesome depth calculation stuff too: https://www.google.com/?ion=1&espv=2#q=im2calories+type:pdf
Edit: oops, should have read further. I see that you pulled appropriate terms from WordNet and used ImageNet to gather training images from Flickr. Cool!
I find myself sometimes not logging food on that app because I couldn't be bothered to search for it.
On the other hand if the product has a barcode I always scan it in because that is convenient.
Also, don't bulk packages of vegetables tend to have barcodes on them inherently?
The more general point is that food in the US has to be at a certain level of quality to be sold, literal barcodes aside.
Sorry about this!
Looking forward to trying the app once I can install it though, looks neat
:(
Email: hack702j002d77s@gmx.us Phone number: +14437633327 ( Text me First if you want me to pick your call )
I wonder if you could use this visual distance algorithm to somehow judge portion sizes/plate sizes?
Aside from that though, I figured Id do portion estimation down the road should this preliminary version take off.
PS, another plus for the Metric System. The Imperial System form includes Feet+Inches, while the metric one only needs Centimeters for the height.
[1] https://www.infino.me/join/
0: http://annals.org/article.aspx?articleid=2499472 (this paper used to be available, because I read the whole thing, but I guess it's paywalled now)
Very cool!
If you're a powerlifter or a sprinter, don't change your habits just because your BMI is high. On the flip side, if you sit behind a desk all day, don't blame a high BMI on the metric being inaccurate.
Muscle density and bone density, amongst other factors, will vary by ethnicity enough that researchers are discussing the value of one singular set of windows for the BMI[0], and although I don't have evidence I would also say that it really doesn't take much exercise for a person to skew their BMI by a noticeable amount - I would venture the top 30% of all exercisers would see their BMI affected by their habits.
So while there is value in the BMI for being easy to collect its data and make recommendations based on its value, in this day and age you'd really expect a more granular and precise measurement for use in research and disseminating health information.
It shouldn't be such a big leap for smartwatches / fitbits to measure body fat %, for example.
[0]http://www.ncbi.nlm.nih.gov/pubmed/19221673?dopt=Citation
Note that for women the low end of normal is probably sub-optimal.
Don't be absurd. There's no reason one couldn't choose to input inches only, or decimal feet, or metres, decimetres and centimetres, or decimal kilometres.
That is checked manually today.
I had do the entire process twice because there is no (obvious) way to change the serving size to 2.
Aim high ...
Will probably opensource this, since I want to grow up a nice research community around quantified self fanatics.
That way when I look at a Krispy Kreme donut I get immediate feedback!
[1]: http://pavlok.com/hello.php
Let's call it: "American Dystopia"
With today's technology we can do it again.
Would decrease cheating perhaps?
Maybe a combination of spectrometer, weight, and photos would be a future solution. I certainly don't have an idea of how it would be implemented, but I think using data from all 3 of those sources could be a good step.
In the lab, we can use doubly-labelled water to precisely measure calorie intake and burn, but its so damned expensive. If only it were commercially viable.
ah, good ole toilet humor
1. Knowing what the hell a (example) rhubarb is or looks like.
2. Knowing if it looks fresh or about to go bad.
Being able to aim a camera slowly across an isle with it getting highlighted would solve first problem. Spinning vegetable around in front of the camera would could solve second. So, try that and make it a paid app.
Had never seen a beet before un-cut. Ashamed of my ignorance ;)
That said, you should try training it on different kinds of produce that look very similar. One example I recall is Zucchini vs Cucumber as the stem at one side is the giveaway. Another is Nappa Cabbage vs Romaine Lettuce. Only two examples I could think of off top of head that would confuse people.
I won't trust an AI until it has a massive database and a million users. Until then, it's just not going to work well enough. You also need to convince users why they should use your app, instead of MyFitnessPal (which already has a massive database, including barcodes.)
It's how companies such as Expensify were able to handle their receipt OCR gracefully for such a wide variety of invoices and receipts.
I'd err on the side of saying this is a more palpable sales approach to selling it as AI. Letting your customers know that while AI is running the show, things are being monitored closely and fixed by real people when misclassifications arise.
Take for example soda, if I gave you a picture of soda in a glass could you tell it was diet or regular? You might scoff at such an edge case but it quickly becomes more common when looking into food perpetration techniques. This is why caloric estimation is a really difficult and causes restaurants to not list their calories as the calories of a meal do not equal the sum of it's parts.
All these solutions are common as people want a signal to tell them to stop eating, but these are insufficient as people will simply ignore it due to hunger cravings (as happens on diets). Any nutritionist service in addition to detailing calories would need to incentive the patient to recognize the need to lose weight or setup a helpline.
That seems to violate the laws of thermodynamics.
Biology is not thermodynamics. Humans are not perfect combustion engines.
Thermodynamics should add up once you account for heating, cooling, evaporation, enzymes, waste, etc
In my area, restaurants list the calories for each menu item, exactly to the extent required by law (chain restaurants are required to public calorie counts), plus some promotional "under 500 calories" or what-have-you for diet-targeting places.
But people would probably still pay as long as it's a good faith attempt, since they can't necessarily do better themselves, so it would be just as accurate and save time.
And with geolocation, you could start to figure out where people are and then you'd know exactly the calories if they're at a chain restaurant.
For me I would be ok with knowing weather the meal was in the 500 or 800 range.