Show HN: Nutrient insights through your grocery receipts (nutri.adrianstobbe.com)
Nutri is still in beta and the GPT-powered results are sometimes inaccurate. The nutrient information accuracy is good to get an overview, but there are still outliers at times. I'm looking to improve the accuracy through food databases. Furthermore, I'd like to add additional tips for combining / preparing food to improve its nutritional value. For example, iron absorption is improved through vitamin C, so combine chickpeas or leafy greens with lemon. Or combine beans with rice to get all amino acids.
On the UX side, I'd like to integrate a QR code on the desktop version to easily upload receipts through the phone. Furthermore, it would be great to have analytics over weeks on nutrient improvements over time. Nutri could also be a great accountability partner to track items high in sugar / processed foods.
What do you think?
83 comments
[ 2.9 ms ] story [ 141 ms ] threadOne great resource is also: https://world.openfoodfacts.org/
One feature suggestion would be to accommodate specific diets (keto, vegan, etc.) and generally allow excluding food (e.g. I hate the taste of beet roots and eggplants, so I’d always exclude those).
One thing that’s not clear, is if it understands multiples, I uploaded a receipt [0] with 2 feta, 2 broccoli, and 3 arugula, but I’m not sure if it recognized that? I also don’t know how well that can even work without some calculation check, for this store (REWE) the amount is always below the product, for Aldi it’s the opposite https://i.imgur.com/YvqJCle.png
[0]: https://imgur.com/a/Xhw3Acs
[1]: https://i.imgur.com/YvqJCle.png
The main issue I see though is that if you're going to include processed foods, things will get complicated.
The names for various ingredients change in some countries ie: Sorbitol is also known as glucitol, D-Sorbitol, Sorbogem and Sorbo .
I use this example because Sorbitol is a sugar alcohol that affect FODMAP types.
You might do well having a chat with the Monash University, they have done a lot of work on this and have even published an app for people to check and see what sensitivities particular foods might have.
https://www.monashfodmap.com/
That database might be crucial for your site.
You could stitch all of this together for this product in a day or two.
I see a lot of examples that use receipts themselves like this, but one idea I had that's kind of similar would be to look at just the "ingredients" list on product labels and parse those, like these examples (under the nutrition labels):
Example 1: https://i.imgur.com/MqpL6yh.png Example 2: https://i.imgur.com/3FSK0CD.png
However, using things like pytesseract and Google's Cloud Vision API returns mixed results, sometimes missing things, transposing lines, etc.
Any ideas on what I could do to improve being able to extract ingredients lists from food labels? Would I have to start looking into something like Vertex AI and training custom models?
Then again, as I'm thinking out loud, I realized if these tools can extract all the text pretty reliably, the order and place doesn't really matter if you create some extractor that's able to just pluck out which words are actual "ingredients" based on some master list or something.
If you can get image working via vision that’s great. On the cloud ocr side, I know tooling like Textract is good enough to generally provide output as if you were reading left to right. So in theory the text should not be that transposed or fragmented and nutrition labels are standard enough that you can probably pull the portion you want. On top of that, like you allude to, LLMs are pretty good and figuring things out.
Can you customize target intakes? Say, Food Pyramid / Food Wheel / etc.
And vitamin B12 is crucial...
My kid loves superheroes but every meal has to be accompanied with a story about why that day’s food item is good for him. I wanted to use GPT for this. Start by reading grocery bills, then have GPT understand the bill and come up with stories since it likely knows what health benefit each item in the bill confers.
Assuming the number of meals you make is limited, you can use ChatGPT directly for this. Just ask it to make stories for your meals.
I once started working on something similar, but the goal was to suggest a list of ingredients for me to buy for the week so that I had a good total nutritional values. I don't work on it anymore though.
1: https://fdc.nal.usda.gov/index.html
Also, if it's going to give protein advice it should probably recognise meat - unless it's aimed solely at vegetarians & vegans. My receipt had a whole chicken, a lamb shoulder, and a couple of different kinds of fish - but it said I was getting 0 protein and recommended quinoa.
Cronometer has a feature that does this – it checks to see where you aren't hitting your daily targets for macro/micronutrients and then suggests foods that would allow you to hit those targets. Quite nifty.
I tracked my intake for 1-2 weeks with Cronometer and it helped me see what kind of minerals and vitamins I might be missing. I mean, these things are "ballpark" only anyways, and I wouldn't have the discipline to track my every meal. But good to get a general idea - kind of like a check-up, just doing it from time to time.
Ultimately there arent that many grocery stores and there arent that many popular items, so you'd get an 80/20 approach.
Also curious about the GPT approach, are you worried about the cost exploding with a GPT approach?
You could partner with grocery stores. The user would have an account, not tied to any one particular grocer. It would track all of your grocery purchases, and like your demo, give you nutrient insights as well as recipes and maybe coupons and money-saving tips. Lots of stuff you could do with that.
https://sifter.shop/
people who are conscious about this dont need an app ( its not at all hard to tell food is bad) and ppl who don't care won't use the app.
so there is no market for this kind of an app.
No need to be so negative about something that apparently doesn't apply to you.
It also excludes going out to eat. That being said, if someone is concerned about their health enough to use an app like this, they SHOULD be eating at home more. They only way to know what is in your food is to make it yourself.
Not to mention trying to guess what's inside of those dumplings or ravioli...
I should stop talking because this is too much work for me either way. Cooking is enough work without all this!!
Overall, my groceries are never consumed at the same rate, so how can this possibly track daily nutrition?
If you have good data of all the households purchases, you can do so much good stuff with it, the problem is getting this data reliably.
Am I missing something?
However, I do think the stores likely already have such data about you, if you take advantage of any of their offers to track you.
It's a little disturbing to see this slow normalisation of mass surveillance, one little thing at a time.
Food studies are currently based on recall... and peoples recall is horrible. "What did you eat last Monday for breakfast, and how many grams of it did you eat?" The inputs are garbage and the outputs are garbage. Linking actual food intake to healthout comes would be VERY powerful.