If I were planning to use A.I. to do what Amazon Fresh does I'd build a human-powered prototype that collects training data and break the decision making process down into small tasks that can be farmed out to different people or automated. Eventually I'd try to automate as many tasks as I could but a project like that might well reach an asymptote where you can't get rid of the people.
It seems like a nightmare to me to send people a receipt later because they could always say I ripped them off and charged them too much, maybe they are right or maybe they are just trying to rip me off. If the transaction is complete in a short amount of time then the window for disputes is less.
Regarding second paragraph, note that the store literally has a dozen high res video feeds from all angles that recorded the entire duration of the visit. There is ample of evidence to prove things one way or the other. Customers caught repeatedly filing fake disputes would simply be banned from entering those stores, which could even be enforced using the same hardware (banned person trespassing detected through AI? autocall the cops). I understand this has a slight dystopian/surveillance state note though.
You're right but I totally wouldn't want to go there. Not sure the police would be in a great to chase after somebody just because an A.I. sent them a bundle of paperwork.
It takes hours, therefore it must not be a computer? I know the author has industry experience with ML and scientific computing, so this conclusion is surprising to me. Big data pipelines, especially those that involve a lot of money, especially especially the ones that are messy and potentially unreliable and have large ups and downs in demand, are not necessarily fast. Maybe there was a bunch of sales and there's a queue. Maybe there's a buffer where they check every few hours to see if inventory matches sales. Maybe they manually validate a couple of orders in each batch before releasing the automated results. Maybe the fleet doesn't autoscale and it's overprovisioned at night but underprovisioned when you're actually shopping. Maybe it sometimes handles the work in large batches, and maybe that job periodically fails due to regular bugs and has to be restarted.
There are lots of "this works but not instantly and not 100% reliably" answers. "It was a mechanical turk all along" is not the only logical conclusion from the presented evidence.
And "usually takes a couple of hours"...isn't very bad? Is it ideal? No. Is it enough to ask "How Has Amazon Come To This?" Also no.
This would be a good excuse for a smaller company that doesn't have the resources but amazon is literally the largest cloud provider in the world. Amazon could scale instantly to deal with queue size no problem and at cost since they own the aws services needed to do so. I don't think queue size or under-provisioning makes sense here at all. Validation and some human element checking inventory matches sales makes sense but that's essentially what the author is suggesting is going on here.
Prompt receipt of a receipt is valuable to the customer. It enables them to promptly dispute mistakes, submit expense reports, allows for return of purchases (valuable also to Amazon: imagine the enormous headaches adapting a system to allow for the return of an item your system doesn't currently believe someone bought yet), etc.
Amazon sends the receipt the moment they charge you. Amazon Go does not do "returns". You just click "request a refund" and they give you your money back. This is more convenient than other stores.
It must cost them some revenue because quick receipts is a much better user experience. The receipt system will inevitably make mistakes and it's much easier for the customer to deal with those before leaving the area and getting involved in something totally different.
I see little correlation between such minor user experience quibbles and sales in US retail.
You can't even tap a MasterCard to pay in any of the top 3 brick and mortar retailers in the US. Nobody is going to walk 2 more blocks past an Amazon Go/Fresh because they want their receipt faster. Or at least, the demographic who would is likely not the demographic Amazon is targeting. These stores are going up in neighborhoods where people are relatively unlikely to look at their receipts anyway.
Because if they made a mistake what evidence would I have to prove otherwise? Sounds like a nightmare waiting to happen, especially given how arcane AWS billing and Amazon account management is in general. What if they billed you for $1,000 worth of groceries?
That's my guess. They wait until the end of the day and see what they have in the store, since that's got to be the easiest of all of this to automate with cameras.
Then they see how far off the AI was for the day and work on error correction and send receipts.
I agree with your overall point that cloud resources probably shouldn’t be a problem for Amazon that factors into the hours of processing time. But is it true that AWS could provide at-cost resources to sister companies?
More of a legal/organizational question. But assuming Amazon Fresh is a legally-distinct entity with service agreements with Amazon Web Services, wouldn’t offering those services at below-market-rate be considered anticompetitive (and also call into question the legal separation of the two entities)?
My understanding internally at amazon is they have the same cloud tools that smaller companies have with AWS. I seriously doubt amazon is keeping some secret AWS sauce for internal-use-only.
The simplest explanation, IMO, is to look at the piece Amazon doesn't own. The bandwidth to send a continuous stream of 4K video from dozens (hundreds?) of cameras from the store to the cloud for processing. That can't be cheap.
Netflix 4K seems to be about 7GB per hour. A camera aimed at a store should be able to get away with much less, since the image is mostly static. But let’s say 7GB per hour.
If all the processing were in the cloud and the data were going in the other direction, AWS might charge 5c / GB, so 35 cents / hour. Peanuts.
But AWS owns the infrastructure, and they don’t charge for ingress anyway. A 1Gbps link might cost $2k per month (wild guess, but not far enough off to matter). That covers 64 streams. This cost is minimal.
Of course, AWS could also stick enough of their processing at the edge to essentially eliminate the bandwidth cost entirely.
> Maybe the fleet doesn't autoscale and it's overprovisioned at night but underprovisioned when you're actually shopping.
It's further worth remembering that Amazon happens to, y'know, own AWS. Since this is not a time sensitive result at all, they could be trying to do it in spare capacity by specifically downscaling when their customers tend to use things. It's complicated to pass that efficiency on to customers, but it's probably super easy to take advantage internally.
Every time my groceries arrive I'm somewhat clueless as to what I ordered, though past me always seems to have made good choices. Checking my recipt after 50 hours would effectively be useless.
With the invention of self checkout I never get charged for an item I didn’t purchase. I have never encountered that issue with a cashier either.
Hell I cannot remember a time when I was _charged for something I didn’t purchase_.
I have been charged MORE than advertised but paying attention while ringing up the item and escalating to the cashier or manager typically solves the problem.
I have also been charged less and notified the cashier of the issue.
Why would you check your receipt _after_ you left the store? Now you have to drive back, go inside, go to customer service and wait.
Seems way easier to pay attention and scan the receipt when it is handed to you.
In what world is "knowing how much you spent" not a reasonable design requirement for a supermarket?
Just to pick one user story, it's a lot harder to check / challenge an itemized receipt if you receive it hours later, once you're already packed things away or eaten some of them.
It's easier to challenge an Amazon Go receipt than any other store. I just click the link in the email and get my money. At other stores, I have to go inside and wait in line behind other people at the customer service desk. I prefer to get on with my day rather than argue over the price of a cabbage.
My wife and I tried to get into our local Amazon Fresh. Sign outside said open Amazon app and scan upon entry. We were ok with that idea.
Got to the turnstiles, and the devices there demanded a palm print, no apparent scanners to scan a phone anywhere. We walked out and will never walk back in, especially considering the jackass carting out in the parking lot ridiculed us for refusing to surrender biometric data to give Amazon our money.
No, you misunderstood the interface. While they certainly encourage you to use the weird biometric scanner it's in no way required. However you do have to have the Amazon app installed on your phone. When you open the app there's an option to reveal a QR code which you scan at the turnstiles.
Sounds like Amazon deliberately made a confusing interface to trick people into revealing more biometric data than necessary. I wouldn't say that's any better than just making it a requirement.
I did not. Had the app open, and QR code on my screen as I had previously done so many times before at Whole Foods. There was literally no device resembling a barcode scanner on the turnstile.
PS I also have a ton of experience with barcode printing and reading. As a matter of fact, software I have written has printed over 30M unique QR Codes for track and trace purposes.
Thanks for posting the photo. To my recollection there was only the palm scanner, no white box to the right of the palm scanner at this store. Maybe they were running an A/B test. I went back outside to read the instructions on the signs and they did not align with the reality inside.
The author clearly wants to push it as some sort of 'Amazon faking it bigtime' thing.
I am sure there are manual reviews when it gets flagged, and I am also sure there are mistakes in receipts (I had an error once out of 20 or so purchases).
I have a much simpler explanation of why billing can take anywhere from a few minutes to a few hours: they process the video feed in batches, and maybe even optimizing for cost depending on compute load in AWS. Video processing takes time, and each store probably has hundreds of cameras - if you want to do multi-person multi-object detection at scale on HD feeds from hundreds of cameras at so many stores, the amount of compute needed is probably not trivial. Instead of processing the video everytime after each person leaves the store, it is much more efficient to do it at intervals and process multiple individuals at the same time.
Even if this is true is it a problem? It obviously has limited how Amazon can scale the product, but I don't think its a scandal worthy of a "gate" suffix. It's just a product/engineering issue. I'm sure someone in a meeting said, "We tried an experiment to scale our image recognition technology, but it turns out the problem was more complex than we thought and not worth more engineering investment"
Amazon fresh spent 2 years building their store in my town and then just decided recently to scrap it before even opening. They took up an entire grocery store for 2 years that could have been running the whole time.
It should say "Amazon Fresh (retail stores)" because Amazon Fresh delivery is wildly successful. I can't see the point of having yet another brand other than Whole Foods.
Amazon Go doesn't make any money and doesn't scale.
Amazon palm pay at Whole Foods doesn't work. There's no way to easy way to opt-out of it or change cards, and took WF employees 30+ minutes to figure out how to remove it.
I was also thinking of Amazon Fresh delivery. It looks like the title of this article is actually "Amazon’s cashierless stores: artificial intelligence or major deception?", so the HN title should probably be updated to that.
Whole Foods still has very strange retailing choices (afaik, you can't get standard Heinz Ketchup or Hellman's Mayo at the big one near me - just the organic varieties).
Obviously this is in line with the history of the brand of a fancy, organic- and artisanal- focused grocery. (Does it have better profit margins than other groceries?)
My point was that Amazon could support another retail brand beyond WF.
TL;DR: Buy those at Costco or Trader Joe's (Aldi Nord). Don't waste your money buying staples at WF.
Their target demo is Mercedes or BMW SUV driver (I'm mostly joking): elitist, hyper-self-absorbed, conspicuous consumption Before Amazon acquired them, their prices were relatively absurd and the products were even bougier.
It's also true you can't get some staples at WF because they need to ensure each SKU makes enough profit / shelf area.
The priciest of WFs are the small ones in major metros in mixed use commercial-residential areas that are essentially convenience stores.
WF post-Amazon also discontinues items that seem to me to be desirable, but I guess I'm not an average WF customer.
One positive point of WF is many locations have bulk dry goods that can sometimes be almost sanely priced.
Disclosure: I live ~1 mi from 3 WF and their HQ.
PS: I recall I was walking on the sidewalk crossing a WF parking garage, and out of nowhere, a white woman driving a white Mercedes SUV with the window rolled down tried to physically push me with their vehicle, laid into their car horn, and shouted profanities at me for the crime of not moving fast enough. Makes one consider wearing several GoPros when they venture forth into the world.
> However, self-checkout machines have already reduced queues to virtually zero in many supermarkets
This line surprised me because it isn't even remotely true in my part of the country, except during times when the stores aren't busy. The self-checkout machines almost always have a line and it's often quicker to go to one of the cashiers instead.
People are smart - they'll pick a self-checkout line or a cashier depending on which will reduce the effort they have to expend. So, even with varying shopper preferences for which is intrinsically better/worse, it'll average out to roughly the same effort required across all queues.
Not quite on-topic, but I have to wonder: why go and develop these long-winded solutions when we can come up with obvious better ones? -- order on your phone (which is nice and easy because you can add to your grocery list with a smart home speaker/smartphone list throughout the day) and then it's automatically packaged (in an automated warehouse: https://www.youtube.com/watch?v=ssZ_8cqfBlE ) and delivered?
And for folks who drive to work, have them shipped to their place of work. Places of work make accommodations like have parking for electric cars and cafeterias -- having a place for delivery storage strikes me as an obvious perk that has tremendous benefits for both employers and employees as well as the environment.
Can confirm that one's workplace accepting packages on one's behalf is a big perk. I don't think they'll accept grocceries through, maybe that's the next frontier.
Uh, no. Pickup/delivery frequently delivers my food cold and unpalatable. I'm not going to trust that these folks know how to pick fresh vegetables correctly.
Automated warehouses work well for larger items. The number of small items and also the variety of those small items make it hard to justify the space and equipment needed for an automated warehouse solution for groceries, but I really like the thought. Something like a personal robo-shopper would be very cool.
It depends how this works. Use of AI does not imply that everything is processed in real-time while you're walking through the store. Raw or preprocessed footage may be queued for processing at a data centre. If there are issues it is also possible that 'uncertain' AI results might indeed be flagged for human review.
I only shopped once at an Amazon Fresh store and the total receipt took less than an hour to show up.
Personally I'm not too fussed if it takes a long time, but I'm very interested in how their procedure works if there is an error to my disadvantage.
In my opinion, using humans in the loop to generate training data in a store like this is perfectly legitimate.
If I were building it, I would do it the same way: start out with the whole thing being people watching cameras and labeling training data, and slowly apply software/ML engineering to improve the automation rate.
In fact, this approach is the opposite of putting the cart before the horse: you test the market risk before you solve all the technical problems. The right way to find out if customers want this type of experience is to bring it to market and find out.
Amazon announced in early March that they are closing cashless stores in Seattle, NYC, etc [0]. Seems that technology is not yet mature for profitability.
The author wrote a book about how AI is all fake and its a bubble so consider the source. I don't think you could get further from the truth right now. Basically the analysis here is 1/10 level.
i helped build this! it's not what i do anymore, and i don't want to go into too much detail, but the TL;DR is that, yes, just walk out (JWO) really does use machine learning models to identify which products customers are picking.
the non-obvious challenges of working in this space are that you have to deal with real world constraints that you just don't have to think about in 100% cloud based software. hardware goes down! internet connections go out! electricity goes out! how much processing can you do in store vs out of store?
that's just the tech in the store. what about humans? humans are now in your programming state moving shit around. in-store associates miss-stock items. kids do kid shit. people don't place things back exactly where they found them.
Then you have interesting distributed problems: how do you handle late data? what should be a massively parallel problem is really a graph of interactions that have to be resolved in just the right order so you can generate an accurate receipt.
and you know what's crazy? the vast majority of the time it works! exactly how they say it does: with machine learning models.
From direct experience in my professional career I know for a certified fact this article is just flat out wrong. Amazon Go is precisely what it says it is. I don’t know what causes the long delays now but that wasn’t always the case. I suspect the fact Amazon go doesn’t make any money and is launched means it basically hollowed out and is on autopilot with two SDE2’s on call 24/7 keeping it alive on life support. Amazon services are always well staffed up to the moment they’re launched, because at Amazon you only get promoted on launching something - not making it better.
I don't understand why I'm supposed to be outraged here.
As a customer, I get the experience of a cashierless store. I don't really care how that's accomplished.
As an investor, I might care that I'm being sold snake oil, but I doubt that I chose to invest in Amazon solely over the Fresh stores.
As someone working in the industry, it might certainly affect my perception of what's possible or feasible, but I'm already used to the fact that most of what's being sold as "sophisticated AI" is snake oil.
It just feels to me like this article strikes a bizarre tone about it all, not to mention the fact that other comments in this thread have offered up equally plausible explanations.
IIRC when you add items to your shopping cart, the little display shows what you added?
I think it would actually be harder to have a bunch of people watching and recording your items manually, than the actual system they set up with 4 scanners on the sides of your shopping cart reading bar codes.
73 comments
[ 3.5 ms ] story [ 87.8 ms ] threadIt seems like a nightmare to me to send people a receipt later because they could always say I ripped them off and charged them too much, maybe they are right or maybe they are just trying to rip me off. If the transaction is complete in a short amount of time then the window for disputes is less.
Regarding second paragraph, note that the store literally has a dozen high res video feeds from all angles that recorded the entire duration of the visit. There is ample of evidence to prove things one way or the other. Customers caught repeatedly filing fake disputes would simply be banned from entering those stores, which could even be enforced using the same hardware (banned person trespassing detected through AI? autocall the cops). I understand this has a slight dystopian/surveillance state note though.
...and the recently renovated Whole Foods on Riverside in Sherman Oaks, CA.
Used it myself last week. All the bananas come in 6 banana bundles.
There are lots of "this works but not instantly and not 100% reliably" answers. "It was a mechanical turk all along" is not the only logical conclusion from the presented evidence.
And "usually takes a couple of hours"...isn't very bad? Is it ideal? No. Is it enough to ask "How Has Amazon Come To This?" Also no.
You can't even tap a MasterCard to pay in any of the top 3 brick and mortar retailers in the US. Nobody is going to walk 2 more blocks past an Amazon Go/Fresh because they want their receipt faster. Or at least, the demographic who would is likely not the demographic Amazon is targeting. These stores are going up in neighborhoods where people are relatively unlikely to look at their receipts anyway.
That's my guess. They wait until the end of the day and see what they have in the store, since that's got to be the easiest of all of this to automate with cameras.
Then they see how far off the AI was for the day and work on error correction and send receipts.
More of a legal/organizational question. But assuming Amazon Fresh is a legally-distinct entity with service agreements with Amazon Web Services, wouldn’t offering those services at below-market-rate be considered anticompetitive (and also call into question the legal separation of the two entities)?
If all the processing were in the cloud and the data were going in the other direction, AWS might charge 5c / GB, so 35 cents / hour. Peanuts.
But AWS owns the infrastructure, and they don’t charge for ingress anyway. A 1Gbps link might cost $2k per month (wild guess, but not far enough off to matter). That covers 64 streams. This cost is minimal.
Of course, AWS could also stick enough of their processing at the edge to essentially eliminate the bandwidth cost entirely.
It's further worth remembering that Amazon happens to, y'know, own AWS. Since this is not a time sensitive result at all, they could be trying to do it in spare capacity by specifically downscaling when their customers tend to use things. It's complicated to pass that efficiency on to customers, but it's probably super easy to take advantage internally.
Hell I cannot remember a time when I was _charged for something I didn’t purchase_.
I have been charged MORE than advertised but paying attention while ringing up the item and escalating to the cashier or manager typically solves the problem. I have also been charged less and notified the cashier of the issue.
Why would you check your receipt _after_ you left the store? Now you have to drive back, go inside, go to customer service and wait.
Seems way easier to pay attention and scan the receipt when it is handed to you.
Just to pick one user story, it's a lot harder to check / challenge an itemized receipt if you receive it hours later, once you're already packed things away or eaten some of them.
It's easier to challenge an Amazon Go receipt than any other store. I just click the link in the email and get my money. At other stores, I have to go inside and wait in line behind other people at the customer service desk. I prefer to get on with my day rather than argue over the price of a cabbage.
amazon seller inventory reports may have 2 or 3 days of delay on a bad day.
Got to the turnstiles, and the devices there demanded a palm print, no apparent scanners to scan a phone anywhere. We walked out and will never walk back in, especially considering the jackass carting out in the parking lot ridiculed us for refusing to surrender biometric data to give Amazon our money.
PS I also have a ton of experience with barcode printing and reading. As a matter of fact, software I have written has printed over 30M unique QR Codes for track and trace purposes.
It’s that little white thing next to the palm reader. It even has an Amazon app icon.
https://www-ocregister-com.cdn.ampproject.org/i/s/www.ocregi...
I am sure there are manual reviews when it gets flagged, and I am also sure there are mistakes in receipts (I had an error once out of 20 or so purchases).
I have a much simpler explanation of why billing can take anywhere from a few minutes to a few hours: they process the video feed in batches, and maybe even optimizing for cost depending on compute load in AWS. Video processing takes time, and each store probably has hundreds of cameras - if you want to do multi-person multi-object detection at scale on HD feeds from hundreds of cameras at so many stores, the amount of compute needed is probably not trivial. Instead of processing the video everytime after each person leaves the store, it is much more efficient to do it at intervals and process multiple individuals at the same time.
Amazon Go doesn't make any money and doesn't scale.
Amazon palm pay at Whole Foods doesn't work. There's no way to easy way to opt-out of it or change cards, and took WF employees 30+ minutes to figure out how to remove it.
My point was that Amazon could support another retail brand beyond WF.
Their target demo is Mercedes or BMW SUV driver (I'm mostly joking): elitist, hyper-self-absorbed, conspicuous consumption Before Amazon acquired them, their prices were relatively absurd and the products were even bougier.
It's also true you can't get some staples at WF because they need to ensure each SKU makes enough profit / shelf area.
The priciest of WFs are the small ones in major metros in mixed use commercial-residential areas that are essentially convenience stores.
WF post-Amazon also discontinues items that seem to me to be desirable, but I guess I'm not an average WF customer.
One positive point of WF is many locations have bulk dry goods that can sometimes be almost sanely priced.
Disclosure: I live ~1 mi from 3 WF and their HQ.
PS: I recall I was walking on the sidewalk crossing a WF parking garage, and out of nowhere, a white woman driving a white Mercedes SUV with the window rolled down tried to physically push me with their vehicle, laid into their car horn, and shouted profanities at me for the crime of not moving fast enough. Makes one consider wearing several GoPros when they venture forth into the world.
This line surprised me because it isn't even remotely true in my part of the country, except during times when the stores aren't busy. The self-checkout machines almost always have a line and it's often quicker to go to one of the cashiers instead.
And for folks who drive to work, have them shipped to their place of work. Places of work make accommodations like have parking for electric cars and cafeterias -- having a place for delivery storage strikes me as an obvious perk that has tremendous benefits for both employers and employees as well as the environment.
Bananas, in particular, I need them at a specific point in their ripeness otherwise they go bad on me way too quickly.
Why is everyone so quick to defer thinking to AI? Do we all want to be dumb as fuck?
How many basic skills have been largely forgotten because people have instead learned to lean on technology services like a crutch?
We're all turning into the fat lazy fucks on WALL-E
I only shopped once at an Amazon Fresh store and the total receipt took less than an hour to show up.
Personally I'm not too fussed if it takes a long time, but I'm very interested in how their procedure works if there is an error to my disadvantage.
If I were building it, I would do it the same way: start out with the whole thing being people watching cameras and labeling training data, and slowly apply software/ML engineering to improve the automation rate.
In fact, this approach is the opposite of putting the cart before the horse: you test the market risk before you solve all the technical problems. The right way to find out if customers want this type of experience is to bring it to market and find out.
[0] https://www.geekwire.com/2023/amazon-closing-eight-amazon-go...
This video from a year ago goes into more engineering depth than promotional videos: https://www.youtube.com/watch?v=S5t6aYhj6pU
the non-obvious challenges of working in this space are that you have to deal with real world constraints that you just don't have to think about in 100% cloud based software. hardware goes down! internet connections go out! electricity goes out! how much processing can you do in store vs out of store?
that's just the tech in the store. what about humans? humans are now in your programming state moving shit around. in-store associates miss-stock items. kids do kid shit. people don't place things back exactly where they found them.
Then you have interesting distributed problems: how do you handle late data? what should be a massively parallel problem is really a graph of interactions that have to be resolved in just the right order so you can generate an accurate receipt.
and you know what's crazy? the vast majority of the time it works! exactly how they say it does: with machine learning models.
bonkers.
As a customer, I get the experience of a cashierless store. I don't really care how that's accomplished.
As an investor, I might care that I'm being sold snake oil, but I doubt that I chose to invest in Amazon solely over the Fresh stores.
As someone working in the industry, it might certainly affect my perception of what's possible or feasible, but I'm already used to the fact that most of what's being sold as "sophisticated AI" is snake oil.
It just feels to me like this article strikes a bizarre tone about it all, not to mention the fact that other comments in this thread have offered up equally plausible explanations.
I think it would actually be harder to have a bunch of people watching and recording your items manually, than the actual system they set up with 4 scanners on the sides of your shopping cart reading bar codes.