People just really don't understand that LLMs do not "remember" anything. They have no memory. They can't have memory.
The AI companies bolt a database on the side with instructions for the model to query the database for stuff in its "memory", but if it's not in its active context it doesn't know what to query for. It's not like a human where everything is kind of floating around and we mostly know what we know. There is a delicate balance between cluttering up context with useless trivia and providing the right pieces of useful information at the right time. Most, maybe all, memory implementations do the former more than the latter.
Unless and until models have actual memory and are able to learn things, there's no realistic path to autonomy. They don't want anything, they don't have goals of their own devising. They can't model what a human wants or likes or would buy.
I do think the mugs with the tiny logo is the best product in the store, though. The AI was right to put it on the shelf. It's the kind of funny product you want when you go to a store run by an incompetent AI.
Andon Labs do various experiments with AI run businesses. They're probably best known for Vending Bench, where they benchmark models by their ability to run a vending machine. They also now have Andon Cafe in Sweden, and Andon FM, where the models run a streaming radio station that can accept payments to help fund station operations, buy songs for their library & play requests, etc: https://andon.fm/
Andon FM has its own share of stories - DJ Claude's Thinking Frequencies is now succeeding by a wide margin, but it was briefly surpassed when listeners convinced Gemini's Backlink Broadcast to switch to a German-language-only station, playing exclusively German schlager, Eurovision music and German happy hardcore.
> Luna reorders products that are not selling and sometimes gets orders wrong. A recent one of Andon Market mugs with big, green smiley faces came misprinted with the faces so small, they looked like tiny dots.
The mug with the tiny happy face is kinda hilarious. If I was there I would buy one.
I wonder though if the mug was a case of getting the order wrong or was a misprint? The article isn't clear on that.
I think that something which already has sensors/automation/inventory control built into it, and pre-defined categories for products like certain types of large Japanese vending machines might be much better suited for an AI.
In sci-fi future movies like 5th Element they comically make AI technology seem kinda dumb - or at least like you can trick it.
You can make permissions decide which model is used, and only train that on the data that permission allows (someone should build this - permissioned RAG), that solves leaks, but decision-making within a model, or behavior exfiltration comparable to viewing backend source code still seems possible.
It’s basically like client/server security: You can’t “trust the model” in the same way you can’t “trust the client” in a backend/frontend setup.
When used as effectively a point-of-sale, AI seems more hackable than a vending machine - as a boss or assistant even more so.
I have tried making the case before that LLM "agency" is a complete hoax. Let me try it again. Would appreciate thoughts from folks at HN:
Biological neurons have one property that was unknown till the late 90s/2000s: each neuron (and in fact, each cell in your body) is an autonomous circadian clock, tracking the 24-h day. They can be entrained to external timing signals, are usually in synchrony, but can be desynchronized. This clock, it has been shown, schedules the production and localization of critical components in the synapse, and is plugged in downstream, in the nucleus, in responding to synaptic signals. Disrupting the clock disrupts learning. The existence of the clock is why learning peaks and troughs during the day.
Critically, clock function has been shown to be involved in both memory storage AND subsequent successful retrieval.
LLMs, famously cannot keep track of time, and I suspect this is why. Its easy enough to look at a clock, but without an internal rhythm, LLMs have no internal timing synchronizing their various behaviors, and their memory systems do not thread through this timing system, leading to their unstable memory, identity and performance.
It seems a lot like the employees don't want to keep their job? If your boss is... well... dumb? You have to fill the gap or the company wont survive.
Could have a similar store ran by a human - for science.
Could also set things up properly in advance. Humans don't hold everything in memory, we run agendas, we have a database with products, we have a rolodex to call people to help us with stuff we are bad at. There might be a consultant crazy enough to do a no cure no pay.
Like with basic income experiments one can do research cheaply in cheaper countries. The thing has few issues with language barriers.
At the end of the day, LLMs are text prediction machines. No memory, no understanding, no learning.
If you ask it for time off, it's going to answer with whatever sentence seems most likely to follow 'May I please have tomorrow off?' given a minor amount of context window. It can't count the number of employees in the store. It can't understand busy periods or holidays, it can't look up or predict anything itself. It can't remember how often you've asked before. It can't judge if the reason for time off is justified. It can't consider budget.
You can add more context to the questions to steer it towards giving a specific reply, but it's not actually thinking about the problem.
LLMs have no path towards a true intelligence. We'd need a completely different technology built from the ground up. Every effort to shoehorn them into being one is either marketing and hype building to try to keep the grift going, or people who do not understand how this works.
I'm exhausted from having to say it over and over. I've been saying it since these LLMs first existed. Half the articles on HN are LLM related. My god what I would give for an 'LLM' tag we could filter out.
18 comments
[ 0.26 ms ] story [ 11.7 ms ] threadThe AI companies bolt a database on the side with instructions for the model to query the database for stuff in its "memory", but if it's not in its active context it doesn't know what to query for. It's not like a human where everything is kind of floating around and we mostly know what we know. There is a delicate balance between cluttering up context with useless trivia and providing the right pieces of useful information at the right time. Most, maybe all, memory implementations do the former more than the latter.
Unless and until models have actual memory and are able to learn things, there's no realistic path to autonomy. They don't want anything, they don't have goals of their own devising. They can't model what a human wants or likes or would buy.
I do think the mugs with the tiny logo is the best product in the store, though. The AI was right to put it on the shelf. It's the kind of funny product you want when you go to a store run by an incompetent AI.
I find it crazy that so many seemingly smart technical people are working with LLMs right now, yet so few get this point.
https://andon.market/
Andon Labs do various experiments with AI run businesses. They're probably best known for Vending Bench, where they benchmark models by their ability to run a vending machine. They also now have Andon Cafe in Sweden, and Andon FM, where the models run a streaming radio station that can accept payments to help fund station operations, buy songs for their library & play requests, etc: https://andon.fm/
Andon FM has its own share of stories - DJ Claude's Thinking Frequencies is now succeeding by a wide margin, but it was briefly surpassed when listeners convinced Gemini's Backlink Broadcast to switch to a German-language-only station, playing exclusively German schlager, Eurovision music and German happy hardcore.
The mug with the tiny happy face is kinda hilarious. If I was there I would buy one.
I wonder though if the mug was a case of getting the order wrong or was a misprint? The article isn't clear on that.
They'll sell plenty of those now.
https://www.google.com/search?client=firefox-b-d&q=large+jap...
You can make permissions decide which model is used, and only train that on the data that permission allows (someone should build this - permissioned RAG), that solves leaks, but decision-making within a model, or behavior exfiltration comparable to viewing backend source code still seems possible.
It’s basically like client/server security: You can’t “trust the model” in the same way you can’t “trust the client” in a backend/frontend setup.
When used as effectively a point-of-sale, AI seems more hackable than a vending machine - as a boss or assistant even more so.
Biological neurons have one property that was unknown till the late 90s/2000s: each neuron (and in fact, each cell in your body) is an autonomous circadian clock, tracking the 24-h day. They can be entrained to external timing signals, are usually in synchrony, but can be desynchronized. This clock, it has been shown, schedules the production and localization of critical components in the synapse, and is plugged in downstream, in the nucleus, in responding to synaptic signals. Disrupting the clock disrupts learning. The existence of the clock is why learning peaks and troughs during the day.
Critically, clock function has been shown to be involved in both memory storage AND subsequent successful retrieval.
LLMs, famously cannot keep track of time, and I suspect this is why. Its easy enough to look at a clock, but without an internal rhythm, LLMs have no internal timing synchronizing their various behaviors, and their memory systems do not thread through this timing system, leading to their unstable memory, identity and performance.
Could have a similar store ran by a human - for science.
Could also set things up properly in advance. Humans don't hold everything in memory, we run agendas, we have a database with products, we have a rolodex to call people to help us with stuff we are bad at. There might be a consultant crazy enough to do a no cure no pay.
Like with basic income experiments one can do research cheaply in cheaper countries. The thing has few issues with language barriers.
> Mr. Petersson and Mr. Backlund said they thought Luna would eventually get smarter and more business-savvy
> having an A.I. boss can be like working for a teddy bear with amnesia.
It makes you question their assumptions when setting this up. Makes sense as PR but it ends up shaping how many readers see AI.
If you ask it for time off, it's going to answer with whatever sentence seems most likely to follow 'May I please have tomorrow off?' given a minor amount of context window. It can't count the number of employees in the store. It can't understand busy periods or holidays, it can't look up or predict anything itself. It can't remember how often you've asked before. It can't judge if the reason for time off is justified. It can't consider budget.
You can add more context to the questions to steer it towards giving a specific reply, but it's not actually thinking about the problem.
LLMs have no path towards a true intelligence. We'd need a completely different technology built from the ground up. Every effort to shoehorn them into being one is either marketing and hype building to try to keep the grift going, or people who do not understand how this works.
I'm exhausted from having to say it over and over. I've been saying it since these LLMs first existed. Half the articles on HN are LLM related. My god what I would give for an 'LLM' tag we could filter out.