These things will have the same flaw as no-code tools.
No one is going to remember how the system works and all those prompt engineers are going to find out that programming languages are well documented, but things like migrations, multitenancy, ... aren't.
Good luck when an AI api implements a breaking change in an API and people rely on it.
Or when a issue happens and it can't find logs, ... ( If it was even implemented :p )
The article starts out as if it's headed for "and that's how we did it". But no. There's no implementation.
"Imagine a virtual team of AI agents, each with its workflow’s own specialism, collaborating to solve problems and make decisions just like a human team would."
OK. Where does that go?
So far, multi-agent systems have been delegating simple and well-bounded tasks, such as "fetch the weather info for Outer Nowhere" or "check airline schedules for flights from JFK to ORD", or even "what is 25% of $50". Those are questions inexpensive to answer, and don't need much management. If the subagents are complex, they will need management, and probably budgeting.
Subagents need to know when to stop and when to approximate. If the subagents are themselves generative AI systems, there's potential for hallucination at the lower levels generating info that the higher levels take as valid. Subagents also need to be able to query their managers - "is this enough detail" is a reasonable question to pass upwards. They may need to talk to their peer agents.
Now you have all the problems of organizational dynamics within a multi-agent AI system.
I look forward to reading papers with titles such as:
- "Teams of generative AI agents for coding - scrum or waterfall?"
- "Span of control - how many subagents should an agent manage?"
- "Does the agent org chart influence the solution too much?"
- "Resolving disagreements between specialized subagents".
That's where this is going. It has to. Once you start to cut a problem into pieces to be handled by different units, all those problems arise.
Its a goal looking for a problem, aka AI hype. It’s the big thing now, like NFTs before.
I agree with your blog, the definition itself is vague, and what people want to get out of it as well. There is a big “we’ll figure it out when we get there” attitude it seems to me.
Imagine a society where everything can be and is done by AGI (and its drones) what then? What do we want out of it? What will define humans in such an environment?
People push back on the Crypto/NFT <—> LLMs comparison but I think it’s spot on.
Crypto failed because it could never be honest about its trade offs and the reasons for centralization in the first place. It couldn’t be honest about its flaws because then VCs and private equity would make less money.
LLMs as an immediate panacea is failing because it can’t be honest about what actual intelligence is, where human-computer-interaction is, and its ultimate goal of culling jobs. It couldn’t be honest about its flaws because then VCs and private equity would make less money.
If only there was some pattern involved here we could avoid the pitfalls of these hype cycles - the pitfalls that end up in a whole lot of people being worse off and a couple new Lake Tahoe vacation home purchases.
You say no. I will have to disagree. Not in the fuzzy sense of "emergent = intelligent" people seem to be using. But in that LLMs are under the hood doing complex linear algebra, and when you pair numbers with words (or token word fragments), something like human language seems to emerge.
LLMs have no concept of language, just matrices. The fact that we can map language to matrices with training, and then matrices to language with generative transformers, doesn't mean there's anything like intelligence going on. It's just pattern matching all the way down. But a surrogate of NLP emerges out of this.
It's wonderful to have new problems to solve. I'm looking forward to Robert's Rules of Order finally being considered a networking protocol between autonomous agents.
those agents might end up with the same solution we came up with - endless meetings and then make the simplest piece of crap possible to meet half of the requirements.
Maybe this is why multi agents are not the answer, or at least not in an individualist proto human sense.
But does every agent have to be completely separate or could this kind of process land somewhere in sub-surface level processing within a single model?
We are already there in some ways with node clusters handling specific subjects at the lowest levels - even being standard human is like being a large collection of processes all running in parallel.
> But does every agent have to be completely separate or could this kind of process land somewhere in sub-surface level processing within a single model?
Some agents won't be LLMs.
Wolfram Alpha (now with ChatGPT) is close to this type of system.
> The productivity benefits perhaps take us closer to the aspiration Keynes had when he wrote Economic Possibilities for our Grandchildren in 1930, in which he forecast that in a hundred years, thanks to technological advancements improving the standard of living, we could all be doing 15-hour work weeks.
Well, you see, the benefits have to be split between capital and labor.
We have a higher standard of living than when the book was written. Even the basics have changed a lot:
> It wasn't until the 1930s that new houses were built with indoor toilets and bathrooms as standard, says Zoe Hendon, head of museum collections at Middlesex University's Museum of Domestic Design and Architecture. "At that time, bathrooms were seen as a luxury."
Capitalism isn't about standards of livings, it's about exploitation of the majority by and for a minority, whether you give the exploited shackles or golden shoes.
Capitalism, The Wealth of Nations, the observation that people making the best choices for themselves is often good for the collective benefit of society, because this reduces waste.
A century later, communism, The Communist Manifesto, was the observation that in practice capitalism is, much like its various predecessors, yet another way for minorities to rule over the masses.
Unfortunately, the utopian attempts to replace capitalism with communism have thus far demonstrated exactly the same problems that capitalism has.
A century later, we got Nash game theory, formalising the tragedy of the commons, the prisoner's dilemma, etc., — I suspect "the next big thing" will be based on this. (Scare quotes because it may already exist: communist thought had precursors before the Manifesto, and there was a big gap between the publication of the Manifesto and the Russian revolution).
Anything economic related is ruined on this board by what sound like ignorant 14 year old children on economics. Ignorant 14 year old children with an axe to grind with daddy.
Most people's parents weren't around in 1930 to see the publication of the book saying "in 2030, we will be able to enjoy our current (1930) lifestyle with only 15 hours of work each week" (or whatever the actual quote in the book was).
If anything, that prediction was pessimistic — if you really want a 1930 lifestyle, you can probably do it with 5 hours a week.
In 1930, fewer than 10% of farms in the US had access to electricity; that kind of land is still cheap.
Mocking responses like yours seem to keep missing how bad 1930 living conditions were: such a lifestyle is with one where the average person had no electricity and no indoor plumbing, no phone service, no internet, no TV: sure 5 hours per week is sufficient for that lifestyle.
You want land commensurate with that lifestyle? Sure, OK. $20/h (IIRC, the average American is about $43/h) * 5h/week * 52 weeks in a year = $5200, here's three options in the US where you could buy that in a year, which is much faster than what you need to pay off a house today:
What do you live in on this land? Again, I want to emphasise that the average person in 1930 lived in conditions that Westerners would no longer accept including no indoor plumbing, no electricity, no TV, no internet, no phone (mobile or landline).
And even adding $15k for the house to $5k for the land gets a total of $20k — a four year cost for someone working 5h per week on half the US average hourly rate.
The current ratio of USA wages to house prices is $495,100 to $77,463/year, or 6.4 years.
The WebArena leaderboard[0], which benchmarks LLM agents against real-world tasks, shows that even the best-performing models have a success rate of only 35.8%.
I'm curious about the limitation? Wouldn't a team of AI agents that is well designed complete such tasks? I assume the challenge would be in coordination and autonomy?
All of these require manipulating browser DOM/vision, best one is at sub 40%. (On mobile, will take too long, but I'll check later if anyone wants a paper link)
> You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.
> Alas, that does not remotely resemble how people are pitching this technology.
This assumes its consequent, and that's it. I think we're all against wrong AI hype. Vast assertions like "(they) claim AI replaces thought" require a wee bit more fleshing out, too abstract.
29 comments
[ 7.7 ms ] story [ 77.5 ms ] threadhttps://en.wikipedia.org/wiki/Multi-agent_system
No one is going to remember how the system works and all those prompt engineers are going to find out that programming languages are well documented, but things like migrations, multitenancy, ... aren't.
Good luck when an AI api implements a breaking change in an API and people rely on it.
Or when a issue happens and it can't find logs, ... ( If it was even implemented :p )
"Imagine a virtual team of AI agents, each with its workflow’s own specialism, collaborating to solve problems and make decisions just like a human team would."
OK. Where does that go? So far, multi-agent systems have been delegating simple and well-bounded tasks, such as "fetch the weather info for Outer Nowhere" or "check airline schedules for flights from JFK to ORD", or even "what is 25% of $50". Those are questions inexpensive to answer, and don't need much management. If the subagents are complex, they will need management, and probably budgeting. Subagents need to know when to stop and when to approximate. If the subagents are themselves generative AI systems, there's potential for hallucination at the lower levels generating info that the higher levels take as valid. Subagents also need to be able to query their managers - "is this enough detail" is a reasonable question to pass upwards. They may need to talk to their peer agents.
Now you have all the problems of organizational dynamics within a multi-agent AI system.
I look forward to reading papers with titles such as:
- "Teams of generative AI agents for coding - scrum or waterfall?"
- "Span of control - how many subagents should an agent manage?"
- "Does the agent org chart influence the solution too much?"
- "Resolving disagreements between specialized subagents".
That's where this is going. It has to. Once you start to cut a problem into pieces to be handled by different units, all those problems arise.
I agree with your blog, the definition itself is vague, and what people want to get out of it as well. There is a big “we’ll figure it out when we get there” attitude it seems to me.
Imagine a society where everything can be and is done by AGI (and its drones) what then? What do we want out of it? What will define humans in such an environment?
Crypto failed because it could never be honest about its trade offs and the reasons for centralization in the first place. It couldn’t be honest about its flaws because then VCs and private equity would make less money.
LLMs as an immediate panacea is failing because it can’t be honest about what actual intelligence is, where human-computer-interaction is, and its ultimate goal of culling jobs. It couldn’t be honest about its flaws because then VCs and private equity would make less money.
If only there was some pattern involved here we could avoid the pitfalls of these hype cycles - the pitfalls that end up in a whole lot of people being worse off and a couple new Lake Tahoe vacation home purchases.
You say no. I will have to disagree. Not in the fuzzy sense of "emergent = intelligent" people seem to be using. But in that LLMs are under the hood doing complex linear algebra, and when you pair numbers with words (or token word fragments), something like human language seems to emerge.
LLMs have no concept of language, just matrices. The fact that we can map language to matrices with training, and then matrices to language with generative transformers, doesn't mean there's anything like intelligence going on. It's just pattern matching all the way down. But a surrogate of NLP emerges out of this.
But does every agent have to be completely separate or could this kind of process land somewhere in sub-surface level processing within a single model?
We are already there in some ways with node clusters handling specific subjects at the lowest levels - even being standard human is like being a large collection of processes all running in parallel.
Some agents won't be LLMs.
Wolfram Alpha (now with ChatGPT) is close to this type of system.
Well, you see, the benefits have to be split between capital and labor.
The system is called "capitalism."
Figure it out.
> It wasn't until the 1930s that new houses were built with indoor toilets and bathrooms as standard, says Zoe Hendon, head of museum collections at Middlesex University's Museum of Domestic Design and Architecture. "At that time, bathrooms were seen as a luxury."
- https://www.bbc.com/culture/article/20210407-how-the-bathroo...
A century later, communism, The Communist Manifesto, was the observation that in practice capitalism is, much like its various predecessors, yet another way for minorities to rule over the masses.
Unfortunately, the utopian attempts to replace capitalism with communism have thus far demonstrated exactly the same problems that capitalism has.
A century later, we got Nash game theory, formalising the tragedy of the commons, the prisoner's dilemma, etc., — I suspect "the next big thing" will be based on this. (Scare quotes because it may already exist: communist thought had precursors before the Manifesto, and there was a big gap between the publication of the Manifesto and the Russian revolution).
Anything economic related is ruined on this board by what sound like ignorant 14 year old children on economics.
Anything economic related is ruined on this board by what sound like ignorant 14 year old children on economics. Ignorant 14 year old children with an axe to grind with daddy.
If anything, that prediction was pessimistic — if you really want a 1930 lifestyle, you can probably do it with 5 hours a week.
Mocking responses like yours seem to keep missing how bad 1930 living conditions were: such a lifestyle is with one where the average person had no electricity and no indoor plumbing, no phone service, no internet, no TV: sure 5 hours per week is sufficient for that lifestyle.
(Also, according to this graph, just under half of USA households in 1930 owned land, likewise just under half owned a car: https://oldurbanist.blogspot.com/2013/02/was-rise-of-car-own...)
You want land commensurate with that lifestyle? Sure, OK. $20/h (IIRC, the average American is about $43/h) * 5h/week * 52 weeks in a year = $5200, here's three options in the US where you could buy that in a year, which is much faster than what you need to pay off a house today:
1. 0.17 Acres, IN: https://www.land.com/property/Michigan-City-Indiana-46360/12...
2. 0.66 Acres, AR: https://www.land.com/property/1001-Ash-Flat-Dr-Horseshoe-Ben...
3. 1.13 Acres, NV: https://www.land.com/property/Spring-Creek-Nevada-89815/2031...
What do you live in on this land? Again, I want to emphasise that the average person in 1930 lived in conditions that Westerners would no longer accept including no indoor plumbing, no electricity, no TV, no internet, no phone (mobile or landline).
We won't accept them today, but glorified garden sheds like this are what a lot of people called homes in 1930: https://www.kaufland.de/product/396861319/?vid=396861513
And even adding $15k for the house to $5k for the land gets a total of $20k — a four year cost for someone working 5h per week on half the US average hourly rate.
The current ratio of USA wages to house prices is $495,100 to $77,463/year, or 6.4 years.
The WebArena leaderboard[0], which benchmarks LLM agents against real-world tasks, shows that even the best-performing models have a success rate of only 35.8%.
[0] https://docs.google.com/spreadsheets/d/1M801lEpBbKSNwP-vDBkC...
A zero multiplied by whatever is still zero.
It can not solve anything with one broad category of exceptions as https://hachyderm.io/@inthehands/112006855076082650 brilliantly explains:
> You might be surprised to learn that I actually think LLMs have the potential to be not only fun but genuinely useful. “Show me some bullshit that would be typical in this context” can be a genuinely helpful question to have answered, in code and in natural language — for brainstorming, for seeing common conventions in an unfamiliar context, for having something crappy to react to.
> Alas, that does not remotely resemble how people are pitching this technology.