The answer to that is no. A lot of these are released on vibes and showing that it does what it says it does. For most organizations that care about processes and procedures thats... enough? I've worked on agentic development in enterprise and they just want their way to stamp their old methods on to the new agentic world so it can pass through their release hoops.
I've tried to benchmark them the most common public skills including those for ADLC (ie SuperPowers) couple of times here https://orcabot.com/benchmarks/agent-skills. Annoyingly I haven't been able to release one in a while because its been next to impossible to stop the new frontier models from cheating one way or another.
I'm not sure that it needs to be this complicated. But yes, I am interested in the problem of hard guardrails between multiple agents, and it is not an easy problem to solve.
Would someone tell me how this happened? We were the fucking vanguard of agent DLC in this country. The AWS Bedrock AgentCore was agent DLC to own. Then the other guy came out with an eight-agent team. Were we scared? Hell, no. Because we hit back with a little thing called the Bedrock Managed Agents Cloudtrail Datalake. That’s eight agents and a data lake. For moisture. But you know what happened next? Shut up, I’m telling you what happened—the bastards went to nine domain experts, a reviewer and a composer. Now we’re standing around with our cocks in our hands, selling eight agents and a datalake. Moisture or no, suddenly we’re the chumps. Well, fuck it. We’re going to fourteen agents with eleven domain experts.
Sure, we could go to nine domain expert agents next, like the competition. That seems like the logical thing to do. After all, eight worked out pretty well, and nine is the next number after eight. So let’s play it safe. Let’s make a thicker sandbox, add some IOPS and call it the Bedrock9SuperTurbo. Why innovate when we can follow? Oh, I know why: Because we’re a business, that’s why!
You think it’s crazy? It is crazy. But I don’t give a shit. From now on, we’re the ones who have the edge in the multi-agent game. Is Devin the best a man can get? Fuck, no. Bedrock AI-DLC is the best a man can get.
Here’s the report from Engineering. Someone put it in the bathroom: I want to wipe my ass with it. They don’t tell me what to invent—I tell them. And I’m telling them to stick two more agents in there. I don’t care how. Make the agents so token efficient they’re invisible. Put some on an Alexa. I don’t care if they have to cram the fourteenth agent in perpendicular to the other ones, just do it!
This is a riff on this [1] classic for those who forgot like I did.
Earlier this year I spent considerable time making a six-agent multi-LLM configuration in OpenCode with a Bifrost multi-model router before just switching to OpenSpec and GLM-5.3 and calling it a day.
Doesn't this project sound similar to the agent skills discussion? What I mean by that. When skills became available, people started to use it and created sophisticated systems having numerous skills. But as models become better and better there are some sayings that the skills can actually produce worse results than not using a skill and just asking agent to do it.
30 comments
[ 0.24 ms ] story [ 19.0 ms ] threadI've tried to benchmark them the most common public skills including those for ADLC (ie SuperPowers) couple of times here https://orcabot.com/benchmarks/agent-skills. Annoyingly I haven't been able to release one in a while because its been next to impossible to stop the new frontier models from cheating one way or another.
Yeah, I kind of figured. I was hoping was AWS-affiliated lab would be a little more rigorous
My assessment has been just use the native harness.
Evals showed that speed and token use improved when it was removed.....
The models have improved and internalized some of these principles.
They still have value, IMO, because they help improve consistency (at the expense of speed and cost).
I check every benchmark site weekly, compare harnesses and models, and its still pretty much just vibes
> 14 agents: 11 domain experts, 2 reviewers, and an adaptive composer
wow, that's a lot of domain experts, I'm convinced
Staying well behind the frontier, just in case.
https://github.com/daitangio/take-ai-control/tree/demo/nello...
branch demo/nello-openspec
Sure, we could go to nine domain expert agents next, like the competition. That seems like the logical thing to do. After all, eight worked out pretty well, and nine is the next number after eight. So let’s play it safe. Let’s make a thicker sandbox, add some IOPS and call it the Bedrock9SuperTurbo. Why innovate when we can follow? Oh, I know why: Because we’re a business, that’s why!
You think it’s crazy? It is crazy. But I don’t give a shit. From now on, we’re the ones who have the edge in the multi-agent game. Is Devin the best a man can get? Fuck, no. Bedrock AI-DLC is the best a man can get.
Here’s the report from Engineering. Someone put it in the bathroom: I want to wipe my ass with it. They don’t tell me what to invent—I tell them. And I’m telling them to stick two more agents in there. I don’t care how. Make the agents so token efficient they’re invisible. Put some on an Alexa. I don’t care if they have to cram the fourteenth agent in perpendicular to the other ones, just do it!
Earlier this year I spent considerable time making a six-agent multi-LLM configuration in OpenCode with a Bifrost multi-model router before just switching to OpenSpec and GLM-5.3 and calling it a day.
[1] https://theonion.com/fuck-everything-were-doing-five-blades-...