One thing that teaching hospitals have is a longer ladder of experience. For example many hospitals have medical students, interns, residents, chief residents, fellows and attendings. One idea we contemplated was to have "treatment" start on the cheapest possible model and see how it ended up, only escalating to more capable/expensive models as necessary. We rejected this idea because we suspected that it would end up in more time/tokens on the capable models to fix any problems.
In the end, medical students learn to become doctors. Cheaper models never learn to become more capable models, so the metaphor is not apt.
Did you do any ablation studies on what is actually useful vs what happens to just work because these systems can work around whatever people do?
I compare this to OpenAI's symphony prompt which, at a high level, does exactly the same thing outlined here except model selection and doesn't really have the need for roles or hospital metaphors.
No, we haven't performed any ablation studies yet - it's a good suggestion.
The comparison with OpenAI's Symphony is valid (the two systems do broadly the same thing). One thing that's different about Sinai is that it keeps the code author and reviewer in separate agents with separate context. At the time we built it, we suspected that this would lead to better outcomes, but again, we haven't validated that it does.
I confess that there's a lot more validation we could do with this model but we just haven't found the time. One thing we don't cover in the post is that this is a side project for both of us, so we have less time than we'd like to perform experimentation and validation.
> One thing that's different about Sinai is that it keeps the code author and reviewer in separate agents with separate context. At the time we built it, we suspected that this would lead to better outcomes, but again, we haven't validated that it does.
I use a similar process for a local agent swarm approach (locally hosted Qwen-3.8-27b or Qwen-Flash-Next), used so far for personal-grade projects. Through tool and process accretion, the review stages are told to check both the work and reasoning of the implementation stage, via processing the pi.dev session log.
Through parsing the jsonl log, the reviewer sees the subagent prompt, the tool call sequence, and non-thinking narration along the way. That allows the reviewr to audit the implementer's process (e.g. were tests run?) and spot procedure violations or gross hallucinations. The reviewer also independently runs the test suite, so even a hallucinated pass is caught.
It's relatively expensive both in tokens and time spent running ideally duplicate tests, but the independent workflow has nonetheless caught errors that would very likely have been missed by a same session, same context review.
Sorry about that. The repo is internal to Cockroach Labs for now. It was an issue about adding a progress rollup comment to decomposed parent issues. We’ll get the post updated.
This, with an extremist take on code quality via linting in ci, is no doubt the future, at least for maintenance and extending the interface kind of work.
1. How does this work with greenfield lifts where the scope and final vision are not yet figured out?
2. Sorry if I missed this in the post, but will you open source this system?
The Migration Assistant was mostly a greenfield project and the roadmap was steered directly by feedback we continually received during development. It was built on top of our existing CLI-based tools for migrations, but the web UI and step-by-step wizard with AI assistance was all brand new.
What worked was a human writing the big-picture issue and the hospital decomposing it into a tree of smaller issues, which people could review with more rigor. We also did a few thing that didn't work; in one case we got a bit too over eager in trying to resolve a large slate of issues all at once, and left ourselves with a giant backlog of plans and code to review.
We're discussing internally what can be released in the open, but nothing to share yet.
Could you fix a bug in this codebase yourself, without the help of agents?
Maybe trying this would be a good experiment? Say route 10 or 20% of any future issues to the “Chief” to troubleshoot and fix directly, and record those for further work?
Yes, the codebase still allows you to make a PR the normal way, and getting another engineer to review. That is, a human could make a PR totally by hand or by steering a Claude session on their machine.
I still find myself taking that approach fairly regularly. For most framework-level changes, or changes to core packages, I think the added scrutiny and deliberation is valuable. But for "leaf" components that don't have many dependents, using the hospital to move rapidly has been invaluable. I also encourage "old-fashioned" PRs for anyone who wants to gain a deeper understanding in a certain area of the code.
I like the control-group idea. It’s close to something we’re already building, where a human can label an issue to opt-in to write the plan or the code themselves. Routing a fixed % of issues that way would give us a baseline we don’t have right now.
To be clear, when I said "can you", I didn't mean "does the system allow this", more like "are you, the person I am speaking to who is the architect of the system, confident that you could make a fix to this system without the LLM's help?".
> I also encourage "old-fashioned" PRs for anyone who wants to gain a deeper understanding in a certain area of the code.
Yeah, this is kind of what I meant. I think in some future, these are going to be really valuable for maintainers.
This task seems adjacent to the sort of work agents excel at - there's a language / grammar / specification / examples to follow, verifying correctness mechanically seems like it is possible. So this seems similar to writing a compiler with a specification / test suite available (like the "building a c compiler" thing Anthropic experimented with).
Would you say this is accurate? Have you had challenges generalizing to other sorts of work (you mentioned you're spreading it out to other repos)?
One thing I've been experimenting with in my own agent orchestration system is an agent that hangs around and does post-merge acceptance testing after the work ships. Any plans to add a follow-up phase? Travel nurse?
The closest thing we have is a fuzz testing framework that generates random schemas, drives the Migration Assistant end-to-end using Playwright, has an LLM judge the results, and files issues back into the hospital. It’s run manually, not after every merge.
This is Microsoft scale, I'd be surprised if it made any difference these labs. It's more likely it's plain managerial mishandling of the infra in chasing new profit heights.
I posted in another threat about this: I am seeing a lot of people building their own little bespoke factories. They introduce endless quality gates until it slows development down, and then they add more agents to decide when to run certain actions, and on and on. The end result from what I've seen and personally participated in, is that it often winds up providing negative value in the software development lifecycle. It creates a whole lot of heat, but IMO is not helping the teams utilizing them to ship value any faster than they would with a more limited setup.
I just finished ripping out one of these “dark factory” setups. Removed about 70k lines of code and 750k words of generated documentation. For what is effectively a 5-screen app.
These software factories are just s/code/specs/g. 5000 lines of (agent generated and poorly reviewed) specs for 2500 lines of code. All because the CTO dreams of getting a few likes from Elon Musk.
The closest thing we have is if a merged change has to be reverted, the Safety Department holds a Morbidity & Mortality conference and writes up what went wrong. A patient that needs three or more rounds of rework gets one too. There have been seven reverts so far, so not many funerals.
Since we are all building our version of this, what mistakes did you made until you arrived to a well-balanced solution? I really like that you know how much an issue is, because then you can start optimizing.
I'd say that the first "mistake" we made was having it run in auto-merge mode. It was incredible to see what it could produce, and the speed with which it worked, but while the results seemed good, they were being produced at a rate that we couldn't human-verify. This is not to say that they were bad, but that we had no way to convince ourselves that they were good.
Once we started to review the code in more detail (and put the hospital into "human approval mode" to stop the momentum) we found one major thing that we corrected.
When issues were decomposed into a DAG, sibling issues would often defer work that they thought that their sibling(s) would handle, and that work sometimes just got dropped. This was because there was no way for sibling issues to communicate with each other. We've since added a deferred scope ledger and policy. If any issue is going to defer scope it must comment about the deferral in the code, and write it to a special ledger with the rationale, and how/when the deferral should be actioned. This has prevented several items from falling through the cracks, but we're constantly refining what is a valid deferral and what should be fixed immediately.
There are several other things that we've corrected over time, which makes me think that another blog post is in order. The full list would be too extensive to try and address in the comments section.
Is a code comment and ledger the best way? Should the agent just fill out a form or something and attach it to the sub issue. This is how the hospital would work.
It often does that too, but we keep the ledger and the code comment as well, to ensure that if it doesn't get resolved by the sibling, that it's not lost. If the sibling does resolve it, it's removed from the ledger and the code.
> When issues were decomposed into a DAG, sibling issues would often defer work that they thought that their sibling(s) would handle, and that work sometimes just got dropped.
Could you structure the DAG so that after each node that contains the work, you have one dependent node that verifies each distinct requirement was implemented as expected?
This way if a single requirement is dropped the system alerts you rather than it being silently dropped.
It makes sense intuitively that if a task has nothing that depends on it the LLM might accidentally attempt to drop it (even purposefully as an optimization).
I like this. Today the parent issue closes automatically once every child is checked off, and nothing re-reads the parent’s original ask against what actually shipped. There is a deferred-scope auditor that catches the deferrals an agent admits to. Your "verify node" would catch the ones it doesn’t, which is the more dangerous kind. I think it would fit naturally as one final child per decomposed parent.
You give a very precise measure of redundancy in the skills. Can you give a bit more detail in how you decided you needed to audit them and how you went about it? Was it fully agent driven? Mostly human?
All that credit goes to Rafi. I believe that he either noticed that they were getting long winded in a code review, or suspected that they needed trimming after reading this blog post from Anthropic: https://claude.dev/blog/the-new-rules-of-context-engineering....
It was definitely human driven, but I believe that the agents did the actual trimming.
It's true that we don't have long-term maintainability data just yet. We've just shipped the first product of this model to customers and likely won't have any detailed maintainability data for several months (and for good data, several years). We hope to author more blog posts on this experiment in the future.
This feels to me like a lovely example of both high-quality systems design and information theory. From my perspective, one of the more abstractly interesting thoughts surfaced by this write-up is the degree to which you've "encoded" a highly complex set of relationships through use of metaphor.
That type of encoding (starting with the statement: you're at a teaching hospital) is profoundly more efficient than having to deeply and reliably articulate what each of the various roles your agents embody are, let alone their interactions.
Perhaps there is more room for any of us to consider what existing systems or identities are described abundantly in training sets and can be leveraged to ritualistically encode these social/civic/cultural dynamics saying: "act as though you're ___." Not exactly a new point, but one this write-up certainly is pulling me towards.
Thank you for sharing and the care you put into writing this!
Thanks for the thoughtful response. We too were surprised by how deeply we could take the model of the teaching hospital to software engineering. Whenever we think to expand the system in one dimension or another, the teaching hospital model seems to have a nearby analogy.
I will confess however that some people internally find the model confusing. For example, one user couldn't remember that to get an issue actioned, they needed to put it into the "waiting room". They wanted to be able to action their issues without learning how complex medical systems operate. There seems to be more work on our plates to make this resonate with all users.
Now imagine what it is like to run an actual hospital with real people whose lives (or the lives of their loved ones) are often at stake, with underpaid and overworked staff and with messy biological creatures as the subjects instead of bits and bytes. If anything this whole exercise should also give you a much deeper appreciation of the people that feel themselves called to help others.
I had a similarly useful analogy of the legal system emerge, in a similar way. I think there's a great analogy between policy in the legal system and policy in software engineering, and they have some awfully good (and very, very historical) ways to think about e.g. amending, repealing, and adjudicating things based on those policies.
Funny, right now we have a super bare bones of something like that. When the Chief (a human) makes a decision that generalizes (for example, a project-wide best practice), it goes into a precedent log. Before an agent escalates something to the Chief, it has to check the log, and if a precedent applies it cites it and follows it instead of escalating. It’s been one of the ways to cut down escalation noise but it definitely could be more mature.
I'll be very interested to read any followups about the part you described at the end about getting it to do more "teaching," both from the perspective of how we can use this to help newer devs out but also from the perspective of helping ourselves understand the software being built.
One thing I've been worrying about recently is the notion of comprehension debt and making sure the humans can still understand the system (so as to be able to do incident response or something if the LLMs are down). The slower, more methodical workflow described here should definitely help by allowing checks like proper docs whenever a piece of the architecture changes for instance, but I'm curious what else could be done to help the humans understand as much of the implications of what the AI is doing as possible.
yeah this is really cool. i've been thinking about sort of a mirrored idea, where you end up with wizards, mages, clerics etc and fantasy terminology used. idk if it would be as effective as this is, but maybe it would be more creative somehow?
i kinda want to try to build this off of github. it's essentially just an event bus / message queue that workers (agents) tap into.
i think role-play and utilizing the full power of language, stories, character, and narrative will unlock very sophisticated use-cases and in general a new dimension to agentic systems much in the way you're describing.
probably what will work best in the future is specific training or fine-tuning against curated datasets of narrative fiction and/or texts in general? not really sure.
Might reread this post a few times to unpack what they’re saying in a more practical way.
It feels like everybody is trying to build a lot of the same concepts and converging onto something. I threw my hat into the race, open sourced it, as I’ve been using this workflow for several months on a serious system and it works for me.
We actually have a /sinai-advocate skill, where a human can advocate on behalf of a stuck patient. We use it every once in a while when the labels get screwed up, or a workflow fails for some reason.
I can think of nothing I'd like to use less than a database or filesystem vibecoded by Gas Town-flavored psychosis. Roleplaying with LLMs is not the secret to producing amazing code.
I absolutely love how you are able to pull so much latent behavior from the underlying LLM. I wonder what other analogies can be pulled into agentic coding that come baked into the existing weights.
I’m surprised at the reasoning behind using the most expensive model which likely led to the “doing too much”. I think it would have been nice to have A/B tested different hospital setups in order to maximize not only efficiency but to see where the strength of each model started to emerge in a given role.
Create a “token guy” that others can consult with to figure out the best model to dispatch something to. Token guys role can be defined as an expert in the current LLM landscape. Not sure whose role this is analogue to in medical (somebody here said insurance agent… that is probably getting close).
Token guy’s role gets even better when you let them recommend entirely different hospitals for a given step in patient care (OpenAI models, random shit on openrouter, etc).
I have a “token guy” in my workflow and the dude has looked at underlying tool call patterns, cache use, etc and made all kinds of recommendations to the clients calling it all on their own and completely unprompted—just following a simple role.
I’d experiment with expanding their role into prompt engineer or some kind of specialist on that domain.
We have a small version of this, but as rules rather than an agent. A token guy role that looks at tool-call patterns and cache use could catch a lot of the wastage that we're missing.
I set different kinds of structures for different projects,
Complete with setting and ambiance. It’s like agents work better if they are role-playing. It’s extremely disorienting and people with marginal mental stability are going to really have a bad time. What have we wrought?
If I wanted to build something like this, at least conceptually with the roles, where’d I start? My first guess would be to give Claude your blog post; but any other pointers to make it work reliably? Do you happen to have the system open source?
I am curious why so many of these systems are based on GitHub issues. Why not use a proper ticket system?
I use Redmine (lightly customized) and have found that to work very well. Especially for performing review / UAT of the work or providing response to the agents questions when it wants me to pick an option.
The underlying data model of a ticket / project management system is so rich and well suited to bodies of work.
I don't get your maths? [1] 160/9 months != 600 for any given number of days a week e.g. 5,6,7 (something between 6 and 7), but where did the 9 come from, is it some kind of adjustment for weekends? Anyway there is also a concept of "fully loaded employee cost".
In any case the way to think of this is not "/day".
The reason is simple. If you buy 1000 barrels of oil, you buy 1000 barrels of oil not 17.4 days of oil. There is now a disconnect between work done and time. Infact you would be sane if you said "that result I can get in 2 days for $4172, if you can get me that same result in 1 hour, I'd pay $8344". See where this is going?
Yes a lot of thought work is now a commodity, and if you want the commodity faster (last minute booking, uber to come quicker etc.) you pay more not less. Value being $/hour is over.
True but the pace of buying doesn't matter for this argument (of course it matters for business longevity!). What matters is if you spend $160k on tokens or an employee then ... what gets more done.
For some jobs the employee comes out top, for others the tokens. This landscape will continually shift. However definitely the "per hour" is just some intermediate part of the value calculation can can't be directed compared.
If there were a way to pop out the next Lovable with $160k tokens and you can have that land in 4 hours, would you pay. Well yeah someone would, just like they'd buy $160k of oil if they needed it. It would be even better than paying $80k to an employee who took 6 months.
I have qualms and disagreements with the whole "software factory" concept, but your math doesn't help the argument. The cost of human labor is more than salary—or even hourly rate, if you're thinking of paying a contractor.
Rather than per-day costs, I think it might be more meaningful to look at per-result costs. It took us about 9 months (and $160k using our napkin math) to add support for Oracle. Adding Db2 support using this agent pipeline took 2 days and $4,172 in tokens.
One caveat to the number for DB2 is that it doesn't consider review costs. We weren’t reviewing the hospital’s code during that sprint. Since we’re still confirming correctness for Db2, a more accurate comparison would add the human time to verify this.
I’m not going to claim this tripled our revenue. But revenue is also a function of time to deliver too. When a missing feature in the migration tool is what’s blocking a customer’s migration or a deal, shipping it in days instead of months can move revenue directly if it means the customer doesn't choose a different alternative.
Thank you for the reply. My question is - will you be able to charge the same amount for work done in days instead of months? Or even a proportional amount?
If its work that was blocking an existing deal, I can see this playing out in your favour. Whereas, if you were trying to negotiate a deal where you wanted the customer to pay x$ for a feature you have yet to develop, you might now have to charge lesser.
141 comments
[ 4.4 ms ] story [ 28.9 ms ] threadIn the end, medical students learn to become doctors. Cheaper models never learn to become more capable models, so the metaphor is not apt.
I compare this to OpenAI's symphony prompt which, at a high level, does exactly the same thing outlined here except model selection and doesn't really have the need for roles or hospital metaphors.
The comparison with OpenAI's Symphony is valid (the two systems do broadly the same thing). One thing that's different about Sinai is that it keeps the code author and reviewer in separate agents with separate context. At the time we built it, we suspected that this would lead to better outcomes, but again, we haven't validated that it does.
I confess that there's a lot more validation we could do with this model but we just haven't found the time. One thing we don't cover in the post is that this is a side project for both of us, so we have less time than we'd like to perform experimentation and validation.
I use a similar process for a local agent swarm approach (locally hosted Qwen-3.8-27b or Qwen-Flash-Next), used so far for personal-grade projects. Through tool and process accretion, the review stages are told to check both the work and reasoning of the implementation stage, via processing the pi.dev session log.
Through parsing the jsonl log, the reviewer sees the subagent prompt, the tool call sequence, and non-thinking narration along the way. That allows the reviewr to audit the implementer's process (e.g. were tests run?) and spot procedure violations or gross hallucinations. The reviewer also independently runs the test suite, so even a hallucinated pass is caught.
It's relatively expensive both in tokens and time spent running ideally duplicate tests, but the independent workflow has nonetheless caught errors that would very likely have been missed by a same session, same context review.
1. How does this work with greenfield lifts where the scope and final vision are not yet figured out?
2. Sorry if I missed this in the post, but will you open source this system?
What worked was a human writing the big-picture issue and the hospital decomposing it into a tree of smaller issues, which people could review with more rigor. We also did a few thing that didn't work; in one case we got a bit too over eager in trying to resolve a large slate of issues all at once, and left ourselves with a giant backlog of plans and code to review.
We're discussing internally what can be released in the open, but nothing to share yet.
Maybe trying this would be a good experiment? Say route 10 or 20% of any future issues to the “Chief” to troubleshoot and fix directly, and record those for further work?
I still find myself taking that approach fairly regularly. For most framework-level changes, or changes to core packages, I think the added scrutiny and deliberation is valuable. But for "leaf" components that don't have many dependents, using the hospital to move rapidly has been invaluable. I also encourage "old-fashioned" PRs for anyone who wants to gain a deeper understanding in a certain area of the code.
I like the control-group idea. It’s close to something we’re already building, where a human can label an issue to opt-in to write the plan or the code themselves. Routing a fixed % of issues that way would give us a baseline we don’t have right now.
> I also encourage "old-fashioned" PRs for anyone who wants to gain a deeper understanding in a certain area of the code.
Yeah, this is kind of what I meant. I think in some future, these are going to be really valuable for maintainers.
Would you say this is accurate? Have you had challenges generalizing to other sorts of work (you mentioned you're spreading it out to other repos)?
One thing I've been experimenting with in my own agent orchestration system is an agent that hangs around and does post-merge acceptance testing after the work ships. Any plans to add a follow-up phase? Travel nurse?
It does! It helps getting promotion with tokenmaxxing.
Please do not flag him into oblivion again. Sheesh.
Once we started to review the code in more detail (and put the hospital into "human approval mode" to stop the momentum) we found one major thing that we corrected.
When issues were decomposed into a DAG, sibling issues would often defer work that they thought that their sibling(s) would handle, and that work sometimes just got dropped. This was because there was no way for sibling issues to communicate with each other. We've since added a deferred scope ledger and policy. If any issue is going to defer scope it must comment about the deferral in the code, and write it to a special ledger with the rationale, and how/when the deferral should be actioned. This has prevented several items from falling through the cracks, but we're constantly refining what is a valid deferral and what should be fixed immediately.
There are several other things that we've corrected over time, which makes me think that another blog post is in order. The full list would be too extensive to try and address in the comments section.
Could you structure the DAG so that after each node that contains the work, you have one dependent node that verifies each distinct requirement was implemented as expected?
This way if a single requirement is dropped the system alerts you rather than it being silently dropped.
It makes sense intuitively that if a task has nothing that depends on it the LLM might accidentally attempt to drop it (even purposefully as an optimization).
It was definitely human driven, but I believe that the agents did the actual trimming.
That type of encoding (starting with the statement: you're at a teaching hospital) is profoundly more efficient than having to deeply and reliably articulate what each of the various roles your agents embody are, let alone their interactions.
Perhaps there is more room for any of us to consider what existing systems or identities are described abundantly in training sets and can be leveraged to ritualistically encode these social/civic/cultural dynamics saying: "act as though you're ___." Not exactly a new point, but one this write-up certainly is pulling me towards.
Thank you for sharing and the care you put into writing this!
I will confess however that some people internally find the model confusing. For example, one user couldn't remember that to get an issue actioned, they needed to put it into the "waiting room". They wanted to be able to action their issues without learning how complex medical systems operate. There seems to be more work on our plates to make this resonate with all users.
One thing I've been worrying about recently is the notion of comprehension debt and making sure the humans can still understand the system (so as to be able to do incident response or something if the LLMs are down). The slower, more methodical workflow described here should definitely help by allowing checks like proper docs whenever a piece of the architecture changes for instance, but I'm curious what else could be done to help the humans understand as much of the implications of what the AI is doing as possible.
i kinda want to try to build this off of github. it's essentially just an event bus / message queue that workers (agents) tap into.
probably what will work best in the future is specific training or fine-tuning against curated datasets of narrative fiction and/or texts in general? not really sure.
It feels like everybody is trying to build a lot of the same concepts and converging onto something. I threw my hat into the race, open sourced it, as I’ve been using this workflow for several months on a serious system and it works for me.
https://github.com/pullboard-dev/pullboard
And see the concepts: https://pullboard.dev/tutorials/doctrine/
Fable feels like overkill for this also.
Token guy’s role gets even better when you let them recommend entirely different hospitals for a given step in patient care (OpenAI models, random shit on openrouter, etc).
I have a “token guy” in my workflow and the dude has looked at underlying tool call patterns, cache use, etc and made all kinds of recommendations to the clients calling it all on their own and completely unprompted—just following a simple role.
I’d experiment with expanding their role into prompt engineer or some kind of specialist on that domain.
I use Redmine (lightly customized) and have found that to work very well. Especially for performing review / UAT of the work or providing response to the agents questions when it wants me to pick an option.
The underlying data model of a ticket / project management system is so rich and well suited to bodies of work.
AI Software Factory: $4172 / 2 days = $2086/day
This seems unsustainable, unless you're also generating 3x the revenue.
In any case the way to think of this is not "/day".
The reason is simple. If you buy 1000 barrels of oil, you buy 1000 barrels of oil not 17.4 days of oil. There is now a disconnect between work done and time. Infact you would be sane if you said "that result I can get in 2 days for $4172, if you can get me that same result in 1 hour, I'd pay $8344". See where this is going?
Yes a lot of thought work is now a commodity, and if you want the commodity faster (last minute booking, uber to come quicker etc.) you pay more not less. Value being $/hour is over.
[1] The ? acts as both a question and a regex.
I dont think this is comparable to buying a barrel of oil at all. A software factory like this, I'm assuming is going to be run 24/7.
I get what you're saying, but thats assuming someone's buying their software at the faster pace as well, which we don't know.
For some jobs the employee comes out top, for others the tokens. This landscape will continually shift. However definitely the "per hour" is just some intermediate part of the value calculation can can't be directed compared.
If there were a way to pop out the next Lovable with $160k tokens and you can have that land in 4 hours, would you pay. Well yeah someone would, just like they'd buy $160k of oil if they needed it. It would be even better than paying $80k to an employee who took 6 months.
One caveat to the number for DB2 is that it doesn't consider review costs. We weren’t reviewing the hospital’s code during that sprint. Since we’re still confirming correctness for Db2, a more accurate comparison would add the human time to verify this.
I’m not going to claim this tripled our revenue. But revenue is also a function of time to deliver too. When a missing feature in the migration tool is what’s blocking a customer’s migration or a deal, shipping it in days instead of months can move revenue directly if it means the customer doesn't choose a different alternative.
If its work that was blocking an existing deal, I can see this playing out in your favour. Whereas, if you were trying to negotiate a deal where you wanted the customer to pay x$ for a feature you have yet to develop, you might now have to charge lesser.