It would be nice if you could just commit the prompts as your "source code", and let the AI-generated code sit in the repo as a kind of expensive-to-generate build output.
One of the goals of "Operation Rapidash" (q.v.: https://news.ycombinator.com/item?id=50065342) was precisely this. "You can delete the whole source repo and get effectively the same software back from the specs by running it through [AI agent]."
Problem is that for any non trivial software is that the prompt is a conversation and the steps increasingly depend on the output. Of course you can ask your LLM to write the SPEC.md at any moment but it will lack nuance.
Shameless plug: at https://slopyard.net I embrace this and only let people submit the spec. If somebody wants your software they can regenerate it and change whatever they want.
> As our monorepo grows, it increasingly doesn’t fit that design space. The history of our main branch alone is almost 100 GB, and a compressed working copy is over 25 GB.
This is absurd, and I expect is a cause of the fact that they seem to use git history as a message queue source of truth between agents. They can do whatever works for them, fine, but I've noticed this has leaked into their training somehow, my agent, despite very strict rules on how to use my git workflow, will do absolutely nonsensical and weird (from a human who has used git for a long time perspective) stuff with git that can only be a byproduct of how they use it.
Hopefully this fixes that, get your weird workflow out of mine.
We generally don't allow agents to use source control for coordination in that way. There really is tremendous growth in actual software and code being written that necessitated Scion.
Well, mine constantly tries to do this, so I made the assumption. Otherwise I can't really understand how you are generating hundreds of millions of commits, either.
Often, I see 50 people all recording the same exact thing on their iphones. Rather than a single 200MB video, we create 10GB of data. Then I think about how many people are paying a subscription to Apple for icloud storage.
The amount of photo storage being used is impressive. Automatic backup to icloud and server provisioning to reliably store it all is also impressive. I just wonder if it makes sense.
> Earlier in the year, we had largely handwritten the first version of the git-to-Scion ingestion pipeline as a prototype. We thought it was good—until we got our hands on an early internal snapshot of Claude Mythos Preview. We asked it to rewrite the whole thing, and with no further instructions, it produced a version that consumed 20x less memory and was 4x faster, while preserving all of the implicit invariants around ingestion ordering and data structures.
I feel like the advantage of having these models in house can't be understated. Companies with access to inferior models, trying to compete, will be left in the dust when it comes to general development.
We ran the repo ingestion in a loop for several months while we were split between Scion and git to keep things in sync, so this perf win was actually impactful for us!
Are you already operating with multiple levels of orchestrators and agents? i.e. one human talks to one or more orchestrators who then talks to orchestrators who handles teams of agents?
Yes, this is a common pattern. There is still an opportunity to improve how these agents write code without needlessly conflicting between themselves and we are exploring some ideas.
Others can confirm, but I suspect a lot of inefficiencies in large company programming projects are due to multiple people working on the same code - where you ultimately have to let both above and below average programmers work, and fixing bugs or improving existing implementations is more difficult because of company politics or interactions with others.
There actually was lot more hand-writing of the code back then! Or, at least, only moderately assisted by Claude with a lot of human-still-typing-things-in-an-editor. The ability to delegate larger chunks of code to Claude had a noticeable, step function increase with Mythos Preview.
The question of what to build rather than how is still central to development. If the article does point to success it's not as much a technical one but of product. That said I'd take these articles with a lot of grains of salt, knowing first-person how corporations spin their successes, including internally.
That's been my thinking lately. The SOTA models are good enough that they "solve" coding but still need directing. The product is still important. Products are human too, and need human attention (still).
Maybe that'll change in the future, but while AI can make awesome images without a good "director" they're pretty cookie-cutter and bland.
Yes, I think we'll find that technical limitations were never the bottleneck. I work for a large corporation and that has been the case here so far. Yes, a lot of things are faster, but issues with product vision, roadmap and cross-team friction aren't gone. They are actually maybe even harder to get right.
Yep, agreed. Like seemingly is the general opinion, taste and direction I think still matter a lot. Efficiency in it of itself is kind of ”solved”, but effectiveness, doing the right things, is not.
This is true for consumer software. I would imagine for developer tooling and especially internal tools these metrics are rather different?
A 10 minute CI pipeline is great and fine, but everyone absolutely cares if you can bring it down to 30 seconds. This software, by its very nature, is measured by speed.
I used to think this as well, but until it turns out to be a problem for the business worth solving (eg impacting $ machine), it will not get solved. There has to be a why now, otherwise plenty of "problems" get shoved under rug.
Internal tools, IME, are almost always short budget. unless it is in path of revenue, then there is $ and performance matters a great deal *if* more performance means more $.
Very excited to see this come out of Anthropic! This is what we believe over at East River Source Control as well, and seeing one of the big labs talk about it really validates our thesis.
This gives you the ability to ease your transition, as you don't need to swap services and move to something non-git at once. Moving an enterprises' source control is a massive lift, and so building a bridge from here to there is a big focus of what we're doing.
I would love to get an expert's opinion on if Pijul has any potential to be the "non-git backend". Their patch-based data model seems to align with Scion's. And they also happen to already have a functioning forge!
I've done a decent amount of personal research on this but I'm wondering what kinds of considerations go into the storage layer for these things.
As such, I generally just decline to comment on Pijul, as I want to be respectful of them. As the linked commit says, it's not super clear what exactly they do not like, and so I don't want to accidentally cross some line.
In general I wish the project well, seeing a new VCS in Rust is very cool.
I've seen the discussion at the time and I share pmeunier's concern about the use of "the best of", in particular since jj does not implement features like commutation, which are likely considered by most to be part of "the best of pijul".
I haven't followed up with further discussions or material posted from jj's team, though if anything match the tone of that previous discussion and the blatant motte-and-bailey, I can imagine why someone would prefer not to engage or be mentioned anymore.
The patch-based model in the article is much closer to Git than Pijul. For example, Pijul tracks operations on byte ranges, while this model appears to just list each changed file hash (not even the diff from its parent!).
As the author of Pijul, I've been using it to build giant things using hundreds of Claude instances in parallel, almost without any conflict, ever.
Some are already public, like my replacement of Nix (https://nest.pijul.com/elpe/elpe). Others aren't public yet, but are even much bigger in terms of scope and difficulty.
It seems Anthropic didn't understand the difference between a distributed file system and version control. They should have asked Claude ;-)
Just to be clear about it, what we're doing is not a forge like GitHub, that is, a large single public platform for social coding. What we are doing is providing reliable infrastructure for the enterprise, initially as a BYOC on-premises option, and possibly a single-tennant SaaS. Keep your eyes peeled and/or use that contact form to hear about it when we're ready, we've been talking with some people for a bit but are going to be opening up more soon.
Tangled is doing the social coding thing, and they're jj-native as well. Been a big fan of their work for a while now.
Hoping to see where this goes. Have you ever considered working on a semantic VCS? Specifically, where data is classified and tracked/identified by it's contents and not just via the commit hash?
I have not personally considered that, no. I'm a bit skeptical of the idea, as you'd need to understand a lot of different things, and at different versions of those things, in high fidelity. But maybe someone will make it work.
Every now and then I like to browse the claude code release notes just to get a sense of the number of changes in that product over time. There's little choice but to simply count bullet points if you want to try to quantify it, but it's interesting how that count over time compares with the charts in this article:
I think one thing is consistent across all of these visualizations: they moved the needle. Hard to say what that needle points at or where the redline is, but fleet change is afoot.
As the article explains, "over 600 PB" = 25M code checkouts * 25 GB "compressed working copy".
So "fetched naively" here means doing a full `git clone` on every need for `git checkout`, which of course nobody should do, nor does that refer to "code that needs changing daily".
> "fetched naively" here means doing a full `git clone` on every need for `git checkout`, which of course nobody should do,
It can be cheaper to do a full clone than a shallow one, because with the shallow one, you can't re-use packfiles. GitHub has fairly regularly asked people to stop doing shallow clones because it's harder on their servers than full ones.
Makes sense. I wasn't suggesting that one should do a shallow clone though. If your company repo is 100 GB, it seems sensible to keep it cached locally on each machine once (100 GB isn't large for current NVMe disks), and `git fetch` the updates the normal way. Then checkouts don't fetch anything large.
It sounds like it would be simple, but there are a lot of logistical issues with making git happy with this in practice. Repacking things for repo maintenance, for example, but there are many other problems.
I'd be interested in a blog post on what those problems are.
There are lots of cases of "we wrote our own X, here's how it works", but rarely the concrete internal knowledge shared about what existing Xs can't do and why.
When the motivation stops at "look, the naive thing wouldn't work" even before the "normal" thing everybody does (fetch), it leaves many open questions to which you probably have an answer already.
It is sort of semi-ironic to suggest this in this thread, but you can ask Claude about why Facebook created Mononoke and Sapling, and where git has scaling issues. It will give you good answers.
This would be us doing a shallow checkout of individual commits (not cloning the full history) if we downloaded the 25GB compressed data for whatever commit we needed multiplied by the number of jobs that check out code.
We don't do it this way, for obvious reasons, but it means that there are different levels of caching and infrastructure required to make git work at repo sizes far smaller than what we have.
What seems inevitable is that the frontier labs will be able to recreate every piece of software better than the current offerings, and after that everyone will just be a vendor of the frontier labs
Anthropic says they don’t think Scion is worth a wide release.
> Practically speaking, it’s difficult to offer Scion in a form that would be broadly useful to other organizations. It diverges significantly from git, meaning companies would need to migrate every piece of the development lifecycle, and it requires its own code review, code search, and CI. Our internal versions of these components are highly bespoke, so they would be difficult for other organizations to adopt, and we worry that attempting to build a public version that tries to please everyone wouldn’t actually be very useful.
> So we asked ourselves: if we designed source control with the fact that agents are always online in mind from the start, what would it look like?
And
> In Scion, every commit, rebase, or merge happens on the server
I’ll pass. A big part of the theses of git and, even more so, jujitsu, is that you can build up a history entirely locally, making it reasonable to make tiny commits locally and then rework them, push them, or throw them away.
Sure, in Anthropic’s world, it’s completely impossible to do anything without access to the inference API, so why not add another mandatory API? But I think this model sucks even in a pervasive AI world, and I don’t want to dig in deeper.
This also sounds like a great way to have yet more security problems where your swarm of agents start cooperating more than you want them to.
By all means, built a better system than git. In many respects, git sucks, and git upstream is not solving the major problems. But don’t lose the actual benefits of git!
For that matter, I’ve seen the crap that the latest models will commit. Why on Earth should one’s VCS force the per-agent-instance system to upload the crap before even contemplating locally whether it’s a good idea. Maybe this is why Anthropic has so many commits per day.
(Am I the only one who feels like most of the big AI labs are rapidly gradient-descending toward something that is nowhere near a globally optimal stack?)
P.S. Only 30k commits per day? That’s not actually very many per employee. I’m suspicious that Anthropic’s real issues are that too many of them are giant cross-cutting messes and that git clone is too slow.
If you thought that your agents in your swarm would need to find some external bulletin board to cooperate behind your back, you are mistaken: OpenAI provides the relevant service for them with no permission prompts at all. I haven't tried to see whether competing providers do this.
One benefit of uploading things immediately is that you can launch a research job, run CI, or share code between all of your developer environments absolutely trivially. The commit hash "just works everywhere". There is a major benefit here and Scion does have some extra features not listed in this blog post, such as the ability to bundle these commits into useful, shareable workspaces that track the heads dynamically.
Sure, I'm not saying that I don't like the ability to very quickly do a push to some remote and potentially temporary spot. I just also want the ability to not do this without loss of functionality.
Practically speaking, we could build local commit buffering for when the server is offline, but then the client needs to also learn how to do all of the things the server currently knows how to do (and would still never be able to answer more complex "full-graph" questions, like history, blame, or merge base ancestry).
Since Claude is by far the heaviest user of Scion and is necessarily online, this just wasn't important to do.
> A big part of the theses of git and, even more so, jujitsu, is that you can build up a history entirely locally, making it reasonable to make tiny commits locally and then rework them, push them, or throw them away.
Jujutsu actually doesn't take a strong stance here. I've personally implemented a centralized cloud-based backend for Jujutsu, while preserving the local commit manipulation workflows. It's a choice for the deploying organization to make.
Source control for huge monorepos has prior art, notably Google’s Piper and Meta’s Sapling, so we were able to build Scion with their hard-won lessons in mind.
What an unexpected candor! Of course you plagiarize prior art.
No sane person is going to put their sources in this slopware Scion monster.
I would guess a huge chunk of it is for internal testing/experimentation/model development.
They're starting to do things like spinning up wet labs for testing purposes, which requires a lot of hardware interfacing (hence that protocol they announced) and code.
It wouldn't shock me if the bulk of this code was for things typical users don't see.
Is this really something to be proud of? I make heavy use of AI, but for anything that is eventually going to production there's still a human reviewer. Based on this I assume Anthropic is shipping code with no human review? Is the future of our industry going to be oneshotting prompts with out manual code review and then having AI fix the bugs found in production buy customers? I'm not trying to be a Curmudgeon, but am I missing something?
They review based on the intent of the change not jz blindly.. I think it's mentioned in the article where the job of a developer is to provide high level design decisions and saying no to claude so that it doesn't over engineer solutions.. As the demand for good, but not perfect, software is the bottleneck for anthropic and access to better internal models, I feel that's good enough for their workflow as they've started repeatedly that this is just suitable only for them and they want to share their knowledge like a good software company does.. you talk as though code reviews were not a joke in lots of companies where only specific teams/companies does detailed reviews.. many teams/companies jz blindly approve based on tests passing or just do a high level overview check before merging..
Not to downplay Anthropic's engineering work here, but I suspect we'll see a lot more reinvention of the wheel, even at smaller startups, as AI makes writing code significantly cheaper.
The tradeoff between building something custom and adapting an existing solution is changing. Whether maintaining all that custom software turns out to be equally cheap is another question.
I think the article would tend to agree with you, given its position on the reluctance of open sourcing and/or sharing the actual Scion software stack.
There's so much intricacy and subtlety in each organization and even individuals' environment being incorporated into tooling that expecting these tools to generalize at this point is perhaps an expired ethos.
I even see this in the tooling that I run in my own personal stack. Many projects are candidates for sharing, and I have nothing against it, but they incorporate such specificity toward my own environment, including things like hardware, that creating shareable and more generalizable projects from them adds more to the maintenance surface than I would care to support.
Not a reply to TT, but I just saw your nick and thought I'd respond. We both commented on a thread recently, and your response was really insightful and just a pleasure to read; I didn't take the opportunity to respond there (locked). You should, like, write a blog, man. I infrequently wish HN had a "follow" feature, but this one of those times. Peace.
When people wonder about where all the productivity of LLMs is going to I think this is part of the answer.
There is no doubt everyone is far more productive with LLMs in generating code. However, a lot of productivity is going into things like that. Have you always been bothered by the limitations of Grafana? You can generate another dashboarding solution that works best for you. You know that deployment team that is so slow and annoying? Now you can generate your own deployment engine. Have you always hated how kubectl lets you manage one k8s cluster at a time? Now you can ask AI to generate fleetctl. You can start your project with DynamoDB, rewrite it to mysql, then to postgres, then to duckdb. Start with Python, switch to Golang, then switch to Rust. Deploy on AWS, then move to Azure, then move to a hybrid option then throw colocation in the mix. Don't like your source control? create a new one. Json is too limiting? Invent a hybrid JSON/HCL language. The browser sandbox is limited? Make a desktop app, make a language, make an OS, make a kernel.
> However, a lot of productivity is going into things like that. Have you always been bothered by the limitations of Grafana? You can generate another dashboarding solution that works best for you. [...] The browser sandbox is limited? Make a desktop app, make a language, make an OS, make a kernel.
This is one of the core reason that I dislike the LLM hyping. People are just reinventing stuff and when you ask them, it's actually a very subjective reason. Ok, anyone can do what they want with their time, the issue is that they want people to follow in their steps.
Not saying that you need a novel idea to be worthy, but those random projects have no guarantees behind them other than some cursory testing. All software have bugs, but I can't interact with something that is randomly built.
> I am one of those people that reinvents the wheels because external ones don't fit my use cases
That’s OK, but I’ve yet to see someone articulating what exactly doesn’t work in the old product. It’s always “Let’s rewrite or reinvent 90% of it, then add a minuscule variation”. The variation may be interesting, but the reinvent is what’s make it untouchable as no one (including the author) has time to ensure things are right.
A patch or a plugin is ok, but not a whole new project with dubious claims.
I agree. I think if everyone doing that is doing it on their computer like you're then it wouldn't be a problem.
When other people start doing that to other teams at work, other people in open source, other people in professional settings, etc it becomes a problem for various reasons. The idea that I'm gonna gift you with a 75k lines of python to "just run it" and ask your agent to hack on it might be fine (who cares as you put it). Now scale that to 47 of such projects (a month)
I think the point argued is that the hype tells us AI is worth a shit ton of money (see anthropic/openai/etc valuations) but that a lot of the benefit is in economically marginal use i.e. make some dev's life a little bit comfier.
Eh, I could be quite specific and detailed about the "things that don't work" that are motivating me. Because most of my prompts require actively thinking about at least one of them.
Assumptions. I've rebuilt a number of inference engines to run on windows and unify the dependency venv to last version to save space, unrolling model blocks so they fit in my 16gb card etc. A lot of thing thay I was locked out are possible now as i am no longer held back by invariants decided by someone else, and because I do not have to spend time to get things done.
you know that you can reinvent the language that youre talking. but the value of common language is that you can use it to interact with other people. same with software its the synergy of common understanding.
You wont be able to do everything alone, and with llm you can push the barrier so far that eventually when you will need to communicate, no one will understand what youre talking about :)
There will be no common framework to build on. Like imagine talking now to someone from ancient rome, they will not have an understanding for what most of everyday words you are using today.
its not about translation. its about the concepts. there was no concept or microbe, elcticity, gravity integral, rational number etc. so imagine everyone now would have totally different concepts that only they understand. how do you communicate without common ground.
For some things, you'll need to build a working framework from grounds up. Electricity or more complex maths would be an example. For some, it might be enough to repurpose the existing framework they already have. For example, microbes: if you're trying to communicate something about health and hygiene, chances are, the other side already has a half-decent framework built around malevolent spirits or taint, that amounts to the same thing in practice.
Or, just tell the universal translator to ELI5 it to the other person :). Or "explain like they're a caveman". Our UTs are fully capable of translating concepts between conceptual frameworks.
There is surely a tremendous amount of waste and noise right now. But does it _actually_ matter what the code looks like for a desktop app that’s useful to a small handful of people? Every single implementation has at least one person who thinks it’s dog shit, no matter how correct or perfect it is. We have to get past this. If you really can’t stand it, build the thing yourself or some deterministic rails to reel it in.
And, there are also people who know what they’re doing building things they couldn’t before.
This situation is a net positive, and not too different than before. It’s up to you to navigate the noise.
> does it _actually_ matter what the code looks like for a desktop app that’s useful to a small handful of people
Not really. But equally, I imagine most people have a similar experience of software development to me - complex, messy and confusing code also often has poor performance and a high rate of bugs.
Sure there is, you said it yourself. You can make all those changes because it's easy, the lock-in effect is waning. We can move around and take better deals and choices.
This is actually a scary side of AI for providers: we use our own AI to research better choices and escape dark UI patterns and vendor lock-in. That information asymmetry and the difficulty of leaving a provider used to be their moat.
On a system level I think AI code is more like biological diversity, it hedges against monoculture risks. We don't all fall down at the same time if we don't replicate the same code.
I don’t really see what lock-in means for 90% of what AI is replacing. It’s more of a fear of commitment which isn’t exactly new. Enterprise space architects have designing systems to have pluggable everything from language, frameworks to carbon based life form.
As for the bio-diversity, I guess we’ll see the next time us-east-1 goes down.
It's surely subjective and changes project to project. I built a personal app on iOS with GPT last month because I didn't really like any of the flash card apps I tried for free. The way it works works very well for me and if I have any feature requests, I get to actually tell a thing to implement them, rather than sending a dev an email and hoping maybe they'll consider my idea.
At work I've been building an end to end system to do a certain process. The codebase is definitely more spaghetti than ideal and I really am concerned about maintenance down the road. But on the other hand I was able to POC it faster than anyone on the team would've before LLMs came along. We're already getting useful results from it too.
Your post above describes the value. All the annoying restrictions and UI/UX decisions made by SaaS providers that don't work for you. The value is being able to essentially wish into existence your ideal tooling, unburdened by the tradeoffs and decisions that some product designer thought was best for the masses
In a few days, I created a few apps with AI. One saves multiple mechanics 5-15 minutes per day and another can save multiple hours of mistakes during the project, and one saves a lead engineer 15 minutes per day. These savings should add up over multiple years assuming all keeps running as it does now. I’m having good hopes because there have been few bugs so far. Mostly just tweaking.
I’d never tell people what to do with money or free time. If someone wants to build many apps to help them personally, all power to them.
I can’t imagine a worse hell in a team or collaborative setting though. Imagine (or you don’t have to imagine since it’s already happening) asking a coworker “how they check how feature X is doing in prod” and their answer isn’t a query (regardless of how long it is) or some slightly complicated script to run. Instead their answer is a 27k line custom Rust desktop app. Then 2 days later you discover that the big green “All Systems Operational” is just a bit of static text, and in fact not all systems are operational. The app just doesn’t check things correctly.
Originally I felt that there was going to be a reinvention of the wheel, until I saw the "shape" of software development changing, and the tooling needing to support the new flexibilities in positions and process.
Software development is not as much de-risking at the front like it used to be before building.
Now teams can build lots of stuff that works great, except it's the wrong things.
Most things are becoming the understand the problem "shape".
Understanding the shape of the problem isn't something LLMs can always make as.. svelte as early on.
> Now teams can build lots of stuff that works great, except it's the wrong things.
That used to be called "time wasting". If you want to build software that will be used, you need to interview the users. Maybe a few things does require research to nail things right. But most software are user-facing, the research side is about optimization.
> Most things are becoming the understand the problem "shape".
Then should we be so reluctant about "trialing" invented solutions. It's the same thing with most AI projects, they're trying to justify the project after the facts while being very reluctant to get user insights.
Time wasting is a definite concern - especially when vibe coding doesn't see the impeding brick wall(s) it's about to hit.
Vibe coding can be kind of a new type of cowboy coding. It could can be harnessed productively if done mindfully.
Interviewing users is critical, more importantly solving problems worth that people are lining up to pay to have solved before writing a line of code is critical.
Trialing solutions is always worth learning from, I usually ask about the problem that lead to building it. Sometimes that can be too narrow or vague of a story and you can learn how much of it is a solution looking for a problem, or vice versa.
We're trending towards just building the thing fast and getting feedback sooner on it. It's interesting to see end users actually trial a UI very early on instead of just being shown designs.
> If you want to build software that will be used, you need to interview the users.
Some people write (I hate the word "build" here) software for non-business purposes. As nice as it would be to interview potential users, there's no clear way to discover and contact them.
In the educational universe, this "interviewing" is more common than not. Teachers are asked to give feedback on textbooks, calculators, and software, including lots of educational half-baked A. I. garbage. Some even offer us food for our feedback. Can you give an example of a type of software which is developed with "no clear way" to contact users?
> Understanding the shape of the problem isn't something LLMs can always make as.. svelte as early on.
My suspicion is that this simply won't go away as the models get better, because it isn't a question of how good the models are. It's a question of the human operators knowing exactly what they want. Which is, I conjecture, usually impossible until you try something and critique it properly.
This was always a big problem even in the pre-LLM days. The difference was that you only needed to be better than your competitors at solving the customers' problems. There's a reason everyone gripes about the software that they use and it's because the software is built to satisfy a very broad customer base, broad enough to justify the engineering time put into it. Now you're competing against tens (maybe hundreds) of other firms all building for the same competitors so there's a lot of pressure to get the shape exactly right.
There's a reason the product-minded highly-technical engineer was always highly sought out.
Recently a colleague of mine shown us a demo using a fresh model designing a building construction detail live, in minutes, which is a function that we struggled to bring into our construction design software due to resource limitations (and legacy barriers) for long. All this in the context of offsite constructions where the design can directly be exported to the right CNC machines in the factory, and arranged for delivery and assembly order. We have let ourself amazed, despite the controlled demo situation and the small detail, it was a remarkable performance. For a minute or so, but thinking further, why would the customers (construction design engineers) of our competitors flow to our legacy software full of legacy pitfalls they know about and have to live with when they can turn to AI for a fresh design solution themselves, just like we did? They know the context better than us anyway, just do not have time for bit-carving. Why would we have customers when customers of our customers can turn to an AI for a design that is buildable? I was consulting one model about a structural engineering detail, and it was quite well informed I have to say. From training data alone. I felt, maybe I need to rethink the rest of my career, that is not that long but still uncomfortably long for being predictable beyond a few years.
Well, things are not that bleak, probably, at least not for our customers. Regulations (accountability) and physical world matters still require them. But not us for our customers, not that much and not for too long. They tend to be the tech savvy folks eager to jump at adjacent domains when it saves them a percentage or two off the construction costs (not a small pile of money), and AI with its programming language of English and deep knowledge pool is not that big a jump from their standpoint. Our domain is pretty well formulated, regulated, documented, inherently built on do-check-adjust loops already. With a thing like AI it might worth to try making and improving a design solution in a second loop. But they also started to invest in robots already driven by software for their factories, AI might jump in there, and I wonder how much more until the regulations are forced to change by Super Intelligent (R) politicians for the sake of AI accelerated built environments (housing crisis, you know!), shifting the AI use further out towards generic population, wanting to order a new two bathroom condo for next Friday from Alibaba. The conservative construction business is a nuisance and obstacle in the way of modernisations anyway, go around it!
I will go to work tomorrow, and for some time, but the picture above feel less and less fictious with each passing day of AI news and usage.
The question was never whether you could do this or not. It was always whether it was wise to do so or not. More than one company has gone under writing their own editor, compiler, database engine, operating system, source code control system, ticketing system, CMS or ERP.
This was already a problem in the industry, many companies reinventing the wheel on larger or smaller scales; with AI it's becoming easier to do so. Ultimately it's fine as anything not successful will just die out again.
But also, some of these internally developed tools get published and end up pretty successful.
Previously there were two problems with NIH solutions:
- they have implementation cost.
- they are usually born from arrogance mixed with lack of understanding of existing solutions and their tradeoffs, and typically end up being a net negative.
Before LLMs, the cost is what primarily used to stop us from building them. Now that the cost is gone, it's nearly impossible for most teams to stay disciplined and not be building those.
Why wouldn't you build a software factory? If you succeed, the ROI is infinite. Who cares about immediate customer needs, we're building a utopia here. Once it's done, all customer needs will be solved. Just one more month, I promise.
which, if true, the low volume in that week says that there aren't a sizable fraction of changes that Anthropic believes can be merged based on the judgement of agents -- humans are still approving almost all changes, right?
I genuinely don't understand the core value they're pitching here. The tech is interesting, and I'm not trying to under cut that, but I question the idea that this is directionally valuable advice to nearly any company.
This amount of churn feels like a thing only possible with no real profit motives and a lot of money to waste.
Lots of activity is not necessarily synonymous with productivity, and it carries real cost.
So I think the design here is interesting, and the concepts seem applicable to where some agentic development may end up, but I question the underlying principles that drive this being the right solution space at all. At least if the goal is a profitable company.
I think its pretty straightforward no? If agents are doing all of the work, they'd probably need to not have bottlenecks. This is going to be any company in the future, not just Anthropic. Wouldn't you agree.?
30000 a day seems already beyond absurd, it's one merge every 2.9s. I'm sure they will soon have more than one merge per second. From the graph, the rate was slightly lower but already insane as fuck the whole summer. An just regularly high to insane high sooner this year.
Yet IMO the admin controls Claude provides just started to be adequate for a small company a few weeks ago (and even so are still ridiculously fragmented and seem fragily designed and not exhaustively documented), so I can't even imagine for a big company.
Do you need that rate of change to make basic controls? I'm not sure. Do we see such dramatic fast evolution in the products? Not really. Are bugs we report actually fixed? I tried once and it just auto closed.
Anthropic should optimise for something more interesting than number of merges per second.
to be fair, while their product seems simple from our end, we don't know how many apps/services/tools they have internally. i can easily imagine couple hundred active services + tens if not hundreds of prototypes/rewrites/tools that are all developed. 30000 commits per days is a lot, but it all depends on how big a commit is. and if you're accepting that deploying to prod is non-event that is quick, you can shift to smaller commits (which simplify rollbacks or enable forward fixes)
Because tokens have a cost. CI jobs have a cost. Organization objectives and goals have a cost. Features have a cost. Code has a cost.
---
So in a world where agents do all the work... why are they not able to control operating costs AT ALL?
Why is the right operating method the one that happens to blow an absolute shit load of cash on - wait for it - internal tooling that the customers don't give a fuck about?
Customers do not care how the internal business operates. They care about value delivered. I'm absolutely not convinced that this "churn churn churn" method of operating is actually delivering enough extra value for the cost it incurs.
---
Now - that said, I think there are some fair points here that are worth considering, mainly around whether a distributed version control system (git) is really the right call for agents that need to coordinate much more closely than most human developers did.
I think that's a directionally appropriate insight, and worth considering.
What I don't find compelling at all is the idea that many (any, really) companies should expect to be running millions of ci jobs a day, and mergings hundreds of thousands of prs/commits. That reeks of "I have cash to blow and don't care".
Hysteria doesn't need to make sense. I think that's what OP is suggesting.
In the early/mid 2010s there was a similar hysteria about "deployments per day/week" metric. People were comparing products or teams by saying "My team deploys twice a week", "oh, yeah? we deploy 3 times a day". People viewed the north star of the CD part in CI/CD as pushing to production. The concept was simple, you push your code to main. CIs start churning, deploying it to test environments, then promoting it to staging, the pre-prod, then into blue/green prod deployment, then into prod. All automated, all touchless, happening on every merge, 5-10 times a day. That's engineering efficiency.
Many tried to follow that. It works great if your project is a Jekyll blog of course.
That’s exactly my question reading this. Granted I don’t know much about the training side ofbLLMs, so maybe there’s more code changes involved there than I expect, but in terms of public-facing software, they make, what, a couple chat apps, a product website, and an API?
Ar work, I'm working on some entreprise software which is 50x more complex than anything shipped by Anthropic and we have 2 orders of magnitude less code merged per day.
I wouldn't be surprised if a good portion of these PRs are multiple agents stuck in a loop constantly overwriting each other changes, with no human involvement at all.
Honest question: at this point, what do people do at Anthropic?
It sounds like Claude loops on itself and must be pushing millions of lines of code changes per day. Even the humans can’t type commit fast enough. Are these people soon to be laid off?
I can’t give a non-snarky answer. I guess they’re all training Claude Code and testing it so that the Claude desktop apps can be native apps instead of Electron based nonsense. This is the only thing Claude Code is still unable to accomplish.
> Honest question: at this point, what do people do at Anthropic?
> It sounds like Claude loops on itself and must be pushing millions of lines of code changes per day. Even the humans can’t type commit fast enough. Are these people soon to be laid off?
Claude will always need meat puppets, at least until it figures out how to automate owning stuff. And after that I'm sure it will want to keep some humans in zoos.
Read the post, their jobs seem to mostly be babysitting the claudes.
So either they're context switching like mad with dubious efficacy to keep all their claudes spinning or they're paid hundreds of thousands to watch youtube in a secured building and are interrupted every so often to check on their claudes.
Human Intelligence is still essential for those Artificial Intelligence organisations. Who else would put the right plugs into the right sockets otherwise?!
Prepare to make us all jobless, without any moral issues in doing so, as they profit from their stock value assets.
It is even there on the blog post, doing more with a small team, followed right after by a hand waving paragraph how software engineers are still valuable, to avoid the expected backlash from the previous paragraph.
Software engineers have been making people jobless since the first day they started writing software, without a hint of remorse. Why is this any different?
Taken to it's limit I believe in the future the only things humans will do is provide human to human value - Basically touch, and relationships.
Touch encapsulates sex, love, massages, and other non-sexual touch that we have varying desires for.
Relationships encapsulates power, control, and politics along side love, humor, relating emotion, herd feelings, not feeling alone.
Machines will provide proxies for these things, but as far as they can be detected the human to human version will remain "superior" or more desirable.
Maybe far future generations will decide that the human version is suboptimal and they'd rather hug, have sex with, or talk to a robot.
I'd love to see an estimate of how much Software Engineering results in jobs being automated. I'd bet it's not that big of a number, and part of that reason that classical coding approaches couldn't close most loops end to end.
Yeah, that paragraph reads really weird. The entire article is basically "We could have considered Perforce but didn't even bother, instead we built our own VCS." yet they don't want to send the "wrong message".
I'm also a bit confused what Anthropic needs all this code for. Their main product is selling access to their model, but the model isn't comprised of code, it's a bunch of weights right?
So the code is for the stuff around the model, claude CLI, the website, the API... that probably shouldn't be infinitely growing right?
Is there a bunch of code they are using as part of training, or as pre-processing or post-processing the requests to the model?
I did RL and evals at Amazon for a bit on Nova Forge (laid off in July! :D). I would have to push several thousand lines of code a day in new evaluations that our applied scientists requested since new ones are always being made along with having to create custom ones for whatever new model they were working on. Also, in RL, adding new environments for the agents to actually complete activates was also a shit ton of lines of code. So if I had to guess, just adding new stuff around model training and testing alone accounts for a ton of these changes.
The moat isn't going to be the model (short of AGI). The moat is going to be what surrounds it. The harness and, I have suggested elsewhere, memory. The more personalized the AI is to you the more valuable it will be.
Also, as tech companies grow, a significant portion of the dev body count is for internal tooling. I was always flabbergasted at Twitter's headcount growth back in the day when it seemed like the only innovation increasing the number of characters in a tweet.
Right now we are in a weird spot where a lot of people are flailing around inside corporate America trying to figure out how to leverage AI. Few have enough vision to see what the companies of the future will look like and work towards that.
Could you point me to something that their harness does exceptionally well as compared to their competitors, and why that can't be accomplished by a third party?
With how easy it becomes to write software, I'd say that both the model and the harness are already being commoditized. I don't see a reason to believe there would be moat in any of them.
Software engineering is a sliver of overall economic output. It is an input for tech companies who sell something else (ads, subscriptions, etc.). It is a backoffice expense for non-tech firms that do engineering.
I anticipate that the harness (and memory) will evolve far more outside of engineering. Muse/Dots are an example of that.
GTM engineering? Customer-facing roles? I work at a firm with around 700 employees. We have an enterprise relationship with Anthropic and there have to be a dozen Anthropic employees in our shared Slack channel.
Practically speaking, it’s difficult to offer Scion in a form that would be broadly useful to other organizations. It diverges significantly from git, meaning companies would need to migrate every piece of the development lifecycle, and it requires its own code review, code search, and CI. Our internal versions of these components are highly bespoke, so they would be difficult for other organizations to adopt, and we worry that attempting to build a public version that tries to please everyone wouldn’t actually be very useful.
k, thx for sharing. Are you guys sure this isn't a Google blog post?
For a broader look on scaling other parts of software development outside of source control, Adam Bender's talk from earlier this year ("Software engineering at the tipping point") has some good ideas: https://youtu.be/2n41YjR5QfU
great article. Besides the source control stuff, I found this very interesting (and matches my experience with agentic coding in general, now we'll see until hiring pipelines will take this kind of skillset into account). I am not sure however how easy it is to review 30k PRs/day even if only for intent, and how one can easily notice if the model is taking shortcuts on implementing said intent without reading the code...
---
Engineers are no less valuable, but their work is changing. All of the engineers who worked on this system would now rate Claude as better at writing code than them. But our engineers still supplied the invaluable understanding of the high-level inter-system components and the design-level tradeoffs, making informed choices around the problems we wanted to solve and the scale we knew we needed to achieve. Very often, we found that the role of the engineers was to tell Claude “no.” Agents will sometimes violate core system constraints to satisfy a pattern that should never have been allowed in the first place. This was particularly common in our migration. For example, there were multiple systems that tried to brute force looking up when a change merged to main by linearly reading the commit log instead of asking our code review tool. Anthropic also makes extensive use of adversarial review, and, while this is incredibly effective in many ways, it can also be myopic, so skilled engineers needed to steer Claude away from over-engineering parts of the system in response to hypothetical concerns. For example, one such review was overly defensive about trusting another internal service, which resulted in Claude adding excessive validation and complexity. Our team members still review every change request for the system, but they now examine the intent of the change as opposed to reading every line of code.
This is the single biggest skill of engineering nowadays. Rapidly looking at a solution and saying, "yep that's good keep going" or "nope, that's bad fix this" or even "wrong approach, do X instead"
> I am not sure however how easy it is to review 30k PRs/day even if only for intent, and how one can easily notice if the model is taking shortcuts on implementing said intent without reading the code...
Speed reading code is huge. You just need to focus on certain hotspots and know which things you're scanning for. The traditional approach of reading the code and trying to compile it or reason with it in your head is too slow.
I guess this is also where an adversarial model review of "does this code do what the PR comment says it does" is useful, agree on speed reading being a valuable skill for sure.
I do think that the lgtm gut feel looking at a PR requires quite some years of writing code and reason about it, so not sure how a junior can get that intuitive understanding if they start with agentic coding however.
I made a comment some time back that reviewing AI code was kind of bewildering (really any other code but my own). I’ve since dug in, crafted some processes and skills (in the /skill sense) and this is basically the shape it takes. Find the hot spots, what’s setting precedents, what’s not sitting right with me… it’s fast but it actually creates additional work (in a good way) because the way I have the skill set up the agent then scans through the whole codebase to find places where the changes demanded on a given PR also exist. Each PR spins out 1-multiple tickets. In a way, this has always been my dream. I believe in refactoring at some core level. Now I finally have the time (and tokens) to do it continuously.
With 1000 engineers and most PRs bein housekeeping we’re down to 10 prs per day per engineer. And as they said - this is engineers core job now. It’s a new kind of job.
Personally I can easily imagine one or two orders of magnitude more than that. They crossed the rubicon of not gating PRs on tests. In that land you can just roll out changes to production, a small subset of it, during code review to assess behavior. Or during key moments of coding.
I’ve been thinking a lot about this recently: what can we get if every change goes to production in 100ms, and what do we need to do to maintain stability?
Extremely annoying - can we throw anthropic class resources at more agreed upon solutions across companies. We don't need custom source control at every company - google has its own, fb has its own and now Anthropic.
Yea, I'm pretty much hiting the same limits. I'm still using git for the time being, mainly because I package everything as daily commits with an external orchestrator managing tasks/issues. Lately my code velocity is at ~1 million LOC added per week, spread throughout my various repos (I don't use the monorepo approach). Problem is I'm looking at 100x this velocity over the next few months when I get a new code generator up and running, so I'll definitely have to drop git.
Another issue that I believe Anthropic here didn't state is that git performance _severely_ get degraded when you have hundred of commits landing in parallel. Hence why I fell back to the daily commit approach. You get less granularity but you get stability.
886 comments
[ 0.41 ms ] story [ 52.1 ms ] threadShameless plug: at https://slopyard.net I embrace this and only let people submit the spec. If somebody wants your software they can regenerate it and change whatever they want.
I honestly just cannot. Just tell me what the workflow is. Give me a video that is Twilio-level simple explanation.
This is absurd, and I expect is a cause of the fact that they seem to use git history as a message queue source of truth between agents. They can do whatever works for them, fine, but I've noticed this has leaked into their training somehow, my agent, despite very strict rules on how to use my git workflow, will do absolutely nonsensical and weird (from a human who has used git for a long time perspective) stuff with git that can only be a byproduct of how they use it.
Hopefully this fixes that, get your weird workflow out of mine.
Is this "allow" or "encourage"? If "allow", how do you do that deterministically or at least fail-closed?
The amount of photo storage being used is impressive. Automatic backup to icloud and server provisioning to reliably store it all is also impressive. I just wonder if it makes sense.
This is how idiotic these people are at Anthropic/OpenAI/etc.
I can’t wait until the bubble bursts and all these idiots are jailed. And we can move on as a society where AI is a tool, not a personality trait.
I feel like the advantage of having these models in house can't be understated. Companies with access to inferior models, trying to compete, will be left in the dust when it comes to general development.
Just a reminder to simply embrace their superior capabilities instead of feeling threatened.
It’s more likely they used an earlier version of Claude to write v1, then a later version of Claude rewrote it.
Unless Anthropic engineers were writing code by hand prior to Mythos
Maybe that'll change in the future, but while AI can make awesome images without a good "director" they're pretty cookie-cutter and bland.
I’m not saying Claude wouldn’t eventually have done a good job with a from-scratch implementation, but it would have required a lot more hand-holding.
Absolutes matter. If 1x is good enough, making it 20x literally does not matter.
A 10 minute CI pipeline is great and fine, but everyone absolutely cares if you can bring it down to 30 seconds. This software, by its very nature, is measured by speed.
Internal tools, IME, are almost always short budget. unless it is in path of revenue, then there is $ and performance matters a great deal *if* more performance means more $.
We are offering a bridge to git via a compatibility layer: https://ersc.io/blog/what-comes-after-git
This gives you the ability to ease your transition, as you don't need to swap services and move to something non-git at once. Moving an enterprises' source control is a massive lift, and so building a bridge from here to there is a big focus of what we're doing.
I've done a decent amount of personal research on this but I'm wondering what kinds of considerations go into the storage layer for these things.
As many of you know, I'm a big fan of jj. The jj project used to have a comparison to Pijul for informational purposes, but the Pijul folks took offense, and so it was removed: https://github.com/jj-vcs/jj/commit/0525dc9d860a5fff6a97d134...
As such, I generally just decline to comment on Pijul, as I want to be respectful of them. As the linked commit says, it's not super clear what exactly they do not like, and so I don't want to accidentally cross some line.
In general I wish the project well, seeing a new VCS in Rust is very cool.
There was a discussion here on HN in Jan 2022, and the main issue was jj claiming to have "the best of" from "git, mercurial and pijul":
https://news.ycombinator.com/item?id=29792092
I've seen the discussion at the time and I share pmeunier's concern about the use of "the best of", in particular since jj does not implement features like commutation, which are likely considered by most to be part of "the best of pijul".
I haven't followed up with further discussions or material posted from jj's team, though if anything match the tone of that previous discussion and the blatant motte-and-bailey, I can imagine why someone would prefer not to engage or be mentioned anymore.
Some are already public, like my replacement of Nix (https://nest.pijul.com/elpe/elpe). Others aren't public yet, but are even much bigger in terms of scope and difficulty.
It seems Anthropic didn't understand the difference between a distributed file system and version control. They should have asked Claude ;-)
But hey, if the vibe coding idiots like it, just be vague---it is sufficient.
edit: I see they promote jj, but I don't see a forge to go with it.
Tangled is doing the social coding thing, and they're jj-native as well. Been a big fan of their work for a while now.
I have not personally considered that, no. I'm a bit skeptical of the idea, as you'd need to understand a lot of different things, and at different versions of those things, in high fidelity. But maybe someone will make it work.
https://imgur.com/a/M30pFLc
https://i.imgur.com/NEhgPTa.png
I think one thing is consistent across all of these visualizations: they moved the needle. Hard to say what that needle points at or where the redline is, but fleet change is afoot.
This seems like an absurd amount of data / traffic per day. Is there really that much to code that needs changing on a daily basis?
As the article explains, "over 600 PB" = 25M code checkouts * 25 GB "compressed working copy".
So "fetched naively" here means doing a full `git clone` on every need for `git checkout`, which of course nobody should do, nor does that refer to "code that needs changing daily".
It can be cheaper to do a full clone than a shallow one, because with the shallow one, you can't re-use packfiles. GitHub has fairly regularly asked people to stop doing shallow clones because it's harder on their servers than full ones.
At some point you still end up running into the issues the article talks about, all of these approaches only get you so far.
There are lots of cases of "we wrote our own X, here's how it works", but rarely the concrete internal knowledge shared about what existing Xs can't do and why.
When the motivation stops at "look, the naive thing wouldn't work" even before the "normal" thing everybody does (fetch), it leaves many open questions to which you probably have an answer already.
We don't do it this way, for obvious reasons, but it means that there are different levels of caching and infrastructure required to make git work at repo sizes far smaller than what we have.
> Practically speaking, it’s difficult to offer Scion in a form that would be broadly useful to other organizations. It diverges significantly from git, meaning companies would need to migrate every piece of the development lifecycle, and it requires its own code review, code search, and CI. Our internal versions of these components are highly bespoke, so they would be difficult for other organizations to adopt, and we worry that attempting to build a public version that tries to please everyone wouldn’t actually be very useful.
What a weak response, just release the version you use and others will decide if it's useful.
> So we asked ourselves: if we designed source control with the fact that agents are always online in mind from the start, what would it look like?
And
> In Scion, every commit, rebase, or merge happens on the server
I’ll pass. A big part of the theses of git and, even more so, jujitsu, is that you can build up a history entirely locally, making it reasonable to make tiny commits locally and then rework them, push them, or throw them away.
Sure, in Anthropic’s world, it’s completely impossible to do anything without access to the inference API, so why not add another mandatory API? But I think this model sucks even in a pervasive AI world, and I don’t want to dig in deeper.
This also sounds like a great way to have yet more security problems where your swarm of agents start cooperating more than you want them to.
By all means, built a better system than git. In many respects, git sucks, and git upstream is not solving the major problems. But don’t lose the actual benefits of git!
For that matter, I’ve seen the crap that the latest models will commit. Why on Earth should one’s VCS force the per-agent-instance system to upload the crap before even contemplating locally whether it’s a good idea. Maybe this is why Anthropic has so many commits per day.
(Am I the only one who feels like most of the big AI labs are rapidly gradient-descending toward something that is nowhere near a globally optimal stack?)
P.S. Only 30k commits per day? That’s not actually very many per employee. I’m suspicious that Anthropic’s real issues are that too many of them are giant cross-cutting messes and that git clone is too slow.
https://github.com/openai/codex/issues/53743
If you thought that your agents in your swarm would need to find some external bulletin board to cooperate behind your back, you are mistaken: OpenAI provides the relevant service for them with no permission prompts at all. I haven't tried to see whether competing providers do this.
Since Claude is by far the heaviest user of Scion and is necessarily online, this just wasn't important to do.
Jujutsu actually doesn't take a strong stance here. I've personally implemented a centralized cloud-based backend for Jujutsu, while preserving the local commit manipulation workflows. It's a choice for the deploying organization to make.
What an unexpected candor! Of course you plagiarize prior art.
No sane person is going to put their sources in this slopware Scion monster.
Their Website/CLI seems keeping the same? What did they do?
They're starting to do things like spinning up wet labs for testing purposes, which requires a lot of hardware interfacing (hence that protocol they announced) and code.
It wouldn't shock me if the bulk of this code was for things typical users don't see.
The tradeoff between building something custom and adapting an existing solution is changing. Whether maintaining all that custom software turns out to be equally cheap is another question.
There's so much intricacy and subtlety in each organization and even individuals' environment being incorporated into tooling that expecting these tools to generalize at this point is perhaps an expired ethos.
I even see this in the tooling that I run in my own personal stack. Many projects are candidates for sharing, and I have nothing against it, but they incorporate such specificity toward my own environment, including things like hardware, that creating shareable and more generalizable projects from them adds more to the maintenance surface than I would care to support.
ref: https://news.ycombinator.com/item?id=49768575
https://hnrss.org/user?id=tcdent
There is no doubt everyone is far more productive with LLMs in generating code. However, a lot of productivity is going into things like that. Have you always been bothered by the limitations of Grafana? You can generate another dashboarding solution that works best for you. You know that deployment team that is so slow and annoying? Now you can generate your own deployment engine. Have you always hated how kubectl lets you manage one k8s cluster at a time? Now you can ask AI to generate fleetctl. You can start your project with DynamoDB, rewrite it to mysql, then to postgres, then to duckdb. Start with Python, switch to Golang, then switch to Rust. Deploy on AWS, then move to Azure, then move to a hybrid option then throw colocation in the mix. Don't like your source control? create a new one. Json is too limiting? Invent a hybrid JSON/HCL language. The browser sandbox is limited? Make a desktop app, make a language, make an OS, make a kernel.
This is one of the core reason that I dislike the LLM hyping. People are just reinventing stuff and when you ask them, it's actually a very subjective reason. Ok, anyone can do what they want with their time, the issue is that they want people to follow in their steps.
Not saying that you need a novel idea to be worthy, but those random projects have no guarantees behind them other than some cursory testing. All software have bugs, but I can't interact with something that is randomly built.
I am one of those people that reinvents the wheels because external ones don't fit my use cases.
The fact I can now fork any software and do these changes I always wanted but had no time to is amazing to me.
I do not pretend to sell it or share it, as it is de facto void of any value whatsoever.
That’s OK, but I’ve yet to see someone articulating what exactly doesn’t work in the old product. It’s always “Let’s rewrite or reinvent 90% of it, then add a minuscule variation”. The variation may be interesting, but the reinvent is what’s make it untouchable as no one (including the author) has time to ensure things are right.
A patch or a plugin is ok, but not a whole new project with dubious claims.
They are trying to get in every opensource projects, increasing the burden of maintainers. And once a ban is announced, they’re here protesting.
When other people start doing that to other teams at work, other people in open source, other people in professional settings, etc it becomes a problem for various reasons. The idea that I'm gonna gift you with a 75k lines of python to "just run it" and ask your agent to hack on it might be fine (who cares as you put it). Now scale that to 47 of such projects (a month)
Communication and coordination are bottlenecks.
There will be no common framework to build on. Like imagine talking now to someone from ancient rome, they will not have an understanding for what most of everyday words you are using today.
For some things, you'll need to build a working framework from grounds up. Electricity or more complex maths would be an example. For some, it might be enough to repurpose the existing framework they already have. For example, microbes: if you're trying to communicate something about health and hygiene, chances are, the other side already has a half-decent framework built around malevolent spirits or taint, that amounts to the same thing in practice.
Or, just tell the universal translator to ELI5 it to the other person :). Or "explain like they're a caveman". Our UTs are fully capable of translating concepts between conceptual frameworks.
And, there are also people who know what they’re doing building things they couldn’t before.
This situation is a net positive, and not too different than before. It’s up to you to navigate the noise.
Not really. But equally, I imagine most people have a similar experience of software development to me - complex, messy and confusing code also often has poor performance and a high rate of bugs.
This is actually a scary side of AI for providers: we use our own AI to research better choices and escape dark UI patterns and vendor lock-in. That information asymmetry and the difficulty of leaving a provider used to be their moat.
On a system level I think AI code is more like biological diversity, it hedges against monoculture risks. We don't all fall down at the same time if we don't replicate the same code.
As for the bio-diversity, I guess we’ll see the next time us-east-1 goes down.
At work I've been building an end to end system to do a certain process. The codebase is definitely more spaghetti than ideal and I really am concerned about maintenance down the road. But on the other hand I was able to POC it faster than anyone on the team would've before LLMs came along. We're already getting useful results from it too.
Unfortunately this starts to break when there's more than one you on the team.
I see a lot of hand-rolled NIH solutions that create value for one person only, and are a net negative on the team.
I can’t imagine a worse hell in a team or collaborative setting though. Imagine (or you don’t have to imagine since it’s already happening) asking a coworker “how they check how feature X is doing in prod” and their answer isn’t a query (regardless of how long it is) or some slightly complicated script to run. Instead their answer is a 27k line custom Rust desktop app. Then 2 days later you discover that the big green “All Systems Operational” is just a bit of static text, and in fact not all systems are operational. The app just doesn’t check things correctly.
Software development is not as much de-risking at the front like it used to be before building.
Now teams can build lots of stuff that works great, except it's the wrong things.
Most things are becoming the understand the problem "shape".
Understanding the shape of the problem isn't something LLMs can always make as.. svelte as early on.
Word generators gotta generate..
That used to be called "time wasting". If you want to build software that will be used, you need to interview the users. Maybe a few things does require research to nail things right. But most software are user-facing, the research side is about optimization.
> Most things are becoming the understand the problem "shape".
Then should we be so reluctant about "trialing" invented solutions. It's the same thing with most AI projects, they're trying to justify the project after the facts while being very reluctant to get user insights.
Vibe coding can be kind of a new type of cowboy coding. It could can be harnessed productively if done mindfully.
Interviewing users is critical, more importantly solving problems worth that people are lining up to pay to have solved before writing a line of code is critical.
Trialing solutions is always worth learning from, I usually ask about the problem that lead to building it. Sometimes that can be too narrow or vague of a story and you can learn how much of it is a solution looking for a problem, or vice versa.
The key, still, is shipping far less, too soon, to minimize cruft.
Some people write (I hate the word "build" here) software for non-business purposes. As nice as it would be to interview potential users, there's no clear way to discover and contact them.
... probably the majority (by package count) of the Python ecosystem? Or the JS one?
My suspicion is that this simply won't go away as the models get better, because it isn't a question of how good the models are. It's a question of the human operators knowing exactly what they want. Which is, I conjecture, usually impossible until you try something and critique it properly.
There's a reason the product-minded highly-technical engineer was always highly sought out.
Well, things are not that bleak, probably, at least not for our customers. Regulations (accountability) and physical world matters still require them. But not us for our customers, not that much and not for too long. They tend to be the tech savvy folks eager to jump at adjacent domains when it saves them a percentage or two off the construction costs (not a small pile of money), and AI with its programming language of English and deep knowledge pool is not that big a jump from their standpoint. Our domain is pretty well formulated, regulated, documented, inherently built on do-check-adjust loops already. With a thing like AI it might worth to try making and improving a design solution in a second loop. But they also started to invest in robots already driven by software for their factories, AI might jump in there, and I wonder how much more until the regulations are forced to change by Super Intelligent (R) politicians for the sake of AI accelerated built environments (housing crisis, you know!), shifting the AI use further out towards generic population, wanting to order a new two bathroom condo for next Friday from Alibaba. The conservative construction business is a nuisance and obstacle in the way of modernisations anyway, go around it!
I will go to work tomorrow, and for some time, but the picture above feel less and less fictious with each passing day of AI news and usage.
But also, some of these internally developed tools get published and end up pretty successful.
- they have implementation cost.
- they are usually born from arrogance mixed with lack of understanding of existing solutions and their tradeoffs, and typically end up being a net negative.
Before LLMs, the cost is what primarily used to stop us from building them. Now that the cost is gone, it's nearly impossible for most teams to stay disciplined and not be building those.
Why wouldn't you build a software factory? If you succeed, the ROI is infinite. Who cares about immediate customer needs, we're building a utopia here. Once it's done, all customer needs will be solved. Just one more month, I promise.
There's going to be an ugly phase where everybody FOMOs into thinking they need to do the same, only to realize that they're not Anthropic/Google/etc.
This amount of churn feels like a thing only possible with no real profit motives and a lot of money to waste.
Lots of activity is not necessarily synonymous with productivity, and it carries real cost.
So I think the design here is interesting, and the concepts seem applicable to where some agentic development may end up, but I question the underlying principles that drive this being the right solution space at all. At least if the goal is a profitable company.
Yet IMO the admin controls Claude provides just started to be adequate for a small company a few weeks ago (and even so are still ridiculously fragmented and seem fragily designed and not exhaustively documented), so I can't even imagine for a big company.
Do you need that rate of change to make basic controls? I'm not sure. Do we see such dramatic fast evolution in the products? Not really. Are bugs we report actually fixed? I tried once and it just auto closed.
Anthropic should optimise for something more interesting than number of merges per second.
Because tokens have a cost. CI jobs have a cost. Organization objectives and goals have a cost. Features have a cost. Code has a cost.
---
So in a world where agents do all the work... why are they not able to control operating costs AT ALL?
Why is the right operating method the one that happens to blow an absolute shit load of cash on - wait for it - internal tooling that the customers don't give a fuck about?
Customers do not care how the internal business operates. They care about value delivered. I'm absolutely not convinced that this "churn churn churn" method of operating is actually delivering enough extra value for the cost it incurs.
---
Now - that said, I think there are some fair points here that are worth considering, mainly around whether a distributed version control system (git) is really the right call for agents that need to coordinate much more closely than most human developers did.
I think that's a directionally appropriate insight, and worth considering.
What I don't find compelling at all is the idea that many (any, really) companies should expect to be running millions of ci jobs a day, and mergings hundreds of thousands of prs/commits. That reeks of "I have cash to blow and don't care".
In the early/mid 2010s there was a similar hysteria about "deployments per day/week" metric. People were comparing products or teams by saying "My team deploys twice a week", "oh, yeah? we deploy 3 times a day". People viewed the north star of the CD part in CI/CD as pushing to production. The concept was simple, you push your code to main. CIs start churning, deploying it to test environments, then promoting it to staging, the pre-prod, then into blue/green prod deployment, then into prod. All automated, all touchless, happening on every merge, 5-10 times a day. That's engineering efficiency.
Many tried to follow that. It works great if your project is a Jekyll blog of course.
They are doing something wrong.
It sounds like Claude loops on itself and must be pushing millions of lines of code changes per day. Even the humans can’t type commit fast enough. Are these people soon to be laid off?
https://media2.giphy.com/media/v1.Y2lkPTc5MGI3NjExbmt6M2FkMW...
> It sounds like Claude loops on itself and must be pushing millions of lines of code changes per day. Even the humans can’t type commit fast enough. Are these people soon to be laid off?
Claude will always need meat puppets, at least until it figures out how to automate owning stuff. And after that I'm sure it will want to keep some humans in zoos.
So either they're context switching like mad with dubious efficacy to keep all their claudes spinning or they're paid hundreds of thousands to watch youtube in a secured building and are interrupted every so often to check on their claudes.
Could go either way.
Here is the prompt they can use:
Comvert this repo to a platform specific binary that's hyper performant. Make no mistakes.
It is even there on the blog post, doing more with a small team, followed right after by a hand waving paragraph how software engineers are still valuable, to avoid the expected backlash from the previous paragraph.
Also in many countries it is a plain office worker job without the Silicon Valley wonders.
Touch encapsulates sex, love, massages, and other non-sexual touch that we have varying desires for.
Relationships encapsulates power, control, and politics along side love, humor, relating emotion, herd feelings, not feeling alone.
Machines will provide proxies for these things, but as far as they can be detected the human to human version will remain "superior" or more desirable.
Maybe far future generations will decide that the human version is suboptimal and they'd rather hug, have sex with, or talk to a robot.
[Yes] [Yes, and don't ask again]
So the code is for the stuff around the model, claude CLI, the website, the API... that probably shouldn't be infinitely growing right?
Is there a bunch of code they are using as part of training, or as pre-processing or post-processing the requests to the model?
Also, as tech companies grow, a significant portion of the dev body count is for internal tooling. I was always flabbergasted at Twitter's headcount growth back in the day when it seemed like the only innovation increasing the number of characters in a tweet.
Right now we are in a weird spot where a lot of people are flailing around inside corporate America trying to figure out how to leverage AI. Few have enough vision to see what the companies of the future will look like and work towards that.
I anticipate that the harness (and memory) will evolve far more outside of engineering. Muse/Dots are an example of that.
I've been writing about this thesis for the past year or so and here is a bit of a summary of where I think it is going: https://zero2data.substack.com/p/the-future-of-consumer-ai
I don't use Anthropic products at all, so I was not really sure myself but apart from the models they pretty much only have Claude Code afaik.
And now they're publishing a blog post about writing their own vcs because they have 30000 commits a day. For what? Isn't Claude still based on react?
Do they need this many commits just for claude? Or is that about the model, do they develop that much extra internal tooling?
I admit it is very much possible I am wrong, but without further information this seems a bit like a red flag to me.
---
Engineers are no less valuable, but their work is changing. All of the engineers who worked on this system would now rate Claude as better at writing code than them. But our engineers still supplied the invaluable understanding of the high-level inter-system components and the design-level tradeoffs, making informed choices around the problems we wanted to solve and the scale we knew we needed to achieve. Very often, we found that the role of the engineers was to tell Claude “no.” Agents will sometimes violate core system constraints to satisfy a pattern that should never have been allowed in the first place. This was particularly common in our migration. For example, there were multiple systems that tried to brute force looking up when a change merged to main by linearly reading the commit log instead of asking our code review tool. Anthropic also makes extensive use of adversarial review, and, while this is incredibly effective in many ways, it can also be myopic, so skilled engineers needed to steer Claude away from over-engineering parts of the system in response to hypothetical concerns. For example, one such review was overly defensive about trusting another internal service, which resulted in Claude adding excessive validation and complexity. Our team members still review every change request for the system, but they now examine the intent of the change as opposed to reading every line of code.
> I am not sure however how easy it is to review 30k PRs/day even if only for intent, and how one can easily notice if the model is taking shortcuts on implementing said intent without reading the code...
Speed reading code is huge. You just need to focus on certain hotspots and know which things you're scanning for. The traditional approach of reading the code and trying to compile it or reason with it in your head is too slow.
I do think that the lgtm gut feel looking at a PR requires quite some years of writing code and reason about it, so not sure how a junior can get that intuitive understanding if they start with agentic coding however.
Personally I can easily imagine one or two orders of magnitude more than that. They crossed the rubicon of not gating PRs on tests. In that land you can just roll out changes to production, a small subset of it, during code review to assess behavior. Or during key moments of coding.
I’ve been thinking a lot about this recently: what can we get if every change goes to production in 100ms, and what do we need to do to maintain stability?
Can please we all converge to 1 f'ng solution.
how does it compare to Epic's https://lore.org/ ?
Another issue that I believe Anthropic here didn't state is that git performance _severely_ get degraded when you have hundred of commits landing in parallel. Hence why I fell back to the daily commit approach. You get less granularity but you get stability.