I was running into something similar where I had it working on a single Python file. Maybe part of the issue is that I had memory turned off, and the conversation might have spilled into two conversations. I wasn't using version control either, but since it was a single file with about 1000 lines of code, it didn't seem too hard to eventually get all of my features back.
I just learned that manually invoked skills MUST be the _first_ thing in the prompt, e.g. you can't have `disable-model-invocation: true` and say "Hi, /grill-me on X".
Worse, everything after the /grill-me is injected a second time into the prompt as a <command-args> so your prompt is literally added to the context TWICE. (first time in the original user prompt.)
I swear Anthropic is intentionally bloating your context so you burn tokens on useless bullshit like this.
To be clear: you can certainly do "Hi, /grill-me on X" anytime you want. You can even write "hi, /grill-me and use /search-docs", even though the /search-docs skill absolutely doesn't work. But the Claude Code _harness_ won't autoload the skill into context, so claude-the-model has to go "hmm, where can I find that skill, when I call the Skill tool with that name it no worky, so now I have to search the filesystem for that skill...".
The <command-args> I know because I just audited my raw Claude transcripts via ccglass. You can verify by just asking Haiku "hey, /grill-me does this skill autoload for you?" and then "/grill-me hey, how many times does this text appear in your context?"
You can trivially doublecheck me. The above prompts won't even eat 1% of a 5 hr on 20$ plan.
Okay, yes, I agree that that’s how it works. I thought when you said MUST you meant that it doesn’t work; it does work it’s just that it burns a turn or two to do the lookup there. I certainly agree it would be nice to not do so.
(And I didn’t know the double args thing, that’s very interesting! I should have said I only meant the first bit.)
I snorted laughing at this. "We are experiencing tremendous growth" and it's just a codebase with 1,000 agents endlessly adding markdown documentation for every 1 line change.
I believe their peak month was >13k issues opened, but worth bearing in mind: "This issue has been automatically locked since it was closed and has not had any activity for 7 days.".
Codex had over 15k issues last I checked including legit months old issues with the sandboxing/bubblewrap config.
Human contributions to issues, comments or submissions, were drowning in a sea of AI vomit. Tons of the issues had those AI issue submissions with the classic confidence of having identified the problem, but that smell I know a lot of us recognize of it not actually knowing what the problem is at all because it hadn't failed to fix it 5 times yet.
lol right? So youre telling me they shipping 150,000 features this week? Or, maybe, they whack-a-moled 149,995 perf or bug regressions and shipping 5 new things. :shrug:
Absolutely, at this rate most software would be complete in a few hours. I may lack imagination but wtf are you changing? Endlessly refactoring? That’s spinning your wheels renamed as productivity.
This might be one of those things were everyone on the team takes a 30 minute lunch instead of 60 and just nobody clocks out to get the 1 hour extra over time.. And everyone just kinda ";) ;) ;) Working from our desk while eating lunch ;) ;) ;)"
I'm somewhat surprised they would say Claude is writing 90% of their code at this point. Did they get tired of reporting new milestones, or has there been a plateau?
Here I am still unable to stop myself from doing line-by-line code review. I notice too many things (redundant code, duplicate code, overly complex architecture, etc.) by doing it that I feel it would be careless not to.
No one cares about good code. It’s a means to an end. Your code doesn’t need to look pretty or be well documented or any of that stuff anymore because people are not looking at it. The only thing people care about is performant and error free code.
The way you phrase it makes me realize its almost like a GPL contamination. Eg, once you pull in GPL code, your project is infected and can never use a BSD (or other GPL incompatible) license.
Similarly, once you pull in terrible AI slop code, the project becomes unmaintainable by humans and you need to maintain it with AI.
There’s a little irony about the example you picked of the GPL, just because in the US the output of an LLM cannot be covered under copyright. The GPL works as an assignment of rights under copyright.
Code that isn't duplicated in 15 places, that doesn't read the whole database to fetch just 1 record, that doesn't do 15 database reads where one would suffice, that uses an optimized algorithm instead of an exponential one recreated badly from scratch... all that is also good code, and is just as important in LLM-slop era.
No code is error free. All code will need a fix of some kind eventually. When it does, it's better that the internals are not a total mess. In the same way that you don't want the insides of your car to be total alien nonsense either.
My advice is to start a totally vibe coded side project. You will never gain confidence until you experiment, find the failure cases, learn to deal with them, etc.
Choose something you want to do but don't really care that much about. Be more ambitious than you think you could be, have a high bar on the outcome but purposefully choose to completely ignore the actual code. Make it write tests, dashboards, monitoring - whatever you need other than code to feel confident.
Now is the cheapest time it will ever be to experiment.
Ridiculous suggestion. If someone already takes issue with ai generated code (as well they should, it often is _still_ pretty bad), getting them to write more of it is useless.
Additionally, do you really believe that 'Now is the cheapest time it will ever be to experiment'? Why so? Because of token prices? Not a good omen if so.
I'm not sure what is ridiculous about suggesting people take a stance of openness and curiousness when confronted with new situations. I genuinely don't understand what people have to lose by building some throw-away experiments without reading the code.
I do believe the subsidies offered on Pro accounts driven by the current competition make access to the best coding models and harnesses as cheap as they will ever be.
I have done this. It's exactly the reason I went back to doing code review. I don't look at the code with a fine toothed comb, but it does help me catch bad patterns early.
I guess the question is: did the project you worked on fail? As in, you couldn't achieve the outcome you were trying to achieve? Or at some point did you sneak a look at the code and recoil in revulsion? What was the failure state?
Was this with the latest models (Fable/Opus 5.5 or equivalent) or an older model in a 3rd party harness?
Are you in the camp of: "if at first you don't succeed, never ever try again".
Not parent, but project I was involved in was shipped very quickly, had many bugs, and burned a lot of customer trust.
This was a few months ago, so obviously not with the latest models, but:
1. That argument is evergreen. You will always be able to make it, and it doesn’t really add much. It is impossible to refute.
2. Gone back to - when writing code - having a strict mental modal and good control over the codebase. A mix of hand coding (to establish the mental modal) and prompting (for boilerplate). It’s not quite as fast, but the quality is much higher when you mix in using LLMs in quality enhancing capacities (review, debug, etc). This is with the latest model. Hands off still atrophies over time (even the boilerplate), and in some cases is very inefficient (as in wasting my time, nothing to be said about tokens).
> Are you in the camp of: "if at first you don't succeed, never ever try again".
This is great in some domains, but “don’t make the same mistake twice” is much more important in mine. I don’t have the luxury to ship low quality, especially when LLMs make even higher quality so much easier.
I think people are moving the goal post here. My post suggested starting a vibe coded side project, one where failure is low stakes and has no consequences. You are talking about shipping a production system to customers.
If you cannot tolerate failure then don't experiment. Don't mix the two.
I did this, but now I have a useful piece of software that I have zero clue regarding how it works. My only chance of "maintaining" it is giving it to another LLM to explain.
I am happy that I have a useful tool, but it simply isn't maintainable without a machine or me investing much time to understand it.
I don't see how the former is a sensible future for humanity. I am happy to say I won't be releasing this software I've created (I will find it useful myself), but deferring all understanding to a black box seems a poor future for brain activity.
One thing I do is ask the LLM to write me a walkthrough of the code. LLMs are pretty good at generating documentation like HTML slideshows or whatever else.
I mean, I've had useful software that I don't understand how it works where I am stuck hoping some small company continues to maintain it. And I've had some of those companies go out of business and leave me with useless software that I eventually have to replace once it no longer works on my machine.
Having the code that I can dump into an LLM to understand myself or have it alter it seems better than that.
It does, but then this admission that I have to feed the code into an LLM for it to generate documentation on how it works is just proving my point that we are now dependent on an LLM to understand the code that the LLM has generated.
I've done this, and always get subpar results. I cant get the LLM to make the minor tweaks I want to see, and now I have no mental model of the code to make the fix myself. Sometimes they arent minor things.
I've been working on a large (~10MLOC) mostly vibe-coded project for months now. It's actually quite frustrating:
* The agent will fix a bug when I tell it to, but only in exactly one place. I'll notice that another place has that bug later and have to redo it. It's the sort of thing that I would know to check for if I were fixing it myself. Yes, I can ask it to look elsewhere for similar bugs, but even that's not super reliable.
* The user-facing text it generates is just as awkward as ever. I have to rewrite all of it, and sometimes it slips through. It's also hard to write text when you don't know what the code is actually doing.
* It doesn't have a good model of what kind of information is relevant to humans. It's just as likely to produce a paragraph of random jargon as a clear explanation of what's actually going on. It shows excessive detail for things that don't matter, and not enough detail for things that do. For example, it doesn't understand that $0 can logically mean "null" in terms of how humans think about things (e.g. "This item is free" vs. "This item is $0").
I kind of wish I hadn't gone down this path but now it's far too late, I can't really manually maintain this amount of slop. I'm quite sad if this is going to be the end state of this profession.
Society would never, ever, ever, ever, ever let civil engineering get away with this level of carelessness. Why it is allowed in software is beyond my comprehension.
if you take yourself out of the loop it will hurt your skills and career. Engineering blogs used to have some best practices you emulate, but the big ai companies you cannot mimic without their infrastructure
The UI has improved substantially over the past six months. New capabilities are prominently featured and less-used ones are de-emphasized or removed. Every product goes through this type of evolution. The difference with Claude Code is that at the speed they iterate, it happens a lot faster and is therefore more noticeable.
customers don't like big changes to workflows they get used to; adding a new feature or a new button or menu item isn't a big deal but rearranging things so often start to feel random and shows little respect for the customer (or signs of any cohesive product design)
what are people using for automated code review these days? the claude code review product originally mentioned it cost "$15-25 per review" (which is absurd), but it seems like they've removed that from their docs with the launch yesterday. we've looked at various products in the space (coderabbit, greptile, etc.) but have gripes with each.
honestly i'm thinking it might be easier (and cheaper) to launch our own agent from GH webhooks and trigger `/code-review`..
in a vacuum, sure. the average pull request will have multiple commits once opened (i.e. to address feedback or perhaps it was opened too early), each triggering a review. it's very easy to rack up $100+ per pr (and an eng may open a dozen prs in a single day)
Using a different lab's model to review the code to the one that wrote it. Claude model plans/codes, OpenAI model reviews, or vice versa, or DeepSeek or GLM or whatever.
I do it this way too. I think having the same lab's model review the code in a fresh session also helps, so I use both: I generally start a new Claude Code and a Codex and have them review the code with a focus on 3 things: 1. bugs, 2. cut corners, 3. code style.
This generally works for me. I think a solid spec is the key for the models to review against. I ask them to follow the spec, and if the spec doesn't say, follow first principles.
Also, if there is a user experience change, they need to notify me.
We've had great success with a custom gh action on a self hosted runner. It sets up codex and all the same stuff a dev has on their machine:
So dbs, seed files, etc, and then just a review prompt and some skills and a few scripts.
It triggers on a new published PR or if tagged @bot-name. It's way better than the other services we've tried (which was admittedly ages ago) because you can get it to work exactly how you want it and it's dead simple.
With Codex, for the time being at least, you can use a subscription, so it's basically free if you're already paying for one. But on API pricing it's still not too bad as it doesn't produce that much output.
I set this all up in about half a day. Codex does 99% of the heavy lifting.
this: gh actions using an opencode harness (so we can use cheap models through OpenCode Zen). With the right model, and the right, scoped access, it's just a prompt away.
bugbot was costing us $1k per month for a small team. EVEN IF we decide to go with a paid or say something like GPT Sol for our custom harness it'll cost us maybe $100. We've already implemented all of bugbot's interesting features like cross-PR memory.
the cool thing is we can leverage this for things that bugbot cannot do. Just today I added a security-reviewer action reusing almost all of the code that we wrote for the code reviewer.
I've set up a claude code on a vm with a bunch of skills that can use git and the Azure Dev Ops MCP via a service user, the regular build agent pops a record in a db that serves as a queue/review and a bash script picks things up and fires off claude code in headless mode.
It's pretty effective and lets you do nice stuff like compare the code to the tickets, reply to comments and stuff like that.
Depends on model, promt and amount of agents that review. I found that default code-review in Claude code is pretty mid for finding all issues especially if you downgrade model and effort.
During my evals on my own 3 agents that review in parallel they are much more cost efficient and cover 75% issues. Even on model like haiku it’s better score than default.
Multiple passes or raising effort will cover 90+. And this was on tested on large diff, about 3500 LoC changes
Depends on model, promt and amount of agents that review. I found that default code-review in Claude code is pretty mid for finding all issues especially if you downgrade model and effort.
During my evals on my own 3 agents that review in parallel they are much more cost efficient and cover 75% issues. Even on model like haiku it’s better score than default.
Multiple passes or raising effort will cover 90+. And this was tested on large diff, about 3500 LoC changes
We built zeroshot.sh to automate our entire review and testing workflow. We found that the only thing that really worked was adverserial review cycles and that they had to be mechanically enforced, not based on instructions in skills. Should be relevant for you to check out :)
How is the ability of a human to understand the code via hands-on familiarity not a direct dependency of their ability to do manual code review? I don't get it. Having claude write an explainer (something I do a lot!) doesn't get you to the point where you can confidently assess a complex diff. And you've funneled your priors on what the diff says and does through the same system that already reviewed and approved it. This article isn't bad but it doesn't give enough weight to the human context-building part of the problem.
Yes it used to be said, the code is not the asset, the people who know what the code does and how to change it are the asset.
Now we’re just going to outsource our capacity to design and review because it’s faster that way, as if the thing that was wrong with the software industry was that releases weren’t being pushed out often enough.
I think there is no magic trick here. The answer is simply that humans will no longer understand the code.
Although, following Anthropic’s ethos and taking this to its conclusion, I’m not even sure why they need human reviewers at all. What, they bring in a human to look at a diff of an “important” section of code that was written by agents and has changed a dozen times since last week through 12 PRs of churn? Obviously they don’t know what’s going on. So what’s a human to do? Do what the other approvers (if any) did: fire up a Claude to command it “verify this change has no regression; check carefully for edge cases”, then come back after a half hour of High Thinking to see “Change is clean (with some non-blocking findings below, in order of significance)” and click Approve. Not sure what the purpose of the human is in such a process.
Fair assumption. I think humans, at least those who like to code and build solutions, will become better builders than coders. Understanding the code will become less important, but knowing how to put it all together will be essential.
I agree with you. A code reviewer from the same company, even with different models or efforts, shares the same blind spots. To review my code, I use a different company; if I code with Claude (usually Sonnet 5.5-medium), Codex reviews it, and vice versa. I get much better code this way. But more importantly, I always tell the reviewer to build its own test-scenario ledger and refute the coder's claim that it is done. And the verdict is shared with the coder, but the coder cannot reject it.
It looks like they are specifically optimizing for incidents/reverts and shipping faster
> The goal hasn't changed: less human time per change, and no more reverts than before.
Is this Goodhart's law in action? Less reverts and less incidents was measured and became the target. Is this then still a good measure? I guess Revenue is a more direct measure, but we don't have access to that one.
From my perspective as a power user, the product has improved substantially over the past six months. It has gained new features that are actually useful, lots of UI/UX/perf improvements, and bug fixes as well.
New features have actually been remarkable. For me they are almost always in one of two categories: things I've always wanted (but never actually bothered communicating to the devs), and things I didn't know I needed but did. The product team is clearly very tuned in.
The Workflow engine, the Mod system. I've tried to explore other coding harnesses and nothing is as powerful as claude code (barring investing a lot of effort into custom plugins for Pi or something). Other harnesses have workflow engines, but none are as powerful as Claude Codes.
I really like the the new ability for agents to start and manage other top-level agents. Makes it super easy to have one agent as the orchestrator for a feature and have it launch new sessions, answer questions from those agents, review their work and build merge trains. This was possible before, but required a fair amount of manual work. Now the experience feels first-class and it's really nice.
What I’m more worried about is feature identification and how to monitor them. How can you get the product to the next level?
Honestly I’m not worried about code anymore in my codebase I’ve not had bug problems for a while and incidents get fixed automatically.
I monitor new features with an agent who checks recordings.
Most actually good features need still 20-30 iterations to design the userflow to actually be interactive, but then there is also another challenge to translate usage patterns into features and how to present functionality with the least amount of friction as possible.
How to efficiently run monitors without wasting your token spends. A best 'monitor' best practice would be nice.
Still this article was very helpful and gave me a lot of new ideas.
After investments. Investments they can cut if they need to. Their daily operation is very profitable, they just decided to use that to bring cash in and go on the crazy growing trajectory, instead of taking it slow and steady. But if AI progress suddenly stops, they can just scale investments back and became profitable immediately. Calling them unprofitable is highly misleading. It's not like they are an early venture company still trying to figure out their product-market fit, with totally inflated numbers based merely on a theoretical number of potential customers (while having close to zero paying ones).
I'm not arguing that their product is not valuable or that it doesn't have customers.
I'm stating that their current business model, prices, and behavior they are trying to instill is not profitable. For example, token maxing every code change is great for Anthropic and terrible advise for Anthropic customers.
The moment they stop training, they become irrelevant. The moment they become irrelevant their valuation crashes down to normal valuation ranges for a company with their revenue and future growth
Antropic's revenue is up [1], [2], [3] and if you calculate adjusted operating income (by excluding most expenses like training costs) then by that measure it is profitable. However, if you factor in all the numbers it is operating at a loss.
"Anthropic told investors that it will profit on a measure called adjusted operating income.
"According to the report, adjusted operating income excludes stock-based compensation, which is not unusual for technology companies. However, the report also quotes its gross margins - above 80% - exclude revenue shared with distribution partners, as well as the cost of training its models."
I wonder how they account for free credits given in this spend. We got given US$5k free API credits from them a couple months ago and even accounting for our Claude Code subscription spend, I think we're still ahead overall - they haven't made any money from us! And they just gave us another few hundred bucks a month of API credit as part of our subscription.
It’s gotten pretty good too. My employer has a team that wrote their own code review bot with the checks and values we care about and it has gotten quite good over the last six months.
I look at it as a giant experiment. Any research tends to be very investment heavy, and people are given a lot more leeway to try things which are wildly unprofitable.
I'm glad they're doing this, because it lets me see an extreme of LLM development.
Why not? Code review is a crazy time sync. If I'm working on a project without code review (e.g. a personal tool) using Claude I can easily churn out 100 commits per day. And it isn't even slop these days.
It does make me wonder what on earth they are working on. Feels like most of their products could be basically finished already.
On the other hand Spotify somehow employs 3000 programmers, and that's just a music player.
All good ideas, I have been doing something similar for half a year. The most important thing is to reduce bugs and ensure the code does what you wanted. Writing code tests, logging, interacting with the running app are essential. For intent realignment I like to have the agent re-read all user messages posted in that project. It only takes a few minutes even on a 10,000 message project. Then use this fresh derivation of user intent to check code and tests. For long spans of work I prefer to generate a todo list which remains as a document in the project. Each closed gate tracks feedback from the agent on what happened. This task file is also reviewed twice - once before implementation and once after. So the formula is - review agents - tests - user message logs.
You don't need to read my message if you think you know what is the best formula for your AI usage. Whenever someone complains to be "tired" of reading some online message I take it as an act of verbal aggression. I rarely disclosed my formula online, and probably never will again after such reaction. Do you want to regulate HN? Was my message off?
369 comments
[ 2.6 ms ] story [ 59.4 ms ] threadWorse, everything after the /grill-me is injected a second time into the prompt as a <command-args> so your prompt is literally added to the context TWICE. (first time in the original user prompt.)
I swear Anthropic is intentionally bloating your context so you burn tokens on useless bullshit like this.
https://github.com/anthropics/claude-code/issues/92769
The <command-args> I know because I just audited my raw Claude transcripts via ccglass. You can verify by just asking Haiku "hey, /grill-me does this skill autoload for you?" and then "/grill-me hey, how many times does this text appear in your context?"
You can trivially doublecheck me. The above prompts won't even eat 1% of a 5 hr on 20$ plan.
(And I didn’t know the double args thing, that’s very interesting! I should have said I only meant the first bit.)
their claude code changelog 'fixed' % over time.
https://i.imgur.com/KJcem6c.png
I believe their peak month was >13k issues opened, but worth bearing in mind: "This issue has been automatically locked since it was closed and has not had any activity for 7 days.".
Human contributions to issues, comments or submissions, were drowning in a sea of AI vomit. Tons of the issues had those AI issue submissions with the classic confidence of having identified the problem, but that smell I know a lot of us recognize of it not actually knowing what the problem is at all because it hadn't failed to fix it 5 times yet.
I hope this is only for low-risk false positives. Dismissing a critical finding as a false positive should still involve AppSec.
You could improve revert rates while simultaneously introducing more security vulnerabilities.
And the linked "claude-code-security-review" explicitly states in the README
> This action is not hardened against prompt injection attacks
That's a significant limitation for an AI security review control, no?
Similarly, once you pull in terrible AI slop code, the project becomes unmaintainable by humans and you need to maintain it with AI.
This is, of course, tangential to your point.
I do, my coworkers do, my friends do, etc etc.
Your end users certainly don't either, heck they'll gladly sit there and scroll through their phone until the 50GB React app loads.
Choose something you want to do but don't really care that much about. Be more ambitious than you think you could be, have a high bar on the outcome but purposefully choose to completely ignore the actual code. Make it write tests, dashboards, monitoring - whatever you need other than code to feel confident.
Now is the cheapest time it will ever be to experiment.
Additionally, do you really believe that 'Now is the cheapest time it will ever be to experiment'? Why so? Because of token prices? Not a good omen if so.
I do believe the subsidies offered on Pro accounts driven by the current competition make access to the best coding models and harnesses as cheap as they will ever be.
Was this with the latest models (Fable/Opus 5.5 or equivalent) or an older model in a 3rd party harness?
Are you in the camp of: "if at first you don't succeed, never ever try again".
This was a few months ago, so obviously not with the latest models, but:
1. That argument is evergreen. You will always be able to make it, and it doesn’t really add much. It is impossible to refute.
2. Gone back to - when writing code - having a strict mental modal and good control over the codebase. A mix of hand coding (to establish the mental modal) and prompting (for boilerplate). It’s not quite as fast, but the quality is much higher when you mix in using LLMs in quality enhancing capacities (review, debug, etc). This is with the latest model. Hands off still atrophies over time (even the boilerplate), and in some cases is very inefficient (as in wasting my time, nothing to be said about tokens).
> Are you in the camp of: "if at first you don't succeed, never ever try again".
This is great in some domains, but “don’t make the same mistake twice” is much more important in mine. I don’t have the luxury to ship low quality, especially when LLMs make even higher quality so much easier.
If you cannot tolerate failure then don't experiment. Don't mix the two.
I am happy that I have a useful tool, but it simply isn't maintainable without a machine or me investing much time to understand it.
I don't see how the former is a sensible future for humanity. I am happy to say I won't be releasing this software I've created (I will find it useful myself), but deferring all understanding to a black box seems a poor future for brain activity.
I mean, I've had useful software that I don't understand how it works where I am stuck hoping some small company continues to maintain it. And I've had some of those companies go out of business and leave me with useless software that I eventually have to replace once it no longer works on my machine.
Having the code that I can dump into an LLM to understand myself or have it alter it seems better than that.
* The agent will fix a bug when I tell it to, but only in exactly one place. I'll notice that another place has that bug later and have to redo it. It's the sort of thing that I would know to check for if I were fixing it myself. Yes, I can ask it to look elsewhere for similar bugs, but even that's not super reliable.
* The user-facing text it generates is just as awkward as ever. I have to rewrite all of it, and sometimes it slips through. It's also hard to write text when you don't know what the code is actually doing.
* It doesn't have a good model of what kind of information is relevant to humans. It's just as likely to produce a paragraph of random jargon as a clear explanation of what's actually going on. It shows excessive detail for things that don't matter, and not enough detail for things that do. For example, it doesn't understand that $0 can logically mean "null" in terms of how humans think about things (e.g. "This item is free" vs. "This item is $0").
I kind of wish I hadn't gone down this path but now it's far too late, I can't really manually maintain this amount of slop. I'm quite sad if this is going to be the end state of this profession.
Society would never, ever, ever, ever, ever let civil engineering get away with this level of carelessness. Why it is allowed in software is beyond my comprehension.
How is this a proper product? Every time you open the desktop app the buttons may be on a completely random place
A random twitter said that, 6 month ago, it was 100% !
DECLINE ! SELL THE STOCK OPTIONS ! (wait, there are none, confused!)
honestly i'm thinking it might be easier (and cheaper) to launch our own agent from GH webhooks and trigger `/code-review`..
If an engineer paid $150,000 a year reviews some code manually for about 30 minutes, the cost of their time is about $36.
This generally works for me. I think a solid spec is the key for the models to review against. I ask them to follow the spec, and if the spec doesn't say, follow first principles.
Also, if there is a user experience change, they need to notify me.
How about your pattern?
https://theopenengine.com/blog/review-loop/
So dbs, seed files, etc, and then just a review prompt and some skills and a few scripts.
It triggers on a new published PR or if tagged @bot-name. It's way better than the other services we've tried (which was admittedly ages ago) because you can get it to work exactly how you want it and it's dead simple.
With Codex, for the time being at least, you can use a subscription, so it's basically free if you're already paying for one. But on API pricing it's still not too bad as it doesn't produce that much output.
I set this all up in about half a day. Codex does 99% of the heavy lifting.
bugbot was costing us $1k per month for a small team. EVEN IF we decide to go with a paid or say something like GPT Sol for our custom harness it'll cost us maybe $100. We've already implemented all of bugbot's interesting features like cross-PR memory.
the cool thing is we can leverage this for things that bugbot cannot do. Just today I added a security-reviewer action reusing almost all of the code that we wrote for the code reviewer.
It's pretty effective and lets you do nice stuff like compare the code to the tickets, reply to comments and stuff like that.
During my evals on my own 3 agents that review in parallel they are much more cost efficient and cover 75% issues. Even on model like haiku it’s better score than default. Multiple passes or raising effort will cover 90+. And this was on tested on large diff, about 3500 LoC changes
During my evals on my own 3 agents that review in parallel they are much more cost efficient and cover 75% issues. Even on model like haiku it’s better score than default. Multiple passes or raising effort will cover 90+. And this was tested on large diff, about 3500 LoC changes
> None required once automated review and security checks are green
How anybody can let an LLM write docs and not even read them is pure insanity.
Now we’re just going to outsource our capacity to design and review because it’s faster that way, as if the thing that was wrong with the software industry was that releases weren’t being pushed out often enough.
Although, following Anthropic’s ethos and taking this to its conclusion, I’m not even sure why they need human reviewers at all. What, they bring in a human to look at a diff of an “important” section of code that was written by agents and has changed a dozen times since last week through 12 PRs of churn? Obviously they don’t know what’s going on. So what’s a human to do? Do what the other approvers (if any) did: fire up a Claude to command it “verify this change has no regression; check carefully for edge cases”, then come back after a half hour of High Thinking to see “Change is clean (with some non-blocking findings below, in order of significance)” and click Approve. Not sure what the purpose of the human is in such a process.
> The goal hasn't changed: less human time per change, and no more reverts than before.
Is this Goodhart's law in action? Less reverts and less incidents was measured and became the target. Is this then still a good measure? I guess Revenue is a more direct measure, but we don't have access to that one.
New features have actually been remarkable. For me they are almost always in one of two categories: things I've always wanted (but never actually bothered communicating to the devs), and things I didn't know I needed but did. The product team is clearly very tuned in.
All in all, I'd say their approach is working.
Honestly I’m not worried about code anymore in my codebase I’ve not had bug problems for a while and incidents get fixed automatically.
I monitor new features with an agent who checks recordings.
Most actually good features need still 20-30 iterations to design the userflow to actually be interactive, but then there is also another challenge to translate usage patterns into features and how to present functionality with the least amount of friction as possible.
How to efficiently run monitors without wasting your token spends. A best 'monitor' best practice would be nice.
Still this article was very helpful and gave me a lot of new ideas.
Authors should really try and view this article from the perspective of someone who has to think about cost to use the models.
https://youtu.be/T-oXyXwD6sE?si=tauzXVyGmQ6MhI_5
I'm stating that their current business model, prices, and behavior they are trying to instill is not profitable. For example, token maxing every code change is great for Anthropic and terrible advise for Anthropic customers.
The moment they stop training, they become irrelevant. The moment they become irrelevant their valuation crashes down to normal valuation ranges for a company with their revenue and future growth
Antropic's revenue is up [1], [2], [3] and if you calculate adjusted operating income (by excluding most expenses like training costs) then by that measure it is profitable. However, if you factor in all the numbers it is operating at a loss.
[1] https://finance.yahoo.com/technology/ai/articles/anthropic-v...
[2] https://fortune.com/2026/09/29/anthropic-leaked-ipo-prospect...
[3] https://www.cnbc.com/2026/10/15/anthropics-third-quarter-rev...
"According to the report, adjusted operating income excludes stock-based compensation, which is not unusual for technology companies. However, the report also quotes its gross margins - above 80% - exclude revenue shared with distribution partners, as well as the cost of training its models."
From https://www.morningstar.com/news/marketwatch/2026091414/the-...
And what they're reporting is adjusted even further than EBITDA
I'm glad they're doing this, because it lets me see an extreme of LLM development.
It does make me wonder what on earth they are working on. Feels like most of their products could be basically finished already.
On the other hand Spotify somehow employs 3000 programmers, and that's just a music player.
good model
Anyway, flagging as misinformation.
Its like telling other ppl about your dreams. Not interesting.
Eventually I did, but it was hours and hours of effectively random stuff happening that I had to write down or dictate with speech to text.
It was a waste of time and I'm now glad I forget almost all of my dreams. My point is even my own dreams are uninteresting.
I'm not that interested in my own agent setup either - usually the tools become better on their own after a few months and render it all obsolete.