It is empty, for me and my team (I'm an engineering manager). We still have a support rotation but it's just so someone is aware they're supposed to respond to any alerts that come up.
Edit: product doesn't have a deep backlog either. Usually we can only plan somewhere between a couple days and a month in advance. Major recent product launch so we are longest-horizon we have been for a while.
I can confirm the same, backlog is sort of empty unless we try to attack a new domain, and that's usually a 2-3 month party, after that you need to almost imagine tasks. this really smells like end of backlog situation to me
Genuine question... How secure do you feel your job is, in that case? There's only so far that road goes, I imagine? I doubt your management are gonna buy you building a custom OS to run your custom CRM on...?
I have no idea what the fate of our industry will be. It cannot possibly survive Fable 7, even if it would still be advisable to hire engineers, people simply won't. Opus 5.5 is too good for things to be normal, but any more and it's simply over.
I don't know if I'm doing something wrong but bug fixing with AI has always been a crapshoot. You can absolutely tell an agent to fix an issue, and it will fix that issue, but the vast majority of the time that solution will be by adding code to cover the specific symptom in the bug report. It rarely considers or recommends solutions that have tradeoffs that may change behavior but will eliminate an entire class of similar bugs.
This is mostly a problem of agents not having the autonomy of managing an entire project and having the ability to make tradeoffs to increase simplicity or maintainability, but when given that autonomy no AI model I've used uses that privilege to write simpler, more maintainable code.
It makes me wonder if it could be solved with better context management
A real developer builds up global context of the whole project over time, you always hold the overall architecture in your head
Whereas with agents there is a still a lot lost between sessions
Its similar in a sense that a new developer to a project does more hacky local solutions whereas the senior on the team knows when the architecture needs to change as part of a fix
> It makes me wonder if it could be solved with better context management
There's a reason that repos are now bloated with AGENTS.md files that try to cram context in as pre-text. And that there's so many vibe-coded context/memory aggregation frameworks.
I think it's rather a problem with the training pipeline incentivizing correctness and endlessly defensive code over simplicity and maintainability of the implementation.
The problem with context I see primarily in mixing up user intent and hard requirements with random half-correct notes some other agent left in the comments.
> Whereas with agents there is a still a lot lost between sessions
Are y'all using AI agentic frameworks? I don't have a lot of issue with anything being lost between sessions, I have active memory and context management and have my agents write up documentation for each project to pick up where they left off or to hand it off if needed.
* It rarely considers or recommends solutions that have tradeoffs that may change behavior but will eliminate an entire class of similar bugs.*
Are you specifically asking it to do that? You can just point AI at a bug and say "fix" or you can ask it to review, plan, offer alternatives. You can get a lot more out of it if you actively engage.
Whenever I bug fix, I prompt it specifically to solve the problem not fix the symptom of the problem, and do so with the least amount of code changes possible to fix the problem. And I also tell it that if there's a more maintainable way to solve the problem(like prevent similar issues in other parts of the project, ie problems with a particular dependency or library being used) to suggest alternatives.
I think it's also important that the AI either has context to your entire project(like ability to read the source), or you put the context of what it's solving in the prompt you give it specifically.
Neither of these prompts effectively address the problem though. Fixing the root cause and doing so with the least amount of code often is contradictory. Including instructions to make suggestions to improve code maintainability will cause the AI to do it every single time it makes a change, even if it's completely unnecessary, drowning out useful suggestions in noise. When a bug is caused by a requirement elsewhere in the application, current models are more likely to break that functionality and implement a fix anyway rather than pause and get confirmation on the requirement. They may leave a note about the broken functionality at the end, but by that point the AI has already been iterating with an incorrect assumption. Any code the agent wrote in the same turn is going to be tainted by that assumption and it often does not remove or rewrite that code when given clarified requirements.
It's tempting to blame all AI issues on prompting because it's the only thing that we're able to control, but if a task falls into the long tail of work unable to be satisfactorily completed with AI, there's no prompt that will work. When there's always the alternative of doing the work yourself, it's not worthwhile to spend the tokens or time to get an AI model to do something after the first two or three prompts.
”current models are more likely to break that functionality and implement a fix anyway rather than pause and get confirmation on the requirement”
On the contrary, Claude Opus 5.5 consistently refuses to break requirements even if I insist on fixing something. It genuinely knows more about the project than I do.
I think this gets to something quite fundamental which is in the information loss when delegating a task in plain English.
It's very hard and actually a lot of work to draw an appropriate buffer around a problem in terms of how much agency you want to be exercised in considering other concerns when fixing it.
It takes a lot of skill on both sides to negotiate this even in an ongoing conversational loop.
There comes a point of course where essentially doing this communication and negotiation is equivalent to the work. At that point, you really aren't saving all that much time by not writing the code yourself in a lot of cases.
Maybe greenfield type work is the exception, where generating the code can actually save hours of typing, but certainly for bugfixes on mature systems where the right fix with all factors and competing priorities considered and negotiated only results in say 10 lines of new code, you start to wonder what actually is the benefit of delegation and automation here.
I don’t find this to be the case when using a good model (Opus 5.5 currently) and a decent process. I constantly have agents check out user bug reports, error alerts, little things I find when using my app, etc. I forward in emails, tag them in slack, drop screenshots, whatever. My first ask is that they investigate and figure out wtf is going on, what the root cause is, what the blast radius. And then from there, propose a few different approaches to fix with the tradeoffs for each, and let me choose.
Combined with a solid set of skills for how to do work in my codebase, this means that in 95% of cases, I can get this fixed with a few mins of my time and attention, and move on with my day.
The 5% of issues where the root cause is more fundamental do fall under what you’re saying though.
It should work, unless you're working on a very large codebase. I give the AI all the information that I would consider before implementing the solution, my prompts sometimes are multiple paragraphs, it's tedious but it gives good results. Information that I provide may include links to different repositories that may impact the feature being implemented, broad description of some terms, instructions to use a local temp/ directory to create temporary/reusable scripts when needed. Sometimes when I don't trust it, I explicitly ask it to come up with a plan without changing anything. Other times, I ask it to implement, build deploy, test deployed app and loop till the feature works.
One of the instructions that I have in Agents.md is to periodically save session summaries in ./docs/ directory, you can use an external file. I haven't tried any context-management tools other than context-mode, which seemed to make it worse. But I haven't needed anything more than the docs/ directory yet, and that's when I leave it working for hours at a time to in implement/test/build/deploy loop.
Interrupting it at every stage is going to make it slow, so it's better to see the end result and if you don't like the end result, you can try again with new instructions.
Mine is. I've been telling my senior managements that it's cheaper to rewrite stuff from scratch with AI then fix OLD products. They ate that up and everything we have now never reach v1.0 status.
1) The agent couldn't find a fix after spending multiple hours with it debugging. For example a very rare non-reproducible C++ crash. Or the person operating it got fed up and just manually figured it out.
2) The AI proposed a fix but people aren't comfortable just submitting it unless someone reviews and understands it.
3) Even when it does find a fix, its still time consuming to operate the agent and feed it info/test. There are also token limits at my company so you can't just burn through things ad infinitum.
Also in my case the product is a huge codebase and has millions of users you have to worry about breaking so maybe that plays into it.
1. That does sound difficult. Did the agent have access to C++ debugging tools? And was it the latest and greatest model?
2. I understand, but that's an organization issue.
3. Another organization issue, but also very understandable. It gets very expensive to let engineers just rip.
The huge codebase is definitely an issue - is there documentation geared toward an agent that's designed to give it a quick overview of the architecture and locations of things so that it can look up code in separate areas quickly? Obviously the agent, on its own, will grep its way through the codebase, but giving it an innate understand of how everything is structured is often very helpful.
Altogether it sounds more like the difficulty of a very large codebase with a difficult language and an org that's (rightfully) hesitant to go all in. The first can be mitigated, the second perhaps, and the third maybe one day.
AI only can improve velocity if it mostly works unsupervised. But most people don't trust their agents to run alone.
And to build that trust will take some time, people will need to learn how to use AI effectively.
I think the PS3 hardware could be a good analogy. It was really powerful, but nobody really knew how to use it. After a few years people got really good at it . And you can view this evolution by the improvement of graphical fidelity we got over the years.
I'm not sure how you can claim it takes time to build trust with AI as if AI isn't actively demonstrating every day why it can't be trusted. Just this morning I had it review clinical notes from a procedure I had done and it mathed things wrong (declaring an injection was 2.5ml when it was 2ml of one thing plus 1ml of another).
I ask it to fix one bug and it introduces three more. I ask it to add a feature and it adds four features.
It has very little common sense. I asked it to build a feature that required api calls to openai, and it decided the best approach was to hand roll the interface code with some bonkers defaults (explicitly zero retries, explicit 1 second timeouts to non steaming api) and wrote the most fragile code that didn't generally work. Even when directed to use the published library by openai it still tried to apply its insane invited requirements to make even that fragile.
I don't think it's nobody knows how to use it, its just not that reliable.
What model are you using/complaining about? Just seems so very removed from my work day where I am more usually impressed about the amount of common sense displayed by the AI..
I find it to be very random. Sometimes it'll do something very reasonable on its own that solves a problem I haven't considered and then give up and say it's very hard when prompted to do the same in another project. Or it'll start adding some static HTTP headers somewhere deep in sever code instead of one line in apache config. And I'm mostly using Opus 5.5 on medium.
> declaring an injection was 2.5ml when it was 2ml of one thing plus 1ml of another
Maybe you could write a custom tool to give your agent to help him with these tasks they seem to struggle?
Sometimes even a simple instruction in the AGENTS.md file can solve some problems.
And another way to improve is to split task into easier ones (first create a csv with all injections then create a summary from that csv). The main tradeoff with this approach is that takes more time, but it can improve your reliability.
I agree with you sentiment that these tools are pretty unreliable. But my approach is generally along the lines of 'how do I use something unreliable to do real work?'. All tools have limitations, and as an engineer I'm always trying to find ways to do more despite the limitations.
> can improve velocity if it mostly works unsupervised.
I would refute that a bit. I've found quite a nice workflow using a small dumb model (haiku mostly), and basically use it as editor tooling. Change function signature here, split out section into a function and update all similar usage locations there, add a new file and update the make system over there. Trivial tasks that just takes longer to write myself; things i would trust are mostly correct because it still compiles.
It's not a 10x workflow, but keeps me enjoying the coding without loosing track or stagnating my own learning. "commit this" is probably my most used prompt at this point. (with a few strict commit hooks to make it behave and keep brevity).
Mabie a 50% velocity bump to the coding itself on a good day.
For my team projects, I still need my teammates to review code, so that bottlenecks how quickly we can burn down the backlog. If I didn't need code review I would have the backlog burned down in weeks, but I'd create a bunch of outages in the process (we do catch things in code review!)
Also, some of my solo projects are gamedev-related, and gamedev is also bottlenecked on actually playing your game to see if its fun :) but it is great to get out of a 2 hour playtest session with notes and have a new build in like 15 minutes with the feedback incorporated.
My day job is in a regulated industry, we are required to have humans review everything. WE do have many many thousands of tests verifying behaviors, linters, autoformatting, we can get agents for an extra review, but human review is not going away in my industry for a long time.
At work you might have millions of users, and you might have dozens or more teams working on dozens or more systems working together.
I too have solo projects, mostly enjoy making casual consumer apps. The most users I’ve gotten with them peaked at 10k DAU.
For comparison I worked somewhere on enterprise software which had a few million DAU.
I get to learn new things on my solo projects and it is fun. It is a completely different environment when compared to working on multi-billion dollar products servicing many more billions/trillions worth.
Standards are super low, so slop gets approved and no one is checking. We get graded on how many Jiras we close. A bug gets a Jira. So we are doing great!
Maybe it's just me, but I've never worked at a company and on any substantial product that had an empty bug backlog. If a bug queue is empty, the product is either not used much (or not a priority), not tested well enough, or not monitored enough.
I don't know that a world where AI has mostly replaced programmers is a world with an empty bug queue.
While AI is reducing the _quantity_ of programmers needed in many cases, it is also an output multiplier for the remaining programmers, while often raising the bug-to-code ratio.
AI has led to more output, more projects, and more bugs.
It pretty much is? Every easily-described bug has been fixed. The only things that remain are "this isn't wrong per se, it just violates customer expectations" and someone needs to figure out what the product should do in this case.
We burnt through our backlog of "obviously we should do this when we have time" stuff in the spring. It was pretty fun to kill pretty much all of my multi-year refactor and cleanup list.
Now we are pretty much bottlenecked on the business side.
Our bug backlog is also practically empty (backend, financial). It’s a bit more slow moving and low bug domain for us in general but still. I went looking the other day and all that’s left is some dependency updates and such. I don’t know what models you guys are using if it can’t fix most bugs pretty easily with a decent set of steps defined
According to our metrics there is a crash in the wild that occurs 1 in 10000 launches of the product (desktop C++ app), all we have is a lightweight crash dump with call stack
I put Opus 5.5 on it and it spins and spins and comes back with some theories, some sound convincing but I can't test them as no one can reproduce the bug (tricky timing or some unknown combination of states)
Eventually I submit one as a speculative fix and it comes back in a few days that the crash is still occurring
I would love to just aim the agent at it and get a fix but it just doesn't happen a lot of times
So it sounds like AI is no worse than your dev team? I mean if you think up some way that you figure it out, like extra instrumentation or some experimental setup, then that's the first 1% and you can tell the agent to do the other 99%. But otherwise this stuff is hard and AI is not (yet) more intelligent than humans. It's just as intelligent as I need to be for most things.
Performance, Defect rate, feature velocity (5x), cycle time on defects to prod are all at the best they've been on our 7 year LOB 1.5 million loc system. 3 devs/2 qa, 2 sme's. 200 paying customers, moves money so it has to work.
First time in 3 decades I don't have any tech debt that I NEED to tackle. Customers can't keep up with us. We are considering new products/adjacent markets. Strangely, our competitors (about 5 serious ones) don't seem to be matching us.
A hidden assumption in your question is that the ai tools themselves don't create bugs or new backlog items. This is false. Even with perfect AI tools, bugs would slip in because of imprecise specifications.
There are many universes where the number of bugs grows faster with better agents.
103 comments
[ 8.6 ms ] story [ 126 ms ] threadEdit: product doesn't have a deep backlog either. Usually we can only plan somewhere between a couple days and a month in advance. Major recent product launch so we are longest-horizon we have been for a while.
This is mostly a problem of agents not having the autonomy of managing an entire project and having the ability to make tradeoffs to increase simplicity or maintainability, but when given that autonomy no AI model I've used uses that privilege to write simpler, more maintainable code.
A real developer builds up global context of the whole project over time, you always hold the overall architecture in your head
Whereas with agents there is a still a lot lost between sessions
Its similar in a sense that a new developer to a project does more hacky local solutions whereas the senior on the team knows when the architecture needs to change as part of a fix
There's a reason that repos are now bloated with AGENTS.md files that try to cram context in as pre-text. And that there's so many vibe-coded context/memory aggregation frameworks.
It's an AI-hard problem.
The problem with context I see primarily in mixing up user intent and hard requirements with random half-correct notes some other agent left in the comments.
Are y'all using AI agentic frameworks? I don't have a lot of issue with anything being lost between sessions, I have active memory and context management and have my agents write up documentation for each project to pick up where they left off or to hand it off if needed.
It cannot, in my current company we have extensive tech documentation, one of the best I've seen in my career and it still doesn't help enough.
Are you specifically asking it to do that? You can just point AI at a bug and say "fix" or you can ask it to review, plan, offer alternatives. You can get a lot more out of it if you actively engage.
Whenever I bug fix, I prompt it specifically to solve the problem not fix the symptom of the problem, and do so with the least amount of code changes possible to fix the problem. And I also tell it that if there's a more maintainable way to solve the problem(like prevent similar issues in other parts of the project, ie problems with a particular dependency or library being used) to suggest alternatives.
I think it's also important that the AI either has context to your entire project(like ability to read the source), or you put the context of what it's solving in the prompt you give it specifically.
It's tempting to blame all AI issues on prompting because it's the only thing that we're able to control, but if a task falls into the long tail of work unable to be satisfactorily completed with AI, there's no prompt that will work. When there's always the alternative of doing the work yourself, it's not worthwhile to spend the tokens or time to get an AI model to do something after the first two or three prompts.
On the contrary, Claude Opus 5.5 consistently refuses to break requirements even if I insist on fixing something. It genuinely knows more about the project than I do.
It's very hard and actually a lot of work to draw an appropriate buffer around a problem in terms of how much agency you want to be exercised in considering other concerns when fixing it.
It takes a lot of skill on both sides to negotiate this even in an ongoing conversational loop.
There comes a point of course where essentially doing this communication and negotiation is equivalent to the work. At that point, you really aren't saving all that much time by not writing the code yourself in a lot of cases.
Maybe greenfield type work is the exception, where generating the code can actually save hours of typing, but certainly for bugfixes on mature systems where the right fix with all factors and competing priorities considered and negotiated only results in say 10 lines of new code, you start to wonder what actually is the benefit of delegation and automation here.
Combined with a solid set of skills for how to do work in my codebase, this means that in 95% of cases, I can get this fixed with a few mins of my time and attention, and move on with my day.
The 5% of issues where the root cause is more fundamental do fall under what you’re saying though.
currently i am able to resolve most of the issue with AI Model when i talk in plain English rathe than prompt.
That's not a place you want to get to.
1) The agent couldn't find a fix after spending multiple hours with it debugging. For example a very rare non-reproducible C++ crash. Or the person operating it got fed up and just manually figured it out.
2) The AI proposed a fix but people aren't comfortable just submitting it unless someone reviews and understands it.
3) Even when it does find a fix, its still time consuming to operate the agent and feed it info/test. There are also token limits at my company so you can't just burn through things ad infinitum.
Also in my case the product is a huge codebase and has millions of users you have to worry about breaking so maybe that plays into it.
2. I understand, but that's an organization issue.
3. Another organization issue, but also very understandable. It gets very expensive to let engineers just rip.
The huge codebase is definitely an issue - is there documentation geared toward an agent that's designed to give it a quick overview of the architecture and locations of things so that it can look up code in separate areas quickly? Obviously the agent, on its own, will grep its way through the codebase, but giving it an innate understand of how everything is structured is often very helpful.
Altogether it sounds more like the difficulty of a very large codebase with a difficult language and an org that's (rightfully) hesitant to go all in. The first can be mitigated, the second perhaps, and the third maybe one day.
And to build that trust will take some time, people will need to learn how to use AI effectively.
I think the PS3 hardware could be a good analogy. It was really powerful, but nobody really knew how to use it. After a few years people got really good at it . And you can view this evolution by the improvement of graphical fidelity we got over the years.
I ask it to fix one bug and it introduces three more. I ask it to add a feature and it adds four features.
It has very little common sense. I asked it to build a feature that required api calls to openai, and it decided the best approach was to hand roll the interface code with some bonkers defaults (explicitly zero retries, explicit 1 second timeouts to non steaming api) and wrote the most fragile code that didn't generally work. Even when directed to use the published library by openai it still tried to apply its insane invited requirements to make even that fragile.
I don't think it's nobody knows how to use it, its just not that reliable.
It's certainly not at the point where you can just hand it over all responsibility and trust that it will be fine.
BUT I also wouldn't trade it for anything and improves my productivity by around 3-5x (gut feeling).
Maybe you could write a custom tool to give your agent to help him with these tasks they seem to struggle?
Sometimes even a simple instruction in the AGENTS.md file can solve some problems.
And another way to improve is to split task into easier ones (first create a csv with all injections then create a summary from that csv). The main tradeoff with this approach is that takes more time, but it can improve your reliability.
I agree with you sentiment that these tools are pretty unreliable. But my approach is generally along the lines of 'how do I use something unreliable to do real work?'. All tools have limitations, and as an engineer I'm always trying to find ways to do more despite the limitations.
I would refute that a bit. I've found quite a nice workflow using a small dumb model (haiku mostly), and basically use it as editor tooling. Change function signature here, split out section into a function and update all similar usage locations there, add a new file and update the make system over there. Trivial tasks that just takes longer to write myself; things i would trust are mostly correct because it still compiles.
It's not a 10x workflow, but keeps me enjoying the coding without loosing track or stagnating my own learning. "commit this" is probably my most used prompt at this point. (with a few strict commit hooks to make it behave and keep brevity).
Mabie a 50% velocity bump to the coding itself on a good day.
For my team projects, I still need my teammates to review code, so that bottlenecks how quickly we can burn down the backlog. If I didn't need code review I would have the backlog burned down in weeks, but I'd create a bunch of outages in the process (we do catch things in code review!)
Also, some of my solo projects are gamedev-related, and gamedev is also bottlenecked on actually playing your game to see if its fun :) but it is great to get out of a 2 hour playtest session with notes and have a new build in like 15 minutes with the feedback incorporated.
I too have solo projects, mostly enjoy making casual consumer apps. The most users I’ve gotten with them peaked at 10k DAU.
For comparison I worked somewhere on enterprise software which had a few million DAU.
I get to learn new things on my solo projects and it is fun. It is a completely different environment when compared to working on multi-billion dollar products servicing many more billions/trillions worth.
"Work expands so as to fill the time available for its completion"
https://en.wikipedia.org/wiki/Parkinson%27s_Law
[1] https://github.com/scala-js/scala-js/issues?q=is%3Aissue+sta...
Thus Y is prioritized over bugs.
I don't follow.
Maybe it's just me, but I've never worked at a company and on any substantial product that had an empty bug backlog. If a bug queue is empty, the product is either not used much (or not a priority), not tested well enough, or not monitored enough.
I don't know that a world where AI has mostly replaced programmers is a world with an empty bug queue.
While AI is reducing the _quantity_ of programmers needed in many cases, it is also an output multiplier for the remaining programmers, while often raising the bug-to-code ratio.
AI has led to more output, more projects, and more bugs.
We burnt through our backlog of "obviously we should do this when we have time" stuff in the spring. It was pretty fun to kill pretty much all of my multi-year refactor and cleanup list.
Now we are pretty much bottlenecked on the business side.
According to our metrics there is a crash in the wild that occurs 1 in 10000 launches of the product (desktop C++ app), all we have is a lightweight crash dump with call stack
I put Opus 5.5 on it and it spins and spins and comes back with some theories, some sound convincing but I can't test them as no one can reproduce the bug (tricky timing or some unknown combination of states)
Eventually I submit one as a speculative fix and it comes back in a few days that the crash is still occurring
I would love to just aim the agent at it and get a fix but it just doesn't happen a lot of times
First time in 3 decades I don't have any tech debt that I NEED to tackle. Customers can't keep up with us. We are considering new products/adjacent markets. Strangely, our competitors (about 5 serious ones) don't seem to be matching us.
I haven't seen AI make a single mistake yet while coding my apps
There are many universes where the number of bugs grows faster with better agents.