113 comments

[ 4.6 ms ] story [ 97.7 ms ] thread
> it sure feels like software has become a brittle mess, with 98% uptime becoming the norm instead of the exception, including for big services

As somebody who has been running systems like these for two decades: the software has not changed. What's changed is that before, nobody trusted anything, so a human had to manually do everything. That slowed down the process, which made flaws happen less frequently. But it was all still crap. Just very slow moving crap, with more manual testing and visual validation. Still plenty of failures, but it doesn't feel like it fails a lot of they're spaced far apart on the status page. The "uptime" is time-driven, not bugs-per-lines-of-code driven.

DevOps' purpose is to teach you that you can move quickly without breaking stuff, but it requires a particular way of working, that emphasizes building trust. You can't just ship random stuff 100x faster and assume it will work. This is what the "move fast and break stuff" people learned the hard way years ago.

And breaking stuff isn't inherently bad - if you learn from your mistakes and make the system better afterward. The problem is, that's extra work that people don't want to do. If you don't have an adult in the room forcing people to improve, you get the disasters of the past month. An example: Google SREs give teams error budgets; the SREs are acting as the adult in the room, forcing the team to stop shipping and fix their quality issues.

One way to deal with this in DevOps/Lean/TPS is the Andon cord. Famously a cord introduced at Toyota that allows any assembly worker to stop the production line until a problem is identified and a fix worked on (not just the immediate defect, but the root cause). This is insane to most business people because nobody wants to stop everything to fix one problem, they want to quickly patch it up and keep working, or ignore it and fix it later. But as Ford/GM found out, that just leads to a mountain of backlogged problems that makes everything worse. Toyota discovered that if you take the long, painful time to fix it immediately, that has the opposite effect, creating more and more efficiency, better quality, fewer defects, and faster shipping. The difference is cultural.

This is real DevOps. If you want your AI work to be both high quality and fast, I recommend following its suggestions. Keep in mind, none of this is a technical issue; it's a business process isssue.

It's a systems engineering job. You need to provide context, acceptable failure modes, and test at each level for validation. Identify false coupling, poor interfaces, things that don't match business context during agent planning phase. Then communicate / translate to others so their decisions improve instead of destroying the system by optimizing only for their local situation.
Super good take - the Andon cord is needed everywhere.
> One way to deal with this in DevOps/Lean/TPS is the Andon cord.

Many years ago, I started working for chip companies. It was like a breath of fresh air. Successful chip companies know the costs (both direct money and opportuity) of a failed tapeout, so the metaphorical equivalent of this cord was there.

Find a bug the morning of tapeout? It will be carefully considered and triaged, and maybe delay tapeout. And, as you point out, the cultural aspect is incredibly important, which means that the messenger won't be shot.

I think before even being able to entertain the thought of slowing the fuck down, we need to seriously consider divorcing productivity. Or at least asking a break, so you can go for a walk in the park, meet some friends and reflect on how you are approaching development.

I think this is very good take on AI adoption: https://mitchellh.com/writing/my-ai-adoption-journey. I've had tremendous success with roughly following the ideas there.

> The point is: let the agent do the boring stuff, the stuff that won't teach you anything new, or try out different things you'd otherwise not have time for. Then you evaluate what it came up with, take the ideas that are actually reasonable and correct, and finalize the implementation.

That's partially true. I've also had instances where I could have very well done a simple change by myself, but by running it through an agent first I became aware of complexities I wasn't considering and I gained documentation updates for free.

Oh and the best part, if in three months I'm asked to compile a list of things I did, I can just look at my session history, cross with my development history on my repositories and paint a very good picture of what I've achieved. I can even rebuild the decision process with designing the solution.

It's always a win to run things through an agent.

Eh I think its self-correcting problem

Companies will face the maintenance and availability consequences of these tools but it may take a while for the feedback loop to close

I suppose everyone on HN reaches a certain point with these kind of thought pieces and I just reached mine.

What are you building? Does the tool help or hurt?

People answered this wrong in the Ruby era, they answered it wrong in the PHP era, they answered it wrong in the Lotus Notes and Visual BASIC era.

After five or six cycles it does become a bit fatiguing. Use the tool sanely. Work at a pace where your understanding of what you are building does not exceed the reality of the mess you and your team are actually building if budgets allow.

This seldom happens, even in solo hobby projects once you cost everything in.

It's not about agile or waterfall or "functional" or abstracting your dependencies via Podman or Docker or VMware or whatever that nix crap is. Or using an agent to catch the bugs in the agent that's talking to an LLM you have next to no control over that's deleting your production database while you slept, then asking it to make illustrations for the postmortem blog post you ask it to write that you think elevates your status in the community but probably doesn't.

I'm not even sure building software is an engineering discipline at this point. Maybe it never was.

Engineering is two things:

1. Applied physics - Software is immediately disqualified. Symbols have no physics.

2. Ethics - Lives and livelihoods depend on you getting it right. Software people want to be disqualified because that stuff is so boring, but this is becoming a more serious issue with every passing day.

> What are you building?

This x1000. The last 10 years in the software industry in particular seems full of meta-work. New frameworks, new tools, new virtualization layers, new distributed systems, new dev tooling, new org charts. Ultimately so we can build... what exactly? Are these necessary to build what we actually need? Or are they necessary to prop up an unsustainable industry by inventing new jobs?

Hard to shake the feeling that this looks like one big pyramid scheme. I strongly suspect that vast majority of the "innovation" in recent years has gone straight to supporting the funding model and institution of the software profession, rather than actual software engineering.

> I'm not even sure building software is an engineering discipline at this point. Maybe it never was.

It was, and is. But not universally.

If you formulate questions scientifically and use the answers to make decisions, that's engineering. I've seen it happen. It can happen with LLMs, under the proper guidance.

If you formulate questions based on vibes, ignore the answers, and do what the CEO says anyway, that's not engineering. Sadly, I've seen this happen far too often. And with this mindset comes the Claudiot mindset - information is ultimately useless so fake autogenerated content is just as valuable as real work.

Software engineering is real engineering because we rigorously engineer software the way real engineers engineer real things.

Software engineering is not real engineering because we do not rigorously engineer software the way "real" engineers engineer real things. <--- YOU ARE HERE

Software engineering is real engineering because we "rigorously" engineer software the way "real" engineers engineer real things.

Edit: quotes imply sarcasm.

Hey Visual Basic is still there, and last time I checked it was still the goto option to do OLE Automation.

RoR is no longer at its peak, but is still have its marginal stable share of the web, while PHP gets the lion part[1]

Ok, Lotus Notes is really relic from an other era now. But it’s not a PL, so not the same kind of beast.

Well, also LLMs are different beast compared to PL. They actually really are the things that evocate the most the expression "taming the beast" when you need to deal with them. So it indeed as far away as possible of engineering as one can probably use a computer to build any automation. Maybe to stay in scientific realms ethology would be a better starting point than a background in informatics/CS to handle these stuffs.

[1] https://w3techs.com/technologies/comparison/pl-php

As far as I can tell, the only reason agents exist is because large context increase the probability of context poisoning, purely by the inability of these models to actually make conceptual decisions about the context.

I was interested in making a semi-automous skill improvement program for open code, and I wired up systemd to watch my skills directory; when a new skill appeared, it'd run a command prompt to improve it and cohere it to a skill specification.

It was told to make a lock file before making a skill, then remove the lock files. Multiple times it'd ignore that, make the skill, then lock and unlock on the same line. I also wanted to lock the skill from future improvements, but that context overode the skills locking, so instead I used the concept of marking the skills as readonly.

So in reality, agents only exist because of context poisoning and overlap; they're not some magicaly balm to improving the speed of work, or multiplying the effort, they simply prevent context poisoning from what's essentially subprocesses.

Once you realize that, you really have to scale back the reality because not only are they just dumb, they're not integrating any real information about what they're doing.

> What are you building? Does the tool help or hurt?

> People answered this wrong in the Ruby era, they answered it wrong in the PHP era, they answered it wrong in the Lotus Notes and Visual BASIC era.

I'm assuming you're saying these tools hurt more than help?

In that case I disagree so much that I'm struggling to reply. It's like trying to convince someone that the Earth is not flat, to my mental model.

PHP, Ruby and VB have more successful code written in them than all current academic or disproportionately hyped languages will ever have combined.

And there's STILL software being written in them. I did Visual Basic consulting for a greenfield project last week despite my current expertise being more with Go, Python, C# and C. And there's a RoR work lined up next. So the presence gap between these helpful tools and other minor, but over index tools, is still increasing.

It's easy to think that the languages one see mor often in HN are the prevalent ones but they are just the tip of the iceberg.

Absolutely agree.

I'm watching a team which is producing insane amounts of code for their team size, but the level of thought that has gone into all of the details that would make their product a fit predator to run at scale and solve the underlying business problem has been neglected.

Moving really fast in the wrong direction is no help to anyone.

Exactly! I’ve noticed a resounding amount of people are writing the same pieces recently, it’s almost like everyone’s sounding their alarm for the upcoming tsunami. Who’s listening? Here’s my piece: https://humantodo.dev
Agreed. I've been building software for 25 years+.

At some point I became so burnt out I couldn't look at an IDE or coloured text for that matter.

I found the way back by just changing my motto and focus... Find good people, do good work. That's it, that's all I want.

I don't care whether the 'property is hot' or what the market is doing anymore, I just build software in my lane, with good people around.

> It's not about agile or waterfall or "functional" or abstracting your dependencies via Podman or Docker or VMware or whatever that nix crap is.

It is though. Picking the right approaches and tools makes more difference than anything else. Sure, you don't need the right tools if you can make the right choices - but it's much easier to pick a better methodology than to hire smarter people.

Perhaps this is the wrong place to plant this thought. Maybe nobody will read it. These comments are now many hours old and HN has a way of walking away once they have had their turn shouting into the void.

I once received a "bonsai" seed kit from a former boss during a holiday dinner. I think it was meant as a joke, but even now I'm not so sure. I planted those seeds anyway. I told some people about it and they immediately mocked me saying it was a waste of time and going to take 30 years. This interaction immediately said everything to me about the expectations and attitudes of others.

Obviously, they grew like any other plants and actually quite nicely. Of course they're a commitment, but not a huge one.

I just wanted some plants for my apartment and they fit the bill. In a few years I had good looking plants. A decade later, I still have them and they're now more recognizably "bonsai". My home now looks nicer, I have a story to tell, and I learned a little bit from a very low stakes hobby.

My point is, I think it's nice when people have projects. I think it's nice to see what comes of it. I guess my only regret is ever saying "I planted bonsai" too soon just because that's what the box said. I didn't know how else to describe what I had done that weekend to those people who threw theirs in the trash.

that's a great story!
Maybe it's more about a rush to share how awesome it is that you compressed your time-to-release down to days and not weeks or months - when in reality that's a good thing in the sense that you get to a failure state much FASTER, and failure states are good, because that means that you get to iterate and get past those failures FASTER.

I don't think people were releasing at this pace, so the failure states are fast and furious so there is just that much more viability. I think the microslop windos failures lately are just them being the same "them" that they've always been .. just MUCH faster. (they just need to stop monkeying with windows and stop adding more features on top of an already shaky foundation.) Maybe we just need more of the stories like Anthropic working with Mozilla to squash 5x the amount of bugs in a similar time frame first, AND THEN "vibe a browser together from nothing but specification files and an army of bots in a weekend".

> What are you building?

I think AI really pushes this higher up the abstraction layer:

> What problem are you solving?

I've spent a good amount of my careering using engineering and math to solve specific problems, I'm usually adjacent to software teams.

What I've seen happen with agentic coding is that traditional software engineers keep focusing on using it to build software, while ignoring the problem they're trying to solve.

Meanwhile I've seen junior data analysts start interfacing with applications and tools they never dreamed of before, and delivering results to stakeholders in record times. Things that were previously blocked by engineering no longer are.

But many engineers today are not really problem solvers, they're software builders. The idea that solving the end users problem is the goal, not building them software, is incomprehensible.

And so they continue to struggle to use AI effectively because they're trying to build software with it. Which it's not terrible at, but it's really the wrong tool for that job.

Sometimes software is necessary to solve a problem, a few years ago, software was necessary for a fairly large problem surface area (though, to your point, even then a lot of software was not really built to solve those problems). Today that surface area is shrinking, and as economic constraints loom on the horizon, I believe it will increasingly be people who are solving problems (with or without AI) that will be the ones surviving.

The kind of jobs an analyst are doing are probably the most amenable of everything to LLM assistance. Small, bounded, etc.

The bigger the problem set and context the less helpful an LLM gets.

I am just using Go at this point and stopped caring about my own opinions.

I live in the happy place in negligence. Go software has almost zero maintenance costs and it will continue to build my programs in 10 years with zero changes to my codebase being necessary.

I probably will never touch C++ again, even though CGo is the most painful FFI/ABI implementation I've dealt with.

Just today I tried to build a project that's using bergamoth and a shitload of broken C++ dependencies and decided to not give a damn after 5 hours of trying to fix crappy code that changed for whatever reasons between c++14 and c++15, well, or the dependencies are broken, or the dependency versions are broken, or the maintainer's code never compiled in the first place... I just don't care.

My hopes were higher during the conan peak days, but now the ecosystem is just so broken even with jinja and whatever build framework the new kids are using.

I guess I just really hate the C++ ecosystem, and the lack of self reflection in there about the self inflicted pain that shouldn't be necessary in 2026.

In regards to agentic coding: I am toying around with codestral:22b right now and xiaomi's mimo models, and am building my own local dev environment which makes this kinda nice.

It's local and I like it, sometimes need to use claude still but it's getting there. But I am delegating only the gruntwork, not decisions, so I use temperature usually below 0.3. My approach is to make this sandboxed per folder I run it in and that agents are only allowed to communicate via notes or tasks, so that they are forced to use better documentation. Specific roles don't have write access to certain things, e.g. coder can't touch tests, and tester can't touch code.

I think the core idea here is a good one.

But in many agent-skeptical pieces, I keep seeing this specific sentiment that “agent-written code is not production-ready,” and that just feels… wrong!

It’s just completely insane to me to look at the output of Claude code or Codex with frontier models and say “no, nothing that comes out of this can go straight to prod — I need to review every line.”

Yes, there are still issues, and yes, keeping mental context of your codebase’s architecture is critical, but I’m sorry, it just feels borderline archaic to pretend we’re gonna live in a world where these agents have to have a human poring over every single line they commit.

> It’s just completely insane to me to look at the output of Claude code or Codex with frontier models and say “no, nothing that comes out of this can go straight to prod — I need to review every line.”

It's insane to me that someone can arrive at any other conclusion. LLMs very obviously put out bad code, and you have no idea where it is in their output. So you have to review it all.

Depends on your prod.

For an early startup validating their idea, that prod can take it.

For a platform as a service used by millions, nope.

Not having a code review process is archaic engineering practice at this point(at any point in history, really), be it for human written or AI written code.
The article didn't say to read every line though. Just the interesting ones. If you don't know where the interesting ones are, you have already lost.
I only have so long on earth. (I have no idea how long) I need things to be faster for me. Sometimes that means I need to take extra time now so they don't come back to me later.
If there is anyone who absolutely should slow down, it's the folks who are actively integrating company data with an agent -- you are literally helping removing as many jobs as possible, from your colleagues, and from yourselves, not in the long term, but in the short term.

Integration is the key to the agents. Individual usages don't help AI much because it is confined within the domain of that individual.

We reduce jobs every time we e.g. fix a bug. Where do you stop?
i just wish someone would explain why i prefer cline to claude code so much
> You installed Beads, completely oblivious to the fact that it's basically uninstallable malware.

Did I miss something? I haven't used it in a minute, but why is the author claiming that it's "uninstallable malware"?

Maybe they meant un-uninstallable?
oh yeah, that's actually how I read it though now I realize it's nonsensical... like when someone says "I could care less" when they actually mean "couldn't"
It's not even the complexity which, you have to realize: many managers and business types think it's just fine to have code no one understands because AI will do it.

I don't agree, but bigger issue to me is many/most companies don't even know what they want or think about what the purpose is. So whereas in past devs coding something gave some throttle or sanity checks, now we'd just throw shit over wall even faster.

I'm seeing some LinkedIn lunatics brag about "my idea to production in an hour" and all I can think is: that is probably a terrible feature. No one I've worked with is that good or visionary where that speed even matters.

Useful context here is that the author wrote Pi, which is the coding agent framework used by OpenClaw and is one of the most popular open source coding agent frameworks generally.
That's a great shout because I'm sure a lot of people would otherwise just discredit this take as just another anti-ai skeptic. But he probably has more experience working with LLM's and agents than most of us on this site, so his opinion holds more weight than most.
> “Heard joke once: Man goes to doctor. Says he's depressed. Says life seems harsh and cruel. Says he feels all alone in a threatening world where what lies ahead is vague and uncertain. Doctor says, "Treatment is simple. Great clown Pagliacci is in town tonight. Go and see him. That should pick you up." Man bursts into tears. Says, "But doctor...I am Pagliacci.”

https://www.goodreads.com/quotes/141645-heard-joke-once-man-...

> While all of this is anecdotal, it sure feels like software has become a brittle mess

That may be the case where AI leaks into, but not every software developer uses or depends on AI. So not all software has become more brittle.

Personally I try to avoid any contact with software developers using AI. This may not be possible, but I don't want to waste my own time "interacting" with people who aren't really the ones writing code anymore.

> Companies claiming 100% of their product's code is now written by AI consistently put out the worst garbage you can imagine. Not pointing fingers, but memory leaks in the gigabytes, UI glitches, broken-ass features, crashes

One thing about the old days of DOS and original MacOS: you couldn't get away with nearly as much of this. The whole computer would crash hard and need to be rebooted, all unsaved work lost. You also could not easily push out an update or patch --- stuff had to work out of the box.

Modern OSes with virtual memory and multitasking and user isolation are a lot more tolerant of shit code, so we are getting more of it.

Not that I want to go back to DOS but Wordperfect 5.1 was pretty damn rock solid as I recall.

Is the price of speed bloat? Where does the tolerance for less reliable software come from?
This assumes that only (AI/Agentic) stupidity comes into play, with no malice on sight. But if things go wrong because you didn't noticed the stupidity, malice will pass through too. And there is a a big profit opportunity, and a broad vulnerable market for malice. Is not just correctness or uptime what comes into play, but bigger risks for vulnerabilities or other malicious injected content.
Nature will handle this in time. Just expect to see a "Bear Stearns moment" in the software world if this spirals completely out of control (and companies don't take a hint from recent outages).
I’m worried we end up with an AIG moment, and we all end up on the hook.
subprime mortgages sprinkled on top of prime ones, treated as prime ones. because they were printing money. subprime code sprinkled on the backbone of software we use everyday. because they are printing code. reckoning
It's 2026, the "fuck" modifier for post titles by "thought leaders" has been done already ad nauseam. Time to retire it and give us all a break.
I for one look forward to rewriting the entirety of software after the chatbot era
> And I would like to suggest that slowing the fuck down is the way to go. Give yourself time to think about what you're actually building and why. Give yourself an opportunity to say, fuck no, we don't need this. Set yourself limits on how much code you let the clanker generate per day, in line with your ability to actually review the code.

This is a great point.

I have been avoiding LLM's for awhile now, but realized that I might want to try working on a small PDF book to Markdown conversion project[0]. I like the Claude code because command line. I'm realizing you really need to architect with good very precise language to avoid mistakes.

I didn't try to have a prompt do everything at once. I prompted Claude Code to do the conversion process section by section of the document. That seemed to reduce the mistake the agent would make

[0]: https://www.scottrlarson.com/publications/publication-my-fir...

This aligns with my observation from product design point as well.

Product design has a slightly different problem than engineering, because the speed of development is so high we cannot dogfood and play with new product decisions, features. By the time I’ve realized we made a stupid design choice and it doesn’t really work in real world, we already built 4 features on top of it. Everyone makes bad product decisions but it was easy and natural to back out of them.

It’s all about how we utilize these things, if we focus on sheer speed it just doesn’t work. You need own architecture and product decisions. You need to use and test your products with humans (and automate those as regression testing). You need to able to hold all of the product or architecture in your mind and help agents to make the right decisions with all the best practice you’ve learned.

What the article doesn't touch on is the vendor lock-in that is currently underway. Many corps are now moving to an AI-based development process that is reliant on the big AI providers.

Once the codebase has become fully agentic, i.e., only agents fundamentally understand it and can modify it, the prices will start rising. After all, these loss making AI companies will eventually need to recoup on their investments.

Sure it will be - perhaps - possible to interchange the underlying AI for the development of the codebase but will they be significantly cheaper? Of course, the invisible hand of the market will solve that problem. Something that OPEC has successfully done for the oil market.

Another issue here is once the codebase is agentic and the price for developers falls sufficiently that it will significant cheaper to hire humans again, will these be able to understand the agentic codebase? Is this a one-way transition?

I'm sure the pro-AIs will explain that technology will only get cheaper and better and that fundamentally it ain't an issue. Just like oil prices and the global economy, fundamentally everything is getting better.

I'm beginning to develop the opinion that the next step in this process will (or at least should) be local and/or self-hosted inference.

The latest qwen models are already very useful, and the smaller ones can be run locally on my laptop. These are obviously not as good as the latest frontier models, and that's extremely noticeable for the development workflow, but maybe in a year or two, they will be competitive with the proprietary models we have today, which are incredibly capable. I also expect compute for inference to continue getting cheaper.

The current lock in for me is the UX of Claude Code / codex cli, but this is a very small moat that will definitely be commoditized soon.

What do you mean about vendor lock-in? I haven’t yet seen any meaningful barriers to switching between different companies’ coding agents. Are you talking about AI market lock-in and not vendor-specific lock-in?

> these loss making AI companies will eventually need to recoup

This is true, and while AI spend continues to rise, I’m starting to think once the dust settles and the true costs emerge and stable profits are achieved, that it may be expensive enough that it’s a limiting force.

If only the AI understands your code, then vendor lock-in and exposure to price hikes will be the least of your problems. I don't think that you will be able to add Claude as the Dev-On-Call to your pagerduty schedule. If you are in an industry that requires due diligence and you get sued for bugs that cause material damage and human suffering, then I don't think the "blame it on Claude" defense is going to land well in court. I cover these topics on https://www.exploravention.com/blogs/soft_arch_agentic_ai/ which is a blog I wrote recently.
(comment deleted)