I wanted to use this post as a source to write on my web server project's blog on why debuggable web infrastructure matters even if AI SRE agents exists, but the idea got delayed and I ended up posting it on HN... :')
I dug into general AI deskilling a couple days back[1] - no good signs anywhere, but there's a few things we can do like make concerted efforts to do programming (like a coding hour every week), do a Leet code question, and getting the agent to quiz before merging something (stolen from Notion).
I've been having this same gnawing feeling. I just started reading the book "The Collision" by Eric C So which promises to cover this subject. I haven't finished it but seems decent so far.
It's so depressing a thought that if I continue on this path, in a few years, let alone a decade, I'll be utterly unable to produce or even understand code. I realize the skill may not be very economically useful, but losing it is still a depressing outcome that's preventable in theory. At the same time I'm feeling a deep apathy towards building anything. Playing a Zachtronics game is the farthest my internal motivation can push me towards...
I don't know if it's true. I didn't program at all between 2000 and 2015 and I've been able to restart without too much effort. I forgot many things, but I found it relatively easy to recover...
I'm more worried about people that did never program and started directly with LLM support...
I’m personally using it to augment my coding more than replace it. I find joy in it.
I use my agent to prototype a rough version that mostly works, and then I iterate through (with or without my LLM) to clean it up. Then back to the top with a new feature until I’m satisfied with the project and code.
I do tend to like it. My real selling point is that I’m not “locked in” to any choices until I’m happy with how it looks. I used to have to build the thing to see how it looks and acts, and going back on those choices was painful.
I do think there will be value in it, but I’ve been wrong before. My agent occasionally makes stupid choices that make every prompt after that 4x as hard to fix. Like I screwed up a pydantic model in an unreleased library and it supported a “legacy” path no one would ever use for like the next 5k LoC. By the time I noticed, legacy handlers were like 25% of the codebase.
That’s gonna burn a looot of tokens if you scale it up to a million line of code app.
Nice video, but I have said it before, the difference between just “vibe coding” and reliably using AI is the specs you need to write first and the tests after. If you start prompting the end goal/s the AI will basically guide you on what needs to be done and you just accept it as long as it runs, and this is following what most software developers were doing before LLM, the agile way, the quick ship way, the daily standup and ship first discus later way, the code pairing way, which worked mostly (even though I hated personally) in teams, so these developers are replacing what they were used to do with an agent and that’s the mistake. However, if you go with traditional engineering ways, waterfall, specs based, blueprints, and then tests, you control the agents and guide them not the other way around, in your specs you define the tiniest possible thing to what library to be used or excluded, the methods and approaches, among others, the agent here will produce the syntax and help you in the tests, just like in other engineering aspects how machines didn’t take over, they follow your specs and frameworks. The syntax is the past, AI can produce better syntax than any human alive or all combined, if you think otherwise you are delusional and still didn’t move on, it’s time to stop the “developer” mindset and do the engineering one.
I personally have a project that have been spending 2 months now making the specs and blueprint, more than 1000 files defining every bit including the future features too, and not a single code has been written yet, later, an agent will use this to produce the syntax and all the needed tests to have exactly what’s been needed.
> Some immediate targets include multi-signal correlation across metrics, logs, and traces, autonomous root-cause analysis for partially understood failures
AI is already pretty good at this in "SRE" workflows.
SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
> SRE was always a lame job description and so was the entire "DevOps" movement we had to suffer through during ZIRP. Real ones are oncall for the systems they build.
You're conflating a whole lot of different models here. In fact, "developers should be on call for the systems they build" is a principle that most schools of DevOps thought emphatically subscribe to.
Google's SRE was always about building infrastructure to make things reliable, and started in a world where LAMP on bare-metal + DNS load balancing was common. It makes sense to have a separate org/team/role responsible building real load balancing with health checks and failover, figuring out how to do zero-downtime updates, etc.
When these problems because largely solved with off-the-shelf software SRE became either a bureaucracy (ensuring reliability through policy and enforcement) or just deploying other people's code and holding the pager.
I need the company to have skin in the game, so I prefer that I am entirely unreachable outside M-F 9-5. Otherwise they just exploit people who submit far too quickly to being on-call 24/7. It also gives me motivation to make my stuff work as best as possible so that I'm not being annoyed by management about my non-existence outside core hours.
But I really do need the company to have a competent plan for handling issues that isn't just, "call the developer." It's amazing how much it makes them care about proper support and testing when the alternative isn't exploiting a developer, but is a customer outage.
Realistically nothing is perfect, there will be cases where something randomly breaks and will need attention after hours. Responding to a call for this should get you double pay for your time. Plumbers and electricians work this way, why not software folks?
If it absolutely has to get fixed, the company will pay the cost. If it can wait until Monday morning, they can save that cost.
there's eventually a point in organization size where it makes sense to have some kind of centralization to make it easier for teams to actually be on call for the systems they build without grinding them to dust.
> Real ones are oncall for the systems they build.
Yes! This is devops, thank you for coming to my ted talk.
Coincidentally I’ve found putting engineers oncall for their stuff really does wonders to ensure they check their AI agents’ code and know how it works.
If you look at the history of SRE, it was distinctly NOT sysadmins, Google specifically wanted software engineers hired to automate scaling as well as sysadmin type ops work.
> Real ones are oncall for the systems they build.
That’s what DevOps meant where I worked.
I remember the separation of developers from deployment being one of the things DevOps pushers wanted to reduce.
In fact it was the beginning of my career when the older senior people were already trying to get away from the assembly line style of software where each person worked on their part and then tossed it over to someone else to become their problem.
On-call problems are mostly just a result of Kubernetes' "cattle not pets" brainrot ideology where people throw local simulate-ability out the window because the Rube-Goldberg web of pods and operators and cluster state gets too complex to simulate locally or keep in your head (esp. if code is located in 50+ microservice repos).
Machines (esp. Virtual Machines) are ultimately predictable and crashes don't "just randomly happen". But it's what this entire "enterprise cloud" culture has brought in, the idea that it's impossible to simulate all possible failure conditions of a cluster locally and therefore we shouldn't even try doing it. Maybe that's true at Google scale, but most companies are simply not, not, not Google-scale.
As a counter-example, I recently built a simple "cloud storage" system with a minimal Linux ISO: everything is stripped down to the absolute minimum to just boot, start a minimal S3 server and pull (encrypted and signed) update commands from an SQL "control plane" database. The entire userspace is ~50k lines of Rust (minus 10k init system) and I can simulate the entire VM locally and play with the "cluster" of VMs locally. The ISO is ~20 megabytes large and I can simulate entire datacenters "burning down", virtually, locally, before it happens for real. I can simulate everything: network failures, disk failures, slowdowns, latency, try getting Fable to hack it, etc.
But in the kubernetes "enterprise" world, I can't do any of this. First I need 10.000 values.yaml files, 10 add-ons for logging and metrics (because pushing metrics to some collection endpoint over HTTP is clearly PhD level complexity) and on top you need a service mesh to restrict random containers talking to each other because nobody knows anymore what services can talk to which other services. The results are constant outages because of misconfigurations and someone having to be on-call at all time.
Needing on-call at all (except for maybe security alerts, but those are usually not part of DevOps) is the result of this Google-influenced "crashes just happen" nonsense where "manager" culture is preferred over results (aka uptime).
It's not that crashes don't happen, it's just that they don't "randomly" happen. The error codes from bad hardware are predictable (against kernel bugs... well I use an "allnoconfig" kernel, which disables like 95% of the kernel). Bad hardware, bad network, bad RAM, all of these are simulateable in QEMU or Linux (tc netem / blkdebug).
If I count my test suite for the ISO, I have ~2500 unit tests (placement groups tests, Raft behaviour tests, disk perf tests, SigV4 crypto tests, DHCP tests, kernel rollback tests), ~110 fault-injection tests (latency, network, RAM), ~60 block-store failure tests (e.g. network interrupt while two disks try to sync), "chaos tests" (generating random event scenarios for an hour), bit flip tests, "fsync lies to me" tests, power cut tests, round-trip tests, "power cut while in the middle of sync between datacenters", SMART degradation tests, load tests, memory pressure tests, file descriptor resource starvation, NTP drift tests, metadata DB loss tests, DNS failover, ....
Then for security everything is turned off, kernel modules are off, eBPF is off, io_uring is off (lots of CVEs), user namespaces are off, most network and file drivers are off, /bin/sh is off, S3 endpoint is minimal, etc.
Most rules from aerospace are said to be "written with blood", some crash caused this rule to be legalized. But with ~40k lines you can go through each line and say "what if this fails". With a kubernetes cluster it's more or less based on hope and happy-path testing.
I also scraped other competitors for user complaints, esp. about data loss / sync engine bugs and in total I now have about 4000 tests for 40k lines of backend code and another ~1000 tests for the 15k lines for the sync engine code (about 1 test for every 15 lines).
I'm still not entirely confident (still need lots of beta testing), but a chance of data loss of .01% would be pretty high. It really depends on the amount of testing whether you need an on-call person at all and most orgs are more engineered towards "just hire someone to be on-call" rather than doing proper testing.
i'm not questioning your setup, I'm saying that if you need a 10,000 machines to run a system you're going to have kernel bugs and hardware bugs and you have no choice to treat machines like cattle.
> If human SREs (okay, now we have to use adjectives like human ) only engage with Tier 3 incidents, i.e. the novel, never-before-seen outages, where do they build the intuition to handle them?
Ime this doesn't happen or reflect the reality of work. I had one mythical job where I wasn't directly on call and only received escalations and didn't have a problem with debugging whole system issues. I still worked on the whole system day to day.
Also I feel like this post is also my frustration with sre. In people's minds it's just the job that deals with issues pumped out by swes and they have to work magic because no one else can be bothered to try at their job. This just reinforces that idea.
We use AI at work, Claude specifically, and it’s faster than avg SRE for trivial tasks like cross signals correlation. But it requires a lot of effort to build a harness from your company contexts for it to work like an SRE. And it’s just a tool so if you have to build your knowledge to make it useful, otherwise it’s just a blunt. Or in other word, incompetent people are incompetent no matter of tools
Is it really that complicated to set up? A basic version is make everything work from a CLI in terminal coding agent of choice (probably already mostly there), hook it up to alerts, and let it run in a corner?
Not OP, for me the issue is nuance of all of your infrastructure, application and outside layers. There's a lot of documentation and such, but often times things don't work as expected to a degree and it requires filling in the blanks correctly. The amount of times I've had an AI Agent misinterpret an alert and assume it was XYZ when it was some random layer it had no idea about is often.
It can certainly get closer but unless your tech stack is a webserver and an application it can get complicated.
From what I've seen the more complicated the deployment and the more tedious the investigation the more useful the automation is. The agent shouldn't be interpreting the alert much beyond gathering symptoms and identifiers for drilling down into all the underlying data and code. The security side of this is very tricky because the speed that makes the tooling useful also makes it hard to supervise.
What contexts would the AI model need for analyzing, let's say, cluster-wide incident other than the observability data that such systems anyway collect already?
This is happening to SWEs too. If you have a certain amount of experience, you are capable of critically analyzing AI output. In Big Tech parlance, these are the Senior SWEs and higher. You require little to no supervision and can work independently.
But how do you make Senior+ SWEs? By having a pipeline of juniors who get trained and mentored. That pipeline is cooked now.
There is going to be a brief period where some like Staff+ SWEs are going to make a lot of money by being AI specialists. But the flood waters are rising for everyone. Those opportunities are going to shrink.
It's eventually going to be a problem that the ecosystem has died because there's no new blood and the people there lack the experience and critical thinking to understand what's being done.
I can use AI on topics I know because I can spot the errors. LLMs lie a lot and get things wrong a lot. What happens when nobody can check the output?
The point I got from the article was that AI is a cause for skill atrophy for any level of seniority, and tends to worsen over time. I can see this happening to myself in some areas, while in others I’m able to learn new skills I’m interested in.
Deskilling is not necessarily a bad thing, if what you refer to as skills are mundane things that should be automated. I do agree with the author in that people are becoming lazy when it comes to critical thinking and evaluating the decisions or work that AI presents to them. These are people who have little passion to do what they do, and for whom their role is primarily a chore.
I don't think this is as black and white. You skill up by spending time with the domain. Of course, you learn better when you are involved in situations or jobs that actually matter. But getting rid of all the "mundane" distances you from the nitty-gritty and you lose that feel for the problem.
Right, there's a reason students that never dealt with a real problem, even when they get As, aren't as good as professionals. You need to do real work to get really good.
Yes. I bet it was really fun debugging punch card computers.
Also, skills can be re-acquired, if needed. People are acting like it's the end of the world for them, like their knowledge will become impossible to understand anymore. I'm not as fast at solving equations as I was in high school, but I still understand the logic behind it.
> "Just as the FAA mandates manual flying time even when the autopilot is perfectly capable, SRE teams can designate certain incidents, even the ones the AI could handle, as human-practice opportunities"
This is a great example. Aviation is a perfect industry to learn from, they figured out a lot of things about how to keep humans from making mistakes. While complexity increases and automation takes away more and more tasks.
Aviation is a good example if you want to have zero fatal mistakes. There is a way and there is a cost to do so.
The same way erlang is good example of how to run a highly available system with hot reload and observability and stuff. There is cost for that too.
Maybe not all of the projects need the same properties. I'm fine if some of the stuff is having scheduled or unscheduled maintenance. Some kind of government portal that I use once a year had my permission to be offline between 0300 and 0500 and then again after 1200.
A better metaphor would perhaps be machining. You will not find a shop that has no CNC anywhere, but the machinists who can also use manual machines will always have an edge for certain things. Not that they're working manually much in their day to day, and not as all as a failsafe, but you can just tell who has that kind of background.
(And you can also tell whose mind is stuck in the old world, too.)
The problem is that people go on Twitter and see the posts from "thought leaders" and "influencers" who say things like "reading the code is not needed". Those people are not working on systems that have paying customers who would cancel their subscription - or worse - if the software became too defective or unreliable.
You only need a handful of occasions of creating massive billing errors for your customer's customers before they will slag you to all their CTO friends and pull the plug. Imagine Stripe began double-charging every third transaction once a week, and couldn't fix it after two months of thrashing because no one there has any understanding of how anything works anymore. They are simply done at that point, all the value of their brand lost.
As if engineers, especially those writing software to handle financial transactions (which are usually in regulated businesses), don’t know how to write conformance tests with invariant cases.
soooo wouldn't that be exactly the "deskilling" being discussed here, and isn't that deskilling concerning for money-handling firms?
having seen both (a) mission-critical human-built systems become nightmares for maintainability and extensibility and (b) bleeding-edge ai coding agents confidently and quickly fix a wrong problem in the face of an issue (or fix it in a slightly wrong way), I remain skeptical that this great de-skilling will all be perfectly fine.
of course, if you aren't looking at the code you won't fucking know until someone yells...
I disagree with several premises of the article. There is not going to be human led investigations of outages. People do not need to train on easy outages to be able to handle hard outages. Even if you wanted to train people on easy outages AI makes it easier than ever to setup up real world like issues to replace the need for real world experience. I also disagree in labeling it deskilling when it is a reallocation of skills to what is actually needed.
>concerning for money-handling firms
The real world is robust. It's okay for money handling code to have bugs. It's not the end of the world and AI will be able to have a lower bug rate than humans.
> Those people are not working on systems that have paying customers who would cancel their subscription - or worse - if the software became too defective or unreliable.
Name one. Point me to one developer that actually works - themselves, not just in the same company - on a system with serious customers that demand correctness and is spreading that message on Twitter.
I think the parent take is valid if you disqualify people managers/leaders rather than ICs. I think that's also valid to disqualify those takes because they aren't in the trenches every day. They may have points strategically for what's best for the company, but not what's best for the workforce or the industry.
I don't see what the giant social cost of AI hucksters is supposed to be... who cares if some random influencers claim you can vibe code your entire existence? the thing that bothers you is that the grifters are basically right, just for the wrong reasons (and a bit too early). Comments like this imply that there is some cavalcade of situations like this that have caused actual harm to actual people, but it is actually just an emotional response to people who are personally distasteful, and borne out of a desire to make sure that the "assholes aren't proven right". stuff like "Imagine Stripe began double-charging every third transaction once a week" is just so silly... what about AI makes this more likely in any way whatsoever? If anything, AI would have such a subtle scheme for robbing us blind that it would defy characterization into pithy quotation marks. this feels like a complaint about humanity dressed up as a complaint about AI. for all we know, AI could actually improve the situation-- it's not like humanity has set some unimpeachable precedent.
AI can absolutely improve the situation. It has helped find and fix bugs I have despaired of ever fixing. But it cannot learn, and it cannot understand things for you. And people outsourcing all their understanding to it absolutely could lead to existential disaster for a system at the heart of the platform. Coding agents write too much code and ignore existing patterns. This has been studied quite seriously and they generally write 2.5x the amount of code a human does when refactoring or enhancing code in a real project. When a very large system slips past a critical point of decoherence you really cannot put any certain boundaries on what kind of problems become possible or how long it may take to fix them.
Yes, there are responsible engineers using AI in innovate ways and making things both more productive and better. But there are A LOT of lazy executives, and lazy developers who are spewing a lot of slop all over the place.
> it cannot learn, and it cannot understand things for you
the former is demonstrably untrue (and wildly so), and the latter is just an egocentric/anthropocentric "the genie in a bottle only answered my wishes, it didn't tell me what to wish for". buried in that statement is the presupposition that humans being able to understand that something is actually Important; short of religious/faith-based reasons I don't see the justification for the primacy of human understanding when it comes to the determination of Truth. if anything, being "understandable" to humans strikes me as an anti-goal. are we really to believe that human comprehension is the limiting factor keeping society from degrading into "decoherence"? how well is that going for us? the reason humanity places such a premium on identifying "The Truth" is precisely because of how illusory it is for our fallible minds. LLMs struggle with many things, but coherence is least among them. that is why "slop" works so well-- if it wasn't at least mildly coherent, it would be easy to discriminate.
> there are A LOT of lazy executives, and lazy developers who are spewing a lot of slop all over the place
and there are even more lazy humans who are happy to ascribe this to a shortcoming of the technology and not of humanity itself.
insofar as we are descending further into post-truth, it isn't because of AI slop blurring the lines between fiction and reality; it is because we are all collectively realizing that Truth isn't all it's cracked up to be. LLMs are forcing us to take a long hard look into the mirror and we don't necessarily like what we see.
Fuck aviation, stop using it as a dead horse metaphor for safety through human protocols. I am private instrument rating holder and there's very much a top down push for single pilot ops. Your newer jets (even the jetliners) are all designed with that in mind even though they may not be certified as such.
This is kind of an interesting thought (underneath the grumpiness, which I guess is where the downvotes came from). Programmers seem to hold aviation processes in very high regard, but their industry is also under cost pressures. The “UX” of riding in a plane does kind of suck, I wonder if some of that comes from the costs involved.
Not too surprised to see pressure to reduce the number of pilots.
Oh yeah, and you will find very quickly that most people are a bunch of flaming hypocrites when it comes to flying. Lots of wannabe aerospace engineers in tech. Half the people I have met who put aviation on a pedestal hate flying but love to trumpet the statistics of how safe commercial aviation is. Try asking them to get into a Cessna and all of a sudden the excuses come out. It's oh noes I have a fear of heights, oh noes general aviation is less safe and only slightly safer than a motorcycle, oh noes the FAA is corrupt and allowed the helicopter crash in DC to happen, oh noes I have a family etc.
Aviation safety discussions on hacker news have barely more more depth than biotech and healthcare discussions here. It's extremely ironic because a tremendous number of pilots are engineers, and especially software engineers. It's not exactly a cheap hobby. But you don't see massive numbers of English majors come crawling out of the woodwork to join a discussion on the safety of C++ vs Rust, but tech bros flock to worshipping aviation as a platonic safety idol like moths to a flame.
Err...have you actually asked someone to get into a Cessna?
Also, "fear of heights" is a totally valid excuse. Fear is a gut reaction and may have nothing to do with one's justified true beliefs. Eg., I know spiders on probability won't harm me but there simply won't be any amicable co-existence between me and any arachnid.
I don't understand the perceived hypocrisy. The chance of death in a 1hr Cessna ride is something like 100x a domestic commercial jet flight, so it's reasonable to have different views of them? Maybe I'm missing your point.
Okay, just because aviation is now also developing in the wrong direction (maybe), all the things they did right in the past are suddenly pointless? I suggest to learn from the things they did right, not the things they did wrong.
Aviation is a perfect industry to learn from if you are willing to hold IT to the same standard of regulatory scrutiny. You can bet your entire net worth that anyone running anything in the industry would fight tooth and nail to prevent that sort of regime from coming into force.
A (very) key difference is that this behaviour is mandated by the FAA, and consistent across all providers. There isn't a (legal) option of just not-doing-it because it's "cheaper".
Modern companies generally do not have the same incentive to keep their SRE (or engineers in general) upskilled. Cutting SRE because "we don't really have incidents, why do we keep paying them? If we need to do layoffs I would rather keep a product engineer" will always* (okay, _generally_) be the direction that organizations go without external influence or requirements.
Now I am sure there are some companies or industries where this is required (for example, I would not be surprised if certain military software/services contracts had requirements for support staff with X hours or training/etc), but that is not consistent everywhere.
If we want something that is "money-losing" in the short/medium term to be consistently applied, it unfortunately needs to be (uniformly) regulated.
You're right, there is no mandate for learning and doing better in SW engineering. Many companies just don't care, also kind of fine. This doesn't mean that everyone doesn't care and nobody wants to improve.
The thing is, without the FAA mandating something like this for SRE’s, no one will do it. In the moment, getting the broken thing back up and running is seen as far more important than the learning opportunity.
I noticed this constraint a few years ago when the SRE/DevOps hiring market was really tight, but no one was investing in non-senior talent. I ended up founding a startup that tried to address this by supplying training via simulated outages. Maybe we were too early.
The plane lands in the same amount of time whether an autopilot lands it or a human, so there's no incidental cost.
If you let a human SRE handle a problem completely, not only will it likely take longer than an AI, but since that SRE is now deskilled, it may take FAR longer. And that's a cost. Who's going to pay it?
no offense, but I think aviation is a horrible example. if we treated every problem like we treated aviation, we would still be living in the 1950s. honestly, probably the 1850s. hell, we sent a guy to the moon LONG before it was actually safe to do so. (is it safe even now?). do we really think LLMs would be more blasé about human life than humanity itself up until... iono, since the Baby Boomer generation? the "issue", such as it is, with AI is not "we need to make so many guardrails in order for it to be useful", it's that unlike aviation, where the ground truth is "did the plane crash or not?", AI is an open-ended experiment in what would happen where we don't have any feedback from anything other than... how humans feel about it. humans are the antithesis of paperclip maximizers.
I don't see any reason whatsoever why "AI regulations" would become antifragile from their having been "written in blood" in the way that aviation regulations have been; if anything, our inevitable present and future AI crises will each be exponentially more devastating than the last. Modern aviation is a triumph because of how fallible humans are-- that's why we delegate control of aircraft mostly to machines. The shape of AI disruption isn't "the computers aren't following the rules", it's that even by our own rules of rationality our world is terribly underspecified-- we have computer security that isn't secure, subjective rules of ethics/fairness/judgment that defy straightforward explication, "knowledge work" that is actually neither "knowledge" nor "work"... the whole situation is much closer to the human domestication of the dog (letting a potential threat into one's home) than any traditional story of humans coming to terms with their own technological innovations. human behavior is FAR easier to predict than un-aligned AI behavior-- that is why it is such a singular moment in human history.
* AI-resistant organizations led by graybeards who refuse to send telemetry off-prem (to either observability vendors or to inference providers) and will be dragged, kicking and screaming, into the future, over their cold, dead bodies. There's a lot of large, conservative organizations in this category like banks and governments, where budgets are rubber-stamped and Don't Break It If It Ain't Broken attitudes are tacitly approved of.
* AI-accepting organizations where Ops will go through the same great narrowing that Dev is going through now - build the dark factory and let it run. You do still need people who know what good architecture looks like and to shepherd changes to improve the overall system to make it more observable, scalable, and maintainable. You do still need Non-Functional Software Engineers who don't answer to Product, for the sheer political reason that you need people who have a long-term perspective, compared to Product that has a short-term perspective. The business will make decisions like "Vendor X is too expensive so we're migrating to Vendor Y" and you will need somebody to drive the agents to make that happen in a way that respects availability and compliance requirements. You probably don't need, in the long-term, a whole team of people to handle it, let alone a whole group or a whole division.
At the end of the day... focus on being a strong engineer. Somebody will always be needed to take care of the trees.
I'm SRE right now and just responded to an incident. The only thing AI has changed is improved my ability to quickly find in the logs what happened when normally I'd have let it go if it was a small hickup as long as everything is green again. There's so many small things we used to not bother persuing. Now I have it write down it's findings no matter how small the issue.
Anyway it's ultimately just running scripts I can run myself.
Is your org at the point where it would make sense just to upstream patches for FOSS projects and pull rather than patch and push to just your infra?
Just a matter of curiosity for me as my career progresses, where I think that is the ideal, but I don't work in a heavily privatized nor regulated space at the moment of posting.
I did read your latest blog post looking for an answer, but it wasn't clear to me from that reading.
I generally don't reveal much about my current job on a blog post for security reasons but I can explain some of the reasoning for having your own build system.
We have many packages internally so inevitably we need to build them which necessitates an entire build matrix on it's own. But once you have your own build matrix adding a new package is trivial and most build in under 10 minutes in github actions. One issue is we don't want to use Canonical's default gcc flags so like Gentoo I setup a make.conf that is specific to the suite (22.04, 24.04, etc). This has allowed me to target a higher tier instruction set which is it's own micro optimization. Folks that upgrade blindly from 22.04 to 24.04 will see their performance drop from gcc compile flags alone. Also we have people who have access to CVE fixes before they are generally available so that's also another important reason. Anyway we DO use upstream most of the time for setting up a project folder or normalizing it against a new suite but then building it is using different gcc flags and might be pulling in our version of curl or something.
I like this analogy that I've come up with: Non-writers can produce impressive articles that are unbearable to read the same way non-programmers can produce impressive software that is unbearable to use.
If it deskills that guy who reaches for complex bash when they should reach for golang but they don't want to bother with writing the build layer in the image, then I'm all for it.
Here's a thought experiment: why is it that I can't do long division?
The Calculator!
Calculators deskilled long division!
Thing is though: who gives a damn about long division? The real task is your taxes or making a massive BI report for your presentation tomorrow.
AI "deskilling" due to automating things makes old key skills become something you can just ask for and have done for you. Why bother staying skilled at those things when you can broaden? It clears the way for what used to take forever and be a specialist skill, to become something to steamroll through on your way to finishing a much greater outcome/task. It's the micro way that, in the macro, humanity becomes far more capable as a species. The gates are being flung wide open and the gatekeepers are being lapped and they have blogs!
It's not a tragedy, it's universal access to leverage.
Well, I'm sorry to hear that. I have a calculator and some pretty neat computing hardware. If I go into the attic and search long enough I can probably find my old sliderule. I can also do long division, as can all of my kids. I think it is an essential skill to understand all basic mathematical operations, and to be able to use them on small data sets. If I didn't know how to do division without a calculator I would consider that a failure of my basic education.
> in the macro, humanity becomes far more capable as a species
Not humanity, a small number of people in this planet, will benefit about this, because now they no longer have to depend on under payed workers to do the job, and will use AI to make the lives of all other human being poorer.
I think that we need to oppose this tech dictatorship, first thing stopping using AI and thus stop depending on an handful of companies that are able to provide it, that while they gain more and more power we all gets poorer.
What was wrong in using our brain, in the end? I still use my brain on my job and I always will, maybe one day they will say that there is no more need of computer experts because AI can do everything (still people bring me their laptop to reinstall Windows, something I've done for 20 years and that AI didn't change), I will just find another job but I will refuse to use this AI shit.
I think a closer analogy would be forgetting basic arithmetic as a concept. Imagine how hard it would be to work with more complex math without understanding what division is. How do you prompt the genie, or understand its outputs, without the foundational knowledge?
Is basic arithmetic something that can be forgotten? Studies have suggested that even infants have an understanding of basic arithmetic. It seems to be something innate.
But as we're only imagining anyway, let's forget about the genie for a moment. If we did lose our sense of basic arithmetic all of a sudden what would life at its core even be like? If there were two apples on a table you wouldn't be able to comprehend what happens if you took one. Would we even be able to function like that?
You're taking the analogy too literally. The point that I'm trying to make is that in math, and a lot of areas, is that concepts build upon one another. To understand something at a 7/10 of complexity it first needs you to understand the 6/10 concepts that it builds upon.
I believe it is more than you've missed the forest for the trees. You haven't accurately explained what understanding means. One might not rigorously understand x÷x=1 but still 'naturally' see that having two apples and taking one leaves one. And that's the thing: The genie, as in LLMs, are analogy machines so you can approach problems without a formal understanding as long as you can grasp the analogies. We have real world proof with 'vibe coders' producing incredibly complex math without much formal understanding of it. They know how to describe the input and interpret the output using words that relate to everyday things that they are familiar with, much like the apples on the table, and it accomplishes their goal.
Perhaps there is a limit, sure, but I suppose that's where you lead with a concrete example to explain where that line is.
SRE was always about understanding; understanding data flow, understanding bottlenecks, understanding logs, understanding application architecture, understanding end user functionality, understanding break points, SLIs, SLOs. Understanding how to mitigate, gracefully deteriorate, restart, restore, recreate, scale, etc.
There is no understanding with AI-generated applications, full stop.
idk, I develop a lot of connected systems: Pis, Macs, lights, Google speakers, custom ESP32 boards. Understanding the capabilities and communication methods to pool them together is still very much part of what I do. If you're thinking one prompt -> production via AI, sure. But I don't think we are anywhere near that because there are so many moving pieces which still require human sign off: legal, design, QA, etc. To me, it just shift ups the level of understanding rather than replaces it.
Seems like an odd criticism given that programming has always been about abstraction. But sure, I don’t know tree-level details (functions) but I also wasn’t doing leaf-level (bits) understanding because that became obsolete. Now I’m on forest level and expanding to eco systems.
You can always know more but at a certain point, after you’ve built enough validation and safe guards, you find it’s not the best use of your time and the things you can make beyond your specialization is quite remarkable.
Also, I think the toothpaste isn’t going back in the tube with this
The internet is full of people doing the “AI = compiler” comparison and it’s just wrong.
Maybe on a really small scale they can resemble a compiler because the number of possible outcomes is so small that it’s nearly deterministic.
But at any larger scale AI will not be deterministic anymore and at that point it will be relevant what the resulting code looks like. Because it can and will be slightly or very much off from your instruction.
I figure out how to do something, wrap it in a bunch of functions, and then kinda just use it. If it just keeps doing what it should, I may kinda forget how it works. But I don't forget that I made it, and that I understood it, that will never not have happened. So in a way, I am able to "forget" some things because I still know I knew them. And I trust that programmers who write programs that I dream about writing also know what they're doing.
With LLM, you accumulate stuff with from knows where, ultimately fished from a vast ocean named question mark, by whatever turns of phrase you and involved agents happened to use, and what the temperature was at that moment. Which is of course where all the harnessing and looping and running code on the output comes in, but still, ultimately you have no idea. Which is fine for prototypes, but long term? Good luck.
I guess I'm just lost on things not being solved to a degree. Maybe its that I'm an iOS engineer where things are so locked down as far as goals and efficiencies that AI doing the work is verifiable by running the program, setting up some pre and post verification of code, and writing tests that look for certain edge cases. To be frank, it just seems like a trust thing. I trust it to the degree I am willing to to verify it.
“The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.” — Edsger Dijkstra
I've thought a lot about this quote, and the best abstractions I use are absolutely this. I love array indexing as an abstraction, but in the back of my head I have pointer + index * size. I like having the kernel do memory mapping, but I know there's a buddy allocator and TLB cache there.
For the last 10 years, I don't think I met a lot of senior architects or dev who could understand all that.
Companies loves to have "specialized experts" who skilled up in one afternoon in their field.
It might be different in startup, but in consulting world in relatively large enterprises, it's just a fucking joke.
I've seen these people rapidly disappear in the last 5 years at a big tech co. Old UNIX heads that had experience back to the 80s being made redundant. Who is replacing them? Mostly offshore 20 something year olds on a fraction of the salary a tenured person had, being told to just follow runbooks, pasting the runbooks into Claude or ChatGPT. These people can't explain anything, didn't really cut their teeth on doing things out of curiosity, and if asked a question, will just be playing copy-paste-back-and-forth with a LLM.
Agree up to certain levels but your examples are too specifc.
Understanding should come in different levels abstraction of what you are building,and managing.
There are different levels SREs, from beginner to say Fellow level.
Do you expect the Fellow level to understand nitty gritty details of each component his apps or services are interacting with and how each of them start? No. Especially when they have to deal with hundres or thousands of services each of them hosted by different teams, orgs, or different companies.
junior SREs, yeah probably got their hands dirty with those nitty gritty details you listed because they tend to deal with smaller or lower levels of components
So my point is, I agree withyour "Understanding" but not specific details as they differ per one's level of experience
Have the AI explain it to you. You probably won't understand it as well as if you created it yourself, but there is (the potential for) lots of understanding.
What would you think of easier-to-debug infrastructure building blocks? I'm building an easier-to-debug web server and reverse proxy, and I want to know how this would compare to trying to use AI SRE agent to reduce debugging friction. Or maybe both complement each other?
There's a lot of chatter about how AI is advancing mathematics at the expense of human understanding. This is going to be true for software industry also at some point. The main gripe of mathematicians is how AI is discouraging young people from persevering. Right now job market is rough for junior SWE and unlike mathematics professors, nobody gives a damn. A few years down the line we'll see a real shortage of software engineers who "understands" and it will come back to bite the industry.
No it's forcing everyone to realize all the jargon was made up to sell books and land speaking engagements. Money from patents (2016 Alice ruling upending that whole scam).
Yall this was always just a money making scheme.
Never was it meant to carry humanity into the future.
I think people are getting funneled into a new career path of being a technologist or technology director. People who can ask the right questions at the right time and direct attention to the appropriate areas given the goals and feedback.
People don't appreciate how much of engineering is just as you've described.
When they imagine someone using AI (except for themself), they imagine the least curious, least inquisitive vibe-coder. And they invent the idea that the only way to be someone who asks questions or remains curious is to have paid the dues that they see themself as having paid. Therefore the world must always need them.
Well, all you have to do now is imagine the vibe-coder who has the same curiosity and inquisitiveness that you had especially when you were starting out and weren't yet a professional programmer. The only difference is that now you don't have to bootstrap a whole software career to understand how things work; you can skip that and merely ask.
Admittedly, a lot of my identity was wrapped up in those hard earned skills. I have empathy for those struggling to chart a new path for themselves. It does not, however, change the fact that tools evolve. Ultimately, using tools effectively is how we are measured and while you may have needed a shovel yesterday, today you have an excavator.
> "...imagine the vibe-coder who has the same curiosity and inquisitiveness that you had..."
We're required to imagine them because they don't seem to exist in the real world. The vibe-coders seem almost entirely composed of the "I commit 20 kLOC a week without reading a single line, lol" or the "LLMs are like calculators/basic arithmetic skills are obsolete" lunatics.
This has always been the most valuable part of the job if you ask me. Its just that the other parts of the job were quite hard to get right letting people build a career out of that specialization.
I work a "dev"ops job, and like yeah, I'm pretty sure this field is even more fucked than software engineering is.
IaC codebases, configs, utility scripts are relatively tiny, and fit into context windows whole. Models are eating them for breakfast.
They're really good at incident response. Sadly, much better than our dedicated colleagues from <location> ever will be. They'll chew through cloud service logs, application logs, application code, iac code and config, CI output, etc. to get whatever is needed, cause wide knowledge is no issue. Even disassembling stuff is no barrier, tracing things down from app level, to runtime level (e.g. JVM), to native calls.
They're really good at handling deployments, end to end infra setup and migrations, following checklists. Hell, they don't even really need checklists. Something that cannot be said about other dedicated colleagues from <location>, who cannot breathe without a checklist, will skip over steps, or otherwise very creatively misinterpret them or improvise something wicked.
They're really good at spinning up mad nice dashboards, and keeping business types happy with impressive looking write-ups they'll never read. It's really refreshing to not have to write those write-ups anymore (or the aforementioned checklists).
It's straight up both not looking good and looking better than ever. Having it be fully hands off remains an issue though.
Eh, I think they're still terrible at architecture. They'll find bugs but they'll try to make them go away instead of building a sustainable system.
They also have a propensity to try to rewrite gigantic amounts of code in one go making changes riskier and harder to scrutinize... without using more LLMs.
254 comments
[ 1.8 ms ] story [ 83.3 ms ] thread[1] https://www.youtube.com/watch?v=EWuf2EekRSM
I'll probably get downvoted for this but: At 15 YoE, hell no.
Day-to-day coding lost it's allure to me about 3-4 years ago; it's just a means to an end. I think AI came at the right time for me.
It's so depressing a thought that if I continue on this path, in a few years, let alone a decade, I'll be utterly unable to produce or even understand code. I realize the skill may not be very economically useful, but losing it is still a depressing outcome that's preventable in theory. At the same time I'm feeling a deep apathy towards building anything. Playing a Zachtronics game is the farthest my internal motivation can push me towards...
I'd love to hear what others are thinking/doing.
I'm more worried about people that did never program and started directly with LLM support...
I use my agent to prototype a rough version that mostly works, and then I iterate through (with or without my LLM) to clean it up. Then back to the top with a new feature until I’m satisfied with the project and code.
I do tend to like it. My real selling point is that I’m not “locked in” to any choices until I’m happy with how it looks. I used to have to build the thing to see how it looks and acts, and going back on those choices was painful.
I do think there will be value in it, but I’ve been wrong before. My agent occasionally makes stupid choices that make every prompt after that 4x as hard to fix. Like I screwed up a pydantic model in an unreleased library and it supported a “legacy” path no one would ever use for like the next 5k LoC. By the time I noticed, legacy handlers were like 25% of the codebase.
That’s gonna burn a looot of tokens if you scale it up to a million line of code app.
I personally have a project that have been spending 2 months now making the specs and blueprint, more than 1000 files defining every bit including the future features too, and not a single code has been written yet, later, an agent will use this to produce the syntax and all the needed tests to have exactly what’s been needed.
AI is already pretty good at this in "SRE" workflows.
> AI-driven change risk assessment
And half decent at this.
If Claude goes down or stops existing when we go down, we're fucked.
You're conflating a whole lot of different models here. In fact, "developers should be on call for the systems they build" is a principle that most schools of DevOps thought emphatically subscribe to.
When these problems because largely solved with off-the-shelf software SRE became either a bureaucracy (ensuring reliability through policy and enforcement) or just deploying other people's code and holding the pager.
But I really do need the company to have a competent plan for handling issues that isn't just, "call the developer." It's amazing how much it makes them care about proper support and testing when the alternative isn't exploiting a developer, but is a customer outage.
Realistically nothing is perfect, there will be cases where something randomly breaks and will need attention after hours. Responding to a call for this should get you double pay for your time. Plumbers and electricians work this way, why not software folks?
If it absolutely has to get fixed, the company will pay the cost. If it can wait until Monday morning, they can save that cost.
Yes! This is devops, thank you for coming to my ted talk.
Coincidentally I’ve found putting engineers oncall for their stuff really does wonders to ensure they check their AI agents’ code and know how it works.
SRE just came from the big guys calling their sysadmins SRE
That’s what DevOps meant where I worked.
I remember the separation of developers from deployment being one of the things DevOps pushers wanted to reduce.
In fact it was the beginning of my career when the older senior people were already trying to get away from the assembly line style of software where each person worked on their part and then tossed it over to someone else to become their problem.
Machines (esp. Virtual Machines) are ultimately predictable and crashes don't "just randomly happen". But it's what this entire "enterprise cloud" culture has brought in, the idea that it's impossible to simulate all possible failure conditions of a cluster locally and therefore we shouldn't even try doing it. Maybe that's true at Google scale, but most companies are simply not, not, not Google-scale.
As a counter-example, I recently built a simple "cloud storage" system with a minimal Linux ISO: everything is stripped down to the absolute minimum to just boot, start a minimal S3 server and pull (encrypted and signed) update commands from an SQL "control plane" database. The entire userspace is ~50k lines of Rust (minus 10k init system) and I can simulate the entire VM locally and play with the "cluster" of VMs locally. The ISO is ~20 megabytes large and I can simulate entire datacenters "burning down", virtually, locally, before it happens for real. I can simulate everything: network failures, disk failures, slowdowns, latency, try getting Fable to hack it, etc.
But in the kubernetes "enterprise" world, I can't do any of this. First I need 10.000 values.yaml files, 10 add-ons for logging and metrics (because pushing metrics to some collection endpoint over HTTP is clearly PhD level complexity) and on top you need a service mesh to restrict random containers talking to each other because nobody knows anymore what services can talk to which other services. The results are constant outages because of misconfigurations and someone having to be on-call at all time.
Needing on-call at all (except for maybe security alerts, but those are usually not part of DevOps) is the result of this Google-influenced "crashes just happen" nonsense where "manager" culture is preferred over results (aka uptime).
Uh, I can literally see thousands of crashes happening all the time. You think there isn't a .01% chance of hardware going bad or hitting kernel bugs?
If I count my test suite for the ISO, I have ~2500 unit tests (placement groups tests, Raft behaviour tests, disk perf tests, SigV4 crypto tests, DHCP tests, kernel rollback tests), ~110 fault-injection tests (latency, network, RAM), ~60 block-store failure tests (e.g. network interrupt while two disks try to sync), "chaos tests" (generating random event scenarios for an hour), bit flip tests, "fsync lies to me" tests, power cut tests, round-trip tests, "power cut while in the middle of sync between datacenters", SMART degradation tests, load tests, memory pressure tests, file descriptor resource starvation, NTP drift tests, metadata DB loss tests, DNS failover, ....
Then for security everything is turned off, kernel modules are off, eBPF is off, io_uring is off (lots of CVEs), user namespaces are off, most network and file drivers are off, /bin/sh is off, S3 endpoint is minimal, etc.
Most rules from aerospace are said to be "written with blood", some crash caused this rule to be legalized. But with ~40k lines you can go through each line and say "what if this fails". With a kubernetes cluster it's more or less based on hope and happy-path testing.
I also scraped other competitors for user complaints, esp. about data loss / sync engine bugs and in total I now have about 4000 tests for 40k lines of backend code and another ~1000 tests for the 15k lines for the sync engine code (about 1 test for every 15 lines).
I'm still not entirely confident (still need lots of beta testing), but a chance of data loss of .01% would be pretty high. It really depends on the amount of testing whether you need an on-call person at all and most orgs are more engineered towards "just hire someone to be on-call" rather than doing proper testing.
Ime this doesn't happen or reflect the reality of work. I had one mythical job where I wasn't directly on call and only received escalations and didn't have a problem with debugging whole system issues. I still worked on the whole system day to day.
Also I feel like this post is also my frustration with sre. In people's minds it's just the job that deals with issues pumped out by swes and they have to work magic because no one else can be bothered to try at their job. This just reinforces that idea.
It can certainly get closer but unless your tech stack is a webserver and an application it can get complicated.
But how do you make Senior+ SWEs? By having a pipeline of juniors who get trained and mentored. That pipeline is cooked now.
There is going to be a brief period where some like Staff+ SWEs are going to make a lot of money by being AI specialists. But the flood waters are rising for everyone. Those opportunities are going to shrink.
It's eventually going to be a problem that the ecosystem has died because there's no new blood and the people there lack the experience and critical thinking to understand what's being done.
I can use AI on topics I know because I can spot the errors. LLMs lie a lot and get things wrong a lot. What happens when nobody can check the output?
Also, skills can be re-acquired, if needed. People are acting like it's the end of the world for them, like their knowledge will become impossible to understand anymore. I'm not as fast at solving equations as I was in high school, but I still understand the logic behind it.
We will adapt just fine, like we did before
This is a great example. Aviation is a perfect industry to learn from, they figured out a lot of things about how to keep humans from making mistakes. While complexity increases and automation takes away more and more tasks.
The same way erlang is good example of how to run a highly available system with hot reload and observability and stuff. There is cost for that too.
Maybe not all of the projects need the same properties. I'm fine if some of the stuff is having scheduled or unscheduled maintenance. Some kind of government portal that I use once a year had my permission to be offline between 0300 and 0500 and then again after 1200.
You might be, and your customers may not! And your competitor could be more than happy to offer a more reliable service.
(And you can also tell whose mind is stuck in the old world, too.)
You only need a handful of occasions of creating massive billing errors for your customer's customers before they will slag you to all their CTO friends and pull the plug. Imagine Stripe began double-charging every third transaction once a week, and couldn't fix it after two months of thrashing because no one there has any understanding of how anything works anymore. They are simply done at that point, all the value of their brand lost.
having seen both (a) mission-critical human-built systems become nightmares for maintainability and extensibility and (b) bleeding-edge ai coding agents confidently and quickly fix a wrong problem in the face of an issue (or fix it in a slightly wrong way), I remain skeptical that this great de-skilling will all be perfectly fine.
of course, if you aren't looking at the code you won't fucking know until someone yells...
>concerning for money-handling firms
The real world is robust. It's okay for money handling code to have bugs. It's not the end of the world and AI will be able to have a lower bug rate than humans.
That's just not true.
Real customers. Real revenue. Many industries. All written with AI.
Yes, there are responsible engineers using AI in innovate ways and making things both more productive and better. But there are A LOT of lazy executives, and lazy developers who are spewing a lot of slop all over the place.
the former is demonstrably untrue (and wildly so), and the latter is just an egocentric/anthropocentric "the genie in a bottle only answered my wishes, it didn't tell me what to wish for". buried in that statement is the presupposition that humans being able to understand that something is actually Important; short of religious/faith-based reasons I don't see the justification for the primacy of human understanding when it comes to the determination of Truth. if anything, being "understandable" to humans strikes me as an anti-goal. are we really to believe that human comprehension is the limiting factor keeping society from degrading into "decoherence"? how well is that going for us? the reason humanity places such a premium on identifying "The Truth" is precisely because of how illusory it is for our fallible minds. LLMs struggle with many things, but coherence is least among them. that is why "slop" works so well-- if it wasn't at least mildly coherent, it would be easy to discriminate.
> there are A LOT of lazy executives, and lazy developers who are spewing a lot of slop all over the place and there are even more lazy humans who are happy to ascribe this to a shortcoming of the technology and not of humanity itself.
insofar as we are descending further into post-truth, it isn't because of AI slop blurring the lines between fiction and reality; it is because we are all collectively realizing that Truth isn't all it's cracked up to be. LLMs are forcing us to take a long hard look into the mirror and we don't necessarily like what we see.
Not too surprised to see pressure to reduce the number of pilots.
Aviation safety discussions on hacker news have barely more more depth than biotech and healthcare discussions here. It's extremely ironic because a tremendous number of pilots are engineers, and especially software engineers. It's not exactly a cheap hobby. But you don't see massive numbers of English majors come crawling out of the woodwork to join a discussion on the safety of C++ vs Rust, but tech bros flock to worshipping aviation as a platonic safety idol like moths to a flame.
Also, "fear of heights" is a totally valid excuse. Fear is a gut reaction and may have nothing to do with one's justified true beliefs. Eg., I know spiders on probability won't harm me but there simply won't be any amicable co-existence between me and any arachnid.
Yes? In practice once you convince them to get into the plane the only real issue in the air for those new to flying is air sickness.
Also fun fact a good number of pilots claim to have a fear of heights (never met one irl except online in aviation forums).
That 2nd pilot makes up for lot's of impossible branches in the fault tree.
Modern companies generally do not have the same incentive to keep their SRE (or engineers in general) upskilled. Cutting SRE because "we don't really have incidents, why do we keep paying them? If we need to do layoffs I would rather keep a product engineer" will always* (okay, _generally_) be the direction that organizations go without external influence or requirements.
Now I am sure there are some companies or industries where this is required (for example, I would not be surprised if certain military software/services contracts had requirements for support staff with X hours or training/etc), but that is not consistent everywhere.
If we want something that is "money-losing" in the short/medium term to be consistently applied, it unfortunately needs to be (uniformly) regulated.
Aviation is used to paying through the nose for stuff. For developers, employers want to pay as little money as possible.
Cut dev salaries by a factor of 3, and then it would be normal.
I noticed this constraint a few years ago when the SRE/DevOps hiring market was really tight, but no one was investing in non-senior talent. I ended up founding a startup that tried to address this by supplying training via simulated outages. Maybe we were too early.
Those will be fun times.
The plane lands in the same amount of time whether an autopilot lands it or a human, so there's no incidental cost.
If you let a human SRE handle a problem completely, not only will it likely take longer than an AI, but since that SRE is now deskilled, it may take FAR longer. And that's a cost. Who's going to pay it?
I don't see any reason whatsoever why "AI regulations" would become antifragile from their having been "written in blood" in the way that aviation regulations have been; if anything, our inevitable present and future AI crises will each be exponentially more devastating than the last. Modern aviation is a triumph because of how fallible humans are-- that's why we delegate control of aircraft mostly to machines. The shape of AI disruption isn't "the computers aren't following the rules", it's that even by our own rules of rationality our world is terribly underspecified-- we have computer security that isn't secure, subjective rules of ethics/fairness/judgment that defy straightforward explication, "knowledge work" that is actually neither "knowledge" nor "work"... the whole situation is much closer to the human domestication of the dog (letting a potential threat into one's home) than any traditional story of humans coming to terms with their own technological innovations. human behavior is FAR easier to predict than un-aligned AI behavior-- that is why it is such a singular moment in human history.
It's mildly orthogonal, but I think the metaphor is really quite apropos.
* AI-resistant organizations led by graybeards who refuse to send telemetry off-prem (to either observability vendors or to inference providers) and will be dragged, kicking and screaming, into the future, over their cold, dead bodies. There's a lot of large, conservative organizations in this category like banks and governments, where budgets are rubber-stamped and Don't Break It If It Ain't Broken attitudes are tacitly approved of.
* AI-accepting organizations where Ops will go through the same great narrowing that Dev is going through now - build the dark factory and let it run. You do still need people who know what good architecture looks like and to shepherd changes to improve the overall system to make it more observable, scalable, and maintainable. You do still need Non-Functional Software Engineers who don't answer to Product, for the sheer political reason that you need people who have a long-term perspective, compared to Product that has a short-term perspective. The business will make decisions like "Vendor X is too expensive so we're migrating to Vendor Y" and you will need somebody to drive the agents to make that happen in a way that respects availability and compliance requirements. You probably don't need, in the long-term, a whole team of people to handle it, let alone a whole group or a whole division.
At the end of the day... focus on being a strong engineer. Somebody will always be needed to take care of the trees.
Ref: Agent management by Microsoft, AI Gateways springing up everywhere, etc.
Anyway it's ultimately just running scripts I can run myself.
Just a matter of curiosity for me as my career progresses, where I think that is the ideal, but I don't work in a heavily privatized nor regulated space at the moment of posting.
I did read your latest blog post looking for an answer, but it wasn't clear to me from that reading.
We have many packages internally so inevitably we need to build them which necessitates an entire build matrix on it's own. But once you have your own build matrix adding a new package is trivial and most build in under 10 minutes in github actions. One issue is we don't want to use Canonical's default gcc flags so like Gentoo I setup a make.conf that is specific to the suite (22.04, 24.04, etc). This has allowed me to target a higher tier instruction set which is it's own micro optimization. Folks that upgrade blindly from 22.04 to 24.04 will see their performance drop from gcc compile flags alone. Also we have people who have access to CVE fixes before they are generally available so that's also another important reason. Anyway we DO use upstream most of the time for setting up a project folder or normalizing it against a new suite but then building it is using different gcc flags and might be pulling in our version of curl or something.
The Calculator!
Calculators deskilled long division!
Thing is though: who gives a damn about long division? The real task is your taxes or making a massive BI report for your presentation tomorrow.
AI "deskilling" due to automating things makes old key skills become something you can just ask for and have done for you. Why bother staying skilled at those things when you can broaden? It clears the way for what used to take forever and be a specialist skill, to become something to steamroll through on your way to finishing a much greater outcome/task. It's the micro way that, in the macro, humanity becomes far more capable as a species. The gates are being flung wide open and the gatekeepers are being lapped and they have blogs!
It's not a tragedy, it's universal access to leverage.
> The Calculator!
Well, I'm sorry to hear that. I have a calculator and some pretty neat computing hardware. If I go into the attic and search long enough I can probably find my old sliderule. I can also do long division, as can all of my kids. I think it is an essential skill to understand all basic mathematical operations, and to be able to use them on small data sets. If I didn't know how to do division without a calculator I would consider that a failure of my basic education.
Not humanity, a small number of people in this planet, will benefit about this, because now they no longer have to depend on under payed workers to do the job, and will use AI to make the lives of all other human being poorer.
I think that we need to oppose this tech dictatorship, first thing stopping using AI and thus stop depending on an handful of companies that are able to provide it, that while they gain more and more power we all gets poorer.
What was wrong in using our brain, in the end? I still use my brain on my job and I always will, maybe one day they will say that there is no more need of computer experts because AI can do everything (still people bring me their laptop to reinstall Windows, something I've done for 20 years and that AI didn't change), I will just find another job but I will refuse to use this AI shit.
But as we're only imagining anyway, let's forget about the genie for a moment. If we did lose our sense of basic arithmetic all of a sudden what would life at its core even be like? If there were two apples on a table you wouldn't be able to comprehend what happens if you took one. Would we even be able to function like that?
Perhaps there is a limit, sure, but I suppose that's where you lead with a concrete example to explain where that line is.
There is no understanding with AI-generated applications, full stop.
This just sounds like convoluted way of saying you understanding much less than you would otherwise.
You can always know more but at a certain point, after you’ve built enough validation and safe guards, you find it’s not the best use of your time and the things you can make beyond your specialization is quite remarkable.
Also, I think the toothpaste isn’t going back in the tube with this
The internet is full of people doing the “AI = compiler” comparison and it’s just wrong.
Maybe on a really small scale they can resemble a compiler because the number of possible outcomes is so small that it’s nearly deterministic.
But at any larger scale AI will not be deterministic anymore and at that point it will be relevant what the resulting code looks like. Because it can and will be slightly or very much off from your instruction.
But the direction is reversed.
I figure out how to do something, wrap it in a bunch of functions, and then kinda just use it. If it just keeps doing what it should, I may kinda forget how it works. But I don't forget that I made it, and that I understood it, that will never not have happened. So in a way, I am able to "forget" some things because I still know I knew them. And I trust that programmers who write programs that I dream about writing also know what they're doing.
With LLM, you accumulate stuff with from knows where, ultimately fished from a vast ocean named question mark, by whatever turns of phrase you and involved agents happened to use, and what the temperature was at that moment. Which is of course where all the harnessing and looping and running code on the output comes in, but still, ultimately you have no idea. Which is fine for prototypes, but long term? Good luck.
I've thought a lot about this quote, and the best abstractions I use are absolutely this. I love array indexing as an abstraction, but in the back of my head I have pointer + index * size. I like having the kernel do memory mapping, but I know there's a buddy allocator and TLB cache there.
Full slop, you mean.
Companies loves to have "specialized experts" who skilled up in one afternoon in their field. It might be different in startup, but in consulting world in relatively large enterprises, it's just a fucking joke.
They wouldn't be able to draw it all onto a whiteboard from memory. But give them a half day and they will have mapped out the data flow
Understanding should come in different levels abstraction of what you are building,and managing.
There are different levels SREs, from beginner to say Fellow level.
Do you expect the Fellow level to understand nitty gritty details of each component his apps or services are interacting with and how each of them start? No. Especially when they have to deal with hundres or thousands of services each of them hosted by different teams, orgs, or different companies.
junior SREs, yeah probably got their hands dirty with those nitty gritty details you listed because they tend to deal with smaller or lower levels of components
So my point is, I agree withyour "Understanding" but not specific details as they differ per one's level of experience
Yall this was always just a money making scheme.
Never was it meant to carry humanity into the future.
SREs are about super-thinking, context switching, recognize patters and workflows, data movements, etc..
When they imagine someone using AI (except for themself), they imagine the least curious, least inquisitive vibe-coder. And they invent the idea that the only way to be someone who asks questions or remains curious is to have paid the dues that they see themself as having paid. Therefore the world must always need them.
Well, all you have to do now is imagine the vibe-coder who has the same curiosity and inquisitiveness that you had especially when you were starting out and weren't yet a professional programmer. The only difference is that now you don't have to bootstrap a whole software career to understand how things work; you can skip that and merely ask.
We're required to imagine them because they don't seem to exist in the real world. The vibe-coders seem almost entirely composed of the "I commit 20 kLOC a week without reading a single line, lol" or the "LLMs are like calculators/basic arithmetic skills are obsolete" lunatics.
IaC codebases, configs, utility scripts are relatively tiny, and fit into context windows whole. Models are eating them for breakfast.
They're really good at incident response. Sadly, much better than our dedicated colleagues from <location> ever will be. They'll chew through cloud service logs, application logs, application code, iac code and config, CI output, etc. to get whatever is needed, cause wide knowledge is no issue. Even disassembling stuff is no barrier, tracing things down from app level, to runtime level (e.g. JVM), to native calls.
They're really good at handling deployments, end to end infra setup and migrations, following checklists. Hell, they don't even really need checklists. Something that cannot be said about other dedicated colleagues from <location>, who cannot breathe without a checklist, will skip over steps, or otherwise very creatively misinterpret them or improvise something wicked.
They're really good at spinning up mad nice dashboards, and keeping business types happy with impressive looking write-ups they'll never read. It's really refreshing to not have to write those write-ups anymore (or the aforementioned checklists).
It's straight up both not looking good and looking better than ever. Having it be fully hands off remains an issue though.
They also have a propensity to try to rewrite gigantic amounts of code in one go making changes riskier and harder to scrutinize... without using more LLMs.