CoPilot is down too, so it's probably Microsoft engineers trying to vibecode their replacement Azure infrastructure before the current garbage fire finishes burning out.
another day, another anthropic outage. Yes, this is because of Microsoft, but they do seem to have a significantly higher amount service disruptions compared to the other LLM providers.
I think it should be fine for most people. Hopefully people don't rely on Claude for Ops, and if it's not Ops-related but just regular coding, I can't see why it shouldn't be able to wait a few days. I mean, even if Claude is down for a month, people should have enough competent developers to keep going, just at a slightly slower pace, right?
Was viewing it more through the lens of less technical companies, Excel, Powerpoint, the backbone of traditional office work has become very model dependent in some companies I have seen and those are the owners and fellow employees that tend to struggle more in getting that the local IT guy can't fix this themselves vs developers.
I really don’t feel comfortable merging a PR not reviewed by both of our AI reviewers. They catch so many bugs no human could. Seriously changed my view of code since we started using it in earnest. Bugs shall not pass. Without AI it feels like the Wild West again.
Ditto. I can do it, and thankfully I’ll say AI mostly catches weird edge cases that often aren’t worth the trouble, but having a 2nd set of eyes has been great for quality.
Apologize for strong words, and sorry, nothing personal, but I find this mindset bizarre and even comical. Will we reach a stage where surgeons on the operating table will second-guess their experience and education and refer to an AI assistant?
Imagine we had an AI system that could monitor 38 different patient signals (from blood pressure electrical activity in the brain) during a surgery and give the surgeon warnings whenever these drifted out of the expected range. It feels plausible to me that using such a system might improve surgery outcomes by, say, 20%.
I would prefer to have a surgeon who used the AI assistant rather than a surgeon who chose not to.
Potentially. Probably not in my lifetime, but it's not impossible given how most sci-fi works depict it. You can argue we had a bit of that paradigm shift already with specialized machinery. I'm sure people can still do hand surgeries, but existing tools and sensors increase the precision.
it is a bit sad comparing that to "I can't check for bugs without an AI", though.
This mindset of "I don't trust myself to do it" is a bit (sorry) pathetic. I think one should realize that mistakes happen and to a certain extent, though professionals have lower rates of failure, everything is a tradeoff.
Some people have asked me, aren't you afraid you'll make a mistake if you fix your own car? Well, aren't they afraid their mechanic will make a mistake?
I think it's good to realize that AI (famously) also makes mistakes, and not to infantilize yourself to the level of not feeling "comfortable" trusting your own thinking.
Though, I do see younger people today whipping out their phone calculator to "double check" something like a 75% increase on 120.
Are you comfortable writing a C application that handles internet traffic on the internet? Well that’s the feeling I have now, if humans were reliable programmers there would be no bugs. AI is already able to pick up most mistakes, by humans and by machines. And I can tell you we all make a lot of mistakes! Be humble and use the tools at your disposal to make that better. This is not infantile, this is what a professional programmer has always done: we had static analysis, type checking, lots of tests. Would you ship code without those? If not, add AI review to your tool belt and you will see it’s probably more valuable than your other tools combined.
> Are you comfortable writing a C application that handles internet traffic on the internet?
Yes, of course. Why wouldn't I be?
Everybody makes mistakes, and of course tools to prevent that are good. But what you're saying makes no sense, have you never written code without using type checking or static analysis?
Sometimes good enough is good enough, it's silly to think that a word without bugs is even possible. And besides, using LLMs to get there is a fallacy in itself, because it's a fundamentally non-deterministic system.
> nobody is wondering why kernel developers are still employed or what they bring to the table.
For now. Wait until we start hearing things like "Mythos reverse-engineered Apple ARM architecture and implemented the whole roadmap of the Asahi Project".
> Hopefully people don't rely on Claude for Ops, and if it's not Ops-related but just regular coding, I can't see why it shouldn't be able to wait a few days.
IME if anything, people are relying on LLMs for ops more than for development! Devops convinced management that ops isn't actually all that valuable and that devs can just do it. IaC is usually a mess of boring YAML based syntax that no one in their right mind want to learn anyway. Learning Ansible doesn't feel like a transferable skill unlike learning a programming language or paradigm, and in most teams the quality of the IaC stuff (abstraction, code reuse etc.) never mattered much.
As soon as LLMs became good, generating IaC stuff was like the first thing to go full vibe coding, at least for me. I'm proud of my craft, I don't produce bad code with my name on it, I very carefully review all the code changes etc. Devops YAML stuff? Couldn't care less.
Right, perhaps I used the wrong term. I'm referring to sysadmin stuff, like what MS got wrong in their latest outage. "DevOps" or YAML stuff is to actual Ops like Excel is to actual programming.
I agree that it's an opportunity to "do it the old way", as a way to preserve the skill. I actually would like to do that regularly, like half a day every week or something.
But I find it difficult to motivate it from the "productivity" angle. Like if I work for a day without an LLM today, it feels like I could go skiing instead, and the LLM will do what I would have done today in 10min tomorrow...
Seriously, if the LLM was down and a dev submitted 10 PRs, I think it would raise some good/bad curiosity from the team with where things stands today.
Which is the way it should be if we were to share the fruits of our labour more evenly in society. Of course in practice the lions share of the product gains go to the capital class and the while they suck the surplus employees and the remaining workers have to carry the burden of supporting the ones now sidelined.
Snow day / skiing? Since this has affected our customers, and they're asking us for help, it means I was up all last night.
Ironically, I also had a change window last night to make some configuration changes to improve site reliability. It was a little challenging to get customer approval for both the cost & downtime for this. Let's just say... this will be a much more fun conversation to have next week, and for the account manager to have. "Yes, we'e proactively improved your reliablity, and here's why you were very smart to approve it and pay for it."
No, and that's also not necessary. We are reliant on our systems. The correct response is to harden our systems, not to foster some illusion of independence.
(We can of course do that privately, for just the heck of it, but it's not the right response for anything at societal level)
What's wrong with failover? I don't use anything from Anthropic except their API. So when Claude is having issues, I switch to GLM 5.3 or something, doesn't take more than a few seconds.
I realise people on the monthly plans can't do it as easily. But I would say that as far as possible, having a plan B is the foundation of resilience.
Not hard on monthly plans either. I have ant, OpenAI and cursor (mostly for grok and gemiii). Taxonomy of model steps, each has a pre-tuned failover from a different plan and I just need to flip a switch when a lab goes down. Switch isn’t really required but occasional enough that I want eyes on and the system is only for my personal usage so outage until switch is acceptable…
I think that's the 6 '9's problem. If you have something with 99% uptime, how much more are you willing to pay for 99.9% uptime, then 99.99% uptime. It's risk vs cost. Many places in the US do the same thing with things like say snow removal. Instead of having expensive systems ready to go for the 1-2 days of year of snow, you just shut things down for a day and slow down a little.
That doesn't work for all systems, and not for all businesses, but I would say the mass majority can 'suffer' 24 hours without email or an LLM without everything going up in flames.
There is always something that you can do that isn’t gated by the api. It isn’t critical path for literally everything. Go organize your expense reports or something.
What do you think I use to organize my expense reports? If there’s something substantial you can do entirely without tokens that’s probably an opportunity for additional automation. But please have failover across multiple labs!
>> If there’s something substantial you can do entirely without tokens that’s probably an opportunity for additional automation.
Well, I just finished a meeting where we diagrammed out how change authority is to be structured as we split one group into two, which teams will retain what authorities. Zero claude involvement. Ten people in a room. Whiteboard and markers with a "scribe" taking notes. I am now editing old SOPs to reflect the decisions made in the meeting ... not something we would trust to claude or any other AI.
You’re not wrong, but some companies swing real hard at the biggest players assuming they won’t all be down at the same time. My work has hard dependencies on 365 and Cloud Entra and only allows Claude company wide. Today is going to be a snow day since we couldn’t get into anything yesterday but we could at least get projects done using connections Claude had into various platforms that cached the data in 365.
>harden our systems, not to foster some illusion of independence.
Let me know when the tech industry workers stop doing this and gains some solidarity, and I'll join you in that sentiment.
Until then, I pretty much need to bolster myself on the idea that one day I will need to switch frameworks, IDE's, and even OS's themselves. I'm the ones guiding the tools, not vice versa.
>We simply need to work harder for more of these 9s
The only 100% thing I will guarantee is that these tools will align to the incentives of their leadership over time, not the customers using it. add 10 9's if you want, that just guarantees that humanity is the weakest link.
>No, and that's also not necessary. We are reliant on our systems. The correct response is to harden our systems, not to foster some illusion of independence.
I'm not sure if this is your intent, but this paragraph reads like satire given current real world events.
The systems you mention in your second sentence are failing at the "right response" you mention in your third, which dictates that contrary to your first sentence alternative action is necessary.
You cannot harden systems you have no control over. Societal level assurances require law and goverment regulation and enforcement. It is juvenile to pretend otherwise. If we were talking about government services, then as a taxpayer you would have some avenue to enact change.
Your only option when dealing with private companies is to become anti-fragile at a personal level or at the level of your own company.
Reading the site guidelines, so I can be Claude-like and properly adjust my answer:
> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
You are also a system. And, if you, as a system, don't run with Claude, you're not sufficiently hardened. People have made all sorts of helpful suggestions, like de-risking with multiple providers or just getting yourself a better Mac to run a local model.
The idea that: "No, no, we must have Claude, b/c it's the same as having a pen and paper!" is woefully naive. It's hard to attack the supply chain of pen and paper if you have a few extra reams and a few extra sticks. It's pretty easy for our geopolitical enemies to attack computers located far away from you, especially those systems are designed like fragile Rube Goldberg machines; furthermore, if we build an entire society (a system) to be productive only when we have LLMs, our issue isn't "insufficient-hardening". It's that the design is wrong.
Is there not a single SaaS, cloud company or similar that your org depends on? No AWS, no Cloudflare, all internal mail servers, no Docusign, no outside inference, nothing of that sort?
I can't imagine a construction worker saying, power is out, gotta find a way to be productive, I guess instead of the paint sprayer, nail gun and circular saw, we'll just find other ways to be productive with our hands. The same might apply to law, medicine, etc. I think we're almost there with software.
I understand what you’re trying yo day, but my family ran a construction company for generations (until my father retired) and they had contingencies for every adverse scenario to stay on schedule: Generators, battery-operated tools, hand-operated tools, heaters for pretty much anything you can think of for extreme cold, wind coverings, etc. I don’t remember them ever stopping work except for a trade union strike.
I’m pretty sure doctors don’t abandon their patients during a blackout either, or even a war for that matter.
Mature professions have procedures and backup procedures for pretty much every non-catastrophic situation.
>Yes, I believe US hospitals are legally required to maintain and test backup generators in case power goes out.
And even that only does so much, it was 45 years ago but my Grandmother had a stroke on the operating table because the power went out and generators didn't kick on until someone could run and manually start them because the autostart failed.
> Microsoft being down is like having a contingency for a tornado.
I'm probably misreading your point, so, apologies, but I'd still just like to say:
Microsoft is a profit-driven, human-run corporation that mostly offers business infrastructure services. I think contingency planning for Microsoft service failures is almost entirely unlike contingency planning for adverse weather events?
I may be misunderstanding what you wrote, but I think that says more about your imagination than about software. Yes, medics do improvise a lot to keep patients alive when the power is out, or when essential tools or medicines are not available. "Well, the power failed, so did the sustainable backup, let's everyone go home, today's ICU is a writeoff".
There's also the option where it doesn't affect you at all, because you do not rely on LLMs—or even just do not rely specifically on Claude—at all, and can easily do your job without ever noticing that it's down.
I get what you're saying and I would wholeheartedly agree 5-7 years ago but after hearing Our Lord and Savior Claude Code as the solution to all our technical problems with ever increasing fervor and intensity for the last two years, I'm gonna take the cynical route, maybe fence on office chairs, not because my code is compiling, nor my prompt is running, but Claude is down management, what do you want me to do now, actually think?
From what I hear, the project managers are having a hard time sending emails anyway. We'll call it quits, winkwink.
Sure, like if your company’s internet is down, you can start sending invoices to customers by paper mail. Just to prove that you’re not reliant on a technology that’s made work more efficient.
Why bother? You do a little bit of work, fine, but when Claude returns all that work will be done in an instant anyway, so unless you really need results right now, just wait.
Friendly reminder that Microsoft has been engineering Office 365 for a while so it simply cannot interoperate with a self-hosted e-mail server or a small hosting provider's e-mail platform.
Likewise, Anthropic / Dario has been busy proclaiming why nobody except themselves should be allowed to develop frontier AI, that we need "kill switches" in all AI models, and that we need a pause / slowdown on AI development because of the threat of open source models.
Essentially, they want complete lock in to their platform, so yes, it's very appropriate to ask questions when they can't meet basic 3 9's of reliability.
I was unable to do things beyond basic banking with my bank yesterday (and today) due to this outage. It just involves making changes to a merchant processing account for a nonprofit, but the point is that normal business activity grinds to a halt when it relies on an unreliable provider like this.
I have fond memories of the “we’re a tech company, why can’t we get a video conferencing call going” and gently reminding them that reliable, synchronous, multiparty video conferencing was a lot harder than anything this company was trying to do.
Yep. This is what happens. Once you give up control it's mostly gone. The interesting thing is, if you maintain control, you can always choose to use llms here and there. But expecting humans to dive into 10000 file 40million line code bases branded todo apps isn't going to happen.
10k files, 40m LOC is rookie numbers for a todo app.
I recently asked Claude to edit a few targets of a Makefile. It decided: "OH BOY HERE WE GO LETS FIRE UP ALL THE B200S BABY" and went way deep adding tests...I stopped it when I saw something like: "Adding guards to prevent homebrew injection threat...". It had written an entire fucking build subsystem, probably larger than make. For a 5-line change.
If humans can't maintain a 10,000 file 40 million LoC code base, then three things are true:
1. We haven't learned anything from Google, who maintained Chromium and Android before LLMs
2. Your LLM is not generating code as maintainable as Chromium and Android (which are gigantic open-source projects that probably 99% of your devs have used)
3. Your company has no faith or interest in teaching devs how to maintain code nor how to build maintainable code
And that's pretty pathetic. It's solvable, but it's pathetic.
I will be amused if those 3 points lead to this conclusion:
4. If you aren't asking your LLMs to write maintainable code, and they aren't smart enough to do it by default, you are pretty quickly gonna hit a brick wall where your LLMs are also just slopping mud onto the big ball of mud. Then nothing will save you, if your leads haven't heard of "quality".
What does “maintainable” mean in a world where people aren’t even reading the code that gets merged?
There are companies telling their developers that they’ll never write code again. They manage agents now. It’s like asking PMs how maintainable the codebase is.
You doing your work manually is like going back to the 90s or further even. It feels like dropping into 6502 manual assembly mode. It's fun but it's not "work" anymore. I honestly think without AI my company is better off if we programmers sat around and talked like humans about business. Coding is for the clankers now.
If all the professionals leave coding to clankers, and discussions about coding to people who code, I see nothing I don't love about that. Just like the web was more interesting before there was money in it, coding is more interesting without people who can't shut up about their work.
That's exactly my point, programming isn't your IT department, "cool things that can be done with computers" isn't even on speaking terms with it, and the web is not a sector of marketing, either. Bores turn everything they touch boring, and now they're telling us we'll be left behind. And I'm like, YES, finally, can't wait, what can I do to accelerate it?
No not really. Most the companies I know that haven't gone full vibes are doing great.
People can and do still write code. Especially in regulated industries. The internet is just full con artists and hype followers not thinking about the future at all.
but I can say that, anyone experienced will notice how you really can't just "let them loose" on the codebase because they'll lose context, not understand the architecture, not have good vision/ideas about the codebase etc.
so I'm bottlenecked by following the code generation and then having to review and respond
and there's so many times where I go "wait but can't we just do x?" and then the model goes, "oh let me see.. yes we can"
if you were to just use agents w/o looking at code you'd have complete spaghetti in no time
maybe the future is that you embrace the spaghetti and just treat code as a black box where all you care about is the input/output
but my intuition tells me that it's not really viable yet. you'll take many steps forward and many steps back.
Hahahahaha it’s like boom and bust, I’m hitting that limit right now with something I’m working on. You push velocity too hard then hit a wall of bugs, then the agent gets a stern talking to and you go back and refactor :)
I wrote code (I am old) before stack overflow. Google existed, and it was even still in the honeymoon phase where "don't be evil" was almost plausible. But search engines were not yet a good tool for solving esoteric programming questions.
But honestly, the practice of programming then was not dramatically different from programming in, say, the mid-2010s. Obviously we leaned more heavily on the official documentation, and tbf, a lot of the docs were provided in PDFs, which is probably the single worst way to provide documentation.
Sometimes, the docs were provided as collections of papers bound together by a "spine". This is actually a wonderful documentation format because our brains can associate physical locations in the physical media with specific pieces of information.
(this is the oldest I have ever been in my entire life)
I suppose I can't bring myself to do the drudgery of coding. I enjoy some of the problem solving but not the workflow frictions, spending 20 minutes due to a stupid typos, syntax fuckery, API lookups, etc.
It's asking tractors why the ground is full of dirt, despite having tractors to move the dirt, and tractors to confirm there is absolutely no dirt left before all the tractors can rest.
I've started noticing at $WORK that a lot of systems, procedures, etc. that fall under SDLC or devX are being made ever more onerous or complex because "LLMs can do it."
Some examples include requiring that someone addresses every review comment separately, or that a PR description be formatted a specific way, or that support requests have existing JIRA numbers etc. All of these make the developer experience much worse and would never fly if generating text (that no one reads) wasn't cheap now. A lot of this is cargo culting to make vibe coding feel more serious and entreprise.
The teams that have gone full specs driven development are not AFAICT more productive but good luck getting any change in without 10$ of API usage, even for a one liner. In some cases development is now slower with LLM than it was before!
I've seen every one of the things you listed, sometimes all at the same company. None of them add much in the way of effort/time to do the work.
- JIRA ticket - audit trail
- PR format - easier for people to scan, and to make sure things aren't forgotten (you can even have the template inserted when it's created)
- address review comments separately - not sure what to say on this one because I don't understand how _else_ you would address them - I've always addressed each one individually
The team I'm on prides itself in providing great service to our clients (other devs in the company). This includes responding to support requests even if the client isn't entirely sure how to ask or who to ask.
> not sure what to say on this one because I don't understand how _else_ you would address them - I've always addressed each one individually
What I mean is that it used to be enough for you to address PR tasks by changing the code and pushing the updates and marking the tasks as "resolved". Now in some projects at work you also need to reply to the comment explaining how the change you made addressed the task. The idea obviously is to stop LLM from just ticking tasks without actually doing the work, but it's onerous and based on distrust. Not trusting LLMs is warranted but those rules get applied to everyone.
> PR format - easier for people to scan, and to make sure things aren't forgotten (you can even have the template inserted when it's created)
The team I just left had a PR template that was probably 60 bullet points (never bothered to count, just grumbled). Certainly was not easy either to scan or to update manually.
Really makes me wonder how much some programmers really understood programming 5 years ago if this is the reaction to a very recent knowledge center being down.
I suppose the equivalent would be Google search being down 10 years ago and being forced to read man pages again. Annoying but not literally "snow day" levels of lost productivity.
Software documentation was a little different when it went through a publication process as a part of a larger waterfall. I was going to say better but I'm not sure about that part, but it was certainly verbose.
I did an experiment with some co-workers around a decade ago, and one of the results was most of that group learning for the first time it was possible to train yourself to read code more easily. I was self-taught and had been doing it for so long, combined with some very specific pressures as a teacher's assistant for years, that I learned to read code in programming languages I understand about as easily as I can read English - something they had never even considered possible before.
I never did find out how they see code, but I have to imagine the right verb for them was "deciphering" instead of "reading"...
To be fair most code in most shops I worked for since 2011 was crap with some notable exceptions. People came and went, teams forgot why code was written or what it did, or randomly rewrote stuff that just worked to replace it with crap that didn't, over-enthusiastic rookies wrote over-complicated code to impress their seniors, early code debt was calcified into impossible monstrosities that couldn't be refactored without bringing entire systems down, etc.
But I'm sure in a few years AI will be so good we won't have to worry about any of that anymore. After all, it's not like those systems learn how to code by reading our code, is it?
Just as you described. Snowday for some...
I was caught in between snowday and stress today with these things.
Got too used for claude to just do most things.
But at the same time was surprised I could still google and resolve things myself when I started thinking again lmao.
The most likely scenario, is their team of high achievers, deployed a massive pile of stinking technical debt, against the advice of the experienced folks who are constantly overridden.
On a normal environment the old Graybeards would fix it, after grumping on being taken away from playing with their latest Emacs config.
But this is the Valley, and we dont do experience only VibeOps. And half reading this post, are wondering if Emacs is a new frontend framework they never heard about....
As people used to say "No one was ever fired for choosing IBM."
If people lose enough money then they will come up with contingency plans etc. that may even work in the next outage! Absolutely no one will be held responsible though. The kind of management folks that made those decisions will just turn around and work on those contingency plans, and everyone (including themselves) will believe that they had been doing so all along.
At least that's my experience in big corporations.
The whole point of dumping all your infrastructure onto these companies in the first place is CYA. “Microsoft is down? Well that’s basically force majeure as far as I’m concerned.”
You get to pass the buck and management can’t be mad at you (sort of)
If you are that reliant on LLMs for your workflow, then how do you not have any redundancy in place? Assuming you've have any kind of basic abstraction in place, swapping in API keys for a different provider should be a simple config change...
We are now in year 3-ish of LLMs going mainstream. Do people immediately forgot writing code manually the moment Claude was out? I understand the LLM acts as a multiplier but does coding immediately stop when Claude is down? My line of business is not in software development so this is a truly genuine question.
What scenarios are there where Execs need to have a "Claude is down" meeting? Surely we aren't far enough along in the singularity where this needs to happen anywhere yet.
This why every corp should have an always on fallback. Self hosted, single dgx spark qwen flash next level.
It's like the 5G backup line on optic fiber router, because when it's down, you're out.
313 comments
[ 3.8 ms ] story [ 35.5 ms ] threadOAuth request failed Temporarily unable to authenticate. Please retry.
Can this be fall-out from the Microsoft outage?
94 points by cloudbreaker 22 hours from now | flag | hide | 64 comments
CoPilot is down too, so it's probably Microsoft engineers trying to vibecode their replacement Azure infrastructure before the current garbage fire finishes burning out.
https://status.cloud.microsoft/m365/
Imagine we had an AI system that could monitor 38 different patient signals (from blood pressure electrical activity in the brain) during a surgery and give the surgeon warnings whenever these drifted out of the expected range. It feels plausible to me that using such a system might improve surgery outcomes by, say, 20%.
I would prefer to have a surgeon who used the AI assistant rather than a surgeon who chose not to.
Claude is going to write a banger medical drama about this
Amusingly, I did the same and it told me there's no point in taking more than a single dose in my uncomplicated case (prophylactically).
it is a bit sad comparing that to "I can't check for bugs without an AI", though.
Some people have asked me, aren't you afraid you'll make a mistake if you fix your own car? Well, aren't they afraid their mechanic will make a mistake?
I think it's good to realize that AI (famously) also makes mistakes, and not to infantilize yourself to the level of not feeling "comfortable" trusting your own thinking.
Though, I do see younger people today whipping out their phone calculator to "double check" something like a 75% increase on 120.
Yes, of course. Why wouldn't I be?
Everybody makes mistakes, and of course tools to prevent that are good. But what you're saying makes no sense, have you never written code without using type checking or static analysis?
Sometimes good enough is good enough, it's silly to think that a word without bugs is even possible. And besides, using LLMs to get there is a fallacy in itself, because it's a fundamentally non-deterministic system.
Why are you still employed? What do you bring to the table if you’ve off loaded so much to AI that you can no longer review a pull request…?
What more needs to happen before you accept that human review does not catch all the bugs that AI does?
For now. Wait until we start hearing things like "Mythos reverse-engineered Apple ARM architecture and implemented the whole roadmap of the Asahi Project".
IME if anything, people are relying on LLMs for ops more than for development! Devops convinced management that ops isn't actually all that valuable and that devs can just do it. IaC is usually a mess of boring YAML based syntax that no one in their right mind want to learn anyway. Learning Ansible doesn't feel like a transferable skill unlike learning a programming language or paradigm, and in most teams the quality of the IaC stuff (abstraction, code reuse etc.) never mattered much.
As soon as LLMs became good, generating IaC stuff was like the first thing to go full vibe coding, at least for me. I'm proud of my craft, I don't produce bad code with my name on it, I very carefully review all the code changes etc. Devops YAML stuff? Couldn't care less.
My soul just dry-heaved from laughter. Nice.
O364...
But I find it difficult to motivate it from the "productivity" angle. Like if I work for a day without an LLM today, it feels like I could go skiing instead, and the LLM will do what I would have done today in 10min tomorrow...
If I wait until tomorrow, though, my company won't see the difference.
You sound self-employed.
Ironically, I also had a change window last night to make some configuration changes to improve site reliability. It was a little challenging to get customer approval for both the cost & downtime for this. Let's just say... this will be a much more fun conversation to have next week, and for the account manager to have. "Yes, we'e proactively improved your reliablity, and here's why you were very smart to approve it and pay for it."
(We can of course do that privately, for just the heck of it, but it's not the right response for anything at societal level)
I realise people on the monthly plans can't do it as easily. But I would say that as far as possible, having a plan B is the foundation of resilience.
Another reason to have good Evals…
That doesn't work for all systems, and not for all businesses, but I would say the mass majority can 'suffer' 24 hours without email or an LLM without everything going up in flames.
That's not always easy to do, but with something like inference providers it is, so why not kill two birds with one stone?
Well, I just finished a meeting where we diagrammed out how change authority is to be structured as we split one group into two, which teams will retain what authorities. Zero claude involvement. Ten people in a room. Whiteboard and markers with a "scribe" taking notes. I am now editing old SOPs to reflect the decisions made in the meeting ... not something we would trust to claude or any other AI.
Let me know when the tech industry workers stop doing this and gains some solidarity, and I'll join you in that sentiment.
Until then, I pretty much need to bolster myself on the idea that one day I will need to switch frameworks, IDE's, and even OS's themselves. I'm the ones guiding the tools, not vice versa.
>We simply need to work harder for more of these 9s
The only 100% thing I will guarantee is that these tools will align to the incentives of their leadership over time, not the customers using it. add 10 9's if you want, that just guarantees that humanity is the weakest link.
I'm not sure if this is your intent, but this paragraph reads like satire given current real world events.
The systems you mention in your second sentence are failing at the "right response" you mention in your third, which dictates that contrary to your first sentence alternative action is necessary.
You cannot harden systems you have no control over. Societal level assurances require law and goverment regulation and enforcement. It is juvenile to pretend otherwise. If we were talking about government services, then as a taxpayer you would have some avenue to enact change.
Your only option when dealing with private companies is to become anti-fragile at a personal level or at the level of your own company.
> Comments should get more thoughtful and substantive, not less, as a topic gets more divisive.
You are also a system. And, if you, as a system, don't run with Claude, you're not sufficiently hardened. People have made all sorts of helpful suggestions, like de-risking with multiple providers or just getting yourself a better Mac to run a local model.
The idea that: "No, no, we must have Claude, b/c it's the same as having a pen and paper!" is woefully naive. It's hard to attack the supply chain of pen and paper if you have a few extra reams and a few extra sticks. It's pretty easy for our geopolitical enemies to attack computers located far away from you, especially those systems are designed like fragile Rube Goldberg machines; furthermore, if we build an entire society (a system) to be productive only when we have LLMs, our issue isn't "insufficient-hardening". It's that the design is wrong.
Or maybe more prescient; machine operators going home on a cold day because they can’t get their loader started.
I’m pretty sure doctors don’t abandon their patients during a blackout either, or even a war for that matter.
Mature professions have procedures and backup procedures for pretty much every non-catastrophic situation.
Of course proof by analogy is silly anyways, my email going down is not nearly as bad as a patient's medical equipment going down.
And even that only does so much, it was 45 years ago but my Grandmother had a stroke on the operating table because the power went out and generators didn't kick on until someone could run and manually start them because the autostart failed.
The equivalent for a dev is laptop, another AI, mobile hotspot and work at a cafe/library.
Sounds like a normal thing to do if you live in Kansas. And perhaps we are all about to start living in the software equivalent of Kansas?
I'm probably misreading your point, so, apologies, but I'd still just like to say:
Microsoft is a profit-driven, human-run corporation that mostly offers business infrastructure services. I think contingency planning for Microsoft service failures is almost entirely unlike contingency planning for adverse weather events?
From what I hear, the project managers are having a hard time sending emails anyway. We'll call it quits, winkwink.
Yesterday was day 1 of using claude code for me, so i almost made it into the "didnt notice" group
Likewise, Anthropic / Dario has been busy proclaiming why nobody except themselves should be allowed to develop frontier AI, that we need "kill switches" in all AI models, and that we need a pause / slowdown on AI development because of the threat of open source models.
Essentially, they want complete lock in to their platform, so yes, it's very appropriate to ask questions when they can't meet basic 3 9's of reliability.
I was unable to do things beyond basic banking with my bank yesterday (and today) due to this outage. It just involves making changes to a merchant processing account for a nonprofit, but the point is that normal business activity grinds to a halt when it relies on an unreliable provider like this.
yeah, couldn't be me
That's a scary thought.
I recently asked Claude to edit a few targets of a Makefile. It decided: "OH BOY HERE WE GO LETS FIRE UP ALL THE B200S BABY" and went way deep adding tests...I stopped it when I saw something like: "Adding guards to prevent homebrew injection threat...". It had written an entire fucking build subsystem, probably larger than make. For a 5-line change.
1. We haven't learned anything from Google, who maintained Chromium and Android before LLMs
2. Your LLM is not generating code as maintainable as Chromium and Android (which are gigantic open-source projects that probably 99% of your devs have used)
3. Your company has no faith or interest in teaching devs how to maintain code nor how to build maintainable code
And that's pretty pathetic. It's solvable, but it's pathetic.
I will be amused if those 3 points lead to this conclusion:
4. If you aren't asking your LLMs to write maintainable code, and they aren't smart enough to do it by default, you are pretty quickly gonna hit a brick wall where your LLMs are also just slopping mud onto the big ball of mud. Then nothing will save you, if your leads haven't heard of "quality".
(Yes, this is projection. Yes, I'm late for work)
There are companies telling their developers that they’ll never write code again. They manage agents now. It’s like asking PMs how maintainable the codebase is.
Most people people here do not code. They wish they do, but they don't.
People can and do still write code. Especially in regulated industries. The internet is just full con artists and hype followers not thinking about the future at all.
but I can say that, anyone experienced will notice how you really can't just "let them loose" on the codebase because they'll lose context, not understand the architecture, not have good vision/ideas about the codebase etc.
so I'm bottlenecked by following the code generation and then having to review and respond
and there's so many times where I go "wait but can't we just do x?" and then the model goes, "oh let me see.. yes we can"
if you were to just use agents w/o looking at code you'd have complete spaghetti in no time
maybe the future is that you embrace the spaghetti and just treat code as a black box where all you care about is the input/output
but my intuition tells me that it's not really viable yet. you'll take many steps forward and many steps back.
Is going back to just a couple of months ago. Jesus.
But honestly, the practice of programming then was not dramatically different from programming in, say, the mid-2010s. Obviously we leaned more heavily on the official documentation, and tbf, a lot of the docs were provided in PDFs, which is probably the single worst way to provide documentation.
Sometimes, the docs were provided as collections of papers bound together by a "spine". This is actually a wonderful documentation format because our brains can associate physical locations in the physical media with specific pieces of information.
(this is the oldest I have ever been in my entire life)
Tractor breaks, you complain about shoveling again and someone says you’re in the wrong job.
It's asking tractors why the ground is full of dirt, despite having tractors to move the dirt, and tractors to confirm there is absolutely no dirt left before all the tractors can rest.
Some examples include requiring that someone addresses every review comment separately, or that a PR description be formatted a specific way, or that support requests have existing JIRA numbers etc. All of these make the developer experience much worse and would never fly if generating text (that no one reads) wasn't cheap now. A lot of this is cargo culting to make vibe coding feel more serious and entreprise.
The teams that have gone full specs driven development are not AFAICT more productive but good luck getting any change in without 10$ of API usage, even for a one liner. In some cases development is now slower with LLM than it was before!
- JIRA ticket - audit trail
- PR format - easier for people to scan, and to make sure things aren't forgotten (you can even have the template inserted when it's created)
- address review comments separately - not sure what to say on this one because I don't understand how _else_ you would address them - I've always addressed each one individually
The team I'm on prides itself in providing great service to our clients (other devs in the company). This includes responding to support requests even if the client isn't entirely sure how to ask or who to ask.
> not sure what to say on this one because I don't understand how _else_ you would address them - I've always addressed each one individually
What I mean is that it used to be enough for you to address PR tasks by changing the code and pushing the updates and marking the tasks as "resolved". Now in some projects at work you also need to reply to the comment explaining how the change you made addressed the task. The idea obviously is to stop LLM from just ticking tasks without actually doing the work, but it's onerous and based on distrust. Not trusting LLMs is warranted but those rules get applied to everyone.
The team I just left had a PR template that was probably 60 bullet points (never bothered to count, just grumbled). Certainly was not easy either to scan or to update manually.
Those examples sound like responses to being swamped with LLM PRs:
* "every review comment separately" [so the bot doesn't get a chance to miss it]
* "a PR description be formatted a specific way" [so the bot has guardrails]
* "support requests have existing JIRA numbers etc" [so we can have a chance of understanding the context]
> The teams that have gone full specs driven development are not AFAICT more productive
I suppose time will tell. It's difficult to generalise my experiences beyond my immediate team/company/industry.
> In some cases development is now slower with LLM than it was before!
Yes, pre-LLM I wouldn't sign up for formatted PR descriptions!
I suppose the equivalent would be Google search being down 10 years ago and being forced to read man pages again. Annoying but not literally "snow day" levels of lost productivity.
I never did find out how they see code, but I have to imagine the right verb for them was "deciphering" instead of "reading"...
But I'm sure in a few years AI will be so good we won't have to worry about any of that anymore. After all, it's not like those systems learn how to code by reading our code, is it?
I’ve been in this situation before, an external API is basically down and my managers solution is to blame me.
Didn’t last there too long.
But at the same time was surprised I could still google and resolve things myself when I started thinking again lmao.
On a normal environment the old Graybeards would fix it, after grumping on being taken away from playing with their latest Emacs config.
But this is the Valley, and we dont do experience only VibeOps. And half reading this post, are wondering if Emacs is a new frontend framework they never heard about....
If people lose enough money then they will come up with contingency plans etc. that may even work in the next outage! Absolutely no one will be held responsible though. The kind of management folks that made those decisions will just turn around and work on those contingency plans, and everyone (including themselves) will believe that they had been doing so all along.
At least that's my experience in big corporations.
You get to pass the buck and management can’t be mad at you (sort of)
well, it works on my machine.