250 comments

[ 3.9 ms ] story [ 125 ms ] thread
(comment deleted)
How many Microsoft employees are working on this? It's concerning that one of the biggest, most valuable companies on the planet, that runs services internationally, can have one of its primary services have gone down for this long.

6 hours ago they seemed to not exactly understand what the problem was, and that was 6 hours into the outage.

This is concerning.

I don't understand why they haven't asked copilot to fix it yet, make no mistakes.
Don’t be silly, they tried that but copilot is/was down!

That’s why it’s taken over 12 hours now, no one’s left that knows without copilot’s help.

Let me put this way, the Visual C++ standard library has officially one FTE working on it, everything else are open source contributions.
Maybe acting like jerk-offs got them to four trillion but there are consequences to burning all your social capital. A large camp of software developers have genuinely hostile feelings towards Microsoft, I would bet that this affects the quality of talent they can hire and these are downstream affects.

Lately it seems like more non-techies are souring on the company.

Jerk-offs can skim the profit of going big yesterday while the consequences affect everyone, but only in the future of yesterday. Who does such things, one might ask? Jerk-offs, off course. And PE too (likely not a disjoint set).
Was there ever a time when this was not true? M$ was always among top hated companies since 90s, arrogance, fuck-U to users all around, absolutely ignoring any standards larger than them, and famous 'Embrace, extend, and extinguish' which was pathetic to watch for any engineer worth 2 cents.

Top notch user experience peak was Windows XP and every subsequent release was massively larger, clunkier and slower while requiring way more resources, was spying on the users more and more. Office has similar story, peak of usability was maybe 2007 with ribbon introduction (which still a lot of power users hated with passion since it was far from clean design, more seemingly designed by committee of marketing guys).

>A large camp of software developers have genuinely hostile feelings towards Microsoft

which, besides hatred, manifests in malice, like Nightmare Eclipse. I wouldn't be surprised if this outage was an engineer leaving a parting gift, using a bug they probably knew of for years but management didn't allow them to fix.

(comment deleted)
Agree. I was having suspicions around that
Wow Nightmare Eclipse is fascinating. It sounds like he is operating in a gray area and could be charged if Germany really wanted to?
I've been hearing that since 2002, at least.

Enterprises have been failed time and time again by poor MS products, but techies will steadfastly refuse to move off it, so the business has no option.

> Enterprises have been failed time and time again by poor MS products, but techies will steadfastly refuse to move off it, so the business has no option.

My suspicion is that lots of the companies are so attached to all the MS stuff (and that's what their staff know), that it would be an incredibly large lift for them to move off it.

Like, I'm at a very windows/azure focused shop now (for the first time in my career), and literally everything is already on MS, or moving towards MS.

There are some good reasons for that, and some bad reasons, but I suspect that most of the sales process happens at the executive level, rather than the technical level (and Excel runs the world, so why not let MS do everything else?).

Tellingly, I will not go to MS for UI stuff. When I have to anyway, I use their decades-old APIs. It is a canary with regards to microsoft, that they have no known champion for their UI stuff, no trusted face for their UI stuff; the company all just appears to coast along on "I assume someone is handling it?" The main microsoft employee I trust is Raymond Chen, they have no current generations of Raymond Chen / that we know of.
>the Visual C++ standard library has officially one FTE working on it

One guy but it's Stephan T. Lavavej the guy who the "STL" was literally named after.

I think STL was developed before Stephan T. Lavavej was born.
This is what I don't get about big companies and their software; you'd think every application, every moving part would have a (small?) team of core maintainers, people that get paid well to know everything there is to know about something, ranging from (in microsoft's case) Paint to a particular configuration panel.

I get that this doesn't scale though - a lot of features are just "done" at some point and keeping one or more people whose sole job is to maintain this may lead to them bolting on features just to stay relevant / stop being bored. But big tech can easily afford to keep people like that on.

(I mean they keep (kept?) tons of middle management on, who knows what they even do? Not emailing right now that's for sure am I right?)

Companies just don't want to compensate people to hang around and maintain a system, they want a consistent output of widgets that can be measured. Somehow our work culture is still stuck in factory worker mentality despite being well into the information revolution now.
> every moving part would have a (small?) team of core maintainers, people that get paid well to know everything there is to know about something,

I work at a company with a few thousand employees in the Netherlands, and have been moved around a few teams in the last 7 years (not by choice, the team is nuked and I'm shuffled off elsewhere).

Technically, the product is supposed to have a "handover" to another team who can do the maintenance. What happens in practice is the person who worked on it gets saddled with maintaining it forever along with taking up the responsibilities of the newer work that comes along their way.

I've accumulated a string of such projects, including one that's used extensively by a few large companies, and the only person maintaining it is drumroll ... me, in whatever time I can spare for it.

Maybe Nadella is still trying to figure out which of the 8000+ sacked employees knew how the system works /s
he was such a nice and effective CEO the first few years. he's really changed.
Executive compensation corrupts, chief executive compensation corrupts absolutely.
Even with some of the Microsoft <3 Open Source, I do miss the Steve Balmer era in regards to Windows development experience.
Not really. He killed the QA department immediately and consciously put MS onto the "slide into mediocrity" path.
Google killed its test engineers (or, rather, forcibly merged them into SWE) at around the same time without measurably denting uptime.
Can't have downtime if you kill everything you create before it can fail :)
Was he? I struggle to see it even in hindsight, and definitely his moves were bunch of FU to common users. Maybe in some specific area you care about, but not their core products like windows and office
yeah very effective, he fired 90% of our Windows Store team (reviewing the apps) in the first year to replace people with automation, I survived up to the last round of firing after which they kept ~20 employees out of original ~200
The ones who actually knew how the system works were too expensive to retain, so now it's just some interns typing frantically into Claude Code.
Company policy is to use copilot but copilot is down so they are just hitting F5 and hoping
Haha this is ridiculous and yet there is some truth in it.
It may not be true now but 10 years down the line? This is disaster in the making yet while most people recognize that (when filtering out most fanatical pro-ai folks claiming everywhere how they get 10x or 100x work done) nobody is able to do anything. Sort of legacy technical debt being delivered but of different kind than usual, that train that will hit you since you stand on the rail line yet its moving 5kmh and you see it from 200m, but unable or unwilling to move away (Austin powers scene with steamroller does it better than me [1]).

Imagine if say BMW decided they don't want to manufacture their motors, suspensions, drivetrain etc. Outsourcing 100% of that knowledge to external suppliers, for all models, from now on forever. How do you think market would reacted to such move ie by valuation of stocks?

[1] https://www.youtube.com/watch?v=y_PrZ-J7D3k

Given how unreliable and poorly engineered modern BMWs are the market might react positively
When you outsource everything, you’re not relevant anymore. I agree on that, anyone else can do it just as good as you.

But the biggest fear is how is ai going to improve then ? We’ve seen it can’t feed itself on data it produced, the model quality decrease. But if there is no more human to produce ever better quality code because they rely on llm to generate code, we’re going to reach a wall.

This should be the top comment, my eyes are watering to this one
"Too expensive" as measured by bean counters that don't understand the actual risk profile - or do understand it but have such a short window of strategic concern it leads to this. It would be an interesting analysis to stack up costs to Msoft vs cost of labor cut as this outage evolves.
And which made a load of staff redundant in the last month.
/s Their internal AI is full steam on it. The issue is just that their internal AI infrastructure is down too, so AI just need to fix itself first. Then it will focus on fixing the issue. Oh, wait...
This is HN. We should be asking why frontier models couldn't still dolve this! How many parallel agents does MS have working on this?
the vibe coding group in charge of this product has decided to delete the production database and backups
In our field, once you understand what the problem is, the problem is usually solved. Unless it's broken hardware, obviously
But Copilot is down too, so maybe they're screwed? :)
Not so long ago, I was assigned to a project that had to be built on Azure due to client requirements. Because we were underhanded and it was an MVP, we'd elected to use PaaS as much as possible: Postgres Flexible Server, Static Web Apps, Easy Auth, etc. On paper, these all looked fine. They had lots of documentation, example projects, GitHub repos, blogs, roadmaps, etc. Then we started building and running into issues, and we realised a bunch of these offerings were basically unmaintained. There were features that had been stuck in "preview" status for literally years. The blogs and roadmaps were no longer updated, and issues were piling up in GitHub.

I started digging into the historical blog posts, GitHub comments, etc. and I realised everybody at MS who'd been working on them, SWEs, PMs, architects, etc. had either left the company, or moved to other teams. And the one constant about the internal moves was that people were now working in some "AI" related project or product. Literally, I didn't find a single person who wasn't now working on AI.

Clearly, there has been a huge internal push to get into AI. Resources have been reallocated, and anything non-AI related has been starved. And since working on an AI project is probably considered the only route to advancing your career, anybody with any ambition has moved to a team doing something AI-related. The result is that huge numbers of products and teams doing vital but non-AI related work have been hollowed out, and now we're seeing the consequences.

That's related to something I realized which I think a lot of people might find amazing. They think of companies like Apple and Microsoft as trillion-dollar, globe-spanning companies, so it must be massive teams that work together on everything.

But no, read Hacker News or other niche places enough and eventually you realize a lot of the things we all know or use was a solo effort or a very small number of people, sometimes not even an assigned project but something they did on the side. Cue the xkcd about the lone open source project maintainer holding up the entire modern web but apply it to the tech giants and there you go, it happens there too.

This state of Azure saddens me. I learnt everything cloud with it. I've built many things (big and small) in it. It was the one part of Microsoft I thought they'd not neglect.

Lots of its features - App Services, DNS, managed DB, Front Door, Blobs, Durable Functions, Service Bus, App Insights - are very powerful. The frontend portal is mostly fine for management. The Az CLI makes it easy to provision things quickly. Coupled with .NET Core on MacOS, I've not needed to touch Windows locally for years.

I could learn AWS or GCP, but that's just going to cost more time when I'd rather use what I already know to build more things. With Azure, I'd hate to see the rot set in further.

Mythical man month, maybe throwing everybody in the company at it isn't the right thing to do. But they must be in a panic, and their own communications internally may well be hampered, could be MS everything is out for MS as well as everybody else.
I've seen the response to an emergency be "get every dev in a big room with some non-technical middle managers to direct them, and have them all talk at once". My jaw was on the floor.

"When in danger, when in doubt, run in circles, scream and shout".

Someone's going to have to write the Mythical AI Month after this.

If you lose 90% of developers to AI, but everything breaks, how much money have you saved?

The underlying moral issue is that the management and money people have contempt for people who don't manage and don't have money. Which includes customers and engineers.

It's not just an inhumane system - although it is, that, brutally - but it's also technologically dysfunctional and ultimately unstable.

"This is on the money!"

Sorry, had to.

211 muted people on the Teams call, 2 guys trying to figure it out, an incident manager asking for an ETA and status update every so often.
I wonder if it’s an AI death spiral: “the service is down. Diagnose the failure.” … Hello?
Why would anyone introduce a single point of failure out of their control into their company?

We're doing anything to avoid single points of failure in our products, but our IT is doing the opposite.

The difference between a distributed system and a monolith is that in a monolith you have some idea what your single point of failure is, but in a distributed system you don't know your dozen single points of failure until they happen in production.
I don't see a reasonable alternative here. Pay for another email/office suite provider, and duplicate all mails and files? Twice the cost and exposure surface just to cover 0.01% of downtime.

Sure, it makes sense for mission critical companies (utilities, emergency services), but I don't think it does for an average company. Especially that you don't take much blame if everyone is down.

Email in a large part is just a tool for communication(and mostly low priority at that). Where I work, we have backups: Teams( same) provider, Cellphone ( different provider ) , Landline( different provider)

We had a situation recently where cellphones were down, we switched to landline.

You don't need a 1 to 1 mode backup for everything, just a failover plan.

Unless your company is infrastructure or medical, I doubt they have a backup/failover for loss of power. Or water.

The repeated comments on their twitter feed are weird, is it an LLM writing it? why can't they just wait until its fully fixed?
Regular communication is an important part of outage/crisis communication, even if that communication is "nothing new to share".
It's actually nice to have some updates on what they're doing that go further than just "we're investigating."
Apparently this single outage means Office 365 uptime this year drops to 3 nines (2 nines for this month). You can claim SLA service credit worth 50% service fee by applying before December 31.
I couldn't specifically say since my admin.microsoft.com fails pukes on a wildcard certificate for azureedge.net.

Also, multiple bankers I'm speaking to about Anthropic's IPO can't send or receive work e-mails. So that's a thing.

I just realized your name is the entire song / artist. Are you Jermaine Dupri?
"Office 364" is apparently a trending term.
Microsoft Outofluck
Office 360 at one point in history.

We just don’t know how good we’ve got it, that’s all!

Do you have official reference for this?
There's two parts, the actual downtime and the SLA agreement...

At my day job, we have a calendar integration that polls Microsoft's Graph API across hundreds of Microsoft 365 tenants. We saw our first metric spike in 5xx errors at 2026/11/12 17:33 UTC. As of writing it's 2026/11/13 09:44 UTC and we are still encountering a full outage, so 16 hours and 11 minutes.

With a SLA calculator [1], assuming no other downtime this month and entering 16h11m, we get:

* Yearly SLA: 99.82%

* Monthly SLA: 97.78%

Next, what is the actual agreement? I found an SLA document list here [2].

I downloaded Service Level Agreement for Microsoft Online Services (WW) English (2026/10/01). There's a heading for Exchange Online that says:

Uptime Percentage : Service Credit

< 99.9% : 25%

< 99% : 50%

< 95% : 100%

Which seems to agree with what the original commenter said for this month.

[1] https://uptime.is/reverse

[2] https://www.microsoft.com/licensing/docs/view/Service-Level-...

> Uptime Percentage : Service Credit > < 99.9% : 25% > < 99% : 50% > < 95% : 100%

These targets feel very very low nowadays.

99% uptime for email means 3.5 days/ year no email.

(although 99.9 = about 8 hours/year would seem bearable - but maybe not for a company like microsoft)

> 99% uptime for email means 3.5 days/ year no email

Depends on how it appears, if it was one or several large chunks that is much more disruptive than minutes per day. 3.5 days/year is equivalent to less than 15 minutes/day.

It sucks no matter how you slice it, but the severity/impact depends on more than the aggregate downtime.

Who needs senior engineers. We are going to have LLMs troubleshoot billion-scale systems!

What could go wrong, said all of the hyperscalers. Enjoy your multi day downtimes.

Funny, my self hosted email has a better uptime than this pos
Email is actually designed with intermittent connectivity in mind. The sending server buffers outgoing emails until they're accepted by the receiving server. Typical settings are to retry every 4 hours, with a permanent failure being counted after 3-7 days.
Work's outbound email server for server/API/website emails (like password reset requests) has 127 emails in the queue, all to MS365 users.

Some of the big mail senders must currently have enormous queues of delayed messages.

It'll be partitioned by receiving domain. The server will rotate through each domain that has queued messages and try to deliver the first message in the queue. If it can't reach the server it'll move on to the next queue. Or something like that but not necessarily exactly that.
26 Oct: Microsoft offers voluntary buyouts to 60,000 employees as AI leads to efficiency push

It's just a wild theory, but maybe there are still too much employees left, standing in the way of AI? I would try firing another 60,000 first.

Ugh. My sarcasm detector is down. Shouldn't have hosted on MS servers.
Welcome to Outlook/364
If only this year were a leap year Microsoft would be in the clear
Their current engineers must have stored their KT notes from their laid off and leaving senior engineers in OneNote on their Mac, and might have been working with copilot to find the root cause /s
I have always felt it's less about email redundancy and more about having a backup plan. If outlook goes down tomorrow, does everyone know how to keep work and client communication moving? that is something worth thinking about.
For the amount of uneeded emails about future emails about other emails, probably no real work was lost.
Who wants to bet that Microsoft is using slack to communicate now?
Good morning and happy Friday the 13th everyone
Having friends who work at Microsoft, this is not really surprising at all. The company has always had deep structural problems. The AI change has really just exposed it and is aggressively accelerating the inevitable implosion. It's truly in a free fall that is planned and orchestrated perfectly by their upper management. Their CEO and VPs are convinced you don't need engineering know-hows any more because of AI, and they got all the management layers to sincerely agree with them. Not in a malicious or subverting way, but in a "you can't get a man to understand something his salary depends on him not understanding it" way.

Microsoft has a moat, but their moat is having billions of dollars to spend on data centers. It's a failure everywhere else. It'll continue to rake in billions in revenue because people need compute now, but it's rotten at the top.

The amount of racist memes targeting Indians on the replies.
it's twitter, what do you expect?
I thought:

"Have you tried firing all the white men that invented everything and replacing them with people from third world countries where people shit in the street?"

was hilarious.

Really? It's not very funny and mainly just white supremacy.
Yes, really - immediately showed my wife who also burst into laughter.

We're not white. Not sure why I'm getting downvoted by merely reporting I found an obvious joke funny.

Or did you think that tweet was actually serious advice from a concerned samaritan?

I don’t know why I need to explain this, but racism isn’t limited to white people.
Care to explain the humour in the 'joke'? It is obvious that the statement is not earnest, but it is framing its criticism in potentially the most racist way possible. It wouldn't be particularly funny even without that.
What an excellent question. I am no comedian, but let me try and break it down as best I can.

What this joke is: Crude, absurdist satire of corporate DEI culture and cheap "offshoring".

Why it works: It packs quite a lot into so few words:

a) It hijacks a polite IT support cliche "Have you tried... (turning it off and on again)", setting the reader/listener up with a familiar rhythm.

b) It then seamlessly pivots into taboo-breaking; just like South Park (for one of a zillion examples, this stuff is literally older than Shakespeare), the "saying something you are not supposed to say" is what draws laughter.

c) It ends having made a good point: an aggressive focus on profit, rather than quality, has increasingly become one of Microsoft's quintessential characteristics. The outage is unsurprising in that light.

If Teams is down I take that as a service upgrade
Alas, Teams is working, although parts of it which are integrated from other services are not, like calendar (exchange), or file sharing (share point).
At least Microsoft 360 still has a nice sound to it.

No Outlook, they should make this a public holiday!

Its microsoft 360 not 365 (days in year). So they have four more days.
All 3 of my outlook accounts are still dead.

I am looking at other current events and beginning to wonder if there is some kind of correlation.

This is going to cause absolute chaos. Basically every government organisation and business in the UK runs on outlook. The whole day is going to be a write off. It’s like a bank holiday!
This observation applies to many other countries, too. Mmmm centralization is pretty fun!
We're coming to the end of the Friday 13th business day in East Asia. Can confirm it caused chaos for the whole day.
Same is Aus. Wife works for government, they got sent home early (typical gov that means 30 minutes before official clock off)

The biggest issue has been so many security and approval flows depend on email, so all processes have ground to a halt.

Citation needed.

Government I have no idea, but loads of businesses don't use Outlook - alternatives do exist. I've personally not used Outlook for well over a decade now.

Quick google AI mode check says:

- MS 365/Outlook: ~40%

- Google Workspace: ~25%

- Zoho: ~2%

And then a bunch of others - Rackspace, Zimbra, IBM WebSphere Lotus Notes Blue For Enterprise XML Watson Edition et al

Every government globally with all parts and agencies is purely microsoft. Some EU states are trying to move away from it for obvious reasons (not this event, more like national security and sovereignty which you don't have with MS products).

Exchange and Active directory are the biggest obstacles by far, bureaucrats can learn how to work with open source office alternatives easily, and underlying OS can be anything as long as you have few icons to click on.

> Zimbra, IBM WebSphere Lotus Notes Blue For Enterprise XML Watson Edition et al

I see that you're a hater. Outlook sucks but your solution is to cure cancer (Outlook) with Alzheimer's (Zimbra, Lotus Notes)?

Depends what market you are in. In the UK I would estimate M365 usage is ~100% of local/national govt.

And in terms of classic 'corporate' sectors like law, finance, etc, it also close to 100%.

Whereas in "tech" it is far more likely to be Gmail.

I _still_ get people who are non technical that work in 'corporate' sectors that can't join a Google Meet because they think it is Teams. They literally haven't ever used a Google meeting before.

> Lotus Notes Blue

I believe nowadays the general solution is HCL Traveler (server side) wit HCL Verse as client. The latter offers email access over pretty nice looking mobile, desktop and web apps.

Now's the time for those NHS fax machines to shine, I guess.
> We're aggressively pursuing all pathways to reduce impact

I love this kind of wording. It makes everything suddenly more serious lol.

What a fancy way to say, "we highkey have no idea why it's broken"
(comment deleted)
“Co-pilot, fix the outage. Make no mistakes.”
This is very serious: I cannot access Viva Engage.
TIL that Yammer turned into that
What turned into MS Teams then?
Didn't they turn Skype for Business into Teams?
Just by looking at the O365 UI changes they have been doing tells me there’s a massive competency crisis within that team.

It would not surprise me if the entire project is complete and utter (human-written) slop