613 comments

[ 2.7 ms ] story [ 88.2 ms ] thread
Down detector shows like everything is down. AWS, Azure, Office, Gmail. Everything. https://downdetector.com
Teams is working!

Or, rather, it's acting like it usually does.

Yup. I just tried to search my chats for a specific phrase. It came back with nothing, even though I know I used it in a recent conversation.

I just assumed it was Teams being Teams.

I asked someone in our teams meeting if they were having issues with their outlook today. What followed was a 5 minute hate fest on outlook. I'm not a big fan, but was kind of surprised so many people genuinely hate it.
If you could create a real world Outlook, it'd be like the Office Space fax machine.
The new version which is a skinned web browser is terrible
Office on Mac is like malware. It brings the system to a halt.

The worst part about Outlook is the broken search. In the world of infinitely smart AI, the e-mail search in Outlook is embarrassingly stupid.

It’s not just that it’s stupid. It’s broken. It can’t handle email that is on the server and not on the machine, and they’ve configured the app to store most email only on the server.
That choice is also so stupid. Why doesn't Outlook just store the text of every single email on all devices. The stupid software needs gigabytes of storage, why can't it spend a couple of megabytes on making e-mail actually work.
down detector always looks like this during big cloud service outages. i think partly because people don't actually know what services they're using and report wrong things, and partly because downdetector parses social media posts like "outlook is down but gmail still works" as outlook and gmail both being down.
(comment deleted)
I work in IT and people are complaining about Atlassian and about AWS but in reality if your organization is using SSO then that would get impacted.

If your org is using Intune or Defender EDR then you probably also are impacted.

Entra is not impacted, SSO is working for non-M365 apps.
Downdetector does not measure service availability. It's more of a vibes thing.
Downdetector is a survey, not a detector, and suffers from user confusion when a high-profile service is actually down, as users start reporting unrelated things
Especially when looking constitutes reporting (Schrödinger's downdetector?) and people look at everything after showing up for one thing.
It also suffers from people reporting functionally related (but not technical related) things.

Any other service that interacts with Outlook in some way will create downstream effects in other services expecting Outlook to work normally.

The biggest problem with downdetector is how they visualise their outage graphs.

The Y axes are entirely independent, and so when there's an outage for Microsoft Outlook (which peaked at ~130k reports) and you get a negligable spike in pepole who get confused and report gmail (~1k reports), the graphs all look correlated and connected.

They need to show the graphs with an absolute y-axis set by the worst outage, or scale the graphs by estimated average daily users or something.

Seems like URL scanning is down entirely and some intermittent failures in Sharepoint.
Anybody else was served raw text of source by default ? (gwire is down too)
The GPUs are having a moment of silence for their fallen leader
Maybe everything relied on an old server running a cronjob in Jensen's basement.
Sure, outlook.com is down, which is bad for everyone that chose to host their email there, I guess, but as an interesting side-effect, everyone with an on-premise Exchange server and "new Outlook" clients is affected as well.

Because, well, once upon a time there was Exchange Server and MAPI (the "Messaging API"), and that actually worked pretty well, and pretty much everyone started using it (or one of its EWS/ActiveSync offshoots), but then the Internet happened and ruined everything, but we got MAPI-over-HTTPS eventually, which by that time had grown so many warts that nobody at Microsoft really understood it anymore, so that got deprecated, even though, well, everyone and everything still uses it.

The new hotness was Azure-slash-Office-365, and that's where the "new Outlook" web interface lived and (sort-of) continues to live, and the on-prem customers got stuck with the "Exchange Subscription Edition", which was basically Exchange 2019 plus security hotfixes at extortionate rates, but with none of the new functionality.

And, since MAPI continued to be really, really hard even in the age of Copilot, access from the new hotness to the deprecated on-prem servers was implemented via IMAP (LOL), but force-proxied via the outlook.com cloud/clownshow.

Which is why, right now, many Microsoft customers that thought they were self-hosting their email are entirely down. Lessons might be learned here, but I sort-of doubt it...

Exchange has been a circus for years. I think that's why everyone went O365 in the first place. Managing to have a paid up configuration where you go down when a cloud provider goes down but you're hosting it locally is absolutely stupid and I am laughing my ass off here.

Also laughing at our ops guys who are now absolutely helpless and have people phoning them. I warned about this on our risk call and it was seen as low probability low risk. I hope it's down for a few hours :)

I don't think many orgs are using "new Outlook" to connect to on-prem Exchange because you miss a lot if you're only doing it via IMAP and "new" still doesn't support on-prem Exchange any other way. So all my clients are using classic Outlook with their on-prem Exchange servers (and really if they have EXO too because new Outlook isn't very good).
Do e-mail messages that self-hosted Exchange receives always get proxied through the outlook.com cloud?
Is this related to GTA6 being pushed to consoles a week prior to launch? Would they ever admit it if that were the case.

I want to believe this is the cause. It'd be especially funny because businesses were already anticipating the productivity loss when GTA6 comes out, but now we have actual productivity loss because of this outage (and it's affecting people who were never going to play the game anyway, lol)

I want to believe that cause is that all the world's computing power is dedicated to calculating the incredibly large balance in my checking account but my desire is the only thing I can find to support that hypothesis.
An estimated 2.6 million pre-orders for Xbox alone. 2.6 million people downloading 200+ GB is not nothing.

If they roll it out evenly the week prior, then today (1/7th) is just 75 PB. To get that done in a single day would require a sustained 6.9 Tbps.

I'm sure this is all handled by a CDN at some level... but again it's not nothing.

It is basically nothing once you remember the cloud isn't one computer either. I want to say Azure was already millions of (beefy ass) severs over hundreds of datacenter locations organized into dozens of zones, and not all of those servers in each region are assigned to the same sets of services.
At that traffic, ISP level cache servers matter more than centralized clouds "(beefy ass) severs.
Why would this be the case? Games preloading has been a thing for many years. This isnt like an MMO launch that there is additional impact when the game is being played.

Sure GTA6 may be a slight uptick in the amount of data being downloaded like any new game being released, but I would wager that the amount of data they are normally pushing out for downloads just on a regular basis (wether new purchases, redownloading games, game pass, etc) is already quite high.

Nice, so at least your messages sent to their SMTP servers will actually bounce instead of being silently dropped /s
Anything sent to my work or home Outlook email from gmail doesn't bounce back, and anything sent from those is still stuck in the outbox. Microsoft.
It won't actually bounce until either gmail gives up on its retries (which generally is 24 hours) or the outlook server gives an explicit rejection (rather than a temporary error or timeout)
Normally if server down, smtp will wait a little while and try again.
Yep.

  421 4.3.2 The maximum number of concurrent connections per resource forest has exceeded a limit, closing transmission channel. xyzxyzxyz.outlook.com. PRX8 [Name=InboundProxyTrackingAgent][AGT=IPT][MxId=xyzxyzxyz] [xyzxyzxyz.ausprd01.prod.outlook.com 2026-11-13T01:08:15.373Z xyzxyzxyz]
Outlook is fine. Exchange is experiencing issues.
(comment deleted)
That kind of depends on what you mean by "Outlook".

Sure, the program on your desktop computer is fine, but the Outlook app on your phone routes everything through MSFT's servers, so you may be impacted. Outlook.com is running EWS, so it would be down as well.

And finally, if you're talking to a layperson, they're not going to know what Exchange is. They just know they opened Outlook and it isn't working.

No, it doesn't depend. Outlook is an app. If you don't have internet access, do you say Chrome is down?
Amateur hour at Microsoft. It baffles me how many companies can’t figure out how to make a status page that can handle the high loads it’ll have to sustain when the service they monitor is down. That’s pretty much the only reason to have a status page in the first place.
quite often, and after 10,000 such hours the expert amateur emerges.
GitHub Copilot strikes again, perhaps?
[dead]
(comment deleted)
I dislike Microsoft as much as the next guy, but many businesses impacted by these outages are completely innocent and don't deserve these consequences.
That's a very painful lesson in "the cloud is just someone else's computer" to a crowd that normally are exposed to ideas like "software and IT are not our core competency"
When your system is down once a decade your boss yells and it’s your fault

When your outsourced system to Microsoft goes down every few months your boss can’t yell at you

Outsourcing is about shifting blame elsewhere

Which is why service level agreements were invented.
The idea is that it's someone else's computers that are managed more comprehensively than you could reasonably do yourself. But if you choose Microsoft to provide such a service for you, I don't know what to tell you.
The truth is they are still managed better than most can reasonably so themselves.
I begrudgingly have to agree.
If your system keeps people alive by relying on 100% Microsoft uptime, maybe they deserve some blame?
They do government and medical infra, those systems should be resilient but...
To be honest, Microsoft has much better chance of actually achieving that (almost) 100% uptime than your company's own IT department self-hosting things.
Github would like a word.
If the self-hosted email infrastructure goes down, roughly one company is deeply affected. When Microsoft's server goes, entire industries and governments will simultaneously be unable to conduct business as usual.

From the individual's perspective Microsoft might seem like the better choice if they can achieve higher availability, but the societal cost is so much greater when the one basket you've put all your eggs in breaks, even if everyone holding their own egg on average results in more broken eggs per year.

I like to dogpile on M$ as much as the next guy, but this feels like a heavy strawman.

> I dislike Microsoft as much as the next guy, but many businesses impacted by these outages are completely innocent and don't deserve these consequences.

At no point is "keeping people alive" mentioned. Businesses use email for everything. Businesses usually don't care about where their email server is hosted. They just want to send emails. They shouldn't have to care either. I don't put blame on a business that happened to choose Outlook, one of the largest email providers, to host their email.

> Businesses usually don't care about where their email server is hosted.

Deeply irresponsible.

> They shouldn't have to care either.

According to what realistic principle?

their bosses decided for the cloud and they pretended not to care!
It's a shame there isn't a well-standardized, free and open protocol for delivering email that could be run on inexpensive modestly provisioned hardware.
It’s a shame we still use email in 2026
They then get you by spam filtering any emails not coming from the "industry leaders".
I mean, they made us take down our departmental run email (presumably postfix) and forced us to make them aliases to university-wide exchange (with crappy support for open source mail clients no less). So I don't think my org is Innocent here...
Of course they're innocent, but many business also can't afford nor want to employ people to host something like an email service for them.

I'd rather Microsoft host it for me than have to maintain a team of people with Microsoft Exchange skills running either on a cloud environment (similar issues) or on my own infrastructure.

I for one don't miss the days of Exchange going offline or having issues and being the person having to try figure it out.

Microsoft has much better reliability than MOST companies.

Remember, my guys : “Microsoft Works” is an oxymoron.
Outlook is down AND Jensen is out? it's like some cosmic convergence
Jensen: If I'm going down I'M TAKING EVERYBODY WITH ME
Neither email nor Teams is working at my non-tech megacorp employer. Wonder how much of the economy this is impacting.
Productivity up 20%
My PRs need approval but I have no way to ping anyone (screams internally)
Is Excel down too?
license servers are down, so we can't use even the desktop products at our large enterprise
Then it’s a huge impact on economy
Teams has been working all day for me but I haven't been able to use outlook (app, not web) for about 7 hours now. Fortune 20 company, BTW.
Copied this from my post in the other thread:

The Status page is not working well, but I managed to subscribe to their RSS feed and it has been sending out updates periodically. I'll post some of them here.

7:52 PM UTC: We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service.

7:30 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.

7:16 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've also moved traffic for some services off the affected infrastructure to further reduce load, with additional traffic moves in progress.

6:59 PM UTC: We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.

Absolutely bizarre wording of these updates. Strangest corporate speak with urgency turned up to the max.
copilot, make this doom and gloom situation sound better. assume you're using the voice of a friendly but intensely focused sysadmin. use only neutral language.
So the servers are ok but many of the health checks are failing and so the servers are taken off rotation, increasing load on other servers. I'm sure it's more complicated than it seems but from my POV this looks like amateur hour at Microsoft.
At Microsoft every hour is amateur hour.
Don’t worry, I’m sure it’ll get better after they layoff 60,000 people.
Don't worry, their replacements will receive very clear instructions instructing them to make no mistakes.
I'm not thrilled with what they've done to Windows lately myself, but this is just rude.

You know a lot of talented people who work there also post here, and it's not fair to insinuate that they aren't talented because some vastly complex system somewhere in the world had a bad day.

Stuff happens, things break. That's why lots of nines cost a lot of money. It doesn't automatically mean that anyone is incompetent.

i agree with everything you said here except the last part.

lots of nines means not having significant downtime. we passed 99.99% hours ago, and we're on the way to 99.9%

Lots of nines means they have SLAs of no significant downtime. It does not guarantee that the SLAs can never be missed, just that there will be consequences when it is missed. Lawyers and accountants and sales folks will sort out the compensation payments that come back to the customers and try to negotiate relationships back to a healthy state.
That’s purely a choice by those claiming the nines. I’ve worked in companies selling software with more than 6 nines uptime (so that we can be more confident about the 5 nines). Not 4 nines uptime plus some money back for the other one.
Stuff happens and things break everywhere but right now at Microsoft they break badly and often, that points to actual issues
When was the last major M365 outage?
There are lots of talented amateurs. What separates professionals is standards, quality, teamwork, etc.

I agree there are many talented people at Microsoft, but I don’t see the professionalism in this incident.

Individual talent is irrelevant when you have a company culture that rewards the opposite of that. Any talented person working at Microsoft knows that they are dealing with structural problem that force MS products to be massive piles of shit. There isn't much an talented individual can do about that.
An organization can become fundamentally opposed to the purpose for which it was created. Organizations simply don't follow the same laws as the individuals inside it do. A company can be made of geniuses and be totally ineffective; it can be full of talented go-getters and be completely unable to fix problems.
Satya sock account found
ButtSlut5099, the finest detective since Sherlock Moans.
I realize I'm responding to somebody who isn't paying much effort, but still…

Isn't that comment saying the direct opposite? If it's a cultural problem, its precisely Satya's fault for not fixing it; that's the executives' job.

I was confused too, but I said hey, it's probably typical buttslut5099
the rot happens from the top

and at the top is an indian

and thus, the indian failure mode commences.

4

The enshitification of Windows was not technical incompetence; it was intentional product direction.

They transformed Windows from a useful OS to a banal surveillance and advertising platform.

Microsoft's superpower has always been hiring smart people and making them work under dumb management to create flimsy, mediocre outcomes.
I'm not actually sure this is true anymore. It seems most of their talent has left.
A few weeks ago I saw a clip of a lady walking us through a day at her Microsoft life.

Between breakfast, 1-hour lunch, 15-min ice cream break, 30-mins chill, 30-mins foosball games I think she worked 4 hours max.

At that point she caught some flak since it was timed with the layoffs. And I get that is not necessarily representative of the MS culture…

… but talk about timing and poor choices!

The people writing these updates are very obviously not technical and are least one step removed from the actual work being done to fix this.
To be fair: I want the people who can speak most competently to a problem to be working on it versus writing commentary. Those highly-technical people can shine in the postmortem report.
I disagree; the communication aspects - including external - are a hugely important part of any incident response. IME the biggest challenges in these situations are the many levels & combinations of communication and coordination. You don't need your best DevOps person doing it, but there should be plenty of competent technical product managers or engineering managers who could do this well.
And how do those product or engineering managers get information on what’s being tried and what is being discovered without slowing down the process of fixing the issue?
Microsoft is not a small startup where the CTO is getting pulled into firefighting.

They should have established procedures for large-scale events, and they must include a coordinator whose entire job is to manage the external and internal communication.

And that is a good thing. Well, maybe not good. But the last thing you need in a crisis is to pull people away from fixing the problem and make them type up status updates. Admittedly, having tech knowledge would help the writers - a technical incident management team would be better. But ain't nothing wrong with letting the PR folks write up words while the tech folk work on problem solving.

Actual tech write-ups can come later.

Amateur hour? Not sure if you've lived through one of these cascading failures, but they are hard to design against, hard to load/chaos test for, and hard to recover from. The key is that this is a dynamic load, not a static load; and it's normally amplified by uncontrolled retry behavior by clients.

In retrospect, the cause and cure are obvious to everyone.

Many of us insist upon not building Rube Goldberg machines that cannot be reasoned about precisely for this reason.

Most of the time we get demonized by resume-driven developers job-hopping every few years (seems like the lack of deep expertise is working out). Today we get to laugh at their expense. These problems were entirely foreseeable and avoidable.

> building Rube Goldberg machines that cannot be reasoned about

What a great phrase!, can I use that in conversation?

Isn't a complex computer system, at the lowest levels, an example of a Rube Goldberg machine ?

If we could see the memory bits and the CPU internals, I think it would be good candidate, particularly branching and branch prediction.

There are methods for building complex systems our of smaller pieces. Humanity has done this forever. An internal combustion engine is very complex. An airplane is very complex. You put a boundary around your system, you build it to a spec, you test it to a spec, you define the behaviour across anything you can see. When two engines fail on the airplane it can maybe still at least glide. Maybe there's an ejection seat. Because someone is doing the system thinking with some level of discipline.

A brick is very complex. A bearing is complex. A bolt and a nut and a washer are complex. An integrated circuit is complex. We build stuff out of that. Like buildings that don't fall when you sneeze on them.

In software we use duct tape - at best. But that's not just Microsoft, that's pretty much everyone. With maybe few exceptions.

Software systems are by nature more complex than physical systems. They also lack any tangible interfaces beyond code and diagrams and collective knowledge in the minds of the humans who build and maintain them.. The rapid evolution of software is no help either..if anything AI is accelerating that rapidity as well.
In my experience it is always the "business" people that do not want to take the time to do it correctly.
Except for the fact that it seems like most Microsoft outages are self owns. I'm not sure how many times we've had outages due to bad deployments that somehow made it to their global infrastructure and only got noticed when customers notified.

Plenty of internal accounts from previous employees highlight this.

To be fair to Microsoft (and I'm having trouble believing I just wrote that), the vast majority of outages are "self owns". If you don't change anything, you're not going to break anything, and major external events (fires, cable cuts etc) are rare and can usually be worked around.
I have lived through and debugged cascading failures and large scale distributed systems.

The problem is amateur hour is the entire industry. Nobody specs anything, nobody tests overload conditions, nobody thinks failure modes. The stuff that every engineer does when building anything else other than software doesn't happen. The orgs are bloated. Too many teams. Too many silos. Incentives are stacked the wrong way. People are motivated to not think about systems but stay in their lane. Responsibilities are split in weird ways.

Yes. Big systems are hard. But we can build, manage and understand big systems with the right approach and people. Even one computer is complicated and many people get stuck trying to debug a piece of software running on one machine. At the end of the day it's not magic. All of humanity is big systems but when I have a power failure the power company is generally able to understand why and resolve it and the water in my tap doesn't stop flowing and the cell towers still work (generators, who would have thunk).

A friend who is really good had to leave Microsoft because of how AI was pushed. Before that he was telling me some other stories that make the culture seem pretty not great. Heard similar stories from other people. It used to be a great place filled with great people until at some point it's just another big company.

I'll be the downvoted comment, since HN doesn't like to hear this, but the other flavors of engineering do stuff we don't because they're regulated.

My bet is we'll eventually get there. Planes kill people when they fall, software used to not be as dangerous, but it's been a while since it became critical infrastructure.

I can't think of any other engineering discipline where demand might increase by a factor of a million in a few seconds because some DNS record was propagated incorrectly or something - regulation won't magically solve the issues unique to software. Maybe it's doable with vastly increased costs (a classic would be running fully redundant 1:1 warm failovers, which nobody could economically do at the cloud provider level.)

That's not to excuse incompetence or whatever, just to note that the benefit of software is precisely how easy it is to change, which will always lead to more problems.

Electrical engineering of the power grid has the exact same failure mode: cascading failures of interconnected power grids can cause loads to increase by factor of a million in a few seconds.

I remember a power outage in 2003 that knocked out power for 55 million people and took over a week to restore in many places.

Also known as *the* blackout. https://en.wikipedia.org/wiki/Northeast_blackout_of_2003

Note that buggy software was a major contributing factor! The software was designed to, and would have, detected a deteriorating system an hour before anything broke, which would have provided ample time to intervene. However, a race condition caused the whole monitoring system to display stale data and then crash, including the redundant backup. This wasn't the only problem by far, but it was one layer of swiss cheese.

Also note the power companies have clear procedures to reboot the grid, known as a "black start[up]". And they test these procedures, though without taking down the grid.

I found out they have prepared, in advance, a list of which disconnects have to be open at subatations in order for power stations to supply each other's startup needs (for stations that are not capable of unpowered startup) while supplying no or minimal actual load. There must be a software analogy to this. Does your company have a plan to bring up the system on a rackful of blank servers and a database backup? I doubt it.
Which is the kind of exceptionalism that really creates these problems. Software is the only industry that allows people untrained in running global infrastructure to run global infrastructure.
> My bet is we'll eventually get there.

Until the cost is mentioned. People love free or nearly free software.

i think it'll go the other direction with vibe-coded software.
The problem is that you are only as strong as your weakest link. At some point, something somewhere will be architected poorly and it will one day line up perfectly with 2 other bugs that were incredibly hard to predict and then you're stuck in Microsoft's current situation.

Also, power grids are a poor example. There's been multiple instances of absolutely catastrophic failures so hoity toity capital E Engineers also make mistakes.

https://en.wikipedia.org/wiki/2025_Iberian_Peninsula_blackou...

https://en.wikipedia.org/wiki/Northeast_blackout_of_2003

We took some of the hardest problems in computer science: distributed systems, scale, concurrency and networking, and made them the standard for every To do app.

This is the inevitable consequence.

When I look across the scale of all the failures that have occurred in big tech over the past few years, my self-hosted solutions have been cheaper and more reliable. They are simple. They are small. They are easy to fix and diagnose. Because they're not trying to solve all the world's problems.

I think scale is the keyword there..
People expect software to be feature rich, complex and solve all the world’s problems. And if you don’t do that, the vibe coded competition will beat your cleanly architected product commercially.
Indeed, it's an expectation problem, and it's even worse in higher management.
It's not even better though. It doesn't have more features because it's a distributed system. It isn't faster because it's a distributed system. Maybe it has a bit higher uptime but nobody was paying too much attention to that anyway, except when it's down. And it's much better for your PR if, when it's down, it isn't down for everyone at once.
Users don't expect infinite features.

Managers and marketers are the ones asking for all those useless features. And god forbid they run out features to ask for, because then they start asking for architecture v2.0 projects.

This is the same as what's been happening with entire industries: Build for the critical path. Eliminate redundancies. Eliminate resiliency. Lean staffing. Just-in-time manufacturing. Single-vendor mandates.

In the name of efficiency and profit, we have stripped away all the safeguards that should prevent cascading failures after the first domino falls. Anything that doesn't return profit now, today, every day, must be purged. Because what if that domino never falls? Then you wasted your money! Think how much you can save if you don't need to make backups! (And every dollar saved is a dollar in my pocket!) Etc.

And torben-friis is right: the only solution to this, demonstrably, is regulation. Require multiple vendors for every major component. Require staffing sufficient to cover a third the company being on vacation. Require resilient systems, that can take a single disaster and keep on trucking, because they are the backbone of our economy.

Regulation will just make a bad situation worse- complicated things rarely get good regulation.
So what's your answer? Just ask the giant megacorporations siphoning billions of dollars a year into the pockets of oligarchs to pretty please stop ruining things for everyone else while they enrich themselves?

This attitude of "regulations don't work" "taxes will just cause wealth to flee" "increasing the minimum wage will just be inflated away" is all propaganda. Propaganda meant to make us believe all we can do is give in to the megacorporations and the oligarchs who lead them, and let them do whatever they want.

The best defense against cascading failures is limiting the blast radius, which is something AWS seems to do well but Azure does not.
From this we can tell where Jim Roskind works.
How did that AWS DNS incident go recently? Unplanned outages are unfortunate but common, and hubris is the problem when everyone assumes that they (or their pet vendor) are immune.
Exchange Online and Microsoft 365 in general doesn't run on Azure.
We learned the hard way that there's a very non-obvious inflection point in large distributed systems, where the your design needs to change from accommodating load resiliently via queueing, auto scaling, and such, to a posture of actively scaling back certain features and even shedding load when in a defensive situation.

It's really not intuitive, and at least for me, the need to admit that we couldn't just comp-sci our way out of some conditions was painful.

This sounds suspiciously what occurred when I was working at a large motor sports company. We had a major release of new year models coming out on the website. The deployment went super smooth and then in about 30 mins the site was crashing. Since we were a huge MS client, almost everything we ran was all MS top to bottom.

Site was taken offline, war rooms ensued. 48 hours later, an Ops team from Redmond was flown out. Another three days and they determined it was several legacy servers were going down and the load balancing servers were getting hit so hard, they also failed. It was a cascading effect that killed everything.

CIO was pissed. The company got absolutely lit up on every social media channel for their ineptitude. The CIO vowed it would never happen again, they were upgrading the entire network and infrastructure to avoid it happening again. Fast forward six months to the next major release. Same thing, same panic, same cascading failures because servers were collapsing under the traffic to the website. Another war room, another Ops team flies out. Two days later they figured out it was some buggy code that was causing it.

This after 8 rounds of load balancing testing to the tune of 10X the amount of traffic the site sees on these major release dates without so much as a hiccup.

I left two months later, but the overall feeling was the same - total amateur hour from Microsoft after our company was gifted major discounts to go all in with their hardware & network gear. The company found out the hard way it wasn't a smart decision.

The scary part was this some 20 years ago - seems like not much has changed considering how frequently Github is down now too.

Newest update says that wasn't the issue after all. They say relatively small loads are actually causing servers to become unresponsive and have to be restarted.

They still don't know why this "core process" is maxing out the processor for no reason.

I'd start with "what has changed" but surely they've looked at that.
Could be something adjacent - the last time I majorly blew up a release was because something had changed (which was fine) but it caused a list of items to be broken over two pages, but only in prod. Me and a team mate were scratching our heads for the longest time because "what had changed" was quite mundane itself.
The classic 'database is smaller in testing than in prod' problem which is responsible for some outages IME. Often the 'query was not optimised but that didn't show until prod' one.
They did look at recent changes but none seemed to be related, but it turned out recent network changes triggered a failure mode in a much older network change. Dependent or cascading issues remain hard to diagnose or find.
I can't stand it when I see armchair analysts making claims like this.

I'm sure it's more complicated than that.... but from my POV, which is extremely limited and informed by what others online tell me.... that this is amateur hour

What would it take for it to be professional hour?

Professional hour would be invisible. Outages don't happen because they were anticipated and prevented.

You can't do much to work around nuclear war or a major asteroid strike, but keeping important things working under normal operating conditions should not be a high bar.

Of course keeping services online is professional hour.

Keeping important things working isn't not a high bar, however inevitably it will go down and the ability for a well experienced team to identify a complex issue and resolve it is also part of their job and is what professionals do.

I am worked with various technical incident response teams, some are people with their arms up, some of them are good displays of technical knowledge and communication.

Professional hour applies when things are online and when things go down.

This is like getting transmissions from Cassini before it falls into saturn.
Why is there not a single mention that this could be a cyberattack? Is the issue only affecting the USA?
No, my family is in Europe and Australia and also seeing it.
I can connect to US account services, but while first VPNing through France, rather than directly from US.
Nope. Europe (eastern) for sure included too.
UK seeing it
Brazil too
I work for a global company, it's affecting every single of our 50+ offices world through US, Europe, Asia, Middle East, etc.
Because there's no indications that it is the case, at least for the time being.
Australia affected as well.
Japan is also seeing the issue. It is probably a global issue.
It doesn't look like a cyberattack at all, talking about fixing health checks and the like looks much more like broken automation.
Impacting us here in Australia, our company is having fun today.
Don't attribute to malice that which can be explained by stupidity.
> Don't attribute to malice that which can be explained by DNS.

Perhaps more accurate in our world.

How would that help.

The issue now is fixing this so status of what happens is needed and not how it happened in the first place.

The cause will ans should only be published when the issue is fixed and reviewed.

Cortana, play My Heart Will Go - oh.
This thread is now complete. We can all go home now.
Day 12: It has been 3 days since the last ping returned, I fear I will not make it back to my family. Give my love to dear Liza.

I feel for those stuck in there. The outage going on for this long has got to be fucking brutal.

Why do you think this is a problem for Eliza?
She uses outlook.
The astronauts going to the moon reporting "we have two outlooks on the laptop here and neither work" will be funny for decades from now.
I though she communicated through RF emissions from CRTs and other monitors.
From the 11:06:31 PM UTC update:

Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We've also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.

We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.

And I can't reach my self-hosted mail server in the Outlook desktop app because of this.

For reasons that I cannot fathom, Microsoft wants to act as the relay to my self-hosted e-mail account, if I add the account in Outlook.

> For reasons that I cannot fathom, Microsoft wants to act as the relay to my self-hosted e-mail account

I can conjecture some reasons. Datamining would be high on the list, since MS can read your mail and thus market to you more effectively. Governments might also be happier if MS can provide them with one-stop access (like Prism) to otherwise self-hosted e-mail accounts proxied through a central server.

Or, you know, similar reasons as why people use Cloudflare as a reverse proxy.
I would strongly doubt (though not completely rule out) that Microsoft is reading your personal mail hosted on your personal mail server. Like what would that even look like, their IMAP client is forwarding email to headquarters?

BUT, more likely, it's some sort of "check in" or "dial home" feature that isn't working. Or plausibly, maybe they count "local user checked email" and their analytics endpoint is down.

Regardless, it's definitely surprising that a traditional desktop application would be blocking when only polling from local/trusted data sources.

It probably is a mirror for caching purposes for the mobile app or something. It lets them control the protocol between the client and their server as well.
But a check in or analytic counter wouldn’t stop you from receiving email. This is most likely due to Microsoft’s server (not your local computer) being the one fetching the mail.

It is normally probably more reliable; except when their server is down.

They proxy the whole ball of wax. Have done for years.
My understanding is: the IMAP client is hosted. The local app talks to a HTTP service, and doesn't have mail processing capabilities of its own.
[delayed]
This is your own fault for using their BS app.
you're getting downvoted but I agree.

play stupid games win stupid prizes.

maybe its the support desk burnout talking, but if you rely on MS apps and services you're gonna have a bad time.

Don’t get the downvotes. Why would you put in all the effort to self host mail and use something that proxies your mail?
I mean, I can still switch to Thunderbird anytime I want, because I use IMAP and don't yank the mail messages from my mail server. And I can still receive mail on my iPhone anyway.

It's just funny that Outlook breaks this badly.

"New" Outlook does route all traffic through Microsoft servers. I assume it's so you can get a "unified" mail look on all your devices (phone, etc), but likely a nice source of data mining
And I hope all of us have learned a valuable lesson.
Personally, I can't reconcile having a self-hosted email server and using Outlook to access said email.

I'm being an ass in saying this, because I'm aware human motivation is a beautiful kaleidoscope of variables, but I would have guessed at the venn diagram of 'those willing to compromise and use a Microsoft product' is mutually exclusive with that of the 'I'm hardcore enough to self-host email'.

I will say, however, that Outlook used to be one of the better desktop email programs, so I'll grant some legacy lenience.

What's your preferred email client? Just wondering what other people are using.
Best one I've ever used was Fastmail's web mail. But it's locked to their platform, so yeah.
Mac/iOS Mail is pretty lightweight. I used to use Thunderbird on Linux.
Surprisingly, Mac mail is a really nice client. It reminds me of how quick eudora was back in the day.
Pegasus Mail.

Almost as good as mutt.

eM Client for various M365 and IMAP accounts. ElectronMail for the 9 ProtonMail accounts I have (allows "tabs" for each account in one client).
Thunderbird still reigns supreme
Eudora
That’s a name I haven’t heard in a longgg time. Google says it’s still being used by some enthusiasts. Were you being sincere?
Thunderbird (desktop), K9 (Android).
The Bat!
The Bat! was my email client of choice for many years, many years ago. Will take another look, thank you for the reminder.
You're not being an ass. You're being a logical person who is advocating for individuals taking responsibility for their choices/actions.

Unfortunately, being someone who speaks up nowadays has been portrayed as being an asshole.

How dare you!?

> Personally, I can't reconcile having a self-hosted email server and using Outlook to access said email.

In a corporate environment, I don't think this would be that unusual. But yes, it's hard to imagine someone would do that in a personal setup.

It does have to do with the fact that Outlook still looks better than Thunderbird.
outlook 2010 works just fine
> For reasons that I cannot fathom

Biggest data heist and most obvious US intelligence operation in human history. I can't fathom why AVs don't treat "Outlook New" as info stealer malware.

I’m pretty certain their desktop client applies filters server-side.

I’m pretty certain because my company’s phishing-test emails skip my filters (I assume admins have a “skip filters” toggle). Which, hilariously, makes it even easier to ensure there’s a place that contains only phishing test emails (filter all not on allow-list of domains to one place, filter all on that list to another, ta-da, the main inbox now contains only phishing test emails)

I assume they do the same thing with non-MS email if you use their client, rather than maintain two filter implementations.

All around, yeah, wow, super gross.

Why would you self host, then use a client with a dependency like this? Seems like the worst of both worlds.
Allegedly, It's so they can use push notifications efficiently from a central place for new mail notifications. Mobile IMAP clients cannot reliably keep a TCP connection open, due to on-device power-management and network management purposes. IMAP natively supports push-based notifications, but mobile networks make that impossible to work reliably.
Which is odd. I also have Outlook for iOS, and the accounts I add in Outlook for Windows don't automatically appear in my Outlook for iOS, even though I have the same Microsoft account logged in in both. So I don't understand how they plan to make push notifications work better for me on my iPhone by relaying my IMAP e-mail account.
Surely you can just use another email client to get the email from your self-hosted server right?
Because the new Outlook app is actually a web app talking to a Microsoft Web Server that does the actual mail retrieval. This is all clearly explained in their documentation.
Outlook Classic also refused to start for many users - looks like it is trying to check/sync something upon start, and when 365 went down it borked completely

Turning off network completely during start-up of Outlook Classic helped to start it at least

but yeah, it all wants to be connected at all times these days...

It took 30 seconds to open a "local" file in Word, and my Outlook email wasn't updating.

In other words, a day that ends in "y".

My company was recently acquired by a company that uses Office, so I'm using Microsoft products for the first time in 15 years. It's absolutely APPALLING how bad they are. I assumed they got fairly usable in the last 15 years, but it seems they've gone backwards.

For all my gripes about GSuite, ... I don't know how businesses can justify using software this bad.

I'm in a similar situation, using Microsoft again after 13+ years. Outlook and teams I expected to be garbage, but Word and Excel was really surprising. I get it that everyone has that one feature of excel that is not available in gSuite so they need it. But UX in general is so much superior at Google it's baffling how they let it's workhorse fall behind so much.
GSuite was written from the start as an online collaborative platform.

O365 is Microsoft Office shoehorned into running in a web browser.

Not surprising. Azure's data centers are out of capacity. In our region they've announced they won't be able to build out capacity for several years.
Also noticed the Partner center is down e.g for uploading apps to the Microsoft Store.