I remember there was a comment from a MSFT-er talking about how unreliable Azure is. Combined with AI coding, is the whole thing collapsing under its own complexity and weight?
It shouldn't be news in the first place, why do we all use such centralized systems where everything breaks down because of one bottleneck is the better question.
Still ongoing, I have not been able to access my personal outlook email for more than 12 hours and as a result have not been able to access my accounts that use email 2FA including my virtual medical appointments!
Ha nearly the same setup here, FreeBSD/ZFS/opensmtp/dovecot/rspamd
And yes uptime (with short outages(updates etc)) is not that important for a mail-server, since the sender re-transfers in intervals (mine is 5min the 8 then 15 then 60 then 180 then something up to 2days)
So the uptime for the service is 100% (maybe 99.999) over the last 8 years.
Getting a clean ip is hard. I have my personal email server setup and had to get it off of some black lists maintained by baracuda. Although i agree personal self hosted email is fairly straightforward and ideally more people should do it. Hetzner provides cheap and reliable vms.
True, i have some hetzner (not vm's but metal), and Vultr (just vm's), i just had one bad IP until now (i just changed it instead of trying to get it removed from a blacklist)
I should've perhaps done that, but i got an additional static ip from them which i pay for so i thought it would've been a good one, then getting off of the blacklist was easy once you have dmarc and dkim and spf.
So i kept it.
Reliable is a flexible word. Hetzner does frequent, albeit often brief, maintenance operations that would incur a down time on a high-throughput live service. Not impossible to design around with enough redundancies, but far from "I have one FreeBSD box that's been up for 467 days with 100% uptime" that people used to brag about in the early 2000s. I have few VPSs and bare metal instances that I just `sudo reboot` when there is a kernel update with practically "zero" impact or downtime for myself. Something unimaginable to do in a production system where you need to cordon, traffic drain, update, reboot, uncordon then orchestrate that across an entire fleet of instances with different roles and characteristics in a specific order.
Every time the cloud koolaid come in to defend the latest drop of a SAAS system from 99.9 to 99% they seem to think it's impossible to run a self hosted service with a better uptime.
It's not, it's easy. We've been doing this stuff for years.
Doing it at billion user scale is hard. You are not google.
> Still ongoing, I haven't been able to access my personal outlook email for 12+ hours and as a result can't access my accounts that use email 2FA, including my virtual medical appointment!
I guess software is still not solved, I'll let Dario know
Why would any company as big as Microsoft choose to centralize software services so much that if something goes wrong, it effects everyone around the world?
You have a good point, but maybe more will de-Microsoft than would have in the past. This is a more severe outage than Crowdstrike for most businesses. But also, LLMs might make it easier for businesses to identify alternatives and realize the transition.
Because not centralizing it means you now have countless separate environments that you need to manage, and if you manage them centrally, congrats, you reintroduced a SPOF again.
You can reduce the blast radius of outages at the cost of increasing the frequency a lot.
Also, while they claim it is not a cybersecurity attack (without saying how they know), these are exactly the kinds of things I would expect with the soon/maybenowww future of automated hacking agents wrecking world's cyber infrastructure. Right now, its embarrassing, when it happens to the banks, it will be something more than that.
A successful cybersecurity attack on a major email provider like Outlook or Gmail is an attack on the banks. Access to a user's email account gives attackers most or perhaps all they need to access their bank accounts.
This take is bizarre and i’m really beginning to think that a scant few people are pushing it in bad faith in an attempt to spread rumours. It has no connection with reality. Extended downtime and faffing about with distributed systems issues all happen without it being caused by an attack.
One, coincidences happen all the time. Two, the world is always undergoing some sort of event that would make you go, "the timing is suspicious". There, distributed systems are hard.
I could and probably am was wrong, but it is not bizarre, nor was I starting rumors. Starting rumors involves a little more than that. Rather it is a natural line of speculation given the times we are in, and the fact that, now well documented, AI hacking attacks are a thing.
I mean why is outlook even centralised? If a company has 600,000 employees, why don’t they get their own infrastructure? Why is it all dependent on central Microslop infrastructure in America
I have administered both Exim and Exchange and would very much like not to do so again. Especially since in the time since then the number of requirements for effective delivery and antispam has only gone up.
It's on days like this that Microsoft could really learn a thing or two from Google's SRE culture. These large tech companies have far too many internal dependencies, creating massive blast radii when something goes wrong.
The reality for most companies, up until now at least, is that if they manage email servers themselves they will have more annualized downtime than if Microsoft does it. Of course if this outage lasts long enough (who knows what's going on over there), that will stop being true.
IMHO a bigger concern is that Russia hacked Microsoft in 2022 and China hacked Microsoft in 2023. Even that wasn't enough to scare companies off MS.
Basically, MS customers are locked in, the cost of migrating to anything else is huge, and there's a "no-one gets fired for buying MS" herd mentality, so MS has little to lose no matter how hard their product fails, just like Cloudstrike.
There's also the question of what alternative would really be better, or at least more reliable. My impression is that Google's infrastructure and engineering are genuinely better, but I'm recently ex-Google so take that with a grain of salt.
That's all predicated on having a team of people with the appropriate knowledge of how to securely self host an email system. Some organisations aren't big enough for that to be worthwhile, others have decided to forego that cost to let the cloud providers pick it up.
Email is not a particularly easy thing to reliably host and secure for a sizable organisation, especially if its not your main focus.
Annualized outage minutes is the wrong metric. If my company's email is down for fifteen minutes, twice a day, at 2am and 3am, that's over a week of downtime. But we might not even care.
If I lose a whole day, we lose all the productivity for that day and delay countless upstream and downstream efforts by a day.
That's true, but it's more true for some companies than others. A lot of big companies cross time zones.
I suppose "unplanned outage minutes" would be a better metric. I still expect most companies to be worse on that metric. Most companies are just bad at running services reliably, especially when it's not their core business.
My small US gov org has been pushing more of our infra into centralized Microsoft services. The rationality seems to be that outages would be worse if we managed our own stuff. Nobody trusts each other and choosing Microsoft is an easy way to shift the blame. Nobody ever gets fired for choosing Microsoft and any issues they have must be justified.
With Europe now being fully woken up and still no emails working alongside Copilot this seems to be a rather more destructive issue then the last ones which mostly were solved at night and lasted ~8hours max in my memory.
We will also discuss and see how to failure proof our infrastructure against Microsoft outages. We made the mistake to base our VPN 2factor on Email. So now no VPN for anyone.
Keeping your own server(s) up is work that takes time and costs money. Microsoft is probably still cheaper than that,[citation needed] and has more resources to keep everything running than even a medium-sized government. So it is understandable.
Yes. M365 gives "free", secure emails in a scalable, hands-off way. The downside is it practically being a black box, making email issues undebuggable.
In a sense you're also paying for delegating responsibility. You can put the blame on Microsoft. Same way companies (internal project sponsors) pay consultants to take the blame if a projects fails.
It's been a while (~10y) since I administered windows servers, but I recall they were much harder to automate w.r.t. maintenance. So Microsoft created M365 to fill that niche and have organizations offload operational cost to them.
With a linux fleet can run via SSH/ansible and you can patch thousands of machines programmatically in minutes and control the deploy at a fine level. With Microsoft/Windows it often involves/d logging in to the one AD server where everything runs, back it up via 3rd party software, apply the patches and hope it doesn't break, and if it does, call MS support, because rolling back is really your last resort (I feel it's become better since 10y ago, but I don't have hands-on experience anymore).
Our company runs on-prem Exchange since 2019, and... well... it's a PITA to work with (Legacy Outlook/New Outlook), we downgraded from M365 and it was really a step down in terms of features and reliability. OTOH I guess it's keeping us afloat today when M365 is having a really bad day.
There isn't much if anything that can't be automated via Powershell and other methods. At this kind of scale, I would highly doubt that any of the servers running Exchange for EO are ever logged in interactively.
You can run your own email easily and cheaply, but if you wan to run with a team big enough and experienced enough to recover quickly if there's a problem it gets expensive fast.
If everyone had their own servers, either they're centrally enough managed that they aren't really independent servers or the management overhead and chaos from version drift etc. creates a mess that overall lowers the reliability of the entire system. Sure, you won't have concentrated outages like this but your instance's overall uptime is likely to be lower.
It's useless though. One passkey max. And login still requires email and there's no button on login page for passkey login. And it can't be f*ing removed. It's a joke. A trillion dollar company, lol.
I think in the future, we need stable, minimal implementations of services: Minimal because a small attack surface is crucial in the age of AI and slop coding/attacks.
Minimal also because core services, like the Unix philosophy, do one thing and do it well. We don't need big tech breaking core services by continuously pushing new 'features' (slop, surveillance-ware, ad trackers, etc.).
The future demands small, stable core services, and mail service should be a small core service commodity that has been defined and solved (protocol wise) a long time ago already - we only need to make it an accessible commodity for all without the big tech entanglement.
The problem is that making a business out of "less" is hard. Especially when dealing with the hellscape that is "enterprise sales". Both the marketing people and the customer love piling on additional features!
Two decades ago I was at a startup that moved from a Linux email solution to a Microsoft one. Why? For the CEO's blackberry and calendar integration.
There are loads of FOSS email solutions and providers. There are fewer "email+calendar" providers, and there are even fewer "email+calendar+desktop logon SSO" providers.
Running a "simple stable core service" at this sort of scale is far, far harder and more complicated than you understand. Even responding to ICMP ping requests at AWS/GCP/Azure scale requires custom solutions. I've seen very many near-misses in high profile global services due to the complexity of running at this scale that have nothing to do with the causes you're assuming here.
if this winds up publicized as update-related, i would question the operational decisions taking place within Microsoft.
first - were the updates fully tested in a legitimate test environment? (robust mirror with proper configuration and data) or were they "modeled" on a digital twin?
second - did any canary deployment precede the full rollout which seems to have a performance/capacity issue that borks (highly technical term) servers under load?
third - regardless of the answers for the first two, given their release quality history, should someone be suggesting "hey, this could be an impactful change, we should roll it during a quiet period like 2am Sunday" to mitigate customer impact and provide a longer restoration window option?
or has Microsoft simply asked OpenAI to manage it all...?
134 comments
[ 4.5 ms ] story [ 88.7 ms ] threadone of the all time tech gossip blog posts
Of course I haven't received a single email because every important email is sent via Microsoft but I still feel superior because I'm not affected
FreeBSD(ZFS+jails old but gold) + postfix + dovecot.
And yes uptime (with short outages(updates etc)) is not that important for a mail-server, since the sender re-transfers in intervals (mine is 5min the 8 then 15 then 60 then 180 then something up to 2days)
So the uptime for the service is 100% (maybe 99.999) over the last 8 years.
https://mail-tester.com/
and
https://mecsa.jrc.ec.europa.eu/ (offline atm Azure??)
True, i have some hetzner (not vm's but metal), and Vultr (just vm's), i just had one bad IP until now (i just changed it instead of trying to get it removed from a blacklist)
It's not, it's easy. We've been doing this stuff for years.
Doing it at billion user scale is hard. You are not google.
Sending out is harder but that 1% when I send something I need it delivered.
Hosting e-mail for friends and family is harder and mostly what I would call stupid.
I guess software is still not solved, I'll let Dario know
Because "personal outlook" at https://outlook.live.com/mail has been working this whole time.
Because I know people here are devs but you guys dont know IT lmao
Thank you!
As long as it works its good for them.
You can reduce the blast radius of outages at the cost of increasing the frequency a lot.
Malice or gross incompetence isn't really required to break it, that's its natural state. You need high levels of competence to get it to work
The timing is suspicious but could indeed be a coincidence.
No need for conspiracy here.
Truly next-level corporate speak.
I hope this sort of stuff pushes us to a distributed model rather than a centralised mono-provider one.
IMHO a bigger concern is that Russia hacked Microsoft in 2022 and China hacked Microsoft in 2023. Even that wasn't enough to scare companies off MS.
Basically, MS customers are locked in, the cost of migrating to anything else is huge, and there's a "no-one gets fired for buying MS" herd mentality, so MS has little to lose no matter how hard their product fails, just like Cloudstrike.
There's also the question of what alternative would really be better, or at least more reliable. My impression is that Google's infrastructure and engineering are genuinely better, but I'm recently ex-Google so take that with a grain of salt.
We've had more downtime from all our SaaS and cloud services in the last year than the 20 years before when self hosting.
Email is not a particularly easy thing to reliably host and secure for a sizable organisation, especially if its not your main focus.
If I lose a whole day, we lose all the productivity for that day and delay countless upstream and downstream efforts by a day.
The distribution of those minutes matters.
I suppose "unplanned outage minutes" would be a better metric. I still expect most companies to be worse on that metric. Most companies are just bad at running services reliably, especially when it's not their core business.
We will also discuss and see how to failure proof our infrastructure against Microsoft outages. We made the mistake to base our VPN 2factor on Email. So now no VPN for anyone.
The fact that the government can get taken down by a central server is mental.
Is it some sort of spying thing? Incompetence?
With a linux fleet can run via SSH/ansible and you can patch thousands of machines programmatically in minutes and control the deploy at a fine level. With Microsoft/Windows it often involves/d logging in to the one AD server where everything runs, back it up via 3rd party software, apply the patches and hope it doesn't break, and if it does, call MS support, because rolling back is really your last resort (I feel it's become better since 10y ago, but I don't have hands-on experience anymore).
Our company runs on-prem Exchange since 2019, and... well... it's a PITA to work with (Legacy Outlook/New Outlook), we downgraded from M365 and it was really a step down in terms of features and reliability. OTOH I guess it's keeping us afloat today when M365 is having a really bad day.
You can run your own email easily and cheaply, but if you wan to run with a team big enough and experienced enough to recover quickly if there's a problem it gets expensive fast.
Google Workspace isn't going to be any different.
LLM generated code and/or configs?
- No emails, calendar, file sharing, etc. - Claude Code broke as well because I cannot login back. - Multiple services I use with 365 are also broken.
The future demands small, stable core services, and mail service should be a small core service commodity that has been defined and solved (protocol wise) a long time ago already - we only need to make it an accessible commodity for all without the big tech entanglement.
Two decades ago I was at a startup that moved from a Linux email solution to a Microsoft one. Why? For the CEO's blackberry and calendar integration.
There are loads of FOSS email solutions and providers. There are fewer "email+calendar" providers, and there are even fewer "email+calendar+desktop logon SSO" providers.
first - were the updates fully tested in a legitimate test environment? (robust mirror with proper configuration and data) or were they "modeled" on a digital twin?
second - did any canary deployment precede the full rollout which seems to have a performance/capacity issue that borks (highly technical term) servers under load?
third - regardless of the answers for the first two, given their release quality history, should someone be suggesting "hey, this could be an impactful change, we should roll it during a quiet period like 2am Sunday" to mitigate customer impact and provide a longer restoration window option?
or has Microsoft simply asked OpenAI to manage it all...?