158 comments

[ 1.9 ms ] story [ 97.2 ms ] thread
A former Azure Core engineer’s 6-part account of the technical and leadership decisions that eroded trust in Azure.
What's your assessment of AWS and GCP? Do you think it's likely they suffer from some of the same issues (eg the manual access of what should be highly secure, private systems, the instability, the lack of security)?
Why do you speak about yourself in the third person?

Also, after this:

https://news.ycombinator.com/item?id=20341022

You continued to work at Microsoft and now there is this takedown?

I'm no friend of MS (to put it very mildly) but it seems to me your story is a bit inconsistent as well as the 7 year break between postings.

It's a nice read. Thank you for sharing this.

> Microsoft, meanwhile, conducted major layoffs—approximately 15,000 roles across waves in May and July 2025 —most likely to compensate for the immediate losses to CoreWeave ahead of the next earnings calls.

This is what people should know when seeing massive layoffs due to AI.

I honestly thought this was one of the weaker points of the article.

The OpenAI deal almost certainly related purely to GPU capacity, which had little to do with the article. The layoffs would have happened regardless.

IMO - churn, and generalization is the root cause. Engineers are thrown on projects for a year with little prior experience, leave others to pickup the pieces, etc. There's no longer a sense of ownership, and I'm sure the recent wave of layoffs isn't helping with this.

What a fascinating view into how the sausage is made
> The direct corollary is that any successful compromise of the host can give an attacker access to the complete memory of every VM running on that node. Keeping the host secure is therefore critical.

> In that context, hosting a web service that is directly reachable from any guest VM and running it on the secure host side created a significantly larger attack surface than I expected.

That is quite scary

(comment deleted)
Scary is the understatement of the day. I can't imagine the environment where someone think that architecture is a good idea.
Like, what did the OP expect?
This is well documented: https://learn.microsoft.com/en-us/azure/virtual-machines/ins...

Why would an Azure customer need to query this service at all? I was not aware this service even exists- because I never needed anything like it. AFAI can tell, this service tells services running on the VM what SKU the VM is. But how is this useful to the service? Any Azure users could tell how they use IMDS? Thanks!

Instead of zero trust, it is 110% trust.
Well I have zero trust in Microsoft, so they've achieved that at least.
It is kind of a fundamental risk of IMDS, the guest vms often need some metadata about themselves, the host has it. A hardened, network gapped service running host side is acceptable, possibly the best solution. I think the issue is if your IMDS is fat and vulnerable, which this article kind of alludes to.

There’s also the fact that azure’s implementation doesn’t require auth so it’s very vulnerable to SSRF

"For fiscal 2025, Microsoft CEO Satya Nadella earned total pay of $96.5 million, up 22% from a year earlier." -CNBC.com

and

"I also see I have 2 instances of Outlook, and neither of those are working." -Artemis II astronaut

Microsoft’s annual revenue ($245Bn) is 2.5x Tesla ($95Bn), and Musk was angling for a trillion dollar compensation package.

Artemis II astronaut was piloting a spaceship not a Tesla.

Makes you think.

Nadella gets the money for getting Outlook onto the ship. Having it actually work would have been a bonus, sure, but it's not the goal.
Title: How Microsoft Vaporized a Trillion Dollars
As an investor, this is exact how I feel. Everything was skyrocketing until OpenAI “diversified” mid-2025. The company’s market value has dropped more than 1 trillion since late October 1025, so the title is factual. You can rightfully argue and be skeptical about the link I make, but not about the numbers :)
What are we reading here? These are extraordinary statements. Also with apparent credibility. They sound reasonable. Is this a whistleblower or an ex employee with a grudge? The appearance is the first. Is it? They’ve put their name to some clear and worrying statements.

> On January 7, 2025… I sent a more concise executive summary to the CEO. … When those communications produced no acknowledgment, I took the customary step of writing to the Board through the corporate secretary.

Why is that customary? I have not come across it, and though I have seen situations of some concern in the past, I previously had little experience with US corporate norms. What is normal here for such a level of concern?

More, why is this public not a court case for wrongful termination?

Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience?

Yes it is that unreliable. Even when given free credits, I would rather pay for the offerings from Amazon/Google.
He is, I think, Swiss, perhaps a cultural difference?
Azure is when you have a different version of the same product/api in each region.
I recall seeing some pretty damning reports from a security pentester that was able to escape from a container on Azure and found the management controller for the service was years old with known critical unpatched vulnerabilities. Always been a bit sceptical of them since then
Large orgs make decisions that prioritize short-term metrics over long-term quality all the time and nobody tracks whether those tradeoffs actually paid off. The decision to ship fast and fix later sounds reasonable in a meeting setting until articles like this surface and the reality comes through clearly.
Yeah I thought that was extreme. An engineer going to the board of any corporation let alone Microsoft is not normal or customary IME. That could explain why they got no response.
“customary” referred to the path through the Secretary, as opposed to writing directly to members. Besides that, depending on the nature of the communication, if everything fails, you may need to be sure you talk to people who will unconditionally put the best interests of the company ahead of any other consideration. The Board is one such group. See what Boeing did with the report of the mechanic who saw flaws in the 737 MAX’s door plugs. Was that worthy of a letter to the CEO, then the Board if no reaction? Or just talk to your dismissive manager and let the planes crash? I made a judgment call, which I entirely own.
>Is Azure really this unreliable? There are concrete numbers in this blog. For those who use Azure, does it match your external experience?

IME, yes.

I'm currently working as an SRE supporting a large environment across AWS, Azure, and GCP. In terms of issues or incidents we deal with that are directly caused by cloud provider problems, I'd estimate that 80-90% come from Azure. And we're _really_ not doing anything that complicated in terms of cloud infrastructure; just VMs, load balancers, some blob storage, some k8s clusters.

Stuff on Azure just breaks constantly, and when it does break it's very obvious that Azure:

1. Does not know when they're having problems (it can take weeks/months for Azure to admit they had an outage that impacted us)

2. Does not know why they had problems (RCAs we're given are basically just "something broke")

3. Does not care that they had problems

Everyone I work with who interacts with Azure at all absolutely loathes it.

I am sort of confused how NDA and such agreements employees sign would allow for an employee to post such an article without being sued by Microsoft?
As a former MSFTy it does sound weird to me too. I didn’t see what Axels level was but a lot of people work for Microsoft and not many of them can expect to email the CEO and get a response. It seems a bit like a crash out, not the first I’ve seen levied at Azure, won’t be the last. They probably think it’s a mental health episode, if you’re an important CEO crazy people will email you all the time and the staff probably filter them out before they see it. Also this is a lot of internal gossip, I would be worried that airing this publicly would impinge on future career opportunities, even healthy orgs would appreciate some discretion.

I’m sure everything he said is completely true, Azure is one of the few tech stacks I refuse to work with and the predominant reason I left.

If you’ve joined an org and nothing works the reason is usually that the org is dysfunctional and there is often very little you can do about it, and you’re probably not the first person who’s tried and failed at it.

> What are we reading here? These are extraordinary statements. Also with apparent credibility.

I left Microsoft in 2014. Already back then I could see this sort of stuff starting to happen.

The Office Org was mostly immune from it because they had a lot of lifers, people who had been working on the same code for decades and who thought through changes slowly.

But even by 2014 there were problems hiring developers who knew C++, or who wanted to learn it. COM? No way. One one team we literally had to draw straws once to determine who was going to learn how to write native code for Windows.

It wasn't even a talent thing, Windows development skills are a career dead end outside of Microsoft. They used to be a hot commodity, and Microsoft was able to hire the best of the best from industry. Now they have to train people up, and Microsoft doesn't offer any of the employment perks that they used to use to attract top talent (Seattle used to be a low CoL area, everyone had private offices, job stability).

When I started at Microsoft in 2007, the interview bar included deep knowledge of how computers worked. It wasn't unusual to have meetings drop down to talking about assembly code. Your first day after orientation was a bunch of computer parts and you were told to "figure out how to setup your box".

Antivirus wasn't mandatory. The logic was if you got a virus, they made a mistake hiring you and you deserved to be fired.

When your average developer can go that deep on any topic, you can generally leave engineers well enough alone and get good software.

What I meant is that it’s customary to write to the Board through the Secretary as opposed to write directly or through some other channel.
I notice the title mentions the author is a former employee but he never mentions the terms on which he left.
What an epic takedown.

Microsoft should have promoted this guy instead of laying him off.

Did Microsoft really lose OpenAI as a customer?

The answer to your question is in the public releases. MS went from primary partner (under ROFR) to one of the options. They retain IP rights and API hosting, although in recent weeks we learned that OpenAI was planning a workaround with AWS and Microsoft said they might sue them for that. So the happy marriage is over, it’s more like a custody battle now: https://www.reuters.com/technology/microsoft-weighs-legal-ac...
Any complex system - and these cloud systems must be immensely complex - accumulate cruft and bloat and bugs until the entire thing starts to look like an old hotel that hasn’t been renovated in 30 years.
It’s not inevitable. Absolutely this is true without significant effort, but if you’ve been around the traps for long enough (in enough organisations), you get to see that the level of quality can vary widely. Avoiding the mud-pit does require a whole org commitment, starting from senior leadership.

This story is more interesting, in my opinion, in how quickly things devolved and also how unwilling the more senior layers of the org were to address it. At a whole company level, the rot really sets in when you start to lose the key people that built and know the system. That seems to be what’s happening here, and it does not bode well for MS in the medium term.

This is an insanely blunt look into some serious issues with microsoft.
(comment deleted)
I had the misfortune of having to use Azure back in 2018 and was appalled at the lack of quality, slowness. I was in GitHub forums, helping other customers suffering from lack of basic functionality, incredible prices with abysmal performance. This article explains a lot honestly.

Google’s Cloud feels like the best engineered one, though lack of proper human support is worrying there compared to AWS.

> Google’s Cloud feels like the best engineered one, though lack of proper human support is worrying there compared to AWS.

Also the lack of locations in general. GCP's fleet is tiny compared to both AWS and Azure

GCP's support sucks compared the quality on AWS. On everything else I find GCP very pleasant.
The post is so dramatized and clearly written by someone with a grudge such that it really detracts from any point that is trying to be made, if there is any.

From another former Az eng now elsewhere still working on big systems, the post gets way way more boring when you realize that things like "Principle Group Manager" is just an M2 and Principal in general is L6 (maybe even L5) Google equivalent. Similarly Sev2 is hardly notable for anyone actually working on the foundational infra. There are certainly problems in Azure, but it's huge and rough edges are to be expected. It mostly marches on. IMO maturity is realizing this and working within the system to improve it rather than trying to lay out all the dirty laundry to an Internet audience that will undoubtedly lap it up and happily cry Microslop.

Last thing, the final part 6 comes off as really childish, risks to national security and sending letters to the board, really? Azure is still chugging along apparently despite everything being mentioned. People come in all the time crying that everything is broken and needs to be scrapped and rewritten but it's hardly ever true.

The grudge is simple and doesn't detract one thing from a very well articulated blog: you do you job as an engineer of pointing out problems, even proposing solutions, and they fire you for doing exactly the job. It's infuriating enough just from reading it, idk how you can't see any legitimacy on what the guy is complaining. You have your right of free speech to complain about shitty jobs if you want, there's no honor bound to maintain silence here.
Do you contest the fact that Microsoft royally fumbled OpenAI out of sheer incapability of providing what's supposed to be its core business despite having all deals in its favor? Because that's the most damning validation against Azure in recent times.
This reads pretty bad, and I believe it was. I worked on (and was at least partly responsible for) systems that do the same thing he described. It took constant force of will, fighting, escalation, etc to hold the line and maintain some basic level of stability and engineering practice.

And I've worked other places that had problems similar to the core problems described, not quite as severe, and not at the same scale, but bad enough to doom them (IMO) to a death loop they won't recover from.

I don't know if any of this is true, but as a user of Azure every day this would explain so much.

The Azure UI feels like a janky mess, barely being held together. The documentation is obviously entirely written by AI and is constantly out of date or wrong. They offer such a huge volume of services it's nearly impossible to figure out what service you actually want/need without consultants, and when you finally get the services up who knows if they actually work as advertised.

I'm honestly shocked anything manages to stay working at all.

The first couple of paragraphs felt like a parody of a guy who goes to a diner and gets upset the waitress doesn’t address him as Dr.

It didn’t get any better.

(comment deleted)
Some previous colleague of mine has to work with Azure on their day to day, and everything explained in this article makes a lot of sense when I get to hear about their massive rantings of the platform.

12 years ago I had to choose whether to specialize myself in AWS, GCP or Azure, and from my very brief foray with Azure I could see it was an absolute mess of broken, slow and click-ops methodology. This article confirms my suspicions at that time, and my colleague experience.

So this is why GitHub is having so many problem…
The personal account makes a lot of sense, although I could easily see why the OP was not successful. Even if you are an excellent engineer, making people do things, accept ideas, and in general hear you requires a completely different skill altogether - basically being a good communicator.

The second thing is that this series of blog posts (whether true or not, but still believable) provides a good introduction to vibe coders. These are people who have not written a single line of code themselves and have not worked on any system at scale, yet believe that coding is somehow magically "solved" due to LLMs.

Writing the actual code itself (fully or partially) maybe yes. But understanding the complexity of the system and working with organisational structures that support it is a completely different ball game.

from part 2:

> Worse, early prototypes already pulled in nearly a thousand third-party Rust crates, many of which were transitive dependencies and largely unvetted, posing potential supply-chain risks.

Rust really going for the node ecosystem's crown in package number bloat

(comment deleted)
I've said it before and I'll say it again. I'm glad rust has good package management I really am. However given that aspect, it ends up forming a dependency heavy culture. In situations like this it's hard to use dependencies due to the amount of transitive dependencies some items pull in. I really which this would change. Of course this is a social problem so I don't expect a good answer to come of this....
(comment deleted)