Ask HN: Azure has run out of compute – anyone else affected?
Last week we at n8n ran into problems getting a new database from Azure. After contacting support, it turns out that we can’t add instances to our k8s cluster either. Azure has told they'll have more capacity in April 2023(!) — but we’ll have to stop accepting new users in ~35 days if we don't get any more. These problems seem only in the German region, but setting up in a new region would be complicated for us.
We never thought our startup would be threatened by the unreliability of a company like Microsoft, or that they wouldn’t proactively inform us about this.
Is anyone else experiencing these problems?
358 comments
[ 2.7 ms ] story [ 296 ms ] threadInstead of providing you with a list of the resources they do have, you have to play this weird game where you ask for specific instances in specific regions and then within several hours someone emails back to say yes or no.
If it’s no, you have to guess again where you might get the instance you want and email them again and ask.
I envisage going to an old shop, and asking the shopkeep for a compute instance in a region. He hobbles out the back, and after a long delay comes back and says “nope, don’t have no more of them, anything else you might want?”.
It’s surprising this how it works. Not the auto scaling cloud computing used to bring to mind.
I just Googled it. Gartner estimated $125 billion.
https://www.fiercetelecom.com/telecom/cloud-and-colocation-d...
And sometimes, that is hard. I've had Azure support not able to understand what quota they need to raise / what quota is being requested. I had to at least link them to their own documentation on it… (partly the confusion is that quota support tickets allow selecting the quota as a piece of metadata on the ticket, but only for some quotas, and of course, mine was for one of the ones not listed. Why they don't just list all of them is anyone's guess.)
you have to do this for every single instance type they have, can't even experiment or test other instance types cause its too much trouble to get quota
> Yes it’s weird that you have to ask them for instances which some actual physical person looks at your request, thinks about it and says yes or no to.
and low quota is low, like 10 cpu, so start a 2 node k8s cluster with 8cpu each? nope, go request quota increase
21st century man…. it’s coming.
Computers don't fix everything. They just allow you to f*ck up bigger, harder, and faster, usually in the most banal way imaginable.
I've never done K8 on Azure, but my understanding is that Azure is pretty good about coordinating between your own datacenter running windows and Azure. Maybe you can spin up some windows boxes in a cheap datacenter to make it work?
I sure hope so, as a German company
Unlike GCP and Azure, all AWS regions are (were) partitioned by design. This "blast radius" is (was) fantastic for resilience, security, and data sovereignty. It is (was) incredibly easy to be compliant in AWS, not to mention the ruggedness benefits.
AWS customers with more money than cloud engineers kept clamoring for cross-region capabilities ("Like GCP has!"), and in last couple years AWS has been adding some.
Cloud customers should be careful what they wish for. If you count on it in the data center, and you don't see it in a well-architected cloud service provider, perhaps it's a legacy pattern best left on the datacenter floor. In this case, at some point hard partitioning could become tough to prove to audit and impossible to count on for resilience.
UPDATE TO ADD: See my123's link below, first published 2022-11-16, super helpful even if familiar with their approach.
PDF: https://docs.aws.amazon.com/pdfs/whitepapers/latest/aws-faul...
There are aws region partitions - general, china, us gov cloud (public), us gov secret and us gov top-secret.
Inside a partition, there can be some regions that are opt-in - see https://docs.aws.amazon.com/general/latest/gr/rande-manage.h...
My understanding is that opt-in regions are even more isolated inside a specific partition for partition-global services like IAM and maybe some other stuff.
Could you elaborate on this a little? We use AWS, but are evaluating OCI for certain (very specific) cases, and I'll love to know what questions to ask for comparison purposes.
Here is how partitioned/isolated OCI is by design:
https://www.wiz.io/blog/attachme-oracle-cloud-vulnerability-...
While that's fixed, it speaks volumes to the architecture. Very little has changed since 2018: https://www.brightworkresearch.com/how-to-understand-the-pro...
As noted there, I'd argue OCI is more akin to Softlayer/Bluemix than to GCP, Azure, or AWS, but depending on your certain very specific cases OCI may still be appropriate.
> setting up in a new region would be complicated for us.
Sounds to me like you've got a few weeks to get this working. Deprioritize all other work, get everyone working on this little DevOps/Infra project. You should've been multi-region from the outset, if not multi-cloud.
When using the public cloud, we do tend to take it all for granted and don't even think about the fact that physical hardware is required for our clusters and that, yes, they can run out.
Anyways, however hard getting another region set up may be, it seems you've no choice but to prioritize that work now. May also want to look into other cloud providers as well, depending on how practical or how overkill going multi-cloud may or may not be for your needs.
I wish you luck.
Without knowing the details about your services and infrastructure, it's hard for me to know what's involved in going multi-region now. Are you sure it's such a a gargantuan effort? I would've thought one person working full-time on this for a week or two would be enough, but again I don't know the details of your setup.
One option would be to pay a consultant who is an expert in Azure/cloud stuff to come in and help. May not be cheap, but could be a lot better and quicker for you and better for the business, especially if none of you are really big experts in Azure.
I've been here before (I think)...had to wear many hats and scramble to make sales, build the tech, act as de facto DevOps person even without a lot of experience doing it, etc. That is the way, but stuff happens.
Happy to chat about specifics if you want to bounce ideas off of me or go through your particular situation. Can't promise I'll have concrete advice, but happy to talk it through.
Very poor position to be in, apparently this happened in azure UK recently too.
Let the scar you get from this is be a learning experience, hopefully you will not fall into the same trap again to trust this company.
In my career I'm in a place where anyone suggesting I do work on Azure gets an instant doubling of my asking day-rate and I really hope the will be put off and find another victim for this gig.
That said, another learning experience would be to use terraform or something (tbh for azure the only sane thing is terraform, ARM templates are just garbage). Having terraformed your one region switching to the other would be much easier, tho not trivial.
I'm totally on board with the idea of being scrappy and taking shortcuts in order to get to PMF as soon as possible. However, it seems the proof is in the pudding here. If you can't service customers due to lack of compute resources, you can't get to PMF.
Also, yes there are certain infrastructure and network topologies that would absolutely be overkill for a young startup. I don't think multi-region is one of those things. I don't have experience with Azure directly, but on every other cloud providers, going multi-region is not something that requires huge amounts of time or resources. You just need to be mindful of it from the outset. And if you decide not to be, then at least be intentional and conscious about the risk and have a plan in place for what happens when you get bit by deciding not to go multi-region.
*privacy invasive that GDPR is explicitly set up to make harder, so duh
Also, lots of companies assert GDPR compliance via magical thinking. They most often are wholly wrong. Shopify can say whatever they want, but there’s no certification body.
Source: I’m the person who evaluates and builds compliance systems for a range of services you almost definitely use.
Sounds like customers are coming in thick and fast.
If this is the dynamic and the company can't spare a few weeks to solve it, something has gone seriously wrong in a very interesting way.
If you rely on Kubernetes for orchestration and have minimal cloud API dependency, it may be worth that evaluating this option.
Also, do you have a TAM associated with your account? Are you just going through regular support channels? Can they deliver different instance types (not sure what the Azure parallel is), can they deliver short term capacity, etc?
I would try to push Microsoft more here. It's not like they've stopped on-boarding new customers into that region right? What happens if you create a new account in that region?
If you don't you're at a big disadvantage.
Similarly, just keep trying to change the size. Often it’ll go through when someone else decommissions something.
This is doubly worthwhile as if this stumble kills the startup (it can happen) this will be excellent experience to take to the next employer :)
The way to get more from most cloud is by becoming a partner, not just a customer. And the way to do that is increase dependency and usage.
(As an aside I also agree that multi cloud from the get go is a YAGNI violation. Just keep in the back of your mind “could we have an alternative to this?” when using your provider’s proprietary features.)
Just having the plan is already expensive enough.
None are ideal.
This is how the iphone was able to nuke windows phones which were designed to meet the needs of IT
Those MS products have an “IBM mainframe” problem. New businesses won’t choose them.
That’s why I say “why do I want them?” If I was starting a new business I’d have no reason to use them.
AD = identity, access, privileging, SSO
In my mind they’re legacy business products.
Sure, most businesses use them, but I don’t necessarily believe that is a forever thing. At one point most businesses had mainframes.
I'm going to guess that the 5% that don't use Azure/Office 365 represent newer entrants to the Fortune 500.
71% of Fortune 500 companies use mainframes [3], and yet they are considered a dead technology with essentially no future. Do you know anyone or anyone who knows anyone who learned how to develop on mainframes in college in the current millennium? It sure wasn't part of my CS curriculum!
Small businesses represent almost half of US economic activity [1] and represent 99.7% of firms with paid employees [2]
[1] https://advocacy.sba.gov/2019/01/30/small-businesses-generat...
[2] https://cdn.advocacy.sba.gov/wp-content/uploads/2021/12/0609...
[3] https://www.precisely.com/blog/mainframe/9-mainframe-statist...
Multi-cloud is really not a big deal. Main nuisance is billing differences, followed by slight variations in e.g. Terraform config.
Multi-cloud should only be for mission critical infrastructure. Very little infrastructure is mission critical. Most other use cases can be temporarily wallpapered over with an "Under maintenance" page unless there's a good reason otherwise.
Multi-cloud introduces more risk than it prevents. Which is why things like simulated failovers and BCP testing is constantly required.
I don't want to be snarky, but when large service providers like AWS have their own crossregion downtime because one snowflake of a service in us-east-1 is down, I kind of dismiss the virtue signaling of high resilient multi-(az/region/cloud) ever existing in practice.
If you can somehow have a separate database per region/cloud, sure, I can understand that, but if you have to shard your database across many clouds, I'd dread having to tame such a beast, especially within a startup.
So you're saying it's impossible to improve reliability from 97% to 99% because you can never make it to 100%.
Multi-AZ and multi-region add complexity and cost much more quickly than they add reliability.
Sometimes it is worth it. Sometimes it is not.
Everyone? That's not going to help.
You will be threatened by your own unreliability of building something that's dependant on one region or one cloud.
I just watched the second video on the page, and it does discuss multi-region a bit.
At least on AWS you typically can find capacity (outside of accelerators) by being flexible on instance types (C, M, R), instance sizes, and availability zones. Sounds like this region OP is in for Azure is constrained such that even this advice doesn’t work.
I suspect that the skills for real HA are atrophying because for 99% of the people multi-AZ is enough and most of the AWS stuff supports multi-az automagically.
The problem with multi-region is that it means configuration, and there are probably lots of services that you can't actually configure to be multi-region. Cognito is one off the top of my head. It looks like the various aurora flavors do multi-region, but what about Neptune? SQS? API Gateway? AWS Lambda? MediaLive?
Maybe you can hide all that behind DNS failover, maybe you can't.
Real multi-region is basically means going back to old-school HA, and that was hard to do when it was your data centers. On AWS it'll be even harder.
That isn't to say it's not possible, it's just a tremendous amount of work.
I mean really, if us-east-1 is down 80% of the internet is screwed...so from an expectations point of view does HA of your particular service matter if that happens? Even for a financial outages happen.
Once you have enough people it might be worth it. For a non mission critical startup? No fucking way.
Everything is for sure until it’s not.
This has vast benefits for agility and fast development when developers are not always fighting the build system and have a "no fear" attitude about deployment.
If you have that, you can build a system in another region and be able to migrate wholesale to another region with more capacity and not be particularly concerned about the general problem of coordinating the service across multiple regions at the same time.
For some clouds that seem to be run on a manual process (IBM, Oracle) that would be expected, since they're sort of clunky. For other places (Rackspace, etc) it would uncommon. For a major provider like Azure, well, it's bizarre. I mean, the whole point of cloud is that it's all-you-can-eat.
You would think that this would be something they would advertise/talk about up-front. But who would sign up if that was disclosed?
Yikes, this is totally the first thing you need to come to expect when working with MSFT.
Building on cloud requires a lot of trade offs, one being a need for very robust cross-region capability and the ability to be flexible with what instance types your infrastructure requires.
I’d use this as a driver to either invest in making your software multi regional or cloud agnostic. Multi regional will be easier. If you’re already on k8s you should have a head start here.
Ideally you have a script that goes from credentials to the service to a complete working instance.
The major cloud services are expensive. This extra cost is supposed to provide for cloud services' high level of flexibility. Running out of capacity should be a rare event and treated as a high priority problem to be fixed asap.
Without the ability to rapidly and arbitrarily scale, they're just overpriced server farms.
They can't magic chips into existence, but leaving a major region like Germany high & dry for almost half a year sounds like planning went wrong frankly. If it were a matter of chips I would have thought on a 3+ month timescale they can steal a few from another region that has a bit of fat
That's exactly what a cloud is. It's someone else's datacenter with an API.
I mean that's what cloud is (outsourced server farm). Sure they also offer services on top, but that's mostly because they want to lock you in, and can charge more for, so it's a win win for them.
And there is no magic here, someone has to get the chips, build servers and connect them to network. And while they will often overbuild for capacity, they will never do it to a degree, where they can't run out, because that would be way to expensive and not financially viable.
I don't think any cloud will ever be able to guarantee to never run out of resources.
I agree with this, but clearly there's a disconnect between how often people expect these kinds of issues and how often they actually happen. The whole point of the cloud is you pay a premium for the added flexibility. If it turns out that flexibility isn't there when you need it then maintaining your own servers becomes a lot more attractive.
I also sort of suspect the spot market is less robust there. Lots of Azure is lift and shift on premises workloads, and those aren't using spot. Without people using spot, it's even harder to have spare capacity...
The strategy they helped us arrive at was two-pronged:
1. Pre-launch all needed infrastructure. Yes, for all their "cloud scale", it was actually suggested that we preallocate all of our servers the week before, rather than rely on autoscaling.
2. Order capacity reservations for all of those instances (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-capa...). This ensure that, if any of those instances go bad, we'd be able to relaunch them without going to the back of the line, and finding out that there was no more compute capacity available.
List what you do you have available so we can choose.
Do not force users to randomly guess and be refused until eventually finding something available.
I don’t know about price point. Dedicated servers can be cheaper than cloud in many cases, if you have the appropriate know-how, and the cloud business is very profitable for a reason.
It’s infuriating that AWS doesn’t have an API that returns a list of AZs with available inventory for a given instance type.
There’s lots of providers apart from AWS/Azure/GCP.
Or buy a machine and put it in your office.
Self hosting can often be cheaper and more available and probably faster than using a cloud.
I can see why they wouldn't want to do this.
It's not the exact metric but you can find which have more availability without knowing the exact number (which is constantly changing anyway)
I don't always care if you give me a E8_v4 or a D8 instead, just give something. With all the 100 of variants of VMs that are available, finding an exact match is obviously an unnecessary constraint. Maybe they already simulate this behind the scenes, I don't know, though given the sizes are advertised with HW capabilities I'd imagine they can't really simulate a v4 using a v5 and vice versa.
Only place I've seen compute be treated this fluidly is in Container instance, which is a bad choice for many many other reasons.
(You could depend on another startup with no revenue).
Now, they could go the GDPR/Cookies route and prompt absolutely every user on pageload, but doing so would annihilate the purpose of the law into monotonous smithereens, just as it did with Cookies. Good on them for defaulting to the "more secure" mode, but yes this is a potential consequence.
Happy to hear from any German amigos present if I've got something wrong. (But watch out... you might be putting HN at risk - their servers aren't (likely) in Germany!)
[1]: https://incountry.com/blog/which-german-data-privacy-laws-yo...
https://learn.microsoft.com/en-us/previous-versions/azure/ge...
In particular, GPU availability has been a continuing problem. Unlike interchangeable x64 / arm64 instances with some adjustments based on the new core and ram count... if no GPU instances are available then I simply cannot run the job. AMD's improved support has increasingly provided an alternative in some situations but the problem persists.
I recommend doing the work to make the business somewhat cloud agnostic, or at the very least multi-region capable. I realize this is not an option for some services that have no equivalent on other clouds but you mentioned databases and k8s clusters which are both supported elsewhere.
All cloud providers charge much, much more for GPUs than if you run a local machine.
Cloud GPUs are also a lot slower than state of the art consumer GPUs.
Cloud GPUs: much slower, less available, much more expensive.
Sure you could buy all that equipment but I’d wager it’s cheaper, more agile, and greater velocity from it being in the cloud
Local GPUs are a big up-front cost. But assuming that your workload is stable, in the long run I think local GPUs ends up being cheaper per-hour than cloud.
For startups, it doesn't make sense to make the up-front purchase, fine. But if you're optimizing for long-term (amortized) costs, I'd be curious if cloud is cost-effective.
But yes, If a single workstation can meet your gpu training needs then it’ll be cheaper with sufficient usage
However, lots of people only need those accelerators once in a while, so time sharing (aka cloud computing) makes a lot of sense and saves a ton of money overall. For FPGAs and some compute GPU applications, not having to handle support for your accelerators is also nice.
GPUs are better run close to your data. If you're training on-prem then your data needs to be on-prem too.
Terraform has different providers for each cloud provider and the code is not transferable any more than saying if you use Python to script your infrastructure it will be transferable.
The reason for Terraform, and it's a good one, is your Terraform-related tooling doesn't have to change, e.g. if you route all your infra change approvals through Terraform Cloud), and you can coordinate multi-service changes, e.g. update Auth0 infra to do X, then AWS to do Y.
The biggest advice I can give is 1. keep trying and grabbing capacity continuously, then run with more than what you need. 2. Explore migrating to another Azure region that runs less constrained. You mention a new region would be complicated, but it is likely much easier than another cloud.
1. https://www.zdnet.com/article/azures-capacity-limitations-ar...
... wait, what? How are they defining 'reserved'?
Dedicated capacity exists, but it’s different (compute reservation groups or dedicated hosts).
You can combine CRG/DH with RI for the desired effect, although IMO it’s a bit confusing.
(Azure employee)
The billing thing became more of the point as big AZ failures are so rare.
This is also a problem internally for Microsoft. GitHub and LinkedIn still operate in private datacenters due to Azure capacity issues