HN Explorer

Azure DevOps is down globally

100 points by vimwizard ↗ HN
https://status.dev.azure.com/

US Central Azure cloud is down also, must be related: https://azure.status.microsoft/en-us/status

54 comments

[ 2.7 ms ] story [ 113 ms ] thread
MrLeap
Been down hours. Those poor 9's
gmuslera
Those 9's are still there. Just that the decimal places were misinterpreted,
MrLeap
The decimal is going door to door, menacing the 9's.
wsdookadr
99.999999999999999999999999%

high availability

fault tolerant and resilient infrastructure

akokanka
Entire Azure been offloaded to WITCH beware.
[dead] WantonQuantum
[flagged]
imadethis
Please don't post ChatGPT spam here. If you're curious, the meaning of WITCH is

Wipro Infosys TCS Cognizant HCL

Commonly included is Accenture (India)

dheerkt
WITCH is the acronym for Indian tech consulting companies for WiPro, InfoSys, Tata Consultancy Services, C something, H something. They're stereotyped as being cheap and low-quality.

Not agreeing/disagreeing here, just stating author's intent.

Rapzid
I have noticed something along these lines in the GitHub repos. Tons of bugs piling up, issues closed out as "by design" where clearly is was not. Etc.

I honestly get the impression Azure is in decline, particularly around the .Net integrations which is their flagship ecosystem..

TeaBrain
I don't necessarily share this impression, but the teams that simply close items after being unable to sufficiently develop a solution, leading to the original issue raiser giving up after realizing that those who are supporting the issues are useless, are not helping with the public's impression of azure.
Rapzid
It's a culture issue. The culture of the teams they are.. Offshoring ownership of these projects to.

A lot goes into company and team culture, but I'm sure a lot here are familiar with the dynamics of outsourced development; offshore or not. Blame seeking, blame shifting, CYA, etc.. "by design"..

It's all on full display in Microsofts Azure GitHub repos.

xtracto
It's a matter of culture in outsourcing.

I'm a CTO in Mexico and outsource to companies in Mexico (insource modality) and it is evident outsourcing people don't give a danm about the product . Why should they? They get paid by the hour, and if the client goes bust after they are paid, another will bite.

leosanchez
Do you have a source for that? I couldn't find anything.
planewave
My web interface and pipelines appear to be down, but it seems like git operations are still working.
cgijoe
Guessing this is related: Minecraft is down, and possibly other games that use XBox Live Login: https://support.xbox.com/en-US/xbox-live-status

"You may not be able to sign-in to your Xbox profile, may be disconnected while signed in, or have other related problems. Features that require sign-in like most games, apps and social activity won't be available."

[deleted]
(comment deleted)
candiddevmike
GitHub Actions is also degraded, coincidence?

https://www.githubstatus.com/

For me, pipelines are queuing and not even scheduling jobs on self hosted runners.

planewave
No, from the documentation it seems they share a lot of infrastructure between GitHub Actions and Azure DevOps pipelines.
jacobwg
Not a coincidence, no.

From the "self-hosted" perspectively, interestingly what we're seeing at Depot (https://status.depot.dev/clyrvud6i57402igofm6jtb7id) is API requests to provision new runners or receive runner jobs are receiving rate limit errors, but without the regular rate limit status HTTP headers[1].

I imagine the GitHub API / Actions control plane is rather overloaded at the moment with the outage.

[1] https://docs.github.com/en/rest/using-the-rest-api/rate-limi...

blondin
very surprised by all the infrastructure hosted in us central. the page claims codespaces is functional but it clearly is not. it fails at loading vscode after bootstrapping the environment.
MrLeap
Is it surprising?

Dallas, STL, KC, Chicago all count as central. More or less equidistant for east/west coast markets. Seems like natural edge node coordinates along the scale ladder.

emunday_bbc
Feel free to use this free time to take a peek at Bitbucket Pipeline, now running up-to 8x faster ;)

https://bitbucket.org/blog/announcing-our-new-ci-cd-runtime-...

Come check out some of our cool new features while you're at it:

- https://bitbucket.org/blog/introducing-dynamic-pipelines-a-n...

- https://bitbucket.org/blog/custom-merge-checks-are-now-gener...

cr125rider
Great to see you guys engaging the community. Bitbucket may be 10 years behind GitHub, but the simplicity of pipelines is nice.
hypeatei
They're claiming only Central US is having issues on the cloud side. But, the portal itself is also having trouble loading pages and "global" services like identity management.
lukeschlather
It seems like Azure DevOps migrated to some new login mechanism recently which is visibly slower than the old one, and I was seeing random failures prior to today.
vimwizard
I have also noticed strange behavior that was not present before this perceived update, e.g. basic auth challenges and pop-up windows if I'm waking up my computer and still have azdo pages open.
raxxorraxor
I believe that was a change to auth from a few weeks ago and is affecting every MS service. Not sure though and it does work, but it doesn't work gracefully.
eBombzor
ADO vs Teams, who takes the crown for worst software of all time?
barryrandall
Teams, and it's not even close. Azure DevOps usually fails in understandable ways. Teams usually fails in ways that make me doubt my understanding of technology, society, and the nature of reality.
hsfzxjy
VSCode extension marketplace is also unaccessible
dkarras
spent a lot of time trying to troubleshoot why vscode can't access extension marketplace as I just installed a new computer and am trying to bring in my profile but getting XHR errors, and devtools showing CORS errors... after an hour+ of trying to get it to connect, finally figured the services are down.
tzs
I suppose this explains why OneDrive is down.

Attempting to save to my OneDrive folder on my Mac gets an error in Excel, and suggests that I save a copy to somewhere else.

Attempting to view my OneDrive files via the web interface shows no files and then eventually says something went wrong. Retrying cycles among different errors includes GeneralError, ServerTooBusy, and telling me to make sure I don't have any firewall or plugins that could be blocking access to api.onedrive.com.

Rapzid
Wider than DevOps. My entire company has been dead in the water for over 2 hours.
MuffinFlavored
What could possibly cause this kind of outage? Loss of a datacenter? Physical problem?

How can a "DNS" problem (or something similar) be this widescale?

BandButcher
I remember something with facebook? where their internal DNS was misconfigured.

edit:

"This change caused a complete disconnection of our server connections between our data centers and the internet. And that total loss of connection caused a second issue that made things worse. "

https://engineering.fb.com/2021/10/05/networking-traffic/out...

https://blog.cloudflare.com/october-2021-facebook-outage/

https://www.thousandeyes.com/blog/facebook-outage-analysis

Not sure if MS is going through the same issue but DNS could be a reason why

barkingcat
DNS is definitely capable of widescale disruption.

A dns error can take down the entire internet.

[deleted]
(comment deleted)
ctyc
Airports are also at a standstill. Handwriting tickets in KC at the moment and “awaiting the manifest before we can board” - where do you think they store that manifest, a drawer in Mr Airline’s basement?? Uhhh same place as their entire system, which is DOWN
BandButcher
"Big Paper and Pencil" can thank the recent MS and CDK Global hacks for making writing fashionable again
[deleted]
(comment deleted)
xtracto
It's funny how this story has not that much traction because it is Azure ... if it was AWS, half of the internet would be in flames.
mike503
Xbox Live was totally messed up earlier, I bet it's related.
hnburnsy
Frontier Airlines is dead in the water...

>Our systems are currently impacted by a Microsoft outage, which is also affecting other companies. During this time booking, check-in, access to your boarding pass, and some flights may be impacted. We appreciate your patience.

shscs911
This seems to be affecting Windows VMs hosted in AWS, as well. I'm seeing status check failure for Windows VMs across multiple AWS accounts in our Org.
whoknowsidont
We definitely won't be getting a postmortem.
red-iron-pine
never do.

i can recall just before COVID hit seeing Azure AD go down -- broke all of our shit.

but you look at the dashboard and it's 100% green. "it's fine" they said, and fought us for months.