> We're not observing an increase in successful connections after rolling back a recent change. We're working to evaluate additional mitigation solutions while we investigate the root cause.
Yep, I can imagine the feeling of doom when the rollback didn't work. Push back from the desk, head droops down, hand on forehead. Then I whisper, "Fuck".
We (like many other orgs, I expect) have gone all-in on AAD, so this has pretty much taken down everything company wide with the exception of the factories.
Good thing the plant floor isn't tied in.
It looks like it was copied from a text file, since if you view the source the actual formatting is there; but HTML's default whitespace/newline treatment causes it to turn into one big blob of text.
> Current status: We've identified a recent change that appears to be the source of the issue. We're rolling back the change to mitigate impact.
Current status: We've identified that reverting the recent change did not alleviate impact to Microsoft services as expected. We're working to explore additional options for mitigation.
[I actually enjoy these updates but they indeed read more like our internal incidents channel]
Most of the elements you'd want on a status page are there, but the tone and structure are really off. I think you nailed it that this reads like an internal communication to a manager, not to a customer.
It’s the “oh crap, there is an unsurmountable obstacle in between the end users and the admin portal. We have to say something now” page. Most of the time, service alerts are supposed to be reported through the Azure or MS365 admin portals.
Microsoft's "official" status page, which they will usually refer you to, can be found inside the admin center. They've been criticized in multiple previous incidents however, for experiencing an outage then telling people they can get full updates in the admin center, when the admin center requires a functioning logon to a service which is down at the time.
The page you've linked is kind of the "our real page is down" page, and after the issue is resolved what we usually see is all that replaced with something like "go check the admin center for outage information".
It looks like an actual status page written by a developer, instead of some Market speak gobbledygook attempt to not say anything for fear of liability or whatever
This is the first time since 2015 that I remember an outage of this magnitude (scope + impact) with the platform. Not saying my memory is authoritative, but.. I certainly wouldn't call this regular.
Ahh, thank you. I simply dabble in PS on the side and write util-like scripts as a brain stretch. This is the first time I've ever had someone look at and 'clean' anything... much appreciated.
"Current status: Rolling back the previously described change did not resolve the incident as expected. We're evaluating additional options to remediate the problem."
I'm assuming this is related: I can't do any development using Visual Studio 2019 at the moment because it calls home to verify the license and it is currently unable to log me in to to verify my license. So VS is basically bricked for me until this gets fixed :-(
That is awful and why I have always been very sceptical of software moving to the cloud.
Just yesterday I had a similar issue with Resharper, where our local licence server was down for the day. Luckily I was able to enable a 30-day trial. (and I guess Resharper isn't that mandatory to get work done)
I have always been very sceptical of software moving to the cloud.
Me too. After a disastrous episode with DropBox while travelling in foreign countries nearly a decade ago, I instituted my own File Server that's accessible via the Internet. All of my data is 'safe at home' with daily backups.
Either sftp/ssh with key-only logins (ie. no passwords), or Syncthing (depending on the volume of data, rate of changes, speed and costs of your network transits)
I now keep my primary 'work' and 'private' collections (10-30g each) and a large shared set (350g) synced across desktop & laptop, and also 3 off-site archive servers, using Syncthing. This includes a literal satellite connection for one site.
Not sure if it's a "best practice", but you can get a fairly Dropbox'ish experience by running a nextcloud/owncloud instance. It even comes in docker containers, so setup is quite simple.
Its less of a hassle to use something like nginx reverse proxy docker container in front of your web services, as it comes preconfigured with some best practices regarding TLS. (TLS1.2+ only by default) https://hub.docker.com/r/jwilder/nginx-proxy
If you use docker, be sure to google around how to manage firewall with containers (must use DOCKER-USER chain instead of INPUT)
Ofcourse, you will get a bit involved in setup step if you choose to tie nginx proxy, letsenccrypt and nextcloud containers. But search engine is your friend.
It was probably my fault in configuration, however the idea was to have a DropBox whereby I could put in files while travelling with my laptop and those files would be taken up with my home server.
Except that it worked in reverse. My laptop was 'emptier' than the home server, so the home server's files were deleted to match the laptop. Hundreds of megabytes of files were destroyed before I managed to stop it.
On top of that, I only had a 2 gig per month quota for internet access while travelling, and all the DropBox activity soon exhausted that in days.
On later trips my own home server coped very well, with less overall unwanted activity.
Welcome to Visual Studio, the home of updates that take hours to install and sometimes end up bricking your IDE.
To be fair, that hasn't happened in a pretty long time for me - but in the earlier days it was commonplace. It used to be bleeding edge features that would do it, other times it was just because they took so long, that interrupting the process of upgrading or installing caused a mess. Or installing the newest version alongside the older version, etc.
Visual Studio is a massive and complex creature (much like other "big IDEs", such as Eclipse) - although it has become much more streamlined in more recent releases - considering how complex it is.
EDIT: To be clear, problems often involved corruption of registry values, library/DLL hell, small database or configuration files and caches, etc. that happened with Visual Studio in the scenarios above.
Yeah I have been using VS for nearly a decade now but I've never 'bricked' it.
Sure, sometimes installing new versions alongside old versions could cause issues. But that's a managed process that one team member would do before the whole team migrated and any issues would be worked out during that stage. (one of our old upgrade steps was to install 2015 before 2013 because doing it the other way around would break something in 2013; 2013 still being needed for some projects).
I have never been in the situation where the entire company (or even entire team) was locked out of their IDE.
Just to back this up, I've also bricked my VS installation multiple times when I worked in that environment. However, some of that was self inflicted when I was trying to figure out how to restore older sub versions of vs 2019 before they started publishing the manifests publicly.
Also, while this isn't bricking the IDE, you can definitely ruin solutions & project files in a very hard to debug way if you ever need to manually update those files.
I've been using it since Visual Studio .NET 2003.. so we as developers didn't all have virtual machines, image-based backups or things like Docker/Vagrant back in late 2003/early 2004.
Yes, since the inception of these kinds of tools, recovery from these corruptions became easier by restoring from an image. Still, in the early days, you could burn a lot of time doing anything of these things!
> I don't think I've ever 'bricked' my IDE. How does one go about bricking an IDE?
Visual Studio major version updates (uninstall old plus install new) used to be a good way to do that and often wreak havoc on the overall stability of your system as well. At least, on a system where you didn't want to strip it to fresh OS install as an intermediate step.
Yes? I have never heard the concept of "bricking your IDE" before. How badly could you mess up and not be able to reset it? If your IDE can break dramatically without an obvious method to revert to a good state, consider a different IDE.
It used to be you bought a key to unlock it permanently, but nowadays it ties your license to your Microsoft account. There’s a link to “unlock with a product key“, but I have no idea how you get one of those. I hate renting software rather than owning it, but also VS2019 is where I’m most productive so I put up with it.
This isn’t even the first time this has happened. I do most of my development on my desktop computer, sometimes I go long stretches without using my laptop. More than once I opened up my laptop on a flight to try and get some coding work done, but surprise, license is expired and I don’t have any connectivity to renew it. Absolutely ridiculous.
Time travel (reverse) debugging is a killer feature and only available for 2019 onwards. Prior to that you have to use their "IntelliTrace" stuff, which isn't nearly as good.
You used to be able to get keys for all windows software through an MSDN account. No idea if that's still the case, would have been around 2017 last time I had one.
The compiler and the .net platform in general are all accessible without a paid license or internet connection, and actually there’s a free edition of visual studio you can use without a paid license as well. But downloading and installing a fresh copy of Visual Studio Community Edition would take a while and I don’t feel like dealing with that hassle right now. I assume this will be fixed by tomorrow.
You (or your employer) knowingly chosen a proprietary and hostile development platform and now you are reaping the benefits. I am sorry for you, not for you inability to work, but for your incorrect choices.
There was a big one in 2018 involving authentication. A handful of datacenters outside of Houston couldn't get enough power and the shutdown procedures prioritized saving data. Some critical Azure system service was only located in that geographic zone and the data recovery priority caused that low level service to fail its migration. It caused a huge authentication failure across all Azure services globally. MS got blasted about it at Ignite pretty soon after.
is this a national risk? - think other countries taking out these massive, saas offerings. must be easier than simultaneously targeting thousands of disparately configured, onprem implementations
I know many posts in this thread have been jokes, but I do hope this outage helps MS reconsider their offline iterations of Office. I use Office on a near constant basis (writing and building lecture PowerPoints). The subscription model isn't feasible and would ultimately be more expensive in the long term. When Office 2019 was released I quickly picked it up out of worry that MS would shift their academic licensing programs to 365, which they ultimately did.
I just finished convincing old-school management to move to Office365/Azure/Teams after we've been doing everything on-premise, saying we can reduce our maintenance and increase in reliability since we're a small operation. Needless to say this isn't a good look for me.
The variable here is not just the size & skill of the team but their needs. In most cases a single company or team will need a simpler, smaller-scale deployment, probably running on one or a handful of machines and not requiring the insane complexity that a public cloud provider is running behind the scenes to provide their service.
This eliminates a lot of moving parts and things that can fail. Imagine an Nginx server running on a single machine. There's very little that can go wrong here beyond hardware failure (and certain types of failures can be mitigated with things like RAID), and yet it is probably enough to host most internal websites.
Now compare this to something like Azure App Service which is obviously more complex to be able to support many tenants, load-balance, etc. There is much more that can go wrong with the entire App Service infrastructure (due to its complexity and moving parts) than with a single machine running Nginx, and the complexity will also delay disaster recovery efforts (this outage is now lasting for more than an hour - you can reinstall an entire Linux web server from scratch in half that time).
If they have enough resources. When everyone is busy working on the revenue generating products for the company, taking time away to work on internal network infrastructure immediately becomes a cost center.
If it makes you feel any better, I've been that guy on both sides of the coin - both on the cloud and on-prem. Things go down.. even the cloud.
I worked for a company, where I really did have nearly 100% uptime at my local data center in our office. I wasn't a luddite, and the cloud wasn't something I was afraid of or didn't understand. I just, at the time, had an infrastructure in place before the cloud existed, and it worked well enough at a great price point. We were on the same electrical grid as a hospital, our infrastructure handled our scale, and just really didn't have problems for a long span of time.
Still, someone came in and said.. what, you are still doing stuff on-prem and not the cloud!? How legacy! How dated! And so the push to the cloud came next.
We migrated to the cloud (and some colo facilities for some equipment we chose to keep), and on the 3rd day of being freshly migrated, the cloud provider had a major outage and went down for longer than we'd ever had on prem. The next month, the colo went down and their diesel generators failed, and it was down for an entire afternoon.
Oh sure, there was some SLA money returned in that event from the colo...
I'm just saying, I've been on both sides. Tell them the truth - even the cloud has outages, but it is certainly more "convenient" to have hundreds of engineers work on fixing an outage at global scale, and all you have to do is wait for it to start working again - then it is to have to fix it all yourself.
> Tell them the truth - even the cloud has outages, but it is certainly more "convenient" to have hundreds of engineers work on fixing an outage at global scale, and all you have to do is wait for it to start working again - then it is to have to fix it all yourself.
Yep, that’s the argument I’ve made, both to management and even our non-techie customers. When we show them our new SaaS product, they get concerned about cloud outages. Our response is “if AWS goes down, you have bigger things to worry about since other things will be down too.”
The plus side I’ve mentioned that our company’s software developers are freed from doing network IT maintenance and infrastructure and can continue focusing on our actual core products instead.
Why are your software developers doing "network IT maintenance and infrastructure" in the first place? Is it because of "DevOps" maybe?
It's one thing to experience the sysadmin practise getting taken over by developers. The next thing is usually the latter needing to be "freed" from this burden... Seems like a big fad.
Microsoft Teams (The chat/collaboration service) and team foundation server (The source code repository and DevOps platform) are two completely different products that happen to both have the same word in their name. Although I can understand the trauma ;-)
I've gone all in on Microsoft for my company (high Office usage) with O365, Azure and friends and I'm regretting this decision.
OneDrive for Business is almost unusable on Windows 10. The web apps are incredibly slow. Teams is very inconsistent for users between desktop and mobile usage. It's a mess.
I'm planning on switching to Amazon next year. I hope I can find an alternative to the Office situation (maybe offline licenses).
If I were offering a free (as in price) alternative to MS Office, I'd first recommend WPS Office over LibreOffice, even though it's closed-source. It has a much better interface out of the box. But I'd use something to block it from phoning home to its probably Chinese government-beholden servers, like OpenSnitch for Linux* or LuLu for macOS.
If I may, in my experience Softmaker is not bad at all (still closed source and Commercial, but affordable), LibreOffice simply does not fit, for very light use LibreOffice is fine, but still has - here and there - too many "quirks".
I have found a Dropbox to be more reliable at file sync than OneDrive, especially if you have many, many small files. Then again, the pricing is so far off between the two that I can’t justify Dropbox anymore and just put up with the horrible OneDrive performance. Does anyone have any better alternatives?
Luckily Microsoft still sells offline licenses for Office 2019. There are several features missing that are exclusively tied to the Office 365 subscription, but overall it’s still the same Office productivity suite you would expect. Here’s to hoping they keep doing that in the future and don’t pull an Adobe…
My kids' school district is doing all online over MS Teams; things started crashing and burning around 3PM PST (no "Dad Support" can fix MS's servers though).
Azure DevOps' Git is also down (a git pull hangs forever) which is a bit surprising. I wouldn't have expected SSH key authentication to depend on Azure Active Directory.
Microsoft Teams also appears to be down so it's not just authentication that's affected as I was already logged in and it still doesn't work, just hangs trying to load forever.
Edit: appears to be resolved at 1:07 am London time.
At least the guy with the change request was vindicated for now. That must have felt good - about to be crucified for putting in a change that brought down AAD globally, they roll it back and shit's still broken. :D hahaha wudn't me mofos!
165 comments
[ 3.1 ms ] story [ 270 ms ] threadOooh that's panic stations.
Imagine there's some foxhole prayers happening.
Big exhale.
We (like many other orgs, I expect) have gone all-in on AAD, so this has pretty much taken down everything company wide with the exception of the factories. Good thing the plant floor isn't tied in.
How frequently do you think this sort of thing would have to happen before your org considered a significant change in arch?
Looking at it now, Azure AD is the only service that isn't green.
https://status.office365.com/
Title: Can't access Microsoft 365 services
User Impact: Users may be unable to access multiple Microsoft 365 services.
More info: Any Microsoft 365 service that leverages Azure Active Directory (AAD) authentication may be impacted by this issue.
Current status: We've identified and are reverting a recent change to the service which may be causing or contributing to impact.
Scope of impact: Any user may experience access problems for Microsoft 365 services.
Anyways the Azure status page is a bit more grown up:
https://status.azure.com/en-us/status
"Title:" makes it look like a grade school book report.
Title: We're investigating a potential issue affecting Outlook.com
User Impact: Affected users may be unable to access Outlook.com services or features.
Current status: We've identified a recent change that appears to be the source of the issue. We're rolling back the change to mitigate impact.
Next update by: Monday, September 28, 2020, at 11:00 PM UTC
Current status: We've identified that reverting the recent change did not alleviate impact to Microsoft services as expected. We're working to explore additional options for mitigation.
[I actually enjoy these updates but they indeed read more like our internal incidents channel]
The page you've linked is kind of the "our real page is down" page, and after the issue is resolved what we usually see is all that replaced with something like "go check the admin center for outage information".
It just seems that office 365 is part of that older monolithic Microsoft that we all used to hate.
Edit: maybe not that short, in 20, 30 years maybe?
1990s hosting company: Linux Server 99.99% uptime. Windows - no guarantee. 2020 - same
At line:1 char:16 + $loop = "loop" do {$response = $null; Start-Sleep -Seconds 5 try {$re ...
+ ~~
Unexpected token 'do' in expression or statement.
+ CategoryInfo : ParserError: (:) [], ParentContainsErrorRecordException
while ($true) { $response = $null; try { $response = Invoke-WebRequest "https://login.microsoftonline.com/common/oauth2/" -ErrorAction SilentlyContinue | select status*; if ($response) { "OK $($response.StatusCode) $(Get-Date)" | Write-Host; } } catch { "Down $(Get-Date)" | Write-Host; }; Start-Sleep -Seconds 5; }
https://status.azure.com/en-us/status
https://status.office365.com
https://portal.office.com/servicestatus
Screenshot: https://rog.gy/ss/3b14f7a2ac.png
Now, the pirated version never phones home and always works even when MS decides to steal your products back.
Just yesterday I had a similar issue with Resharper, where our local licence server was down for the day. Luckily I was able to enable a 30-day trial. (and I guess Resharper isn't that mandatory to get work done)
Me too. After a disastrous episode with DropBox while travelling in foreign countries nearly a decade ago, I instituted my own File Server that's accessible via the Internet. All of my data is 'safe at home' with daily backups.
I now keep my primary 'work' and 'private' collections (10-30g each) and a large shared set (350g) synced across desktop & laptop, and also 3 off-site archive servers, using Syncthing. This includes a literal satellite connection for one site.
[0] https://syncthing.net/
Its less of a hassle to use something like nginx reverse proxy docker container in front of your web services, as it comes preconfigured with some best practices regarding TLS. (TLS1.2+ only by default) https://hub.docker.com/r/jwilder/nginx-proxy
If you use docker, be sure to google around how to manage firewall with containers (must use DOCKER-USER chain instead of INPUT)
Ofcourse, you will get a bit involved in setup step if you choose to tie nginx proxy, letsenccrypt and nextcloud containers. But search engine is your friend.
What was that? They cut your access because you logged from a different IP?
Except that it worked in reverse. My laptop was 'emptier' than the home server, so the home server's files were deleted to match the laptop. Hundreds of megabytes of files were destroyed before I managed to stop it.
On top of that, I only had a 2 gig per month quota for internet access while travelling, and all the DropBox activity soon exhausted that in days.
On later trips my own home server coped very well, with less overall unwanted activity.
Because you're less likely to brick your IDE than Microsoft is?
To be fair, that hasn't happened in a pretty long time for me - but in the earlier days it was commonplace. It used to be bleeding edge features that would do it, other times it was just because they took so long, that interrupting the process of upgrading or installing caused a mess. Or installing the newest version alongside the older version, etc.
Visual Studio is a massive and complex creature (much like other "big IDEs", such as Eclipse) - although it has become much more streamlined in more recent releases - considering how complex it is.
EDIT: To be clear, problems often involved corruption of registry values, library/DLL hell, small database or configuration files and caches, etc. that happened with Visual Studio in the scenarios above.
Sure, sometimes installing new versions alongside old versions could cause issues. But that's a managed process that one team member would do before the whole team migrated and any issues would be worked out during that stage. (one of our old upgrade steps was to install 2015 before 2013 because doing it the other way around would break something in 2013; 2013 still being needed for some projects).
I have never been in the situation where the entire company (or even entire team) was locked out of their IDE.
Also, while this isn't bricking the IDE, you can definitely ruin solutions & project files in a very hard to debug way if you ever need to manually update those files.
Restores operating system image in a few minutes.
Yes, since the inception of these kinds of tools, recovery from these corruptions became easier by restoring from an image. Still, in the early days, you could burn a lot of time doing anything of these things!
Visual Studio major version updates (uninstall old plus install new) used to be a good way to do that and often wreak havoc on the overall stability of your system as well. At least, on a system where you didn't want to strip it to fresh OS install as an intermediate step.
This isn’t even the first time this has happened. I do most of my development on my desktop computer, sometimes I go long stretches without using my laptop. More than once I opened up my laptop on a flight to try and get some coding work done, but surprise, license is expired and I don’t have any connectivity to renew it. Absolutely ridiculous.
I don't think you can target core 3.0+ at all in VS2017, and only 3.1 is supported now. What that means for security updates I don't know.
https://news.ycombinator.com/newsguidelines.html
The link is javascript:window.external.AddFavorite(location.href,%20document.title);
I'm not sure if even that number is low enough.
I think I once did something stupid like try to patch GoOffice into OpenOffice and couldn't get it to build. But the distro packages still worked!
Wasn't there also a widely-publicized outage in 2016?
Microsoft can still probably do a much better job compared to a small team.
This eliminates a lot of moving parts and things that can fail. Imagine an Nginx server running on a single machine. There's very little that can go wrong here beyond hardware failure (and certain types of failures can be mitigated with things like RAID), and yet it is probably enough to host most internal websites.
Now compare this to something like Azure App Service which is obviously more complex to be able to support many tenants, load-balance, etc. There is much more that can go wrong with the entire App Service infrastructure (due to its complexity and moving parts) than with a single machine running Nginx, and the complexity will also delay disaster recovery efforts (this outage is now lasting for more than an hour - you can reinstall an entire Linux web server from scratch in half that time).
I worked for a company, where I really did have nearly 100% uptime at my local data center in our office. I wasn't a luddite, and the cloud wasn't something I was afraid of or didn't understand. I just, at the time, had an infrastructure in place before the cloud existed, and it worked well enough at a great price point. We were on the same electrical grid as a hospital, our infrastructure handled our scale, and just really didn't have problems for a long span of time.
Still, someone came in and said.. what, you are still doing stuff on-prem and not the cloud!? How legacy! How dated! And so the push to the cloud came next.
We migrated to the cloud (and some colo facilities for some equipment we chose to keep), and on the 3rd day of being freshly migrated, the cloud provider had a major outage and went down for longer than we'd ever had on prem. The next month, the colo went down and their diesel generators failed, and it was down for an entire afternoon.
Oh sure, there was some SLA money returned in that event from the colo...
I'm just saying, I've been on both sides. Tell them the truth - even the cloud has outages, but it is certainly more "convenient" to have hundreds of engineers work on fixing an outage at global scale, and all you have to do is wait for it to start working again - then it is to have to fix it all yourself.
Yep, that’s the argument I’ve made, both to management and even our non-techie customers. When we show them our new SaaS product, they get concerned about cloud outages. Our response is “if AWS goes down, you have bigger things to worry about since other things will be down too.”
The plus side I’ve mentioned that our company’s software developers are freed from doing network IT maintenance and infrastructure and can continue focusing on our actual core products instead.
It's one thing to experience the sysadmin practise getting taken over by developers. The next thing is usually the latter needing to be "freed" from this burden... Seems like a big fad.
OneDrive for Business is almost unusable on Windows 10. The web apps are incredibly slow. Teams is very inconsistent for users between desktop and mobile usage. It's a mess.
I'm planning on switching to Amazon next year. I hope I can find an alternative to the Office situation (maybe offline licenses).
(*) https://github.com/gustavo-iniguez-goya/opensnitch
Luckily Microsoft still sells offline licenses for Office 2019. There are several features missing that are exclusively tied to the Office 365 subscription, but overall it’s still the same Office productivity suite you would expect. Here’s to hoping they keep doing that in the future and don’t pull an Adobe…
Was worried there for a second.
Microsoft Teams also appears to be down so it's not just authentication that's affected as I was already logged in and it still doesn't work, just hangs trying to load forever.
Edit: appears to be resolved at 1:07 am London time.