Considering almost everything public facing at Google supports IPv6 and almost no GCP services support it outside of a few regions (1 in the US), I’d guess none of it.
> Folks on Azure are active-active in multiple availability zones
Why would people on Azure be more likely than people on AWS to have redundancy? Anyway, in the recent AWS outages, multiple AZs didn't help much. Needed to be in multiple regions.
In my experience, a higher portion of Azure customers are larger enterprise customers that engage professional services from Microsoft to architect things, etc.
AWS is far more popular in general, but especially with smaller teams that are more self-directed, start with prototypes and iterate from there. It wouldn't surprise my if the latter often only built in redundancy after learning lessons the hard way, even though AWS provides the ability to do so from the get go.
Purely conjecture, but that skew in the customer type is definitely there.
Before “the cloud”, it was already common to run Active Directory installs as active-active in on- prem scenarios, since those installs would’ve hit outages pretty early after installing AD. Maybe not across data center regions, but AD admins learn real quick to have redundancy and backups.
Porting over to Azure AD probably wasn’t such an issue then, as most firms would lift&shift their on-prem to the cloud, keeping the same configurations.
This is what I didnt get about the multi-AZ claims. If you're in multiple zones within the same region when that region goes down, you're still going down too. Yet, this seems to be the agreed upon solution decreed far & wide.
It helps. It's not a silver bullet. AZs tend to go down more often than whole regions, is the belief (I don't think there's sufficient data, but a priori it does make sense).
We are on Azure but no one was working when this hit and we don't host any critical workloads so its not a big deal for us this time.
AWS and GitHub seem to crater right when everyone is using them the most (i.e. 10am EST on a Tuesday), so there is bound to be far more discussion around these occurrences since everyone is wired on coffee, ass-in-chair, frustrated that they can't get any work done. People usually don't get out of bed and then get on the internet to complain about a transient outage that occurred while they were asleep.
This is probably the correct answer. We run a lot of stuff at work on AAD but I don't care, it's 10:45p EST. If I were on our infra team I might care a bit, mostly monitoring to verify it would be up before users start trying to log on tomorrow but otherwise, meh.
I'd wager that Azure customers tend to run classic workloads as VMs. Those workloads are usually not impacted by API outages. When AWS has API outages, all those Lambdas, autoscaling groups, subscribers to queues and topics, etc have a very bad day.
Doesn't pretty much everything depend on IAM to know what it's allowed to do? I suppose if you had stuff already running then it might be able to keep going on cached authorizations, but that still sounds likely to cause a lot of problems.
Probably their most important consumer and b2b off the shelf cloud service, yeah, but when people talk about Azure I presume they mean more like, the compute, database, hosted kubernetes etc products. Most of which people run Linux on and tend(?) not to integrate with ActiveDirectory, broadly.
In my limited experience working with Azure, parts of it, including AAD, go down way more often than AWS. Also, very large swaths of the Internet run on AWS directly or indirectly, so it makes sense that it would be larger news.
It looks like Azure Active Directory, so O365 would be hit harder than Azure I imagine because most services in Azure are still using storage access keys and whatnot given that AAD auth for data plane is fairly recent in most offerings.
I.e my VM running and having network connectivity doesn’t necessarily depend on AAD.
The main idea with Google creating K8s was to commoditize it's complement. Now they are pushing for Anthos with is basically GCP with BYOD on-prem (or other vendor) hardware. So what you will see is people using Hetzner like services with cloud software provided by these companies.
It is kind of already happening with Teleco world. Companies like Dish and Vodafone partner with AWS & GCP. These telecos buy their own OpenRAN hardware from Nokia or Ericsson and manage it using AWS, GCP, Azure software.
Azure has a lot of features around this model.For instance, you can put a kind of azure agent on your on prem servers and then provision azure virtual machines and managed databases onto them, or cloud active directory for on-prem desktops.
I hadn't thought of them in the context of "hire cheaper non-cloud servers for your baseline load and storage".
One of my favourite quotes is from "Romance of the Three Kingdoms". It notes an observation that I find generally applicable to many things in life:
"Those long divided shall be united;
those long united shall be divided:
such is the way of the universe."
The dynasty of Han collapsed and was divided into three kingdoms. After many years of war, one of them finally defeats the other two and it becomes united. This pattern is replayed again and again in the history, and possibly in the future.
Our production infrastructure is on Azure and we're seeing basically no impact from this event. I'm not sure what constitutes a major outage but we've had a few others this year that were much worse than this in terms of business impact.
The us-west outage did not affect connectivity to the west coast. Our west coast customers had no problems with that connectivity, but our east coast and overseas customers had problems for a couple of hours.
Also see https://en.wikipedia.org/wiki/Semisopochnoi_Island ("At 179°46' East (+179.7667) (in the Eastern Hemisphere), the easternmost tip of Semisopochnoi is, by longitude, the easternmost land location in the United States and North America.")
FWIW, Semisopochnoi Island is a TIL moment for me, too =)
> The latitude and longitude of a point don’t depend on that point’s time zone.
They also don't have anything to do with how far east or west a point is.
You have two options:
1. East is the direction in which the Earth rotates, and West is the opposite direction. By this definition, no point on the earth has an east-coordinate or a west-coordinate.
2. East is the direction in which time gets later, and West is the direction in which time gets earlier. This is the only definition which will allow you to claim that one point is "more east" than another point. And it tells you that the difference between East and West is the International Date Line.
“ Longitude is a geographic coordinate that specifies the east–west position of a point on the Earth's surface relative to the prime meridian.”
It’s looking good for longitude so far. Looks like people somehow manage to measure relative longitudes ok after all. Let’s check time zones.
“ You may notice, however, that the French collectivity of Saint-Pierre and Miquelon just to the south of Newfoundland is a half-hour ahead despite being at exactly the same longitude. By traveling west on the ferry from Newfoundland, you actually travel forward in time a half-hour.”
Oops. So by travelling consistently west in the same direction you can actually go back into a time zone you’ve previously left without crossing the date line. Not looking good. There are actually several places on Earth where this is possible. Here’s another good one. Is Kaliningrad east of Lithuania, or west of it?
“ All together, there are now three different areas where you gain two hours crossing time zones instead of just one. This also creates an unusual situation in the Kaliningrad Oblast exclave, which is now an hour ahead of Lithuania to its east, and two hours ahead of neighboring Poland.”
Good luck constructing a consistent reference frame out of that.
> “ Longitude is a geographic coordinate that specifies the east–west position of a point on the Earth's surface relative to the prime meridian.”
What was the disagreement? So you're 3° west of the prime meridian. How far west is that? Are you to the west of the prime meridian, or are you to its east?
Alaska is the western, northern, and eastern state in the U.S. The Aleutian Islands cross the International Dateline. Hawaii is the southern most state.
Azure has data centers all over the globe, how likely is it to impact all of them? So even if all your infrastructures were provisioned in one single physical data center, it's highly likely the incident had nothing to do with your stuff.
Wasn't the last meaningful AAD outage in February sometime? This was an outage that lasted on the order of an hour, outside of normal US business hours.
I think the crazy uptime we normally get from the cloud has seriously skewed people's perceptions of how things used to be on-prem. I personally remember when my Fortune 50 former employer had multiple multi-day email outages in one year.
Perhaps we should put computers inside each businesses location, to minimize the blast radius of any specific outage and make people take responsibility for their own stack.
I wouldn't call this a major outage. We had almost no issues with our applications ( it's resolved by now) and customers are in US, Canada and EU.
The update that caused an issue, was at a pretty good time ( 20:42 EST ). So that probably explains it.
> A shared component of the Microsoft account (MSA) and Azure AD sign-in services stopped responding when a combination of a configuration error and a routine update caused multiple redundant endpoints to become unreachable. This caused sign-in failures in Microsoft services for both Microsoft personal accounts and Azure AD accounts.
94 comments
[ 2.9 ms ] story [ 174 ms ] thread[1] - https://downdetector.com/
- Folks on Azure are active-active in multiple availability zones
- Most of those using Azure are not impacted by this particular outage
- Azure isn't widely used
...
Why would people on Azure be more likely than people on AWS to have redundancy? Anyway, in the recent AWS outages, multiple AZs didn't help much. Needed to be in multiple regions.
AWS is far more popular in general, but especially with smaller teams that are more self-directed, start with prototypes and iterate from there. It wouldn't surprise my if the latter often only built in redundancy after learning lessons the hard way, even though AWS provides the ability to do so from the get go.
Purely conjecture, but that skew in the customer type is definitely there.
Porting over to Azure AD probably wasn’t such an issue then, as most firms would lift&shift their on-prem to the cloud, keeping the same configurations.
AWS and GitHub seem to crater right when everyone is using them the most (i.e. 10am EST on a Tuesday), so there is bound to be far more discussion around these occurrences since everyone is wired on coffee, ass-in-chair, frustrated that they can't get any work done. People usually don't get out of bed and then get on the internet to complain about a transient outage that occurred while they were asleep.
This is more like Amazon’s IAM product being down than AWS as a whole.
https://wiebetech.com/products/mouse-jiggler-mj-3/
https://portal.office.com/servicestatus
Edit: looks like it's AAD that's having a bad day.
https://status.azure.com/en-us/status
Source: I'm a former Azure intern
The cycle will continue. What was old will become new again.
It is kind of already happening with Teleco world. Companies like Dish and Vodafone partner with AWS & GCP. These telecos buy their own OpenRAN hardware from Nokia or Ericsson and manage it using AWS, GCP, Azure software.
I hadn't thought of them in the context of "hire cheaper non-cloud servers for your baseline load and storage".
Original quote in Chinese: ＂话说天下大势，分久必合，合久必分。＂
To the extent the phrase has meaning, the claim is still untrue; the International Date Line goes out of its way to avoid crossing Alaska.
https://en.wikipedia.org/wiki/Eastern_Hemisphere
Also see https://en.wikipedia.org/wiki/Semisopochnoi_Island ("At 179°46' East (+179.7667) (in the Eastern Hemisphere), the easternmost tip of Semisopochnoi is, by longitude, the easternmost land location in the United States and North America.")
FWIW, Semisopochnoi Island is a TIL moment for me, too =)
Ask a geographer or cartographer about hemispheres.
They also don't have anything to do with how far east or west a point is.
You have two options:
1. East is the direction in which the Earth rotates, and West is the opposite direction. By this definition, no point on the earth has an east-coordinate or a west-coordinate.
2. East is the direction in which time gets later, and West is the direction in which time gets earlier. This is the only definition which will allow you to claim that one point is "more east" than another point. And it tells you that the difference between East and West is the International Date Line.
Latitude is absolute. Longitude is relative.
It’s looking good for longitude so far. Looks like people somehow manage to measure relative longitudes ok after all. Let’s check time zones.
“ You may notice, however, that the French collectivity of Saint-Pierre and Miquelon just to the south of Newfoundland is a half-hour ahead despite being at exactly the same longitude. By traveling west on the ferry from Newfoundland, you actually travel forward in time a half-hour.”
Oops. So by travelling consistently west in the same direction you can actually go back into a time zone you’ve previously left without crossing the date line. Not looking good. There are actually several places on Earth where this is possible. Here’s another good one. Is Kaliningrad east of Lithuania, or west of it?
“ All together, there are now three different areas where you gain two hours crossing time zones instead of just one. This also creates an unusual situation in the Kaliningrad Oblast exclave, which is now an hour ahead of Lithuania to its east, and two hours ahead of neighboring Poland.”
Good luck constructing a consistent reference frame out of that.
> “ Longitude is a geographic coordinate that specifies the east–west position of a point on the Earth's surface relative to the prime meridian.”
What was the disagreement? So you're 3° west of the prime meridian. How far west is that? Are you to the west of the prime meridian, or are you to its east?
I think the crazy uptime we normally get from the cloud has seriously skewed people's perceptions of how things used to be on-prem. I personally remember when my Fortune 50 former employer had multiple multi-day email outages in one year.
i know it's extremely unlikely, but that kind of cleverness would be pretty cool.
"our new sexy feature x went down for an hour today."
"wait, what? they offer x as a service now?"
https://status.cloud.mongodb.com/
The update that caused an issue, was at a pretty good time ( 20:42 EST ). So that probably explains it.
> A shared component of the Microsoft account (MSA) and Azure AD sign-in services stopped responding when a combination of a configuration error and a routine update caused multiple redundant endpoints to become unreachable. This caused sign-in failures in Microsoft services for both Microsoft personal accounts and Azure AD accounts.
https://portal.azure.com/#blade/Microsoft_Azure_Health/Azure...