In Southern Europe some smaller web servers are intermittently not working, while big servers like YouTube are working fine. But I don't think it is related.
> The other AZs in the region are functioning normally. Customers who were running their applications redundantly across the AZs are not impacted by this event.
Interesting adjacent theory is how much are datacenters becoming military target to strike as part of disrupting initial defenses. It doesn't seem it was the case in this instance, but I could see this becoming a more important target in future.
Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services.
Again, this is just me thinking out loud on a tangent and this doesn't have much to do with this story, but I felt it was an interesting thought to share nonetheless.
Data centers in space no longer look so unreasonable when the requirement is “redundancy against multi site bomb strikes mid op”. A little depressing when some pieces start to fit together.
Has this ever happened ever in history of Cloud providers before this because of war?
They mention that the datacenter had fires and sparks and they are mentioning hours of downtime but given the situation, How does that prevent the situation from happening again. It's best for people to use safer regions than the middle east in the moment as missiles might target the same datacenter seeing that some damage was caused.
Moving forward, will there be a demand (all be small) for nuclear bunker esque datacenters which can withstands missiles? I know absolutely nothing about constructing underground but can explosives not be used to create underground datacenters comparatively cheaply? One can also use revamped Nuclear bunkers (although the scale of AWS datacenters might be huge tho who knows)
I am curious but what are the safety attempts made by Internet Exchange Providers or (had to search it up) but Submarine Cable landing stations, to me it feels like blowing these up leads to internet downtime across whole country / between providers.
We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally.
This reminds me of a visit to an Equinix data centre where the sales person was droning on and on about how incredibly reliable their power supplies were, how uninterruptible everything was, etc, etc…
Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances.
A bit bored and annoyed, I pointed to the giant red button conspicuously placed in the middle of a pillar and asked what it is for.
“Oh, that’s in case there’s a fire!”
“What does it do?”
“It cuts… the power… uhh… for the safety of the fire department.”
“So… if there’s a wisp of smoke in a corner somewhere, the fireys turn up, the first thing they do is… cut the power?”
> Due to the ongoing conflict in the Middle East, both affected regions have experienced physical impacts to infrastructure as a result of drone strikes.
27 comments
[ 2.8 ms ] story [ 43.5 ms ] thread> The other AZs in the region are functioning normally. Customers who were running their applications redundantly across the AZs are not impacted by this event.
I bet that was an interesting sev2 ticket!
God forbid we'd ever say that it was struck by a missile or a munition in an act of war.
Seems like it should be somewhat easier to bomb 50 datacenters than it would be to hack and disrupt 1000s of different services.
Again, this is just me thinking out loud on a tangent and this doesn't have much to do with this story, but I felt it was an interesting thought to share nonetheless.
To me it absolutely looks like the case in here, what are the chances three availability zones across two different countries have been attacked?
They mention that the datacenter had fires and sparks and they are mentioning hours of downtime but given the situation, How does that prevent the situation from happening again. It's best for people to use safer regions than the middle east in the moment as missiles might target the same datacenter seeing that some damage was caused.
Moving forward, will there be a demand (all be small) for nuclear bunker esque datacenters which can withstands missiles? I know absolutely nothing about constructing underground but can explosives not be used to create underground datacenters comparatively cheaply? One can also use revamped Nuclear bunkers (although the scale of AWS datacenters might be huge tho who knows)
Had some ideas which show that this idea might be interesting, https://www.nature.com/articles/s44284-026-00406-2
I am curious but what are the safety attempts made by Internet Exchange Providers or (had to search it up) but Submarine Cable landing stations, to me it feels like blowing these up leads to internet downtime across whole country / between providers.
Those already exist. See for example Bahnhof's "Pionen - White Mountain" data center in Stockholm, or Cyberfort's "The Bunker" a bit west of London.
Mar 01 9:41 AM PST
We want to provide some additional information on the power issue in a single Availability Zone in the ME-CENTRAL-1 Region. At around 4:30 AM PST, one of our Availability Zones (mec1-az2) was impacted by objects that struck the data center, creating sparks and fire. The fire department shut off power to the facility and generators as they worked to put out the fire. We are still awaiting permission to turn the power back on, and once we have, we will ensure we restore power and connectivity safely. It will take several hours to restore connectivity to the impacted AZ. The other AZs in the region are functioning normally.
Essentially, he was trying to assure us that no-no-no, we don’t need multiple zones like the public clouds, they can instead guarantee 100% uninterrupted power under all circumstances.
A bit bored and annoyed, I pointed to the giant red button conspicuously placed in the middle of a pillar and asked what it is for.
“Oh, that’s in case there’s a fire!”
“What does it do?”
“It cuts… the power… uhh… for the safety of the fire department.”
“So… if there’s a wisp of smoke in a corner somewhere, the fireys turn up, the first thing they do is… cut the power?”
“… yes.”
“Not 100% then, is it?”
Well that's a way of phrasing it.
I assume these were missiles.
> Due to the ongoing conflict in the Middle East, both affected regions have experienced physical impacts to infrastructure as a result of drone strikes.
So they're confirming actual aggression