Description: We are experiencing an issue with Google Compute Engine in all zones in us-east1 where existing VMs may be unavailable or unreachable, and new VM creation may fail beginning on Monday, 2020-06-29 08:15 US/Pacific.
Symptoms: Existing VMs may appear as unavailable and may be unreachable. New VM creation may fail.
Our engineering team continues to investigate the issue.
We will provide an update by Monday, 2020-06-29 09:52 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Description: We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.
We expect services in us-east1-d to be recovering within the next 30 minutes, there is no eta for service recovery for us-east1-c yet.
Impact is due to power failure. A more detailed analysis will be available at a later time.
Our engineering team is working on recovery of impacted services..
We will provide an update by Monday, 2020-06-29 10:00 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.
Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.
Got hit by this, multiple production systems down... No mitigation available because we can't migrate workloads to another zone - can't detach PD from affected nodes, can't add more node pools to K8S cluster, can't do anything but wait...
The issue with issue with Google Compute Engine in zones us-east1-c and us-east1-d where existing VMs may be unavailable or unreachable, and new VM creation may fail beginning on Monday, 2020-06-29 08:20 US/Pacific has been resolved for all affected users as of Monday, 2020-06-29 09:45 US/Pacific.
We are experiencing an issue with Google Compute Engine in zones us-east1-c and us-east1-d where existing VMs may be unavailable or unreachable, and new VM creation may fail beginning on Monday, 2020-06-29 08:20 US/Pacific.
The impact to us-east1-d looks has been mitigated by Monday, 2020-06-29 08:45 US/Pacific.
The impact to us-east1-c looks has been mitigated by Monday, 2020-06-29 09:45 US/Pacific.
We thank you for your patience while we're working on resolving the issue.
We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.
Services in us-east1-d have been restored. Services in us-east1-c are still being restored. No ETA as of now.
Impact is due to power failure. A more detailed analysis will be available at a later time.
Our engineering team is working on recovery of impacted services.
We will provide an update by Monday, 2020-06-29 10:30 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.
Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.
We are experiencing an issue with Cloud Networking in us-east1-c and us-east1-d, beginning on Monday, 2020-06-29 07:54 US/Pacific, affecting multiple Google Cloud Services.
Services in us-east1-d have been restored. Most services in us-east1-c are restored except for Persistent Disk. No ETA for Persistent disk recovery as of now.
Impact is due to power failure. A more detailed analysis will be available at a later time.
Our engineering team is working on recovery of impacted services.
We will provide an update by Monday, 2020-06-29 11:00 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.
Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.
6 comments
[ 2.9 ms ] story [ 28.4 ms ] threadDescription: We are experiencing an issue with Google Compute Engine in all zones in us-east1 where existing VMs may be unavailable or unreachable, and new VM creation may fail beginning on Monday, 2020-06-29 08:15 US/Pacific.
Symptoms: Existing VMs may appear as unavailable and may be unreachable. New VM creation may fail.
Our engineering team continues to investigate the issue.
We will provide an update by Monday, 2020-06-29 09:52 US/Pacific with current details.
We apologize to all who are affected by the disruption.
We expect services in us-east1-d to be recovering within the next 30 minutes, there is no eta for service recovery for us-east1-c yet.
Impact is due to power failure. A more detailed analysis will be available at a later time.
Our engineering team is working on recovery of impacted services..
We will provide an update by Monday, 2020-06-29 10:00 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.
Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.
We are experiencing an issue with Google Compute Engine in zones us-east1-c and us-east1-d where existing VMs may be unavailable or unreachable, and new VM creation may fail beginning on Monday, 2020-06-29 08:20 US/Pacific.
The impact to us-east1-d looks has been mitigated by Monday, 2020-06-29 08:45 US/Pacific.
The impact to us-east1-c looks has been mitigated by Monday, 2020-06-29 09:45 US/Pacific.
We thank you for your patience while we're working on resolving the issue.
Services in us-east1-d have been restored. Services in us-east1-c are still being restored. No ETA as of now.
Impact is due to power failure. A more detailed analysis will be available at a later time.
Our engineering team is working on recovery of impacted services.
We will provide an update by Monday, 2020-06-29 10:30 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.
Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.
Services in us-east1-d have been restored. Most services in us-east1-c are restored except for Persistent Disk. No ETA for Persistent disk recovery as of now.
Impact is due to power failure. A more detailed analysis will be available at a later time.
Our engineering team is working on recovery of impacted services.
We will provide an update by Monday, 2020-06-29 11:00 US/Pacific with current details.
We apologize to all who are affected by the disruption.
Diagnosis: Some services in us-east1-c and us-east1-d are failing, customers impacted by this incident would likely experience a total unavailability of zonal services hosted in us-east1-c or us-east1-d. It is possible for customers to experience service interruption in none, one, or both zones.
Workaround: Other zones in the region are not impacted. If possible, migrating workloads would mitigate impact. If workloads are unable to be migrated, there is no workaround at this time.