Potential GCP SSD Data Corruption Issue
Subject: Urgent: Action Required - Identified Issue with GCP SSD Disks Affecting ScyllaDB Clusters
Dear Customer,
We trust this message finds you well. We are reaching out to bring your attention to a critical update concerning your ScyllaDB servers hosted on the Google Cloud Platform (GCP).
Google has recently identified an issue affecting certain SSD disks that may have implications on the data reliability of your ScyllaDB clusters on GCP. Recognizing the urgency of the situation, we highly recommend taking immediate action to minimize any potential disruption to your operations.
To address this promptly, we advise contacting Google Cloud Support for personalized assistance. Follow these steps to assess and mitigate the impact:
Reach out to Google Support to verify whether any of the cluster nodes you are operating on GCP are utilizing the affected SSDs.
If Google identifies a susceptible node, our recommendation is to perform scrub and removal of the corrupted sstables, (contact our support team for details), then bootstrap a new node to replace the potentially compromised one. Subsequently, decommission the affected node. During this process, any existing corruption will be identified, and appropriate steps, such as using removenode or deleting corrupted SSTables, can be taken with our technical support team. While there is a slim chance that the replacement node may be susceptible, Google has assured us that the probability is low. Unfortunately, there is no proactive way to determine susceptibility.
Google is actively deploying a fix, although an ETA for completion has not been provided. While the probability of encountering a problematic node is expected to decrease over time, we advise addressing this issue proactively.
Already reached out to Google support, but past experience would suggest they will not be helpful. Anyone have any additional information about this issue?
0 comments
[ 4.4 ms ] story [ 11.7 ms ] threadNo comments yet.