Yes, my servers and all associate services are down: DNS, email, websites. It's a major outage at the whole Newark datacenter, which is their main one, no less.
Thankfully all my nodes are back up. But the DNS server as down so long that Namecheap registered it as a personal DNS server, so I am working with them to get it back up.
These are secondary effects of outage, not Linode directly, but caused by the outage itself.
I woke up to a few hundred messages from Icinga - thankfully my phone is on do-not-disturb overnight. Some of my servers in Newark are up and responding, some are not.
Happy Sunday! Cleaning up the automatically-created maintenance/alert tickets generated by this is going to be a fun time.
"The issue is related to heating/cooling complications in the data center due to a power outage . The power outage has been fixed and we are working quickly to bring our services back online."
They said it started with a power outage, but I thought Linode data centers ran on UPS and didn't have power outages? And, given the length of this outage, the problem seems more serious.
Over 22 hours of downtime for the one VPS I have in that region.
My infrastructure is redundant and spread out among hosting providers and DCs so there's no real impact, but I'm pretty sure this is the longest outage I've ever had with any provider. And the communication level has been so dissapointing. 4 hrs to say it's a power / HVAC issue? Updates that basically just say we're still working on it since then.
It is still ongoing and impacting multiple services and regions. Looking into the status page history, it seems they're facinga different issue on Kubernetes for the last couple of days
Of 60-something Linodes in Newark across a few accounts (we don't use LKE, Node Balancers, etc)
- Many came back up yesterday. Most of the rest came back up this morning.
- All but two are back online. One of those is "Powered off" but can't be turned on because "Linode busy". The other is online but unreachable, same behavior as most of them during the outage.
- Three required me to put them in Rescue Mode and run fsck.ext4 -F /dev/sda to get them back online.
Thankfully all my nodes are back online and accessible via SSH but there are currently secondary effects due to the length of the outage that I am dealing with, such as my DNS server being deregistered with Namecheap from serving as a personal DNS server (their name for that feature). The consequence of this is also that my email is not currently work, since that affects the hostname the email server operates under serving multiple other domains. I am still grateful that it seems I suffered no data loss, thank God!
Also, I've taken it upon myself to PROPERLY implement a better, redundant backup strategy (since I was mainly relying on Linode's service, but now I feel I should go beyond that). I am using restic backing-up to a Backblaze bucket via the S3 interface. Nice thing is I can put all hosts into the same bucket and restic will organize by host but still full deduplication support. Not sure how much that'll net me but it's nice to have.
18 comments
[ 2.7 ms ] story [ 42.6 ms ] threadThese are secondary effects of outage, not Linode directly, but caused by the outage itself.
Happy Sunday! Cleaning up the automatically-created maintenance/alert tickets generated by this is going to be a fun time.
"Mal de muchos, consuela de tontos."
"The issue is related to heating/cooling complications in the data center due to a power outage . The power outage has been fixed and we are working quickly to bring our services back online."
My infrastructure is redundant and spread out among hosting providers and DCs so there's no real impact, but I'm pretty sure this is the longest outage I've ever had with any provider. And the communication level has been so dissapointing. 4 hrs to say it's a power / HVAC issue? Updates that basically just say we're still working on it since then.
- Many came back up yesterday. Most of the rest came back up this morning.
- All but two are back online. One of those is "Powered off" but can't be turned on because "Linode busy". The other is online but unreachable, same behavior as most of them during the outage.
- Three required me to put them in Rescue Mode and run fsck.ext4 -F /dev/sda to get them back online.
Also, I've taken it upon myself to PROPERLY implement a better, redundant backup strategy (since I was mainly relying on Linode's service, but now I feel I should go beyond that). I am using restic backing-up to a Backblaze bucket via the S3 interface. Nice thing is I can put all hosts into the same bucket and restic will organize by host but still full deduplication support. Not sure how much that'll net me but it's nice to have.