Tell HN: Rackspace apparently lost some storage volumes
1st of Oct:
"The above volume has been selected as available to clone and migrate to prevent being affected by the maintenance listed below. If the volume migration is not completed before October 16 at 22:00 CST, it will be affected by the below work. New volumes and clones of existing volumes will be placed in the new datacenter location and will not be affected by the below work."
5 hours ago:
"Unfortunately you have one or more volumes that we have not yet been able to bring back online. The following volumes are impacted by this issue:"
Then a few minutes ago:
"Following extensive troubleshooting we have been unable to bring the host server on which your volume is hosted back online and as such we are unable to recover data for your device.
You have the option to deploy a new Cloud Block Storage device."
66 comments
[ 181 ms ] story [ 3081 ms ] threadAWS and Azure might not be better, but they're definitely cheaper.
I've found that these systems are often there to make the business-minded customer feel better vs. offering up any sort of highly-competent technical support. Rackspace fits this bill for me although I haven't been their customer for close to 10 years due to how badly they've burnt me.
On the contrary, free-tier Linode support has always been incredibly technically competent and has never let me down for the 10+ years I've been with them. If anyone from Linode is reading this - you guys are awesome, and your support staff is amazing. I can't say enough nice things about them as they've done so much to support me for a solid decade!
All I'm saying is paying for support vs. not paying doesn't mean you'll have any more technically competent help in my real-world experience. It just means "you're paying for support" on paper, and typically is just an up-sell from legacy hosting providers like Rackspace or GoDaddy. Free from Linode is 100X paid from Rackspace - all depends on the company.
All of my issues actually happened with "managed" hosting 10 years ago. I was in my early-20's and still getting my feet underneath me as a Linux administrator so having a backup was something I advocated when I was provisioning our hosting.
Frankly, it was probably pretty good for career development because their support staff was clueless with anything Linux administration, and if I needed an answer quick I was my only trustworthy resource. Like - basic LAMP (Linux, Apache2, MySQL, PHP) was completely beyond them even though that was the bread and butter of their business.
I haven't sprung for "managed" hosting ever since because it just taught me that I was the best person to handle my interests... and it lit a fire under my butt to get way better at my Linux chops!
Azure has "ZERO % Annualized Failure Rate". It gives me such a cozy feeling.
I can't speak for AWS as we don't use it but as a former client of Rackspace and a current client of Azure I can assure you that Azure is better and cheaper by a country mile.
Migration sounds to me like you migrate data from one state to the other state, then delete the old state. To me, it would be logical that before you delete the old state, you verify whether the migration was successful.
However, for example, the hetzner cloud has an explicit choice between triple-redundant ceph-based storage for VMs, or hypervisor local SSD storage. That's good and explicit. I can make an informed choice about speed or durability.
Just to point out, that's still not a backup system. That's only one software bug (in your systems or theirs) away from having no working current data set. ;)
If you find yourself on Rackspace hosting I would absolutely migrate to something more mature. Personally I've had great luck with Digital Ocean and Linode for basic/affordable VPS hosting, with AWS being a great cloud provider with an incredible tool-box at your disposal. Hell, GPC, Azure... there's just so many better options.
IMO Rackspace has been on a steady decline for years. They were a choice provider back in the early 2000's when a managed/unmanaged beige box/shared cPanel/Plesk hosting was common. Back then their support was incredible. Sometime around 2010 though things started to go down hill, and I experienced more and more support issues with them. These would range from annoying all the way to "that shouldn't happen" (like losing volumes, which happened to me).
The final straw was when a dedicated server was turned down by mistake, causing a major outage for 20+ of my clients at the time. This was 10 years ago and I've never looked back. They did a pretty good job at turning me into an advocate against their services.
Oh and the Slicehost transition was crap too. That sucked to get over to a first-gen VPS provider only to have it sucked right back into the beast that you were trying to escape from in the first place...
Sorry for the mini rant. I'm still not over the PTSD that Rackspace has caused me.
They claim it's a security measure.
I've never had an issue with service or availability. But, the price per performance on the VM instances is so much better elsewhere.
Was working to migrate to DO, but now I'm fast tracking that process.
This is insanity to me. I'd leave a VPS/cloud provider so fast if they started this. Even back in the day I always had the option to delete a Rackspace Cloud instance.
Wow. Just wow.
MBA1: So we've figured out that we make a lot of money from people just being too lazy to delete a server.
CFO: Great, how do we make it even harder to delete a server?
MBA2: We could remove the delete button.....
MBA1: I don't think that's a very good idea
CFO: Then how will they delete servers?
MBA2: Fax us!
CFO: Brilliant, I'll tell an engineer to develop the web code.
Rackspace has actually been around longer than DigitalOcean. I'm not arguing that they're better or worse though because I haven't used either in years.
Thanks for the "correction"
We didn't have anything important there, migrated to GCP a while ago.
This is kinda surprising this ever happened, no cloud company allows this with nuremous safe guards in place. This is very embarrassing for RS.
Nahhh - it went public and everything is a race to the "bottom line" while exec/investors skim money off of the company vs. re-invest in an ever-changing market. Their stagnation has effectively put them back in the crib maturity wise.
It's not surprising when a company goes public and everything becomes about the almighty dollar first, and any sort of technical/support excellence second. Seen it happen time and time again in tech...
Vendor problems... amirite?
Fortunately for us, we don't have anything mission-critical on DigitalOcean. Running a similar load on Vultr (high frequency CPU nodes) simply got us a polite email asking us to switch to dedicated nodes, which we happily did.
I worked for a Rackspace competitor in the mid 2000's and can confirm that every business decision in that space is driven by short-term growth and revenue. When I worked there, the pay was terrible (although I didn't know it at the time) and the "servers" were literally rows upon rows of cheap tower PCs shoved into wire shelving. The owner of the company made millions but never invested it back into the company beyond whatever was required for horizontal expansion. (He preferred to spend it on exotic cars and vacation homes.)
There is close to zero long-term thinking in this space and as a result, a decade or two later, cloud providers are eating these guys alive.
I know it seems like it could be right out of a Simpsons episode or w/e but managers can have some pretty backwards egos. The same manager tried telling my colleagues that I had been fired and like 2-3 people were like, "Nope - actually he quit. Thanks for screwing us out of 2 weeks of project hand-off..."
According to my experience, a company that passes from an owner to another one is basically another, completely different company.
1. The second a company is bought out by PE, all the good employees run for the exits. Corporate America isn't exactly known for its empathy and compassion, but working for a PE-owned company means you can be sure the ownership gives 0 shits about anyone working at the company. I have never heard anything but horror stories of those working for PE-owned shops.
2. Given #1, it means the products of said companies always turn to shit pretty fast.
This makes me really wonder about how PE firms can really work their financial magic in the long term. Perhaps they're just better at extracting value out of companies that are already declining in the first place, so they just suck the blood dry before they discard of the carcass?
I'd agree with another comment here. It doesn't even take a declining company, just one small enough to purchase (realistically, you can't buy Apple, it's just too big) with a stable amount of cash coming in. Getting the cash coming in is hard, that takes creativity and creating value, so they need that part in the bag before purchasing the company. But if you have the cash coming in, any fool can just start cutting costs.
The stable income also helps with leverage. You can use the company you're buying as security for a loan to by the same company based on the stable income. Ta-da! That's another reason they need companies with cash coming in, leverage so they can flip bigger companies.
I'm convinced any idiot with a billion dollars laying around could do it.
Reporting the open Elasticsearch server to then was one of the most bizarre experiences of my life. Ultimately the system was exposed for about 2 weeks.
You can find some screenshots of what their backend looked like, which came in at about 3 TB daily, here: https://blog.12security.com/rackspace-bluefood-breach/
> Amazon EBS offers a higher durability volume (io2), that is designed to provide 99.999% durability with an annual failure rate (AFR) of 0.001%, where failure refers to a complete or partial loss of the volume. For example, if you have 100,000 EBS io2 volumes running for 1 year, you should expect only one io2 volume to experience a failure. This makes io2 ideal for business-critical applications such as SAP HANA, Oracle, Microsoft SQL Server and IBM DB2 that will benefit from higher uptime. io2 volumes are 2000 times more reliable than typical commodity disk drives, which fail with an AFR of around 2%. All other Amazon EBS volumes are designed to provided 99.8%-99.9% durability with an AFR of between 0.1% - 0.2%,
https://aws.amazon.com/ebs/features/
I'm not trying to blame you, just trying to figure out what happened.
Reading October 1st message, it didn't sound like they were going to clone and migrate it, but that they were letting you know something you could do to avoid the below work (which you didn't quote).
What was the description of the work?
Did you initiate a clone and migrate?
All storage will fail at some point, this storage failed during a maintenance window that was advertised two weeks out.
I'm somewhat surprised they are able to declare defeat in less than 24 hours, I would expect moving the drives to another host would at least be able to recover the data, but I don't know how they designed their system, and some systems are more byzantine than others.
How is now being able to bring a server online connected to the storage that server provides?
Surely they'd remove the physical storage devices from the dead server, put them in another one, and bring it up?
... unless they're hosting their servers with a third party they can't access? eg AWS or similar, which would be kind of ironic. ;)