Hetzner Cloud – Data loss incident
Dear Customer, Unfortunately, we have to inform you that there was a data loss incident that affects a small amount of your snapshots on Hetzner Cloud. All snapshots you create are stored on our highly available storage systems. The snapshot contents are distributed over multiple internal servers and data is stored in a way that allows up to two separate disks to fail without impacting data integrity. This means the snapshot can still be accessed, even if two disks fail at the same time. Due to a recent, very unfortunate series of events in one of our clusters, multiple disks failed in short succession and caused a small number of snapshots to become unavailable. We immediately tried to recover the affected snapshots but unfortunately the data is lost and we have exhausted all our options.
Affected snapshots in your account: XXXXXXXXX
The snapshots have been removed from our system as they are no longer accessible. We sincerely hope this doesn’t cause too much trouble for you; we know losing data is the worst-case scenario. Also, we have added 20€ as Cloud Credits to your account (valid for one year). While we know that this will not bring back your data, we still hope that you will accept the gesture. In response to this we will re-evaluate our snapshot cluster data replication strategies as well as our strategies for replacing disks and rebuilding redundancy after replacement.
Best Regards, Hetzner Cloud
98 comments
[ 6.2 ms ] story [ 208 ms ] threadYes it's offtopic, but that's one really frustrating thing. Maybe someone here has a solution.
1: https://docs.hetzner.com/cloud/servers/iso-installation-gate...
The problem is that preseed is a very minimal linux environment and either relies on DHCP to work, or to use the usual network config DHCP sets (address, netmask, gateway). Arbitrary routes are not a possibility there. And ways to run abitrary commands to just <ip r> something come after the network initialization, which fails in that setup. The same holds for the kernel network bootstrapping parameters, I haven't found a way to add an arbitrary route there yet.
And for some reason, kickstart is able to handle this with it's centos/RHEL base.
Aside from backups, the snapshots can also be used as base images to provision new nodes though, losing those could indeed be inconvenient.
Edit: people use snapshots to compliment backups too, you can do snapshots hourly and back a snapshot up daily. (They don't act as backups, more like "oops" revert solutions, humans are more likely to fail than machines)
If you're using VMware or Hyper-V the backups rely on a dirty "bitmap" of which blocks has been written since last backup so it only has to pull those from the system, this doesn't work with more snapshots for some reason. Enterprise backup solutions will just pull the entire machine and compare what's changed anyways, but you usually get a warning about it. So grumpy admins
I believe EBS snapshots are equivalent to you temp-snapshot, incremental/deduplicated backup/delete temp snapshot combination. Logically it's still a snapshot, because it contains all data of the volume at a specific point in time, but the cloud provider abstracts away the details of how it achieves durability.
> Snapshots cost €0.0125/GB/month (incl. 25 % VAT)
so €20 would cover a month for about 1.6TB of snapshots. Sounds pretty reasonable to me. But I guess it depends on how much data OP lost.
For simple websites/blogs etc. you don't need much.
I host 12 different sites on this setup. And I could put up more.
I get around 0.5-1m views combined per month. They are mostly from one time zone, so traffic is not that spread out.
That's without cloudflare in front. I used to have it, but a lot of people said they are getting captched to death.
People* caught up in AWS bubble often don't realize how expensive it is. However I do use AWS at work extensively - right product for right occasion.
* Not saying that's true for you. But I notice that a lot of people I work with are surprised, when they learn this.
The €20 of credit made air accelerate out of my nostrils.
Also they made sure you knew it's only valid for one year!
[1] https://www.youtube.com/watch?v=5eo8nz_niiM
Yeah I've seen way, way worse. Stuff like power cables zip tied to network cables and fibre lines. DUST. delerict hardware just sitting everywhere.
This is nice. What are you looking for, IBM dudes in white shirts running around?
depends how localized the fire extinguishers are and how the power is distributed which is hard to see from the video
not sure what you expect, using 1-2U boxes or blades or something wont change the setup that much, it pretty much boils down to 'how many machines will die in the same time on power/fire' and how can we cool them as cheap as possible.
Ever been in a firewall/router manufacturers R&D/QA lab room? Those are so, so much worse. This is heaven by comparison.
Never had any trouble with their vps offerings, but I'm not liking it.
There are various options here but the one I happened to have read up on (and then contributed documentation to make it easier to use safely) is Restic with Rest-server in append-only mode. It's not perfect (e.g. no compression yet, only dedup) but it's pretty good. If you're an individual or small–medium business, put your favorite raspberry pi equivalent in a closet and invest a hundred bucks in a hard drive big enough for the important documents, pictures of loved ones, etc.
A single drive shouldn't be your only backup, just don't put 100% of your trust in magic cloud. See also Atlassian this week: it may be down, but if you need to access the data of ticket xyz for a lead this week and you have a local data export, you can find the needed data.
For durable backup I'd recommend storing objects in S3/Glacier. If you need to image an entire volume, make an AMI, which is stored in S3. Much more durable and targeted than a snapshot, and in the case of S3 objects, gives you benefits such as logging and version control.
I'm curious now as to how the EBS Snapshot was lost. If I had to guess, I suspect it had something to with that linkage back to the original volume. I maintain that the best backup has no dependencies on another resource existing.
We now run an instance in another region and account and rdiff-backup to it.
[1] https://aws.amazon.com/fr/blogs/aws/new-amazon-ebs-snapshots...
It seems odd that they'd be quite so vague and circumspect about it. Why not just say what it was, so we're not all left to speculate, perhaps for the worst?
One of my favorite is, if you want to store three copies of something on a lot of disks, randomly select which three disks you want to send the copies to so that each bit of content is stored on three random disks. It superficially seems very appealing; I have 3 copies of everything, nothing is likely to go wrong!
However, what you've actually created is a situation where if any three disks go down, you're guaranteed to lose data because as you scale up, the probability that those three disks were the three randomly-chosen disks for some set of data goes to 1. The probability of three disks going down at some point also likewise trends to 1 as you scale up.
Not saying that was the case here, just sharing it as one of the well-known pitfalls I find particularly interesting.
If instead, you break your disks into N/3 groups of three, and store each file three times in one group, you only lose data when all three disks in one group fail simultaneously. If you're lucky, you can lose 2/3rds of your disks before you lose data.
Edit to add:
If you're using mirroring, it's low cost to do mirroring by groups although uniform distribution can help with hotspots. With uniform distribution, even if one device has two popular files, the other copies of those files are likely to be on devices with only one popular file. With grouping, if one group gets two popular files, all devices in the group have two popular files and therefore elevated use.
If you're using a parity scheme (raid6/raidz2/some other erasure coding), the cost to do grouping is more apparent. Each group has the +X parity cost; If you did raidz2, and had 45 drives in a case, you could do one set of 45, and get 43 drives worth of storage, or three sets of 15 and get 39 drives worth of storage, or five sets of 9 to get 35 drives worth of storage, etc. In each of the smaller sets, you still can lose data with the first three failures, but you need an increasingly large number of failures to guarantee data loss. Also, the size of the data lost when there's a loss is smaller.
[1] or at least, the third disk fails before either of the first two failures is repaired
Edit: “proof” by example…
9 disks, replication factor of 3. 9 choose 3 = 84 ways to allocate a file’s replicas across the disks. If 3 disks fail, on average 1/84 of the total data is lost (assuming >>84 files).
Versus… 3 groups of 3 disks. If 3 disks fail, the probability of the second disk failure being in the same group as the first disk failure is 2/8, and then 1/7 that the third failure is also in the same group. (2/8)*(1/7)=1/28.
So a 100% chance of losing 1/84 of the data versus a 1/28 chance of losing 1/3 of the data. Same total expected amount of data lost.
It is generally preferable in reality to take the second choice because additional realities intervene to give you time to deal with the problem, allowing the existence of fairly highly-reliable systems, whereas those same additional realities mean that in practice, you will lose data with some frequency if you select the uniformly random result.
Sometimes the better systems still catastrophically lose data, that is indeed inevitable as you point out, but what it takes to knock out that system would have result in lost data in the bad uniformly-random system, too, so in practice it's not much of a tradeoff.
[1] https://www.hpe.com/us/en/services/sas-ssd-advisory.html
Also a favorite of mine is power cycling a machine. With enough disks, chances are high that a couple won't come back after they spin down.
I have a local backup box and a remote one and they both sync stuff in from rsync on a cron job from multiple servers. Works great. In case of catastrophic failure (Which has happened a few times), putting my eggs in more than one basket has been extremely successful.
Highly available right until they are not. Nice to see that your data is worth 20 bucks to Hetzner. I always liked them but this is a bit rude to put it mildly.
S3 for example is designed for an availability such that it can fail 1 in every 10,000 requests on average and for a durability that can lose one object for every 100,000,000,000 objects you have each year on average.
For data you can’t afford to loose, please, for the love of ${DEITY}, don’t store it with just one vendor. You never know what happens.
https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
For my cloud data, I sync it down to a local server and then sync that to removable offline media.
This seems like an excellent approach. Personally i use BackupPC which internally uses rsync and allows for incremental backups, compression and deduplication of said backups as needed, then copying the data over to additional HDDs with cron.
Works pretty decently on a slow residential connection, since it all happens in the background through WireGuard and one of my homelab boxes. Plus, BackupPC allows to test restoring backups with a single click, on a per-file or per-directory basis.
Of course, internally the solution is a bit of an Eldritch mess with Perl and whatnot, so other pieces of software like Bacula might be more popular in comparison.
Make 3 copies of your systems (just data if you can rebuild infra within your RTO)
Use 2 different media (compress then encrypt)
Move 1 copy off-site. (print the encryption key and store that in a fireproof safe)
(Now, that’s not strictly true any more for JPEGs thanks to projects like lepton and JPEG XL which can indeed compress already-compressed JPEGs losslessly).
Personally, i use regular HDDs for storage of data (SSDs are out of my price point for anything but boot drives) and having fewer larger files seems to result in slightly better copying performance than having more smaller files. Though that might also have to do something with the file system and method of copying as well.
1. Doesn't provide a meaningful size tradeoff for /family photos/
2. Doesn't provide a way to recover the data if it was damaged, at best you get a notification if a decompression routine failed the verification of uncompressed data
10 years ago I had a meaningful dedup ratio, nowhadays it is cheaper for me to have another 50Gbs than bother eith a dedup
My setup has 4-2-2 and using 2 different backup implementations for the remote, so bugs in them wouldn't be corrupting entire backups unknowingly at the same time.
Apparently the 1 local and 2 remotes are having generations of incremental backups as well.
That could be three copies at 3x the storage cost, or it could be a RAID6 or raidz2 style system where the storage cost is essentially two disks out of the storage set size (which we don't know).
You could certainly increase the required failure count by increasing the parity, but if the problem was something like all the disks in the storage system hit the same fatal disk firmware bug at the same time (as speculated elsewhere in the thread, and is unfortunately plausible), then that doesn't really help much.
I agree that this would be good as a last-ditch backup for stuff like Hetzner cloud snapshots, but the primary storage for those snapshots probably has to be ordinary RAID.
Fwiw I have a bunch of data in Hetzner Storagebox and have gotten several notices in the past few months that Storagebox would be temporarily offline for maintenance, which I assume meant raid rebuilds and/or drive swaps. The most recent of these was an "urgent" maintenance with less notice time than usual. It hasn't caused me any inconvenience, but I wonder if they have suffered a spate of drive failures recently.
They communicated what went wrong, how it happened (in a nutshell), apologized and stated how they're planning to do better.
They even threw in 20€ which when compared to the pricing of their snapshot storage is more than fair.
Shit happens, things go wrong. If you lost very important data because you only stored it in one location, then you're equally at fault. Especially when taking into account that they are a rather cheap service.
What more do you want?