Backblaze restore for Personal Backup is awful

The problem is that this is very expensive if you're not in the US. I'd have to pay for customs + return shipping, which would add up to much more than the cost of a new drive.

Would you mind posting the exact information you gave them, and their response - redacting sensitive information of course.

I had a Mac which crashed and that had a whole bunch of files go missing. Backblaze didn't want to do backups with a huge subset of my disk files gone without having gotten notification messages from the file changes API that this stuff should be missing/was deleted. So they safety froze my machine.

It's cool that Backblaze notices this. But yes, the question of what to do next isn't clear. Does one just start a new backup? Does one scour the backup looking for anything that may be missing on your machine, now? Etc.

One thing to notice about the super-affordable Borg plan is that it doesn't include free ZFS snapshots. My understanding is that you can have the SSH key used by the host to push its backup restricted to only Borg, and only append-only, within its repo... but if there's another way to access the ZFS (e.g., with an unrestricted SSH key), the Borg repo could be deleted. And then you might really want automatic ZFS snapshots as an additional layer of protection.

HashBackup (I'm the author) has a lot of ways to verify backups. For B2 uploads, it generates a SHA1 hash before sending the data, stores that in a local database, sends the data with the hash, B2 generates an independent hash of the data, and compares it to the one HB sent. If there is a mismatch, an upload error occurs and the upload is retried.

On download, B2 sends the data and SHA1 hash (I'm assuming it verifies the hash during erasure reconstruction also), HB generates a SHA1 hash for the received data, verifies it against the SHA1 B2 sent, and verifies it against the SHA1 stored when the file was uploaded.

After the upload, files can be verified by:

- downloading the complete file as above, then verifying every block in the file - getting the file size and SHA1 hash from B2 and verifying those against the local database (no data download) - downloading N random samples from each file and verifying the block-level SHA1 for each sample - downloading a small percentage of the files every day to verify the backup over time (this can be done with whole files or random samples)

If a problem is detected and you have other copies of the backup on another storage service or locally, HashBackup's selftest will fetch all copies of a file to reconstruct the original backup data to correct any errors, then upload the corrected file to all destinations.

Can you elaborate on this part?

Then I realized that S3 Glacier and Deep Archive were even less expensive than B2. I took a bit further of a look and found that Glacier/DA files have some fairly chonky metadata that must be stored in normal S3, and for a lot of our images the metadata was larger than the image in question. So Glacier/DA would increase our storage costs. Over all it probably wasn't a money-saving situation.

The ideal use case is to bundle those up into a tar file or something and store those large files, and manage the metadata and indexing/access ourselves.

So, using rclone to copy 11TB of data to B2.

I need a solution for backing up the stored states for my 100 trillion digit PI calculation efforts.

The question this thread seems to be raising is... do you? Do you really? Are you sure?

I have a ZFS based NAS. And periodically do a incremental backup (zfs send) of the entire dataset, encrypt it gpg and pipe it straight up to S3 deep archive. Works like a charm.

The catch with S3 deep archive is if you want to get the data back... It's reliable, but you will pay quite a bit more. So as a last resort backup, it's perfect.

Reading the explanation in the reddit thread, that's not the impression I got at all.

1. If your computer exploded, your backup integrity would not be compromised

2. If gremlins in your computer did mess with the file, your backups could be compromised. That sounds bad, until you realize that gremlins in your computer could also compromise the executable to do other things that could compromise the backup, (eg. telling the server to delete existing backup data because the retention period has passed or whatever, or simply uploading bad data and waiting for the retention period of 30 days to pass). Moral of the story: if the computer doing the backup can't be trusted to operate correctly, all bets are off.

I'd be more interested in the skill of the developers themselves than the CTO.

> This quote especially seemed childish and immature to me:

>

> > "And this one makes me actively angry, because both Microsoft and Apple will happily throw away portions of your files and not tell you about it"

>

> No, this is NOT Microsoft's or Apple's fault. It is 100% HIS fault for not understanding what's going on. Even if a file flush is correctly requested

Several times you seem to jump to the assumption that I don't understand fsync and disk flushing and that is the core issue. You aren't understanding what I'm criticizing. Here is an example of what bothers me:

You take a picture at your wedding, and you store it on your hard drive. You like the photo, it means a lot to you, and you use it as the background for your desktop FOR FIVE YEARS. You have rebooted hundreds of times, and it's always the background for your desktop. Then one day 5 years after your wedding, you reboot your laptop, and it seems to take a little longer to boot, and then after you sign into the laptop half the image you use for your desktop background is scrambled. The middle of it looks like dirty snow. You didn't get any reports of any issues from the OS manufacturer, but now one of your photos is corrupted.

This isn't because the software that wrote the photo 5 years earlier forgot to flush the picture to disk. It just isn't. Behind the scenes, as your laptop was booting from an aging drive, it probably encountered some issue, and it went about fixing the problem as best it could - which I have no problem with. My issue is the drive lost some data, and if the OS manufacturer would let you know this occurred you could take useful actions like order a new drive, prepare a restore from a few weeks ago before that issue occurred, etc.

> No, this is NOT Microsoft's or Apple's fault.

It isn't their fault that the drive is going bad, I agree. Drives go bad, that's why we have backups. My issue is the OS manufacturer try to cover up too much, keep too much hidden from the user, and didn't let the user know data loss has occurred (or might have occurred). And yes, I hold them accountable for not telling customers what is going on. It isn't anybody's "fault" that it occurred, but there is a responsibility to let customers know about it so the customer can take appropriate actions so they don't lose data (or more data).

I try to write incredibly paranoid software. Part of the reason is that is the "average" environment the Backblaze client runs in is more unstable than what most software developers are used to. The whole point of backups is to run when the computer is going sideways, it has bad RAM, it's losing disk sectors, or a customer's cat likes sleeping on the keyboard because it's warm, and the fans are clogged with cat fur. And because the family has teenage children that don't know about computer security problems, they download and install unstable junk from all over the internet because why not? That's the environment my software runs in, and I take my job of trying to protect my customer's data very seriously.

Disks in your friend's basement are more durable when not attached to a network and sitting in a Pelican case instead.

One of the things a personal backup might want to survive is a long period of incapacitation and or neglect or higher priorities. The cloud, because it takes a dependency on credit cards, doesn't do that. A spinning hard disk in a box tends to and if it doesn't spin up there is still the physical media from which the data can be recovered.

S3, etc. are suitable for business records. You can insure against their loss. If you don't have money to pay the bill you're probably not really going to need a backup much longer. etc.

To put it another way, there is a decades long track record of success with physical media. There is about a decade and a half of total experience with the cloud.

A clean solution would be to have stub executables that link to a central libClang.dll/libLLVM.dll (-DCLANG_LINK_CLANG_DYLIB=ON), but this is not supported under Windows because a DLL can export at most 2^16 symbols. Some work would need to be invested to make process launching on Windows work differently than on other platforms, but then disk space is not that much of a problem.

>

> This seems needlessly dismissive. I feel like they definitely know that you can execute the same binary multiple times.

I'm the programmer at Backblaze that made the copies on purpose, I wrote some extra code to do this, and it's meant to help us debug certain things. Yes they are identical, the installer only ships with one copy of the executable, the installer then makes the copies on purpose. I get to explain this from time to time. :-)

In Windows when you want to know what is going on behind the scenes, you can bring up Task Manager and look at the different names of the different processes that are running. On the Macintosh this is called Activity Monitor, same sort of thing. The different names for the executables are for different "threads" which have different roles. Backblaze is multi-threaded to get higher performance.

The parent coordination process is called "bztransmit". But when doing the actual transmission it spawns the bztrans_thread01, bztrans_thread02, bztrans_thread03, etc.

So BEFORE I made multiple copies of the executables with different names, a customer would say "bztransmit is hung" or "bztransmit is using up too much memory". There was very little visibility into this. But now that I made multiple copies with different executable names, when the same customer says "bztrans_thread03 is hung". Or they say "bztransmit is using too much memory". We immediately have narrowed down what to look at.

Here is a screenshot showing what "Chrome" looks like to me in Windows, and how it compares to how Backblaze's bztransmit looks like to me in Windows: https://i.imgur.com/KOJHJ9Q.jpg In that screenshot, you can see there is the "main thread", and "worker threads". Meanwhile chrome is just one big list of processes all named the same thing (see the screenshot). I prefer the Backblaze system, but I understand it upsets some customers that prefer the chrome experience.

That's it. It's not some huge mystery.

One question asked here was do we know you can launch the same executable twice? Yes, and we do that. The bztrans_thread05 is launched for thread 05, thread 25, thread 45, thread 65, etc. It's THREAD_NUMBER mod 20. Here is what it looks like to hit 500 Mbits/sec upload speeds, this isn't photoshopped, it's a real screenshot on my development computer: https://i.imgur.com/hthLZvZ.gif

Another question is: why not use hard links or symbolic links? That's the only real optimization possible here, everything else was on purpose. The answer is not an excuse, it's just an explanation if you are curious. The software we develop at Backblaze is cross platform, so what we like to do is make the most general form first that will always work, then if customers complain or we want to refine it we special purpose code in per file system or per platform. The most general thing to do is make full copies. We could then go on to make links on the Mac WHEN POSSIBLE and the equivalent on Windows WHEN POSSIBLE, but it never became a large priority. The reason I can't use one technology is we support several file systems, and not all of them are the same or support the same technology for links.

Every feature we have is the result of prioritizing it over working on other things. Until recently, we did not have a lot of funding or an infinite supply of programmers, so we had to choose what order to implement each feature in. I'm not saying we got all the priorities correct, or that we did things in the correct order. I don't really even think there is one correct order. For example, some individual home user customers prefer saving 180 MBytes of their valuabl...

I have 26 terabytes on my NAS, so with 2 copies I'd probably use $800 in HDDs? I could swap to a new 10TB HDD every few hours just like a tape, attach group labels just like a tape, it'd have similar speed, and I'd probably write new images to the same HDDs every so often.

The word "just" sounds like you think it's less work, but it sounds the same to me. FWIW, I did roll from HDD to HDD for years, so I'm familiar with the concept.

Put them in safety deposit boxes at the bank, give copies to a friend, bury them in your backyard, and - yes - store a set or two in the closet.

Because they're encrypted, there is the issue of the encryption key needing to be managed. But that's the only issue.

[1] https://news.ycombinator.com/item?id=29535055

I’ve also had issues with downloading my 2TB of data the Backblaze has. Takes about 3 or 4 days, if you’re successful. With retries and failures, wall time was about a week. And I have a 1G/1G internet connection. And doing the work to break it into 500GB chunks sucks.

I’d like to back up more of my files but I worry I would not be able to get it back.