RClone is more about copying folders/files between storage systems, it doesn't pack files together like other tools. I use Rclone for larger files like pictures or media files that don't change or move around, pure sync to targets. You can use cron to run it if you want.
Hey! Thanks for asking. On mobile right now so this isn't comprehensive but a few advantages jump to mind.
1. Compression/deduplication. Your data will (generally speaking) take up about half the space it did before even including the multiple file versions.
2. Automation. You won't have to manually do anything after you configure a schedule once.
3. Management of multiple backups.
4. A really nice native GUI
5. All the advantages that backup with file versioning has over sync (which is what it looks like you have). Safe from ransomware (you'll have an older version always even if your infected ones get synced accidentally), projection from human error by being able to revert to older versions, etc
It's generally faster to backup [1], way faster to restore [1], backups use less space [1], has a native app for each OS and usability (subjective I know but give the two a shot yourself) is better imo.
How does it scale to larger datasets (TBs of data)? For example, Duplicati is much slower with large datasets compared to small/medium datasets, because of it's database. Once, the database grows too large (eg. multiple GBs), it becomes too slow to be usable.
Which version of Duplicati was used in this benchmark?
Duplicacy would have been a better comparison since it and blobbackup are backup tools, whilst Rclone is built for sync and has supporting tech like mounting for that usecase.
Nice work!
If one were to buy this and want to back everything up from local device(s) and cloud accounts back down to a local NAS - supported use case?
Not the parent, but if you're ever looking for features to implement:
I've been thinking about this for quite a while, and one reason I've been unhappy with any sort of cloud backup solution is that they're usually too slow for laptops, unless I leave them awake while I'm sleeping.
An ideal backup solution for me would be laptop > NAS (or desktop) > cloud. A backup over the local network should be magnitudes faster, and the always-on NAS can handle a slower, versioned backup to cloud storage.
Well you could configure BlobBackup to do something kinda like that.
I know some users run backups to their NAS every 6 hours and then run a weekly backup of their NAS to the cloud using whatever NAS software it came with (alternatively you could sync it through your laptop).
Scenario == backup everything from local devices + cloud accounts/storage (e.g., iCloud, Dropbox, etc) all back down into a locally owned NAS. Use case == customer can use whatever backup whenever on whatever device, and rely on this service to hoover it all up and sync it to a big local NAS so it's all there when needed (and optionally allows deleting various cloud sync accounts or restoring/syncing cloud accts from big local NAS if needed)
The one thing that finally works for me for backup is restic (https://restic.net/). I use linux and set my home dir to a different partition, Run a daily script that uses restic to backup to backblaze. Smooth as butter and hopefully I don't need to use it. Curious what this uses under the hood, wouldn't be surprised if it was restic.
> Blob file names are the SHA256 hash of the unencrypted blob contents
Obviously this is needed to make block level deduplication possible, but it also means that for this "end to end encrypted" product someone can still tell whether or not you own a given file. This is probably fine for most people but worth calling out.
Good catch. But the SHA hash is salted per backup destination though. So you can't figure out the content from the hash alone.
EDIT: I should make note of that on the docs page really. It currently gives the impression that I'm just straight up calculating the sha of the content.
I figured borg is stable and does the job but others, including restic have problems like memory usage concerns and I'm just a single person shop and don't have any huge assets to backup and already seeing restic/duplicacy choke consuming GBs of memory.
What are the tools that are performant and stable?
I'd like to at least use another implementation besides borg in case a bug corrupts the data.
Encryption, retention policies, incremental backup without periodical full backup and acceptable performance (that people don't fill GitHub issues with performance problems) are requirements and deduplication is a plus.
I hope it gets to the point not to consume several GB soon enough. Also the pricing got too greedy and I'm not sure if I want to pay that.
It used to be per user but now it's per machine and it's even asking you to pay $50/year per machine even for cheap instances that are $60/year and when you have 20 small machines to look after, the pricing looks a bit too unfair.
That's a heavily requested feature that's in the works. I was focusing on platform independent features in the beginning but this will come eventually.
It is surprisingly hard to come up with a simple and concise UI even if it may seem all too trivial in retrospect.
Given that this one uses secondary control labels that are verbatim of Bvckup's, has the same layout and opening screen, and nearly identical look and feel, this is certainly not an independent effort.
Yeah full disclaimer. I started working on it before I saw bvckup2 and then at some point came across it and then could not unsee it design wise so you're right. Some things are very heavily inspired off their awesome design.
Outside of the main window though, lots of other things are quite different and completely independently arrived at. Bvckup2 doesn't have third party storage support so everything else requires a fundamentally different UI to work in BlobBackup. Look at the settings windows and the menus for example.
Well generally speaking, backups will use about 37-40% of the original size. BlobBackup uses variable length and fixed length block level dedupe and zstandard compression. So you can optimize by adjusting block sizes, changing compressing levels, etc to get things to fit on smaller storage.
First time software seller here so I'm a little slow on some things but rest assured, I'm gonna have a privacy policy soon. The EULA is there when you download the software.
BlobBackup doesn't store any user data anywhere other than the user's computer and their backup storage though (and this is encrypted).
I found this comparison between borg and restic, and it seems like most of these apply. (https://stickleback.dk/borg-or-restic/) seems like blogbackup and restic are very similar. Wish there were more docs though.
borg has been around for a while and I hear it's stable. They also have change logs that mention any corrupting bugs, which is getting quite rare for only on few occasions. I'm migrating from restic to borg.
The downside of borg is that the target is only sftp and not directly to any of S3 and some other targets but there are online services that accept sftp.
On the other hand, restic is new and while this can sync towards many targets, the use of memory gets crazy after a while that my machine choked out of memory and these kind of stability problem is something not mentioned in the docs until you use it.
I've tried several others like duplicacy but this also has a memory problem and both restic and duplicacy seem to have trouble fixing it quickly.
When it comes to backup, I'd like to use more than 1 implementation toward different remote locations in case 1 of the implementation has corruption bugs to render the backup useless.
Yeah the main difference between borg (and their GUI vorta) and a lot of the other options (BlobBackup included) is the non sftp storage support.
I would have used restic directly as a backend for BlobBackup had it not been for the awful prune operation. Downloads a ton of data everytime, prone to errors, and has to repack blobs. Compression would be nice too but they have nice dedupe so it would have been workable without.
> target is only sftp and not directly to any of S3...
This used to be (as in, from 2008 to 2018) difficult, one of my past companies made a good deal of money by offering an sftp gateway to S3. But AWS continues to do a great job commoditizing their complements, and tackled SFTP back in 2018 when they launched...
“... AWS Transfer for SFTP, a fully-managed, highly-available SFTP service. You simply create a server, set up user accounts, and associate the server with one or more Amazon Simple Storage Service (S3) buckets. You have fine-grained control over user identity, permissions, and keys. You can create users within Transfer for SFTP, or you can make use of an existing identity provider. You can also use IAM policies to control the level of access granted to each user. You can also make use of your existing DNS name and SSH public keys, making it easy for you to migrate to Transfer for SFTP. Your customers and your partners will continue to connect and to make transfers as usual, with no changes to their existing workflows.”
“You have full access to the underlying S3 buckets and you can make use of many different S3 features including lifecycle policies, multiple storage classes, several options for server-side encryption, versioning, and so forth.”
Maybe you don't care if you're a larger company but the pricing is insanely higher than competitors.
Just to turn SFTP on at AWS, you need to pay $0.3/h ($216/mo) and it even costs to upload not just for download at $0.04/GB. If you upload 100GB/day, that's also about $120/mo and if you're storing about 1TB you're paying like $350/mo for such little storage you get.
Who wants to pay for upload?
At this point, why don't you just run EBS backed EC2 yourself?
With that kind of pricing, you'd get like 15TB of space at rsync.net and it doesn't leech users for bandwidth and on every parameters as AWS does. (If you use the borg discount, you get 40TB+ at $0.008/GB!)
And at what benefit does it bring for the price difference? I see no point in using it. Maybe people don't care to mention than not knowing it.
Have you thought about adding CLI support to the roadmap? If you were to add that it would be an instant buy for me, as well as many other admins, I'm sure.
Great product though! Always happy to see projects like this
One more question: Has BlobBackup been tested with hundreds of thousands, to millions of files in a single backup archive? I've had issues with backup software in the past choking during backup/restore with a very large number of files.
The limiting factor with BlobBackup will be the number of files the storage backend can handle in one directory. Generally speaking BlobBackup will create slightly more files in your destination than your original source.
With object storage like s3, b2, azure, wasabi etc, this just isn't a problem since you can effectively have as many objects in one "directory" as you want.
With local, nas or hard drive storage, you'd be limited by the number of files per directory the file system allows (4.2 billion for NTFS for example). So you're probably fine here too.
Google drive (which isn't recommended for large backups with BlobBackup) has much stricter restrictions on files per directory. So really it depends on the backend but for most use cases it isn't a problem.
BlobBackup has been tested on millions of files backed up to NTFS and object storage yes.
66 comments
[ 0.21 ms ] story [ 176 ms ] thread[1]: https://rclone.org/overview/, [2]: https://rclone.org/crypt/
1. Compression/deduplication. Your data will (generally speaking) take up about half the space it did before even including the multiple file versions.
2. Automation. You won't have to manually do anything after you configure a schedule once.
3. Management of multiple backups.
4. A really nice native GUI
5. All the advantages that backup with file versioning has over sync (which is what it looks like you have). Safe from ransomware (you'll have an older version always even if your infected ones get synced accidentally), projection from human error by being able to revert to older versions, etc
[1] Benchmark on open data set. https://blobbackup.com/10_gb_benchmark.php
Which version of Duplicati was used in this benchmark?
The largest backup a user has reported using this with has been 5 tb and it ran without a hitch.
I'm pretty sure it was the latest duplicati for the benchmark. Maybe the second latest. I forget.
What is your use case for this?
Do you simply mean is it possible to restore backups to local? Or a reverse backup operation from cloud to local?
I've been thinking about this for quite a while, and one reason I've been unhappy with any sort of cloud backup solution is that they're usually too slow for laptops, unless I leave them awake while I'm sleeping.
An ideal backup solution for me would be laptop > NAS (or desktop) > cloud. A backup over the local network should be magnitudes faster, and the always-on NAS can handle a slower, versioned backup to cloud storage.
I know some users run backups to their NAS every 6 hours and then run a weekly backup of their NAS to the cloud using whatever NAS software it came with (alternatively you could sync it through your laptop).
What tech stack is this application built in?
Its built in a combination of C and python. Using zstandard for compression and openssl for crypto. Qt for the GUI.
[1]: https://blobbackup.com/docs.php#data-format
Obviously this is needed to make block level deduplication possible, but it also means that for this "end to end encrypted" product someone can still tell whether or not you own a given file. This is probably fine for most people but worth calling out.
EDIT: I should make note of that on the docs page really. It currently gives the impression that I'm just straight up calculating the sha of the content.
If, for any reason, you'd like a free test account, just email info@rsync.net.
Thank you for building this!
https://adc.arm.gov/armlogin/login.jsp
"The requested URL /resources/regulatory/pricing.html was not found on this server."
I figured borg is stable and does the job but others, including restic have problems like memory usage concerns and I'm just a single person shop and don't have any huge assets to backup and already seeing restic/duplicacy choke consuming GBs of memory.
What are the tools that are performant and stable?
I'd like to at least use another implementation besides borg in case a bug corrupts the data.
Encryption, retention policies, incremental backup without periodical full backup and acceptable performance (that people don't fill GitHub issues with performance problems) are requirements and deduplication is a plus.
It used to be per user but now it's per machine and it's even asking you to pay $50/year per machine even for cheap instances that are $60/year and when you have 20 small machines to look after, the pricing looks a bit too unfair.
https://duplicacy.com/home.html
The UI is basically a 1-to-1 copy of an older version:
https://blobbackup.com/images/logs.png
http://bvckup.com/img/shot-6.png
"Good artists create, great artists copy", but that's a bit too close for comfort.
Similar to how desktop word processors and browsers all look the same; some patterns just make sense for a specific type of software.
Given that this one uses secondary control labels that are verbatim of Bvckup's, has the same layout and opening screen, and nearly identical look and feel, this is certainly not an independent effort.
Outside of the main window though, lots of other things are quite different and completely independently arrived at. Bvckup2 doesn't have third party storage support so everything else requires a fundamentally different UI to work in BlobBackup. Look at the settings windows and the menus for example.
Website does not inspire trust as there is no mention at all about who the seller is.
Terms of use, privacy policy are also relevant, especially in this context where software gets access to all of customer's data...
First time software seller here so I'm a little slow on some things but rest assured, I'm gonna have a privacy policy soon. The EULA is there when you download the software.
BlobBackup doesn't store any user data anywhere other than the user's computer and their backup storage though (and this is encrypted).
https://borgbackup.readthedocs.io/en/stable/changes.html
The downside of borg is that the target is only sftp and not directly to any of S3 and some other targets but there are online services that accept sftp.
On the other hand, restic is new and while this can sync towards many targets, the use of memory gets crazy after a while that my machine choked out of memory and these kind of stability problem is something not mentioned in the docs until you use it.
I've tried several others like duplicacy but this also has a memory problem and both restic and duplicacy seem to have trouble fixing it quickly.
When it comes to backup, I'd like to use more than 1 implementation toward different remote locations in case 1 of the implementation has corruption bugs to render the backup useless.
Trying out kopia to see if it does any better.
https://kopia.io
Also asuran seems promising but so far too early to test it out.
https://asuran.rs/
I would have used restic directly as a backend for BlobBackup had it not been for the awful prune operation. Downloads a ton of data everytime, prone to errors, and has to repack blobs. Compression would be nice too but they have nice dedupe so it would have been workable without.
This used to be (as in, from 2008 to 2018) difficult, one of my past companies made a good deal of money by offering an sftp gateway to S3. But AWS continues to do a great job commoditizing their complements, and tackled SFTP back in 2018 when they launched...
“... AWS Transfer for SFTP, a fully-managed, highly-available SFTP service. You simply create a server, set up user accounts, and associate the server with one or more Amazon Simple Storage Service (S3) buckets. You have fine-grained control over user identity, permissions, and keys. You can create users within Transfer for SFTP, or you can make use of an existing identity provider. You can also use IAM policies to control the level of access granted to each user. You can also make use of your existing DNS name and SSH public keys, making it easy for you to migrate to Transfer for SFTP. Your customers and your partners will continue to connect and to make transfers as usual, with no changes to their existing workflows.”
“You have full access to the underlying S3 buckets and you can make use of many different S3 features including lifecycle policies, multiple storage classes, several options for server-side encryption, versioning, and so forth.”
https://aws.amazon.com/blogs/aws/new-aws-transfer-for-sftp-f...
Because this did not exist for a decade, it seems most folks don’t know it exists today.
Just to turn SFTP on at AWS, you need to pay $0.3/h ($216/mo) and it even costs to upload not just for download at $0.04/GB. If you upload 100GB/day, that's also about $120/mo and if you're storing about 1TB you're paying like $350/mo for such little storage you get.
Who wants to pay for upload? At this point, why don't you just run EBS backed EC2 yourself?
With that kind of pricing, you'd get like 15TB of space at rsync.net and it doesn't leech users for bandwidth and on every parameters as AWS does. (If you use the borg discount, you get 40TB+ at $0.008/GB!)
And at what benefit does it bring for the price difference? I see no point in using it. Maybe people don't care to mention than not knowing it.
https://aws.amazon.com/aws-transfer-family/pricing/
https://www.rsync.net/products/borg.html
Great product though! Always happy to see projects like this
With object storage like s3, b2, azure, wasabi etc, this just isn't a problem since you can effectively have as many objects in one "directory" as you want.
With local, nas or hard drive storage, you'd be limited by the number of files per directory the file system allows (4.2 billion for NTFS for example). So you're probably fine here too.
Google drive (which isn't recommended for large backups with BlobBackup) has much stricter restrictions on files per directory. So really it depends on the backend but for most use cases it isn't a problem.
BlobBackup has been tested on millions of files backed up to NTFS and object storage yes.
[1]: https://www.reddit.com/r/BlobBackup