"Parts of the programm relevant for security and networking actions will be open source."
"Parts of the program" might not do it for everyone (me included). Will anyone be able to audit the full source code of the client and compile it themselves, like e.g. tarsnap? Otherwise I fear the usual trust issues arise.
If I missed some information at your linked site which says the entire client will be open source, then please correct me.
Hmm, yeah, part of the program might as well be none of it. Ideally, I'd want to be able to compile it myself and use the resulting binary, or have the client be in Python or something like that.
You can still tell what exactly happens. Two separate processes might be used: one for crypto + network which is open source, the one manages synchronisation (which is the secret sauce of all cloud-sync services, if it works really good).
Communication between open and closed source processes runs via IPC. Since all the IPC functions are declared in the open source part, you can exactly check what data is exchanged between the processes -> You can see that keys are not transmitted.
You can make it so that it will only store the key in RAM, but then you'd have to enter the key every time it launches, making automatic backups impossible.
Although, I agree, if you only allow the open source network access, AND can ensure that the key won't somehow be smuggled in the data the closed source process sends, you're probably fine.
I think there's plenty of room for the larger discussion about how much we actually (ought to) trust a lot of our closed source software that runs free and non-jailed on our computers day in and day out - and recent events haven't affected that viewpoint very much in either direction. However, data synchronization software gets tougher love from everyone these days for apparent reasons.
I read the updated FAQ and saw the note about blocking network I/O for the closed source process. If the closed source process in your client were locked down in such a way that it was only ever invoked from the open source process and had no way of doing any I/O whatsoever (network or otherwise) except with the open source process (where what goes in and comes out can be audited), then I'd agree this sounds quite interesting.
I'll be waiting as well, and sincerely hope you deliver something nice!
Two things, one doesn't store the encfs.xml file in the reversed directory, so it doesn't get backed up automatically, and the other, more important (for my use case) is that there's no way to make multiple directories appear under one (symlinks are exposed as encrypted/broken symlinks). This means that you can only back up one dir per repo, which I found too constraining.
I just found out about python-fuse, and it seems that I can use a lot of the code from encbup to write a simple EncFS clone that will accept a list of directories and produce a mountpoint that will expose them encrypted. Then you can just rdiff-backup that and you're done.
Meeting most of the requirements on that list, I have found Arq[1] to be a great backup solution on OS X. It stores encrypted, incremental backups on S3 and although the client itself isn't open source, the data format is[2]. The company also maintains an open source recovery tool[3]. The only requirement that isn't met is cost, because S3 costs about $60/yr for 50GB.
Happy Arq user here, but the cost of S3 is why I have it store in Glacier instead. It all really depends how (and how often) you plan on accessing the backups.
Added in edit: I see that people are saying it's expensive, and I appreciate that. When I bundle it into my considerations of how much I spend on other things, what I get, and my level of confidence in the service, it seems a good deal to me. In the interests of full disclosure, I don't actually use anything like this, as I have a storage solution provided for me. But when I did the sums a while ago, I decided that tarsnap would be my choice if I needed to make one. For people to say it's expensive is a useful additional data point for me. Thank you.
By telling him to charge more, what he's saying is more or less to drop the consumer market and sell to bigger businesses.
It would probably be better for Tarsnap's financial health, but it wouldn't make it a better product for people like the OP who just want to save pictures of their dog.
Can encbup be used for automated backups? Seems like it's using symmetric encryption, so that the passphrase would need to be entered or passed as an argument on each run.
Not sure if it meets all your requirements, but duplicity [1] has support for asymmetric encryption via gpg pub/priv key pairs.
You can specify the passphrase on the command line, but it's less secure. Automated backups are my use case, so that's what it's meant to be used for.
I wanted to also add GPG encryption, but I'd be unable to store the key locally, so I couldnt find a way to get it for subsequent backups. I'll have to see how duplicity does it.
I tried that on 500 KB when I was testing it and it took a few minutes to back things up. I'm not sure if it's linear, but I wouldn't want to try the full set if it's that slow...
http://liw.fi/obnam/ondisk/ attempts to answer that in some detail. In summary: Obnam splits file data into chunks and stores each chunk only once. It looks up chunks using checksums, and does not need to download the chunk data again (unless you turn on verification mode to protect against checksum collisions). Obnam does download B-tree nodes, though, but they should be a fraction of the size of the file content data.
Personally I use crashplan, its more expensive than your max price requirement if you use their cloud services, but I also back up way more data (300+GB), mostly family photos+videos. Also, since its unlimited storage I don't really care if I suddenly have a couple of extra gigs to store after xmas or whatever. Oh and finally I can use the same app to backup to my home server so basically everything is backed up locally and remotely with one app and is encrypted/deduplicated.
I used to use Arq and it worked great but then the backups became too big.
EDIT: forgot you wanted opensource... oh well... anyway this works for me.
I feel that the main problem with online backups is simply a matter of bandwidth. I have in excess of 3 TB of data and even if I only backed up things I considered _critical_, it would still be so much that it would take a week to upload and cost me hundreds of dollars of data overages. Such is Canada.
It's very telling that I'm even considering this, but if there existed a backup service where I could mail them a hard drive to load my data initially, and then upload only incremental changes, I would be all over that. Of course, uploading videos and such would still be out of the question.
"if there existed a backup service where I could mail them a hard drive to load my data initially"
Crashplan provide this option. I've never used that specific part of their service personally, but I've been a satisfied customer of their unlimited family plan for several years.
It's possible to mail drives to Amazon and have them import the initial data set http://aws.amazon.com/importexport/ A former employer of mine did that a few years ago so that they could start serving a lot of high resolution images using S3.
I needed to implement deduplication in my system. Since I controlled the server, I developed a slightly more elaborate system which doesn't have the limitation of a predictable IV (predictable from the encryption key).
So in my system, I derive two keys from the same passphrase (PBKDF2 with different salts). I encrypt as usual with unpredictable IVs. When uploading, the HMAC of the plaintext and SHA-256 of the ciphertext are both loaded.
To check for duplication, the client asks if a certain HMAC is already present. And it's an error (at the server) to upload multiple ciphertexts with the same HMAC.
The vulnerability in that post is for using AES-CBC with AES-CBC-MAC with the same key. I'm using AES-CBC and HMAC-SHA512, which should be okay. The design was reviewed by cryptographers and was given a green light, plus I tried to use as little custom crypto as possible for this exact reason :)
I'm disappointed there's no comment here from tptacek telling you off for that decidedly dodgy-looking mac-then-encrypt, or the timing side channel in your mac verification. He must be slipping :)
Haha, the MAC-then-encrypt should be fine, since it's stored files, and how is anyone going to use the timing attack when the script runs on trusted hardware anyway? :P
I'm surprised no one here has mentioned Bitcasa. They provide unlimited storage for $99/year, and everything is encrypted locally before being sent over the wire.
Yes, they do deduplication. I'm using Cyphertite for some of my backups. Works fine, and the pricing is a bit more convenient than Tarsnap for home users.
On a slightly larger scale, we work with customers to do encrypted logical database backups using GPG and s3/Glacier.
Works pretty well, and we're able to de-duplicate because we keep local copies and make use of hardlinking. A bit of logic around s3cmd to look for hardlinked files before uploading takes care of the remote de-duplication (which isn't realistic after the files are encrypted).
Not quite as easy to adapt to non-linux machines or as practical for smaller files, but it's still quite useful.
49 comments
[ 3.7 ms ] story [ 70.4 ms ] threadOn the link you provided I found the following:
"Parts of the programm relevant for security and networking actions will be open source."
"Parts of the program" might not do it for everyone (me included). Will anyone be able to audit the full source code of the client and compile it themselves, like e.g. tarsnap? Otherwise I fear the usual trust issues arise.
If I missed some information at your linked site which says the entire client will be open source, then please correct me.
Updated the FAQ: https://boldshare.com/?lang=en#faq
Key won't be accessible on disk anyway, only in RAM during crypto and will be destroyed immediately afterwards.
Although, I agree, if you only allow the open source network access, AND can ensure that the key won't somehow be smuggled in the data the closed source process sends, you're probably fine.
I read the updated FAQ and saw the note about blocking network I/O for the closed source process. If the closed source process in your client were locked down in such a way that it was only ever invoked from the open source process and had no way of doing any I/O whatsoever (network or otherwise) except with the open source process (where what goes in and comes out can be audited), then I'd agree this sounds quite interesting.
I'll be waiting as well, and sincerely hope you deliver something nice!
Does anyone know what this refers to? It sounds like a winning combination to me.
What does everyone think?
[1] http://www.haystacksoftware.com/arq/index.php
[2] http://www.haystacksoftware.com/arq/s3_data_format.txt
[3] http://sreitshamer.github.io/arq_restore/
Added in edit: I see that people are saying it's expensive, and I appreciate that. When I bundle it into my considerations of how much I spend on other things, what I get, and my level of confidence in the service, it seems a good deal to me. In the interests of full disclosure, I don't actually use anything like this, as I have a storage solution provided for me. But when I did the sums a while ago, I decided that tarsnap would be my choice if I needed to make one. For people to say it's expensive is a useful additional data point for me. Thank you.
It would probably be better for Tarsnap's financial health, but it wouldn't make it a better product for people like the OP who just want to save pictures of their dog.
Not sure if it meets all your requirements, but duplicity [1] has support for asymmetric encryption via gpg pub/priv key pairs.
[1] http://duplicity.nongnu.org/
EDIT: just noticed the article mentioned duplicity as insufficient.
I wanted to also add GPG encryption, but I'd be unable to store the key locally, so I couldnt find a way to get it for subsequent backups. I'll have to see how duplicity does it.
It's pretty excellent in every other respect.
I used to use Arq and it worked great but then the backups became too big.
EDIT: forgot you wanted opensource... oh well... anyway this works for me.
There's even a front end of it called Deja Dup [2] being distributed by the GNOME team and used in Ubuntu.
[1]: http://duplicity.nongnu.org/
[2]: https://wiki.gnome.org/Apps/DejaDup
It's very telling that I'm even considering this, but if there existed a backup service where I could mail them a hard drive to load my data initially, and then upload only incremental changes, I would be all over that. Of course, uploading videos and such would still be out of the question.
Crashplan provide this option. I've never used that specific part of their service personally, but I've been a satisfied customer of their unlimited family plan for several years.
A comment in the code about why it's OK in your case to use the same key for MAC and encrypt would be useful. I think you're fine. See here: http://security.stackexchange.com/questions/37880/why-cant-i...
I needed to implement deduplication in my system. Since I controlled the server, I developed a slightly more elaborate system which doesn't have the limitation of a predictable IV (predictable from the encryption key).
So in my system, I derive two keys from the same passphrase (PBKDF2 with different salts). I encrypt as usual with unpredictable IVs. When uploading, the HMAC of the plaintext and SHA-256 of the ciphertext are both loaded.
To check for duplication, the client asks if a certain HMAC is already present. And it's an error (at the server) to upload multiple ciphertexts with the same HMAC.
https://www.cyphertite.com/
Not clear to me from a quick overview of their site if they do de-duplication.
Works pretty well, and we're able to de-duplicate because we keep local copies and make use of hardlinking. A bit of logic around s3cmd to look for hardlinked files before uploading takes care of the remote de-duplication (which isn't realistic after the files are encrypted).
Not quite as easy to adapt to non-linux machines or as practical for smaller files, but it's still quite useful.