I was hoping this was going to be something where the files themselves were distributed so that they didn't take up space on everyone's machines, a la MogileFS[1], but alas.
This is always a point of pain for us: in our studios we work from a central file server, so our shiny iMacs with their 1TB drives all have about 90% capacity on them — but we don't have any easy way to pool all that extra capacity and use it even as overflow for our central fileserver.
That's what git-annex is for. Everyone has a list of all the files, and then if you want to access one, you can just do "git annex get [file]" and it'll get it from any of the machines that happen to have that file. If you want to delete it, "git annex drop [file]" will ensure at least N machines keep a copy of the file before deleting it.
I use it to manage files between my VPS, desktop, laptop and NAS, with each machine having a different subset of the whole pool, and it works great.
Is there any way to get around the fact that adding a new machine involves adding N*2 remotes? (That is, one remote on each of the current machines to represent the new one; and then on the new machine, one remote for each existing machine). This seems like it would be the sticking point with Git annex.
That's what the new Assistant (now part of git-annex) is for!
Tons of pairing work, which culminated today in pairing fully working for the very first time. And it works great! Type something like "my hovercraft is full of eels" into two git annex webapps on the same LAN and the two will find each other, automatically set up ssh keys, and sync up, like magic. Magic based on math.
The closed source nature of Bitorrent Sync is biggest drawback IMHO. But I guess FOSS clones are on their way. The bit torrent protocol in itself is very well known after all.
Note that the author seems to misunderstand a few things about BitTorrent sync:
1. Firstly, Btsync is not open-source (unfortunately), as he claims. [1]
2. 24-hour secrets don't give someone read access for 24 hours. They must be redeemed within 24 hours after creation, but if they are redeemed on time they become permanent read-only secrets.
Yup, but for some reason most people don't seem to know about it. They could set up a shared top level project directory with sub directories for each aspect of the project and then people only need to sync the stuff relevant to them.
"git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, time, or disk space."
and
"The git-annex assistant creates a folder on each of your computers, Android devices, removable drives, and cloud services, which it keeps synchronised, so its contents are the same everywhere. It's very easy to use, and has all the power of git and git-annex."
If you want to transfer a big file easily without the need to upload it to a server http://www.sharefest.me does the trick.
bit-torrent like service but instant and without any client.
That looks pretty slick. There was a static HTML version I could host on my site, do you know if this does that? I don't want my files going through any third parties.
Not sure what you mean there, but if you want to share files you can use this platform.
If you want to hose your own version of sharefest the open source is in github.com/peer5/sharefest
26 comments
[ 0.18 ms ] story [ 66.2 ms ] threadThis is always a point of pain for us: in our studios we work from a central file server, so our shiny iMacs with their 1TB drives all have about 90% capacity on them — but we don't have any easy way to pool all that extra capacity and use it even as overflow for our central fileserver.
The search continues...
[1]: https://code.google.com/p/mogilefs/
I use it to manage files between my VPS, desktop, laptop and NAS, with each machine having a different subset of the whole pool, and it works great.
Tons of pairing work, which culminated today in pairing fully working for the very first time. And it works great! Type something like "my hovercraft is full of eels" into two git annex webapps on the same LAN and the two will find each other, automatically set up ssh keys, and sync up, like magic. Magic based on math.
http://www.kickstarter.com/projects/joeyh/git-annex-assistan...
http://git-annex.branchable.com/assistant/local_pairing_walk...
http://labs.bittorrent.com/experiments/sync/get-started.html...
The closed source nature of Bitorrent Sync is biggest drawback IMHO. But I guess FOSS clones are on their way. The bit torrent protocol in itself is very well known after all.
1. Firstly, Btsync is not open-source (unfortunately), as he claims. [1]
2. 24-hour secrets don't give someone read access for 24 hours. They must be redeemed within 24 hours after creation, but if they are redeemed on time they become permanent read-only secrets.
[1] - Answered in these threads: http://forum.bittorrent.com/topic/8816-will-syncapp-be-open-... http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-...
Dropbox has selective sync for years already, doesn't it?
[0] http://duplicity.nongnu.org
[1] https://github.com/boto/boto
Duplicity[0] supports it through a boto[1] backend. [0] http://duplicity.nongnu.org [1] https://github.com/boto/boto
"git-annex allows managing files with git, without checking the file contents into git. While that may seem paradoxical, it is useful when dealing with files larger than git can currently easily handle, whether due to limitations in memory, time, or disk space."
and
"The git-annex assistant creates a folder on each of your computers, Android devices, removable drives, and cloud services, which it keeps synchronised, so its contents are the same everywhere. It's very easy to use, and has all the power of git and git-annex."