rsync can however take a file list, and inotifywait can produce a file list with the help of awk or sed. Which means you can achieve continuous synchronisation with ~5 lines of shell script. A bit more care if you want to sync two ways.
I've found that my file syncing needs tends to be so varied that most of the dedicated tools I've looked at have almost always been just wrong enough not to be worth it compared to just composing something with rsync and inotifywait.
This is a nice little project, but I think rsync is better suited for any serious use. It can do ssh, it's own protocol or any connection program that would tunnel bidirectionally. It does very efficient transfer, can create off-line patch files, and basically do any one-way file/dir sync function you can think of.
Edit: this one is continuous and rsync is one-shot. Personally, I've been using inotify+rsync+timeout (intofy+rsync has a race condition, so limiting the inotify wait and syncing anyway is essential). Does sshync avoid the race?
You might also check out Syncany if you're interested in client-side encryption or Syncthing if you're looking for something more Bittorrent Sync like.
I use similar solution: my own shell script with inotify + timeout + unison over ssh. Unison is much better suited for two-way synchronization than rsync.
"Rsync is a mirroring tool; Unison is a synchronizer. That is, rsync needs to be told "this replica contains the true versions of all the files; please make the other replica look exactly the same." Unison is capable of recognizing updates in both replicas and deciding which way they should be propagated."
Don't know if this is still the case, but in the past, unison was extremely picky about the versions at both ends; I stopped using it when I upgraded my Ubuntu box and as a result I needed to either recompile an older version for the new box or upgrade the version on all other boxes (across windows and various distros).
I now use rsync for bidirectional mirroring (any two versions from the last 15 years or so can talk to each other), git for synchronization (getting history for the price of needing to add/commit). Can't beat the network efficiency and compatibility of these two.
I use rsync. Question for you (or any expert here). Is there any possible way that rsync can alter the source file (unintentionally I mean, by accident or bug)?
And what happens to an rsync file that is modified while in the middle of a transfer? [1](I haven't been able to find satisfactory answers to these questions hence the question here).
[1] Say you have a 1,000,000 line file that you are rsynch-ing and you delete line 50000 and insert at line 750000?
Rsync opens source file in read only mode - so it has the same probability to modify a file that regular copying does. That said, the --remove-source-files option will delete source files after copying; --delete may remove destination files; and used together they drop the idempotence guarantee that rsync gives. So you have to be careful.
Re changing file - IIRC it will keep Resending until file size+mtime at end of transfer is same as in the beginning.
For most uses, rsync -a (preserve all possible metadata, like cp -a) is preferable to rsync -r ;
Also, unless you have a very large (breadth or depth) tree, you can just let rsync synchronize the entire path - it is quite efficient, will pick up additional diretories (you need inotifywait -r for that, and AFAIK you still have an unavoidable race).
On Windows I use WinSCP, from menu "Commands\Synchronize" and "Commands\Keep Remote Directory up to Date", or from scripting "synchronize" and "keepuptodate".
Note however that sshfs makes a file system available through ssh (with Some caching but not e.g. persistent cache across ssh reconnects), and no off-line persistence - whereas other solutions discussed (sshync, unison, lsync, syncthing) all replicate files (think dropbox) which comes with benefits (quicker access, offline availability) and problems (edit conflicts, stale versions)
I appreciate the positive response, but this is just something I put together quickly and needs a little refurbishing before it can be considered a viable alternative to something like lsyncd.
Top priorities right now are to get rid of ssh-client dependency and to address security concerns. I'd also like to implement file diffing.
This is a small thing - but how do you pronounce the name? Looking at it, I want to call it "ssh sync", but it looks more like it should sound like "shync" (which sounds sort of like Garth from "Wayne's World" saying "schwing!") Just curious because it's hard to discuss a tool verbally if I don't know how to pronounce it.
These are some pretty awful scripts, to be honest. Use Unison (https://www.cis.upenn.edu/~bcpierce/unison/). Syncing is much more dangerous and difficult than people assume.
56 comments
[ 3.2 ms ] story [ 128 ms ] threadAnd fewer features.
(but a cool little project none the less!)
My point was it isn't "hipster" to use something that is fast and efficient.
Glibc is used by multiple packages. I simply identified the number of dependencies to install this tool when nodejs/npm are not installed.
You can use rsync if you like too
I've found that my file syncing needs tends to be so varied that most of the dedicated tools I've looked at have almost always been just wrong enough not to be worth it compared to just composing something with rsync and inotifywait.
Of course your mileage will vary.
> this one is continuous and rsync is one-shot.
http://aurelianito.blogspot.com.ar/2014/12/mejorando-el-mirr...
Want to compare any other useless statistics?
Edit: this one is continuous and rsync is one-shot. Personally, I've been using inotify+rsync+timeout (intofy+rsync has a race condition, so limiting the inotify wait and syncing anyway is essential). Does sshync avoid the race?
Edit: I am now noticing this tool isn't even maintained anymore.
"Rsync is a mirroring tool; Unison is a synchronizer. That is, rsync needs to be told "this replica contains the true versions of all the files; please make the other replica look exactly the same." Unison is capable of recognizing updates in both replicas and deciding which way they should be propagated."
(your profile mentions to respond to one of your comments if someone needs to reach you ...)
I now use rsync for bidirectional mirroring (any two versions from the last 15 years or so can talk to each other), git for synchronization (getting history for the price of needing to add/commit). Can't beat the network efficiency and compatibility of these two.
And what happens to an rsync file that is modified while in the middle of a transfer? [1](I haven't been able to find satisfactory answers to these questions hence the question here).
[1] Say you have a 1,000,000 line file that you are rsynch-ing and you delete line 50000 and insert at line 750000?
Re changing file - IIRC it will keep Resending until file size+mtime at end of transfer is same as in the beginning.
Here is a tip on how it can be used ( not tested )
1 : http://linux.die.net/man/1/inotifywaitAlso, unless you have a very large (breadth or depth) tree, you can just let rsync synchronize the entire path - it is quite efficient, will pick up additional diretories (you need inotifywait -r for that, and AFAIK you still have an unavoidable race).
http://anonscm.debian.org/cgit/pkg-ssh/openssh.git/tree/cont...
https://anongit.mindrot.org/openssh.git/tree/contrib/ssh-cop...
This is good to learn!
I think newline is valid in filenames in most *nix filesystems, so the possibility to use an alternative RS would be nice.
I appreciate the positive response, but this is just something I put together quickly and needs a little refurbishing before it can be considered a viable alternative to something like lsyncd.
Top priorities right now are to get rid of ssh-client dependency and to address security concerns. I'd also like to implement file diffing.
Your feedback is invaluable!
This is how I do my email, exim running on a DO instance and I send and receive to my laptop via ssh.