Tangentially the answer is probably compression first, so I'm curious what the current settled opinion is on compression formats and workflows to get this done!
The canonical Unix method is to use tar in a pipeline to transfer them (assuming none exist at the destination):
tar -cf - /source | ssh destination "cd /destination ; tar -xf -"
However there are as many different ways of doing what you ask as there are opinions, so you may get far more "methods" than you ever knew existed.
If some number of them are already at the destination, then rsync will only send the ones that don't exist, and the ones that differ. What kind of performance will result for "thousands of small files" is a different matter (and note that in today's world, "thousands" is a relatively small number).
(It's cheekily what I was hoping for.... A more naunced argument...) But I got just one reply, hah! I appreciate your input!
And yes, probably on the low end, you're right -- but it's the starting point at which the connection speed becomes irrelevant it seems like. So where I'm most curious.
If you care about permissions, you can always run an rsync after rclone has finished the transfer because applying just the permissions is much faster than the actual single-threaded transfers of rsync.
3 comments
[ 3.1 ms ] story [ 20.1 ms ] threadtar -cf - /source | ssh destination "cd /destination ; tar -xf -"
However there are as many different ways of doing what you ask as there are opinions, so you may get far more "methods" than you ever knew existed.
If some number of them are already at the destination, then rsync will only send the ones that don't exist, and the ones that differ. What kind of performance will result for "thousands of small files" is a different matter (and note that in today's world, "thousands" is a relatively small number).
And yes, probably on the low end, you're right -- but it's the starting point at which the connection speed becomes irrelevant it seems like. So where I'm most curious.
If you care about permissions, you can always run an rsync after rclone has finished the transfer because applying just the permissions is much faster than the actual single-threaded transfers of rsync.