HN Explorer

Show HN: Dsync: a tool to sync source directories/files to a target directory (github.com)

7 points by dorjoy ↗ HN
Hello HN. I worked on this small project recently called dsync. I have a few directories where I keep adding stuff and I need to sync them to an external drive from time to time. In the spirit of "write your own tools", I wrote my own. The tool itself is really simple. It tries to sync source directories or files to a destination directory based on size and modification time, basically copying the source files if they don't match.

dsync can use multiple threads (specified via an option). It uses a bounded multi-producer multi-consumer queue (C implementation of Dmitry Vyukov's bounded MPMC queue using gcc atomic builtins). The main thread traverses the given sources and adds the files to the queue. Other threads dequeue entries from the queue and do the sync/copy work concurrently.

dsync performs well with multiple threads compared to GNU cp when copying directories with a lot of small files such as the linux kernel repository. More details are in the README.

8 comments

[ 556 ms ] story [ 1767 ms ] thread
gregjor
Have you heard of rsync?

I don't know about the spirit of "write your own tools," I tend to follow the spirit of "use the tried and tested tools." As a hobby project this looks interesting, but to be fair you should have benchmarked it against rsync rather than cp.

dorjoy
Hey. Thanks for the input. I have heard of rsync but did not try it out because I wanted to implement something simple on my own. And about the benchmark, I think comparing with cp makes sense as when targeting an empty directory it is really just measuring what time it takes to "copy". I guess it could make sense to do "sync" benchmark too but I really just measured the copy performance to be honest.
thewickedaxe
Rsync's name might be a little misleading. It doesn't attempt to sync the directories unless you ask it to, aka just copy. However I remember studies from a while ago showing that rsync actually performs worse than cp/scp under certain circumstances. May that's something you can try to address
gregjor
> Rsync's name might be a little misleading. It doesn't attempt to sync the directories unless you ask it to, aka just copy.

rsync will "just copy" if you point it at an empty destination. Otherwise it will try to sync.

  rsync source/ dest/
rsync has options to preserve ownership, modification date/time, etc. similar to cp's -a (archive) mode. Most important to me and I think a lot of rsync users: it can restart failed transfers. That's more important when transferring over a network but it works on local copies too.

> However I remember studies from a while ago showing that rsync actually performs worse than cp/scp under certain circumstances.

Maybe you're thinking of these threads:

https://serverfault.com/questions/43014/copying-a-large-dire...

https://stackoverflow.com/questions/6339287/copy-or-rsync-co...

Not really seeeing significant performance difference but maybe you can find data. In the case of maintaining a copy of a directory of files that gets added to and changed that's an exact description of what rsync does, and it has years of optimizations for exactly that.

scp, on the other hand, is both slower than rsync for non-trivial cases (more than a few files), and less secure. "The scp protocol is outdated, inflexible and not readily fixed. We recommend the use of more modern protocols like sftp and rsync for file transfer instead." (from OpenSSH 8.0 release notes, 2019-04-17). scp was informally deprecated in favor of rsync a long time ago.

snthpy
On a related note, is there a tool that syncs all filepaths and their hashes to a sqlite database and then also logs all changes, maybe with treesitter or something like that?
vrighter
how is it better than rsync?
dorjoy
Hi. I haven't compared it with rsync as rsync has a lot more features than dsync. dsync only works locally which is what I needed and decided to implement myself.