Yes, the Perl is very short, and has more comments. I missed the Perl era so it was a bit of work for me to figure out how it worked enough to implement it in Rust.
You should make it clearer that the benchmark is perl fsmonitor vs. rust fsmonitor. Without reading the comments here, I would have thought it was without fsmonitor vs. with.
It is not enabled by default yet, since it depends on an external tool to provide repo file update events. Git has one bundled that uses Facebook's watchman (with homebrew if you've installed it like so), but you have to configure each repo (or globally) to enable it. I was just curious how much faster a Rust implementation could be over a Perl script, and wanted to learn a bit of Rust.
I wonder how much faster it would be to use the OS' inotify equivalent (if Linux, inotify itself) directly from rust rather than using an external tool.
This is not at all comparable to what the OP submitted and would be at least a many thousands of lines worth of effort.
What watchman is doing is doing inotify-as-a-service for you, in order for any program to use the Linux inotify API it needs to be running constantly so it can keep consuming events from the kernel, and in the case of the use case watchman fulfills, maintain an internal database of how the current state differs from various times in the past.
There's lots of very nasty edge cases you need to deal with. E.g. if you don't consume the kernel events fast enough (or the buffer is too small) new events will be dropped.
You'll get told about this by the kernel, but now the only way to re-build your in-memory representation to match reality is to recursively walk the FS with readdir(), stat() etc, but at the same time consume new inotify events and update your in-memory structures while doing both.
Needless to say this is a lot more complex than just using watchman off the shelf.
It's also pointless to optimize this area, watchman usually returns in 2-3 milliseconds even on huge trees, but because of bugs / misfeatures in the current git-side implementation it can take hundreds of milliseconds until we make sense of all of that. This is because git doesn't really "know" about watchman/inotify, it just uses it as a replacement for walking the FS, but on big repositories other stuff can still be expensive.
To OP: I highly encourage you to submit this to the Git mailing list for comment. I've been involved in this area (helped write the Perl hook) and there's people who'd be interested in having this be e.g. a part of git's contrib/ directory so it can be compiled with git.
Very cool! I wrote the original (with help) in Perl as it has the advantage of being installed with git but I love the speed advantage of your Rust implementation. Also helps shows the advantage of making core.fsmonitor a pluggable interface.
18 comments
[ 4.4 ms ] story [ 73.8 ms ] threadThis appears to mainly be a lightweight wrapper of watchman.
https://github.com/git/git/blob/v2.16.0/templates/hooks--fsm...
Edit: well, maybe just fewer comments... ha
Its a watchman wrapper written in rust.
But in that article you linked, there is mentioned that git ships with a sample watchman hook already?!
[1]: https://crates.io/crates/notify
What watchman is doing is doing inotify-as-a-service for you, in order for any program to use the Linux inotify API it needs to be running constantly so it can keep consuming events from the kernel, and in the case of the use case watchman fulfills, maintain an internal database of how the current state differs from various times in the past.
There's lots of very nasty edge cases you need to deal with. E.g. if you don't consume the kernel events fast enough (or the buffer is too small) new events will be dropped.
You'll get told about this by the kernel, but now the only way to re-build your in-memory representation to match reality is to recursively walk the FS with readdir(), stat() etc, but at the same time consume new inotify events and update your in-memory structures while doing both.
Needless to say this is a lot more complex than just using watchman off the shelf.
It's also pointless to optimize this area, watchman usually returns in 2-3 milliseconds even on huge trees, but because of bugs / misfeatures in the current git-side implementation it can take hundreds of milliseconds until we make sense of all of that. This is because git doesn't really "know" about watchman/inotify, it just uses it as a replacement for walking the FS, but on big repositories other stuff can still be expensive.
To OP: I highly encourage you to submit this to the Git mailing list for comment. I've been involved in this area (helped write the Perl hook) and there's people who'd be interested in having this be e.g. a part of git's contrib/ directory so it can be compiled with git.
https://github.com/facebook/watchman/tree/master/rust/serde_...
[1]: https://github.com/clibs/entr