This command seems like a no-brainer that should be in all OSes by default.
It's mind-boggling that most systems don't have a standard tool for observing file changes. The only language I know of that has a platform-agnostic API for this is Factor.
The required kernel hooks exist in pretty much any common OS these days, it is a user-space tool that is sometimes missing.
It may not be installed by default, but inotifywait is available in common Linux distributions, usually in a package called something like ionotify-tools, and has been for over a decade-ana-half IIRC. It'll work under WSL on Windows too, though only for ext4 devices not bits of the Windows filesystem made available to Linux.
I can't speak to what other OSs include by default, but as every major OS has a different API for how to register a listener & how it gets messages no built-in tool is going to be cross-platform. There are third party tools which present more cross-platform consistency, most notably https://github.com/emcrisostomo/fswatch#readme (also available in common Linux distros, just an apt install away in Debian for instance).
Not to be confused with Facebook’s file watch daemon, also names watchman, which does the same sort of thing but is more complicated. There’s a bunch of tools that integrate Facebook’s watchman for more efficient change tracking.
Advantages of Facebook’s watchman:
1. Implements efficient file system event watches on macOS and Windows
2. IPC/daemon system reduces resource use because overlapping watches/triggers don’t use more inotify slots.
3. Denounces / waits for changes to settle
4. Client libraries in a few different languages for scripting purposes
Also not to be confused with watchdog and it's command watchmedo, which is what I thought this was when I first read the headline. I don't know what the differences are, but I use watchdog to restart my local celery workers when I make changes. It's scripted out so I don't ever use the command directly.
EDIT:
Now I'm wondering how often I miss out on something because I confuse it for something I already use. Maybe there is room for a "things I use" or "things I know about" browser plugin.
Also to note "watchdog" the term. Is used also in hardware/software context. Usually as a way to mitgate or recover system faulate.
I first I heard of this term while taking in a course about PIC microcontroller[1].
> The Watchdog Timer in the PIC16F819
The watchdog timer can be a real source of pain and can also make a PIC system incredibly robust and reliable. But what exactly is the watchdog timer? Simply put, the watchdog timer is a hardware timer in the PIC that, if not constantly reset by the software, will cause the PIC to reset. This feature can be incredibly useful if the PIC should hang due to a hardware or software issue and guarantees that the PIC will restart from the beginning. Not only does it reset the system, it also flags a bit that can be used to determine if the system just crashed.
use facebook's watchman to watch changes to the source for OPs watchman to commit changes to the source of facebook's watchman to commit changes to OP's
For those use-cases I use entr (https://github.com/clibs/entr). In what sense does watchman differ to it? Didn't see anything related to this at first glance :-)
ifnotifywait doesn't exist on macOS, so in addition to that not being an option on macos, this would be something you could write tools that a team who used multiple platforms, or a person with setups on multiple os's, could use. I suspect this might even be usable on windows.
[edit: fswatch looks very promising, and may be a better choice than either ifnotifywait or watchman]
You are confused about how the tool works. This project uses inotifywait from inotify tools to reimplement what inotifywait already does. The linked article page at github clearly says this.
Adding a new layer atop inotify tools doesn't make that package automatically run on new platforms. There are already tools on macos that use the native FSEvents API for watching for file changes and executing scripts.
This wraps inotifywait, so it doesn’t really address cross-platform needs.
The unique features in this over straight inotifywait is plugging together the command you want to execute, and some event dedupe logic. Grand majority of the code is around colorized logs, command docs, and some options handling code which mostly just gets piped to inotifywait.
Discovered fswatch recently, it's proved very useful, my favorite use being replacing a few situations where I was using sshfs with fswatch+rsync: no network lag on save/load, can even work offline, but everything still makes its way to the network destination!
I've been using fswatch for a project that supports both linux and darwin. Here's an example from a Makefile of how I use it to re-run tests on file change:
test-watch:
make --no-print-directory test || :
fswatch --event Updated -o test/*.c test/*.h src/ | xargs -n1 -I{} make --no-print-directory test
It looks like a simple bash wrapper around inotifywait to provide a simpler interface for a subset of the functionality. You don’t have to remember any
flags.
From my experience, they break every time if you are making changes with tools that replace the file rather than change it in-place, like most text editors.
Yeah, and then filter out the events in the directory to find out when the correct file changed, etc. That is exactly what I would want the tool to abstract for me.
At least with inotify, ie, incron, this is just a matter of setting a less stupid watch.
You can ask the kernel to watch for several different specific kinds of fs events, either at the file or directory level.
Detecting when a file is either created, or closed from writing, is no problem at all. Ie, handles the case of writing a file normally, editing a file, and renaming a file into exitence without having opened it for writing (that happened while it had some other name you weren't watching). This would handle unlinking and recreating the same as editing or uploading.
You can watch specifically for the close event only, and even more specifically for close-from-write-mode to detect updates but ignore until the update is done. And at the same time you can also watch for creation, which handles creating a file under some other name and then renaming it into existence under the name you care about without the name you care about ever having a close-from-write event.
That would fire off your script when a filename you were watching was unlinked and re-created instead of edited in-place, and then it's up to your script to handle the case where it's a "new" file that is really just replacing an old file.
I don't even see why that should be any kind of special problem. Are some tools doing something like maintaining ooen file handles or something for every watched file??? That would break by unlink-create, but that would be insane.
Perhaps these wrappers that try to simplify things are as usual just a way to break the thing they are trying to simplify. Just misguided crap.
If the feature is baked into an ide or media server and failing to handle that case, well they don't have to fail to handle that case. The subsystem totally supplies the functionality.
For something random adhoc where you're writing your own incrontab and handler script, it's no problem at all.
Yeah, this is difficult, though not impossible, to get right with the inotify/kqueue api. Windows and mac do provide apis for recursive directory monitoring but a cross platform tool will have to solve this problem. At a high level, the way to do it is to create an in memory representation of the file system that caches a watch handle for every file. When a deletion of a file is detected, you must create a watch on the parent directory, if there wasn't one already. Then you should be able to detect the ensuing create. To make this more concrete, the problem that a file watcher needs to solve is the problem of keeping its in memory representation of the file system consistent with the actual file system. Watch events are just a useful side effect of this process.
The other fun part is that there is often a lag between the deletion and the create in the text editor case so it is necessary to defer triggering events when a deletion is detected and wait a little while to see if there is a corresponding creation. Otherwise you may rerun your command that depends on the file before it exists.
It is possible to get like a 99+% solution to this problem without polling but it is a lot harder than what these simple tools, including entr do. The upshot is that file monitoring should be looked at as a lowest common denominator solution. A better solution is to build automatic command running into the text editor itself.
If it’s running in a shell, it’d be the terminal session’s shell. If it’s running as a daemon, in a traditional Unix environment, it’d be the init process (pid 0).
Neither init or a shell or any other userspace code. The userspace code like in inotifywait or incron just tells the kernel what it wants the kernel to watch, and then exits, gone, no further parent, not even init.
Ok now that I mentioned incron..., incron being a daemon is managed by init, but incron is not watching any files, not even it's own config files. incron just reads it's incrontab files and registers the desired events with the kernel, and then performs the desired actions when the kernel triggers it. It notices when it's own config files change the same way as any other inotify job, it asks the kernel to tell it when they change.
incron is like cron where you write a crontab-like file that specifies a command of your choosing. Pretty much the same as this thing minus some colorful display.
I guess the point of this thing would be if it provides a consistent wrapper interface over the various fs watching mechanisms on different OS's.
But for linux, I've already been using incron to do this job efficiently for ages.
this project hasn't had a commit in 8 years and only has a handful of stars. It confusingly shares the name with a much more popular tool from Meta—though the use case is slightly different.
There are other much more popular tools that do the same thing, like entr, watchexec, and nodemon. Why is this posted?
No, there is no such thing as “Facebook’s Payroll”. It’s all Meta. I’m a former Meta employee. Watchman is used heavily across the family of apps. It’s not like Alphabet that’s a holding company for different smaller companies.
Watchman not yet being branded with “Meta” is a case of laziness. It doesn’t indicate it’s limited to the Blue app.
In my defense, I didn't know about Facebook's watchman then (did it even exist?).
`watchman` sounded like an appropriate name to me for the task this script was for.
I had not intention about gaining from folks accidentally landing on my project. Apologies if people felt mislead. I have no idea why this was suddenly posted on Hacker News yesterday.
Edit: it may have been a different watchman, but same point still applies.
It’s been a few years, so perhaps it got addressed, but this library used to introduce a “memory leak” by design, by infinity storing pointers to files being watched.
Ultimately I ended up fixing it by writing my own version using native system calls available on Node.
The moral of the story, this library is great if you are building cross platform dev tool. It is not (at least wasn’t at the time) great for long lived processes that target only one type of operating system.
I'm so used to making snap judgements on complex projects, and seeing that every file was last committed 8 years ago is usually a giant red flag to me.
However, this is a single bash script, which definitely could be stable and forgotten about. Something like this would be more appealing as a posix shell script, but if you're using bash anyway and not trying to distribute, then stable and simple goes a long way.
I'm gonna try to use this to revert the .xcscheme file changes that Xcode makes when I switch between various Xcode versions on my machine. These happen in 10+ submodules everyday so this is great!
I've been using fswatch to great effect. The big advantage is that it's truly cross-platform, falling back to polling if no fs notification system is available as long as stat(2) works.
Very composable as you just pipe its output to whatever you want (typically a while read do end), so exclusion is a awk/sed/perl/grep/rg away, and command can be as simple or complex as it needs to while the tool itself stays lean.
Cool features include -o to just be notified that something changed (e.g to fire up rsync), and ability to batch changes.
Also the command is a front end to libfswatch, so one can use that directly if shelling is to be foregone.
> If you are installing fswatch using a package manager and you would like the PDF manual to be bundled into the package, please send a feature request to the package maintainer.
I can't find the PDF manual in the repo. Do I need to contact the author and say please?
97 comments
[ 3.3 ms ] story [ 141 ms ] threadIt's mind-boggling that most systems don't have a standard tool for observing file changes. The only language I know of that has a platform-agnostic API for this is Factor.
It may not be installed by default, but inotifywait is available in common Linux distributions, usually in a package called something like ionotify-tools, and has been for over a decade-ana-half IIRC. It'll work under WSL on Windows too, though only for ext4 devices not bits of the Windows filesystem made available to Linux.
I can't speak to what other OSs include by default, but as every major OS has a different API for how to register a listener & how it gets messages no built-in tool is going to be cross-platform. There are third party tools which present more cross-platform consistency, most notably https://github.com/emcrisostomo/fswatch#readme (also available in common Linux distros, just an apt install away in Debian for instance).
Advantages of Facebook’s watchman:
1. Implements efficient file system event watches on macOS and Windows
2. IPC/daemon system reduces resource use because overlapping watches/triggers don’t use more inotify slots.
3. Denounces / waits for changes to settle
4. Client libraries in a few different languages for scripting purposes
https://facebook.github.io/watchman/
https://pypi.org/project/watchdog/
https://github.com/gorakhargosh/watchdog
EDIT: Now I'm wondering how often I miss out on something because I confuse it for something I already use. Maybe there is room for a "things I use" or "things I know about" browser plugin.
> The Watchdog Timer in the PIC16F819 The watchdog timer can be a real source of pain and can also make a PIC system incredibly robust and reliable. But what exactly is the watchdog timer? Simply put, the watchdog timer is a hardware timer in the PIC that, if not constantly reset by the software, will cause the PIC to reset. This feature can be incredibly useful if the PIC should hang due to a hardware or software issue and guarantees that the PIC will restart from the beginning. Not only does it reset the system, it also flags a bit that can be used to determine if the system just crashed.
Ref: https://maker.pro/pic/tutorial/how-to-get-started-with-pic-m...
[1]: PIC microcontroller, is a small hardware simliar to arduino but more on a lower level.
https://github.com/watchexec/watchexec
Oh, who am I kidding. Nobody, obviously.
> WARNING: This is a (possibly outdated and/or unmaintained) fork of https://github.com/eradman/entr .
[0]https://github.com/ar-/incron
For example:
[edit: fswatch looks very promising, and may be a better choice than either ifnotifywait or watchman]
Adding a new layer atop inotify tools doesn't make that package automatically run on new platforms. There are already tools on macos that use the native FSEvents API for watching for file changes and executing scripts.
The unique features in this over straight inotifywait is plugging together the command you want to execute, and some event dedupe logic. Grand majority of the code is around colorized logs, command docs, and some options handling code which mostly just gets piped to inotifywait.
Not super useful, but not useless.
Do not use if your life depends on it.
You can ask the kernel to watch for several different specific kinds of fs events, either at the file or directory level.
Detecting when a file is either created, or closed from writing, is no problem at all. Ie, handles the case of writing a file normally, editing a file, and renaming a file into exitence without having opened it for writing (that happened while it had some other name you weren't watching). This would handle unlinking and recreating the same as editing or uploading.
You can watch specifically for the close event only, and even more specifically for close-from-write-mode to detect updates but ignore until the update is done. And at the same time you can also watch for creation, which handles creating a file under some other name and then renaming it into existence under the name you care about without the name you care about ever having a close-from-write event.
That would fire off your script when a filename you were watching was unlinked and re-created instead of edited in-place, and then it's up to your script to handle the case where it's a "new" file that is really just replacing an old file.
I don't even see why that should be any kind of special problem. Are some tools doing something like maintaining ooen file handles or something for every watched file??? That would break by unlink-create, but that would be insane.
Perhaps these wrappers that try to simplify things are as usual just a way to break the thing they are trying to simplify. Just misguided crap.
If the feature is baked into an ide or media server and failing to handle that case, well they don't have to fail to handle that case. The subsystem totally supplies the functionality.
For something random adhoc where you're writing your own incrontab and handler script, it's no problem at all.
The other fun part is that there is often a lag between the deletion and the create in the text editor case so it is necessary to defer triggering events when a deletion is detected and wait a little while to see if there is a corresponding creation. Otherwise you may rerun your command that depends on the file before it exists.
It is possible to get like a 99+% solution to this problem without polling but it is a lot harder than what these simple tools, including entr do. The upshot is that file monitoring should be looked at as a lowest common denominator solution. A better solution is to build automatic command running into the text editor itself.
Neither init or a shell or any other userspace code. The userspace code like in inotifywait or incron just tells the kernel what it wants the kernel to watch, and then exits, gone, no further parent, not even init.
Ok now that I mentioned incron..., incron being a daemon is managed by init, but incron is not watching any files, not even it's own config files. incron just reads it's incrontab files and registers the desired events with the kernel, and then performs the desired actions when the kernel triggers it. It notices when it's own config files change the same way as any other inotify job, it asks the kernel to tell it when they change.
incron is like cron where you write a crontab-like file that specifies a command of your choosing. Pretty much the same as this thing minus some colorful display.
I guess the point of this thing would be if it provides a consistent wrapper interface over the various fs watching mechanisms on different OS's.
But for linux, I've already been using incron to do this job efficiently for ages.
[0] - https://www.man7.org/linux/man-pages/man1/inotifywait.1.html
There are other much more popular tools that do the same thing, like entr, watchexec, and nodemon. Why is this posted?
(yes, one owns the other, but Facebook employees writing and releasing code for Facebook are on Facebook's payroll, not Meta's payroll)
Watchman not yet being branded with “Meta” is a case of laziness. It doesn’t indicate it’s limited to the Blue app.
I had not intention about gaining from folks accidentally landing on my project. Apologies if people felt mislead. I have no idea why this was suddenly posted on Hacker News yesterday.
It’s been a few years, so perhaps it got addressed, but this library used to introduce a “memory leak” by design, by infinity storing pointers to files being watched.
It was a pain to isolate (https://techtldr.com/simple-guide-to-finding-a-javascript-me...)
Ultimately I ended up fixing it by writing my own version using native system calls available on Node.
The moral of the story, this library is great if you are building cross platform dev tool. It is not (at least wasn’t at the time) great for long lived processes that target only one type of operating system.
However, this is a single bash script, which definitely could be stable and forgotten about. Something like this would be more appealing as a posix shell script, but if you're using bash anyway and not trying to distribute, then stable and simple goes a long way.
Example .reflex.conf to run one command when an openapi spec changes and another when any go files change:
Then run it with:https://github.com/Cyphrme/watch
{ "path_to_watch":"example_command_to_execute_on_change.sh", }
https://github.com/e-dant/watcher
It hooks into system APIs where viable. (Otherwise, it uses std::filesystem.)
It’s meant to be as or more:
- Easy to use
- Efficient
Than all similar libraries.
Very composable as you just pipe its output to whatever you want (typically a while read do end), so exclusion is a awk/sed/perl/grep/rg away, and command can be as simple or complex as it needs to while the tool itself stays lean.
Cool features include -o to just be notified that something changed (e.g to fire up rsync), and ability to batch changes.
Also the command is a front end to libfswatch, so one can use that directly if shelling is to be foregone.
https://github.com/emcrisostomo/fswatch
I can't find the PDF manual in the repo. Do I need to contact the author and say please?
https://emcrisostomo.github.io/fswatch/doc/1.17.1/fswatch.pd...
I understand this is shell script territory, but a cross-platform tool would be ideal.