Ask HN: Software that treats a folder as the input to a queue?

2 points by elviejo 6y ago ↗ HN

I would like to receive files by FTP in a folder, and then put each file on a queue so that different workers can process it.

however my googling attempts haven't return satisfactory results?

Maybe I'm not searching for the right keywords? How do people with legacy systems, that receive text files through FTP, process each file through a pipeline of work in a reliable way?

2 comments

[ 21.0 ms ] story [ 271 ms ] thread

mtmail 6y ago ↗

We had to build our own 10 years ago. One thing we learned is to never allow FTP users to change, overwrite or delete files once uploaded. There are feature flags for that, I think we were using pureftp. Then our script moved all files out of the directory before processing.

wmf 6y ago ↗

Bunch of suggestions here https://stackoverflow.com/questions/4060212/how-to-run-a-she... https://superuser.com/questions/181517/how-to-execute-a-comm...

Also apparently systemd can do something like this. "DirectoryNotEmpty= may be used to watch a directory and activate the configured unit whenever it contains at least one file." https://www.freedesktop.org/software/systemd/man/systemd.pat...

The keyword is inotify since that's the (Linux) API that allows watching files/directories.