I wouldn't exactly call replacing AWK with Sed a "simplification". And now you have a stew of escaped $s. Invoking `tput` is a good idea, though; maybe you can use the AWK `system()` function for that.
For historical reasons my team uses Makefiles for executing data science workflows[1]. They are auto-documenting which makes it not just helpful on me when I forget an underused target, but it's perhaps even more useful when onboarding new team members. Rather than directing them to some out of band docs, it's convenient to simply direct them to use make to get the help they need[2].
1. There are reasons, like consistency of interface which I value even more than autodocumentation, but that is outside the scope of this discussion.
For data science specifically, I would strongly suggest looking into DVC: https://dvc.org/.
You can easily write DVC stage files by hand as a straightforward Makefile replacement, and integrate other features into your workflow as needed/desired.
PS: quite feature complete but not yet well marketed so to speak. I'm actually recording an asciinema session this week in order for a visitor to grasp quicker the mentioned benefits.
Which is also included in your excellent GNU Make Book¹, which you should've pimped directly ;) It is both a fantastic general guide and an amazing source of usable little recipes, thanks!
If you're using Make as a command runner or a "standard entry point" to a project, instead of using it as a build system that tracks dependencies between files, I highly recommend using `just` instead: https://github.com/casey/just
It has this functionality built-in, and avoids a lot of Make's idiosyncrasies. (Not affiliated, just a fan.)
Make is fiddly, has gotchas and is far from perfect, from the development side - but it's certainly "good enough", and it's great once you've got your Makefile done. And it's available, or at least easily attainable, pretty much everywhere. I even use it on Windows.
Yeah, but if you follow the advice FTA, you now have to rely on a usable python version and make. Still fairly common, but what if I use node instead? I think the point is -- the more usability you add the more you potentially dilute how ubiquitous it is. I think it's okay to point to new tools in the space that shake things up and add usability for the purposes Make currently is used for. Ubiquity doesn't happen over night.
Even if I followed the advice in the article (I don't really plan to), python3 is very likely to already be there in a Linux distro for other reasons, whereas node.js is very likely to not be there.
If you're on another OS where neither make nor python3 nor node.js are there by default, then sure you're already going to manually install make, so the choice of python3 or node.js is basically identical.
Also, if you add a new target that would benefit from tracking dependencies between files, then make is already ready to do that.
I use makefiles for my Rust code with just default+test+install targets that do no dependency tracking (since Rust's build system already does that much better). But if I need to add a target for, say, validating some OpenAPI spec that only needs to run if the spec file updates, then that can go in the same Makefile.
I wanted to use Just at one point, but was disappointed that it doesn't support parallel execution. I ended up making a quick tool to convert a ZSH file into a Ninja file to take advantage of its pool feature.
> GNU Make is a tool which controls the generation of executables and other non-source files of a program from the program's source files.
What is idiosyncratic is to think Make is first a command runner, and only after a file generator, and complain that it is doing a poor job at the former when it clearly says it is focusing on the latter.
I read the "idiosyncrasies" section on the just README and they are all performance related targeting file generation. All of those can be learned in one minute, as part of "I'll always find Makefiles in the wild, it's good to know the basics".
A bit reluctant to share this as it is old (2005?) and would not support features I tend to use now. But if you are just starting,
or stick to the most basic functionality it may be of use as a way to visualize the execution graph (DAG)
38 comments
[ 3.6 ms ] story [ 89.5 ms ] thread(Now if I could integrate that into the autotools we use for some of our stuff....)
[1] https://www.thapaliya.com/en/writings/well-documented-makefi...
It has sections and categorization.
1. There are reasons, like consistency of interface which I value even more than autodocumentation, but that is outside the scope of this discussion.
2. For small orders of help.
You can easily write DVC stage files by hand as a straightforward Makefile replacement, and integrate other features into your workflow as needed/desired.
PS: quite feature complete but not yet well marketed so to speak. I'm actually recording an asciinema session this week in order for a visitor to grasp quicker the mentioned benefits.
¹ https://nostarch.com/gnumake
It has this functionality built-in, and avoids a lot of Make's idiosyncrasies. (Not affiliated, just a fan.)
Make is fiddly, has gotchas and is far from perfect, from the development side - but it's certainly "good enough", and it's great once you've got your Makefile done. And it's available, or at least easily attainable, pretty much everywhere. I even use it on Windows.
If you're on another OS where neither make nor python3 nor node.js are there by default, then sure you're already going to manually install make, so the choice of python3 or node.js is basically identical.
https://github.com/rofl0r/jobflow
I use makefiles for my Rust code with just default+test+install targets that do no dependency tracking (since Rust's build system already does that much better). But if I need to add a target for, say, validating some OpenAPI spec that only needs to run if the spec file updates, then that can go in the same Makefile.
> GNU Make is a tool which controls the generation of executables and other non-source files of a program from the program's source files.
What is idiosyncratic is to think Make is first a command runner, and only after a file generator, and complain that it is doing a poor job at the former when it clearly says it is focusing on the latter.
I read the "idiosyncrasies" section on the just README and they are all performance related targeting file generation. All of those can be learned in one minute, as part of "I'll always find Makefiles in the wild, it's good to know the basics".
https://book.babashka.org/#_discoverability
https://github.com/TomConlin/MakefileViz
I don't use make often, so it also contains beginner friendly comments about the syntax
I published my self-documenting docker Makefiles here:
https://github.com/Infused-Insight/docker_makefiles