I'm pleased to see this being espoused. One thing I always tell junior people is ""don't be tempted to think of make(1) as out-of-date [1], and to view packages available via npm etc as better replacements, as many tasks fit the model of make(1), if not the title "a program for directing recompilation" "" I am often viewed with suspicion until I show them a (much more compact) replacement for their custom python/js program expressed as a makefile without any of the process management requiring debugging. :-)
An alternative approach for the "batches of files to process" situation is to generate the commands and feed them to GNU parallel for execution.
While i agree, I’ve spent literal days fighting with node modules in recent months. Sure make isn’t perfect either but it’s a lot less, confusing? Ie json isn’t a language for humans, but often you really need to read it to correct some issue or bad merge from another contributor.
> Sure make isn’t perfect either but it’s a lot less, confusing?
I think that's a matter of what you're used to. I don't use make all that often, so I find make files beyond a few lines quite a lot to get my head around, whereas I spend all day writing JavaScript so I can read/write JSON in my sleep.
Maybe. I actually use node much more often, i don't even recall the last time i used a make file in production, i just find parsing json as a human to be rather painful.
Make works fine, it's the commands you call from make that don't tend to as compatible. For example, try running rm -rf on windows. It might work if you have GNU tools installed, but it certainly won't work out of the box. On the other hand, the node.js `rimraf` package will do the same thing with no cross-platform compatibility issues.
No, that's why there's a bunch of packages such as rimraf[0] that implements that sort of functionality in a cross-platform way that most people use in their scripts
Biggest pain of package.json to me is having to be crammed on one line. Slap in a conditional and now we have a messy thing to read, compared to its multiline counterpart.
print-%: ; @echo '$(subst ','\'',$*=$($*))'
include Makefile
Save somewhere, eg as ~/Makefile.debug, then when you want to know the evaluated value of some variable or task in some Makefile, you prefix it with print- — for example, LOCALES:
make -f ~/Makefile.debug print-LOCALES
It'll print the evaluated LOCALES of the Makefile in the directory you run this in.
I have very similar use-cases, and I also I found makefiles a bit limiting. I wrote lk[1] to make this sort of thing easier, and you can just write plain bash functions instead of makefiles.
I've only started using a Makefile ~two years ago when I started a big Go project at work, and I quite like it. There's a few caveats here and there - Makefile specific syntax, shell-script specific syntax, and whether I should put .PHONY in front of every command (I guess 'no unless you have a file with the exact same name'?), but it's quite compact and straightforward.
The most complicated command I have is something that removes a folder or set of files, invokes the swagger generator with a list of options, copies it to a target folder, and passes it through a formatter/processor. But it's very straightforward, and it's "just" shell script, not shell script wrapped in a JSON document, or some Java tool invoked indirectly through an XML configuration file like back in the day with Maven.
I like Make, and use it in personal JavaScript ecosystem projects that do actually need to build things with interdependency (eg https://github.com/justjake/quickjs-emscripten/blob/master/M...). I’ve also seen Makefiles grown to be horrendous monstrosities masquerading as command-line tools; for about a year at Airbnb we used a 1000+ line Makefile as the main tool for fiddling with Kubernetes cuz one of our senior engineers didn’t like Ruby, and I’ve seen another one get close to that level of cravenness.
What I learned from supporting Make and shell among a few different audiences is that most developers have no interest in how to write or maintain shell-like tooling. They forget or mess up quoting rules constantly, and eschew learning things and good design in these tools to a much greater degree than in their “normal” work in Java/Python/Ruby/Typescript/Golang.
For every POSIXly Correct HN Commenter (of which I count myself a member), there’s 100 regular software engineers who won’t read a `man` page on what $@ or $< mean in Make. I know that if I start writing a Makefile for $JOB, it’s gonna be me and that one guy who uses tcsh who are gonna maintain it and answer questions.
(Although for what it’s worth we don’t use package.json scripts either thank goodness. All our complicated build steps are typescript commands, and our glue is CI system YAML files.)
I would try to use JavaScript tooling in JavaScript projects, especially if I share it with other developers.
I would love to see better handling of filename transformation, file watching, and parallelization.
A few times over the years I had given up and just used Make instead of Gulp or Grunt. Most recently, I wanted to watch if files had been deleted (if directory entries had been modified). It ended up being a few lines of Makefile.
And once you understand the syntax, it is expressive and simple, and way more portable. There was some good discussion awhile ago on this[1]:
But you don't write scripts in package.json and it also failed to show an example how the javascript dependencies could be loaded with make.
I think make works well in combination with package.json, but not as a replacement. You can also do the same stuff in an NPM command (which just loads node code) that you could do in a makefile script.
Not to mention that the top three package managers will load binary files from node_modules automagically by name only, without having to specify paths. That gets hairy in a monorepo, and package managers running package.json scripts manage them just fine.
For a lot of the same reasons in this article I like to use tup[0] when I can. It doesn't integrate into anything which is both good and bad. I wish my IDE could check with tup to see what dependencies get pulled in by what files. However, it's nice that it doesn't care what language or ecosystem I'm using.
Also, it's very strict about declaring dependencies properly, and will actually fail the build if you've set things up in a way that depends on something not tracked (by watching filesystem access as the compiler runs). That gives me warm fuzzies that my builds are reproducible.
Also I can get a neat dependency graph as a PNG if I want.
Over here in crazytown I'm using Makefiles to automate system deployments.
I thought it was insane at first but actually I've come around to it; Make hits that sweet spot of being ubiquitous, simple, and flexible. There's a little bit of a learning curve but once you're over that initial hurdle it's pretty straightforward, especially for one-way operations, e.g. install without uninstall, build without clean, etc.
This is one of the few fixes I'd really like to see. I'm from a Unix background so I don't tend to have spaces anywhere, but it's still a splinter that needs removing.
I would like to call attention to `just` (https://github.com/casey/just) which is a modern take on "make if we didn't have to keep it backwards compatible", and more focussed on just task running rather than tracking dependencies.
Yes! This is what a modern task runner should be. Includes ability to list recipes, and a consistent format between multiple platforms. Just two things Make doesn't have.
There's no -j option nor mention of parallelism on the docs. Does it run independent targets in parallel? This is easily the main feature of make.
I don't understand how it can work in parallel if targets are not somehow associated to files. What's the advantage to a shell script with separate sub-commands, then?
It sounds like `just` is not a replacement for make, then. It seems like it could be a nice wrapper on top, i.e. `just build` could invoke the make build system.
Parallelism is pretty useful for just a 'task runner' too.
I've (ab)used Make for server pushes, and sometimes you want those to run one at a time, and sometimes you want -j 100 (sometimes -j 100 is even effective).
The key aspect here is that since Make knows what the dependencies are it has enough information to automatically parallelize the task. A simple task runner would need to be specifically configured with the dependency graph, and if you have that you've just re-created Make.
If I'm using the task runner for deployment, it doesn't need to have a sophisticated dependency graph, it "just" needs to be able to have a list of tasks that are declared as independent/parallelizable, and a way to specify how many to run at once.
Sophisticated dependencies are nice to have of course, maybe make push-all transitively includes building the thing locally, but it's still useful to have just the push step (or the reload step, etc), have optional parallelism. If you start to build a tool that does do these things first, in order; then these things in any order; then these other things, that does look a lot like Make. but if you just handle ordered steps and independent steps, and let a person do the mixing, then it's not nearly as much work as Make.
What are you talking about? Make does not deal with "sophisticated" dependencies. Make is insultingly simple, not sophisticated at all. It's just a damn directed tree. The simplest data structure in computer science after the array.
There are also some fancier shortcuts for expressing rules in makefiles, but you can ignore them and use only the simple stuff. That's already a much more powerful tool than "just".
Tracking dependencies is what makes make make. Love it or hate it, Ant got this right (imho), and modern build systems almost universally get this wrong. Every language can run and compose tasks. Dependency tracking, parallelization, and language agnosticism are the killer features of make (again, imho).
I agree with you on all grounds (I personally use make daily) however note that `make` checks dependencies via last updated time flag on files. I think it would have been a better default to cache based on hash of file. This way if document was modified, by hash wasn't changed, make shouldn't rebuild tasks that depend on this file.
But computing the hashes of all the files would take the same time as rebuilding everything, or maybe even slower. And you'd need to re-compute all these hashes even for a no-op. It's absurd.
The nice thing about dates is that they can be checked instantaneously, and independently of file size. If a file has changed, its date has, so it's all the information you need.
Yep, I can take the sha256sum over my all my project's (vendored) dependencies in 10ms using the command:
time find ./vendor -type f -print0 | sort -z | xargs -0 sha256sum | sort -k2
| sha256sum
Note if using this code for practical purposes, you should be sure your collation is set appropriately. That's taken with 1280 files that are in total 19MB.
While 1280 files isn’t exactly small, it’s not really big either. Big as in where incremental builds really show their shine.
This might be approaching niche but I’ve seen a few mega-projects with 100 times more than that (also mostly due to vendored deps). You really don’t want to read 10G from disk for every incremental build. The actual sha computation isn’t the bottleneck, disk is. Ninja on the other hand, using mtime, handles this without a sweat.
The best solution would probably be to integrate the change detection with git, it is already content addressable and has lots of other smart things built in to make it fast.
Rebuilding potentially has side-effects. You can have a build system that, if a file is changed, it makes HTTP calls, generate docker containers, store artifacts in S3 etc. Given this, computing the hash would be negligable amount of work. Besides make does iterate over all dependencies every time to check last modified date filesystem keeps. So, at that point one can compute hash.
idempotent does not mean side effects free; a compile-to-the-cloud system for distributed caching can be idempotent (ocaml is building support in this direction)
Side effects that target on non local dir seems shouldn't be included in the build step. It should be in a standalone script/ target/whatever. Or the build process will not be reproducible at all.
builds should not use non-local inputs, but they can have non-local outputs. the simplest case would be automatically publishing a package or image to a private registry
If the file's contents have changed but the timestamp didn't, then something is broken and _that_ is the thing to fix, not rewriting the fundamental mechanism underneath the oldest relevant build tool.
There are always different preferences on deciding when something needs to be run again. File modification time is a reasonable default that serves many (most?) cases pretty well.
Yes, file hashes might be another way. On the other hand, if I modify a comment, the file hash will change, but re-compilation is not necessary. In this case, you might need yet another way...
You can always write your Makefiles to compute and depend on hashes or whatever else you might consider appropriate for your exact use case.
I've been using make for a long time but make really really sucks at parallelization and language agnosticism.
I mean those things work, but they're by no means nice. Parallelization works, but output of different tasks (and even the log that certain commands have been run) are interleaved, making it impossible to view progress. Language agnosticism works by operating on the level of shell commands, but that's really about it. If shell commands need to be customized on a per-system or per-OS level... well it's possible to hack around that with enough external scripts and even esoteric if statements inside the Makefile, but it's not nice.
Documentation for make feels like it's from the 80s. Which it is.
All these things (with the exception of having to rely on arcane sets of external scripts and if-statements-in-Makefile) are perfectly fixable. But they don't get fixed.
I haven't understood what the point is for a "task runner" over having a bin/ directory in the project with some shell scripts. If it's not going to do anything extra for you (like run commands according to a dependency tree like make), I don't see the point.
`just` seems like a useful tool, but it only covers a secondary use case of `make`. `make` without a dependency graph is like a bicycle without wheels.
All I can say is that I’ve only ever used make as a task runner myself, so just matches my need way better.
IME make’s model is often too simplistic to be of use, so it simply gets in the way of tools already doing things internally with proper insight, or requires force-cleaning everything because it's too dumb and just breaks the build when you e.g. switch branch.
Just’s author is also looking at integrating `redo` for building / dependency tracking, though the issue has no activity since it’s been created.
If you find yourself running "make clean" because it is "too dumb" you've built your Makefile wrong. Usually it means you've tried to manually re-create the inherent Make rules and have gotten it wrong somewhere. This is incredibly common, since the inherent rules are not always the same across various versions of Make and people don't always trust them.
IMHO, this is the primary weakness of Make. The standardization efforts never really happened so you have various forks of 1980s code with annoying incompatibilities. BSD Make is missing some of the implicit build rules from GNU Make (only able to compile simple single source file applications without an explicit rule), and Windows Make tends to be extremely primitive. Worse, Make still in 2022 can't handle spaces in file names, like c:\Documents and Users\.
I would be so much happier if everybody could standardize on one flavor of Make. Probably GNU Make.
> If you find yourself running "make clean" because it is "too dumb" you've built your Makefile wrong.
What's built wrong is make itself. Because the signal it uses for "this dependency has changed" is the mtime, any change which does not move the mtime forwards will be ignored. Switching between concurrent git branches, for instance.
> Usually it means you've tried to manually re-create the inherent Make rules and have gotten it wrong somewhere.
And pray tell how do you "manually re-create the implicit make rules" when such rules don't exist? GNU Make has implicit rules for C, C++, Pascal, Fortran, Raftor, Modula-2, and assembly. That is, frankly, not much. And mostly outdated.
Use make for dependency graph, but also use "npm run" or "yarn run" to make sure you're running with the correct Node and correct paths to all your tools.
I know you can specify those in your Makefile too, but the plus of running via npm/yarn is that they know where your binaries live, where your mode_nodules are, and that sort of thing.
Yes, I usually treat Make as the user-friendly facade to the underlying framework tooling (which might be e.g. Rails, Docker, or npm). `make bootstrap` gets the project setup. `make up` starts the project.
For npm projects these usually just wrap package.json stuff. But I don't need to remember what tech a project is in, or where to look for the commands, if Make is the entry point.
Out in Grand Rapids, Mi where AO is located there seems to be a bit of a tradition of abusing make in a good way. It's where I picked up the habit as well. I think its just institutional knowledge/tradition that has creeped through the various firms here.
I kind of wonder what other "traditions" get passed around on a regional level?
I also use Makefiles, been using them for 40 years. I find that Makefile targets for misc. things like grabbing remote training data, running tests, building documentation, etc., etc. augments information in READ files, that is in addition to functionally saving time, serves as documentation when I haven’t looked at a project in 6 months.
No, make understands the directed acyclic graphs of dependencies, regardless of language, and will walk that graph in parallel. No forcing or ignoring sorting, and no enforced package level blocking on scripts.
It's a shame this doesn't work for Java in particular (not JS) as Java's coding conventions/requirements do not lend themselves at all to manually updating a list of file dependencies.
The Makefile is the facade to the project. So you can have `make run` do `make run-windows` or `make run-nix` under the hood. And when you change hosts, or you redo the `run-X` targets into a single cross-platform binary or you put it in a Docker container, nothing changes. You don't need to remember that you need to invoke X or Y script.
The makefile lets me be very expressive about task dependencies and encode some developer conveniences. Ie make can know `make bootstrap` should invoke bootstap_windows.sh on Windows and bootstrap_osx.sh on Mac. Then the work happens inside the appropriate script.
They’re two tools that solve different problems but they work together quite well.
Imagine a simple two-step process. The first takes a long time. The second one fails on occasion. You don't want to run the first one again. With Make, if the first one succeeds, it won't run again unless you `clean`
It also compares the last-modified timestamps of the source and target files so it only runs if the target is out of date. Plus because the dependencies are explicit, it's trivial to run in parallel the parts that can be.
Stick around in any programming ecosystem long enough, and you live long enough to see strategy, taste, and "discovery" come full circle. We had make, then package.json "scripts," then grunt, then gulp, then the trend shifted back towards package.json "scripts." (Note: Using "scripts" to differentiate the package.json property versus the generic word use)
I like that the author isn't being authoritative, but there could be some additional due diligence. I ran the make-is-faster loose benchmark via PNPM and runtime was 0m0.022s for make and 0m0.012s for PNPM. If I care about those 10ms, PNPM is my horse. Yarn is a glacier compared to PNPM.
Another thing I would've liked to see comment on is the automatic pre and post paradigm that package.json "scripts" affords. The big three package managers all support pre and post, and it makes arranging "scripts" a breeze, it breaks down dependent steps into separate console output, and is generally easy to organize imho.
All in all a nice write up for folks who might not really like package.json "scripts" to begin with, or for those who'd rather not gain more granular insight into how "scripts" works, but I don't see this being the nail in the coffin case against them.
The caveat here is that the binaries aren't "shipped" with node.
`corepack enable` adds shim scripts to node's $PATH to intercept calls to `npm` `yarn` or `pnpm` and downloads the binaries from a URL without much checking along the way. This functionality has had some vocal opposition.
I’ve used pnpm religiously for over a year and have seen it being adopted as a golden standard replacement for npm in my circles. Highly recommend it to anyone who works with JavaScript or Typescript!
This seems to be missing the obvious point: Make isn't anchored to any single language. If you will only ever use npm for your entire life, then you go ahead an live happily in package.json script land.
But I've lost track of the number of languages and environments that I've worked in. Make ties it together by being the Good Enough anchor point that documents and launches all the other tools and compilers.
- Compile this Rust app with Cargo then flash it to the esp32? Make.
- Build this page with JS using Tool Of The Week and push it to a test container? Make.
- Compile ancient C app and build a .deb? Make.
And so on.
Why not use shell scripts you may ask? Because shell scripts are way too free form and your tastes will change over time. Make forces just enough structure that you won't get carried away with yourself. Not for the basic task running stuff anyway.
I agree, but make is usually more abundant on much more systems (which is useful when there is no root privileges available). Also people are used to run make when they encounter a Makefile. I personally use Makefiles even to create Docker images, I find it simpler to run make images or make run, than remembering (or putting in a script and remember what parameters to use, as pointed in the comment above scripts are too free-style) how to do it manually.
Though I also agree that Makefile is not a silver bullet and some more complex/niche methods may be required in particular cases.
I've been working with Bazel by myself. My experience is that it is not a nightmare... it's just the documentation is missing some pretty crucial "how-to" guides, and there are a couple features that changed a lot prior to 1.0 (so using them is a bit difficult).
Some stuff that is very easy with makefiles is a bit harder with Bazel, so I can't give it an unqualified recommendation. The payoff is that some things that are very, very hard with makefiles are much easier with Bazel (using multiple languages, cross-compiling, downloading dependencies automatically, distributed builds, etc).
My experience is mostly through consuming packages that other people have written, and the downloading existing packages is a big part of why it's problematic. If I already have PrettyCommonStandardLibraryX, I don't really want Bazel to download another copy of it for me just to incorporate your project. Other build systems are simple enough to change, or support this straightforwardly. Doing it with Bazel is quite painful IMO.
I like nothing more than dropping into a new project, and, rather than figuring out the commands to make it run, see that it has a Makefile with a handful of targets for common tasks.
I could not agree more. However most people are on windows for which obtaining gnu make is painful, or maybe even impossible - at least the path handling has many sharp edges.
Indeed – although I guess WSL made unix tools slightly easier to use nowadays.
But even without executing the Makefile, simply reading it can tell new developers which language-specific command needs to be run to build the project (and then the command can be copy-pasted and run manually).
I don't comprehend how people can develop on windows (without linux subsystem).
I have to do that every once and then (to ship the occasional C++ or Rust build on Windows) and it's the stuff of nightmare. Stuff breaking randomly from one day to another, env variables to be set in weird ways, GUI installers, 8 different versions of mingw or similar. Recently I've seen that there are a few package managers now (I used chocolatey and at least 2 others just trying to get something to compile) but still, compiling something trivial is always an adventure.
Mac OS X is kind of okay. Brew is barely decent and things mostly work (unless you discover you need to install 12GB of XCode for some dependency or your script is expecting coreutils instead of bsd).
Every linux distro comes with a package manager and compiling is trivial
Homebrew is a subpar choice on macOS; MacPorts is faster, has more packages, and is implemented more correctly than Brew.
Additionally, MacPorts was co-created by an engineer who also created the original FreeBSD Ports system, and thus hews much more closely to standard UNIX/BSD practices.
I’m not sure how and when Homebrew became the standard, but it is definitively worse.
Using Homebrew and multiple users is excruciating and an eye opener on how system-level software should really be installed.
Homebrew insists on avoiding root privileges whilst also installing packages system-wide. That works fine and is invisible with one user but falls down hard otherwise.
Their documentation is incorrect too, saying that this is all fine because “we install in /usr/local/bin”. It’s not easy to change this.
The solution was to embrace MacPorts which correctly requires root privileges to install system-wide packages.
I haven’t looked back since. I haven’t missed brew or any software that’s available on brew alone.
Too bad when I want to install newer fancy tools on Linux, it's easier to unify the environment with Homebrew instead of using Homebrew only on Linux (which has a very weird quirk that it wants to install in /home/linuxbrew/.linuxbrew (or else it'll start compiling so many packages instead rolling binary packages) instead of /opt/homebrew like any sane decision would've been.)
I also don't understand Nix when it wants to make 30 users for build process and a few unintuitive decisions. Otherwise it's good that it works same on macOS and Linux.
For me compiling usually is something like `cargo build`, `ng build`.
I remember having problems with libs that require installing & registering a library somewhere such that CMake can find it. However, I distanced myself a bit from C(++), so that doesn't really happen anymore :)
I avoid mingw, don't use any package manager besides Windows Store (if you want to call that a package manager).
Can't complain. Sometimes, there is stuff that simply doesn't support Windows -> WSL.
When there is docker, it doesn't matter anyways...
My strategy is don't fight Windows and you'll be happy
From your description it sounds like you might be going off the beaten track and hitting problems.
When I was doing C++ on Windows getting a dev environment setup just meant installing Visual Studio with an appropriate Windows SDK version (or the Windows SDK + build tools for a build system).
You can have multiple VS versions installed side-by-side. To get a terminal with environment variables set correctly you just need to use the shortcuts from your VS installation.
For third party dependencies we checked the headers and (pre-built) binaries into the repository. I don’t remember ever having more than a dozen or so in total. It was usually things like boost and zlib.
Having done that you can just point CMake directly at the packages rather than worrying about FindPackage.
Working in tools like Python and Node, personally I often miss the simplicity and stability of this approach.
Well, most developers I work with benefit from a task runner. Now, we use different runners for different repos. I’d prefer being able to go into a repo and do «make test» and have it work, regardless of language, framework, purpose of the repo.
And it still holds true for development today! Really any organization that has multiple languages should consider making their devs lives easier with Make.
The only bad part about Make is that it doesn't provide even the most basic utilities for doing cross platform work like removing a directory. It mostly works with Unix, but writing a Makefile that works on Windows and Unix is really hard.
If Make had just a few more batteries included, it would be perfect. However, maybe adding those batteries would take away from its other qualities.
Nowadays there's Windows Subsystem for Linux. There's no excuse not to successfully run "Linux" scripts on Windows.
I've been running very complex build systems via https://github.com/ysoftwareab/yplatform (disclaimer: author here) since 2016 on Linux, Mac and Windows without a problem.
Make is a file-based dependency tree that only defaults to a specific shell. If you really want to go crazy, you could actually change it to anything that can be invoked with "<executable> <file>", such as python, and use that language instead.
As a really crude example:
SHELL := python
.ONESHELL:
out.csv:
import csv
with open("$@", "w") as f:
writer = csv.writer(f)
writer.writerow(["header1", "header2"])
and so:
$ make
import csv
with open("out.csv", "w") as f:
writer = csv.writer(f)
writer.writerow(["header1", "header2"])
$ cat out.csv
header1,header2
I get that this doesn't exactly solve your "cross-platform work" gripe, but my point is that a lot of what people perceive as make providing isn't actually make in the first place.
The second I see a node project built in make, I remove it
That's a person who doesn't understand portability, tooling, or the need to stick to community norms. Everything about their library is going to be pure pain.
I don't have a problem with make files, but I do prefer to work directly with `package.json` when reasonable. Most notably, most of the IDE's work very well with `package.json`, but don't always handle make as cleanly.
Make has been around for decades, and with good reason. I've enjoyed watching the grunts and gulps and whatevers appear as the New Shiny Thing, become complicated and collapse under their own weight as people struggle with overly complex configs, as I quietly do the thing that has always worked and has not changed in forever.
Pretty convincing writeup. I've only used makefiles when compiling C / C++ but lately I'm finding even those are moving to CMAKE with makefiles becoming less and less common.
181 comments
[ 5.7 ms ] story [ 352 ms ] thread(My last few forays into the build systems domain have mostly turned up “Bazel but for X audience” so I can’t recall a satisfying solution)
An alternative approach for the "batches of files to process" situation is to generate the commands and feed them to GNU parallel for execution.
[1] no pun intended
I think that's a matter of what you're used to. I don't use make all that often, so I find make files beyond a few lines quite a lot to get my head around, whereas I spend all day writing JavaScript so I can read/write JSON in my sleep.
[0]: https://www.npmjs.com/package/rimraf
https://community.chocolatey.org/packages/make
I cannot overstate how annoyed I was by the patronising tone of this comment.
[1] https://github.com/jamescoleuk/lk
The most complicated command I have is something that removes a folder or set of files, invokes the swagger generator with a list of options, copies it to a target folder, and passes it through a formatter/processor. But it's very straightforward, and it's "just" shell script, not shell script wrapped in a JSON document, or some Java tool invoked indirectly through an XML configuration file like back in the day with Maven.
What I learned from supporting Make and shell among a few different audiences is that most developers have no interest in how to write or maintain shell-like tooling. They forget or mess up quoting rules constantly, and eschew learning things and good design in these tools to a much greater degree than in their “normal” work in Java/Python/Ruby/Typescript/Golang.
For every POSIXly Correct HN Commenter (of which I count myself a member), there’s 100 regular software engineers who won’t read a `man` page on what $@ or $< mean in Make. I know that if I start writing a Makefile for $JOB, it’s gonna be me and that one guy who uses tcsh who are gonna maintain it and answer questions.
(Although for what it’s worth we don’t use package.json scripts either thank goodness. All our complicated build steps are typescript commands, and our glue is CI system YAML files.)
I would love to see better handling of filename transformation, file watching, and parallelization.
A few times over the years I had given up and just used Make instead of Gulp or Grunt. Most recently, I wanted to watch if files had been deleted (if directory entries had been modified). It ended up being a few lines of Makefile.
And once you understand the syntax, it is expressive and simple, and way more portable. There was some good discussion awhile ago on this[1]:
[1]: https://news.ycombinator.com/item?id=7622296
I think make works well in combination with package.json, but not as a replacement. You can also do the same stuff in an NPM command (which just loads node code) that you could do in a makefile script.
Also, it's very strict about declaring dependencies properly, and will actually fail the build if you've set things up in a way that depends on something not tracked (by watching filesystem access as the compiler runs). That gives me warm fuzzies that my builds are reproducible.
Also I can get a neat dependency graph as a PNG if I want.
[0] https://gittup.org/tup/
I thought it was insane at first but actually I've come around to it; Make hits that sweet spot of being ubiquitous, simple, and flexible. There's a little bit of a learning curve but once you're over that initial hurdle it's pretty straightforward, especially for one-way operations, e.g. install without uninstall, build without clean, etc.
https://www.thapaliya.com/en/writings/well-documented-makefi...
I don't understand how it can work in parallel if targets are not somehow associated to files. What's the advantage to a shell script with separate sub-commands, then?
So I assume it doesn’t need/want to implement parallelism.
I've (ab)used Make for server pushes, and sometimes you want those to run one at a time, and sometimes you want -j 100 (sometimes -j 100 is even effective).
Sophisticated dependencies are nice to have of course, maybe make push-all transitively includes building the thing locally, but it's still useful to have just the push step (or the reload step, etc), have optional parallelism. If you start to build a tool that does do these things first, in order; then these things in any order; then these other things, that does look a lot like Make. but if you just handle ordered steps and independent steps, and let a person do the mixing, then it's not nearly as much work as Make.
What are you talking about? Make does not deal with "sophisticated" dependencies. Make is insultingly simple, not sophisticated at all. It's just a damn directed tree. The simplest data structure in computer science after the array.
There are also some fancier shortcuts for expressing rules in makefiles, but you can ignore them and use only the simple stuff. That's already a much more powerful tool than "just".
The nice thing about dates is that they can be checked instantaneously, and independently of file size. If a file has changed, its date has, so it's all the information you need.
time find ./vendor -type f -print0 | sort -z | xargs -0 sha256sum | sort -k2 | sha256sum
Note if using this code for practical purposes, you should be sure your collation is set appropriately. That's taken with 1280 files that are in total 19MB.
For raw throughput, you can use BLAKE2 for even more speed https://www.blake2.net/
This might be approaching niche but I’ve seen a few mega-projects with 100 times more than that (also mostly due to vendored deps). You really don’t want to read 10G from disk for every incremental build. The actual sha computation isn’t the bottleneck, disk is. Ninja on the other hand, using mtime, handles this without a sweat.
The best solution would probably be to integrate the change detection with git, it is already content addressable and has lots of other smart things built in to make it fast.
Unless the date hasn't changed -- there is no hard guarantee that mtime will be accurate.
Yes, file hashes might be another way. On the other hand, if I modify a comment, the file hash will change, but re-compilation is not necessary. In this case, you might need yet another way...
You can always write your Makefiles to compute and depend on hashes or whatever else you might consider appropriate for your exact use case.
I mean those things work, but they're by no means nice. Parallelization works, but output of different tasks (and even the log that certain commands have been run) are interleaved, making it impossible to view progress. Language agnosticism works by operating on the level of shell commands, but that's really about it. If shell commands need to be customized on a per-system or per-OS level... well it's possible to hack around that with enough external scripts and even esoteric if statements inside the Makefile, but it's not nice.
Documentation for make feels like it's from the 80s. Which it is.
All these things (with the exception of having to rely on arcane sets of external scripts and if-statements-in-Makefile) are perfectly fixable. But they don't get fixed.
IME make’s model is often too simplistic to be of use, so it simply gets in the way of tools already doing things internally with proper insight, or requires force-cleaning everything because it's too dumb and just breaks the build when you e.g. switch branch.
Just’s author is also looking at integrating `redo` for building / dependency tracking, though the issue has no activity since it’s been created.
IMHO, this is the primary weakness of Make. The standardization efforts never really happened so you have various forks of 1980s code with annoying incompatibilities. BSD Make is missing some of the implicit build rules from GNU Make (only able to compile simple single source file applications without an explicit rule), and Windows Make tends to be extremely primitive. Worse, Make still in 2022 can't handle spaces in file names, like c:\Documents and Users\.
I would be so much happier if everybody could standardize on one flavor of Make. Probably GNU Make.
With included Guile scripting support!
https://www.gnu.org/software/make/manual/html_node/Guile-Exa...
I really wish it was more common because having it would be great.
What's built wrong is make itself. Because the signal it uses for "this dependency has changed" is the mtime, any change which does not move the mtime forwards will be ignored. Switching between concurrent git branches, for instance.
> Usually it means you've tried to manually re-create the inherent Make rules and have gotten it wrong somewhere.
And pray tell how do you "manually re-create the implicit make rules" when such rules don't exist? GNU Make has implicit rules for C, C++, Pascal, Fortran, Raftor, Modula-2, and assembly. That is, frankly, not much. And mostly outdated.
Use make for dependency graph, but also use "npm run" or "yarn run" to make sure you're running with the correct Node and correct paths to all your tools.
I know you can specify those in your Makefile too, but the plus of running via npm/yarn is that they know where your binaries live, where your mode_nodules are, and that sort of thing.
For npm projects these usually just wrap package.json stuff. But I don't need to remember what tech a project is in, or where to look for the commands, if Make is the entry point.
I kind of wonder what other "traditions" get passed around on a regional level?
https://taskfile.dev/#/
Also I'm not sure Windows can run Makefiles out-of-the-box.
I'm pretty sure git-bash provides `Make` on Windows. I believe thats how I tested it for this project: https://github.com/J-Swift/cod-stats
The makefile lets me be very expressive about task dependencies and encode some developer conveniences. Ie make can know `make bootstrap` should invoke bootstap_windows.sh on Windows and bootstrap_osx.sh on Mac. Then the work happens inside the appropriate script.
They’re two tools that solve different problems but they work together quite well.
Imagine a simple two-step process. The first takes a long time. The second one fails on occasion. You don't want to run the first one again. With Make, if the first one succeeds, it won't run again unless you `clean`
Easy, but now imagine many, many steps
if [[ -f myfile ]]; then
But it's trivial to run shell scripts in parallel as well, just add ` &` at the end.
I like that the author isn't being authoritative, but there could be some additional due diligence. I ran the make-is-faster loose benchmark via PNPM and runtime was 0m0.022s for make and 0m0.012s for PNPM. If I care about those 10ms, PNPM is my horse. Yarn is a glacier compared to PNPM.
Another thing I would've liked to see comment on is the automatic pre and post paradigm that package.json "scripts" affords. The big three package managers all support pre and post, and it makes arranging "scripts" a breeze, it breaks down dependent steps into separate console output, and is generally easy to organize imho.
All in all a nice write up for folks who might not really like package.json "scripts" to begin with, or for those who'd rather not gain more granular insight into how "scripts" works, but I don't see this being the nail in the coffin case against them.
Take a look: https://pnpm.io
https://nodejs.org/api/corepack.html
`corepack enable` adds shim scripts to node's $PATH to intercept calls to `npm` `yarn` or `pnpm` and downloads the binaries from a URL without much checking along the way. This functionality has had some vocal opposition.
https://pnpm.io/
But I've lost track of the number of languages and environments that I've worked in. Make ties it together by being the Good Enough anchor point that documents and launches all the other tools and compilers.
- Compile this Rust app with Cargo then flash it to the esp32? Make.
- Build this page with JS using Tool Of The Week and push it to a test container? Make.
- Compile ancient C app and build a .deb? Make.
And so on.
Why not use shell scripts you may ask? Because shell scripts are way too free form and your tastes will change over time. Make forces just enough structure that you won't get carried away with yourself. Not for the basic task running stuff anyway.
Though I also agree that Makefile is not a silver bullet and some more complex/niche methods may be required in particular cases.
Some stuff that is very easy with makefiles is a bit harder with Bazel, so I can't give it an unqualified recommendation. The payoff is that some things that are very, very hard with makefiles are much easier with Bazel (using multiple languages, cross-compiling, downloading dependencies automatically, distributed builds, etc).
With some collegues we wrote an article about the benefits of this approach a few years ago: https://blog.capitaines.fr/2016/09/30/standardizing-interfac...
But even without executing the Makefile, simply reading it can tell new developers which language-specific command needs to be run to build the project (and then the command can be copy-pasted and run manually).
I have to do that every once and then (to ship the occasional C++ or Rust build on Windows) and it's the stuff of nightmare. Stuff breaking randomly from one day to another, env variables to be set in weird ways, GUI installers, 8 different versions of mingw or similar. Recently I've seen that there are a few package managers now (I used chocolatey and at least 2 others just trying to get something to compile) but still, compiling something trivial is always an adventure.
Mac OS X is kind of okay. Brew is barely decent and things mostly work (unless you discover you need to install 12GB of XCode for some dependency or your script is expecting coreutils instead of bsd).
Every linux distro comes with a package manager and compiling is trivial
They want to distribute their app on Windows?
MSVS is really quite nice.
Additionally, MacPorts was co-created by an engineer who also created the original FreeBSD Ports system, and thus hews much more closely to standard UNIX/BSD practices.
I’m not sure how and when Homebrew became the standard, but it is definitively worse.
Using Homebrew and multiple users is excruciating and an eye opener on how system-level software should really be installed.
Homebrew insists on avoiding root privileges whilst also installing packages system-wide. That works fine and is invisible with one user but falls down hard otherwise.
Their documentation is incorrect too, saying that this is all fine because “we install in /usr/local/bin”. It’s not easy to change this.
The solution was to embrace MacPorts which correctly requires root privileges to install system-wide packages.
I haven’t looked back since. I haven’t missed brew or any software that’s available on brew alone.
I also don't understand Nix when it wants to make 30 users for build process and a few unintuitive decisions. Otherwise it's good that it works same on macOS and Linux.
I remember having problems with libs that require installing & registering a library somewhere such that CMake can find it. However, I distanced myself a bit from C(++), so that doesn't really happen anymore :)
I avoid mingw, don't use any package manager besides Windows Store (if you want to call that a package manager).
Can't complain. Sometimes, there is stuff that simply doesn't support Windows -> WSL. When there is docker, it doesn't matter anyways...
My strategy is don't fight Windows and you'll be happy
When I was doing C++ on Windows getting a dev environment setup just meant installing Visual Studio with an appropriate Windows SDK version (or the Windows SDK + build tools for a build system).
You can have multiple VS versions installed side-by-side. To get a terminal with environment variables set correctly you just need to use the shortcuts from your VS installation.
For third party dependencies we checked the headers and (pre-built) binaries into the repository. I don’t remember ever having more than a dozen or so in total. It was usually things like boost and zlib.
Having done that you can just point CMake directly at the packages rather than worrying about FindPackage.
Working in tools like Python and Node, personally I often miss the simplicity and stability of this approach.
It's the same problem with docker-compose files; how do you expect the developers not running in windows to fix your windows problems?
https://community.chocolatey.org/packages/make
I did implement though that uniform interface in https://github.com/ysoftwareab/yplatform
And of course, it plays great with Docker & Compose, here's a write-up we did on using Make with container tooling: https://shipyard.build/blog/makefiles-for-modern-development...
If Make had just a few more batteries included, it would be perfect. However, maybe adding those batteries would take away from its other qualities.
I've been running very complex build systems via https://github.com/ysoftwareab/yplatform (disclaimer: author here) since 2016 on Linux, Mac and Windows without a problem.
As a really crude example:
and so: I get that this doesn't exactly solve your "cross-platform work" gripe, but my point is that a lot of what people perceive as make providing isn't actually make in the first place.Worth noting, each recipe can have its very own SHELL[0] (e.g., ruby recipe, that uses ruby command with -e flag):
[0]: http://agdr.org/2020/05/14/Polyglot-Makefiles.htmlThat's a person who doesn't understand portability, tooling, or the need to stick to community norms. Everything about their library is going to be pure pain.
cmake can help portability across OSes but it has a layer of its own complexity, I mostly work on linux alone, makefile seems more than enough.
google etc produces new tools to build its huge code base but I don't have that large scale source code, makefile so far worked well for my scale.
unless you have specific needs I feel makefile will do just fine. it's simple, readable, manageable, and get the job done fast.
https://github.com/adriancooney/Taskfile