I wish the make documentation was better though. I've also read oftentimes that people say that turning on make concurrency with `-j` is on your own risk.
That's because people write shitty makefiles. Most makefiles I see are full of improperly used double colon rules, phony targets not marked .PHONY and implicit ordering assumptions.
If `-j' was enabled by default, then it would be much harder for writers to develop these issues. But I suspect most of these authors, who are already unwilling to learn make properly would simply give up and use a different tool.
The problem is that most people don't set out to be masters of make. They just want to get their shit built so they can make forward progress on the thing they want to create.
Having to learn all of these subtleties is what gets you into these problems in the first place. I'll happily take a more opinionated build tool with proper escape hatches over something that expects you to know it in detail for even the most basic tasks.
I really hate this attitude. The same people who don't grok Make also don't grok git or bourne shell or who knows what else. Don't take shortcuts in understanding the fundamental tools of your stack. Your build system is just as important to your project as the code it's building!
Make is only a fundamental tool if that's what your library/binary uses. You may have a mix of CMake, MSVC and language specific build tools like Cargo and others. By your logic I should be an expert in all of them.
Make's support outside of the posix world isn't that great(as opposed to CMake) so calling it fundamental seems that you are making assumptions about the environments that development happens in.
Yes, but also it's make's fault -- it is possible to notice and complain if you write something that's not || safe, rather than just giving you a flaky build. (Cf. tup.)
Which, coincidentally, is my main complaint with Make: It's way too easy to write shitty Makefiles. There are way too many wrong ways to do things. Arguably this is a community problem and not a tool problem, but it doesn't change the fact that most Makefiles are written badly.
I used Make professionally for 5 years and have since moved to other tech stacks that don't require Make but have tools that are far worse in every other way (cough Ant cough). But IMO the reason they win out is they are far less flexible, so there are fewer ways to do things, so they work better in aggregate.
Build tooling is something that most devs look at very rarely, so they never build expertise. Ergo, build tooling shouldn't require expertise.
If my theory holds out, I feel Gradle will become the new Make in a few years or a decade - very powerful but too many wrong ways to do things.
Make is more than 40 years old -- 40 fucking years! And we still use it. It's everywhere. It's not going anywhere. Gradle will not replace it.
Why? Because make works. Because it's actually easy to understand if you've read the manual. It's simple. People try to make it do really complicated things, which you are better off putting in a script or another program. At it's core, make works and does its job very, very well.
It's hard to write portable anything without details leaking out. Go is the best implementation I've ever seen here. It works identically everywhere I've tried it. The approach the Go team has taken is extreme. I think that says a lot.
Both these tools are very simple, and limit what you can do. When you try to go outside the box, you hurt yourself. Don't hurt yourself.
Tools like Gradle make it real easy to hurt yourself without realizing it.
Look, this battle has been lost for ages. People only seem to write portable Makefiles by accident and usually just target the implementation of make on the platform they're developing on.
Unfortunately, when you learn Gradle, you also have to learn its procedural description language, Apache Groovy, with its many quirks, such as when to use
I think people write bad makefiles because the documentation is bad. Its "taking the user by the hand and pushing him into the right direction"-level is just not up to date anymore.
Make is a good tool because it allows a lot of flexibility, but it doesn't protect its users from the pitfalls this flexibility unavoidably creates.
Make is possibly the most popular thing people re-invent for specific platforms.
Touch of irony in this project, with that Makefile sitting in the root directory.
EDIT: My apologies if the above comment was cynical. There's a salient need for workflow systems which provide features above/beyond the basics of make. Built-in things like a central broker, adjustable worker processes, scheduling and history. Python/Luigi is an example of such a workflow system, yet it keeps the workflow definitions very similar to Make for conciseness
And tup for the last 10. Lots of people in this thread complaining about Make's syntax might want to check it out; it has a nice, simple "inputs |> command |> outputs" syntax, a similar "output the DAG" mode to Cr, and defaults to parallelism.
That's funny because I would list tup's syntax as a negative for an otherwise very useful tool.
Also, Tup is much more single-purpose than Make or (from a quick readthrough) Cr. It is very strongly focused on the "N files in, 1 file out" for compilation, and specializes at that to the point of not doing other things.
As far as a minimal "better make" I find redo to be the best. Make is a weird combination of the shell and makefile languages, but redo boils it down to just a single language and manages to handle dependency of build rules in an elegant manner.
I love tup, but I find too many places where it doesn't work to use it for things I distribute (sometimes fuse fails, doesn't work on Linux subsystem for Windows, flakey on full windows).
Everyone is going to compare it to Make, but it's not like Make syntax is ideal or consistent; I can see this being especially useful when using it in part of a pipeline, where maybe the YAML is generated elsewhere.
On the topic of syntax, I'll say YAML is not the best format for configuration either. At a glance, it looks simple and easy to write, but YAML can bite its user in an unexpected ways due to its implicit typing[1]. For example this YAML:
on: yes
…will be parsed to:
{true => true}
…which cannot be serialized to JSON in a lossless way (since JSON spec specify that a key must be a string):
{"true":true}
…and translating this back to YAML would result in:
'true': true
This is without considering other features such as anchor, aliases and tags[2], which further make the YAML spec incredibly complex.
> But most sane people would agree that Makefile generation is not a fun business. Autotools, anyone?
I use Make for a range of things (specifically, not c/c++/etc though) and I've never used Autotools/Automake etc (except compiling a project that uses it occasionally)
I disagree with this notion. Code generation is limited by the approach, the input information and the output language. Make is (subjectively) an ugly language, text macros are a bad approach. That's why autotools sucks in my opinion.
OTOH, approaches like Go's 'Stringer' are pretty clever and useful, and generate good code from many PoVs.
>It's not even a useful thing to do.
Look, if you write nice, minimal Makefiles, that's great. It's not what I was talking about. I know autogenerating Makefiles sucks. I prefer Makefiles like those from projects like musl. But some projects are too big and complicated.
If I want to target a huge number of platforms while also supporting cross compiling, Make starts to break down rapidly. Out of tree builds are also a pain.
Tools like qmake and autotools can alleviate this by autogenerating Makefiles and project files abstractly. That way, I don't have to make configuration decisions on every single invocation. This gets especially tedious if you need to run pkg-config many times or sdl-config, or maybe some platform dependent things, or what have you. Or if you want to do things Make isn't good with, like recursing or acting on a wildcard of files. Soon enough your Makefile is calling find and relying on the BSD or GNU version of find, someone tries to run it on MinGW and gets confused by the weird argument error with find.
Lots of people use Make for it's ability to concurrently run tasks with dependencies. This program presents just that part of make with a declarative syntax that is easily generated programmatically.
If you don't see a point, I can understand. Personally, I have no plans to use Cr. But if I was writing a kind of build system that generated a manifest, I'd rather generate a YAML manifest than a Makefile.
Looks nice! Will inspire my project as well. Made something recently as well which is based on the same principle. A simple parallel/sequential task runner which has built in retries and allows steps to be continued even if they return errors.
I'm sure somebody has a use for this. But with no scheduler tuning, no prioritization, no dynamic configuration, no multiprocess logic, etc it's effectively just "xargs" with dependency tracking, written in Go. And I'm pretty sure you can do this with Ansible.
A co-worker of mine recently implemented something very similar.
Our major feature over Make was emitting events as newline separated JSON, so that a downstream process can visualize them in a pretty way, track progress, etc.
We use ours run multiple Chef recipes in parallel.
54 comments
[ 2.0 ms ] story [ 63.2 ms ] threadProblems arise when one of your build steps creates temporary files for each invocation and they end up overwritting each other (race condition).
This would have the same issue
If `-j' was enabled by default, then it would be much harder for writers to develop these issues. But I suspect most of these authors, who are already unwilling to learn make properly would simply give up and use a different tool.
Having to learn all of these subtleties is what gets you into these problems in the first place. I'll happily take a more opinionated build tool with proper escape hatches over something that expects you to know it in detail for even the most basic tasks.
Make's support outside of the posix world isn't that great(as opposed to CMake) so calling it fundamental seems that you are making assumptions about the environments that development happens in.
Correct.
>Make's support outside of the posix world
The world outside of posix doesn't really matter.
Which, coincidentally, is my main complaint with Make: It's way too easy to write shitty Makefiles. There are way too many wrong ways to do things. Arguably this is a community problem and not a tool problem, but it doesn't change the fact that most Makefiles are written badly.
I used Make professionally for 5 years and have since moved to other tech stacks that don't require Make but have tools that are far worse in every other way (cough Ant cough). But IMO the reason they win out is they are far less flexible, so there are fewer ways to do things, so they work better in aggregate.
Build tooling is something that most devs look at very rarely, so they never build expertise. Ergo, build tooling shouldn't require expertise.
If my theory holds out, I feel Gradle will become the new Make in a few years or a decade - very powerful but too many wrong ways to do things.
Why? Because make works. Because it's actually easy to understand if you've read the manual. It's simple. People try to make it do really complicated things, which you are better off putting in a script or another program. At it's core, make works and does its job very, very well.
Here is a very beautiful, readable, portable makefile: https://github.com/git/git/blob/master/Makefile
It's hard to write portable anything without details leaking out. Go is the best implementation I've ever seen here. It works identically everywhere I've tried it. The approach the Go team has taken is extreme. I think that says a lot.
Both these tools are very simple, and limit what you can do. When you try to go outside the box, you hurt yourself. Don't hurt yourself.
Tools like Gradle make it real easy to hurt yourself without realizing it.
> Here is a very beautiful, readable, portable makefile: https://github.com/git/git/blob/master/Makefile
That's a GNU Make Makefile. It's portable to installations where you have GNU Make ported already.
Make is a good tool because it allows a lot of flexibility, but it doesn't protect its users from the pitfalls this flexibility unavoidably creates.
https://news.ycombinator.com/newsguidelines.html
Touch of irony in this project, with that Makefile sitting in the root directory.
EDIT: My apologies if the above comment was cynical. There's a salient need for workflow systems which provide features above/beyond the basics of make. Built-in things like a central broker, adjustable worker processes, scheduling and history. Python/Luigi is an example of such a workflow system, yet it keeps the workflow definitions very similar to Make for conciseness
http://gittup.org/tup/
Also, Tup is much more single-purpose than Make or (from a quick readthrough) Cr. It is very strongly focused on the "N files in, 1 file out" for compilation, and specializes at that to the point of not doing other things.
As far as a minimal "better make" I find redo to be the best. Make is a weird combination of the shell and makefile languages, but redo boils it down to just a single language and manages to handle dependency of build rules in an elegant manner.
It's standardized in POSIX.
"Ideal" is subjective.
OTOH, generating YAML documents is easy. I already write code that generates YAML documents.
P.S.: POSIX make doesn't support concurrency. So, it's not fair to compare this purely to POSIX make.
On the topic of syntax, I'll say YAML is not the best format for configuration either. At a glance, it looks simple and easy to write, but YAML can bite its user in an unexpected ways due to its implicit typing[1]. For example this YAML:
…will be parsed to: …which cannot be serialized to JSON in a lossless way (since JSON spec specify that a key must be a string): …and translating this back to YAML would result in: This is without considering other features such as anchor, aliases and tags[2], which further make the YAML spec incredibly complex.[1]: https://github.com/crdoconnor/strictyaml/blob/master/FAQ.rst...
[2]: http://www.yaml.org/spec/1.2/spec.html
I use Make for a range of things (specifically, not c/c++/etc though) and I've never used Autotools/Automake etc (except compiling a project that uses it occasionally)
If it was the only way of creating Makefiles, it might be a valid complaint against Makefiles, but it isn't. It's not even a useful thing to do.
I disagree with this notion. Code generation is limited by the approach, the input information and the output language. Make is (subjectively) an ugly language, text macros are a bad approach. That's why autotools sucks in my opinion.
OTOH, approaches like Go's 'Stringer' are pretty clever and useful, and generate good code from many PoVs.
>It's not even a useful thing to do.
Look, if you write nice, minimal Makefiles, that's great. It's not what I was talking about. I know autogenerating Makefiles sucks. I prefer Makefiles like those from projects like musl. But some projects are too big and complicated.
If I want to target a huge number of platforms while also supporting cross compiling, Make starts to break down rapidly. Out of tree builds are also a pain.
Tools like qmake and autotools can alleviate this by autogenerating Makefiles and project files abstractly. That way, I don't have to make configuration decisions on every single invocation. This gets especially tedious if you need to run pkg-config many times or sdl-config, or maybe some platform dependent things, or what have you. Or if you want to do things Make isn't good with, like recursing or acting on a wildcard of files. Soon enough your Makefile is calling find and relying on the BSD or GNU version of find, someone tries to run it on MinGW and gets confused by the weird argument error with find.
Lots of people use Make for it's ability to concurrently run tasks with dependencies. This program presents just that part of make with a declarative syntax that is easily generated programmatically.
If you don't see a point, I can understand. Personally, I have no plans to use Cr. But if I was writing a kind of build system that generated a manifest, I'd rather generate a YAML manifest than a Makefile.
https://github.com/abelmokadem/scriptz
I’m running batch jobs of long running single threaded tasks that depend on each other in a somewhat complicated manner.
https://github.com/fungos/cr
It was recently over Twitter.
for similar projects look at:
Bazel https://bazel.build/
Tup http://gittup.org/tup/
Make https://www.gnu.org/software/make/
But again, no IPC, just a simplified "GNU Parallel."
https://www.gabrielcsapo.com/build.sh/
Our major feature over Make was emitting events as newline separated JSON, so that a downstream process can visualize them in a pretty way, track progress, etc.
We use ours run multiple Chef recipes in parallel.