I'm all for good build systems and I have used make quite a bit but the problem in moving to a better build system is that makefiles are so convoluted and hard to reason about that nobody wants to be blamed for breaking the build by migrating.
I wish there were a testing system for builds. Specify updating what should change what and check timestamps (to begin with) of created files.
That's just a start. Tup is much less flexible and forces certain conventions on the build system.
It is also much more rigid about introducing dependencies during the build.
> How many people are actually editing makefiles by hand in non-pet projects?
people that don't want makefiles so convoluted and hard to reason about that they can't edit them by hand..
i suggest taking a look at any of the BSD build systems and seeing what sane use of make can look like (PMake and not GNUMake; imho PMake's language makes this doable; GNUMake's language makes autotools and other mess-generating 'helpers' required)
> people that don't want makefiles so convoluted and hard to reason about that they can't edit them by hand..
You've missed the point. The point was that nowadays editing or even looking at a makefile is far from standard practice, because makefiles are autogenerated by the build system and are closer to temp files than to project files.
Who wastes their time reasoning about a temp file that just works?
I wrote a build system for an embedded system with two processors and lots of code sharing across three platforms in Make because no other tool would let me program the whole DAG in the same language. The makefiles generated themselves.
Anyone aware of an npm library that wraps tup to provide automatic dependency graph construction from objects changing on filesystem?
I'd could use a javascript api that enabled automatically detecting filesystem change dependency graph associated with each of a given set of javascript function calls ... Perhaps with simplifying assumption that all function parameters are serializable or maybe even that the functions being tracked are only allowed to operate on strings that represents filesystem path's ...
I'd love for the functionality implemented within tup of running a command and automatically doing the low-level kernel hacking necessary to track all the filesystem objects read/written to be abstracted into an application-level library ...
What I've seen of Tup is good but it doesn't add enough to convince me to switch away from make which is already everywhere.
CMake and similar tools don't attract me because they're not language independent. The best thing about make (and Tup too) is that it lets you express dependencies and ways to satisfy them in terms of other tools.
Now what make is not good at is being ./configure. I'd like to see a more elegant autoconf-like tool to detect things about the environment and generate configure.{h,mk}.
CMake actually is language independent, but it so happens that only C, C++, Assembly, Fortran are first class citizens. Java and Ada are second class. Anything else, nobody wrote it. You might find a package to find the compiler and dependencies perhaps.
I’ve used tup in a previous project. I was expecting issues that I couldn’t predict before getting my hands dirty, but it was actually a very smooth transition.
For complex C and C++ projects, I can recommend tup. I’ve also used it for Latex documents.
premake¹ feels like an elegant autoconf alternative to me, and can be combined with various build systems including make. The syntax is far nicer than cmake too, but frankly that bar isn’t very high.
Sadly, it isn’t used by all that many projects² which holds me back from wanting to depend on it. I have played with it in a few personal projects though, and have been impressed.
I found the Build Systems a la Carte¹ paper, and NDM’s companion write up² to be good comparison of build systems in a more general manner.
Caveat: It may skewed having been written by heavy Haskell hitters, and an author of Shake. I’ll note that I didn’t notice any bias, but maybe only because it aligned with mine ;)
This reminds me on 'Make vs. X' trend some years ago, where Make would be bashed how is slow, not extensible, had weird syntax and incompatible between implementations. So we got alternatives like Aegis[1], Scons[2], A-A-P[3] or Jam[4] instead, which were much faster and flashier on paper.
But guess what, Make is still kicking, GNU Make got a bunch of new goodies (Guile scripting or loadable modules to name a few) and all those alternatives are pretty much dead now. What I see now is that we are repeating the same mistakes only with flashier tools (node, python3, rust).
Although I always preferred Jam[4], I'm pretty happy with GNU Make now. Not perfect, does the job well and if I ever hit some weird platforms, I can always 'extend' myself to Autotools[5]. Funny thing, I'm even using Make to run Ansible scripts or compile Java/Clojure code and works like a charm.
I suspect a large amount of the issue is sheer availability.
There's an odd proclivity of developers of build tools to assume that their favorite runtime (be it python for scons, etc) is already available or trivial to nicely install. And that's just... not true.
The build tool itself is the one place in the ecosystem of building software that we have the least room for manual dependency handling and manual setup, because there's nowhere left to punt the hard part.
Make isn't better at this either, but it's been so widely packaged and is so nearly-default available (it's remarkably hard to get a working desktop system without some transitive dependency having gotten make installed for you) that it defacto gets a free pass on this.
My hypothesis is that these newer build tools would have a much better shot at reaching adoption if they were well-packaged enough that a single tarball (with zero external deps) or a single-line install script (think: "gradlew", though even that was cheating by assuming an existing jvm) could bootstrap them. Most don't seem to have invested that effort. And it shows.
Tarbals are a poor packaging method as they require users to install applications manually without ensuring dependencies or pre/post installation configuration.
However, taking the time to put together a working installation package for specific platforms is something that does wonders to the app's adoption. It's all about providing a standard distribution method and standard installation in a platform so that it becomes a fixed target for end-users and developers.
> require users to install applications manually without ensuring dependencies or pre/post installation configuration.
Yet my point is the converse of this: by not supporting those things, it means for something to be well packaged as a tarball, it may not have any external dependencies nor may it have pre/post installation hooks. And that's a Good Thing.
Because why would you need any of these things?
And how can you survive that level of complexity when the thing in question is itself the build tool which is traditionally the thing that we use to handle these complexities?
This may just be "my opinion", but I think my build tool should have packages available which have no further recursive dependencies -- the buck has to stop somewhere -- and it certainly shouldn't need post-install hooks for absolutely any purpose -- what would it use them for? Doing something stateful? That's antithetical to the core of what a good build tool should be doing.
I suspect you're right. And make (or whatever your platform default) is Good Enough™ for most people.
The heavy hitters (Microsoft, Google, ...) have code bases that are orders of magnitude larger than what most others have so their needs may just be different.
Anyway, one approach I find intriguing in terms of "availability" is what Jai uses: build "scripts" are just Jai programs evaluated at compile time. So if you are building a Jai program, it'll obviously be available.
A related aspect that somewhat still bugs me is that build systems really just seem to be special-purpose instances of one-way dataflow constraints[1], a feature that is tremendously useful not just for build systems, but also for UI/model interaction (MVC, one-way-dataflow, bindings), spreadsheets, etc.
Combining the two: shouldn't we have a language that supports dataflow constraints, and then use a part of the program that builds the rest of the program?
> The heavy hitters (Microsoft, Google, ...) have code bases that are orders of magnitude larger than what most others have so their needs may just be different.
and yet I can mostly-deterministically build any of the main BSD kernels + base systems + UI + all applications and sometimes even package them into release media using 1-3 PMake invocations.. (without needing to embed a QT-based configuration GUI to do it).. don't think msft or goog need to build anything more complicated than that.. and i'd wager if they do some other process isn't working correctly..
(side note PMake is much better than GNUMake imho; syntax is more 'shelly' and there are build in looping constraints abilty to dynamically define targets based on variable contents (e.g. define a target 'T' who's target name will be the contents of the variable 'V'), unlike GNUmake's bizarre quasi-lispy syntax and functions)
> one-way dataflow constraints
definately not up on the theory; this sounds very similar to make target heirarchies (e.g target X is constrained by presence/absence of artifact Y)
I'm interpreting your comment as implying that it's somehow a waste of time to develop alternatives to an incumbent that just won't evolve. I'm not sure if that's the point you were trying to make or not, but I think it's important to call out that perhaps Make was improved precisely because it was challenged? And whether or not this was the case for Make, it has certainly been the case for other technologies.
> I'm interpreting your comment as implying that it's somehow a waste of time to develop alternatives to an incumbent that just won't evolve.
That assertion has no objective meaning, or even a point. Change for the sake of change does more to create problems instead of fixing them. Is there actually anything that is broken in Make's specification?
I’m not sure why you think anyone is advocating for change for its own sake. Is there something about the thread context that has confused you? The quoted statement seems perfectly valid to me.
> I'm not sure why you think anyone is advocating for change for its own sake.
Your only remotely objective criticism of Make is that "it just won't evolve". That's not a reason at all, and only advocates change for the sake of change.
Perhaps you could start to have a case if you could identify any aspect of Make that has problems that need fixing. Until then, any argument based on vage notions of evolving is pointless.
> Your only remotely objective criticism of Make is that "it just won't evolve".
I'm not the critic. You misunderstood the thread. Another user observed that lots of people have made other tools because they found make wanting and that make evolved to add features (presumably borrowing good ideas from its challengers). If you must be defensive, take it up with the authors of the aforementioned make-rivals.
I totally agree with you. Despite not having much experience with the tool, I recently used Make and M4 in a project to build Docker images for multiple architectures [1] and it worked really well.
Yes, agreed. I went through SCons, Jam, CMake, and a few others. I’ve made projects fully with autotools and manually maintained IDE projects. Make compares well against most of these other tools, especially if you are using GNU Make.
However, I don’t use Make as much these days and tend to use one of two different replacements… either a Ninja build file generated by a Python script, or Bazel. Ninja solves the problems I have writing makefiles (rebuild when args change, rules with multiple outputs, create output directories) and gets out of the way. Bazel has a much higher level of complexity but for complex, multi-language projects I can’t see not using Bazel as a viable option these days.
My two favorite make alternatives make HN in two consecutive days.
I like redo because with make you need to know two languages, with redo you know just one, the interface is simple and you get automatic dependency on build rules for free. Also the implementation is so much simpler than make, and yet I've not run into a feature I miss from make.
I like tup because it is very opinionated and forces you to write your build scripts in a reasonable manner, while also being very fast and very correct. It might be possible to write a tup file that incorrectly handles dependencies, but you'd have to work hard to do so.
That was my take as well, and I honestly don't get the downvotes.
More importantly, in large projects the bulk of the comptational budget is spent actually compiling source files. It's hard to believe that picking which file neess to be compiled next takes more time than actually compiling it.
34 comments
[ 0.31 ms ] story [ 155 ms ] threadI'm all for good build systems and I have used make quite a bit but the problem in moving to a better build system is that makefiles are so convoluted and hard to reason about that nobody wants to be blamed for breaking the build by migrating.
I wish there were a testing system for builds. Specify updating what should change what and check timestamps (to begin with) of created files.
Nowadays Makefiles are largely autogenerated by the build system. How many people are actually editing makefiles by hand in non-pet projects?
people that don't want makefiles so convoluted and hard to reason about that they can't edit them by hand..
i suggest taking a look at any of the BSD build systems and seeing what sane use of make can look like (PMake and not GNUMake; imho PMake's language makes this doable; GNUMake's language makes autotools and other mess-generating 'helpers' required)
You've missed the point. The point was that nowadays editing or even looking at a makefile is far from standard practice, because makefiles are autogenerated by the build system and are closer to temp files than to project files.
Who wastes their time reasoning about a temp file that just works?
I'd could use a javascript api that enabled automatically detecting filesystem change dependency graph associated with each of a given set of javascript function calls ... Perhaps with simplifying assumption that all function parameters are serializable or maybe even that the functions being tracked are only allowed to operate on strings that represents filesystem path's ...
I'd love for the functionality implemented within tup of running a command and automatically doing the low-level kernel hacking necessary to track all the filesystem objects read/written to be abstracted into an application-level library ...
CMake and similar tools don't attract me because they're not language independent. The best thing about make (and Tup too) is that it lets you express dependencies and ways to satisfy them in terms of other tools.
Now what make is not good at is being ./configure. I'd like to see a more elegant autoconf-like tool to detect things about the environment and generate configure.{h,mk}.
It is very much possible to add other languages.
For complex C and C++ projects, I can recommend tup. I’ve also used it for Latex documents.
Sadly, it isn’t used by all that many projects² which holds me back from wanting to depend on it. I have played with it in a few personal projects though, and have been impressed.
1. https://premake.github.io/ 2. https://github.com/premake/premake-core/wiki/Who-Uses-Premak...
Caveat: It may skewed having been written by heavy Haskell hitters, and an author of Shake. I’ll note that I didn’t notice any bias, but maybe only because it aligned with mine ;)
1. https://www.microsoft.com/en-us/research/publication/build-s... 2. http://neilmitchell.blogspot.com/2018/07/inside-paper-build-...
But guess what, Make is still kicking, GNU Make got a bunch of new goodies (Guile scripting or loadable modules to name a few) and all those alternatives are pretty much dead now. What I see now is that we are repeating the same mistakes only with flashier tools (node, python3, rust).
Although I always preferred Jam[4], I'm pretty happy with GNU Make now. Not perfect, does the job well and if I ever hit some weird platforms, I can always 'extend' myself to Autotools[5]. Funny thing, I'm even using Make to run Ansible scripts or compile Java/Clojure code and works like a charm.
[1] http://aegis.sourceforge.net/
[2] https://scons.org/
[3] http://www.a-a-p.org/
[4] https://en.wikipedia.org/wiki/Perforce_Jam
[5] https://en.wikipedia.org/wiki/GNU_Build_System
There's an odd proclivity of developers of build tools to assume that their favorite runtime (be it python for scons, etc) is already available or trivial to nicely install. And that's just... not true.
The build tool itself is the one place in the ecosystem of building software that we have the least room for manual dependency handling and manual setup, because there's nowhere left to punt the hard part.
Make isn't better at this either, but it's been so widely packaged and is so nearly-default available (it's remarkably hard to get a working desktop system without some transitive dependency having gotten make installed for you) that it defacto gets a free pass on this.
My hypothesis is that these newer build tools would have a much better shot at reaching adoption if they were well-packaged enough that a single tarball (with zero external deps) or a single-line install script (think: "gradlew", though even that was cheating by assuming an existing jvm) could bootstrap them. Most don't seem to have invested that effort. And it shows.
However, taking the time to put together a working installation package for specific platforms is something that does wonders to the app's adoption. It's all about providing a standard distribution method and standard installation in a platform so that it becomes a fixed target for end-users and developers.
Yet my point is the converse of this: by not supporting those things, it means for something to be well packaged as a tarball, it may not have any external dependencies nor may it have pre/post installation hooks. And that's a Good Thing.
Because why would you need any of these things?
And how can you survive that level of complexity when the thing in question is itself the build tool which is traditionally the thing that we use to handle these complexities?
This may just be "my opinion", but I think my build tool should have packages available which have no further recursive dependencies -- the buck has to stop somewhere -- and it certainly shouldn't need post-install hooks for absolutely any purpose -- what would it use them for? Doing something stateful? That's antithetical to the core of what a good build tool should be doing.
I suspect you're right. And make (or whatever your platform default) is Good Enough™ for most people.
The heavy hitters (Microsoft, Google, ...) have code bases that are orders of magnitude larger than what most others have so their needs may just be different.
Anyway, one approach I find intriguing in terms of "availability" is what Jai uses: build "scripts" are just Jai programs evaluated at compile time. So if you are building a Jai program, it'll obviously be available.
A related aspect that somewhat still bugs me is that build systems really just seem to be special-purpose instances of one-way dataflow constraints[1], a feature that is tremendously useful not just for build systems, but also for UI/model interaction (MVC, one-way-dataflow, bindings), spreadsheets, etc.
Combining the two: shouldn't we have a language that supports dataflow constraints, and then use a part of the program that builds the rest of the program?
[1] https://blog.metaobject.com/2014/03/the-siren-call-of-kvo-an...
and yet I can mostly-deterministically build any of the main BSD kernels + base systems + UI + all applications and sometimes even package them into release media using 1-3 PMake invocations.. (without needing to embed a QT-based configuration GUI to do it).. don't think msft or goog need to build anything more complicated than that.. and i'd wager if they do some other process isn't working correctly..
(side note PMake is much better than GNUMake imho; syntax is more 'shelly' and there are build in looping constraints abilty to dynamically define targets based on variable contents (e.g. define a target 'T' who's target name will be the contents of the variable 'V'), unlike GNUmake's bizarre quasi-lispy syntax and functions)
> one-way dataflow constraints
definately not up on the theory; this sounds very similar to make target heirarchies (e.g target X is constrained by presence/absence of artifact Y)
That assertion has no objective meaning, or even a point. Change for the sake of change does more to create problems instead of fixing them. Is there actually anything that is broken in Make's specification?
Your only remotely objective criticism of Make is that "it just won't evolve". That's not a reason at all, and only advocates change for the sake of change.
Perhaps you could start to have a case if you could identify any aspect of Make that has problems that need fixing. Until then, any argument based on vage notions of evolving is pointless.
I'm not the critic. You misunderstood the thread. Another user observed that lots of people have made other tools because they found make wanting and that make evolved to add features (presumably borrowing good ideas from its challengers). If you must be defensive, take it up with the authors of the aforementioned make-rivals.
[1] https://github.com/hectorm/hblock-resolver/blob/master/Makef...
However, I don’t use Make as much these days and tend to use one of two different replacements… either a Ninja build file generated by a Python script, or Bazel. Ninja solves the problems I have writing makefiles (rebuild when args change, rules with multiple outputs, create output directories) and gets out of the way. Bazel has a much higher level of complexity but for complex, multi-language projects I can’t see not using Bazel as a viable option these days.
I like redo because with make you need to know two languages, with redo you know just one, the interface is simple and you get automatic dependency on build rules for free. Also the implementation is so much simpler than make, and yet I've not run into a feature I miss from make.
I like tup because it is very opinionated and forces you to write your build scripts in a reasonable manner, while also being very fast and very correct. It might be possible to write a tup file that incorrectly handles dependencies, but you'd have to work hard to do so.
More importantly, in large projects the bulk of the comptational budget is spent actually compiling source files. It's hard to believe that picking which file neess to be compiled next takes more time than actually compiling it.