Maybe I'm a dinosaur, but if the product you're building is so complex that it needs such a complex (and Turing complete!) description for how it needs to be built, maybe the problem is that product's architecture. Creating a meta build is fixing the symptom and not the problem.
Yes, it is unfortunate that "make" syntax is a bit odd. But if you can't describe the hierarchical DAG for how your product comes together, that ought to be the problem to tackle rather than building a Turing complete language to try to describe how you think maybe your product might come together.
I want everything declarative, and mostly things are. However I keep running into something weird that the build system designers didn't think of and so I need something to make it work. Note that I'm the cmake expert here, so when I say I keep running into these situations understand that most of the other developers never hit these things, but when it happens they go to me and so I run into them often even though they are rare.
I've designed and shipped more than one multi-million line of code product that had parts in C, C++, Fortran, Python, Tcl, and even some assembly, plus the usual grab bag of open source dependencies. For none of those was anything more sophisticated than (admittedly fairly complex) hierarchical Makefile description to make it come together.
I'll be the first to admit that I've not seen it at all, but color me suspicious that it is an inherent problem of builds of a multi-language product, rather than unknown/unnecessary complexity in the architecture of the system being built.
There is something very fishy in that article: the order of traversal in a hierarchical make should be entirely expressed by dependency rules, not mere existence of a directory. Failure to describe dependencies is a flaw in the creation of the Makefile, not an inherent property of make itself.
Specifically, this idiom:
all:
for dir in $(MODULES); do \
(cd $$dir; ${MAKE} all); \
done
should be banished from everyone's Makefiles. make isn't a just way of cobbling together some collection arbitrary collection of scripts. Each of those modules should be a target with dependencies. The rule to make that module is the recursive make. But the order is determined by the explicit dependencies, not whatever random order glob gives you. ie, what you should have is:
Every single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG.
> Every single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG.
You can look at the dependency graph in two ways. The first is, in order to build X, you first need to build Y and Z. These kinds of dependencies tend to be relatively easy to figure out. The second way is to ask, if Y changes, what needs to be rebuilt--and these kind of rebuild dependencies aren't going to be correct unless they're autogenerated, and getting autogenerated dependencies done correctly is challenging. (This basically reaches the point in the article that any system that doesn't support dynamic dependencies ends up hacking its way to support for dynamic dependencies).
The other key point I will make is that, while it can be possible to describe the DAG you want with make, it tends to fight you every step in the way. If almost everyone is complaining about how hard it is to get the DAG correct in make, then maybe the problem isn't that everyone can't figure out how to fully describe the DAG, it's that make isn't a good option for specifying the DAG.
file modtime-based dependencies is enough to kill any interest in make I might have had. When I update my working copy from version control to an earlier version to look at how things used to work, but the modtime is later, make does the wrong thing.
I suspect that you can describe pretty muich all builds without turing completeness, but avoiding it is rarely particularly helpful. What you need is a language that has a reasonable amount of expressiveness to describe modules and how they fit together, including whatever decisions you need to allow for a) build configurations and b) dealing with platform variations. It's a lot easier to express this in a turing complete language than a non-turing complete language, and it's hard to make a non-turing-complete expressive enough without accidentally making it turing complete.
The reason make is not particularly great for writing a build system in is the language is pretty arcane, and it has annoying restrictions which make certain things particularly difficult to deal with (Spaces in filenames!). The fact that it's not turing complete is not really the key reason in my mind to use something like cmake instead. But, if you were to try to turn make into something that's more flexible and easier to use, you'd be making a lot harder on yourself if you still wanted to avoid turing completeness.
I have made a (internal only unfortunately) meta build system for my work. Instead of being Turing complete is uses Horn Clauses as the primary mode of logic. This means that the problem of determining "is this compiler flag set" or "is this file used in this build" is always solvable in polynomial time.
It has been in use for over ten years by multiple teams around the world for all sorts of whacky toolchaims and projects.
I don't believe anyone has missed Turing completeness.
Despite all that, I agree that if you need a metabuild system then things are probably too complicated. I guess the main reason I would argue for one is when building a cross platform thing, it is nice if developers using toolchain X don't need to modify build files for all the toolchains that aren't X - making Linux developers have to fight Visual Studio just to add a new file to a build is not ideal. But in principle - I still agree that the resulting question of "how do I build this thing from scratch?" Should probably be answerable in 2 sentences (e.g. "Compile and link all the C files with the top level directory in the include path. Don't compile files that have an operating system in their name unless you are building for that os")
Sure, you can avoid it. But I bet if it was turing complete you would not suffer for it either.
And the answer to 'how to build this' should be answerable by "run build_system". Any further details represents a failure of the build system (what you describe of having multiple build systems for each platform/IDE is especially hellish). Even a relatively small amount of code is likely to have details the matter beyond just 'compile and link'. Especially in embedded, where even if the codebase is all the code that's running on the system (no external libraries), it's far from trivial to actually get a working binary just by invoking the compiler.
Actually, Turing completeness would make it worse. This system means that when you pull in things from multiple places and multiple teams, the interactions between the different things would be really trivial to understand. e.g. if you want to know "why is this preprocessor macro being set?" then the tooling can tell you exactly why - even if the answer spans multiple modules from different teams. But the other side effect was that people tended to not do anything crazy or even just different (compare with the whole find_package business in cmake where different packages do things in different ways and frequently have conflicting compile/link requirements)
I am very familiar with embedded stuff, and this whole system was built precisely because of that. Some people are developing in visual studio so give them a good visual studio project that works well, some are working with a proprietary garbage toolchains, so you need to let them write makefiles and have enough flexibility to let them encode how to do that in the build system (e.g. all the different quirks you get for figuring out header file dependencies). The policy I used was "every toolchains needs some lowest common denominator support" and then "once you have that, let every toolchains be as good as possible with equivalences given abstractions". It did appear to work.
I think you may have misinterpreted me on the multiple build system thing. There is one build system and it produces files that each ide/toolchains can work with. So it can make xcode, visual studio, makefile, etc. No one edits those by hand, just like cmake, gyp, etc
No, it is just a finite set of variables. The problem being solved is "given a known set of variables that are true, find the minimal set of other variables that need to be true" and this is basically a flood fill algorithm.
Could you elaborate on the "find the minimal set of other variables that need to be true" part? What other variables are you referring to? "need to be true" for what exactly?
The user of the build system defines boolean variables for any aspect of the build system that they think might be helpful. There are booleans for types of compiler flags (e.g. should I include debug symbols?) and booleans for specific chunks of code (e.g. should I build the unit test code) and booleans for how the thing is being built (e.g. am I building for linux? am I writing a makefile?)
The build system is then able to figure out a starting set of booleans which must be true (e.g. "I am writing the makefiles for the unit test project so it builds on linux in a debug build"). This translates into some starting assignment of boolean values to the boolean variables.
However, it is most likely, that this assignment will not satisfy every horn clause. For example: "if (debug build) then (include debug symbols = TRUE)" might be a clause, and so known that we have a debug build implies we need to set our boolean for including debug symbols to be true (we can break this out so we can then easily have things like optimized builds with debug symbols). Then once you know that you have debug symbols, we might have a clause like "if (debug symbols AND compiling with gcc AND writing a makefile) then (<magic side effect goes here>)"
Similar stuff happens for dependencies. If some code uses the maths library, then the place where the magic side effect says "add this file to the list of files being compiled" will also have a "link with maths library = TRUE" part to it, so then anyone who sets the boolean to get that file, will magically also be told that they need the maths library, and some other part of the system will be responsible for figuring out what that means (e.g. adding "-lm" to some makefiles)
The syntax for it is all simplified down so it doesn't really look like you are doing boolean arithmetic everywhere, but the semantics are mostly what I'm describing.
Horn clauses just mean you can have a whole bunch of things saying "if x then y is true" where X is constrained to be a monotonic (i.e. no negation) Boolean function (i.e. OR and AND).
Given some starting known true values in the system (e.g. debug_build=true, platform_is_windows=true, build_curl_library=true) you solve to find the minimum set of other variables that need to be true.
Each variable can then have some effects tied to it (e.g. if build_curl_library is true, then make sure curl.c is compiled and curl/include is in the include path)
Being Horn Clauses, the solving is basically a flood fill - as soon as you know a variable is true, it is locked in as true. If there was negation, then you can have weird contradictions and reasoning about the system starts being really difficult.
Ideally, a build system should be just a simple, declarative thing: here is the list of the files that need to be compiled, go do it. But the simple declarative tends to fail when you start getting to complex logic.
Things start out with a simple "oh, we need to do completely different things on different OSes. Compile only these files on Linux and these on Windows." Then, oops, we also have a feature that requires a long build time that half our developers don't want to build (also it's unstable and has a nasty dependency or something), so only build this files when this feature is enabled.
Then something gets complex and you now have part of your source code being autogenerated from some tool that needs to be built as part of the build. And the autogenerator needs to use the support library. And someone complains about the speed of the debug build, so the autogenerator needs to be built with optimizations enabled even when the rest of the program is disabled. And then cross-compilation is now more of a headache...
I haven't even touched on so many other possible complexities in the build system (autodownloading dependencies is another fun one). The point is that the idealized, simple, declarative build system only cuts it like 60% of the time (even that requires some level of conditional compilation which make can do but not nicely). At the upper end, you're going to need some complex logic to handle complex cases, and not providing some form of general-complexity features to achieve those cases is going to result in people trying to emulate them using existing features--ironically probably making it more complex as a result.
Giving users the ability to have Turing-completeness in their build system doesn't necessarily mean the buildsystem has to devolve into incomprehensiblity. I've come to like Mozilla's meta-build system in this regard, for example, this relatively complex build file: https://searchfox.org/mozilla-central/source/browser/app/moz...
This right here. When working with large enterprise clients I'm seeing insanely complex build chains.
Artifactories hosting all the built/allowable dependencies/binaries allowed in the customers software. Systems that check the added dependencies and verify any newly added dependencies match the allowed licenses for the application being built. Static checkers of different types. Open source analysis, dll version security analysis, SBOM, and more. Checks to see if enough build system servers are up and running at the time or if the job needs delayed. Seemingly continuous integration has swept this type of customer and the complexity of the build system to allow this has exploded.
Yup. This is the difference between computer science and software engineering. Build systems in the real world have to be resilient in all sorts of crazy scenarios.
Simple declarative models are great until they ... Aren't.
Essentially you have a DSL. A DSL can be great if you use it frequently enough to know the insurance and outs. Like SQL.
But a build DSL is something you muck with like ... Once every six months? And most build files I one off rather than write from scratch. I'm not going to o learn your DSL. I'm going to stack overflow it.
And build systems no matter how genius prescient the authors are, do weird things sometimes. Integrate with so e proprietary library? Have a required code generation step? Pull a dependency from a weird location? Auto annotate code?
Alas then you have to either be able to decompile the DSL to the base lang.in your head, or inject some "real code".
And ... By the standards of this article when dealing with the problem of building code (which, come on, evolves to a workflow engine eventually, and not a simple one) becomes meta.
Build systems do more than just build files. They find and run tests, analyze test results, analyze coverage. Conditionally generate code based on whether a feature is enabled or disable. Run "find dependency" logic. Read pkgconfig files.
Wasn't a large part of the complexity introduced by autoconf from how loosely prescribed UNIX systems were in where things were installed and how they were implemented? Feels that we are much more homogenous now than we have ever been in the past. Such that it is not that surprising to find a lot of complexity in the past is not as necessary today.
I highly recommend going through the make documentation at least once in your career. Per the lindy effect, as it has been around for 40 years, it has a decent chance of sticking around for another 40.
Regarding dynamic dependencies, the author doesn’t seem to be familiar with the build systems used for Java (Maven and Gradle), which in my understanding use static dependencies (static build graph).
Yes, that’s how they work. To be precise, the direct dependencies are listed, and each dependency (after retrieval from a package repository, or locally cached) provides a static list of its own direct dependencies.
The point of dymanic dependencies is that Rig could run some code to figure out what the main function imports, transitively, and use that info to build the entire project.
All without listing dependencies in the build file.
A good API might look like this:
build_java_package("com.yzena.rig");
So when you add a new package and use it, you do not have to change the build files; Rig would pick it up automatically.
In Java, there is no 1:1 relation between package imports and jars. A jar (dependency) usually contains many packages (namespaces), and of course the same packages can be contained in multiple (usually mutually incompatible) dependencies. (Java imports are often more fine-grained than, say, C/C++ header files.) Dependency selection therefore requires a conscious decision.
IDEs provide tooling to suggest matching dependencies to be added to the build file based on the names of unresolved imports. However, the much more common direction is to first decide which libraries (dependencies) you want to use and add those to your build file, which then allows the IDE to suggest imports when you reference some unresolved type in your source code. So the dependency declarations in the build file generally come first, before you start writing your corresponding code. (This can be likened to specifying coarse-grained includes in your project build file, from which fine-grained sub-includes are then selected in each source file.)
The conscious selection of dependencies is a good thing, IMO. Dependencies also don’t change frequently, so there isn’t much work to be saved by determining them automatically (except for transitive dependencies upon version bumps, which are determined automatically by the build system). Moreover, it’s not only the dependencies as such that are consciously selected, but also their respective version numbers. This prevents unexpected breaking changes.
It's clear to me that any build system that's NOT based on a declarative file format and strong project model is a nightmare waiting to happen. Each generation of maintainer will add it's own customizations until it becomes a super specific brittle undocumented jumbled mess that's nigh impossible to work with and that prevents refactoring of project code.
The engineers wanted to write a thin encabulation module in K++, but management insisted that the engineering team use the industry standard Encabulate.moo library, and the Cows on Crack framework doesn't have a stable ABI.
I'm pretty sure that party is over. "Make Money Online with ${x}" is what you pivot to when the return on ${x} drops below the going rate for "Make Money Online" courses.
Though it would be nice to blame this on Management (who are often the root cause of all sorts of screw ups) - managers generally have no idea of, let alone interest, in build systems. We did this to ourselves.
If turing-completeness is necessary, why should there even exist a build system independent from a programming language now?
Isn't the obvious solution to script your language's default build system in precisely the language you're trying to build from?
So when you build Rig, build it in C++ and make C++ your scripting ... Ok, that left a bad taste in my mouth.
Seriously, though. Build systems as the author describes them are going to compete against the project managers that come with languages (Cargo, Go). The only real use case for Rig would be legacy languages that don't come with language-central packaging repositories, i.e., Fortran, C++, C.
While I applaud the author for the attempt to improve this ecosystem, I really hope it will become less relevant some day.
In all seriousness, I think both our projects are aiming in the right direction, mine is just more focused on minimalism at the cost of having extra-pointy corner cases.
I've a similar question to GP, but from a slightly different angle. I can see the attraction of using a Turing complete language, but on the other hand, an important properly of a build system is that it ought to halt at some point. Something like, say, Starlark, which is used in Bazel, is primitive recursive, so it's not Turing complete, but gets you almost all the way to Turing complete while also being guaranteed to halt. It'd be interesting to see this aspect covered.
The thing about Starlark is that it is effectively Turing complete.
You can simulate a while loop with a long enough foreach, an if statement, and a break statement.
And by "long enough," I mean long enough to make anyone just stop the build, but hey, if I stack the foreach loops, I can make the inner loop exponentially long, even though ints are limited to 32 bits.
By not taking out the break keyword, Starlark remains Turing-complete.
That's not Turing complete. That's still primitive recursive. You're going out of your way make it loop for a really, really long time, but it's still guaranteed to terminate, and thus isn't Turing complete, even if it'd take a really long time to terminate. This isn't some theoretical thing: it's how you can introduce guarantees into the system that it can't get out of control by accident and it also makes it easier to reason about the system.
A primitive recursive language gets you most of the benefits of a general recursive one _without_ needing to be _actually_ Turing complete. So this isn't "bad news" for me. I'm saying that I think you should avoid using a general recursive (i.e. Turing complete) language for a build system and instead have guarantees that the system will terminate by using a primitive recursive language of some kind because you literally, as you did, need to go out of your way to get code written in such a fashion to run for an inordinate amount of time. And yet, an inordinate amount of time isn't forever and unlike a program written in an actually Turing complete language, you can use static analysis to detect that kind of thing if only primitive recursion is supported.
> And yet, an inordinate amount of time isn't forever
If it's only guaranteed to terminate after the heat death of the universe, it won't terminate, so I would call that "forever."
> unlike a program written in an actually Turing complete language, you can use static analysis to detect that kind of thing if only primitive recursion is supported.
And this is exactly why I said that break was necessary.
Sure your static analysis may detect that the loop goes 2^256 times, which would be "forever," and so it returns the answer, "Does not halt."
Ah, but with the break keyword, it still could. In fact, it might always halt rather quickly.
Rule of thumb is that if static analysis can be wrong, it is Turing-complete.
> why should there even exist a build system independent from a programming language now?
I want my build system to be independent of the programming language for two reasons:
1) What gavinhoward said: I regularly use several different languages, sometimes in the same project, sometimes not. But in all cases, I want to use the same build system in order to minimize my environmental complexity. Much like I want to use the same IDE regardless of language.
2) I don't want to risk version dependencies. I want to be able to use the same build system even when I'm compiling something using an old version of a compiler, and I don't want to be forced into a build system upgrade just because I want to upgrade one of my compilers.
For me, the separation of the two roles is very important. If they were tied together, it would mean that I'd have to add complexity elsewhere to cover my use cases. Languages that include their own package managers (like Rust, Go, etc.) give me headaches.
> I regularly use several different languages, sometimes in the same project, sometimes not. But in all cases, I want to use the same build system in order to minimize my environmental complexity.
It’s not obvious to me that one can solve the problem of making N languages co-operate by adding another. Why not just use the most powerful language to orchestrate everything?
> I don't want to risk version dependencies. I want to be able to use the same build system even when I'm compiling something using an old version of a compiler, and I don't want to be forced into a build system upgrade just because I want to upgrade one of my compilers.
That begs the question of whether compiler upgrades should force build system upgrades. In a mature programming language, this should never be the case, although practically speaking it’s definitely possible (e.g. ASDF 3.3, released earlier this year but built on a language which has been static for thirty years, adds compatibility for Allegro Common Lisp!).
> Why not just use the most powerful language to orchestrate everything?
I usually do everything in a single language, but sometimes that's not the best solution. It's not a matter of how powerful the language is in a general sense, but how good it is at a particular kind of task.
But even ignoring my projects that involve multiple languages simultaneously, I still regularly use several languages in total, even if only one per project. So having a build system that's independent of the language is still highly desirable.
> That begs the question of whether compiler upgrades should force build system upgrades.
My answer to that is "no, never", of course. That leads to increased complexity for me, in exchange for no gain. The build system should have no special knowledge or expectation of what language or language tools are being used.
> But even ignoring my projects that involve multiple languages simultaneously, I still regularly use several languages in total, even if only one per project. So having a build system that's independent of the language is still highly desirable.
Ah, that’s a difference between us: I prefer to always use one primary language, and all others are secondary languages selected per-project. So it makes sense for me to just use a system built in my primary choice.
> My answer to that is "no, never", of course.
Agreed, too! In principle upgrading the compiler should have no effect whatsoever on correct, portable programs. OTOH, there sure do seem to be an awful lot of incorrect and unportable programs out there!
I was once on a project that inadvertently used ruby in some way for everything.
Rails
Bundler
Vagrant
Chef
Opsworks configuration
JavaScript bundling
Generating PDFs
Probably other things
The system was basically frozen in time because too many cross-dependencies couldn't update in sync to something newer (the opsworks cookbook in chef were the worst offenders iirc).
It was so bad when I started that I couldn't even get the system up and running locally. I ended up tossing hundreds of lines of random ruby code and with a few lines of docker commands had it up and running.
Since then I've preferred keeping things nicely not dependent on each other whenever possible.
> If turing-completeness is necessary, why should there even exist a build system independent from a programming language now?
It shouldn’t. The only issue is that most folks aren’t using powerful enough languages. As you note, using C++ as a scripting language would be pretty crazy. So don’t do that!
>The only real use case for Rig would be legacy languages that don't come with language-central packaging repositories, i.e., Fortran, C++, C.
What about packages for *nix based OS's? dnf, pacman, apt, etc. essential have there own "meta" build systems to create packages for there particular package flavor weather that be rpm, deb, tar.gz etc.
What can be done about libraries on which you want to depend, but that have too many opinions about how they should be built? For example I use bazel and one large library I use forced -Werror into the C compiler invocation, but with my C compiler it has warnings, so it can't actually be built unless I patch out that line of their bazel configs. Another library only builds correctly when using clang if the clang toolchain is named literally "clang" and not, for example, "clang-17". Consequently I have to adopt their convention or maintain a patch against their project. There are many, many unsolved rough edges in bazel along these lines.
Rig will have multi-project builds. It also has a concept called "context stacks".
The idea would be that you would have a "compiler option" context stack, and you would push `-Wno-bad-warning`, then inside the context, build that library.
Then the library would add `-Werror`, but it would also have `-Wno-bad-warning`, so no problem.
But multi-project builds won't exist in the first release.
Bell Labs had a very nice portability framework also -- there was a Cygwin-like product called U/WIN that made extensive use of it -- might find it associated with ksh93 maybe.
Speaking of Bell Labs, there are some great ideas for a build tool in the Bell Labs "nmake" tool - I believe it was originally created to speed up ESS5 switch builds, which took 3 days before make. It has programmable header file scanning for example, a shell coprocess to avoid all the fork/exec of standard make, build state files, lots of other things I have forgotten.
This is missing something: what does my ide do? I want a key combo or menu select for all the things I do. Build and run tests in address sanitizer. Build, deploy to my embedded target then run in the debugger with these options.
I recently wrote Hancho (https://github.com/aappleby/Hancho) that is designed to be fully dynamic; I thiiiink it would fit most of the desired criteria here (aside from cloud caching) but I'm not positive.
I'll have to do a bit more investigating into what this article means by "dynamic dependencies".
> Say you add a new source file. Wouldn’t it be great if your build system just picked it up?
I have never enjoyed the experience of working with tools that did this. How does it know which files I actually want, and which files are there because I (or somebody else) got something wrong? The answer, of course: it doesn't.
I moan a lot about having to do the computer's work for it, but I think in this case I am happy to put up with some minor drudgery (adding a file name to a list, once) in exchange for the computer's boundless ability to then patiently follow my instructions without trying to second-guess me on every single build.
Looking fwd to reading about rig. Consider pushing a standard for builds to the c++ committee. With modules on the table it's criminally bad c++ has no eta to get substantially closer to golang like builds.
That's a good idea too. For my smaller-scale personal projects I rely on the git status, but if you use some other version control system then you don't always get early warning of this sort of thing. (My work stuff typically uses Perforce, and it's annoyingly blasé about this stuff.)
Yeah, that's intellectually right there with: "Say you mention a new variable. Wouldn't it be nice if the language just defined it for you with a blank value?"
> So there are five features of build systems that you want, and the harsh truth is that meta build systems cannot have some of them! Some of these things are only available on end-to-end build systems.
Did I miss something or did the authors actually didn't explain what prevents meta build systems to implement any of those features.
Looking at the current state of cmake it seems to have everything mentioned here. Including dependencies and header parsing in c++
SCons is listed as a meta build system here, but it is clearly an end-to-end build system. It seems to me SCons even fits all the requirements listed by the author. Plus, you program your SCons build using Python.
He also describes the dynamic discovery of header dependencies of Ninja as a hack. It is true that it is a special case to handle a specific type of dynamic dependencies. But the basic idea behind it (using the compiler's diagnostic as the source of truth) is sane and efficient. The alternative is to scan the sources using a custom parser, which is slower and more fragile.
> The alternative is to scan the sources using a custom parser, which is slower and more fragile.
More fragile, yes, but I dispute the "slower" aspect. waf does this very well. In C you don't need to parse the entire program structure, you just need to parse what the preprocessor does.
Also there's a difference between the complete set of header dependencies (notably #include statements within an #if or #ifdef) and a more sane but incomplete methodology for #include statements (don't do it within #if/#ifdef). Not sure whether this is a big enough difference to try to matter.
Do you have an RSS or Atom feed so I can subscribe and be notified when Rig is available.
Anyways, I’m looking forward to more options in this space. The current pattern of rebuilding everything from scratch every time in a non deterministic way is the gross reality of GitHub Actions and Dockerfiles. It would be great to have more people working on a more cacheable build system.
That said, I wonder what Rug can do about IDE integration. Specifically several IDEs (visual studio and Xcode come to mind) need their own project file format to be able to understand how to interpret source code. One approach that Buck did is to generate those files. But it is always a source of friction when trying to get an IDE to integrate smoothly with an additional build system.
> It would be great to have more people working on a more cacheable build system.
Rig is not that yet, unfortunately. I don't want to get your hopes up.
It can't even do incremental builds yet. And while it has multi-threading, I haven't tested it.
This is very alpha software.
> That said, I wonder what Rug can do about IDE integration. Specifically several IDEs (visual studio and Xcode come to mind) need their own project file format to be able to understand how to interpret source code. One approach that Buck did is to generate those files. But it is always a source of friction when trying to get an IDE to integrate smoothly with an additional build system.
Generating those files was my plan, but friction is not good.
Can you describe the friction and its source? I'll do some rethinking and redesign.
> Rig is not that yet, unfortunately. I don't want to get your hopes up.
No worries, this space has so much inefficiency that I'm sure you will eventually start chipping away at it.
> Can you describe the friction and its source? I'll do some rethinking and redesign.
Some software ecosystems already have their own build tools. Often these build systems are intertwined with IDE support. So care must be taken to ensure the IDE still functions as developers expect. If you ignore the functioning of the IDE, developers will balk at adopting the new build tool. There is some prior art for generating files that an IDE can understand:
The article lists some specific limitations of the meta build systems but doesn't really get to the underlying, fundamental issue with the approach. Which leaves the possibility of these specific limitations being addressed somehow and thus refuting the claim that meta build systems are fundamentally broken.
I think the fundamental issue boils down to aggregation. Normally, aggregation is a good thing, a variant of the divide and conquer technique which we were all taught in kindergarten you can never go wrong with. But in build systems aggregation leads to the loss of precision (rebuilding more than necessary) or accuracy (not rebuilding when necessary, rebuilding things in the wrong order), and usually both. This is the fundamental reason why "recursive make is considered harmful": we split one big graph that has the total, accurate view of all the dependencies into a number of aggregate sub-graphs. And the same happens with meta build systems where we do some part of the build (configuration, pre-build steps where we generate source code, etc) in the meta build system and the rest in the underlying build system, essentially splitting the graph into two.
P.S. And, yes, `build2` supports both dynamic prerequisites/dependencies[1] and dynamic targets[2].
93 comments
[ 399 ms ] story [ 1353 ms ] threadYes, it is unfortunate that "make" syntax is a bit odd. But if you can't describe the hierarchical DAG for how your product comes together, that ought to be the problem to tackle rather than building a Turing complete language to try to describe how you think maybe your product might come together.
My go-to example is multi-language builds. Those get complex, and sometimes, multiple languages are necessary.
I'll be the first to admit that I've not seen it at all, but color me suspicious that it is an inherent problem of builds of a multi-language product, rather than unknown/unnecessary complexity in the architecture of the system being built.
That said, if Rig gives you nothing new, please don't even try it!
I'd rather you wait until you need it to try it; then you'll be more forgiving of the bugs. :)
Specifically, this idiom:
should be banished from everyone's Makefiles. make isn't a just way of cobbling together some collection arbitrary collection of scripts. Each of those modules should be a target with dependencies. The rule to make that module is the recursive make. But the order is determined by the explicit dependencies, not whatever random order glob gives you. ie, what you should have is: etc etc etcEvery single time in my career that I've had someone complain about a flaw in make, what I discover is a significant flaw in their description of the make rules. If it is not behaving correctly, you did not fully describe the DAG.
You can look at the dependency graph in two ways. The first is, in order to build X, you first need to build Y and Z. These kinds of dependencies tend to be relatively easy to figure out. The second way is to ask, if Y changes, what needs to be rebuilt--and these kind of rebuild dependencies aren't going to be correct unless they're autogenerated, and getting autogenerated dependencies done correctly is challenging. (This basically reaches the point in the article that any system that doesn't support dynamic dependencies ends up hacking its way to support for dynamic dependencies).
The other key point I will make is that, while it can be possible to describe the DAG you want with make, it tends to fight you every step in the way. If almost everyone is complaining about how hard it is to get the DAG correct in make, then maybe the problem isn't that everyone can't figure out how to fully describe the DAG, it's that make isn't a good option for specifying the DAG.
The reason make is not particularly great for writing a build system in is the language is pretty arcane, and it has annoying restrictions which make certain things particularly difficult to deal with (Spaces in filenames!). The fact that it's not turing complete is not really the key reason in my mind to use something like cmake instead. But, if you were to try to turn make into something that's more flexible and easier to use, you'd be making a lot harder on yourself if you still wanted to avoid turing completeness.
It has been in use for over ten years by multiple teams around the world for all sorts of whacky toolchaims and projects.
I don't believe anyone has missed Turing completeness.
Despite all that, I agree that if you need a metabuild system then things are probably too complicated. I guess the main reason I would argue for one is when building a cross platform thing, it is nice if developers using toolchain X don't need to modify build files for all the toolchains that aren't X - making Linux developers have to fight Visual Studio just to add a new file to a build is not ideal. But in principle - I still agree that the resulting question of "how do I build this thing from scratch?" Should probably be answerable in 2 sentences (e.g. "Compile and link all the C files with the top level directory in the include path. Don't compile files that have an operating system in their name unless you are building for that os")
And the answer to 'how to build this' should be answerable by "run build_system". Any further details represents a failure of the build system (what you describe of having multiple build systems for each platform/IDE is especially hellish). Even a relatively small amount of code is likely to have details the matter beyond just 'compile and link'. Especially in embedded, where even if the codebase is all the code that's running on the system (no external libraries), it's far from trivial to actually get a working binary just by invoking the compiler.
I am very familiar with embedded stuff, and this whole system was built precisely because of that. Some people are developing in visual studio so give them a good visual studio project that works well, some are working with a proprietary garbage toolchains, so you need to let them write makefiles and have enough flexibility to let them encode how to do that in the build system (e.g. all the different quirks you get for figuring out header file dependencies). The policy I used was "every toolchains needs some lowest common denominator support" and then "once you have that, let every toolchains be as good as possible with equivalences given abstractions". It did appear to work.
I think you may have misinterpreted me on the multiple build system thing. There is one build system and it produces files that each ide/toolchains can work with. So it can make xcode, visual studio, makefile, etc. No one edits those by hand, just like cmake, gyp, etc
The build system is then able to figure out a starting set of booleans which must be true (e.g. "I am writing the makefiles for the unit test project so it builds on linux in a debug build"). This translates into some starting assignment of boolean values to the boolean variables.
However, it is most likely, that this assignment will not satisfy every horn clause. For example: "if (debug build) then (include debug symbols = TRUE)" might be a clause, and so known that we have a debug build implies we need to set our boolean for including debug symbols to be true (we can break this out so we can then easily have things like optimized builds with debug symbols). Then once you know that you have debug symbols, we might have a clause like "if (debug symbols AND compiling with gcc AND writing a makefile) then (<magic side effect goes here>)"
Similar stuff happens for dependencies. If some code uses the maths library, then the place where the magic side effect says "add this file to the list of files being compiled" will also have a "link with maths library = TRUE" part to it, so then anyone who sets the boolean to get that file, will magically also be told that they need the maths library, and some other part of the system will be responsible for figuring out what that means (e.g. adding "-lm" to some makefiles)
The syntax for it is all simplified down so it doesn't really look like you are doing boolean arithmetic everywhere, but the semantics are mostly what I'm describing.
Given some starting known true values in the system (e.g. debug_build=true, platform_is_windows=true, build_curl_library=true) you solve to find the minimum set of other variables that need to be true.
Each variable can then have some effects tied to it (e.g. if build_curl_library is true, then make sure curl.c is compiled and curl/include is in the include path)
Being Horn Clauses, the solving is basically a flood fill - as soon as you know a variable is true, it is locked in as true. If there was negation, then you can have weird contradictions and reasoning about the system starts being really difficult.
Things start out with a simple "oh, we need to do completely different things on different OSes. Compile only these files on Linux and these on Windows." Then, oops, we also have a feature that requires a long build time that half our developers don't want to build (also it's unstable and has a nasty dependency or something), so only build this files when this feature is enabled.
Then something gets complex and you now have part of your source code being autogenerated from some tool that needs to be built as part of the build. And the autogenerator needs to use the support library. And someone complains about the speed of the debug build, so the autogenerator needs to be built with optimizations enabled even when the rest of the program is disabled. And then cross-compilation is now more of a headache...
I haven't even touched on so many other possible complexities in the build system (autodownloading dependencies is another fun one). The point is that the idealized, simple, declarative build system only cuts it like 60% of the time (even that requires some level of conditional compilation which make can do but not nicely). At the upper end, you're going to need some complex logic to handle complex cases, and not providing some form of general-complexity features to achieve those cases is going to result in people trying to emulate them using existing features--ironically probably making it more complex as a result.
Giving users the ability to have Turing-completeness in their build system doesn't necessarily mean the buildsystem has to devolve into incomprehensiblity. I've come to like Mozilla's meta-build system in this regard, for example, this relatively complex build file: https://searchfox.org/mozilla-central/source/browser/app/moz...
This right here. When working with large enterprise clients I'm seeing insanely complex build chains.
Artifactories hosting all the built/allowable dependencies/binaries allowed in the customers software. Systems that check the added dependencies and verify any newly added dependencies match the allowed licenses for the application being built. Static checkers of different types. Open source analysis, dll version security analysis, SBOM, and more. Checks to see if enough build system servers are up and running at the time or if the job needs delayed. Seemingly continuous integration has swept this type of customer and the complexity of the build system to allow this has exploded.
Essentially you have a DSL. A DSL can be great if you use it frequently enough to know the insurance and outs. Like SQL.
But a build DSL is something you muck with like ... Once every six months? And most build files I one off rather than write from scratch. I'm not going to o learn your DSL. I'm going to stack overflow it.
And build systems no matter how genius prescient the authors are, do weird things sometimes. Integrate with so e proprietary library? Have a required code generation step? Pull a dependency from a weird location? Auto annotate code?
Alas then you have to either be able to decompile the DSL to the base lang.in your head, or inject some "real code".
And ... By the standards of this article when dealing with the problem of building code (which, come on, evolves to a workflow engine eventually, and not a simple one) becomes meta.
> Alas, it cannot; the list of stuff to build is passed from the meta build system to the build system, usually by fiat.
It can, actually. At least for make. Just use a wildcard in uour rules: https://www.gnu.org/software/make/manual/make.html#Wildcards
I highly recommend going through the make documentation at least once in your career. Per the lindy effect, as it has been around for 40 years, it has a decent chance of sticking around for another 40.
I mentioned suffix rules and pattern rules.
Unfortunately, meta build systems don't like to generate rules like that.
I have worked with them long enough ago I can't remember.
Do they require the build files to list all dependencies?
The point of dymanic dependencies is that Rig could run some code to figure out what the main function imports, transitively, and use that info to build the entire project.
All without listing dependencies in the build file.
A good API might look like this:
So when you add a new package and use it, you do not have to change the build files; Rig would pick it up automatically.IDEs provide tooling to suggest matching dependencies to be added to the build file based on the names of unresolved imports. However, the much more common direction is to first decide which libraries (dependencies) you want to use and add those to your build file, which then allows the IDE to suggest imports when you reference some unresolved type in your source code. So the dependency declarations in the build file generally come first, before you start writing your corresponding code. (This can be likened to specifying coarse-grained includes in your project build file, from which fine-grained sub-includes are then selected in each source file.)
The conscious selection of dependencies is a good thing, IMO. Dependencies also don’t change frequently, so there isn’t much work to be saved by determining them automatically (except for transitive dependencies upon version bumps, which are determined automatically by the build system). Moreover, it’s not only the dependencies as such that are consciously selected, but also their respective version numbers. This prevents unexpected breaking changes.
It's clear to me that any build system that's NOT based on a declarative file format and strong project model is a nightmare waiting to happen. Each generation of maintainer will add it's own customizations until it becomes a super specific brittle undocumented jumbled mess that's nigh impossible to work with and that prevents refactoring of project code.
You'd think that would be hard for even engineers to screw up, but here we are.
Perhap just one layer of meta isn't enough...
<hides under carpet>
If turing-completeness is necessary, why should there even exist a build system independent from a programming language now?
Isn't the obvious solution to script your language's default build system in precisely the language you're trying to build from?
So when you build Rig, build it in C++ and make C++ your scripting ... Ok, that left a bad taste in my mouth.
Seriously, though. Build systems as the author describes them are going to compete against the project managers that come with languages (Cargo, Go). The only real use case for Rig would be legacy languages that don't come with language-central packaging repositories, i.e., Fortran, C++, C.
While I applaud the author for the attempt to improve this ecosystem, I really hope it will become less relevant some day.
This is a great question that I will make sure to answer in my Turing-completeness post in a week.
But to answer now, it's because of multi-project builds, where projects can be in multiple languages.
If you have build systems that are not independent from their language, how do you get them to play nice?
I designed Rig to be able to act as a babysitter and make them play nice.
If I play my cards right, Rig will be the reason other build systems become irrelevant. :)
End users get to pick a familiar language to them.
Similar to how Pulumi is available in multiple languages.
Then code both builds and meta build concerns as you would code anything else.
https://github.com/aappleby/Hancho
In all seriousness, I think both our projects are aiming in the right direction, mine is just more focused on minimalism at the cost of having extra-pointy corner cases.
EDIT: s/Skylark/Starlark/
The thing about Starlark is that it is effectively Turing complete.
You can simulate a while loop with a long enough foreach, an if statement, and a break statement.
And by "long enough," I mean long enough to make anyone just stop the build, but hey, if I stack the foreach loops, I can make the inner loop exponentially long, even though ints are limited to 32 bits.
By not taking out the break keyword, Starlark remains Turing-complete.
And yes, I tested this.
A primitive recursive language gets you most of the benefits of a general recursive one _without_ needing to be _actually_ Turing complete. So this isn't "bad news" for me. I'm saying that I think you should avoid using a general recursive (i.e. Turing complete) language for a build system and instead have guarantees that the system will terminate by using a primitive recursive language of some kind because you literally, as you did, need to go out of your way to get code written in such a fashion to run for an inordinate amount of time. And yet, an inordinate amount of time isn't forever and unlike a program written in an actually Turing complete language, you can use static analysis to detect that kind of thing if only primitive recursion is supported.
If it's only guaranteed to terminate after the heat death of the universe, it won't terminate, so I would call that "forever."
> unlike a program written in an actually Turing complete language, you can use static analysis to detect that kind of thing if only primitive recursion is supported.
And this is exactly why I said that break was necessary.
Sure your static analysis may detect that the loop goes 2^256 times, which would be "forever," and so it returns the answer, "Does not halt."
Ah, but with the break keyword, it still could. In fact, it might always halt rather quickly.
Rule of thumb is that if static analysis can be wrong, it is Turing-complete.
I want my build system to be independent of the programming language for two reasons:
1) What gavinhoward said: I regularly use several different languages, sometimes in the same project, sometimes not. But in all cases, I want to use the same build system in order to minimize my environmental complexity. Much like I want to use the same IDE regardless of language.
2) I don't want to risk version dependencies. I want to be able to use the same build system even when I'm compiling something using an old version of a compiler, and I don't want to be forced into a build system upgrade just because I want to upgrade one of my compilers.
For me, the separation of the two roles is very important. If they were tied together, it would mean that I'd have to add complexity elsewhere to cover my use cases. Languages that include their own package managers (like Rust, Go, etc.) give me headaches.
It’s not obvious to me that one can solve the problem of making N languages co-operate by adding another. Why not just use the most powerful language to orchestrate everything?
> I don't want to risk version dependencies. I want to be able to use the same build system even when I'm compiling something using an old version of a compiler, and I don't want to be forced into a build system upgrade just because I want to upgrade one of my compilers.
That begs the question of whether compiler upgrades should force build system upgrades. In a mature programming language, this should never be the case, although practically speaking it’s definitely possible (e.g. ASDF 3.3, released earlier this year but built on a language which has been static for thirty years, adds compatibility for Allegro Common Lisp!).
Incidentally, ASDF is pretty cool: https://asdf.common-lisp.dev/
I usually do everything in a single language, but sometimes that's not the best solution. It's not a matter of how powerful the language is in a general sense, but how good it is at a particular kind of task.
But even ignoring my projects that involve multiple languages simultaneously, I still regularly use several languages in total, even if only one per project. So having a build system that's independent of the language is still highly desirable.
> That begs the question of whether compiler upgrades should force build system upgrades.
My answer to that is "no, never", of course. That leads to increased complexity for me, in exchange for no gain. The build system should have no special knowledge or expectation of what language or language tools are being used.
Ah, that’s a difference between us: I prefer to always use one primary language, and all others are secondary languages selected per-project. So it makes sense for me to just use a system built in my primary choice.
> My answer to that is "no, never", of course.
Agreed, too! In principle upgrading the compiler should have no effect whatsoever on correct, portable programs. OTOH, there sure do seem to be an awful lot of incorrect and unportable programs out there!
It was so bad when I started that I couldn't even get the system up and running locally. I ended up tossing hundreds of lines of random ruby code and with a few lines of docker commands had it up and running.
Since then I've preferred keeping things nicely not dependent on each other whenever possible.
It shouldn’t. The only issue is that most folks aren’t using powerful enough languages. As you note, using C++ as a scripting language would be pretty crazy. So don’t do that!
Please no. I don't want to handle filename and other string manipulation in C.
What about packages for *nix based OS's? dnf, pacman, apt, etc. essential have there own "meta" build systems to create packages for there particular package flavor weather that be rpm, deb, tar.gz etc.
C/C++ are big time numerous and serious real. And they need all the help they can get to do builds more like golang
Rig will have multi-project builds. It also has a concept called "context stacks".
The idea would be that you would have a "compiler option" context stack, and you would push `-Wno-bad-warning`, then inside the context, build that library.
Then the library would add `-Werror`, but it would also have `-Wno-bad-warning`, so no problem.
But multi-project builds won't exist in the first release.
Speaking of Bell Labs, there are some great ideas for a build tool in the Bell Labs "nmake" tool - I believe it was originally created to speed up ESS5 switch builds, which took 3 days before make. It has programmable header file scanning for example, a shell coprocess to avoid all the fork/exec of standard make, build state files, lots of other things I have forgotten.
Every IDE is different, unfortunately.
Rig will have to adapt to each IDE, but it will generate files with enough info to call into Rig, which will do the job.
https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dyn...
I'll have to do a bit more investigating into what this article means by "dynamic dependencies".
I have never enjoyed the experience of working with tools that did this. How does it know which files I actually want, and which files are there because I (or somebody else) got something wrong? The answer, of course: it doesn't.
I moan a lot about having to do the computer's work for it, but I think in this case I am happy to put up with some minor drudgery (adding a file name to a list, once) in exchange for the computer's boundless ability to then patiently follow my instructions without trying to second-guess me on every single build.
Rig will do this too. You can statically list dependencies.
Yes, I want my build fast. However, give me a way to kick off a scan to look for extraneous stuff.
why would you put a file "foo.c" into directory "src/" if you don't want to have it compiled? put it somewhere else then.
Did I miss something or did the authors actually didn't explain what prevents meta build systems to implement any of those features.
Looking at the current state of cmake it seems to have everything mentioned here. Including dependencies and header parsing in c++
He also describes the dynamic discovery of header dependencies of Ninja as a hack. It is true that it is a special case to handle a specific type of dynamic dependencies. But the basic idea behind it (using the compiler's diagnostic as the source of truth) is sane and efficient. The alternative is to scan the sources using a custom parser, which is slower and more fragile.
More fragile, yes, but I dispute the "slower" aspect. waf does this very well. In C you don't need to parse the entire program structure, you just need to parse what the preprocessor does.
Also there's a difference between the complete set of header dependencies (notably #include statements within an #if or #ifdef) and a more sane but incomplete methodology for #include statements (don't do it within #if/#ifdef). Not sure whether this is a big enough difference to try to matter.
Anyways, I’m looking forward to more options in this space. The current pattern of rebuilding everything from scratch every time in a non deterministic way is the gross reality of GitHub Actions and Dockerfiles. It would be great to have more people working on a more cacheable build system.
That said, I wonder what Rug can do about IDE integration. Specifically several IDEs (visual studio and Xcode come to mind) need their own project file format to be able to understand how to interpret source code. One approach that Buck did is to generate those files. But it is always a source of friction when trying to get an IDE to integrate smoothly with an additional build system.
But I can email you directly if you wish.
> It would be great to have more people working on a more cacheable build system.
Rig is not that yet, unfortunately. I don't want to get your hopes up.
It can't even do incremental builds yet. And while it has multi-threading, I haven't tested it.
This is very alpha software.
> That said, I wonder what Rug can do about IDE integration. Specifically several IDEs (visual studio and Xcode come to mind) need their own project file format to be able to understand how to interpret source code. One approach that Buck did is to generate those files. But it is always a source of friction when trying to get an IDE to integrate smoothly with an additional build system.
Generating those files was my plan, but friction is not good.
Can you describe the friction and its source? I'll do some rethinking and redesign.
Thank you for your interest!
No worries, this space has so much inefficiency that I'm sure you will eventually start chipping away at it.
> Can you describe the friction and its source? I'll do some rethinking and redesign.
Some software ecosystems already have their own build tools. Often these build systems are intertwined with IDE support. So care must be taken to ensure the IDE still functions as developers expect. If you ignore the functioning of the IDE, developers will balk at adopting the new build tool. There is some prior art for generating files that an IDE can understand:
https://buck.build/command/project.html
https://github.com/bazelbuild/tulsi
Interfacing existing build tools into a meta build tool can be difficult to do efficiently, but I’ve run out of time for talking about that tonight.
I think the fundamental issue boils down to aggregation. Normally, aggregation is a good thing, a variant of the divide and conquer technique which we were all taught in kindergarten you can never go wrong with. But in build systems aggregation leads to the loss of precision (rebuilding more than necessary) or accuracy (not rebuilding when necessary, rebuilding things in the wrong order), and usually both. This is the fundamental reason why "recursive make is considered harmful": we split one big graph that has the total, accurate view of all the dependencies into a number of aggregate sub-graphs. And the same happens with meta build systems where we do some part of the build (configuration, pre-build steps where we generate source code, etc) in the meta build system and the rest in the underlying build system, essentially splitting the graph into two.
P.S. And, yes, `build2` supports both dynamic prerequisites/dependencies[1] and dynamic targets[2].
[1] https://build2.org/release/0.15.0.xhtml#dyndep
[2] https://build2.org/release/0.16.0.xhtml#dyn-target