Agreed. What cmake does under the hood is actually very sensible and useful, but the arcane scripting language makes even simple things hard. Adding a "proper" alternative scripting language would be a good reason to bump the version number to 4.0
CMake as a library with bindings for various scripting languages (e.g. Python) would make a lot of sense to me. Have the power of a proper scripting language and let the CMake library deal with generating the build system.
I once looked into extracting the CMake build system generation code and make it into a standalone library. But I soon realized that it’s not exactly a weekend project.
Again, the way the language works is baked deep into the project. The stringly-typed nature is everywhere. Using Python isn't going to magically fix that list elements that contain `;` "don't work right". Policy scoping is weird and is not lexical (e.g., policies "stick" to function definitions, so calling a function can cross a policy boundary).
It's one of those "ah, that sounds nice" things that is just too much work once you get down to it (and we have enough compatibility to handle without dealing with the vagaries of umpteen languages as it is).
> CMake as a library with bindings for various scripting languages (e.g. Python) would make a lot of sense to me.
Well, if your thing is fragmentation and bit rot and maintenance hell, sure.
There were plenty of competing build systems for C++ that provided Python bindings, such as SCons. For some reason, CMake with its awful BASIC-like DSL became the de facto standard.
Couldn't they still do this? I think there was a project a ways back to add a lua frontend to CMake. It's been since abandoned, but I don't see why it couldn't be revitalized. I guess the project just doesn't see the value in doing it, or they don't have enough manpower.
See my sibling comment. The way CMake's language has evolved is very organic. There is a declarative core now with targets and usage requirements, but due to backwards compatibility guarantees, actually making it the only thing available is nigh impossible.
Lua was also deemed not suitable in the long run because of our backwards compatibility guarantees. We'd need to ship N Lua interpreters internally because Lua is not compatible between 5.1, 5.2, 5.3, etc. Or say "we only support Lua 5.2, we cannot support 5.4" which…distros really dislike.
10 lua interpreters would still be less than 1% of the cmake code base. Shipping all of them would be orders of magnitude better than the current clusterfuck that cmake is. But in reality you would only need one.
Except that our testing and behavior matrix now explodes. It's not the code I'd be worried about maintaining: it's the effects and interactions of that code.
It has already much exploded, fractally several times around. My main portability problem when building packages is the very existence different cmake versions, and hardcoded cmake version requirements in cmakelists.
I think parent is referring to the fact that for many Linux distributions, you must build packages with the version of CMake they have in the repositories.
Even today, in 2021, I have to use CMake 2.8.12 / GCC 4.8.5 in order to build code that will deploy on a CentOS 7 server. That means I have to backport the CMake file of every dependency that declares a minimum_requirement of 3+.
If you want to be in that distro repo, yeah. If you're just building on CentOS 7, download newer binaries (https://github.com/Kitware/CMake/releases) and use them instead. I do this all the time for our CentOS 7 deployments (and we use gcc 8 or 9; I forget when we last uplifted the devtoolset).
FWIW, `cmake3` is packaged in CentOS 7 (though it might be EPEL?).
All that said, given CMake's backwards compatibility guarantees, it is a bit silly that distributions don't update CMake more aggressively.
> it is a bit silly that distributions don't update CMake more aggressively.
this would actually worsen the problem.
Main issue is the message "Compatibility with CMake < 2.8.12 will be removed from a future version of CMake" that appears so often when compiling older codes (not that old, written three or four years ago). When you change the version requirement on the offending CMakeLists.txt, the compilation breaks subtly and then you have to spend a very romantic afternoon with the build system and its beautiful language.
Backwards compatibility. You keep using that word. I do not think it means what you think it means.
"Future version" will have lots of fanfare before that actually happens. There's no timeline for it; it's a warning that, eventually, we will have to drop old behaviors because otherwise we're stuck with terrible behaviors forever. I mean look at the kinds of things we still support (https://cmake.org/cmake/help/latest/manual/cmake-policies.7....):
- CMP0010 is there to allow projects which did `${var` to "work"; it is an error today
- CMP0038 bans `target_link_libraries(tgt tgt)`
- CMP0053 gives a *much* faster variable expansion engine at the expense of no longer expanding `@var@` in regular CMake code
And it's only with projects using pre-2.8.12 behaviors of policies that actually care about that. If the project is policy-warning clean (except for the policies we cannot realistically warn on, but these are obscure in practice), then you're already compatible. For the record, the old behaviors we cannot warn about because detecting reliance on the old behavior is nigh impossible in practice:
- CMP0025: Xcode's `clang` now reports its compiler id as `AppleClang` (due to different versioning schemes); we cannot track if this behavior is used
- CMP0047: Same thing for `qcc` on QNX to use the `QCC` compiler id
- CMP0056: `try_compile` uses link flags
- CMP0060: obscure Unix platform implicit link directory order shenanigans (HP-UX at least)
- CMP0061: Remove the implicit `-i` flag to `make` when doing `cmake -build`
- CMP0065: executables now need to explicitly export symbols if wanted (was implicit)
- CMP0066: `try_compile` now supports per-config flags (instead of only `CMAKE_<LANG>_FLAGS`)
- CMP0067: `try_compile` forwards language standard flags through
- CMP0073: removal of `_LIB_DEPENDS` cache variables (an implementation detail that got into the cache due to historical decisions)
- CMP0082: install rules now happen in declaration order (used to be depth-first order)
- CMP0088: `FindBISON`'s CMake API now uses `${CMAKE_CURRENT_BINARY_DIR}` instead of the source directory
- CMP0089: compiler id change for IBM's clang-based xlc compilers: `XLClang`
- CMP0090: `export(PACKAGE)` default behavior change
- CMP0091: MSVC runtime flags are now an abstraction (instead of baked into `CMAKE_<LANG>_FLAGS_<CONFIG>`)
- CMP0092: MSVC warning flags are removed from `CMAKE_<LANG>_FLAGS` (to avoid having to `string(REPLACE)` if one wanted to change them)
- CMP0093: `FindBoost` version reporting format
- CMP0094: `FindPython*` search behavior changes (used to prefer the newest Python over a specific one)
- CMP0096: Version numbers with leading zeros are preserved (instead of `21.05` being stripped to `21.5`)
- CMP0097: `ExternalProject` handling of git submodules
- CMP0098: `FindFLEX` change to mirror `FindBISON` above
- CMP0099: link usage interfaces (options, directories, depends) are pushed through linking to static libraries privately (just like linking was)
- CMP0101: `target_compile_options(BEFORE)` works on all visibilities (instead of ignoring it in `PRIVATE`)
- CMP0102: `mark_as_advanced` no longer wrecks havoc with local variables
- CMP0105: link options now apply to the device linking step (CUDA)
- CMP0107: `ALIAS` target cannot shadow existing targets
- CMP0108: rejects `target_link_libraries(tgt alias_of_tgt)`
- CMP0112: relaxes target dependencies when generator expressions are used
- CMP0113: internal Makefiles generator fixes to avoid running custom commands multiple times
- CMP0116: Ninja generator transformation of `add_custom_command(DEPFILE)` contents to match CMake's usage of paths in the `build.ninja` file
- CMP0117: MSVC doesn't get the `/GR` flag from `CMAKE_CXX_FLAGS` by default
- CMP0118: any directory ca...
Why are you inflicting this to yourself ? Centos 7 has GCC 9 and cmake 3.16 (last I checked) in its repos. Personally I build my own software against cmake 3.20 and clang-12 and they work fine on centos 7.
> You should always be able to update CMake and have it continue to work.
CMake hard-codes the absolute path to itself all over the place. Any time a package manager that relies on symlinks (eg homebrew) updates the CMake package, every build that relies on it breaks. (last time that happened I ended up downloading the cmake.org build and recreating the symlinks because I couldn't afford hours of rebuild)
Hmm? I know it puts the absolute path of the CMake that generated the cache file in there, but what else cares about the CMake version that shouldn't also be reconfigured when CMake changes anyways?
FWIW, CMake is highly relocatable and doesn't really care where it ends up living. That Homebrew removes versions you were using out from under you is not something we can fix. Or maybe I'm not understanding the problem fully (I don't use Homebrew with any consistency to know what the issue is here).
- Distros like unbundling (vcpkg, Anaconda, Spack, etc. also do too)
- Old releases of Lua are not maintained, so it would be our problem
- Compiled Lua packages need to match our build (and since we'd likely need multiple Lua versions, our symbols would be mangled, so nothing would work unless explicitly compiled against CMake's version). I really don't want to deal with "why can't I use Lua package XYZ?" issues every week
I know this might seem weird, but did you consider JavaScript? It takes backwards compatibility very seriously and with QuickJS (https://bellard.org/quickjs/) you could embed it easily if you wish to do so. I do understand that it could be a big culture shock to ask people using CMake to suddenly learn JavaScript and use it, and it has its fair share of warts.
We hadn't at the time because the only JS engines we could think of were 10x larger than CMake because they were built for browsers and had all kinds of cruft for a DOM that CMake doesn't care about hanging around. Maybe QuickJS would work, but (personally) I don't really think the language would be all that much of an improvement over what we have today (my issues are mostly around the oddities of which JS has quite a number of due to its roots). Then again, some of the ideas that have been tossed around for what could happen are probably interfering and I don't think JS would make those things any easier.
That presumes that we knew it would have been such a problem when CMake was started (back in the late 90's; I started contributing in 2010). The issue is that a lot of semantics are tied up in the way the language works between the list representation, variable lookup, scoping, policies, etc.
Perhaps it's time to cut your losses, throw out the syntax and build a new one? With a solid engine underneath this should in theory be feasible?
I once contributed to and even built a number of projects using CMake, but have since vowed that I'll never EVER open another CMakeLists.txt again, and will toss any CMake-based project that has builder errors I'd otherwise have to debug. CMake is the only build system of the dozens I've used over the last 30 years that has driven me to this. For awhile I held up hope with the advent of "Modern CMake", but alas it's not enough to fix things, and I doubt the syntax could ever be fixed.
Why would you rewrite it in rust? The engine is in C++. It's only the syntax part that's broken. So a new parser, new AST handling code, some engine section overhauls here and there to match the new reality. And they could even separate the engine from the syntax while they're at it, making it even more powerful. Simple? No. Necessary? Yes.
The alternative is to keep smearing more and more lipstick on the pig and hope people don't keep abandoning it, which is a shame because CMake COULD be great.
It's not only the syntax that is broken. The "type system" has quite a few holes in it because of the language. Many of them have no counterpart in other languages either. It would, for example, most likely still be stringly typed, so list representations would still get baked down into `;`-separated lists with all the problems that entails.
Meson has different engine and requires Python. CMake is already nearly perfect in everything except syntax. Moreover, it requires only C++ to be built so bootstrapping is easier than in case of Meson.
Python doesn't seem like a huge barrier to entry to me. CMake might be great in many ways, but the syntax is truly awful. It's just not intuitive to many people. I think due to how spacing works in CMake you can have incorrect things run and not fail. That's a huge issue, especially when onboarding people who aren't familiar with CMake.
Meson has some warts, especially around documentation but it's just easier and is just generally predictable. I've converted some CMake projects internally to meson and people were able to contribute and make changes with minimal learning and training, because the structure is intuitive. That means a lot.
Eh. CMake is a build system. It supports multiple build tools, but it's not like one can just tell CMake "here's some literal Make code, please put it into the generated files". The generated builds are still restricted to the semantics that CMake itself provides via its interface (e.g., no dyndep support in `add_custom_command` because only the Ninja generator supports that reasonably well; it's possible with Make, but I don't know how easy it would be to make it work for arbitrary custom commands).
[Not a C/C++ dev] I am under the assumption CMake do not perform build itself, it produces the build file for Make, Ninja etc. and have those tools run the build file to produce the final binary. If that's correct, wouldn't it make CMake a meta build system?
Right, I'm being more semantic here. CMake is a build system with the concepts of targets, linking, compiling, languages, toolchains, etc. The fact that it gets "rendered" into a `build.ninja`, `.vcxproj`, `.xcodebuild`, or `Makefile` is (largely) ambivalent (I call these "build tools" or "build executors"). Other instances would be `gn` or Meson. They're the build system; ninja is "just" the build tool they use to actually implement their semantics. Still others like build2, Boost.build, or the Bazel family are both: the build system and the build tool.
One cannot use CMake to write arbitrary Makefiles using all the fancy features available through macro expansion, rule templates, etc. All you have are CMake's semantics that can then be executed by a tool that implements POSIX makefile execution behaviors.
A (probably poor) analogy is a network resource. It can be provided over HTTP, Gopher, or whatever. The semantics of the resource are the same, but the transport mechanism is just how it interacts with the rest of the world.
Cargo etc also cache dependencies in a central location.
"Separation of concerns" is ill-defined — you can use cargo as just a package manager, just a build system, or both. Most people use it as both but there are definitely places (e.g. bigcos) that use just one of the aspects of cargo.
Systems that combine the two generally let you use just one or the other. For example, cargo fetch just acts as the package manager. cargo build can be configured (through vendoring or just not using any external dependencies) to not perform any package management.
What is missing is a modern cross-language, cross-platform package manager, I agree. But there are really good reasons for that, chief among them being that I don't think it's possible to define the scope of such a thing in a coherent fashion.
> Systems that combine the two generally let you use just one or the other. For example, cargo fetch just acts as the package manager. cargo build can be configured (through vendoring or just not using any external dependencies) to not perform any package management.
Indeed, and I'm still highly skeptical. Learning Rust has been an amazingly rare sequence of "wow that's perfect" uttered from my mouth – except for Cargo. I couldn't disable the networking functionality of Cargo fast enough!
Yes. Or not, if it's not suited for the task. But it seems insane to me to shoehorn this into the build system. I don't even understand why the latter should contain networking code at all.
> The sanest, of course, is to not update your dependencies unless really necessary, to avoid introducing new bugs.
So instead you live with all the existing (and unknown) bugs and security issues?
And when you finally do update those frozen dependencies it becomes a nightmare because of all the drift, not just in that package but also its dependencies.
You should be updating regularly; if you're concerned about stability then stay a release or two behind latest but that's not guaranteed to more stable than latest.
Too often I see people take the approach you're espousing to only suffer more later when they are forced to take an update, and then they become so scared of future updates that they never update again, manually backporting patches, and creating a nightmare legacy application.
> The sanest, of course, is to not update your dependencies unless really necessary, to avoid introducing new bugs
My experience over the 40 or so packages that I regularly use is that updates always have less bugs that old versions though. Even for my system, I build most of the software I use from git head and it just works better than whatever's in Ubuntu or Debian on average
One of the above, depending on my current situation each of the above (and a few others you didn't mention) make the most sense. That is why the cmake for a project shouldn't pull in dependencies, it should just build with whatever happens to be there or throw an error if something is missing.
Now there do exist "uber build systems" that are written in cmake. I'm fine with those existing (though I would probably not chose cmake to write it in, that is the author's choice). However when building a project you should not force a method to get dependencies on your users. The package management is a system level concern and you should not mess with it.
Note that I just put a lot of hate on systems like Cargo, pip, or hackage. Or google's pull everything into the one repository. Dependencies belong to tools like apt, rpm, yocto, ,snap, conan (when not building packages to install) - Microsoft/Apple are deficient for not providing them. Your program should just attempt to build with what it is given, not try to subvert my system - whatever it is. Yes I know it makes your life easier to not have to deal with more than one [possibly buggy] version of a dependency, but it makes my life worse.
> The package management is a system level concern and you should not mess with it.
This makes anything approaching a hermetic/deterministic/reproducible build basically impossible. That's not good. How do I get consistent builds across systems if I have to hope that some (set of, if I'm building on multiple OSes) external dependencies give me the correct version of the package I'm looking for, especially since many of the tools you listed don't, for example, support multiple simultaneous versions of a particular package.
I'm also confused by precisely what you mean by package management. Tools like pip cargo and bazel etc. Don't manage compiled binaries, but uncompiled sources. They deal with code dependencies, not dynamic library dependencies.
Installing a source file via apt doesn't make a whole lot of sense.
Yes. That seems reasonable. Gp suggested deferring such things to the os package manager (apt or pacman etc.) which doesn't, for example, handle keeping versions X and X+3 both available because I have different projects that depend on different versions.
> How else would you add/manage external dependencies for your project?
Either using a package manager, or doing it "manually" (like you suggest) – a process which can of course also be automated outside of the build system.
> Do you manually install them on your local host? That's not reproducible and doesn't support multiple versions for different projects.
Of course that's reproducible and supports multiple versions. Any build system worth its salt can be directed to look for dependencies at specific locations. Then multiple versions of things can be installed at multiple locations. This is literally how things have been done for ages.
> Do you manually copy all the external deps into your own project repo, and then manually keep them updated with the latest changes?
I abhor bundling dependencies, so I'm probably not the right person to ask, but supposing that you do want to bundle dependencies: yes (a process that can be scripted just as easily as it can be handed off to the build system!)
Why do I want the package manager to fetch and install dependencies for a project? I don't want to put all kinds of random junk into my /usr/lib before running "make" in the project's directory stops failing. And I've seen makefiles with "sudo apt install libqt5core libxml2-dev ..." in them and I didn't like that.
> Why do I want the package manager to fetch and install dependencies for a project?
You don't have to do that. If you separate the task of building from the task of managing software, you can mix and match solutions for each separate task as you wish.
> And I've seen makefiles with "sudo apt install libqt5core libxml2-dev ..." in them and I didn't like that.
Build systems that manage dependencies are that automation you're talking about, except they're easy-to-reuse across many projects. That's incredibly valuable.
Being able to ship software, especially across many platforms is much more important than maintaining some sort of purist separation between build system and package manager. Your view of affairs, after being dominant for decades, is finally receding into the background, and the world is a better place for it.
The separation of concerns is of far greater value than just purism. It can ease cross-compilation, it can help building in offline environments, it can make dependency customization far easier, it can make swapping out compilers much easier, etc.
These are all things that have been valuable in the past, and there's little reason to think that they aren't still valuable and will be valuable in the future.
cargo supports all of the things you talked about, though.
Cross-compiles are part of the fundamental model of cargo (target vs host dependencies).
cargo can be used in a completely offline manner easily, through either caching dependencies in an online step or vendoring them (or just not using dependencies).
Cargo has pretty good support for dependency patching.
The compiler can be swapped out through an environment variable, RUSTC_WRAPPER.
I agree that all of those things are valuable! The cargo developers think so too which is why cargo supports all of them.
I suspect the future is more in making tools like cargo support all the use cases people are concerned about, than trying to find an orthogonality where one doesn't exist.
It's no worse than several other not-very-secure-but-nonetheless-common ways to install software. Would a MS-provided copy-paste-curl-sh-to-install, say, be any worse than a link to an MSI? Either way I'm trusting that MS isn't sending me ransomware or a rootkit or whatever. If that MSI just executes curl under the hood, exactly like the sh command would have, is it better (nb this is what thin web installers for big programs do, basically)? If I add a PPA for some company and apt-install something from it, unless I audit all the files before proceeding, I'm trusting that they won't format my disk without my wanting them to.
How terrible an idea it is depends on who's providing it, same as most other software.
(A) the web server may serve different code for curl's user-agent than what you audited by clicking the link in a browser, leading to a false sense of security; (B) if the network transfer is interrupted, bash will partially execute an incomplete script; and (C) the recurring prevalence of high-profile bugs like https://github.com/valvesoftware/steam-for-linux/issues/3671
You can mitigate (B) by wrapping the whole script in a function, and you can mitigate (C) with `set -euo pipefail` and `trap`, but that doesn't seem to be the kind of bash scripts that people write in practice.
MSI (and deb/rpm) definitely can arbitrary bad things at install time too, but at least their built-in file extraction features are entirely declarative, and they always have an entrypoint for the system uninstaller.
Supporting distribution for all the different kinds of Linux package mangers alone isn't easy, and you take in how macOS and Windows do things, I'm not even sure where you start.
It's kind of sad, but there's multiple walled gardens in play here. I mean, Java's been around for 25 years, and it's just starting to get tooling for making OS installers for applications (jpackage).
So, programming toolchains just invent package management that is easy for everyone to use. Everyone needs a problem solved today for multiple platforms, and it's not like OSes are working together to make this easy.
Personally, I've come to prefer vendored dependencies with the sources, and building scripts around vendoring, separating the build from dependency distribution. But not a lot of programmers are comfortable with this. It's not a common practice, and the most popular version control system (git) isn't very great with binary management.
The point isn't the whole debate about vendoring. I have strong opinions on that matter, and they are different from yours, but independently of whether one wants vendoring or not: It seems to me to be highly advantageous to decouple building from dependency management (nomatter if said dependencies are vendored or not).
This is a worrisome trend. Now one needs to manually check what the build system will fetch (and from where) before starting the build.
Many times most of the dependencies are available through the package manager of ones system.
A good way in my opinion is just to provide a clear list of the dependencies and make the build system complain if it's not there so one can install it (if didn't bother manually to check that the listed dependencies are present).
Of course there are very large and complex projects which may need more sophisticated solutions - Perhaps some sort code notarisation would serve well? Yet even there the dependency management should be clearly separated from the build system and it should prioritize using system provided packages when possible.
There are probably other considerations and implications too. Like preference of static linking over dynamic etc..
Very nice tutorial on the subject! Probably it's a noob question, but - with such CMake setup, will it be possible to step into dependencies' source code when debugging?
Excellent guide. I wish I had had it when struggling to learn CMake initially. There’s good content there for those past the basics as well. For example, I had no idea FetchContent could be used to download release zips. I will be trying that out tonight.
I think FetchContent is quite recent and only really usable since cmake 3.14 or so, released in 2019.
So maybe that was just not yet possible when you learnt cmake initially.
I've much preferred, in recent years, to use Conan for fetching, building and importing external dependencies.
Being able to use Python to manage the individual packages made it somewhat easier to get stuck in to; avoiding, as much as possible, the dark art that is writing scripts for CMake.
I really wanted to use Conan, but the deal-breaker for me was discovering that Conan can only create one CMake target for a package, and that it must be called 'packagename::packagename'. Conan's packages could never work like native installs of the dependencies I wanted to use.
The author of the article will realize their mistake when they try to consume a library that consumes its own dependencies via the same FetchContent mechanism, and one of those dependencies will be the same as what author’s executable needs. Or when they need to compile a build tool (like protobuf compiler) that needs to target build host rather than what CMake currently targets.
All such ad-hoc solutions are not scalable, and as the project grows the moment will come when you’ll have to throw all of that away and do everything differently. Dependency management is a lot harder problem than just downloading and unpacking a zip. Why not leave the job to specialized tools from the start?
The argument against package managers is poor. At least with Conan, your project's CMake list does not have to depend on the package manager; instead one can consume dependencies from Conan and use `cmake_paths` generator and `find_library()` to link to them, allowing consumers of your project to use something other than Conan.
EDIT: Also, many open-source projects are not based on CMake themselves but have very weird very custom build scripts e.g. OpenSSL, FFmpeg, that can easily kill several days of your time each just to get going. All while package managers' recipes already have all of that figured out, just provide `feature1=yes feature2=no ...` and they will handle the rest.
This. For some of my native projects I’ve resorted to building a “fat lib” or amalgamated build of all dependencies into a static library. It’s a pattern I first saw when dealing with gameplay3d. It works for me. I then include the fat lib as a dependency to my library and let the compiler prune my lib of unused while leaving the static dependency lib unchanged. CMake on the surface looks good, is easy to work with, but once you start having dependencies of dependencies (some of which you need too) you run into issues that you have to script your way out of. I’ve tried vpkg as well and it has the same kind of issues.
What does the future hold for native development when so much of your initial time is spent fighting the tools?
I feel that if the author took the time to learn the problem domain well and why conan & vcpkg happen to exist, then this article wouldn't have been written. It's not like people that started conan/vcpkg didn't know about cmake's FetchContent and friends.
Just a simple case of switching branches where some dependencies are different is going to trigger a lot of issues. And it's a big waste of time waiting for everything to compile instead of having a package server like jfrog hosting compiled libraries & source and making sure all your flags and toolchain setup is applied. And dealing with library versions, and making them available to other dependencies outside your project. And having projects that use cmake and others that use autotools, and some that are custom etc. Those issues are a few obvious ones that you'll see if you barely scratch the surface.
Unfortunately, this article will stay on the internet and spread a very problematic way of dealing with dependencies.
> I feel that if the author took the time to learn the problem domain well and why conan & vcpkg happen to exist, then this article wouldn't have been written. It's not like people that started conan/vcpkg didn't know about cmake's FetchContent and friends.
Exactly this.
I would add that the author, for some reason, left out the most basic and I dare standard way to manage third-party dependencies: have a setup/provision script that downloads and installs dependencies locally.
You know, what Conan does, but ad-hoc.
In Linux it might be just a call to the package manager to install stuff from a djstro's dependencies, a kin to NPM's npm install. In Windows it might even be a big whole zip file bundling prebuilt binaries.
83 comments
[ 27.4 ms ] story [ 1605 ms ] threadI once looked into extracting the CMake build system generation code and make it into a standalone library. But I soon realized that it’s not exactly a weekend project.
It's one of those "ah, that sounds nice" things that is just too much work once you get down to it (and we have enough compatibility to handle without dealing with the vagaries of umpteen languages as it is).
Well, if your thing is fragmentation and bit rot and maintenance hell, sure.
There were plenty of competing build systems for C++ that provided Python bindings, such as SCons. For some reason, CMake with its awful BASIC-like DSL became the de facto standard.
Lua was also deemed not suitable in the long run because of our backwards compatibility guarantees. We'd need to ship N Lua interpreters internally because Lua is not compatible between 5.1, 5.2, 5.3, etc. Or say "we only support Lua 5.2, we cannot support 5.4" which…distros really dislike.
Why? You should always be able to update CMake and have it continue to work. If not, that's a regression in CMake and we take those very seriously.
Even today, in 2021, I have to use CMake 2.8.12 / GCC 4.8.5 in order to build code that will deploy on a CentOS 7 server. That means I have to backport the CMake file of every dependency that declares a minimum_requirement of 3+.
FWIW, `cmake3` is packaged in CentOS 7 (though it might be EPEL?).
All that said, given CMake's backwards compatibility guarantees, it is a bit silly that distributions don't update CMake more aggressively.
this would actually worsen the problem.
Main issue is the message "Compatibility with CMake < 2.8.12 will be removed from a future version of CMake" that appears so often when compiling older codes (not that old, written three or four years ago). When you change the version requirement on the offending CMakeLists.txt, the compilation breaks subtly and then you have to spend a very romantic afternoon with the build system and its beautiful language.
Backwards compatibility. You keep using that word. I do not think it means what you think it means.
CMake hard-codes the absolute path to itself all over the place. Any time a package manager that relies on symlinks (eg homebrew) updates the CMake package, every build that relies on it breaks. (last time that happened I ended up downloading the cmake.org build and recreating the symlinks because I couldn't afford hours of rebuild)
FWIW, CMake is highly relocatable and doesn't really care where it ends up living. That Homebrew removes versions you were using out from under you is not something we can fix. Or maybe I'm not understanding the problem fully (I don't use Homebrew with any consistency to know what the issue is here).
That presumes that we knew it would have been such a problem when CMake was started (back in the late 90's; I started contributing in 2010). The issue is that a lot of semantics are tied up in the way the language works between the list representation, variable lookup, scoping, policies, etc.
I once contributed to and even built a number of projects using CMake, but have since vowed that I'll never EVER open another CMakeLists.txt again, and will toss any CMake-based project that has builder errors I'd otherwise have to debug. CMake is the only build system of the dozens I've used over the last 30 years that has driven me to this. For awhile I held up hope with the advent of "Modern CMake", but alas it's not enough to fix things, and I doubt the syntax could ever be fixed.
The alternative is to keep smearing more and more lipstick on the pig and hope people don't keep abandoning it, which is a shame because CMake COULD be great.
Meson has some warts, especially around documentation but it's just easier and is just generally predictable. I've converted some CMake projects internally to meson and people were able to contribute and make changes with minimal learning and training, because the structure is intuitive. That means a lot.
Eh. CMake is a build system. It supports multiple build tools, but it's not like one can just tell CMake "here's some literal Make code, please put it into the generated files". The generated builds are still restricted to the semantics that CMake itself provides via its interface (e.g., no dyndep support in `add_custom_command` because only the Ninja generator supports that reasonably well; it's possible with Make, but I don't know how easy it would be to make it work for arbitrary custom commands).
One cannot use CMake to write arbitrary Makefiles using all the fancy features available through macro expansion, rule templates, etc. All you have are CMake's semantics that can then be executed by a tool that implements POSIX makefile execution behaviors.
A (probably poor) analogy is a network resource. It can be provided over HTTP, Gopher, or whatever. The semantics of the resource are the same, but the transport mechanism is just how it interacts with the rest of the world.
Do you manually install them on your local host? That's not reproducible and doesn't support multiple versions for different projects.
Do you use an uber build system like Yocto to wrap your CMake build, and let it pull the packages?
Do you manually copy all the external deps into your own project repo, and then manually keep them updated with the latest changes?
Also, separation of concerns is a good thing IMO.
"Separation of concerns" is ill-defined — you can use cargo as just a package manager, just a build system, or both. Most people use it as both but there are definitely places (e.g. bigcos) that use just one of the aspects of cargo.
What is missing is a modern cross-language, cross-platform package manager, I agree. But there are really good reasons for that, chief among them being that I don't think it's possible to define the scope of such a thing in a coherent fashion.
Indeed, and I'm still highly skeptical. Learning Rust has been an amazingly rare sequence of "wow that's perfect" uttered from my mouth – except for Cargo. I couldn't disable the networking functionality of Cargo fast enough!
That is the saner option among the ones that you mention.
The sanest, of course, is to not update your dependencies unless really necessary, to avoid introducing new bugs.
So instead you live with all the existing (and unknown) bugs and security issues?
And when you finally do update those frozen dependencies it becomes a nightmare because of all the drift, not just in that package but also its dependencies.
You should be updating regularly; if you're concerned about stability then stay a release or two behind latest but that's not guaranteed to more stable than latest.
Too often I see people take the approach you're espousing to only suffer more later when they are forced to take an update, and then they become so scared of future updates that they never update again, manually backporting patches, and creating a nightmare legacy application.
My experience over the 40 or so packages that I regularly use is that updates always have less bugs that old versions though. Even for my system, I build most of the software I use from git head and it just works better than whatever's in Ubuntu or Debian on average
Now there do exist "uber build systems" that are written in cmake. I'm fine with those existing (though I would probably not chose cmake to write it in, that is the author's choice). However when building a project you should not force a method to get dependencies on your users. The package management is a system level concern and you should not mess with it.
Note that I just put a lot of hate on systems like Cargo, pip, or hackage. Or google's pull everything into the one repository. Dependencies belong to tools like apt, rpm, yocto, ,snap, conan (when not building packages to install) - Microsoft/Apple are deficient for not providing them. Your program should just attempt to build with what it is given, not try to subvert my system - whatever it is. Yes I know it makes your life easier to not have to deal with more than one [possibly buggy] version of a dependency, but it makes my life worse.
This makes anything approaching a hermetic/deterministic/reproducible build basically impossible. That's not good. How do I get consistent builds across systems if I have to hope that some (set of, if I'm building on multiple OSes) external dependencies give me the correct version of the package I'm looking for, especially since many of the tools you listed don't, for example, support multiple simultaneous versions of a particular package.
I'm also confused by precisely what you mean by package management. Tools like pip cargo and bazel etc. Don't manage compiled binaries, but uncompiled sources. They deal with code dependencies, not dynamic library dependencies.
Installing a source file via apt doesn't make a whole lot of sense.
Have a script which generates build project/solution/script files for the actual build.
These scripts can use whatever tools your organisation is most comfortable with, up to and including a mono repo.
Either using a package manager, or doing it "manually" (like you suggest) – a process which can of course also be automated outside of the build system.
> Do you manually install them on your local host? That's not reproducible and doesn't support multiple versions for different projects.
Of course that's reproducible and supports multiple versions. Any build system worth its salt can be directed to look for dependencies at specific locations. Then multiple versions of things can be installed at multiple locations. This is literally how things have been done for ages.
> Do you manually copy all the external deps into your own project repo, and then manually keep them updated with the latest changes?
I abhor bundling dependencies, so I'm probably not the right person to ask, but supposing that you do want to bundle dependencies: yes (a process that can be scripted just as easily as it can be handed off to the build system!)
You don't have to do that. If you separate the task of building from the task of managing software, you can mix and match solutions for each separate task as you wish.
> And I've seen makefiles with "sudo apt install libqt5core libxml2-dev ..." in them and I didn't like that.
That's obviously terrible too.
Being able to ship software, especially across many platforms is much more important than maintaining some sort of purist separation between build system and package manager. Your view of affairs, after being dominant for decades, is finally receding into the background, and the world is a better place for it.
The separation of concerns is of far greater value than just purism. It can ease cross-compilation, it can help building in offline environments, it can make dependency customization far easier, it can make swapping out compilers much easier, etc.
These are all things that have been valuable in the past, and there's little reason to think that they aren't still valuable and will be valuable in the future.
Cross-compiles are part of the fundamental model of cargo (target vs host dependencies).
cargo can be used in a completely offline manner easily, through either caching dependencies in an online step or vendoring them (or just not using dependencies).
Cargo has pretty good support for dependency patching.
The compiler can be swapped out through an environment variable, RUSTC_WRAPPER.
I agree that all of those things are valuable! The cargo developers think so too which is why cargo supports all of them.
I suspect the future is more in making tools like cargo support all the use cases people are concerned about, than trying to find an orthogonality where one doesn't exist.
How terrible an idea it is depends on who's providing it, same as most other software.
There are some specific pitfalls with curl|bash:
(A) the web server may serve different code for curl's user-agent than what you audited by clicking the link in a browser, leading to a false sense of security; (B) if the network transfer is interrupted, bash will partially execute an incomplete script; and (C) the recurring prevalence of high-profile bugs like https://github.com/valvesoftware/steam-for-linux/issues/3671
You can mitigate (B) by wrapping the whole script in a function, and you can mitigate (C) with `set -euo pipefail` and `trap`, but that doesn't seem to be the kind of bash scripts that people write in practice.
MSI (and deb/rpm) definitely can arbitrary bad things at install time too, but at least their built-in file extraction features are entirely declarative, and they always have an entrypoint for the system uninstaller.
Supporting distribution for all the different kinds of Linux package mangers alone isn't easy, and you take in how macOS and Windows do things, I'm not even sure where you start.
It's kind of sad, but there's multiple walled gardens in play here. I mean, Java's been around for 25 years, and it's just starting to get tooling for making OS installers for applications (jpackage).
So, programming toolchains just invent package management that is easy for everyone to use. Everyone needs a problem solved today for multiple platforms, and it's not like OSes are working together to make this easy.
Personally, I've come to prefer vendored dependencies with the sources, and building scripts around vendoring, separating the build from dependency distribution. But not a lot of programmers are comfortable with this. It's not a common practice, and the most popular version control system (git) isn't very great with binary management.
Many times most of the dependencies are available through the package manager of ones system.
A good way in my opinion is just to provide a clear list of the dependencies and make the build system complain if it's not there so one can install it (if didn't bother manually to check that the listed dependencies are present).
Of course there are very large and complex projects which may need more sophisticated solutions - Perhaps some sort code notarisation would serve well? Yet even there the dependency management should be clearly separated from the build system and it should prioritize using system provided packages when possible.
There are probably other considerations and implications too. Like preference of static linking over dynamic etc..
Being able to use Python to manage the individual packages made it somewhat easier to get stuck in to; avoiding, as much as possible, the dark art that is writing scripts for CMake.
All such ad-hoc solutions are not scalable, and as the project grows the moment will come when you’ll have to throw all of that away and do everything differently. Dependency management is a lot harder problem than just downloading and unpacking a zip. Why not leave the job to specialized tools from the start?
The argument against package managers is poor. At least with Conan, your project's CMake list does not have to depend on the package manager; instead one can consume dependencies from Conan and use `cmake_paths` generator and `find_library()` to link to them, allowing consumers of your project to use something other than Conan.
EDIT: Also, many open-source projects are not based on CMake themselves but have very weird very custom build scripts e.g. OpenSSL, FFmpeg, that can easily kill several days of your time each just to get going. All while package managers' recipes already have all of that figured out, just provide `feature1=yes feature2=no ...` and they will handle the rest.
What does the future hold for native development when so much of your initial time is spent fighting the tools?
Just a simple case of switching branches where some dependencies are different is going to trigger a lot of issues. And it's a big waste of time waiting for everything to compile instead of having a package server like jfrog hosting compiled libraries & source and making sure all your flags and toolchain setup is applied. And dealing with library versions, and making them available to other dependencies outside your project. And having projects that use cmake and others that use autotools, and some that are custom etc. Those issues are a few obvious ones that you'll see if you barely scratch the surface.
Unfortunately, this article will stay on the internet and spread a very problematic way of dealing with dependencies.
Exactly this.
I would add that the author, for some reason, left out the most basic and I dare standard way to manage third-party dependencies: have a setup/provision script that downloads and installs dependencies locally.
You know, what Conan does, but ad-hoc.
In Linux it might be just a call to the package manager to install stuff from a djstro's dependencies, a kin to NPM's npm install. In Windows it might even be a big whole zip file bundling prebuilt binaries.