That doesn't strike me as a good characterization. Especially because both Python and Cmake are quite imperative. Meson is declarative with an emphasis on immutability.
It's "cmake in role only" - both are build systems, that's where the similarity falls apart.
The syntax is very pythonic, however, I think that's true of many ergonomic centric formats.
CMake is two smaller systems in a trenchcoat, with a creepy stalker following along.
There's the 3.0+ CMakeLists.txt syntax, which is a nice declarative target-based system.
There's the *.cmake language syntax for finding dependencies and other scripting, which is a C'thonian horror mix of imperative and declarative bits that the trench coat tries to hide.
Then there's the creepy pre-3.0 stalker, the old syntax for both. Non-target based, imperative bits sneaking in everywhere, makes Autotools look sane.
I'm using Meson for a project using C++ with Gtkmm, Libmicrohttpd, catch2 and other libraries. My experience is good neglecting some minor issues. Meson is fast, well readable if written well, uses system libraries by default and allows flexible usage.
On the other side I'm using Java with Maven and it is - a big burden. It's build in dependency retrieval system also isn't helping. Maybe I just don't like XML - because it is not human-readable ;)
I'm sorry to pick a nit, but could you expand on "well readable if written well"? I would say that a Makefile or even a CMakeLists.txt are readable if written well.
For my part, I find that being written well is necessary but not sufficient for readability in a Makefile.
The other big one is that you need to have a strong familiarity with Make, and use it often. Familiar because there's no way a person who hasn't actually been through some form of Make documentation in detail can even guess at what things like $@, $?, and $^ do, or accurately decipher its macro replacement syntax, or any of that. And use it often because, even if you were deeply familiar with Make in the past, if you haven't touched it in a few years, you're unlikely to reliably remember it without help.
I find pom.xml much more readable than deciphering a giant script file like build.gradle or CMakeLists.txt. 99% of the time I automatically know how the build works because its convention.
For me, Gradle's readability depends heavily on how you use it.
If you use it Maven-style, with relatively simple build scripts and all the complexity pushed out to plugins, it's very readable. A fair bit more readable than Maven, in my opinion, but I wouldn't care to argue that point in particular.
If you use it the way Stack Overflow tells you to do it, so that your build scripts are basically glorified ad-hoc Groovy programs, then, yeah, it's just impossible.
Unfortunately, since the only way to really understand any of Gradle is to understand all of Gradle, and understanding all of Gradle is a huge effort, it's kind of a catch-22. For most people, the only really sensible way to use it is to Google for advice and then do what Stack Overflow tells you to do. But that invariably leads to a result that is not even remotely sensible.
Long story short, my hot take is this. Maven is write-only. Gradle (ideal usage) is read-only. Gradle (normal usage) can be neither read nor written.
Meson is okay, it solved a few problems for us on our deployment but the creator has a few hard decisions and style choices that don't really gel well with other projects. We are tempted to go back to cmake.
One typical problem for C/C++ build systems is lack of proper Windows and MSVC toolchain support. Most C/C++ build systems originated in the UNIX world, and either only support GCC-compatible toolchains or if they support MSVC then only as an afterthought. Trying to get this stuff running in a Windows environment is often a mix of frustation and plain hilarity. CMake is one of the few build tools that gets Windows and MSVC support right.
Nix is likely a nonstarter because as far as I can tell it does not natively support Visual Studio and MSVC.
I suspect Bazel was ruled out because it requires the JVM and it has limited open source uptake relative to CMake (huge open source userbase), and Meson (limited presence in open source scientific software, but adopted by GNOME and systemd).
I can't speak for scipy, but I ruled Bazel out for a project that included Python components a few years ago. It was great for C, C++, and Java projects, but, at least at the time, there was no official Python support. I looked into third-party options and the feasibility of building it myself. Most of what I found was a few different conference talks and blog posts where the presenter was enthusiastically talking about how they'd spent a year trying to get it to work and it's not quite there yet but they're feeling really really confident that they'll turn a corner sometime soon. But I could never find any subsequent evidence that the presenter's team actually had turned that corner.
Based on that, I just sort of assumed that, in addition to all the fairly well-documented up-front challenges that these folks had identified and were talking about, there must also be some impassable barrier lurking around in there that nobody finds until they've already sunk a lot of time and money into trying to get it working. I don't know what that is, and I'm not curious enough to spend the better part of a year trying to find it for myself. I ended up choosing Gradle.
Our team started moving to Bazel two years ago with 0.21 or something. Python support is much, much better these days. We’re building Python with requirements and packing it into Docker containers.
It went from “maybe run stdlib Python in an activated venv“ to actually working.
That's great to hear. Yeah, 2 years ago would have been after I made this decision. Looking forward to taking another look at Bazel; I strenuously wanted it to be the one.
SciPy is a multi platform library that is distributed, most famously on pypi but in many other ways (conda, spack, rpm, deb, etc…). How can nix handle that without requiring the user to install nix?
nix-build relies on symlinks, which seemingly can't work on Windows. And it's not really a build system. Bazel relies on a JVM and it's really nice to not have one of those on your system if you don't absolutely need it.
Depending on how minimal you want to get, I think native build systems where you're not just building for yourself, but distributing source bundles that need to be built by others, have to start the discussion with Autotools, CMake, and Meson. The reason is just that they're already required by everything else anyway, so you're not adding any burden to the downstream packagers by using them.
If you're just building for yourself, have at it, get as bespoke and obscure as you want, but SciPy isn't being built and deployed primarily on NumFOCUS's own servers. It's being distributed as a tarball to its users who mostly build it themselves. You should make every effort to use a build system they're likely to already have and already be familiar with.
In particular, much of the focus here seems aimed at users who want to cross-compile for embedded Linuxes running on ARM. I doubt those people want to pull in a JVM.
Kind of reminds me of when I was working for a geointelligence agency charged with building and integrating third-party ground processing algorithms into a user-facing web tasking framework. We have to retrieve and build the dependencies, too. xerces-c, Armadillo, MKL, no problem. Just keeping downloading, either cmake .. or configure, make, then make install, over and over.
Then one of them requires TensorFlow and it becomes a two-month research project trying to figure out how to build it (sorry Google, but the military does not trust your pre-built binaries).
It would be great to port Meson from Python to pure C for portability. There is Boson[1] attempt to do exactly this but it's just a first step towards the goal.
Why is Python not portable, as in, on which systems is "build Python and then use that to run Meson" not a reasonable option?
The CI for boson seems like it runs on platforms where Python definitely is available, but also I notice the CI uses samurai, a reimplementation of ninja with a similar motivation: https://github.com/michaelforney/samurai
Ninja is in C++ so I am even more confused at Sanurai.
Is this just an implementation-diversity thing? (which is great!)
This is interesting. It will certainly give a boost to Meson, which is in part a better CMake (the compilation model is very similar). It probably makes a lot of sense for SciPy given the alternatives, but from an ecosystem perspective it's not clear whether it is better enough to justify the churn. For example, caching seems like a secondary concern and outsourced to Linux-specific technologies. This is unfortunate given the implications of build time for individual and team (CI) productivity, as well as the environmental considerations of redundant compilation at scale.
I haven't followed Meson closely in about 3 years, but I also got the sense that Windows support sometimes lagged. If that's true, it's going to be a tough sell for the many large C++ projects who adopted CMake almost exclusively due to its support for Windows and Visual Studio.
This is great. I used to support an line of Linux-based realtime audio products, and our engineers wanted me to include Scipy in the internal developer builds. It turned out to be a really tough ask, because of Scipy's build system. The result was a tech-debt blemish on an otherwise excellent codebase.
Scipy is very, very hard to cross-compile. Like, almost impossible. I eventually had to compile it on-target and store the outputs for inclusion in later images.
Distutils in general is wretched at cross-compilation tasks, should be retired.
This is interesting to read. I've used CMake on-and-off for mixed python/C++ code bases, and in particular https://scikit-build.readthedocs.io/en/latest/ which aims to integrate with cmake (lightly mentioned here but not evaluated as "getting away from setuptools".
But I have never been satisfied with the solutions for out-of-tree builds with python - being able to debug-in-place extensions _and_ eventually install in a single package structure seems to be somewhat in conflict with the way that Python allows you to structure packaging, without lots of environment variable hacks.
> Cross-compiling will become possible. For years we've told people "sorry, distutils wasn't really made for cross-compiling, let us know if you have any luck". As a result we've completely ignored users on some exotic platforms, and also spent a lot of time fighting with different CI systems to do native builds.
As a user (not a developer) of SciPy, this is the big win. My "exotic platform" is embedded Linux distributions via Buildroot[1]. This opens the door to many downstream libraries becoming available as well, such as pandas and scikit-learn.
33 comments
[ 4.6 ms ] story [ 76.1 ms ] threadIf you want to make that comparison, meson is more restrictive than cmake, but tends to have more functionality built-in to make up for it.
bazel is more all-encompasing than either, as it's more about reproducible and distributed builds than a "better makefile" type system.
It's "cmake in role only" - both are build systems, that's where the similarity falls apart.
The syntax is very pythonic, however, I think that's true of many ergonomic centric formats.
There's the 3.0+ CMakeLists.txt syntax, which is a nice declarative target-based system.
There's the *.cmake language syntax for finding dependencies and other scripting, which is a C'thonian horror mix of imperative and declarative bits that the trench coat tries to hide.
Then there's the creepy pre-3.0 stalker, the old syntax for both. Non-target based, imperative bits sneaking in everywhere, makes Autotools look sane.
On the other side I'm using Java with Maven and it is - a big burden. It's build in dependency retrieval system also isn't helping. Maybe I just don't like XML - because it is not human-readable ;)
The other big one is that you need to have a strong familiarity with Make, and use it often. Familiar because there's no way a person who hasn't actually been through some form of Make documentation in detail can even guess at what things like $@, $?, and $^ do, or accurately decipher its macro replacement syntax, or any of that. And use it often because, even if you were deeply familiar with Make in the past, if you haven't touched it in a few years, you're unlikely to reliably remember it without help.
:) it devolves into a custom shell script
If you use it Maven-style, with relatively simple build scripts and all the complexity pushed out to plugins, it's very readable. A fair bit more readable than Maven, in my opinion, but I wouldn't care to argue that point in particular.
If you use it the way Stack Overflow tells you to do it, so that your build scripts are basically glorified ad-hoc Groovy programs, then, yeah, it's just impossible.
Unfortunately, since the only way to really understand any of Gradle is to understand all of Gradle, and understanding all of Gradle is a huge effort, it's kind of a catch-22. For most people, the only really sensible way to use it is to Google for advice and then do what Stack Overflow tells you to do. But that invariably leads to a result that is not even remotely sensible.
Long story short, my hot take is this. Maven is write-only. Gradle (ideal usage) is read-only. Gradle (normal usage) can be neither read nor written.
Would be great to hear why those were the only two candidates that they considered (and not e.g. Bazel or Nix).
I suspect Bazel was ruled out because it requires the JVM and it has limited open source uptake relative to CMake (huge open source userbase), and Meson (limited presence in open source scientific software, but adopted by GNOME and systemd).
Based on that, I just sort of assumed that, in addition to all the fairly well-documented up-front challenges that these folks had identified and were talking about, there must also be some impassable barrier lurking around in there that nobody finds until they've already sunk a lot of time and money into trying to get it working. I don't know what that is, and I'm not curious enough to spend the better part of a year trying to find it for myself. I ended up choosing Gradle.
It went from “maybe run stdlib Python in an activated venv“ to actually working.
Depending on how minimal you want to get, I think native build systems where you're not just building for yourself, but distributing source bundles that need to be built by others, have to start the discussion with Autotools, CMake, and Meson. The reason is just that they're already required by everything else anyway, so you're not adding any burden to the downstream packagers by using them.
If you're just building for yourself, have at it, get as bespoke and obscure as you want, but SciPy isn't being built and deployed primarily on NumFOCUS's own servers. It's being distributed as a tarball to its users who mostly build it themselves. You should make every effort to use a build system they're likely to already have and already be familiar with.
In particular, much of the focus here seems aimed at users who want to cross-compile for embedded Linuxes running on ARM. I doubt those people want to pull in a JVM.
Kind of reminds me of when I was working for a geointelligence agency charged with building and integrating third-party ground processing algorithms into a user-facing web tasking framework. We have to retrieve and build the dependencies, too. xerces-c, Armadillo, MKL, no problem. Just keeping downloading, either cmake .. or configure, make, then make install, over and over.
Then one of them requires TensorFlow and it becomes a two-month research project trying to figure out how to build it (sorry Google, but the military does not trust your pre-built binaries).
[1] https://sr.ht/~bl4ckb0ne/boson/
The CI for boson seems like it runs on platforms where Python definitely is available, but also I notice the CI uses samurai, a reimplementation of ninja with a similar motivation: https://github.com/michaelforney/samurai
Ninja is in C++ so I am even more confused at Sanurai.
Is this just an implementation-diversity thing? (which is great!)
I haven't followed Meson closely in about 3 years, but I also got the sense that Windows support sometimes lagged. If that's true, it's going to be a tough sell for the many large C++ projects who adopted CMake almost exclusively due to its support for Windows and Visual Studio.
Scipy is very, very hard to cross-compile. Like, almost impossible. I eventually had to compile it on-target and store the outputs for inclusion in later images.
Distutils in general is wretched at cross-compilation tasks, should be retired.
But I have never been satisfied with the solutions for out-of-tree builds with python - being able to debug-in-place extensions _and_ eventually install in a single package structure seems to be somewhat in conflict with the way that Python allows you to structure packaging, without lots of environment variable hacks.
As a user (not a developer) of SciPy, this is the big win. My "exotic platform" is embedded Linux distributions via Buildroot[1]. This opens the door to many downstream libraries becoming available as well, such as pandas and scikit-learn.
[1]: https://buildroot.org/