That's really cool. Fascinating to think about all the problems that get missed due to poor or missing visualizations like this.
I did a lot of work to improve the Mozilla build system a decade ago where I would have loved this tool. Wish they would have said what problem they found.
Is there a tool that records the timestamp of each executed command during a build, and when you rebuild, it tells you how much time is left instead of "building obj 35 out of 1023" ?
Or (for cmake or ninja) use a CSV that says how long each object takes to build and use it to estimate how much is left ?
I am stuck in an environment with CMake, GCC and Unix Make (no clang, no ninja) and getting detailed information about WHY the build is taking so long is nearly impossible.
It's also a bit of a messy build with steps like copying a bunch of files from the source into the build folder. Multiple languages (C, C++, Fortran, Python), custom cmake steps, etc.
If this tool can handle that kind of mess, I'll be very interested to see what I can learn.
> I am stuck in an environment with CMake, GCC and Unix Make (no clang, no ninja) and getting detailed information about WHY the build is taking so long is nearly impossible.
I have a similar problem, with a tangential question that I think about from time to time without really having the time to investigate it further, unfortunately.
I notice sometimes that CMake recompiles files that shouldn't have been affected by the code changes made previously. Like recompiling some independent objects after only slight changes to a .cpp file without any interface changes.
So I often wonder if CMake is not making some file more inter-dependent than what they are, leading to longer compilation times.
Nice, I’ve been looking for something like this for a while.
I’ve noticed on my huge catkin cmake project that cmake is checking the existence of the same files hundreds of times too. Is there anything that can hook into fork() and provide a cached value after the first invocation?
> It also has 6 seconds of inactivity before starting any useful work. For comparison, ninja takes 0.4 seconds to start compiling the 2,468,083 line llvm project. Ninja is not a 100% fair comparison to other tools, because it benefits from some “baked in” build logic by the tool that created the ninja file, but I think it’s a reasonable “speed of light” performance benchmark for build systems.
This is an important observation that is often overlooked. What’s more, the changes to the information on which this “baked in” build logic is based is not tracked very precisely.
How close can we get to this “speed of light” without such “baking in”? I ran a little benchmark (not 100% accurate for various reasons but good enough as a general indication) which builds the same project (Xerces-C++) both with ninja as configured by CMake and with build2, which doesn’t require a separate step and does configuration management as part of the build (and with precise change tracking). Ninja builds this project from scratch in 3.23s while build2 builds it in 3.54s. If we omit some of the steps done by CMake (like generating config.h) by not cleaning the corresponding files, then the time goes down to 3.28s. For reference, the CMake step takes 4.83s. So a fully from-scratch CMake+ninja build actually takes 8s, which is what you would normally pay if you were using this project as a dependency.
I love the visualization, I think it's great information and will be very helpful to whoever uses it.
I would think about a different name. Often names are either meant to be funny or just unique nonsense but something short and elegantly descriptive (like BuildViz etc.) can go a long way to making it seem more legitimate and being more widely used.
Love it!
We did something similar using strace/dtruss back in 2018 with https://buildinfer.loopperfect.com/ and were generating graphs (using eg. graphviz and perfetto.dev) and BUCK files on the back of that
Whilst we regrettably never came around to package it as a propper product, we found it immensly valuable in our consulting work, to pinpoint issues and aid the conversion to BUCK/Bazel.
We used graphviz, https://perfetto.dev/ and couple other tools to visualise things
Recently we cicled back to this too but with a broader usecase in mind.
There are some inherent technical challanges with this approach & domain:
- syscall logs can get huge - especially when saved to disk. Our strace logs would get over 100GB for some projects (llvm was around ~50GB)
- some projects also use https and inter process communications and that needs ot be properly handled too. (We even had a customer that was retriving code from a firebird database via perl as part of the compilation step!)
- It's runtime analysis - you might need to repeat the analysis for each configuration.
the parallels between tech and manufacturing never cease to amaze, this looks so much like the machine monitoring / execution system we use in the car parts plant I want to ask if you've calculated the TEEP and OEE of your build farm
A CMake build is used as an example in the blog post. It's literally just visualizing all spawned processes, so it will work for anything that spawns subprocesses to do the build.
If you use the Visual C++ compiler on Windows, vcperf is worth a look: https://github.com/microsoft/vcperf - comes with VS2022, or you can build from github.
I've used it with projects generated by UBT and CMake. I can't remember if it provides any info that'd let you assess the quality of build parallelism, but it does have some compiler front end info which is pretty straightforward to read. Particularly expensive headers (whether inherently so, or just because they're included a lot) are easy to find.
Really cool tool, but perhaps not for the original use-case. I often find myself trying to figure out what call tree a large Bash script creates, and this looks like it visualises it well.
This would have been really useful 6 months ago, when I was trying to figure out what on earth some Jetson tools actually did to build and flash an OS image.
It visualizes each crate's build, shows the dependencies between them, shows when the initial compilation is done that unblocks dependents, and soon will have link information.
42 comments
[ 4.2 ms ] story [ 61.1 ms ] threadI did a lot of work to improve the Mozilla build system a decade ago where I would have loved this tool. Wish they would have said what problem they found.
Or (for cmake or ninja) use a CSV that says how long each object takes to build and use it to estimate how much is left ?
What limits your tool to compiler/build tools, can it be used for any arbitrary process?
I am stuck in an environment with CMake, GCC and Unix Make (no clang, no ninja) and getting detailed information about WHY the build is taking so long is nearly impossible.
It's also a bit of a messy build with steps like copying a bunch of files from the source into the build folder. Multiple languages (C, C++, Fortran, Python), custom cmake steps, etc.
If this tool can handle that kind of mess, I'll be very interested to see what I can learn.
I have a similar problem, with a tangential question that I think about from time to time without really having the time to investigate it further, unfortunately.
I notice sometimes that CMake recompiles files that shouldn't have been affected by the code changes made previously. Like recompiling some independent objects after only slight changes to a .cpp file without any interface changes.
So I often wonder if CMake is not making some file more inter-dependent than what they are, leading to longer compilation times.
Is there a version available for MacOS today?? I'd love to give it a whirl... For Rust, C++ / Swift and other stuff.
Thanks!
Developers always get it wrong and do it badly.
I’ve noticed on my huge catkin cmake project that cmake is checking the existence of the same files hundreds of times too. Is there anything that can hook into fork() and provide a cached value after the first invocation?
This is an important observation that is often overlooked. What’s more, the changes to the information on which this “baked in” build logic is based is not tracked very precisely.
How close can we get to this “speed of light” without such “baking in”? I ran a little benchmark (not 100% accurate for various reasons but good enough as a general indication) which builds the same project (Xerces-C++) both with ninja as configured by CMake and with build2, which doesn’t require a separate step and does configuration management as part of the build (and with precise change tracking). Ninja builds this project from scratch in 3.23s while build2 builds it in 3.54s. If we omit some of the steps done by CMake (like generating config.h) by not cleaning the corresponding files, then the time goes down to 3.28s. For reference, the CMake step takes 4.83s. So a fully from-scratch CMake+ninja build actually takes 8s, which is what you would normally pay if you were using this project as a dependency.
I would think about a different name. Often names are either meant to be funny or just unique nonsense but something short and elegantly descriptive (like BuildViz etc.) can go a long way to making it seem more legitimate and being more widely used.
Whilst we regrettably never came around to package it as a propper product, we found it immensly valuable in our consulting work, to pinpoint issues and aid the conversion to BUCK/Bazel. We used graphviz, https://perfetto.dev/ and couple other tools to visualise things
Recently we cicled back to this too but with a broader usecase in mind.
There are some inherent technical challanges with this approach & domain:
- syscall logs can get huge - especially when saved to disk. Our strace logs would get over 100GB for some projects (llvm was around ~50GB)
- some projects also use https and inter process communications and that needs ot be properly handled too. (We even had a customer that was retriving code from a firebird database via perl as part of the compilation step!)
- It's runtime analysis - you might need to repeat the analysis for each configuration.
I've used it with projects generated by UBT and CMake. I can't remember if it provides any info that'd let you assess the quality of build parallelism, but it does have some compiler front end info which is pretty straightforward to read. Particularly expensive headers (whether inherently so, or just because they're included a lot) are easy to find.
This would have been really useful 6 months ago, when I was trying to figure out what on earth some Jetson tools actually did to build and flash an OS image.
> Here it is recording the build of a macOS app:
> <gif>
At the top of the page, it should be right under the header.
You made a thing, so show the thing. You can waffle on about it later. Just show the thing.
It visualizes each crate's build, shows the dependencies between them, shows when the initial compilation is done that unblocks dependents, and soon will have link information.