Similar to EdenFS/Sapling VCS, buck2 is being developed as such a fast pace that it's hard to imagine there would be any OSS support for it. Together with buck2 are 3-5 other repos that are supposed to be built and used together with it.
If you want stability, Bazel is still a much better choice with their LTS release program.
I would rather have a look at Pants since it's now developed for the needs of Toolchain's customers, and not following the roadmap of a big company with a pretty unique infra. Its engine was also rewritten from Java to Rust.
I cannot wait until Pants supports Rust as a target language (they were looking for rust consumers as of 3-6 months ago if I recall). For our python needs it was a real breath of fresh air. I'm now at a shop without it and need to deal with slow dependency installs and dependency conflicts again :'(
So one problem we have at the moment in our project is that outdated Cargo.lock will basically break everything. Using `nix develop` will just fail to build the derivation, instead of regenerating the correct Cargo.lock (because it depends on someone not using `nix develop` to build the Rust project).
It seems like the ideal way to mix nix and cargo is to simply trust and delegate to cargo once you reach a rust dependency, not try to nixify it.
I think there was a rustup library for nix that probably would allow doing that but I haven't spent enough time trying (it's such a pain to debug flake.nix files)
You definitely should be leaving a sort of boundary between them. You can use fenix in your devshell to get a toolchain and then use crane for building. You just use cargo like normal when developing, including dependent crates, but CI can be handled with nix from that point.
I've got my flakes organized in a way that they are basically modular, so I can easily reuse files from other ones. If I need a rust toolchain, cargo builds, python toolchail, postgres, or whatever I can just import the file and change it up slightly (mostly names)
One question I have after all these articles and internal company presentations is: how exactly does Bazel track dependencies? Module A importing module B importing module C is a language-level feature, not a build system-level feature.
You basically have to declare all those dependencies in Bazel. For instance, if you look at rules for Go, they declare the whole dependency graph (Modules, their relations) in Bazel. There is tooling to generate that for you, so it's not really that bad.
Where it gets harder is when this language feature is integrated with an existing build system. C# is normally build with it's own build system, and to build it with Bazel you have to essentially rewrite some of it's logic in Bazel, which can be hard, but doable.
There are open-source rules for that, but at my place of work, we found those lacking. We had to write our own set of rules which would basically take all the source files for a given target, generate MSbuild files depending on the target type and call it from Bazel to build the code in this isolated environment. Another problem are nugets: you have to wrap each in a Bazel target in order to make it available during build, and analyze them beforehand to understand what kind of outputs (like .dlls) will they generate during build. It's a bit of a mess, sadly.
You will need to maintain MS Build files alongside the Bazel definitions. In theory the MS Build files can be generated from the Bazel definitions but someone has to write that tool.
I may be missing something, but it’s quite similar to pkgconfig+(cmake|autoconf). If you include a header, your build system probably needs to know where on the filesystem that header exists.
If you only ever need to build Rust code then cargo is all you should need. However if you need to build Rust, C, C++, python, and JavaScript all together in one build then Cargo is inevitably going to make you sad. That is when you reach for something like Bazel.
But really, I think that people don't go extremely granular on this. For Rust projects that are spread over 10-100 crates, I think that most people would just model the crate dependencies rather than at a file level. Mostly for pragmatic reasons, but also because that will likely model your test suite closely and your build requirements.
Bazel is a build tool, so one "module" (in Bazel you talk about target) per build artifact basically aligns with what people want. But you can in theory go really granular, making it so that tests only run if files it depend on change (less granular and that becomes "crates they depend on change")
I mean in the "real world" what happens is you do this once, and then you keep on using it and gradually make changes. Annoying, of course, but I think cargo is fortunately still relatively simple stuff, so the scope is much smaller than, say, Python setup.py shenanigans (though Rust build.rs stuff will definitely require some futzing around with environment variables at the Cargo level).
The "obvious" thing would be to either generate cargo files from Bazel stuff, generate Bazel stuff from cargo files, or have a third source that generates both. And like... yeah, if you are suffering from compile time issues, Bazel can make that stuff order of magnitudes faster (especially if your dep tree is pretty horizontal), so there's a reason to do it!
Not directly to your question, but basically bazel needs a static graph. So if you have a.cpp #include "a.h" - then you need to declare that "a.h" in your BUILD file.
But fear not, (and I'm not sure if BAZEL does this yet, but does it for java). If you've declared only "a.cpp", but forgot "a.h" - then bazel would detect that (/showIncludes in MSVC, or file system sandbox in linux, etc.) - and it'll tell you - you need to add "a.h" to your BUILD file.
Fear not again, it's possible to create a BUILDIFIER command that given what BAZEL told you it can do it automatically for you. Hence the BUILD files needs to be kept simple, for everyone to understand (even non-engineers), while the ugly parts go into the .bzl files and workspaces.
So in tha java world (but any language could do it), anything that was discovered dynamically but not declared in, bazel can offer how to fix it (copy+paste the command offered, or tool makes it for you) - at worst report it and you have to fix it manually
The important thing is - bazel can use only the information provided in the BUILD files to detect - oh hey, things have changed in the CI - so I need to rebuild. Without relying on "cl/gcc/clang" to run this through preproc step to discover actual headers. or keep horrible make-like deps.
Even just for running test suites incrementally, Bazel is pretty nice. If you have CI that takes "too long", Bazel is basically the only serious tool that can get you to incremental CI in a pretty rigorous way.
Though I seriously wish there were a better option, because there are a _lot_ of usability things with Bazel that make me very frustrated when working with a repo that is "merely" quite large.
I'm inclined to agree, but mainly because I can't see an issue where cargo is insufficient, it seems to me that would be an issue somewhere else. Not cargo specific.
Lol, Googler here, I can't deny that I have he urge to force Bazel into the world.
It's just, when I joined Google I found a build system I didn't hate. This was the first time. I don't actually love Bazel (Blaze internally), but I love that I don't hate it.
Every time I have to learn a new build system in open source work I groan. Why are there so many systems for essentially the same thing?
Question for an insider. Bazel code coverage seems to be one of the roughest edges. At least when running C++ tests.
Only lcov is supported, and due to the sandboxing it's impossible to get other tools working since they always spit out extra instrumentation data that's neither cached nor available on the next sandbox.
Does blaze support code coverage properly? Does Google just not care about code coverage? Am I just missing some obvious thing in the documentation?
Code coverage is pretty well supported in the internal ecosystem but I would not be surprised to hear this is because we designed Bazel around the system Google was already using for coverage.
> Code coverage is pretty well supported in the internal ecosystem (...)
What you're actually saying is that internally Google managed to get a single and very specific use case to work and forced all projects to either adopt it or have no code coverage.
You can get coverage working for almost anything by having each test convert its coverage data to LCOV before stopping, in the same sandbox as the test.
I forget the exact details but I think you can do this with a custom run_under test wrapper used during coverage. The wrapper runs your test, with whatever custom coverage system you have, then afterwards it converts to LCOV. Sometimes this might require updates to language specific rules because the test runner itself needs to be configured.
The bottom line is that Bazel upstream does not put a lot of attention into this, so it’s under developed outside of Google, because Google internal does not use the OSS rules.
>I forget the exact details but I think you can do this with a custom run_under test wrapper used during coverage. The wrapper runs your test, with whatever custom coverage system you have, then afterwards it converts to LCOV.
The testing is not all you need to modify. You need to modify every compile and link step, since you get byproducts of the instrumentation on every step.
>Sometimes this might require updates to language specific rules because the test runner itself needs to be configured.
That's what I found, too, however I feel like rewriting the C++ rules is a huge undertaking, since they're not defined in Starlark.
> You can get coverage working for almost anything by having each test convert its coverage data to LCOV before stopping, in the same sandbox as the test.
I have to call bullshit on this. With C or C++ you need to build the whole project with specific compiler flags to be able to generate code coverage reports. This is not something you implement by tweaking a setting in unit tests.
Because the outside world is diverse and open source does not have the ressources to write bazel rules. Nor does it fit their history.
There is definitely a need for better (and Meson is the foss answer, as well as Shake). But Bazel need a context that OpenSource do not have. Time and a clean slate of dependencies cooperating.
Also noone fund tools for DX for OpenSource devs :) only for JS and even that...
Not really. Some open source projects go back a long long time and have a large enough build system that rewriting it would require a lot of effort. Other open source projects are small enough that the simplest build system the author(s) could build was good enough.
The standard suite of open sourced Bazel rules are powerful enough for 99% of use cases. I have seen people argue this isn’t true, or that a rules package is fundamentally broken, but then learn that they are holding it wrong. And that holding it wrong reveals some glaring issues with how they setup their build system.
It’s not solely the user’s responsibility to hold it right. But Bazel is and has been developed in the “open source context” for long enough now that your argument doesn’t hold up imo.
These people hold other build tools wrong too. The difference is those other build tools often have no way of being held correctly. That Bazel is possible to use correctly at all explains why it was copied at least twice (Buck and Pants) before it was open sourced.
And yet noone use neither of them in this world. Maybe we should realise that if none of them can be hold right when people use them, it is because they are not adapted to the use.
on the other hand, there are some of us that really dislike Bazel. Try compiling something for ARM and then realising you can't find a build of the build tool for ARM... That was a special level of pain.
Tools commonly available in distros are typically much easier to use for outsiders.
I think Bazel is actually a good build system, but the issue is that they lack good educational resources. Then others adopt it at smaller companies without a team to manage Bazel and without investing time to onboard others into understanding how it works.
There are some new talks coming out external from the Bazel maintainers which do a better job of this, but you still have to hunt for them.
This is made worse because each of the rules packages, which are super useful, have their own idiosyncrasies which only make sense with a solid foundation in what problems Bazel solves, how it works and how to write/read rules.
My other wishlist for Bazel was an easier time setting up a hermetic sandbox without having caveats. This is improving, so I am optimistic on this front.
For context, I am not a Google (or ex-Google) engineer and I’ve had to work with a ton of different build systems on different levels of the stack.
Not just that it doesn't integrate well with external tools. Cmake works better here.
We had a googler at our company switch everything to bazel and it just slowed everyone down.
There really is something about Google engineers, this sort of subtle arrogance that the Google way is superior to everything else is existence. Maybe I'm the ass hole but I wonder if anyone can relate?
> We had a googler at our company switch everything to bazel and it just slowed everyone down.
I'd love to hear if anyone can make a case for Bazel knowing that cmake+build cache tools like ccache buy far better performance improvements than any full or incremental build.
FWIW performance is not really in the picture when I say "I wish everything was Bazel". Configured properly, CMake and Make are probably unbeatable for speed.
It's just that I really really hate maintaining Makefiles and CMake configs.
Admittedly in the latter case it's because I have never grokked CMake, but also... I don't want to grok CMake, I grok a perfectly good way of building C++ already. And it can also build Go and Rust and Typescript and Proto and JSonnet and it can run my janky codegen shell scripts and if we migrate to Carbon it will build that too and I won't even have to read the documentation.
Not saying I think everyone should migrate to Bazel (I hope I will not become That Ex-Googler), just explaining where the urge comes from.
I don't like bazel, but cmake is a piece of shit too.
The logic is convoluted, documentation is over complicated and the language contains too many primitive functions and ways to shoot yourself in the foot.
If I want near-instantaneous compile times with my build system I'll just make a bash script or Makefile and use a single compilation unit (which Rust does not really do).
When I use Bazel it is because:
- I want to build nearly everything from source in a controlled hermetic environment. No more "well it works on my machine"... only to learn the developer's libs leaked into the build environment.
- I want to do this with a single build system that works across many languages, and to build libraries which are used in many different languages.
- I want to do this within a sandboxed environment for peace of mind when compiling thousands of dependencies.
- I am willing to pay the money and time to setup remote executors to keep performance acceptable.
I don't think it's arrogance, it's just that blaze inside google is really nice, and they assume that using the tool outside google will also be really nice.
The issue is that to make it really nice:
- you have to convert everything to it, not just almost everything
- you need to set up build servers and lightning fast caching servers
- you need a team dedicated to building bazel rules for whatever doesn't already have rules (see: you must convert everything). This team is a bottleneck
Regular google engineers aren't exposed to these issues, because inside google it just works. It's only once they try to deploy it themselves that they realize it's a sisyphean task.
The thing is, bazel actually is really good at solving the issues it was designed for, it's just most companies do not have those particular problems acutely enough that it's worth paying the cost to switch to bazel.
>- you need a team dedicated to building bazel rules for whatever doesn't already have rules (see: you must convert everything). This team is a bottleneck
This is a major issue. A good build tool shouldn't require a dedicated team.
Ideally a build tool should function as sort of a settings menu for the project.
> This is a major issue. A good build tool shouldn't require a dedicated team.
Build systems are complex because they solve issues which only exist because of how OSes handle libraries and executables. I am pessimistic we can ever solve that issue in a capitalist society. I will be optimistic if we have a new major OS that isn't Windows, Linux, or MacOS in our lifetime or we solve death.
This was my fist experience with blaze too. First it was, wtf is this shit, and theze blaze-bin/etc folders. Then after I left Google I realized how much I miss it. No Make is not a replacement, or ninja.
The world has been running on make for decades, and it runs just fine. Meanwhile, does Bazel support basic stuff like shared libs or vended libs that are not deployed system-wide? Last time I checked, it didn't.
I never worked for Google nor worked with anyone from Google who said something like "you should move to Bazel" even once. I also did lots of work on in house build system migrations and optimizations.
Do you think that the problems with cargo mentioned in TFA are exaggerated or solvable? I think Bazel's support for caching, parallelism and custom rules are real advantages for large projects. What's wrong with this argument wrt Rust/cargo?
I worked at a company where a DevOps team was 5 to support a team of 35. They hired someone incredibly talented, knowledgeable and driven for change. They shrunk in size and are still over capacity and could now be 2 only.
Don't look at headcount to gauge complexity, difficulty, etc.
What I've experienced is the larger the team the more likely to have dilution of ownership. The less ownership the less likely each person will spend the energy or fully understand solving root causes.
Oddly, I'd say headcount can often be a driver of complexity.
In two ways, one being that more people legit need a more complicated system. You want that so that they interact with the system, not with each other.
The other being a bit of a Parkinson's Law. That is specifically that you will expand work to fit the allocated time; but I posit that you will also expand work to fit the people doing it. So, more people pushing ideas into the codebase will keep more ideas in the codebase. Even if fewer would work.
Any build system is a nightmare when you don't understand it. I don't know what that team moved to Bazel from. I suppose it was Makefiles, and I'm sure if that's what the team was using, almost all of those 25 people did not understand them either.
What parts of Bazel were a nightmare? What problems were fixed by these 4-5 people that were previously impacting all 25 people routinely?
> There are languages that need bazel, rust isn't one of them
Speak for yourself. Our Rust monorepo has nearly a dozen apps and counting, lots of shared libraries, C++ dependencies, etc. Tests require multiple cargo invocations.
We need a proper build system with distributed build caching. Cargo takes forever and doesn't understand the workloads.
Then again, the main issue of Bazel is that it’s complex to setup. If you already have the know how internally, it’s a reasonably good and well supported build system. It’s definitely better than cargo when it comes to caching and the breadth of targets available. Plus cargo is strictly limited to rust.
> Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything.
I feel like our use case at Grapl wasn't particularly crazy. It was just slightly more than "I'm a dev building a library and publishing it" - we had a workspace with multiple devs and CI/CD. CI/CD was really slow - issues with caching, rerunning unnecessary tests, etc.
Plus we had some Python code so unifying our builds would have been ncie.
This. As soon as you have to integrate multiple languages, jumping from the native build tool to an integrated build tool is a must. Why? So that you can saturate all your threads in an efficient way.
Hey Colin :wave: (I hope you're bringing Rust to DD)
To add onto this, Grapl had a moderate sized codebase with less than 10 engineers. I do think we could have done some optimizations in cargo, but it's simply much easier to use a tool like Bazel/pants etc that does the work for you as long as it supports your language of choice.
In addition to the problems stated in the article that you're ignoring here, in any large enough system, you are going to use multiple languages. Cargo isn't built for that. Bazel is.
> In addition to the problems stated in the article that you're ignoring here
What problems, exactly? The author acknowledged cargo works but hand-waves over vague claims of "it does not track dependencies well or support arbitrary build graphs", followed by acknowledging those issues are either non-issues or solved problems whose solution was dismissed without any good reason (caching tools).
The author's claims boil down to "I wanted to shoehorn Bazel but had no good reason to, so I double-down on Bazel's only selling point while ignoring everything".
It's ok if anyone just wants to try stuff without having any rational or coherent argument to justify it, but let's not call that "problems".
Nonsense. All build systems implement a DAG of build targets.
> why caching tools can't be used effectively
Nonsense. There are a myriad of caching tools freely available, from ccache, sccache, build cache, etc. To get them to work, all you need to do is pass compiler launchers to them, which means setting a single env/build flag.
The author is either trying hard to make a mountain out of irrelevant molehills, or is outright being disingenuous.
Bazel is being sold as a solution desperately grasping at straw-like problems, while being feature incomplete.
> All build systems implement a DAG of build targets.
And the example they gave is one that doesn't work so well.
> ccache, sccache, build cache, etc
The example they gave included the build cache misfiring. ccache doesn't work with Rust. sccache does, but can't be used with incremental compilation or when building binaries, making it essentially useless for what they needed, i.e. speeding up CI and local dev.
I really have no idea why you're still avoiding reading the article you're dismissing.
Bazel isn't about a particular language needing its build facilities - its about ALL of your organizations languages being built with one tool, ideally in one monorepository. Its very attractive when you have lots of microservices that would otherwise live in disparate repositories. Using a monorepo and Bazel allows you to share api definitions (proto files) across your entire code base (rules_proto, rules_grpc) , so you see immediately what api changes impact existing code across your entire organization, regardless of what languages each component is coded in.It allows you to containerize (rules_docker) and deploy (rules_k8s) only the components that need to be redeployed with every commit. It allows your entire development team to live, code, test, build, and deploy in harmony.
When I was at a large money center bank a constant source of problems was trying to deploy the 70+ microservices backing our business unit's main application during "release weekend", or for ad-hoc reasons like the log4shell vulnerability patch. It was very frustrating for our build pipelines to compile / test / static scan the entire millions of lines codebase when only a few libraries have changed. This build often took a few hours for our largest application and at least 10 minutes for each microservice. A few times we ran into breaking api changes and only realized it after it was deployed to production. This is the use case Bazel is serving - god like observability across your organizations code base, quickly compiled.
My problem with Bazel - at least based on my understanding when I evaluated it - is that you can't use the language-specific package managers for each module in your monorepo, so it's all or nothing. I'd prefer a tool that makes it much easier to defer build tasks to my package manager of choice in each project (eg Yarn for TS, Poetry for Python, etc.) and that lets me define rules at one level of abstraction above this.
In our monorepo, we ended up just using a Makefile, since it basically does that. Of course the Makefile is now a few hundred lines and we have about 60 *.compose.yml files, but that's another story.
Our current solution still uses the Makefile, but we basically split the monorepo into a Python part and a TypeScript part, and each of those has its own monorepo build system (Yarn workspaces for TS, and Poetry with Pants for Python).
We use a similar setup where I work. One challenge we have has been generating bazel targets for different sets of crate features for dependencies. We have solved this by writing Bazel tooling, but learning how to do this has cost a fair bit of time. We also rely heavily on cargo-raze, which has rough edges.
As with all things Bazel, it's great when it works, but it has a steep learning curve that could be hard to justify in many situations. You need at least one person on the team who is willing to really spend time on it.
Those look like the problems with rules cc for building C++. But it’s not inherent to Bazel, and would not expect problems producing static binaries in other languages.
We just had the ugly requirement of having to build a rust library/crate that uses C++ libraries (tensorflow lite + libedgetpu). That forced us to use bazel, on/for platforms that i have little experience / love for (Windows, Apple platforms).
Hitting the platform annoyances of each of those platforms just added to the pain. (On Windows, the 260 character path limit + on iOS Apple messing with ar binary used to create static libraries).
Left a very bad taste of trying to use bazel as a build tool. Even handwritten 1000+ lines of makefiles/experimental build systems like qbs did not cause so many headaches.
hmm. this migration would have been possible by staying with nix and developing a rules_rust equivalent (maybe it already exists) - see https://github.com/ghuntley/depot/tree/trunk/nix/buildGo for example but for golang. provides a simple dsl over nix and hides nix.
There is a broad tendency in software to jump onto whatever the big FAANG guys are doing. The trouble with this is often that they are working at a vastly bigger scale than you are, hence their solutions are vastly overfitted to your problem. So it's definitely interesting to read about this stuff but I'm sure Cargo is fine for most folks.
There is also big tendendcy to ignore them "because we are not at that scale" without realizing that you've reached the scale where you need it, even though your scale is magnitudes still less than what FAANG is.
The fallacy here is that just because you are not at the scale of the FAANG does not mean that their tools/libraries/products are not suitable for you - e.g. it's not the case that only if you reach their scale, then maybe you should switch there. This is bs.
This is true, and so you have to look at it case by case. I think if most developers look at the switching costs of bazel vs. just using a decent build tool like cargo, they wont make that tradeoff.
Similarly, they might choose some other tool out of google because it doesnt extract such a high price
I can get behind a story of starting with cargo, trying to improve by adding sccache, and when that's not enough switching to Bazel. I believe 99% of projects/organisations are best served with the first two of these steps, but it's nice to have the option of going to Bazel for the other 1% of cases (like when you have a lot of inhouse C++ in the mix).
I think this would have been useful with some code. My experience with Bazel is that you need a BUILD file in each subdirectory or something like that and every time I add a file I have to add it to that BUILD file.
Exceedingly tedious to use and has poor support in Clion.
It is likely that I don't have the constraints that this company has. The last time I tried this in Java I had to replicate my package dependency graph in Bazel. It was exceedingly pointless.
It is super tedious but for some languages a tool called Gazelle can auto-generate these for you. I know Go works well and there is a Java plugin too [0]. There is definitely some level rust support because that is how external crates are handled.
The [Magma project](https://github.com/magma/magma) uses Bazel. The transition seems to have gone well enough they're looking at ripping out the legacy Makefile based build system they maintained in parallel.
I used bazel for a few internal repositories written in Rust and it's amazing .... until you hit an issue.
Edge cases in Bazel are about as documented as any random fish you might find in a crack at the bottom of the ocean. And the most hilarious thing to deal with is "yeah this was written for a Google project, that other use case didn't apply to our project".
When eventually after the Xth issue I was told "why don't you just fork this standard package" (it was for typescript support inside the Rust repo) I was like "f*** bazel" and never touched it again.
It's amazing when it works. If it doesn't you're f******.
The experience of using Bazel inside Google with support is different from the experience of using Bazel outside Google.
Some ex-Googler may want to use Bazel and may have very good reasons for it, but they’re often unprepared to take over the support duties that, at Google, are provided by the developer tool teams.
In general, yes. Bazel has a lot of potential. It can improve developer productivity, with better correctness guarantees and improved build times. Or it can be a big sink of developer time, with people spending endless hours managing Bazel. It has potential to go both ways.
I found when i started using bazel outside of google that my mental model for how its apis really work was really lacking. Once i learned the apis (and the docs are outstanding at helping with that), and with the help of a few examples easily searchable via github, it became quite manageable
> I used bazel for a few internal repositories written in Rust and it's amazing
Could you explain how it beats for example, a Dockerfile in terms of what you need to write/get your hands dirty with (syntax, tooling, etc.) if your output goal is just an OCI image format to publish/run in container orchestration software? Is that too simple of a usecase?
> "yeah this was written for a Google project, that other use case didn't apply to our project"
Not only is that a very narrow set of use cases (Read: they only deploy statically linked binaries on mostly unix-y environments), Even the fixes for those very basic use cases take for ever.
I'm using Nix Flakes to support a active and medium-sized Open Source development team working on a decently-sized Rust project which includes cross-compilation for mobile devices, CI with larger integration tests, and so on. I had to find solutions to some of the same aspects as described here.
> This nix-alienation bifurcated our build environment: the CI servers built the code with nix-build, and developers built the code by entering the nix-shell and invoking cargo.
I don't anything wrong with this. Why would you throw the ability to simply use the Rust toolchain and all other tools directly?
It might be the fault of `nix-shell` which was a confusing and worked in some contrived ways. Nix Flakes have `nix develop` with an explicit "nix dev shell" construct which starts a shell that simply puts all the tools neccessary to build the project in the PATH (and possibly other relevant envs).
In our project developers simply need to install Nix and run `nix develop` (or use direnv) and from there on they mostly don't need to remember about Nix during their local work. Only when they need to touch something build/CI related they need to touch nix files, which is maybe 5% of the work, and then they can just ask for help.
The higher level tests do require a bit of thought so they reuse Nix building system, instead of fighting against it.
> Our security team chose Ubuntu as the deployment target and insisted that production binaries link against the regularly updated system libraries (libc, libc++, openssl, etc.) the deployment platform provides.
> Furthermore, the infrastructure team got a few new members unfamiliar with nix and decided to switch to a more familiar technology, Docker containers. The team implemented a new build system that runs cargo builds inside a docker container with the versions of dynamic libraries identical to those in the production environment.
I think at this point it was game over for using Nix.
Nix has a great support for building OCI (aka Docker) containers. It's great to be able to write 10 lines of Nix and just import all the required derivations from existing Nix build system and get a quick to build, nicely cacheable container that has only 35M.
But if you're not going to use it, then you turn a benefit into an extra work.
Using Nix to build inside Dockerfile is a terrible combination, eliminating lots of benefits of each done properly in separation.
> The idea of migrating the build system came from a few engineers (read Xooglers) who were tired of fighting with long build times and poor tooling.
It seems to me that it comes down to this. People didn't want to learn Nix, so they constructed bunch of incompatible ways of doing things.
Eventually there were already bunch of people familiar with Bazel so that's what they used. Had you have few engineers familiar with Nix, they would restructure things and make them work with Nix.
Some final tactical advice for people that would want to solve these problems with Rust. Use Nix Flakes, they are soo much better than old style Nix. Use `crane` for Rust - IMO the best compromise between "not rebuilding everything" and "simple wrapper over cargo". Use cachix or some shared Nix caching. Let developers work in `nix develop` shell and do their thing without thinking too much about Nix. Whenever they need part of codebase running reproducibly (fixtures for tests etc.) tactically invoke `nix build` and `nix run` . For normal Linux distros, provide OCI containers built natively with Nix. If you need raw binaries in order of preference: cross-compile to Musl (can't be done for C++), patchelf (a bit risky with C++ as OP mentioned), maintain a parallel build procedure via Dockerfile, but don't use Nix in it, and use it only to build the artifacts and nothing else.
131 comments
[ 5.1 ms ] story [ 203 ms ] thread> This project is not yet polished. We are continuing to develop it in the open, but don't expect it to be suitable for most people…
So it’s not ready yet to be used for anything serious I assume.
If you want stability, Bazel is still a much better choice with their LTS release program.
"Cargo is not a build system" reminded me of "Cabal is not a package manager"
https://ivanmiljenovic.wordpress.com/2010/03/15/repeat-after...
I'm using `crane` and `buildRustPackage` to build wasm front-end that is embedded into server binary (also rust). It works like a charm.
It seems like the ideal way to mix nix and cargo is to simply trust and delegate to cargo once you reach a rust dependency, not try to nixify it.
I think there was a rustup library for nix that probably would allow doing that but I haven't spent enough time trying (it's such a pain to debug flake.nix files)
You definitely should be leaving a sort of boundary between them. You can use fenix in your devshell to get a toolchain and then use crane for building. You just use cargo like normal when developing, including dependent crates, but CI can be handled with nix from that point.
I've got my flakes organized in a way that they are basically modular, so I can easily reuse files from other ones. If I need a rust toolchain, cargo builds, python toolchail, postgres, or whatever I can just import the file and change it up slightly (mostly names)
Where it gets harder is when this language feature is integrated with an existing build system. C# is normally build with it's own build system, and to build it with Bazel you have to essentially rewrite some of it's logic in Bazel, which can be hard, but doable.
So you need several layers of tooling just to get something built.
It's bad
But really, I think that people don't go extremely granular on this. For Rust projects that are spread over 10-100 crates, I think that most people would just model the crate dependencies rather than at a file level. Mostly for pragmatic reasons, but also because that will likely model your test suite closely and your build requirements.
Bazel is a build tool, so one "module" (in Bazel you talk about target) per build artifact basically aligns with what people want. But you can in theory go really granular, making it so that tests only run if files it depend on change (less granular and that becomes "crates they depend on change")
> For Rust projects that are spread over 10-100 crates
It's already 10-100 crates you need to manually declare and keep track of, or use a yet another tool to generate this for you.
And for big projects you'd also want to go more granular and split by parts of your app I guess.
The "obvious" thing would be to either generate cargo files from Bazel stuff, generate Bazel stuff from cargo files, or have a third source that generates both. And like... yeah, if you are suffering from compile time issues, Bazel can make that stuff order of magnitudes faster (especially if your dep tree is pretty horizontal), so there's a reason to do it!
But fear not, (and I'm not sure if BAZEL does this yet, but does it for java). If you've declared only "a.cpp", but forgot "a.h" - then bazel would detect that (/showIncludes in MSVC, or file system sandbox in linux, etc.) - and it'll tell you - you need to add "a.h" to your BUILD file.
Fear not again, it's possible to create a BUILDIFIER command that given what BAZEL told you it can do it automatically for you. Hence the BUILD files needs to be kept simple, for everyone to understand (even non-engineers), while the ugly parts go into the .bzl files and workspaces.
So in tha java world (but any language could do it), anything that was discovered dynamically but not declared in, bazel can offer how to fix it (copy+paste the command offered, or tool makes it for you) - at worst report it and you have to fix it manually
The important thing is - bazel can use only the information provided in the BUILD files to detect - oh hey, things have changed in the CI - so I need to rebuild. Without relying on "cl/gcc/clang" to run this through preproc step to discover actual headers. or keep horrible make-like deps.
Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything.
There are languages that need bazel, rust isn't one of them
Though I seriously wish there were a better option, because there are a _lot_ of usability things with Bazel that make me very frustrated when working with a repo that is "merely" quite large.
It still doesn't support post build steps, and integrates poorly with other tools that are used for more general builds.
It's just, when I joined Google I found a build system I didn't hate. This was the first time. I don't actually love Bazel (Blaze internally), but I love that I don't hate it.
Every time I have to learn a new build system in open source work I groan. Why are there so many systems for essentially the same thing?
Only lcov is supported, and due to the sandboxing it's impossible to get other tools working since they always spit out extra instrumentation data that's neither cached nor available on the next sandbox.
Does blaze support code coverage properly? Does Google just not care about code coverage? Am I just missing some obvious thing in the documentation?
Code coverage is pretty well supported in the internal ecosystem but I would not be surprised to hear this is because we designed Bazel around the system Google was already using for coverage.
What you're actually saying is that internally Google managed to get a single and very specific use case to work and forced all projects to either adopt it or have no code coverage.
I forget the exact details but I think you can do this with a custom run_under test wrapper used during coverage. The wrapper runs your test, with whatever custom coverage system you have, then afterwards it converts to LCOV. Sometimes this might require updates to language specific rules because the test runner itself needs to be configured.
The bottom line is that Bazel upstream does not put a lot of attention into this, so it’s under developed outside of Google, because Google internal does not use the OSS rules.
The testing is not all you need to modify. You need to modify every compile and link step, since you get byproducts of the instrumentation on every step.
>Sometimes this might require updates to language specific rules because the test runner itself needs to be configured.
That's what I found, too, however I feel like rewriting the C++ rules is a huge undertaking, since they're not defined in Starlark.
I have to call bullshit on this. With C or C++ you need to build the whole project with specific compiler flags to be able to generate code coverage reports. This is not something you implement by tweaking a setting in unit tests.
The most useful leads I've found so far are:
* The design doc for the (abandoned?) coverage framework [1]
* A prototype of GCOVR support [2]
* Comment on getting llvm-profdata working [3]
[1] https://docs.google.com/document/d/1-ZWHF-Q-qCKf19ik-t33ie58...
[2] https://github.com/ulfjack/bazel/commit/e9f21bbf562c1f6006eb...
[3] https://github.com/bazelbuild/bazel/issues/8178#issuecomment...
There is definitely a need for better (and Meson is the foss answer, as well as Shake). But Bazel need a context that OpenSource do not have. Time and a clean slate of dependencies cooperating.
Also noone fund tools for DX for OpenSource devs :) only for JS and even that...
But the open source world has the resources to write new build systems (and ecosystems!) time and time again. This argument does not hold up.
What leads you to believe that writing new build systems is a popular hobby?
I mean, cmake is the de facto standard for C++ projects and it has been for what? Over a decade? What does this tell you?
It’s not solely the user’s responsibility to hold it right. But Bazel is and has been developed in the “open source context” for long enough now that your argument doesn’t hold up imo.
Tools commonly available in distros are typically much easier to use for outsiders.
Definitely a ton of rough edges on Bazel but the passage of time is rounding them, even if progress uneven and frustratingly slow.
> Tools commonly available in distros are typically much easier to use for outsiders.
Which part about autotools or ninja is easier?
There are some new talks coming out external from the Bazel maintainers which do a better job of this, but you still have to hunt for them.
This is made worse because each of the rules packages, which are super useful, have their own idiosyncrasies which only make sense with a solid foundation in what problems Bazel solves, how it works and how to write/read rules.
My other wishlist for Bazel was an easier time setting up a hermetic sandbox without having caveats. This is improving, so I am optimistic on this front.
For context, I am not a Google (or ex-Google) engineer and I’ve had to work with a ton of different build systems on different levels of the stack.
We had a googler at our company switch everything to bazel and it just slowed everyone down.
There really is something about Google engineers, this sort of subtle arrogance that the Google way is superior to everything else is existence. Maybe I'm the ass hole but I wonder if anyone can relate?
I'd love to hear if anyone can make a case for Bazel knowing that cmake+build cache tools like ccache buy far better performance improvements than any full or incremental build.
It's just that I really really hate maintaining Makefiles and CMake configs.
Admittedly in the latter case it's because I have never grokked CMake, but also... I don't want to grok CMake, I grok a perfectly good way of building C++ already. And it can also build Go and Rust and Typescript and Proto and JSonnet and it can run my janky codegen shell scripts and if we migrate to Carbon it will build that too and I won't even have to read the documentation.
Not saying I think everyone should migrate to Bazel (I hope I will not become That Ex-Googler), just explaining where the urge comes from.
CMake is a high level makefile/build system generator.
There is nothing to maintain in modern cmake. You just set what executables/libraries you want to build, set their dependencies, and you're done.
And of all alternatives you chose to push, you pick Bazel of all things? Feature-incomplete and requiring all sorts of low-level maintenance?
The logic is convoluted, documentation is over complicated and the language contains too many primitive functions and ways to shoot yourself in the foot.
There's really nothing good that's available.
When I use Bazel it is because:
- I want to build nearly everything from source in a controlled hermetic environment. No more "well it works on my machine"... only to learn the developer's libs leaked into the build environment.
- I want to do this with a single build system that works across many languages, and to build libraries which are used in many different languages.
- I want to do this within a sandboxed environment for peace of mind when compiling thousands of dependencies.
- I am willing to pay the money and time to setup remote executors to keep performance acceptable.
The issue is that to make it really nice:
- you have to convert everything to it, not just almost everything
- you need to set up build servers and lightning fast caching servers
- you need a team dedicated to building bazel rules for whatever doesn't already have rules (see: you must convert everything). This team is a bottleneck
Regular google engineers aren't exposed to these issues, because inside google it just works. It's only once they try to deploy it themselves that they realize it's a sisyphean task.
The thing is, bazel actually is really good at solving the issues it was designed for, it's just most companies do not have those particular problems acutely enough that it's worth paying the cost to switch to bazel.
This is a major issue. A good build tool shouldn't require a dedicated team.
Ideally a build tool should function as sort of a settings menu for the project.
Build systems are complex because they solve issues which only exist because of how OSes handle libraries and executables. I am pessimistic we can ever solve that issue in a capitalist society. I will be optimistic if we have a new major OS that isn't Windows, Linux, or MacOS in our lifetime or we solve death.
Why?
The world has been running on make for decades, and it runs just fine. Meanwhile, does Bazel support basic stuff like shared libs or vended libs that are not deployed system-wide? Last time I checked, it didn't.
Can you actually provide a working example?
I'd love to hear what leads you to believe that adding yet another build system in the form of Bazel would solve that problem for you.
I agree that Cargo is already near perfect and I wouldn't suggest you switch to Bazel if you have a Rust project.
In our case we have a C++ project with some Python, some Rust. Having everything in Bazel here is good for us.
It's a good thing then that cmake+ccache work out of the box by passing the caching tool of your choice like ccache to the compiler launchers.
Do you think that the problems with cargo mentioned in TFA are exaggerated or solvable? I think Bazel's support for caching, parallelism and custom rules are real advantages for large projects. What's wrong with this argument wrt Rust/cargo?
Bazel is a nightmare. I wouldn't wish it on my worst enemy.
Don't look at headcount to gauge complexity, difficulty, etc.
What I've experienced is the larger the team the more likely to have dilution of ownership. The less ownership the less likely each person will spend the energy or fully understand solving root causes.
In two ways, one being that more people legit need a more complicated system. You want that so that they interact with the system, not with each other.
The other being a bit of a Parkinson's Law. That is specifically that you will expand work to fit the allocated time; but I posit that you will also expand work to fit the people doing it. So, more people pushing ideas into the codebase will keep more ideas in the codebase. Even if fewer would work.
What parts of Bazel were a nightmare? What problems were fixed by these 4-5 people that were previously impacting all 25 people routinely?
Hey look another nimwit that make stupid assumptions without understanding the nature of the problem being solved.
> and it's integrated with everything.
Oh wow Cargo integrates with Python, shell scripts, C++ and node JS? Holly sh*t that’s amazing.
/s
Speak for yourself. Our Rust monorepo has nearly a dozen apps and counting, lots of shared libraries, C++ dependencies, etc. Tests require multiple cargo invocations.
We need a proper build system with distributed build caching. Cargo takes forever and doesn't understand the workloads.
I feel like our use case at Grapl wasn't particularly crazy. It was just slightly more than "I'm a dev building a library and publishing it" - we had a workspace with multiple devs and CI/CD. CI/CD was really slow - issues with caching, rerunning unnecessary tests, etc.
Plus we had some Python code so unifying our builds would have been ncie.
To add onto this, Grapl had a moderate sized codebase with less than 10 engineers. I do think we could have done some optimizations in cargo, but it's simply much easier to use a tool like Bazel/pants etc that does the work for you as long as it supports your language of choice.
What problems, exactly? The author acknowledged cargo works but hand-waves over vague claims of "it does not track dependencies well or support arbitrary build graphs", followed by acknowledging those issues are either non-issues or solved problems whose solution was dismissed without any good reason (caching tools).
The author's claims boil down to "I wanted to shoehorn Bazel but had no good reason to, so I double-down on Bazel's only selling point while ignoring everything".
It's ok if anyone just wants to try stuff without having any rational or coherent argument to justify it, but let's not call that "problems".
Nonsense. All build systems implement a DAG of build targets.
> why caching tools can't be used effectively
Nonsense. There are a myriad of caching tools freely available, from ccache, sccache, build cache, etc. To get them to work, all you need to do is pass compiler launchers to them, which means setting a single env/build flag.
The author is either trying hard to make a mountain out of irrelevant molehills, or is outright being disingenuous.
Bazel is being sold as a solution desperately grasping at straw-like problems, while being feature incomplete.
And the example they gave is one that doesn't work so well.
> ccache, sccache, build cache, etc
The example they gave included the build cache misfiring. ccache doesn't work with Rust. sccache does, but can't be used with incremental compilation or when building binaries, making it essentially useless for what they needed, i.e. speeding up CI and local dev.
I really have no idea why you're still avoiding reading the article you're dismissing.
Yeah, that's an effective way to sell someone's promotion, but it's always a promise that is never met and instead it's a constant source of problems.
Sometimes reinventing the wheel just buys you a wobbly, squeaky wheel.
In our monorepo, we ended up just using a Makefile, since it basically does that. Of course the Makefile is now a few hundred lines and we have about 60 *.compose.yml files, but that's another story.
Our current solution still uses the Makefile, but we basically split the monorepo into a Python part and a TypeScript part, and each of those has its own monorepo build system (Yarn workspaces for TS, and Poetry with Pants for Python).
Other big complaints were:
- big CPU hog
- bad IDE integration (esp scala/java for intelliJ)
- it lets people write i-cant-believe-its-not-python, which is far too much rope to hang yourself with. I've seen horrors.
As with all things Bazel, it's great when it works, but it has a steep learning curve that could be hard to justify in many situations. You need at least one person on the team who is willing to really spend time on it.
I successfully migrated recently and it's such a gigantic improvement.
TBH it’s hard to say which operate model is better anyway. They are simply incompatible :)
Related topic: https://youtu.be/Pzl1B7nB9Kc
https://github.com/bazelbuild/bazel/issues/1920
https://stackoverflow.com/questions/32845940/symbols-from-st...
We just had the ugly requirement of having to build a rust library/crate that uses C++ libraries (tensorflow lite + libedgetpu). That forced us to use bazel, on/for platforms that i have little experience / love for (Windows, Apple platforms).
Hitting the platform annoyances of each of those platforms just added to the pain. (On Windows, the 260 character path limit + on iOS Apple messing with ar binary used to create static libraries).
Left a very bad taste of trying to use bazel as a build tool. Even handwritten 1000+ lines of makefiles/experimental build systems like qbs did not cause so many headaches.
to show how awesome bazel is - in one step it can build (compile) and package them into a .zip file ready to deploy to NuGet-like repo or other place.
The fallacy here is that just because you are not at the scale of the FAANG does not mean that their tools/libraries/products are not suitable for you - e.g. it's not the case that only if you reach their scale, then maybe you should switch there. This is bs.
Similarly, they might choose some other tool out of google because it doesnt extract such a high price
Exceedingly tedious to use and has poor support in Clion.
It is likely that I don't have the constraints that this company has. The last time I tried this in Java I had to replicate my package dependency graph in Bazel. It was exceedingly pointless.
I found an example repo for rust and Bazel and it is just as tedious as I recall https://github.com/laramiel/bazel-example-rust
0 - https://github.com/bazel-contrib/rules_jvm/tree/main/java/ga...
a BUILD file declares a "package" and the demarcation line happens when it sees another BUILD file in a sub-directory. Then that's another "package".
Edge cases in Bazel are about as documented as any random fish you might find in a crack at the bottom of the ocean. And the most hilarious thing to deal with is "yeah this was written for a Google project, that other use case didn't apply to our project".
When eventually after the Xth issue I was told "why don't you just fork this standard package" (it was for typescript support inside the Rust repo) I was like "f*** bazel" and never touched it again.
It's amazing when it works. If it doesn't you're f******.
Some ex-Googler may want to use Bazel and may have very good reasons for it, but they’re often unprepared to take over the support duties that, at Google, are provided by the developer tool teams.
Could you explain how it beats for example, a Dockerfile in terms of what you need to write/get your hands dirty with (syntax, tooling, etc.) if your output goal is just an OCI image format to publish/run in container orchestration software? Is that too simple of a usecase?
Not only is that a very narrow set of use cases (Read: they only deploy statically linked binaries on mostly unix-y environments), Even the fixes for those very basic use cases take for ever.
Eg. This 7 year old issue is still open: https://github.com/bazelbuild/bazel/issues/1920 . To be able to create a statically linked library, we had to use: https://github.com/hotg-ai/librunecoral/blob/master/runecora... . Had to use some weird hack to build shared libraries too. Overall, it was just annoying.
> This nix-alienation bifurcated our build environment: the CI servers built the code with nix-build, and developers built the code by entering the nix-shell and invoking cargo.
I don't anything wrong with this. Why would you throw the ability to simply use the Rust toolchain and all other tools directly?
It might be the fault of `nix-shell` which was a confusing and worked in some contrived ways. Nix Flakes have `nix develop` with an explicit "nix dev shell" construct which starts a shell that simply puts all the tools neccessary to build the project in the PATH (and possibly other relevant envs).
In our project developers simply need to install Nix and run `nix develop` (or use direnv) and from there on they mostly don't need to remember about Nix during their local work. Only when they need to touch something build/CI related they need to touch nix files, which is maybe 5% of the work, and then they can just ask for help.
The higher level tests do require a bit of thought so they reuse Nix building system, instead of fighting against it.
> Our security team chose Ubuntu as the deployment target and insisted that production binaries link against the regularly updated system libraries (libc, libc++, openssl, etc.) the deployment platform provides.
> Furthermore, the infrastructure team got a few new members unfamiliar with nix and decided to switch to a more familiar technology, Docker containers. The team implemented a new build system that runs cargo builds inside a docker container with the versions of dynamic libraries identical to those in the production environment.
I think at this point it was game over for using Nix.
Nix has a great support for building OCI (aka Docker) containers. It's great to be able to write 10 lines of Nix and just import all the required derivations from existing Nix build system and get a quick to build, nicely cacheable container that has only 35M.
But if you're not going to use it, then you turn a benefit into an extra work.
Using Nix to build inside Dockerfile is a terrible combination, eliminating lots of benefits of each done properly in separation.
> The idea of migrating the build system came from a few engineers (read Xooglers) who were tired of fighting with long build times and poor tooling.
It seems to me that it comes down to this. People didn't want to learn Nix, so they constructed bunch of incompatible ways of doing things.
Eventually there were already bunch of people familiar with Bazel so that's what they used. Had you have few engineers familiar with Nix, they would restructure things and make them work with Nix.
Some final tactical advice for people that would want to solve these problems with Rust. Use Nix Flakes, they are soo much better than old style Nix. Use `crane` for Rust - IMO the best compromise between "not rebuilding everything" and "simple wrapper over cargo". Use cachix or some shared Nix caching. Let developers work in `nix develop` shell and do their thing without thinking too much about Nix. Whenever they need part of codebase running reproducibly (fixtures for tests etc.) tactically invoke `nix build` and `nix run` . For normal Linux distros, provide OCI containers built natively with Nix. If you need raw binaries in order of preference: cross-compile to Musl (can't be done for C++), patchelf (a bit risky with C++ as OP mentioned), maintain a parallel build procedure via Dockerfile, but don't use Nix in it, and use it only to build the artifacts and nothing else.