Unfortunately for those of us following HN through RSS readers (like Feedly) the URL doesn't show until you mouse over the link ... but even then, I agree adding that additional piece of information such as "build software" would be useful.
Feedly has an API, I started writing a script a while ago to improve some RSS feeds, including HN (by making the number of comments and the URL appear in the body), but never finished it.
If someone had the same idea, please don’t hesitate sharing.
Since far from all TLDs have reliable semantics and it's impossible to know all of them, it doesn't. And compiling a project from source code is not the only meaning of "build", not even in tech.
Not the GP, but as a developer reasonably conversant with build systems and their pain points, I still assumed this was going to be some form of lazyweb[0] related blog post or portal rather than a build system named 'Please'.
For what it's worth, I think the too-short title is pretty normal for HN. I'd blame its less than optimum informativeness on HN's cultural norm of short link titles, as reinforced by the administrators' moderation.
I don’t think this is a fair reason to write the tool off. They support Homebrew, and I’m sure will add other install methods in the future. Piping to bash is no worse than clicking “yes” on every step of an install wizard.
Often these "pipe to Sh scripts" support --help and a variety of configuration options anyway. The benefit of a script over a binary installer at least is that you can inspect the script before running it!
Have you inspected one of these scripts? What have you found? (I've tried it a few times and haven't felt like I learned anything meaningful from doing so.)
Yes, plenty of times. Usually I find that they do what you would expect them to do: set up a bunch of parameters like an installation prefix, and then copy files around. They also handle user input options and maybe prompt for some stuff.
The usual annoying thing is the automated package install. I have not looked at this particular package, but in the past, I have seen:
- installing specific gcc version and making it system-default.
- installing “virtualbox” packages - this was the worst, as the machine had a few KVM VMs running at the same time, and KVM and VirtualBox cannot run at the same time.
In general, I now stay away from every install script, and if I cannot, then I run it in docker container. Life is short, I do not want to spend it fixing my primary workstation.
(And I examine postinst scripts in deb files too, but those are usually much more boring)
I almost always inspect these kinds of scripts before running, more out of curiosity than anything, but also so that I know its not going to do that so stupid that even I can see it's stupid. Usually you can just pipe to `cat`, which is super low effort to do.
I've occasionally seen scripts that install some other application where it was not clear that it was a dependency and there was no heads up to the user that this was going to happen: that kind of behavior makes me more distrustful of the author, so there's a useful signal there.
Most scripts like this seem to amount to "configuration defaults + cli completions installation". To that end, I find looking at them useful because it gives me a sense of the expectations of the tooling and hints at where I might find things if I need to go debugging later.
When they are provided by the same entity as the program you wish to run, I don't see how it's significantly more risky to run these scripts than running the application code itself with the active user's permissions. Still, if there was something, by glancing at the script, you have half a chance of seeing it. If there's something there that doesn't make sense, that's both a "proceed with caution" and a learning opportunity.
It’s not really intended as a defense against being owned per se, it’s more about knowing what’s going on and getting an additional signal about the risk profile (not just from maliciousness) of the thing I’m about to run.
That said, I generally pipe to file and cat the file, yes, if only because it somehow feels wrong to download it twice.
I understand, but cating the saved file or printing from pipe to the terminal directly has identical issues. Terminal ANSI escape sequences are interpreted either way.
The alternative is not “install wizard”, it is either “here is a deb file with no post* scripts” or “here is a tar.gz, extract it anywhere and add symlink”
Both of those are vastly safer, because they do not require root privileges, and guarantee to uninstall cleanly.
This. Simply the decision to eschew standard package management already says unpleasant things about their approach to integrating with my environment, or in whatever it is the installation does being reversible using standard tools. "Install wizards" do indeed have exactly the same issue, which is why those are also terrible.
Yes. For starters, it's much harder to suffer disastrous consequences from running truncated command due to terminated network connection when running an installer.
This is easily rectified by script authors by adding a main function invoked at the end. At a certain point, the argument simply becomes "executing arbitrary code is potentially dangerous", which is trivially true, and also not a helpful security posture for anyone who wants to use their computer pragmatically.
Theoretically yes, it is all turing complete at the nd.
Practically no, it is the matter of established conventions.
For example, most interactive installers I have seen prompt for install location, and write minimal amount of files outside of it. The chance that they break your system is pretty low.
At the same time, many |bash scrips like to install system-wide packages, or use pip as root, or other potentially breaking things. And they do not ask you before installing this, and often auto-confirm, too.
Why? I pragmatically use my computer with packages my distro offers me. When I download something from untrusted sources, I do take a quick look at what it is before executing it. It still is dangerous since I'm not able to spend time on verifying every single line of it, but it's up to me to decide whether that's an acceptable risk or not. All perfectly pragmatic.
I'm not a kind of person who's on a crusade against curling into bash, but I never do that myself because it feels dirty. It gives me no advantages and makes me unable to even take a quick look to reassure myself that things at least look like what I'd expect them to be.
I also read scripts from the user repositories on my distro. It’s perfectly pragmatic for us because we understand shell scripts and system administration. It makes me feel better too. But I would never tell someone to avoid installing developer tooling simply because it pipes curl to bash. It’s not pragmatic to tell a new developer they need to learn to be a sysadmin before installing homebrew.
That's a strawman though. Nobody wants to tell that to new developers. This is all about conventions and normalizing bad behaviors. New developers should avoid piping curl to bash and explicitly download files instead because when they eventually will become experienced enough to inspect those files they will start doing it naturally.
Good practices are rarely black and white, they're often there to influence how you think about stuff and what habits you develop.
I love build tools, and work with them professionally, and this tool seems to be getting one thing quite right: it is not injecting itself into the dependency resolution process. Where I see build tooling fall down is where they try to replace the idiomatic dependency modeling tools that exist in each language.
The build logic and CLI experience looks to be well thought out. I really like the native sandboxing support and the intentional exclusion of the calling environment. There's a lot of polish here.
I read Please website (main page, "Getting Started" and FAQ), and still ain't sure about the benefits.
To the maintainers of Please: please make cases and user stories, also comparisons with other build tools.
Many visitors will jump on each "comparison with tool X" where X=each of the current tools I'm familiar with. Only "Bazel, Buck or Pants" are mentioned. Okay no JVM, looks like Please is a better thought out Blaze/Buck.
Please mention CMake and others, like the C/C++-oriented http://floooh.github.io/fips/ (a high-level build system "similar to Rust’s Cargo or Javascript’s NPM, but for C/C++ projects.").
It's hard to tell what the value proposition is here, apart from vague hand-waving about parallelization. The quick start is not enough to get started actually building something.
Changing build systems is hard, and getting buy-in from the team even harder - so you need to demonstrate value up front.
For me it has the feeling of something I would play around with in my free time. But as a drop in replacement I agree, it's a bit of a hard sell for me.
Yup fair enough. Migrating build systems is hard. I've put some work into making migrating from a `go build` project to Please a bit easier. Banzai cloud recently migrated their [Pipeline](https://github.com/banzaicloud/pipeline) project over to use Please but admittedly it was quite a lot of work for them.
Typically (general purpose) build systems make you choose between reproducibility and performance. Please and other related tools aim to change that (however, in my experience they all fail for one reason or another—not because the problem is inherently intractable but rather because these tools are underinvested-in such that they’re infeasible for smaller organizations to operate and larger organizations can afford to roll their own in-house and get the benefits of a tool suited to their specific needs).
I think this “missing middle” is a common pattern in expert tools. The economics are challenging because people quickly move through the skill level where the tools are most useful.
Hey, I maintain please and will be the first one to admit we’re not the best at marketing it. We’ve mostly been focused on getting it up to scratch and only recently have we been trying to publicise it.
With that being said I have put a good amount of work into the QuickStart.
The code labs are designed to get you up and started in an inviting way. If you have any specific feedback about your QuickStart experience I’d love to hear it.
Thanks for all the hard work! I'm very excited to see where this goes.
Have you considered sponsoring, or accepting donations, to work on IDE integration? That's pretty much the only reason I migrated my current employers codebase over to Bazel instead of Please.
Hey! Glad you liked it :D Please is owned by Thought Machine. They’ve recently (8 months) brought me on the project full time. We’re planning on putting together a dedicated IDE team together but no promises.
Bazel has a lot of things "missing" that had to be cut out from the OSS release. Things like grpc rule sets are extremely fragmented right now. It's a bit of A saying "the B team should maintain the rules" and B saying "the A team should maintain the rules". As such almost every single language has inconsistent support for everything: code coverage, protobuf, grpc, external dependency management, IDE integration. Also because there's no stubby, sponge, or other cool internal tools a lot of the benefits from blaze as a system are not externalized.
Also, the docker support in Bazel is pretty "different". It's not bad, but it's not immediately obvious to newcomers how this is meant to work.
Agree with the OP. I went through both the Python and Go quickstarts. They were easy to follow, but I’m also not seeing the value proposition. The build system has different DSLs for different languages so I’m not seeing the advantage of switching from I.e. go install ./... for Go and setuptools for Python? No offense, but you didn’t answer the question: what’s it good for?
In that case, you can stitch both Python and Go builds together in the same interface, and then tie them into further actions (e.g. Docker builds, Kubernetes), again using the same interface. Neither setuptools nor go build are particularly helpful to that goal.
(I'm one of the original developers, but these days do less maintaining of it than tatskaari does)
We wanted something like Blaze (Google's system); at the time we were using Buck which didn't satisfy us (e.g. only one output per genrule, no directories; and it was more or less impossible to write first-class support for a new language without modifying the core system). Subsequently Bazel got released but we still find that a bit lacking in some areas; e.g. the CLI, the JVM reliance and some of Starlark (e.g. it maintains Python 2 syntax compatibility so can't have type annotations).
We'd also tried Gradle previously but that was pretty awful for anything non-Java which made it a non-starter.
To be clear, I'm one of the original implementors, although I imagine that was obvious already...
For most purposes, Bazel dependency on Java is an implementation detail (users don't need to install a JVM), although you might notice it if you need to bootstrap Bazel.
(I co-designed Starlark and I used to work on Bazel)
I’m not sure if the feature was added after you started building Please but Buck does support multiple outputs from a genrule now. It’s kind of hacky though; you write a genrule that produces a directory containing said multiple outputs, and then you can write rules to extract the subcomponents. (I don’t know why I feel compelled to point this out. I worked on Buck for about a year, several years back. )
At first this wasn't a singular clear answer I was hoping for, but the many points being made here make it clear that it's a complex area and there's room for another advanced/high-perf build system to cover a broader range of usage.
My rubric for documentation is "how many clicks to code". I found the installation instructions, and started to look for usage, but didn't find it. Too many clicks. That's where I'd start.
PS - IMO the gold standard for this evaluation method is Sinatra.rb, which is zero clicks to code.
As someone who has had to run and maintain build systems for several (sadly far too) large projects (5MLoc+), I want a product that sells me on how easy it is to create _correct_ builds, and how impossible it is to create incorrect ones.
So much wasted time goes into diagnosis of incorrect results from incremental builds that most people who implement CI systems never use incremental builds, and always build from scratch. Developers are far too used to having to do things like 'make clean' because their build didn't work quite right. Efficiencies are then gained by doing things like ccache/Gradle build cache, which trace their dependencies better than most naively written build systems do.
Unfortunately, reality is messier than this. For example, Please currently has escape hatches that can be used that nullify these guarantees. What I have found is that if such a mechanism exists it will eventually be abused.
My perspective here is specifically from that of a maintainer of large builds where the code base is always under active development. Murphy's law becomes your enemy at scale.
This is not a criticism of Please - just an observation that as an industry we are not there yet. But, I'm happy to see things moving in the right direction.
I haven't had many issues with non-determinism when using Bazel/Blaze? Especially when you use remote execution or sandboxing it's pretty hard to avoid being hermetic.
This is admittedly contrived, but sadly not unrealistic.
Again, I am not picking on Bazel here. Correct builds are difficult enough to get right even when everyone involved wants to keep the build correct! But, the reality is that a lot of real-world build conditions can be downright hostile. (did you know that ext4 vs xfs can change the order JARs in the same directory are loaded off the classpath? I wish I didn't!)
Yeah fair point. At Google a lot of these non-deterministic rules are either banned through "date" just not existing on remote execution machines, or through determinism tests that ensure rules behave. Maybe in real-life scenarios it's not as easy as I claim, as you suggest.
A few sample build configs of varying complexity would be nice to see at a glance to understand what you are getting into without digging through the weeds.
You hopefully have tried out many other build systems by now, been frusted with all of them, and only then made your own. And with all that experience with the status quo, marketing should be easy.
If you haven't done that, this project is a net-negative on the world, because the proliferation of build systems exacerbates Conway's law and balkanizes our software commons.
Marketing can be hard not because you don't have something worth marketing, but because marketing is a skill-set largely orthogonal to software development.
I'm talking about "marketting" to other software developers, not to non-software developers. Communication with people like oneself is a core skill, not a specialized trade.
This isn't a paid product, after all.
(I'm also a traditionalist that thinks writing good prose and writing good programs are related skills.)
That's only true if you define software development in a pretty particular way.
For me, writing code is a way of solving problems for people. That involves listening to them to see how I can help. Then collaborating with them to iteratively make something that fits their needs. Which includes helping them adopt it and get productive with it. To me, all of that is part of software development.
And that's most of the "marketing" that's needed for a free, open-source tool. When that's lacking, I have to wonder to what extent the tool was actually made to solve somebody's problems. As opposed to being created because the developer was excited to build a thing as a technical exercise. Either path is fine, of course, but it's a mistake to confuse the two.
To me, writing code is very much the same way - it is a human endeavor. But, over the years, I have learned that just not all people are the same way, and that's okay, too.
Growing up, one of the jobs I took was as a system's administrator at the same company my father worked at. It was a small company, maybe six people or so, doing contract work for aerospace. One of the older men who worked there held his Ph. D. in aerospace engineering. He was kind, warm and brilliant... but he struggled with things like sending coherent emails to clients. But that was okay - other people were able to take that on.
That's totally fair. But...going on a hunch, such a person would also make a crude 90s-style website.
Whoever made this clearly knows about slick presentation --- they've got all those colors and shapes pizzas --- so they either capable with slick communications, or work with someone who is. My patience for lack of explanation is therefore a lot thinner.
Do you realize the “sell something to people” industry has been at least as big as the tech industry for many decades and is the one growing faster and faster?
Also marketing has historically been a problem for people who make stuff.
Also, better have the Balkans of software than the USSR of mega corps.
I am not an author, but was around when we decided to create Please in 2015.
When we created Please we were currently using Buck from Facebook. As has been covered elsewhere at the time it had some limitations which meant that it was getting more and more expensive to workaround (no multiple outs from rules so doing things like sourcemaps was hard etc). We had previously migrated from a set of disparate build tools (Gradle, Gulp etc) to Buck after evaluating it against Pants (we actually trialled our repo with both).
We all had experience with Blaze (Bazel did not exist in 2014/2015) and wanted to get closer to the experience we had in Google, hence the final decision to build our own. At the time we rolled it out we had full Buck compatibility (and actually most of our engineers did not notice initially). Once we were happy it worked, we migrated in full to Please.
Overall it allowed us to provide much better tooling for our developers (queries on the build graph for smaller CI/CD footprints, coverage in all languages, and for rules that can output multiple languages (like protocol buffers) we only build variants in the requested languages rather than all languages).
If Bazel had been open-sourced and easy to extend at the time we would certainly have looked to adopt and improve it (as we did with Buck at the time). But by the time they open-sourced it the two system had diverged in their approach and use cases.
I (obviously somewhat biased) think having multiple principled build systems which prioritise the needs of different communities is good for the ecosystem overall (and allows the sharing of good ideas such as the Remote Execution Interface).
Please build is really fantastic and I loved using it. If it had better ide integrations I'd recommend it over bazel for oss work. It's extremely fast and we'll documented and the community seems very welcoming.
Glad it's getting some discussion on HN. Can't wait to hear other's experiences.
Plz is great for c++, and has a better workflow (IMO) vs (e.g) Cmake. The skylark-like syntax is easier to reason through than cmake generative expressions. Also the use of “plz run” vs “cmake -g; make; run”
cmake generator only needs to be run once per project. After that you use make or ninja directly, and it regenerates the build files if necessary. I can't say it's a strong argument against cmake.
This added user complexity is exactly the argument for plz, though.
One of the ideas touted is (to my mind) tight integration, in that plz knows how to handle everything. Whereas cmake just makes project files and leaves it at that.
I've used it for Python, C, and some Go. The huge benifit of bazel-like build systems isn't that it's better then a languages build system, it's that it's the best every language build system.
It's consistent between all languages and supports a lot of functionality. For example the proto_library and grpc_library rules make building microservices extremely easy. You can build an entire service in like 3 lines of boilerplate.
It also makes it easy to do cross-language deps. For example a c library included into a Java library that uses native bindings to wrap it.
I can imagine a project that successfully replicates the entire google developer environment (distributed build system w/ caching, monorepo with presubmit checks, code review, testing, etc) would be successful since ex-googlers would be likely to advocate for it inside their own organizations and most orgs don't have the engineering time to build all this themselves.
Without this tooling large organizations tend to silo which creates high engineering costs (every project starts from scratch, no shared frameworks/libraries, no standardizing around best practices, etc).
There's an interplay between them. Communication always has a cost; there are certainly things you can do to the constant factors, but in a 100-person engineering organization with 20 teams, it's fundamentally going to be much easier for an engineer on one team to write a build system (or pre-merge CI flow, or whatever) that meets the needs of their four teammates working on the same code than one that meets the needs of all 20 teams.
The advantage of a running, pre-existing system (and the theoretical advantage of an off-the-shelf system like this that you can supposedly turn into a running system) is that it's been built to satisfy most of the needs of a wide variety of developers working on all sorts of things, and so each engineer on each of those teams that says "All right, I'll spend some time thinking about how we build and release code" is incentivized to start from the common starting point, even if they have to customize it. That means that if two or three teams end up working on similar-enough projects down the line, the shared tooling can actually support communication between the teams. Without the shared tooling, they can communicate all they want, but it won't be rational for any team to abandon its existing tooling. (And of course you need the communication too - both are required.)
I hear you. I do. How do you get 20 teams working on disparate codebases, and probably languages, to share their knowledge across teams? How does a team working on Java benefit from something someone found in Django from another side of the company? Organizations need standards. Developers require it. We want the thing we are programming against to not change right? We want our codebase to build when we say build. All tests green. Happy. Productive. Engineering leadership needs to organize standards around engineering culture and tooling, it could be championed by a team that did it, or by research. It doesn't matter. What matters is that leadership facilitates communication into what is working, what isn't, why, and what can we do about it?
At the micro-level, tools help, no doubt about it. At an org level, communication helps (documentation is a form...). Even if using different tooling, concepts and strategies are the same. This isn't to take away from the argument that this tool the OP posted is pretty awesome.
If you want change, be the change. Evangelize the tool through your org, not just your team. If your org is silo'd, break those barriers down with a friendly lunch-n-learn or lightning talks session. We tout empathy for our customers, how about empathy for our coworkers too? Others may have the same pain...
A surprisingly hard sell even if your organization has a supermajority of ex-Googlers. There's a really strong meme out there that if you are smaller than Google, which is almost universally true, then you shouldn't adopt their practices even when they are zero-cost and obviously superior.
For me the "obvious" part of "obviously superior" is worth interrogating. Googlers have a very particular way of thinking of things. Which works well enough for Google, I guess. But there are a lot of ways to to good work, and not all of them are compatible.
Sure, I think thought needs to be put into any development decision. I think a new project should carefully consider, for at least several hours of research and discussion, what revision control system to use, instead of just using git without consideration of the consequences.
There are already book-length arguments that I like to refer to for specific topics, such as trunk-based development.
I agree. A big missing link at my organization is a VCS that can make it easier to checkout and operate on only a subset of a large (many GBs) monorepo.
How easy is it to "port" BUILD files between the various open implementations (Bazel, Buck, Pants, Please)? If you have a project someone else wrote with the intention of using it in e.g. Bazel, can you integrate it into your Please build system, or would you need to treat it as an opaque build process (same as if you were shelling out to ./configure && make)?
As a concrete example, one frustration at $DAYJOB is that building certain Google-released OSS is hard to integrate with our existing build system because Bazel is pretty opinionated about compiler-provided headers, output paths, etc. We're calling Bazel as a subprocess during our build and making various files available to it to make our compiler build visible to it, and then we pack up the results. I'm pretty sure if we somehow adopted Bazel for our entire build this would be much better - would it also be better if we adopted one of those other systems?
(Maybe one way of asking this is, are any of the corporate sponsors of Buck, Pants, or Please building third-party Bazel code from source? Or vice versa?)
A broader question: would it be worth defining a compatible subset of BUILD file syntax and library calls (i.e., not just Starlark but also the rules themselves for common things like building C libraries or JARs)?
Are we in the build-system equivalent of the vendor-specific C / C++ / UNIX implementations, and will a cross-vendor standardization effort emerge one day?
For the first part of your question - it is not 1:1. There is some limited capacity for bazel integration, but massaging is needed (IME) for protos, nvcc rules & so on
I've only seriously worked with bazel, but it seems pants and bazel have at least enough compatibility for twitter to migrate to bazel. They gave a talk[0] at bazelcon this year about the effort.
I used to work on Bazel and Starlark. In the past, I talked with engineers working on other build systems to aim for some compatibility. The was some interest (even Microsoft people were interested to support Starlark rules in their tool), but it didn't work.
On the BUILD file level, Buck is using the Starlark. Pants and Please are close enough that some tooling (e.g. buildifier) can be shared. But that's about it.
Admitting ignorance here, I have never intentionally used a build system like this. My normal process for e.g. Python deployment is to write a Dockerfile that installs the dependencies, installs packages, and then copies my user code in. Then I use Pulumi to upload that image to a Docker repository & deploy it to a k8s instance.
What am I missing by doing that? This looks really slick, but I'm not sure how, why, or where to use it.
Speaking from my own limited experience, plz works really well for compiled languages, and total-build approaches (everything from source). I am not sure how much of a multiplier it is for python, apart from finer-grained dependency control and good sandboxing
Those kinds of systems are for building stuff, they assume and have multiple build steps with complex dependency relations (like autogen header -> Compile -> link -> test), and that each step takes dozens of seconds and maybe even minutes.
In your case, it seems you only have one non-trivial step: python dep install. So this system will be quite overkill for you.
Please excels in a mono-repo situation. We built Please because we struggled to get Buck to handle Python, Javasript, and Java while having protobuf code generation for all these languages.
Fast forward 5 years, now we're using Please to build all our code, generate hashes for docker images, template those hashes into our k8s .yamls, generate the documentation website from the docstrings in the proto files etc. etc.
With a language specific build system, you would have a lot of trouble handling things like this.
They make some lofty claims about Python packaging, has anyone used this tool in a reasonably high stakes environment for Python? How was the experience?
"Please supports Linux, macOS and FreeBSD at the moment"
Let me know when I can actually use this on any machine I have to work on, instead of going "fuck you, Windows users". Writing cross-platform build tooling isn't rocket science, it's a choice. And the choice made here is stupidly disappointing in near as makes no difference 2021.
Hi, one of the implementors here... Yes, WSL is an option, and I believe Please works just fine in it. Right now we don't have any CI etc set up so we can't stand behind it and say "this works", but it works in WSL as most Linux things do.
It's open source, if you want it, you can add support for it. Which is a lot more than what you can do for a lot of other development tools that only run on Windows and have lackluster or no support on Linux.
Also, they might just not have Windows developers or a use case for running the tool on Windows, so even if the cost of adding Windows support was zero, they'd have to go out of their way to test it. A lot of open source code might run just fine on Cygwin, but most people choose not to support Cygwin because damn near no-one uses it anyway.
It’s not just a choice. It’s also the kind of tooling the people developing the system have at their disposal.
For example, if all I have is a MacBook running macOS and Docker, how can I reasonably expect to port this to Windows? I haven’t owned a Windows machine for 10 years.
I am the author of Buildbarn, a build cluster implementation for Bazel/Pants/.... Buildbarn also doesn’t support Windows. If someone sent me high quality patches for Windows, I’d merge them for sure, but nobody has. Sorry you are ‘stupidly disappointed’ in me.
I make use of openat() and friends to safely populate and interact with build environments on disk.
I perform scanning of process tables to kill lingering processes. Go has no functions for that in its standard library. There are some third party libraries for that, but they are all of pretty poor quality. They are slow.
Have used it in a go mono-repo. It’s own convention on packages and bad integration with IDE makes it a hard sell for me. I mean I just paid for Goland IDE why would I waste my time integrating with something that might or might not work for others. It’s a very hard sell.
Go tools in general have poor integration with build systems other than "go build". One trick I find useful is to use Go modules (and therefore goimports, not goreturns) and structure your build so that it looks like a standard Go module build, even though it's really being built by some other system. In other words, it's a mix of making the build system conform to Go norms and making your IDE adapt to the remaining differences.
I recently added a config option so plz test will run tests in the package directory rather than the repo root (just like go test). Should make things a little easier.
Hey, Please maintainer here! One of the big differences to the other Blaze like build systems is Please actually doesn't treat any of the built in languages any differently to other languages. They are implemented entirely in the build language which is totally open to you as a consumer of Please.
Another big difference is Please is written in Go so there's no dependency on VMs or runtimes.
I come from a sofware world not typical for HN. I am a (very) low level software and FPGA guy. Despite my best efforts, I don't understand: What do any of these tools do that Make does not? Are they faster and easier to use? Do they work better?
Make build configurations can be difficult to understand for newcomers in the industry. If the goal is to obscure the code, by all means continue using the older tools.
If the goal is continued maintenance, then encouraging new engineers to explore and read the codebase with tools they can comprehend is critical.
disclaimer: have not used these particular tools, but the domain is nice and polite
I have use all of these tools and your take isn't accurate at all. Make doesn't obfuscate anything more than Plz or CMake or whatever.
Here are some real reasons why Make isn't the end-all:
- Make doesn't allow platform selection in a nice way.
- Make doesn't work on Windows natively (no, NMake doesn't count).
- Recursive make doesn't work well at all.
- Make doesn't track byproducts or deleted artifacts.
- Make doesn't have auto-clean.
- Make doesn't facilitate out-of-source builds.
- Make itself isn't a scripting language (arguably good/bad).
- Make doesn't facilitate compiler selection cross-platform, so things like MSVC are nearly impossible to integrate nicely without the use of the developer tools prompt.
- Make can't (elegantly) handle a number of rule cases, such as single-input multiple-output rules (it can, but it's a hack).
Not sure why you think Make is unapproachable. That's like saying shell scripting is unapproachable. No developer worth a damn is going to avoid shell scripting, and as long as they understand "this file turns into this file using this command" then Make makes sense.
As with all tools, Make and Autotools and CMake and probably Plz will be misused by developers that think they're being clever. In actuality, they make things worse at best and unusable/unmaintainable at worst.
As a build systems designer, I've evaluated Plz personally and found it to be neat but not suitable for most of the problems I think build systems should solve.
The problem with Make is that it’s never just Make. It’s also Autoconf, Automake, m4, and god-knows-what-else. If Make is so great, why are people resorting to generating the Makefiles? There’s clearly some kind of deep UX problem there.
I think it’s 1. people find makefiles confusing, and if someone’s introduction to the concept is a large project they may never pick up the basics and 2. Make’s fundamental assumptions: that each command creates a single, sensible artifact that is the input for the next phase, maps poorly to the dynamic build processes of browser and phone ecosystems, which are themselves barely controlled explosions of complexity
No unfortunately. The FPGA "build" realm is mostly a clusterfuck of proprietary tooling. It all has poor to nonexistent integration with, well, anything else. There are some things that are okay for simulation like Cocotb, but for actually building hardware FPGA designs, you are locked into the vendor's tools. The only exception is if you use a small, slow, and outdated FPGA, not suitable for large designs used in industry. For these reasons, there isn't much of an open source community making things like Please and Bazel specific to FPGAs. You can set it up yourself, but a packaged product that just works out of the box doesn't exist to my knowledge.
I work in a similar field and this is accurate for us too, except our tools do at least have some integration with Make (you can generate makefiles from the proprietary project files, although it's not useful to us and we don't use it). Like most (I assume), we've built our own build language/system on top of Make. It works but only in a few well-defined ways that modern software shops would probably baulk at. We don't support parallel builds, no caching, only coarse grained incremental builds (at a subproject/library level) and who knows what else that others benefit from.
I've only looked at modern tools briefly but everytime I come away realising we'd have to re-write everything. There is no out-of-the-box support for our tools/language and it would be a lot of work to learn someone else's DSL, re-implement everything and then potentially discover a load of ways it's broken for our workflow. Personally I think it'd be worth it in the long run but it's a really hard sell.
Makefiles operate at a low level of abstraction. The Makefile abstraction is "run this command when the output files don't exist or are older than the input files." You manually specify every tool to be run and all of its arguments.
These tools operate at a higher level of abstraction. The BUILD file abstraction is something like "I am declaring a C library with these sources and headers." The build system then turns that into a set of commands to run. This higher-level abstraction provides a bunch of compelling features:
- Reliability: it's easy to write Makefiles that are fragile, for example one that works when run on a single CPU but fails when run with -j, because of undeclared dependencies. When the tool is in charge, it has more control and can sandbox each build step so that undeclared dependencies simply won't be available in the sandbox.
- Reproducibility: similar to the previous point, it's easy to write a Makefile that produces incorrect results with an incremental rebuild. For example, unless you implement your own header scanning (or manually list your header dependencies) "make" won't know to rebuild a source file when a header changes. These tools can ensure that builds are reproducible and incremental rebuilds are always correct.
Besides that, working at a higher level generally means writing less code to accomplish the same task, and the code you do write is closer to expressing your actual intent compared with a Makefile.
> The Makefile abstraction is "run this command when the output files don't exist or are older than the input files." You manually specify every tool to be run and all of its arguments.
> The BUILD file abstraction is something like "I am declaring a C library with these sources and headers."
This is wrong. Even ninja has generic rules. Here's an example of a minimal makefile:
> When the tool is in charge, it has more control and can sandbox each build step so that undeclared dependencies simply won't be available in the sandbox.
There's no reason this shouldn't be possible with make; it just hasn't been implemented so. Do bazel/buck/please actually do this? As far as I know tup is the only tool that actually verifies inputs/outputs of rules, and it needs FUSE to do so.
> For example, unless you implement your own header scanning (or manually list your header dependencies) "make" won't know to rebuild a source file when a header changes.
Your example does not contradict what I wrote. You manually specified the tool to be run ($CC) and all of the arguments to that tool.
It's true that there is a level of indirection through the $CC variable, but you're still operating at the level of specifying a tool's command-line.
> There's no reason this shouldn't be possible with make; it just hasn't been implemented so.
Make is 44 years old. If it were an easy extension to the existing paradigm, there has been ample time to implement such an extension.
> Do bazel/buck/please actually do this? As far as I know tup is the only tool that actually verifies inputs/outputs of rules, and it needs FUSE to do so.
Bazel certainly has some amount of sandboxing, though I don't think it's quite as complete as what is available internally at Google with Blaze. I haven't used Buck or Please, so I can't speak for them.
> True, it's a bit of a footgun, but by no means difficult.
Well footguns aren't great. :) As just one example, any header that is conditionally included (behind an #ifdef) could cause this cache to be invalidated when CFLAGS change, but Make has no idea of this.
Notably, linklibrary can be defined according the platform, or according to dynamically set variables, giving you the same level of flexibility as make.
>> There's no reason [sandboxing] shouldn't be possible with make; it just hasn't been implemented so.
> Make is 44 years old. If it were an easy extension to the existing paradigm, there has been ample time to implement such an extension.
‘Nobody's done it yet, ergo it's not possible or easy’ is not a valid argument.
> just one example, any header that is conditionally included (behind an #ifdef) could cause this cache to be invalidated when CFLAGS change, but Make has no idea of this.
Then you specify that the source files depend on the makefile.
> Then you specify that the source files depend on the makefile.
Then you gratuitously rebuild everything whenever the Makefile changes, even if you only changed a comment.
Also this scheme is incorrect if the Makefile was written to allow command-line overrides of variables like CFLAGS, as many Makefiles do.
But these are just details. The larger point is this. The language of Bazel is defined such that builds are automatically correct and reproducible.
While it's true that Make has some facilities like "call" that support some amount of abstraction, it is up to you to ensure the correctness. If you get it wrong, your builds are back to being non-reproducible.
It's like the difference between programming in a memory safe vs a memory unsafe language. Sure, every thing you can do in the memory safe language can be done in the unsafe language. But the unsafe language has far more footguns and requires more diligence from the programmer to get reasonable results.
> ‘Nobody's done it yet, ergo it's not possible or easy’ is not a valid argument.
Ultimately, make and co are text oriented, while bazel and co are object oriented. Hacking object oriented capabilities into a text-oriented language isn't particularly fruitful or ergonomic.
Bazel and make are both text-based languages for describing symbolic, abstract structures. Just like pretty much every other programming language.
Bazel and make both use the same abstract structure, just like pretty much every other build system: a directed graph of build directives and dependencies.
Fundamentally, bazel and make treat "targets" differently. A make target is an invokable thing. That's about the extent of it. You have a dag of invokables, and invoking one will cause you to invoke all of its dependencies (usually, other people have discussed the limitations of make's caching already).
But let's look at how a rule is implemented in bazel[0]. Here's a rule "implementation" for a simple executable rule[1]:
def _impl(ctx):
# The list of arguments we pass to the script.
args = [ctx.outputs.out.path] + [f.path for f in ctx.files.chunks]
# Action to call the script.
# actions.run will call "executable" with
# "arguments", saving the result to "output"
# access to files not listed in "inputs" will
# cause errors.
ctx.actions.run(
inputs = ctx.files.chunks,
outputs = [ctx.outputs.out],
arguments = args,
progress_message = "Merging into %s" % ctx.outputs.out.short_path,
executable = ctx.executable.merge_tool,
)
concat = rule(
implementation = _impl,
attrs = {
"chunks": attr.label_list(allow_files = True),
"out": attr.output(mandatory = True),
"merge_tool": attr.label(
executable = True,
cfg = "exec",
allow_files = True,
default = Label("//actions_run:merge"),
),
},
)
This is, admittedly, not easy to follow at first glance. Concat defines a "rule" (just like cc_binary) that takes three arguments: "chunks", "out", and "merge_tool" (and "name", because every target needs a name).
Targets of this form have metadata, they have input and output files that are known and can be queried as part of the dag. Other types of rules can be tagged as test or executable, so that `blaze test` and `blaze run` can autodiscover test and executable targets. This metadata can also be used by other rules[2], so that a lot of static analysis can be done as a part of the dag creation, without even building the binary. To give an example, a rule like
can be built and implemented natively within bazel by analyzing the dependency graph, so this test could actually run and fail before any code is compiled (in practice there are lots of more useful, although less straightforward to explain, uses for this kind of feature).
Potentially, one could create shadow rules that do all of these things, but you'd need to do very, very silly things like, off the top of my head, creating a shadow filesystem that keeps a file-per-make-target that can be used to query for dependency information (make suggests something similar for per-file dependencies[3], but bazel allows for much more complex querying). That's what I mean by "object-oriented". Targets in bazel and similar are more than just an executable statement with file dependencies. They're complex, user-defined structs.
This object-oriented nature is also what allows querying (blaze query/cquery/aquery), which are often quite useful for various sort of things like dead or unusued code detection or refactoring (you can reverse dependency query a library that defines an API, see all direct users and then be sure that they have all migrated to a new version). My personal favorite from some work I did over the past year or so was is `query --output=build record_rule_instantiation_callstack`, which provides a stacktrace of any intermediate startlark macros. Very useful when tracking down macros that conditionally set flags...
FUSE, strace, and namespacing were all the mechanisms I found. Bazel uses separate wrapper program which you could reuse in other build systems, so there is no fundamental problem with adding a "hermetic builds" feature to other build systems like Meson or Cmake.
Please takes sandboxing a bit further using kernel name-spacing to isolate builds and tests. It's an opt-in feature but you can bind to port 8080 in your tests and run them in parallel if you do ;)
Make is only good at one thing: calculating how to execute a dependency graph to produce a specific target.
Everything else that is needed for a modern build system is lacking:
- expressive scripting language
- platform-related configuration
- feature-related configuration
- handling hierarchical projects
- properly handling rebuilds when the Makefile itself has been changed
- dealing with dynamically generated parts of the dependency graph
- handling third-party dependencies
- build reproducibility
- etc... the list is very long
This is why people these days mostly take the path of generating Makefiles, because make is only good at one thing: executing a statically defined dependency graph.
Make is a programming language: it is only good at executing that language, and that language kind of sucks, but saying Make is somehow only good at executing static dependency graphs is a fundamental misunderstanding of Make (but AFAIK might be a good description of say, ninja). All of the Make scripts I have written for the past 15 years have dynamically generated configuration and autodetected files and calculated version numbers and essentially done all of this work that somehow you think Make can't do.
- Cannot handle multiple outputs for a single rule
- Does not rebuild when flags change
- Make rules may contain implicit dependencies
- Slow for large codebases
- Does not understand how to build for multiple platforms, sucks for cross-compiling
- Recursive make sucks (job control does not work across recursive invocation boundaries)
- You must create output directories yourself
- You must create your own rule to clean
This adds up to a fragile & slow build system, where you have to do a full rebuild to have a reasonable level of assurance that your build is correct—incremental builds aren’t safe.
There have been a few attempts to make a “better make” over the years like Jam, SCons, Ninja, Tup, etc. each with their own pros and cons.
The new generation of build tools—Bazel, Buck, Pants, and Please are all an attempt to redesign build systems so that the design is resistant to the flaws that plague Make build systems. You can use a shared cache (shared between different users) fairly easily, and you have reasonable assurance than incremental builds are identical to full builds (so your CI pipeline can move a lot faster, developers almost never have to "make clean" when they change things, etc).
Personally I’m working on a project right now that uses Bazel (which is similar to Please) and is for an embedded system. It’s been a great experience, and I can pass a flag to Bazel to tell it to build for the embedded target using the cross compiler or for the native host--that makes it easy to share code between tools and the target system, and I can do things like write tests that run on both the target & host. Anyone who does any cross-compiling is missing out if they are using Make—but, do note that setting up a cross-compiling toolchain in Bazel isn’t exactly a cakewalk.
The documentation is not so great. What I’m doing is enabling platforms (https://docs.bazel.build/versions/master/platforms-intro.htm...), defining the OS and CPU target for my system, copying the configuration out of Bazel’s “local_config_cc”, and modifying it to fit my use case. This didn’t take me very long, but it’s also not the first time I’ve done it.
Make does support multiple outputs, though the syntax sucks. Most of what you are annoyed with though is like being annoyed at C for the same reasons: Make is a programming language with a built-in dependency mechanism, and as such you can use it to build whatever you want... now, does it already come with whatever you want? No. I can appreciate wanting something which does. But such systems usually then give you what they want. I don't want my programming language to solve half of these things you want, and yet somehow I have built things on top of Make that solve even the problem of building for arbitrary embedded cross compile targets. (And hell: of cross compile toolchains is what you care most about, the king of that space is autotools, which not only correctly supports compiling to everything always every time out of the box, but somehow manages to make it possible with nothing more than you passing some toolchain configuration as command line arguments to the generated configure script... and one might note that it generates Make, though without taking much advantage of what makes Make great.)
> Make does support multiple outputs, though the syntax sucks.
No, it doesn’t. There’s NO syntax for multiple outputs. If you can show me what the syntax is and prove me wrong, I’d love to see it. At best, there are workarounds for the lack of multiple output support, with various tradeoffs.
> Make is a programming language with a built-in dependency mechanism, and as such you can use it to build whatever you want...
Make is NOT a programming language. End of story. You can… kind of… build things with it, given that variables in Make can be used like functions, but it’s goddamn horrible and you shouldn’t do it because nobody will want to use it and nobody will maintain it.
At most, you can build something on top of Make, but you’re still facing the limitations of Make and working around them. If you are interested in building something on top of a low-level build system, you want Ninja instead of Make, because Ninja is good at that. Make isn’t.
Make is, at best, something you would want to use for a small C program where all the files are in one directory. Once you grow past that use case, my rule is that you shouldn't be using Make any more, because there are just too many things that Make does wrong.
Make is successful because eh, it’s good enough for small things, you can suffer through the pain if you need to use it for big things, and it was the first tool to solve this problem. We have better tools now. We had better have better tools now! Make is in its fifth decade… if we didn’t improve on Make in that many years, that would be a damning indictment of the entire field.
The big advantage of make to me is I understand it and can figure out what happens when things go wrong. When something doesn't work the way I want with cmake or autotools (I haven't used Bazel etc.), I have to randomly start googling things. Sometimes I literally resort to editing the generated cmake Makefiles because I have no idea how to tell cmake what I want it to do...
GNU Make does support multiple outputs, but the feature is very new (it's in the latest release that came out earlier this year), so if you didn't happen to catch that release announcement you probably missed it. The support is called 'grouped targets', documented in the second half of this page: https://www.gnu.org/software/make/manual/html_node/Multiple-... -- the syntax has &: in the rule line.
(One point you don't mention in your list of reasons why Make is successful is that it's reliably available everywhere. For projects that ship as source for others to build, that matters, and it delays uptake of fancy new build systems in that segment.)
I think one good way to think about it is all of these build systems create Make environments. They create the makefiles and have all the other tools that go with it.
So at the end of the day you end up with an executable artifact, but hopefully you did a lot less work to get there.
Only garbage like autotools or CMake generates Makefiles, all remotely sane build systems (including the various bazel/blaze derivatives) do away with make, which even sucks as a low level abstraction.
I think there's a few advantages of bazel-like systems:
1. Reproducible/Hermetic: your builds cannot, say, access the network during a step or produce changing binaries for the same input. This makes sure that once you've downloaded your deps locally everything builds correctly from then on.
2. Caching: Because everything is reproducible in bazel-likes everything is cachable. Dependencies, builds, and even tests! If you've worked with Makefiles you've likely run `make clean; make tests` or something similar. I've never needed to do `bazel clean` to make sure things were working right.
3. Visibility: you don't only control who can see what in your source code. Different dependencies can be marked as private/protected/public to control who can import them. This is a huge boon to large monorepos.
4. Everything is uniform: code generation, compilation, etc is all described in "rules" and "toolchains" and can "easily" expanded to other languages. The community manages a Rust and Golang rules set for bazel itself and they're better then the Google rule sets for Java (only "officially" supported rule set right now) in some areas.
So if you have a lot of code/internal libraries/code generation, what to write a LOT of unit tests and cache their results, and write code in multiple languages bazel is probably for you.
You can also use tools like BuildBuddy [0] to share a cache with your team and extract build performance metrics to identify areas you can improve performance in.
> If you're familiar with Blaze / Bazel, Buck or Pants you will probably find Please very familiar
Yes, so why would I use Please over any of them? I've spent close to 10min reading and have no idea why this exists or why anyone would use it. It looks like Bazel with a different config format, in which case why wouldn't one just use Bazel?
And I would have considered that to be large 20 years ago, because it would have filled an entire CD-ROM disc, and it would have taken over five hours to download.
However, I still have somewhere over 100 GB free on my seven-year-old laptop, and the download will take me ~40 seconds.
But even if the 600MB were a problem, Bazel doesn’t need an external version of the JVM. It bundles its own, and fits the whole thing in under 50MB. Some people are under the mistaken impression that you have to install the JVM first and then install Bazel, which is simply not true.
> In my experience there is a strong correlation between executable size and how hard it is to unravel any issue that arises in it.
Bazel bundles the JRE, that’s why it’s 45 MB.
The correlation is kinda beside the point, though, because my experience with Bazel is that it’s easier to unravel issues with Bazel than unravel issues with Make, and Make is much smaller.
Good to know that you live in a privileged place and access a fast internet connection. Most of the places don't have that kind of bandwidth [1] to download 600 MB of file in 40 seconds. Also, Bazel needs a huge amount of RAM because of JVM and runs a daemon process in the background to speed up the build duration. Running a background JVM daemon process is a NO for me and Bazel wastes system resources.
> Most of the place don't have that kind bandwidth…
Most people don’t have the kind of bandwidth it takes to download a 50 MB binary? Are we still talking about Bazel here, because that’s the size of the download. What about tools like compilers? You have to download those, too.
> Running a background JVM daemon process is a NO for me and Bazel wastes system resources.
What system resources does it actually use up? A half-gig of RAM? This kind of optimization is penny-wise and pound-foolish. You are spending your precious time and energy worrying about a resource whose marginal cost to you is about the same as a cup of coffee.
I run Bazel on a terribly obsolete, seven-year-old laptop which is my daily driver. Sometimes Chrome will choke on a website, sometimes I'm waiting ages for Homebrew to update or for NPM to download some packages, but Bazel is not a problem.
> You are spending your precious time and energy worrying about a resource whose marginal cost to you is about the same as a cup of coffee.
I don't install tons of random garbage because I want to know and understand what is running on the hosts that I maintain.
I need to be able to debug the stuff I run, and complexity makes it difficult.
Tenths of MB of stuff is complexity. Tenths of build time or run time dependencies is complexity. A compiled tool when a script would have been sufficient is also complexity.
> I need to be able to debug the stuff I run, and complexity makes it difficult.
Bazel is way easier to debug than Make. Debugging a decent-sized build system made with Make is just an exercise in suffering.
I am… honestly… no longer interested in understanding the entire software stack. I understand my time on this earth is limited and want to spend it doing other things. With Bazel, I am spending less time fucking around with build systems and more time doing the stuff I care about.
> Tenths of MB of stuff is complexity. Tenths of build time or run time dependencies is complexity. A compiled tool when a script would have been sufficient is also complexity.
As someone who still writes C, I can understand the joy that people feel when you make some cool program and it’s measured in KB, not MB. However, what you’re describing strikes me as fetishistic.
That 600 MB is mostly standard library, and reusing it makes every app smaller. Statically linking everything creates size and version skew problems that I hoped we had left behind in the 1980s.
I also was trying to figure out why I would use this over Bazel. Then I remember reading a story yesterday about how ex-googlers miss their tooling when they leave. Their CTO is an ex-googler, so maybe this is the reason.
But Bazel is developed by Google itself right now. Presumably if you missed Google tooling you'd rather use Bazel, no?
I'm not suggesting Bazel is perfect: I think e.g. Starlark's insistence on being a separate language does more harm than good. (FWIW I also think the JVM objection is a little silly.) But I am saying preferring the Google tooling would ostensibly mean you like Bazel a lot already :)
I've read Bazel without the rest of Google tooling (giant monorepo of everything, distributed build farms, etc) the experience is nowhere near the same.
Starlark is _almost_, but not quite_ Python, and Python 2 at that. You could do everything Starlark does in actual Python and get stuff like static type checking for free.
I think the term DSL is overloaded here? Consider all the Lisp and Ruby stuff that's definitely DSL but doesn't need most of a language implementation.
I've only used Bazel in anger (in a medium-small sized project); I've found it really frustrating because it effectively has two separate languages (inside build files and in custom rules), even though it's both Starlark. It's especially terrible for doing things that Bazel didn't already know about (i.e. running shell commands to generate things, even if you could tell Bazel all the inputs and outputs).
We wanted to switch away from Bazel because of the mismatch; it looked like Pants and Buck had the same issues. At the time Please didn't have sensible documentation so it was skipped. From what they have now it seemed like it's more reasonable in this respect, but I don't actually know yet; perhaps somebody who knows can chime in?
Genrules are okay for commands, but most things I want to do end up being tiny shell scripts, which isn't a great fit.
I mean full on custom rules, exactly because of the inability for genrule to span more than one command (so we end up having to generate tiny shell scripts and running them).
I've also had issues debugging things any time I've tried to use genrule; the error messages tend to be unhelpful in my experience (though that's been a while and I don't recall the details). Things that look like they'd do what I want would just not work in unhelpful ways, and interrogating the system usually didn't work until there were no errors (at which point I wouldn't need to interrogate it anymore…)
Please predates Bazel IIRC. It just didn’t gain as much traction. Please seems considerably easier to operate, but it doesn’t enjoy Bazel’s rigorous usage (although I’ve only encountered bugs when trying to use Bazel to build Python 3 projects).
I hate bazel with passion, I have to use it at work on really large project, and the UX is so wrong... built for google by google and my CTO thinks we can be google if we use it...
What specifically about the UX is annoying to you? I'd be interested in hearing what could be done to improve the experience. Is it just writing BUILD files or is it the cli interface?
From their FAQ:
>Why use Please instead of Bazel, Buck or Pants?
>All four of these systems are quite closely related in the scheme of things, being inspired by (or in Bazel's case, a direct open sourcing of) Google's Blaze.
>Several of us had worked at Google and used Blaze extensively there; we were excited about it being open sourced as Bazel but by then we were already using Please internally. It's a great system but we have slightly different goals, specifically we're aiming Please at being lighter weight and pushing the boundaries of what can be done within the BUILD language. Since Please is written in Go there's no runtime dependency on the JVM.
>We actually used Buck for some months internally before deciding to write Please and before it was capable of building our repo. We preferred it to other options available, but again we're focused on different goals; Please is easier to extend to new languages, has a bunch of features that we specifically wanted (e.g. test sandboxing) and has a stronger focus on BUILD language correctness. Conversely we have much less support for mobile development.
>We're least familiar with Pants; one of our developers briefly tried it and while we liked many aspects we didn't think it was the ideal fit for us at the time.
That's a lot of text to say three things: doesn't use JVM, has test sandboxing, and the DSL is "more correct". This doesn't tell me what I want to know.
First thing every software engineer does after leaving Google is rewrite Blaze. Just like every site reliability engineer rewrites borgmon. What I chose to do myself was make Blaze happen using a Makefile config. There's a blog post somewhere where Google talks about why they switched from GNU Make to Blaze c. 2006. if I remember correctly it basically boiled down to not having strict dependency checking. So I thought, why not simply avoid their published mistakes without changing build systems? I learned everything I could about make, discovered that variables are secretly lambdas, and that enabled me to figure out a way to have a static archive package for each folder, which also avoided the need to write a makefile code generator. Then I wrote a simple C program to check incremental elf symbols so the build graph stays sanely formed. It totally got make working like a dream in a way I simply hadn't seen before. https://github.com/jart/cosmopolitan/blob/master/tool/build/...
A Google engineering manager once told me that “back in the day, SREs got promoted for writing monitoring systems, and so we ended up with a bunch of unsupported monitoring systems”
Reading your words, it looks like you have made a kind of make replacement that just works. Following your link, it looks like... what ? I don't know, some tool that checks some build configuration, augments C and postprocesses executable files?
(Looking at https://github.com/jart/cosmopolitan it's an interesting hack. Might be useful in some cases, for example command-line tools like git.)
Have you made something that looks like a better make? If so, it would be nice to know more. Thanks.
Sorry here's what a build file looks like, which should hopefully clarify: https://github.com/jart/cosmopolitan/blob/master/libc/mem/me... As you can see, it's just vanilla make. The package program is just a helper that checks to make sure DIRECTDEPS variables are correct. That's the part Google said was tricky back in 2006. Have you ever seen people talk online about how it's super difficult to have a single makefile build multiple library files? Particularly static ones? Like the pitfalls of making sure the linker arguments are the correct order, or removing a dependency from one will break all the others that needed it transitively? This is how it can be done. I found out it just requires following a highly specific convention, where it's a little toilsome to type, but so rewarding once it's done. It also means I don't need to ask users to install a cutting edge build system to build my code, since I've just seen too much drama in that space.
Hey guys! Thanks for checking Please out! I see a lot of recurring themes in the comments so I thought I would clear some things up.
What is Please?
Please is a multi-language build system designed for huge mono-repos. It was created by a couple frustrated ex-googlers who were familiar with Blaze (which was later open sourced as Bazel). We found the "real world" alternatives to be somewhat lacking and so Please was born!
Please draws inspiration on the Blaze paradigm. If you're familiar with Bazel, the biggest difference is Please aims to be simpler and have far less magic in the binary. We push the implementation of the build rules into the build language, dog feeding them to ensure Please is flexible enough for any task. Also Please is written in Go so doesn't require a JVM ;)
If you're not familiar with Blaze/Bazel, here's what all the fuss is about:
1) Hermetic builds: builds are run in their own tightly controlled environment. Each step of the build runs in their own temp directory isolating them from other steps and only having access to the files and environment variables they've declared as their inputs. Please also has sandboxing built in taking advantage of the linux kernel to further isolate tests.
2) Scalability through incrementallity: if you've used Make, you're probably familiar with caching problems. Make uses last modified timestamps on files to determine if they need to rebuild each step, which turns out is fallible. Please uses a hash based approach which is far more robust. Most of our developers don't even know how to clean the cache. As a result, we can incrementality build our entire repository locally and on our CI workers no matter how big our repo gets.
3) Flexibility: the build language is a dialect of python. This can be used to write "build definitions" which define a unit of work i.e. compiling a Go package. There's nothing special about the built in definitions; it's totally possible to write your own to automate nearly any part of your development process. You could generate code, template kubernetes .yamls and beyond!
4) Unified developer experience: The please command line provides a unified experience across your codebase. Want to test all the tests under a branch of your repo? `plz test //some/part/of/the/repo/...`. It doesn't matter what language you're using, what those tests depend on etc. etc. Please can always run them for you.
PS: Apologies for the website. We're a small team of build system engineers, not front end types. If you want to offer your skills, I'd be happy to point you in the right direction: https://github.com/thought-machine/please.
Or CMake or the C/C++-oriented http://floooh.github.io/fips/ (a high-level build system "similar to Rust’s Cargo or Javascript’s NPM, but for C/C++ projects.").
If you are working in an environment that needs to mix between different languages, and a monorepo makes sense for you, then a tool like please let’s you stitch those things together and handles dependencies between those things. So if you have a service defined in an IDL like gRPC or OpenAPI, with a backend in python and a web front-end in JS, and API docs, etc, a tool like this can stitch together changes really efficiently.
Please is a lot more lightweight than Bazel, so it’s easier to get deployed and work with different projects. The new maintainer has been shaving off a lot of warts lately and it’s getting a lot better.
If the above doesn’t describe your situation, then a tool like please, pants, buck, or bazel is probably not for you. And that’s ok too.
256 comments
[ 1.7 ms ] story [ 267 ms ] thread'build' kind of indicates that
If someone had the same idea, please don’t hesitate sharing.
> 'build' kind of indicates that
Not the GP, but as a developer reasonably conversant with build systems and their pain points, I still assumed this was going to be some form of lazyweb[0] related blog post or portal rather than a build system named 'Please'.
[0] https://ftrain.com/dear-lazyweb
...one can stop reading there: this product is not aimed at this community.
The usual annoying thing is the automated package install. I have not looked at this particular package, but in the past, I have seen:
- installing specific gcc version and making it system-default.
- installing “virtualbox” packages - this was the worst, as the machine had a few KVM VMs running at the same time, and KVM and VirtualBox cannot run at the same time.
In general, I now stay away from every install script, and if I cannot, then I run it in docker container. Life is short, I do not want to spend it fixing my primary workstation.
(And I examine postinst scripts in deb files too, but those are usually much more boring)
I've occasionally seen scripts that install some other application where it was not clear that it was a dependency and there was no heads up to the user that this was going to happen: that kind of behavior makes me more distrustful of the author, so there's a useful signal there.
Most scripts like this seem to amount to "configuration defaults + cli completions installation". To that end, I find looking at them useful because it gives me a sense of the expectations of the tooling and hints at where I might find things if I need to go debugging later.
When they are provided by the same entity as the program you wish to run, I don't see how it's significantly more risky to run these scripts than running the application code itself with the active user's permissions. Still, if there was something, by glancing at the script, you have half a chance of seeing it. If there's something there that doesn't make sense, that's both a "proceed with caution" and a learning opportunity.
Save to file, and read the file in the editor.
That said, I generally pipe to file and cat the file, yes, if only because it somehow feels wrong to download it twice.
Both of those are vastly safer, because they do not require root privileges, and guarantee to uninstall cleanly.
For example, most interactive installers I have seen prompt for install location, and write minimal amount of files outside of it. The chance that they break your system is pretty low.
At the same time, many |bash scrips like to install system-wide packages, or use pip as root, or other potentially breaking things. And they do not ask you before installing this, and often auto-confirm, too.
I'm not a kind of person who's on a crusade against curling into bash, but I never do that myself because it feels dirty. It gives me no advantages and makes me unable to even take a quick look to reassure myself that things at least look like what I'd expect them to be.
Good practices are rarely black and white, they're often there to influence how you think about stuff and what habits you develop.
The build logic and CLI experience looks to be well thought out. I really like the native sandboxing support and the intentional exclusion of the calling environment. There's a lot of polish here.
To the maintainers of Please: please make cases and user stories, also comparisons with other build tools.
Many visitors will jump on each "comparison with tool X" where X=each of the current tools I'm familiar with. Only "Bazel, Buck or Pants" are mentioned. Okay no JVM, looks like Please is a better thought out Blaze/Buck.
Please mention CMake and others, like the C/C++-oriented http://floooh.github.io/fips/ (a high-level build system "similar to Rust’s Cargo or Javascript’s NPM, but for C/C++ projects.").
Changing build systems is hard, and getting buy-in from the team even harder - so you need to demonstrate value up front.
With that being said I have put a good amount of work into the QuickStart.
The code labs are designed to get you up and started in an inviting way. If you have any specific feedback about your QuickStart experience I’d love to hear it.
Have you considered sponsoring, or accepting donations, to work on IDE integration? That's pretty much the only reason I migrated my current employers codebase over to Bazel instead of Please.
Also, the docker support in Bazel is pretty "different". It's not bad, but it's not immediately obvious to newcomers how this is meant to work.
It's getting better every day though.
(I'm one of the original developers, but these days do less maintaining of it than tatskaari does)
We'd also tried Gradle previously but that was pretty awful for anything non-Java which made it a non-starter.
To be clear, I'm one of the original implementors, although I imagine that was obvious already...
People interested in type annotations can check this discussion: https://github.com/bazelbuild/starlark/issues/106
For most purposes, Bazel dependency on Java is an implementation detail (users don't need to install a JVM), although you might notice it if you need to bootstrap Bazel.
(I co-designed Starlark and I used to work on Bazel)
My rubric for documentation is "how many clicks to code". I found the installation instructions, and started to look for usage, but didn't find it. Too many clicks. That's where I'd start.
PS - IMO the gold standard for this evaluation method is Sinatra.rb, which is zero clicks to code.
[1] http://sinatrarb.com/documentation.html
[1] http://sinatrarb.com/intro.html [2] https://github.com/sinatra/sinatra#readme
So much wasted time goes into diagnosis of incorrect results from incremental builds that most people who implement CI systems never use incremental builds, and always build from scratch. Developers are far too used to having to do things like 'make clean' because their build didn't work quite right. Efficiencies are then gained by doing things like ccache/Gradle build cache, which trace their dependencies better than most naively written build systems do.
Edit: zig is a beautiful exception (https://ziglang.org/download/0.3.0/release-notes.html#cachin...)
My perspective here is specifically from that of a maintainer of large builds where the code base is always under active development. Murphy's law becomes your enemy at scale.
This is not a criticism of Please - just an observation that as an industry we are not there yet. But, I'm happy to see things moving in the right direction.
Again, I am not picking on Bazel here. Correct builds are difficult enough to get right even when everyone involved wants to keep the build correct! But, the reality is that a lot of real-world build conditions can be downright hostile. (did you know that ext4 vs xfs can change the order JARs in the same directory are loaded off the classpath? I wish I didn't!)
You hopefully have tried out many other build systems by now, been frusted with all of them, and only then made your own. And with all that experience with the status quo, marketing should be easy.
If you haven't done that, this project is a net-negative on the world, because the proliferation of build systems exacerbates Conway's law and balkanizes our software commons.
NIH in FOSS is not free.
This isn't a paid product, after all.
(I'm also a traditionalist that thinks writing good prose and writing good programs are related skills.)
For me, writing code is a way of solving problems for people. That involves listening to them to see how I can help. Then collaborating with them to iteratively make something that fits their needs. Which includes helping them adopt it and get productive with it. To me, all of that is part of software development.
And that's most of the "marketing" that's needed for a free, open-source tool. When that's lacking, I have to wonder to what extent the tool was actually made to solve somebody's problems. As opposed to being created because the developer was excited to build a thing as a technical exercise. Either path is fine, of course, but it's a mistake to confuse the two.
Growing up, one of the jobs I took was as a system's administrator at the same company my father worked at. It was a small company, maybe six people or so, doing contract work for aerospace. One of the older men who worked there held his Ph. D. in aerospace engineering. He was kind, warm and brilliant... but he struggled with things like sending coherent emails to clients. But that was okay - other people were able to take that on.
Whoever made this clearly knows about slick presentation --- they've got all those colors and shapes pizzas --- so they either capable with slick communications, or work with someone who is. My patience for lack of explanation is therefore a lot thinner.
I’m sorry, what?!
Do you realize the “sell something to people” industry has been at least as big as the tech industry for many decades and is the one growing faster and faster?
Also marketing has historically been a problem for people who make stuff.
Also, better have the Balkans of software than the USSR of mega corps.
When we created Please we were currently using Buck from Facebook. As has been covered elsewhere at the time it had some limitations which meant that it was getting more and more expensive to workaround (no multiple outs from rules so doing things like sourcemaps was hard etc). We had previously migrated from a set of disparate build tools (Gradle, Gulp etc) to Buck after evaluating it against Pants (we actually trialled our repo with both).
We all had experience with Blaze (Bazel did not exist in 2014/2015) and wanted to get closer to the experience we had in Google, hence the final decision to build our own. At the time we rolled it out we had full Buck compatibility (and actually most of our engineers did not notice initially). Once we were happy it worked, we migrated in full to Please.
Overall it allowed us to provide much better tooling for our developers (queries on the build graph for smaller CI/CD footprints, coverage in all languages, and for rules that can output multiple languages (like protocol buffers) we only build variants in the requested languages rather than all languages).
If Bazel had been open-sourced and easy to extend at the time we would certainly have looked to adopt and improve it (as we did with Buck at the time). But by the time they open-sourced it the two system had diverged in their approach and use cases.
I (obviously somewhat biased) think having multiple principled build systems which prioritise the needs of different communities is good for the ecosystem overall (and allows the sharing of good ideas such as the Remote Execution Interface).
https://please.build/faq.html
Glad it's getting some discussion on HN. Can't wait to hear other's experiences.
It's consistent between all languages and supports a lot of functionality. For example the proto_library and grpc_library rules make building microservices extremely easy. You can build an entire service in like 3 lines of boilerplate.
It also makes it easy to do cross-language deps. For example a c library included into a Java library that uses native bindings to wrap it.
Without this tooling large organizations tend to silo which creates high engineering costs (every project starts from scratch, no shared frameworks/libraries, no standardizing around best practices, etc).
The advantage of a running, pre-existing system (and the theoretical advantage of an off-the-shelf system like this that you can supposedly turn into a running system) is that it's been built to satisfy most of the needs of a wide variety of developers working on all sorts of things, and so each engineer on each of those teams that says "All right, I'll spend some time thinking about how we build and release code" is incentivized to start from the common starting point, even if they have to customize it. That means that if two or three teams end up working on similar-enough projects down the line, the shared tooling can actually support communication between the teams. Without the shared tooling, they can communicate all they want, but it won't be rational for any team to abandon its existing tooling. (And of course you need the communication too - both are required.)
At the micro-level, tools help, no doubt about it. At an org level, communication helps (documentation is a form...). Even if using different tooling, concepts and strategies are the same. This isn't to take away from the argument that this tool the OP posted is pretty awesome.
If you want change, be the change. Evangelize the tool through your org, not just your team. If your org is silo'd, break those barriers down with a friendly lunch-n-learn or lightning talks session. We tout empathy for our customers, how about empathy for our coworkers too? Others may have the same pain...
There are already book-length arguments that I like to refer to for specific topics, such as trunk-based development.
https://trunkbaseddevelopment.com/
Unfortunately, microsoft had to move away from VFS for Git to some a more limited-in-scope approach: https://github.com/microsoft/scalar
As a concrete example, one frustration at $DAYJOB is that building certain Google-released OSS is hard to integrate with our existing build system because Bazel is pretty opinionated about compiler-provided headers, output paths, etc. We're calling Bazel as a subprocess during our build and making various files available to it to make our compiler build visible to it, and then we pack up the results. I'm pretty sure if we somehow adopted Bazel for our entire build this would be much better - would it also be better if we adopted one of those other systems?
(Maybe one way of asking this is, are any of the corporate sponsors of Buck, Pants, or Please building third-party Bazel code from source? Or vice versa?)
A broader question: would it be worth defining a compatible subset of BUILD file syntax and library calls (i.e., not just Starlark but also the rules themselves for common things like building C libraries or JARs)?
Are we in the build-system equivalent of the vendor-specific C / C++ / UNIX implementations, and will a cross-vendor standardization effort emerge one day?
[0] https://opensourcelive.withgoogle.com/events/bazelcon2020/wa...
On the BUILD file level, Buck is using the Starlark. Pants and Please are close enough that some tooling (e.g. buildifier) can be shared. But that's about it.
What am I missing by doing that? This looks really slick, but I'm not sure how, why, or where to use it.
In your case, it seems you only have one non-trivial step: python dep install. So this system will be quite overkill for you.
Fast forward 5 years, now we're using Please to build all our code, generate hashes for docker images, template those hashes into our k8s .yamls, generate the documentation website from the docstrings in the proto files etc. etc.
With a language specific build system, you would have a lot of trouble handling things like this.
Let me know when I can actually use this on any machine I have to work on, instead of going "fuck you, Windows users". Writing cross-platform build tooling isn't rocket science, it's a choice. And the choice made here is stupidly disappointing in near as makes no difference 2021.
Please be a good netizen.
If you’re interested in the project and are a Windows user, contribute to the project.
This is written in go! How isn't it cross platform?!
Also, they might just not have Windows developers or a use case for running the tool on Windows, so even if the cost of adding Windows support was zero, they'd have to go out of their way to test it. A lot of open source code might run just fine on Cygwin, but most people choose not to support Cygwin because damn near no-one uses it anyway.
For example, if all I have is a MacBook running macOS and Docker, how can I reasonably expect to port this to Windows? I haven’t owned a Windows machine for 10 years.
I am the author of Buildbarn, a build cluster implementation for Bazel/Pants/.... Buildbarn also doesn’t support Windows. If someone sent me high quality patches for Windows, I’d merge them for sure, but nobody has. Sorry you are ‘stupidly disappointed’ in me.
I perform scanning of process tables to kill lingering processes. Go has no functions for that in its standard library. There are some third party libraries for that, but they are all of pretty poor quality. They are slow.
The value proposition is also a bit more clearly defined there [2]
[1]: https://www.pantsbuild.org/docs/welcome-to-pants
[2]: https://www.pantsbuild.org/docs/how-does-pants-work
Another big difference is Please is written in Go so there's no dependency on VMs or runtimes.
If the goal is continued maintenance, then encouraging new engineers to explore and read the codebase with tools they can comprehend is critical.
disclaimer: have not used these particular tools, but the domain is nice and polite
Here are some real reasons why Make isn't the end-all:
- Make doesn't allow platform selection in a nice way. - Make doesn't work on Windows natively (no, NMake doesn't count). - Recursive make doesn't work well at all. - Make doesn't track byproducts or deleted artifacts. - Make doesn't have auto-clean. - Make doesn't facilitate out-of-source builds. - Make itself isn't a scripting language (arguably good/bad). - Make doesn't facilitate compiler selection cross-platform, so things like MSVC are nearly impossible to integrate nicely without the use of the developer tools prompt. - Make can't (elegantly) handle a number of rule cases, such as single-input multiple-output rules (it can, but it's a hack).
Not sure why you think Make is unapproachable. That's like saying shell scripting is unapproachable. No developer worth a damn is going to avoid shell scripting, and as long as they understand "this file turns into this file using this command" then Make makes sense.
As with all tools, Make and Autotools and CMake and probably Plz will be misused by developers that think they're being clever. In actuality, they make things worse at best and unusable/unmaintainable at worst.
As a build systems designer, I've evaluated Plz personally and found it to be neat but not suitable for most of the problems I think build systems should solve.
all of which are even worse than make itself.
Also curious to know, is there no work done in the realm of developer productivity inside the FPGA world?
I've only looked at modern tools briefly but everytime I come away realising we'd have to re-write everything. There is no out-of-the-box support for our tools/language and it would be a lot of work to learn someone else's DSL, re-implement everything and then potentially discover a load of ways it's broken for our workflow. Personally I think it'd be worth it in the long run but it's a really hard sell.
These tools operate at a higher level of abstraction. The BUILD file abstraction is something like "I am declaring a C library with these sources and headers." The build system then turns that into a set of commands to run. This higher-level abstraction provides a bunch of compelling features:
- Reliability: it's easy to write Makefiles that are fragile, for example one that works when run on a single CPU but fails when run with -j, because of undeclared dependencies. When the tool is in charge, it has more control and can sandbox each build step so that undeclared dependencies simply won't be available in the sandbox.
- Reproducibility: similar to the previous point, it's easy to write a Makefile that produces incorrect results with an incremental rebuild. For example, unless you implement your own header scanning (or manually list your header dependencies) "make" won't know to rebuild a source file when a header changes. These tools can ensure that builds are reproducible and incremental rebuilds are always correct.
Besides that, working at a higher level generally means writing less code to accomplish the same task, and the code you do write is closer to expressing your actual intent compared with a Makefile.
> The BUILD file abstraction is something like "I am declaring a C library with these sources and headers."
This is wrong. Even ninja has generic rules. Here's an example of a minimal makefile:
> When the tool is in charge, it has more control and can sandbox each build step so that undeclared dependencies simply won't be available in the sandbox.There's no reason this shouldn't be possible with make; it just hasn't been implemented so. Do bazel/buck/please actually do this? As far as I know tup is the only tool that actually verifies inputs/outputs of rules, and it needs FUSE to do so.
> For example, unless you implement your own header scanning (or manually list your header dependencies) "make" won't know to rebuild a source file when a header changes.
At least with GNU make, it's very easy:
True, it's a bit of a footgun, but by no means difficult.Make has problems, but the ones you listed aren't they.
Your example does not contradict what I wrote. You manually specified the tool to be run ($CC) and all of the arguments to that tool.
It's true that there is a level of indirection through the $CC variable, but you're still operating at the level of specifying a tool's command-line.
> There's no reason this shouldn't be possible with make; it just hasn't been implemented so.
Make is 44 years old. If it were an easy extension to the existing paradigm, there has been ample time to implement such an extension.
> Do bazel/buck/please actually do this? As far as I know tup is the only tool that actually verifies inputs/outputs of rules, and it needs FUSE to do so.
Bazel certainly has some amount of sandboxing, though I don't think it's quite as complete as what is available internally at Google with Blaze. I haven't used Buck or Please, so I can't speak for them.
> True, it's a bit of a footgun, but by no means difficult.
Well footguns aren't great. :) As just one example, any header that is conditionally included (behind an #ifdef) could cause this cache to be invalidated when CFLAGS change, but Make has no idea of this.
Notably, I didn't specify the arguments to be used to compile the source files. But if you insist:
Notably, linklibrary can be defined according the platform, or according to dynamically set variables, giving you the same level of flexibility as make.>> There's no reason [sandboxing] shouldn't be possible with make; it just hasn't been implemented so.
> Make is 44 years old. If it were an easy extension to the existing paradigm, there has been ample time to implement such an extension.
‘Nobody's done it yet, ergo it's not possible or easy’ is not a valid argument.
> just one example, any header that is conditionally included (behind an #ifdef) could cause this cache to be invalidated when CFLAGS change, but Make has no idea of this.
Then you specify that the source files depend on the makefile.
Then you gratuitously rebuild everything whenever the Makefile changes, even if you only changed a comment.
Also this scheme is incorrect if the Makefile was written to allow command-line overrides of variables like CFLAGS, as many Makefiles do.
But these are just details. The larger point is this. The language of Bazel is defined such that builds are automatically correct and reproducible.
While it's true that Make has some facilities like "call" that support some amount of abstraction, it is up to you to ensure the correctness. If you get it wrong, your builds are back to being non-reproducible.
It's like the difference between programming in a memory safe vs a memory unsafe language. Sure, every thing you can do in the memory safe language can be done in the unsafe language. But the unsafe language has far more footguns and requires more diligence from the programmer to get reasonable results.
Ultimately, make and co are text oriented, while bazel and co are object oriented. Hacking object oriented capabilities into a text-oriented language isn't particularly fruitful or ergonomic.
Bazel and make are both text-based languages for describing symbolic, abstract structures. Just like pretty much every other programming language.
Bazel and make both use the same abstract structure, just like pretty much every other build system: a directed graph of build directives and dependencies.
But let's look at how a rule is implemented in bazel[0]. Here's a rule "implementation" for a simple executable rule[1]:
This is, admittedly, not easy to follow at first glance. Concat defines a "rule" (just like cc_binary) that takes three arguments: "chunks", "out", and "merge_tool" (and "name", because every target needs a name).Targets of this form have metadata, they have input and output files that are known and can be queried as part of the dag. Other types of rules can be tagged as test or executable, so that `blaze test` and `blaze run` can autodiscover test and executable targets. This metadata can also be used by other rules[2], so that a lot of static analysis can be done as a part of the dag creation, without even building the binary. To give an example, a rule like
can be built and implemented natively within bazel by analyzing the dependency graph, so this test could actually run and fail before any code is compiled (in practice there are lots of more useful, although less straightforward to explain, uses for this kind of feature).Potentially, one could create shadow rules that do all of these things, but you'd need to do very, very silly things like, off the top of my head, creating a shadow filesystem that keeps a file-per-make-target that can be used to query for dependency information (make suggests something similar for per-file dependencies[3], but bazel allows for much more complex querying). That's what I mean by "object-oriented". Targets in bazel and similar are more than just an executable statement with file dependencies. They're complex, user-defined structs.
This object-oriented nature is also what allows querying (blaze query/cquery/aquery), which are often quite useful for various sort of things like dead or unusued code detection or refactoring (you can reverse dependency query a library that defines an API, see all direct users and then be sure that they have all migrated to a new version). My personal favorite from some work I did over the past year or so was is `query --output=build record_rule_instantiation_callstack`, which provides a stacktrace of any intermediate startlark macros. Very useful when tracking down macros that conditionally set flags...
http://beza1e1.tuxen.de/hermetic_builds.html
Everything else that is needed for a modern build system is lacking:
This is why people these days mostly take the path of generating Makefiles, because make is only good at one thing: executing a statically defined dependency graph.At a very minor cost after all: your sanity.
- Cannot handle multiple outputs for a single rule
- Does not rebuild when flags change
- Make rules may contain implicit dependencies
- Slow for large codebases
- Does not understand how to build for multiple platforms, sucks for cross-compiling
- Recursive make sucks (job control does not work across recursive invocation boundaries)
- You must create output directories yourself
- You must create your own rule to clean
This adds up to a fragile & slow build system, where you have to do a full rebuild to have a reasonable level of assurance that your build is correct—incremental builds aren’t safe.
There have been a few attempts to make a “better make” over the years like Jam, SCons, Ninja, Tup, etc. each with their own pros and cons.
The new generation of build tools—Bazel, Buck, Pants, and Please are all an attempt to redesign build systems so that the design is resistant to the flaws that plague Make build systems. You can use a shared cache (shared between different users) fairly easily, and you have reasonable assurance than incremental builds are identical to full builds (so your CI pipeline can move a lot faster, developers almost never have to "make clean" when they change things, etc).
Personally I’m working on a project right now that uses Bazel (which is similar to Please) and is for an embedded system. It’s been a great experience, and I can pass a flag to Bazel to tell it to build for the embedded target using the cross compiler or for the native host--that makes it easy to share code between tools and the target system, and I can do things like write tests that run on both the target & host. Anyone who does any cross-compiling is missing out if they are using Make—but, do note that setting up a cross-compiling toolchain in Bazel isn’t exactly a cakewalk.
https://docs.bazel.build/versions/master/tutorial/cc-toolcha...
No, it doesn’t. There’s NO syntax for multiple outputs. If you can show me what the syntax is and prove me wrong, I’d love to see it. At best, there are workarounds for the lack of multiple output support, with various tradeoffs.
> Make is a programming language with a built-in dependency mechanism, and as such you can use it to build whatever you want...
Make is NOT a programming language. End of story. You can… kind of… build things with it, given that variables in Make can be used like functions, but it’s goddamn horrible and you shouldn’t do it because nobody will want to use it and nobody will maintain it.
At most, you can build something on top of Make, but you’re still facing the limitations of Make and working around them. If you are interested in building something on top of a low-level build system, you want Ninja instead of Make, because Ninja is good at that. Make isn’t.
Make is, at best, something you would want to use for a small C program where all the files are in one directory. Once you grow past that use case, my rule is that you shouldn't be using Make any more, because there are just too many things that Make does wrong.
Make is successful because eh, it’s good enough for small things, you can suffer through the pain if you need to use it for big things, and it was the first tool to solve this problem. We have better tools now. We had better have better tools now! Make is in its fifth decade… if we didn’t improve on Make in that many years, that would be a damning indictment of the entire field.
(One point you don't mention in your list of reasons why Make is successful is that it's reliably available everywhere. For projects that ship as source for others to build, that matters, and it delays uptake of fancy new build systems in that segment.)
So do C, and Bourne shell. And SQL.
And yet, here we are. Are the benefits enough to overcome incumbency?
I would strongly recommend watching this talk if you want to understand why systems like Bazel and Please were created:
Build Systems with Andrey Mokhov (Jane Street)
https://www.youtube.com/watch?v=V9YA32uV3Ls
So at the end of the day you end up with an executable artifact, but hopefully you did a lot less work to get there.
1. Reproducible/Hermetic: your builds cannot, say, access the network during a step or produce changing binaries for the same input. This makes sure that once you've downloaded your deps locally everything builds correctly from then on.
2. Caching: Because everything is reproducible in bazel-likes everything is cachable. Dependencies, builds, and even tests! If you've worked with Makefiles you've likely run `make clean; make tests` or something similar. I've never needed to do `bazel clean` to make sure things were working right.
3. Visibility: you don't only control who can see what in your source code. Different dependencies can be marked as private/protected/public to control who can import them. This is a huge boon to large monorepos.
4. Everything is uniform: code generation, compilation, etc is all described in "rules" and "toolchains" and can "easily" expanded to other languages. The community manages a Rust and Golang rules set for bazel itself and they're better then the Google rule sets for Java (only "officially" supported rule set right now) in some areas.
So if you have a lot of code/internal libraries/code generation, what to write a LOT of unit tests and cache their results, and write code in multiple languages bazel is probably for you.
You can also use tools like BuildBuddy [0] to share a cache with your team and extract build performance metrics to identify areas you can improve performance in.
[0] - https://www.buildbuddy.io/ or https://github.com/buildbuddy-io/buildbuddy
It does have some features to facilitate CI and testing at scale, e.g. `plz query changes` can be used to find a minimal set of tests to run for a PR.
Yes, so why would I use Please over any of them? I've spent close to 10min reading and have no idea why this exists or why anyone would use it. It looks like Bazel with a different config format, in which case why wouldn't one just use Bazel?
However, I still have somewhere over 100 GB free on my seven-year-old laptop, and the download will take me ~40 seconds.
But even if the 600MB were a problem, Bazel doesn’t need an external version of the JVM. It bundles its own, and fits the whole thing in under 50MB. Some people are under the mistaken impression that you have to install the JVM first and then install Bazel, which is simply not true.
In my experience there is a strong correlation between executable size and how hard it is to unravel any issue that arises in it.
Bazel bundles the JRE, that’s why it’s 45 MB.
The correlation is kinda beside the point, though, because my experience with Bazel is that it’s easier to unravel issues with Bazel than unravel issues with Make, and Make is much smaller.
[1] https://en.wikipedia.org/wiki/List_of_countries_by_Internet_...
Most people don’t have the kind of bandwidth it takes to download a 50 MB binary? Are we still talking about Bazel here, because that’s the size of the download. What about tools like compilers? You have to download those, too.
> Running a background JVM daemon process is a NO for me and Bazel wastes system resources.
What system resources does it actually use up? A half-gig of RAM? This kind of optimization is penny-wise and pound-foolish. You are spending your precious time and energy worrying about a resource whose marginal cost to you is about the same as a cup of coffee.
I run Bazel on a terribly obsolete, seven-year-old laptop which is my daily driver. Sometimes Chrome will choke on a website, sometimes I'm waiting ages for Homebrew to update or for NPM to download some packages, but Bazel is not a problem.
I don't install tons of random garbage because I want to know and understand what is running on the hosts that I maintain.
I need to be able to debug the stuff I run, and complexity makes it difficult.
Tenths of MB of stuff is complexity. Tenths of build time or run time dependencies is complexity. A compiled tool when a script would have been sufficient is also complexity.
Bazel is way easier to debug than Make. Debugging a decent-sized build system made with Make is just an exercise in suffering.
I am… honestly… no longer interested in understanding the entire software stack. I understand my time on this earth is limited and want to spend it doing other things. With Bazel, I am spending less time fucking around with build systems and more time doing the stuff I care about.
> Tenths of MB of stuff is complexity. Tenths of build time or run time dependencies is complexity. A compiled tool when a script would have been sufficient is also complexity.
As someone who still writes C, I can understand the joy that people feel when you make some cool program and it’s measured in KB, not MB. However, what you’re describing strikes me as fetishistic.
"slimmed down"
lol.
I'm not suggesting Bazel is perfect: I think e.g. Starlark's insistence on being a separate language does more harm than good. (FWIW I also think the JVM objection is a little silly.) But I am saying preferring the Google tooling would ostensibly mean you like Bazel a lot already :)
I think the term DSL is overloaded here? Consider all the Lisp and Ruby stuff that's definitely DSL but doesn't need most of a language implementation.
Bazel has a lot of magic and crazy abstractions. Please is far easier to get your head around and as a result, far easier to bend to your iron will.
We wanted to switch away from Bazel because of the mismatch; it looked like Pants and Buck had the same issues. At the time Please didn't have sensible documentation so it was skipped. From what they have now it seemed like it's more reasonable in this respect, but I don't actually know yet; perhaps somebody who knows can chime in?
By custom rules do you mean macros, or full on custom rules?
I mean full on custom rules, exactly because of the inability for genrule to span more than one command (so we end up having to generate tiny shell scripts and running them).
I've also had issues debugging things any time I've tried to use genrule; the error messages tend to be unhelpful in my experience (though that's been a while and I don't recall the details). Things that look like they'd do what I want would just not work in unhelpful ways, and interrogating the system usually didn't work until there were no errors (at which point I wouldn't need to interrogate it anymore…)
What do you mean by a "command"? I'm very confused by the nomenclature here.
>All four of these systems are quite closely related in the scheme of things, being inspired by (or in Bazel's case, a direct open sourcing of) Google's Blaze.
>Several of us had worked at Google and used Blaze extensively there; we were excited about it being open sourced as Bazel but by then we were already using Please internally. It's a great system but we have slightly different goals, specifically we're aiming Please at being lighter weight and pushing the boundaries of what can be done within the BUILD language. Since Please is written in Go there's no runtime dependency on the JVM.
>We actually used Buck for some months internally before deciding to write Please and before it was capable of building our repo. We preferred it to other options available, but again we're focused on different goals; Please is easier to extend to new languages, has a bunch of features that we specifically wanted (e.g. test sandboxing) and has a stronger focus on BUILD language correctness. Conversely we have much less support for mobile development.
>We're least familiar with Pants; one of our developers briefly tried it and while we liked many aspects we didn't think it was the ideal fit for us at the time.
(Looking at https://github.com/jart/cosmopolitan it's an interesting hack. Might be useful in some cases, for example command-line tools like git.)
Have you made something that looks like a better make? If so, it would be nice to know more. Thanks.
What is Please?
Please is a multi-language build system designed for huge mono-repos. It was created by a couple frustrated ex-googlers who were familiar with Blaze (which was later open sourced as Bazel). We found the "real world" alternatives to be somewhat lacking and so Please was born!
Please draws inspiration on the Blaze paradigm. If you're familiar with Bazel, the biggest difference is Please aims to be simpler and have far less magic in the binary. We push the implementation of the build rules into the build language, dog feeding them to ensure Please is flexible enough for any task. Also Please is written in Go so doesn't require a JVM ;)
If you're not familiar with Blaze/Bazel, here's what all the fuss is about:
1) Hermetic builds: builds are run in their own tightly controlled environment. Each step of the build runs in their own temp directory isolating them from other steps and only having access to the files and environment variables they've declared as their inputs. Please also has sandboxing built in taking advantage of the linux kernel to further isolate tests.
2) Scalability through incrementallity: if you've used Make, you're probably familiar with caching problems. Make uses last modified timestamps on files to determine if they need to rebuild each step, which turns out is fallible. Please uses a hash based approach which is far more robust. Most of our developers don't even know how to clean the cache. As a result, we can incrementality build our entire repository locally and on our CI workers no matter how big our repo gets.
3) Flexibility: the build language is a dialect of python. This can be used to write "build definitions" which define a unit of work i.e. compiling a Go package. There's nothing special about the built in definitions; it's totally possible to write your own to automate nearly any part of your development process. You could generate code, template kubernetes .yamls and beyond!
4) Unified developer experience: The please command line provides a unified experience across your codebase. Want to test all the tests under a branch of your repo? `plz test //some/part/of/the/repo/...`. It doesn't matter what language you're using, what those tests depend on etc. etc. Please can always run them for you.
PS: Apologies for the website. We're a small team of build system engineers, not front end types. If you want to offer your skills, I'd be happy to point you in the right direction: https://github.com/thought-machine/please.
https://github.com/thought-machine/please/issues?q=is%3Aissu... https://github.com/thought-machine/please/issues?q=is%3Aissu...
Please is a lot more lightweight than Bazel, so it’s easier to get deployed and work with different projects. The new maintainer has been shaving off a lot of warts lately and it’s getting a lot better.
If the above doesn’t describe your situation, then a tool like please, pants, buck, or bazel is probably not for you. And that’s ok too.