Show HN: Hancho – A simple and pleasant build system in ~500 lines of Python (github.com)
Hi HN, I've been taking a break from my big side projects to work on a smaller side project - a tiny build system that's based on what I've learned from using Ninja and ad-hoc Python for my homebrew build systems over the last few years.
It's basically a promise-based dependency graph runner plus a simple text templating engine, and it works quite well for the smallish projects I've tried it out on so far.
If you find Make crufty, CMake inconsistent, Ninja verbose, and Bazel just too much build system, give Hancho a try.
Don't like one of Hancho's defaults? It's only 500 lines - hack it up however you like.
61 comments
[ 3.3 ms ] story [ 119 ms ] threadhttps://www.merriam-webster.com/dictionary/honcho
There are lot of words ending in 長 (chō) in Japanese that refer to people in positions of leadership: shachō (company director), kōchō (headmaster), kichō (captain, of an aircraft).
Another somewhat surprising example of unexpected Japanese etymology: ‘rickshaw’ comes from the Japanese 人力車 (jin-rikisha, literally “person-powered-car”). I would’ve probably expected it to have come from an Indian language, or something.
It’s these particular systems that I find the most difficult. If there were a collection of nice utilities for working with them uniformly, oh, that would be amazing.
I can not help but compare `honcho` to others in this space that I know, particularly `waf` and `snakemake`. I guess these do cover a larger feature space than `honcho` and that comes at the cost of rather greater complexity. I don't think sacrificing the simplicity of `honcho` to add features is necessarily a good thing but I do wonder if / how `honcho` might be used in these ways:
- Execution of a rule adds a new task node to the DAG (a `TaskGen` in `waf`).
- Implicit DAG forming and rule execution (akin to how we define a `rule:` for `snakemake` but it is the system that determines what rules to run and then runs them).
- DAG edge types besides files. Eg, run a downstream rule if some Python data object's value changes (instead of a file change). I believe neither `waf` nor `snakemake` supports this and but one can serialize the Python data to file to make it fit the DAG engine.
- Batteries included such as a cross-platform version of the `c_binary()` function in the `honcho` tutorial. (like `waf` "tools").
- In-system dependency generators (eg `waf` "scanner" pattern).
Thanks for sharing `honcho`!
1. In that case posting the source code to GitHub is less useful
2. More to the point, op said: “Don't like one of Hancho's defaults? It's only 500 lines - hack it up however you like.”
I think OP simply forgot to add a license. In which case, reminding them to do so is useful.
- "Execution" of a rule can add new task nodes to the DAG (via Rules with command=<async Pythonfunction>
- Implicit DAG is there in the form of Rules that depend on promises generated by other Rules.
- DAG edge types are just the promises returned by Rules. A custom rule that returns a promise that resolves to an empty array (or a dynamically-populated array) is totally valid
- 'Batteries included' rules are out of scope for the base release but I will probably add a default rules.hancho with my preferred C++ build commands.
- In-system dependency generators can be as simple as "glob.glob('*.cpp')", or as fancy as you're willing to write.
I've already used Hancho to build FPGA bitstreams with the Icestorm tools and it was delightful, but as the author I'm not a neutral observer. :)
Edit: if you use RSS, keep an eye on https://taoofmac.com, it will show up there...
I looked at the examples and tutorial and it still seems more low level, in that you mention object files and link them?
One reason I like the layering is for build variants -- dbg opt ASAN UBSAN -- something Make does very poorly. IMO you don't want to mention literal object file paths in the build config for this reason.I use a tree layout like
and then this is abstracted with the Bazel-like target syntax This works well - you don't have to clean when rebuilding variants, and all the object sharing between test binaries really speeds up the build.IMO this is 100% essential for writing C++ these days -- all tests should be run with ASAN and UBSAN.
---
I wrote a mini-bazel on top of Ninja with these features:
https://www.oilshell.org/blog/2022/10/garbage-collector.html...
So it's ~1700 lines, though some of that is our own logic, and extra features like computing preprocessed size, which I used to improve compile times.
You get the build macros like asdl_library() generating C++ and Python (the same as proto_library(), a schema language that generates code)
And it also correctly finds dependencies of code generators. So if you change a .py file that is imported by another .py file that is used to generated a C++ header, everything will work. That was one of the trickier bits, with Ninja implicit dependencies.
This build file example mixes low level Ninja n.rule() and n.build() with high level r.cc_library() and so forth. I find this layering really does make it scale better for bigger projects
https://github.com/oilshell/oil/blob/master/asdl/NINJA_subgr...
Some more description - https://lobste.rs/s/qnb7xt/ninja_is_enough_build_system#c_tu...
Comment about Chrome/Android using the same pattern - https://lobste.rs/s/0qc6vp/arcan_0_6_3_i_pty_fool#c_ahqwky
which is explicitly in the Android docs - https://android.googlesource.com/platform/build/bazel/+/7d96...
The main thing my system doesn't have as mentioned is the per-action build sandboxing, which Bazel has OS-specific wrappers for. That's really useful for finding missing deps. Right now it's not a big pain point, since the build will break in a slightly less obvious way if you're missing deps.
1. Incremental/cached builds are reliable.
2. You can reliably avoid rebuilding/testing artifacts that cannot possibly be affected by a PR.
Neither of those are possible in build systems that don't do this sandboxing. If you're doing a big project you really need these, hence why Google came up with this technique.
I'm not sure there's much value in a new build system that doesn't even get that right. We already have dozens.
The reason most build systems don't have it is because sandboxing is OS-specific
Bazel has sort of a complicated bootstrap/deployment for that reason.
I mentioned it a few times, but someone should extract the exec wrappers from Bazel as a separate project :) It's a small amount of code, though fiddly to build and distribute
There are few other strategies like bubblewrap, but last I heard bubblewrap is not installed by default on most distros. It needs setuid root so it's a bit fiddly and has security implications
But then you also need something totally different for OS X and Windows
If you don't need sandboxing, you can ditch a _ton_ of complexity.
When I was heavily using Bazel ~3 years ago we had made substantial changes to the Bazel source to try to support this, but ultimately couldn't get it across the finish line without making changes to Linux's dynamic loader. After talking to a few folks familiar with Blaze, it seems that's exactly what they've done inside Google to solve the problem.
Perhaps the situation has improved in the last 3 years, but my general experience with Bazel is that it made extremely lofty claims and delivered on very few of them.
The binaries your Bazel system built could not be run unless the runtime environment matched the build environment?
(I would go with statically linked binaries anyway tbh.)
I don't remember using runfiles much outside of tests in Bazel, even at Google. My guess would be that runfiles are packaged up into via MPM [1], which in open source world I would guess a container image is the closest equivalent? Then you get hermetic runtime environment, especially because k8s/containers seems to be the default deployment platform these days. I think the bazel OCI rules [2] should package runfiles too (I haven't checked)
[1]: https://www.usenix.org/conference/lisa14/conference-program/...
[2]: https://github.com/bazel-contrib/rules_oci
Both of these details are covered in the tutorial.
I saw "Bazel and Ninja" in the post, and thought it was more similar to what I had built.
Though IMO, to build C/C++, even for small projects, you need something to parse .d files, the output of gcc -M. Otherwise you need to write more build metadata manually, and you can get it wrong
And there is no enforcement of errors
For truly small projects, I just use a shell script, and no incremental build. Works great
---
And Ninja does parse .d files. The compiler is the authoritative source of header deps, because it has the preprocessor.
Ninja is very fast and small. It's meant to be generated, so I wouldn't call it verbose. I don't think anyone really writes Ninja by hand. There are at least 3 common code generators for it -- CMake, the Chrome one, and Android one
Personally, I think doit (https://www.bitecode.dev/p/doit-the-goodest-python-task-runn...) is still the winner in the python space, but there is a lot of it that could be improved.
I think I may steal Doit's "actions can be arrays of commands" feature, as that would be trivial to add to Hancho and potentially useful.
I find it easier to understand and maintain. I've used Makefiles for decades and I still find myself looking up basic Make syntax.
“How is it so awesome? In a typical build system, the dependency arrows go down. Although this is the way they would naturally go due to gravity, it is unfortunately also where the enemy's gate is. This makes it very inefficient and unfriendly. In tup, the arrows go up.”
https://gittup.org/tup/
On a more serious note, the whitepaper: https://gittup.org/tup/build_system_rules_and_algorithms.pdf
It's fascinating to think of this as a deliberate approach or paradigm rather than a shortcoming. The antithesis to the inner-platform effect [1]. Instead of digging an ever deeper rabbit hole of options and configuration, encourage customization via forking. You can't beat the flexibility. And lots of projects have well over 500 lines of build system configuration anyway. With that kind of competition, forking a 500-line build system may well yield the more comprehensible end result. Not to mention that when facing a problem outside of a given build system's anticipated set, I'd much rather have Python to solve that with than a special build system language.
[1]: https://en.wikipedia.org/wiki/Inner-platform_effect
The latter is the worse offense because it's much harder to decouple two systems than it is to notice a similarity in two bits of code and to extract the commonality.
If you only need to release it before Christmas no matter what, and sales will wane anyway in 6 months, then even worse crimes against maintainability may find a justification.
The problem is, of course, is that even code that was intended as throwaway tends to live much much longer than expected, if it actually works.
This meant that when you wanted to change something on all pages, it was a PITA but not always terrible because tools like grep, sed and awk exist for Windows also.
The nice thing was, you could make a change on any page and know that it would only affect that page and had no (or very little) chance to introduce bugs anywhere else in the system.
I wouldn't recommend the approach, but it wasn't all bad.
Smetimes you don't want that but sometimes you do, and it's a counter-productive mistake to only ever go one way or the other in all situations instead of identifying what best serves different situations.
dwm is customized through editing its source code, which makes it extremely fast and secure - it does not process any input data which isn't known at compile time, except window titles and status text read from the root window's name. You don't have to learn Lua/sh/ruby or some weird configuration file format (like X resource files), beside C, to customize it for your needs: you only have to learn C (at least in order to edit the header file).
https://dwm.suckless.org/