Launch HN: Moonrepo (YC W23) – Open-source build system

As I understand it, the primary reason these build systems leverage these Python-variants is so that the build rules, toolchains, constraints, and build definitions can all be written in the same language (since build rules often require some programmatic behavior). Perhaps with a future vision of them being totally interoperable across build systems.

Could you explain why any existing project using Turborepo/Nx should switch to Moonrepo? What are the advantages and disadvantages? The support for multiple languages seems like a big advantage.

(I also worked for Google a while back, and they were very conscious about not using Google as a verb internally, losing a slow battle against the tide of trademark dilution.)

It's a fair bit of effort to change one's already established code repository and build system, and I don't really understand from your pitch what you are offering that is sufficiently advantageous for a decision maker to choose to switch.

In the same vein as those questions how does caching work? Is it content based like Bazel or mtime like Nx et al? If there is no sandboxing does it do input tracking or is there manual "cache key" shenanigans?

If the configuration language is YAML how am I expected to implement new behavior? Is that in Rust? Is there a plugin architecture? Do I need to rebuild the tool itself and redistribute it to my build machines and developers? The main appeal of Starlark/Python in build systems is ability to create macros or in many cases define entirely new rulesets to support new tools and languages without needing to modify the complex and performance sensitive core.

Sorry for the skeptisicm but build systems are very complex beasts and new entrants like Nx don't measure up to tools like Bazel very well.

This seems to be a cyclical thing with build tools. Things are great until you expect customers to pay.

You lost me there. Buck/Blaze aren't perfect but configuration was never an issue.

Moon like it is currently is a glorified task runner.

Most likely you did this because you felt all the other ones are too complicated.

But the reason the “enterprise” ones are so complicated is to serve their enterprise customers, who need “just this one feature” so they can use it. But those customers pay the bills.

So basically you have to choose complexity and profit or simplicity and less (or no) profit.

Good luck! But make sure you’re ready to make that choice.

also: are you looking for any other founders?

I'm a tech lead for the web monorepo at Uber. We talked to the Turborepo guy a few years ago, and he admitted that he wasn't sure if it could handle our scale in terms of all the bells and whistles that a repo of our size leverages - and his is one of the more feature packed commercial offerings in this space.

As a random example: we see thousands of commits a week, so invalidating the whole build graph when the lockfile changes is a complete non-starter, yet most turn-key solutions target small teams and are not equipped to cope with this problem. Phantom dependencies[0] are a problem with disjointed build graphs. Etc.

As someone who's evaluated a ton of these systems, I would love to see where your experience in this space is coming from. A new kid in the block will have a lot to prove.

[0] https://rushjs.io/pages/advanced/phantom_deps/

What you will find is the vast majority of your configurations will invoke a make.sh script that does everything that you want to support in your system.

I've primarily worked in typescript codebases and have used raw yarn workspaces, lerna, nx and recently evaluated moon and turbo.

The funky part is I eventually simply went with nx, not just because I've used it before, but also because I felt like the configuration is just simpler and more lightweight. Esp. since you can pretty much roll with it without defining much more than single simple config file - while moon required some 2-3 separate config files, plus config files in each project (I understand the per-project config is not required, I don't remember why I needed it - something to do with my build tasks)..

In any case I intend to give it another shot soon.

As for the whole config in yaml vs json vs toml or whatever, not a deal breaker since most of the time these sorts of configs are perhaps not something you end up interacting with programmatically - I know a lot of people enjoy editing yaml more than json files

This seems pretty cool. I particularly like how 'gradual' it seems to be relative to things like Bazel, i.e. you can take some shell scripts and migrate things over. I did have a play and hit an initial problem around project caching I think, which I raised at [0].

One comment, from the paranoid point of view of someone who has built distributed caching build systems before is that your caching is very pessimistic! I understand why you hash outputs by default (as well as inputs), but I think that will massively reduce hit rate a lot of the time when it may not be necessary? I raised [1].

Edit: for any future readers, I spotted an additional issue around the cache not being pessimistic enough [3]

As an aside, I do wish build systems moved beyond the 'file-based' approach to inputs/outputs to something more abstract/extensible. For example, when creating docker images I'd prefer to define an extension that informs the build system of the docker image hash, rather than create marker files on disk (the same is true of initiating rebuilds on environment variable change, which I see moon has some limited support for). It just feels like language agnostic build systems saw the file-based nature of Make and said 'good enough for us' (honorable mention to Shake, which is an exception [2]).

[0] https://github.com/moonrepo/moon/issues/637

[1] https://github.com/moonrepo/moon/issues/638

[2] https://shakebuild.com/why#expresses-many-types-of-build-rul...

[3] https://github.com/moonrepo/moon/issues/640

But abstractions are a good thing! They let you separate a system into layers, so that things programmed at a higher layer don't need to know all the details of the lower layers. It's the way we separate interface from implementation.

Let me give an example from Bazel. Suppose you are compiling some C or C++, and you need to define a preprocessor symbol. There are two ways of doing this, "defines" and "local_defines":

    cc_library(
        name = "my_lib",
        srcs = ["my_lib.cc"],
        hdrs = ["my_lib.h"],
        # This will pass -DFOO when compiling this library
        # *or* any library that depends on it.
        defines = ["FOO"],
    )

    cc_library(
        name = "my_lib",
        srcs = ["my_lib.cc"],
        hdrs = ["my_lib.h"],
        # This will pass -DFOO when compiling this library
        # only.  Libraries that depend on us will *not* get
        # the symbol.
        local_defines = ["FOO"],
    )

I did not have to think about how defines will propagate to the libraries that depend on me, I just specified what I need and the build system handles the rest. I did not have to define my own CXXFLAGS variable and handle concatenating all of the right things together. The lower layer takes care of that.

What Bazel lets you do is create abstraction boundaries within the build system. People can write rules that define a self-contained set of build logic, then other people can use those abstractions without knowing all the details of how they are implemented.

Bazel is not perfect, and I have found myself banging my head against the wall many times. But overall the ability to define abstractions in a build system is, I think, one of the biggest things it gets right.

Disclosure: I work for Google, but not on Bazel.

- https://www.buildbuddy.io/

- https://github.com/buildbuddy-io/buildbuddy

The thing that annoys me most about CircleCI, TravisCI, Github Actions and Appveyor is that there is no simple way to run the same thing locally to debug or test workflows without creating either git history mess or temporary hacks like taking off branch restrictions in the yaml.

I'm incredibly skeptical of this.

I'm ex-meta and have worked a lot with the enterprise solutions you're talking about and the choice of starlark (originally python) as the build definition language is one of the killer features of the systems.

People want to create macros, target generators, etc. It's a common use case for a lot of engineers and IMO is a pretty killer feature.

Being able to say "This is an X86 Python Binary", "This is an M1 python binary" and then bundle those into "this is how you build either of those binaries based on inputs" without ever touching the internals or anything other than (more or less) a blob of python is why those tools scale organizationally.

It allows the teams that need to do weird stuff to unblock themselves without drowning the tools org. Sure, it has draw backs. Super deep macro layers are kinda a crime against humanity and debugging/evolving them can be quite expensive, but I think that's just the cost of software.

If that logic isn't in the build definitions it'll expand into a meta layer that generates configuration (I've seen giant "Translate this definition into 30 configs to run stuff" systems time and time again).

I may just be super biased from past mistakes and wins, but I think what you're doing is just moving the complexity out of your tool into neighboring tools and selling it as a win isn't really true, it's shuffling complexity around not removing it.

Just is a task runner (with full shell integration) that calls our turborepo tasks.

We define all of our tasks in one justfile (things like repo setup, syncing env vars, and compiling dependencies) and then link them to turbo processes which cache the result.

Massively reduced our cognitive load working with our monorepo, and is lightning fast.

If we ever want to change it will be simple to remove both, so we're not tied to the ecosystem.

[1]https://github.com/casey/just