AOSP master platform builds now build with ninja instead of GNU make.
The build/core makefiles and Android.mk files are read by kati
(https://github.com/google/kati/blob/master/README.md), which will
interpret them and write out a $OUT_DIR/build_$(TARGET_PRODUCT).ninja
file, and then the execution of the build rules will be handled by
ninja (https://martine.github.io/ninja/).
Building with kati and ninja provides some immediate benefits:
1. Faster startup time for new builds. Running kati and then ninja
results in builds starting about 25% faster than GNU make.
2. Much faster incremental builds. kati caches the mtimes of files
and directories it read, what environment variables it used, etc., and
only re-reads all the makefiles if something has changed. This
includes globs like $(call all-files-under). Incremental builds with
a warm disk cache will start within 5 seconds.
3. Better build command output. Ninja buffers the output of each
command and prints it without mixing output from multiple commands,
meaning error messages will be printed as a group after a build
failure, and not mixed with other concurrent build output. Ninja also
prints a configurable status line that defaults to printing the
percentage of build commands that have been run, the number that have
been run, and the number that need to run, giving a reasonable
approximation of the progress through the build.
4. Incremental builds should be more reliable. Ninja considers
outputs whose command line has changed to be dirty, which will catch
changes to global cflags, etc. and rebuild anything that has changed.
The use of ninja should be transparent, it is wrapped by the top level
Makefile, so all make-based commands should continue to work.
These changes are a precursor to a much larger change that will
replace all the Android.mk files with data files using a format that
provides much better error checking, even faster builds, and 100%
reliable incremental builds.
Google groups, in Chrome at least, requires you to sign in if you have a google account cookie (signed in to any google account at some point), but are currently not logged in. This occurs unless you specifically run google groups in a new incognito window.
Google groups has to be one of my most hated sites. It's always so slow, so hard to read and the forced sign in thing is so annoying. I groan internally every time I get taken there.
There is some good content on there, but it seems like it's going the way of google code (which could have been github if google had played it right).
I appreciate the sentiment, but now I have to side-scroll to read ;-)
Maybe I'm just using an unusually large font for reading hn, but I find that <pre>/indented/code blocks on hn are best limited to around 50 character width - preferably a little less - to avoid scrollbars.
I get at least 109 characters in my viewport (eg: your full post above). I get about 50 characters in the <pre>-box that hn helpfully renders for code/blocks.
Colin's reply to the followup question about the last paragraph is interesting:
On Wed, Nov 4, 2015 at 4:54 PM, Colin Cross (ccr...@android.com) wrote:
>On Wed, Nov 4, 2015 at 4:29 PM, Dmitry Blotsky <dmitry....@gmail.com> wrote:
>> These changes are a precursor to a much larger change that will
>> replace all the Android.mk files with data files using a format that
>> provides much better error checking, even faster builds, and 100%
>> reliable incremental builds.
>>
>> Are you referring to the ninja format, or to something else?
>
>Android.mk files will be replaced with Blueprints files. For an
>in-progress example, see
>https://android.googlesource.com/platform/bionic/+/master/libc/Android.bp.
Blueprint takes build descriptions written in Go and generates ninja files from them:
Can anyone summarise the benefits of this build system vs make? According to the manual, just speed (why didn't they just work on making "make" faster?)
With another four years of perspective, I think the things Ninja does could not have been added to make without forking Make (perhaps even the build file syntax), at which point there's little point in starting with the Make code. For example Ninja builds in parallel by default (like the -j flag to Make); you couldn't make that change in Make because existing users have builds that don't work in parallel.
(Disclaimer: Ninja author, no involvement in the Android work.)
Basically, the way most projects use make (recursive make) creates a whole lot of problems. This can be fixed while still using make, but if you're already revamping your entire build system, you might as well move to something that offers additional niceties like the ones mentioned in the article: better error messages, progress report, machine readable output, etc.
I like this: "Ninja buffers the output of each
command and prints it without mixing output from multiple commands, meaning error messages will be printed as a group after a build failure, and not mixed with other concurrent build output."
To me, the most important feature of Ninja is that it can still track dependencies when a command produces multiple output files.
In Make, if you have a step with multiple outputs, it seems the usual thing to do is to lose track of all but one of them, and instruct the developer to run "make clean" when things stop working.
how long until they get it to build properly i wonder? XD
sorry... just can't help but poke fun at these tools i've never needed except in the linux environment.
every time i get something from google and need to build it its a nightmare of poor build experience. good to know they are making an effort to improve things.
I agree that the GP wasn't .. coherent. But I don't understand your comment. msbuild resides in (for example) C:\Windows\Microsoft.NET\Framework64\v4.0.30319
What would you need to download? It doesn't seem to be part of VS if that's what you were suggesting.
That version of msbuild isn't fully compatible with the one that lives in VS's folder, even when VS is installed. I've had it fail when trying to use it to compile stock projects made by Visual Studio, without anything fancy in the scripts.
have you ever tried to build webrtc, for example? (its the worst case i've hit tbh, not all of google's stuff is /that/ bad)
its a nightmare. some of the dependencies are utterly horrible (systemd!!!!, old deprecated MS headers from VC6) it seems to be custom scripts called from one build system that trigger another... its below the par of what i would expect from an absolute novice.
i hear that internally its a lot easier, because there is infrastructure that downloads all these things and sorts it out for you... but from the outside world its rubbish. i'd rather have a hand crafted visual studio project, xcode project, makefile where i can do one thing and not have to understand the insides of a 'black box' library to use it for anything at all.
generally i use whatever tools come with the standard ide for the platform. so yes, msbuild, if i need to do a build from a command line for protecting my main branch or something like CI. on mac its xcodebuild, on linux i tend to use make.
i've dabbled with cmake but it doesn't solve the problems i care about the most (needing to specify the files to build by hand is a headache - but it saves me having to do it 7 or 8 times at least), for which i write my own script...
i strongly disagree with the command line tools mentality. its not the 1970s any more. this is why i say "one click" build. not "one command" build. pressing any number of keys is too much work. XD
Don't forget Chromium which itself has two meta build systems (gyp and gn - similar to bazel/cmake/etc.) along with an underlying build system (ninja) which can target multiple compilers and runtimes on different platforms.
P.S. Chromium is in the middle of a transition from gyp to gn[1] which is why it has two build systems.
47 comments
[ 5.3 ms ] story [ 98.0 ms ] threadThere is some good content on there, but it seems like it's going the way of google code (which could have been github if google had played it right).
Maybe I'm just using an unusually large font for reading hn, but I find that <pre>/indented/code blocks on hn are best limited to around 50 character width - preferably a little less - to avoid scrollbars.
Try using phone and you will get his point.
The first one has text size comfortable for reading, but it is not 80c.
The second one fits the width of the text, but is not fine by any stretch of imagination.
(The device has 5.2" display).
https://github.com/google/blueprint/
Can anyone summarise the benefits of this build system vs make? According to the manual, just speed (why didn't they just work on making "make" faster?)
The short answer is that Ninja is conceptually similar to Make, but it dropped slow features to speed things up.
This old post from when Ninja was first released discusses "why not just improve make": http://neugierig.org/software/chromium/notes/2011/02/ninja.h...
With another four years of perspective, I think the things Ninja does could not have been added to make without forking Make (perhaps even the build file syntax), at which point there's little point in starting with the Make code. For example Ninja builds in parallel by default (like the -j flag to Make); you couldn't make that change in Make because existing users have builds that don't work in parallel.
(Disclaimer: Ninja author, no involvement in the Android work.)
Basically, the way most projects use make (recursive make) creates a whole lot of problems. This can be fixed while still using make, but if you're already revamping your entire build system, you might as well move to something that offers additional niceties like the ones mentioned in the article: better error messages, progress report, machine readable output, etc.
In Make, if you have a step with multiple outputs, it seems the usual thing to do is to lose track of all but one of them, and instruct the developer to run "make clean" when things stop working.
sorry... just can't help but poke fun at these tools i've never needed except in the linux environment.
every time i get something from google and need to build it its a nightmare of poor build experience. good to know they are making an effort to improve things.
one click build ffs.
What would you need to download? It doesn't seem to be part of VS if that's what you were suggesting.
have you ever tried to build webrtc, for example? (its the worst case i've hit tbh, not all of google's stuff is /that/ bad)
its a nightmare. some of the dependencies are utterly horrible (systemd!!!!, old deprecated MS headers from VC6) it seems to be custom scripts called from one build system that trigger another... its below the par of what i would expect from an absolute novice.
i hear that internally its a lot easier, because there is infrastructure that downloads all these things and sorts it out for you... but from the outside world its rubbish. i'd rather have a hand crafted visual studio project, xcode project, makefile where i can do one thing and not have to understand the insides of a 'black box' library to use it for anything at all.
i've dabbled with cmake but it doesn't solve the problems i care about the most (needing to specify the files to build by hand is a headache - but it saves me having to do it 7 or 8 times at least), for which i write my own script...
i strongly disagree with the command line tools mentality. its not the 1970s any more. this is why i say "one click" build. not "one command" build. pressing any number of keys is too much work. XD
It might be because Google is a big company doing many things in many languages, Bazel was only open sourced recently, and so on.
P.S. Chromium is in the middle of a transition from gyp to gn[1] which is why it has two build systems.
[1] https://chromium.googlesource.com/chromium/src/+/master/tool...