Misses the fundamental point that Make is broken for so many things. To begin with you have to have a single target for each file produced. Generating all the targets to get around this is a nightmare that results in unreadable debug messages and horribly unpredictable call paths.
nix tried to solve much of this, but I agree it can't compete with the bazillion other options.
It does not miss it, just ignores it. The author states that there are lots of things we can improve but the point is that we have too many variations on the theme without converging to a solution that has few (or no) dependencies and comes with built-in build knowledge and the ability to discover what you want rather than make you declare it.
Such a tool should be:
- Zero (or few) dependencies. Likely written in plain C (or C++, D, Rust) and compiled to distribute in binary form.
- Cross-platform
- Support any mix of project languages and build tasks.
- Recognizes standard folder hierarchies for popular projects.
- Easy enough to learn. Not overly verbose (looking at you, XML). Similar to Make if possible.
Examples of the auto-discovery: It can find "src", "inc", and "lib" directories then look inside and see .h files then make some educated guesses to build the dependency tree of header and source files (even with mix of C and C++). Or it could see a Rails app and figure out to invoke the right Rake commands, perhaps checking for the presence of an asset pipeline etc. Or a Node.js project. It could check for GIT or SVN and make sure any sub-modules have been checked out.
The dependencies thing is a killer. I remember a Windows developer co-worker insisting that everyone had the .NET runtime installed, and after shipping it turned out that most of our customers didn't have it installed, to which he finally said, "well, I always have it installed." (To be fair, I should have pressed him harder, and I did ask the question twice, but because I'd never built against the runtime I was unprepared for any challenge.)
Almost every new project I download starts with a sad, manual, and demoralizing installation of a bunch of third-party stuff that you have to google to find out what's missing. And it's not educational at all, because in a few years all these tools will now be obsolete.
(The best project I ever encountered was the Stripe CTF, which almost always used just one command to install a complete working copy of everything you needed and didn't have. I'm still impressed with that.)
Some of these requirements should be built into any build tool. However, most can be added easily enough:
For instance, redux [https://github.com/gyepisam/redux] is written in Go (not compiled for binary distribution, but I could add that), is cross platform, supports any mix of languages and tasks, is very easy to learn.
It uses shell scripts to create targets so everything is scriptable.
Stuff like recognizing standard folder hierarchies and auto-discovery can be added with small scripts or tools.
It can be as simple as you want or as complex as you need.
Could you post an example of what you mean by the single target/file limitation? As stated I can't tell how implicit rules or a rule to build an entire directory wouldn't be a solution, but maybe I'm not understanding the problem.
Sure, consider a compiler that produces an (foo.o) object file and an annotation (foo.a). Now if a target requires both foo.o and foo.a you have to create two targets on them (even though its really one command).
You can do implicit rules which requires a very verbose makefile, which is what automake and other make generation tools do. God help you figure out what went wrong.
If you make people go to a directory approach you've now imposed a new structure on their code. One reason for the multitude of packages is each one matches their target community better.
If both the .o and the .a are created from another file, wouldn't it be safe to just rely on either one of them? (Obviously, you will need to be consistent in choice.)
That is, if every time a .o is created, so is the .a, then where is the difficulty? Just rely on one (the .o). I could conceive of a scenario where the .a updated but the .o didn't, but I don't know of any tools that really work that way right now. I thought the norm was to at least touch all output files.
Further, if that is happening, seems you are safest having two rules, anyway.
Say you have a long build process and do a quick semi-clean by hand to speed up the next buld (not the best idea, but not inconceivable), deleting the .a files, but fogetting to delete the matching .o files. Then, your next build will produce some novel (to you) error messages that may take long to clean up. Worse, the command building on the .o and the .a might just say "OK, I'm given a .o without a .a; fine, then I'll do a slightly different thing"
Also, having two rules means duplicating a command:
Invoking the command twice can also screw up things if you run parallel build, which you should always do! Not only to speed things up it's also a good way to verify that your make file actually is correct. If your make file doesn't work in parallel build it is broken, in the same way as C code that breaks at -O2 and above due to reliance on undefined behavior.
The solution to the multiple target problem is using the built in magic .INTERMEDIATE rule which isn't entirely obvious how it works.
Ok, that makes sense. I'm tempted to rattle the knee jerk, "don't go deleting random crap," but I realize that is a hollow response.
I'm curious how .INTERMEDIATE helps in this case. I did find this link[1], which was a rather fun read down how one might go about solving this, along with all of the inherent problems.
Or if you don't like taeric's suggestion you can just touch a .ao file after the line that creates the .a and .o files and have your further rule(s) depend on that .ao file. Have .ao depend on your source. If you still want to be able to type stuff like 'make foo.a' instead of 'make foo.ao' and have it work, then you can make a rule where .a depends on .ao and all the rule does is touch the .a file. Create the same rule for the .o too.
The target baz has both a .l and a .o, both of which are produced in one command. The line that begins with "%.o" starts an implicit rule, which loosely states, in English: "to produce a .o file, or a .l file, run the following ...". $(*F) is a GNUism that maps to the filename of the source (directory part, if any, is stripped). This works. I tested all three targets (foo, bar, baz) with a "make clean" between each one.
(and for the really curious, a09 is a 6809 assembler; disasm.a is a 6809 diassembler, written in 6809; binary is a 2K relocatable library)
The third rule simulates a compiler producing two outputs. Now if foo.o changes, both "copied" and "o" will be updated, and if foo.a changes, both "copied" and "a" will be updated. (And if either foo.o or foo.a are deleted, the compiler will be rerun, as will everything depending on foo.a or foo.o.)
> To begin with you have to have a single target for each file produced.
Try this next time (only the pertinent lines are included):
SOURCES=$(wildcard $(SRCDIR)/*.erl)
OBJECTS=$(addprefix $(OBJDIR)/, $(notdir $(SOURCES:.erl=.beam)))
DEPS = $(addprefix $(DEPDIR)/, $(notdir $(SOURCES:.erl=.Pbeam))) $(addprefix $(DEPDIR)/, $(notdir $(TEMPLATES:.dtl=.Pbeam)))
-include $(DEPS)
# define a suffix rule for .erl -> .beam
$(OBJDIR)/%.beam: $(SRCDIR)/%.erl | $(OBJDIR)
$(ERLC) $(ERLCFLAGS) -o $(OBJDIR) $<
#see this: http://www.gnu.org/software/make/manual/html_node/Pattern-Match.html
$(DEPDIR)/%.Pbeam: $(SRCDIR)/%.erl | $(DEPDIR)
$(ERLC) -MF $@ -MT $(OBJDIR)/$*.beam $(ERLCFLAGS) $<
#the | pipe operator, defining an order only prerequisite. Meaning
#that the $(OBJDIR) target should be existent (instead of more recent)
#in order to build the current target
$(OBJECTS): | $(OBJDIR)
$(OBJDIR):
test -d $(OBJDIR) || mkdir $(OBJDIR)
$(DEPDIR):
test -d $(DEPDIR) || mkdir $(DEPDIR)
I've been using a makefile about 40 lines long and I've never needed to update the makefile as i've added source files. Same makefile (with minor tweaks) works across Erlang, C++, ErlyDTL and other compile-time templates and what have you. Also does automagic dependencies very nicely.
> Generating all the targets to get around this is a nightmare that results in unreadable debug messages and horribly unpredictable call paths.
If you think of Makefiles as a series of call paths, you're going to have a bad time. It's a dependency graph. You define rules for going from one node to the next and let Make figure out how to walk the graph.
Dunno if the owner of the site will read this, but here's a tip. Don't show a full screen overlay telling me how my visit would be better with cookies enabled.
1) I have cookies enabled.
2) The Eurpoean law is daft, but since you feel you must comply do it in a more user friendly way.
Note that this doesn't exist. Only local law could force sites to implement this, and afaik only the UK had a defunct specification of something like that they did'nt even follow for government pages. So, just the usual law craze.
> Is there a nice JS lib to detect and put up an unobtrusive hover-over popup in the bottom corner or something?
Hmm, maybe the overlay changed since you posted (11 minutes ago), but all I see is a sticky-slim footer with the cookie mention.
I've always liked how rockpapershotgun.com does it...it also uses a sticky-slim footer, but the text reads: "Rock Paper Shotgun uses cookies. For some reason we are now obliged to notify you of this fact. Not that you care"
I know. I don't agree with it much either and I found this least intrusive (wasn't aware of issue on mobile). If I can find a better solution, will change.
It is bloated, difficult to read, tends towards duplication. It also doesn't do dependency management all that well, doesn't cache build results (so it does a complete rebuild every time), and is difficult to extend.
Not an Ant fan. I have used both Rake and Gradle successfully, and have been much happier with each. Their scripts tend to be (much) more compact, easier to read, and less prone to duplication.
I agree with a lot of your points, but can you explain the part about it doing a complete rebuild every time? It doesn't do that for me (unless I specifically tell it to).
My apologies: I just wrote a basic HelloWorld.java and a build.xml to go with it, and it looks like it doesn't recompile the class unless there is a change to the source .java file. So I was mistaken about that.
I'm currently working on a build tool that doesn't work using the traditional "makefile" approach. Instead it's designed as a Python library, and you have the full power of Python at your disposal.
Sadly it's not ready for prime-time yet (early designing stage), so I won't link my highly unfinished project.
I suspect your downvotes are coming because you haven't mentioned http://www.scons.org/. Letting you know in case you're not aware of it. (I would have emailed you out-of-band, but there's no contact info in your profile. Sorry for the noise, everyone else)
Scons uses Python more for configuration rather than the control flow. I'm sure most people think that's ok, but I find it limiting and a bit too derpy.
In fairness that's the core of the argument: a significant proportion of people that have had to maintain build tools over long periods believe they should only be configuration and not contain any control flow.
The problem with adding flow control is that establishing the dependencies without actually executing the whole process becomes next to impossible.
So he's saying use 'make' instead of gulp/grunt and then he submits an example where it's as easy as piping through GCC.
He's making the wrong assumption that you don't need to setup a build environment when building with make, but to have gcc you will still also need to install g++ and build-tools. Also, he refers to building on Windows using yet another specialist tool, but have you recently tried building anything C++ on Windows when you don't have visual studio, or even worse, CYGWIN installed?
When you make such a statement, then please show me a makefile that's not 10.000 lines long, that will do the same as this, but without NPM and 'downloading half of the internet'.
gulp.task('scripts', function() {
// Minify and copy all JavaScript (except vendor scripts)
return gulp.src(paths.scripts)
.pipe(coffee())
.pipe(uglify())
.pipe(concat('all.min.js'))
.pipe(gulp.dest('build/js'));
});
I submit that's impossible, simply because this stuff took 2 years to evolve (for the Javascript toolchain, that is) and a lot of people went through hours of frustration trying out alternative methods.
Indeed. Responding to the fetishization of web technologies by fetishizing Unix utilities is missing the point. They're both tremendously important and tremendously useful, but trying to ignore their shortcomings doesn't help anything.
I think he's saying that re-implementing make in ruby is a bad idea.
Not that I agree with him. Being able to do some printf debugging (or even use a real debugger!) to troubleshoot issues is a big plus that he doesn't mention.
I'm not that familiar with grunt. Can it do the equivalent of make -j4?
PS. I'll happily agree that collecting the list of script files to operate on -- and the list of script files after the transformation (eg $(UGLY)) -- is slightly annoying in make.
This is Make 001: $@ is your output file, $^ are your inputs (prerequisites, technically), and $< is your first input file. They're unfortunately ungoogleable, but just remember that they're called "Automatic variables" in the GNU make documentation.
"-j4" means to run up to 4 jobs in parallel. It's orthogonal to the issues at hand.
-j4 says use up to 4 processes in parallel. Make has a jobserver that does parallelism safely with respect to the dependency graph. So for example, even if we have workers to spare, it won't try to build all.min.js until all of the .ugly files have finished building.
The %.foo : %.bar things are pattern rules. [1]
The $<, $@, and $^ things are called automatic variables. [2] They correspond to the first input file for a rule, the target file for a rule, and the complete list of input files for a rule, respectively. Cryptic at first but really handy.
There are 42,000 issues filed for make. if you could resolve each of these issues in 10 minutes, you'd spend 291 days of frustration.
http://savannah.gnu.org/bugs/?group=make
make is a general-purpose tool for describing dependencies for regenerating files. It would be worth your while to learn make, and try it on your example. Understand that it's a declarative language ("A depends on B"; when "B" changes, here's how to update "A"), and not a scripting tool. This is a good thing.
In my experience, make is coupled to Unix. Make is not coupled to C.
On one hand, make typically comes with built-in rules for .c targets.
On the other hand, make can't cleanly handle #include dependency detection. I doubt that there is any major C project where "make extraclean" (or its equivalent) isn't occasionally necessary.
Well, #include resolution would require that make be able to parse C, which would add a heck of a lot of complexity, and be unscalable. For instance, you'd need to modify make to parse CSS to teach it about @import, or to parse javascript to teach it about require() (but only if you're using RequireJS)
Or, you could use the C preprocessor option "-M" and its variants[0] to get it to generate make rules with C #include resolution for you.
See also Recursive Make Considered Harmful[1] for a good description on how to set up this in combination with GNU make's "include" facility to autogenerate your per-source #include resolution fragments.
In my experience, make is coupled to Unix. Make is not coupled to C.
However, implementations typically include magic that is heavily biased toward building C and C++ code. As someone who works on a lot of projects, some using those languages and some not, I tend to think it's rather too magical at times. Personally, I'd prefer to have that kind of magic explicitly stated in some standard file that comes with the tool, so that file can be included with a one-liner for those projects that want it but there is nothing implicit going on by default.
_make_ was first released in 1977, but that was the PWB/UNIX version. The GNU variant of make didn't come along until sometime in the 1980's. It's hard to pin down the exact date of the first release because the developers didn't keep good records prior to the switch from RCS to CVS. However, the earliest ChangeLog entry now is dated July 15, 1988, and 1985 is the earliest date mentioned in any copyright statement in any of the GNU make source files.
Quite the contrary: I've had to maintain or incorporate some non-standard build processes into several build systems (mostly scons, waf, gyp), and in each one it was difficult or impossible to express what I wanted because the build system wasn't as general as Make.
The single most valuable thing Make has going for it is that the primitive is a Unix pipeline. Anything that can be built with tools you can invoke from your shell can be built with Make, and the language of action is about as universal as it gets for Unix-like systems.
Yes, the dependency syntax is somewhat confusing, and it's not obvious how to understand and debug Makefiles at first glance, but the GNU make documentation is decent, and time spent learning the language and the tools (e.g., "make -d") is much better than trying to reinvent the system, especially without understanding it. Every reinvention I've had the displeasure of working with missed some important (if not widespread) use case.
Have you ever wished there was one-character symbol for the word "pipe", like maybe '|'? Such a symbol would even make all those .'s and ()'s redundant. While we're at it, if our build script is going to be in its own specially-named file, wouldn't it be nice if instead of namespacing under 'gulp', within the special build script file there was a DSL where you could specify the task name and its dependencies with a single character, like ':'? And instead of 'function() { return ...; }', your instructions were delimited with just indentation, like a Tab character?
Your example proves the opposite of the point you're trying to make. Starting from your example and trying to compress it with a DSL, you literally couldn't do better than Make syntax: the "gulp.task('" part is implicit, the "', function() { return gulp.src(" part is a single character (':'), every ").pipe(" is a single character ('|'), and ").pipe(gulp.dest('...'))" is a single character ('>').
I submit that's impossible, simply because this stuff took 2 years to evolve...
Before and during the entirety of those 2 years Make has been a better tool, for those of us JS coders who didn't dismiss it offhand as being always thousands of lines and only for dinosaur C coders.
Make has many problems, but taking thousands of lines to simply pipe together commands has never been one of them. Having to write .pipe() where in shell you could just do '|' has never been one of them, either.
No. You don't. But EU laws requires that if you use Cookies for things such as Google Analytics or even just stats, you must inform the user. And you can switch it off.
[Edit: Note that these are heredoc examples showing how to create the do scripts.]
These are just shell scripts and can be extended as much as necesary.
For instance, one can create a dependency on the compiler flags with these changes:
cat <<EOF | install -m 0755 /dev/stdin cc
#!/bin/sh
g++ -c "\$@"
EOF
# sed -i 's/^\(redo-ifchange.\+\)/\1 cc/' *.do
# sed -i 's}g++ -c}./cc}' *.do
sed calls could be combined; separated here for readablility.
CXX=g++ isn't necessary either; make already knows about $(CXX) and how to link C++ programs. Also, I think you wanted .o, not o.
And compared to that Makefile, the redo scripts you list don't seem simpler at all. I've seen reasonably compelling arguments for redo, but that wasn't one.
> CXX=g++ isn't necessary either; make already knows about $(CXX) and how to link C++ programs.
You're right, of course.
> Also, I think you wanted .o, not o.
I would, yes, but I copied the Makefile ;)
Should have been clearer; I meant that redo is simpler (and more reliable) than make.
For simple projects, redo scripts are a bit longer. However, as the projects grow, the redo scripts reach an asymptote whereas Makefiles don't.
The only way to reduce the growth in make is to add functions and implicit rules
which get ugly real fast.
redo might be simpler and more reliable, but shell isn't. And redo is encouraging even more work to be done in shell. Additionally, the redo version is more verbose and harder to read. While fancier tasks will make's version look horrible relatively quickly, they won't make redo's version look any better.
I think the big difference between redo and make is that make requires knowledge of dependencies up front, and this is sometimes tricky to get right.
"as you can see in default.o.do, you can declare a dependency after building the program. In C, you get your best dependency information by trying to actually build, since that's how you find out which headers you need. redo is based on the following simple insight: you don't actually care what the dependencies are before you build the target; if the target doesn't exist, you obviously need to build it. Then, the build script itself can provide the dependency information however it wants; unlike in make, you don't need a special dependency syntax at all. You can even declare some of your dependencies after building, which makes C-style autodependencies much simpler."
It's actually quite simple. You write a short shell script to produce the output you need and redo handles the dependencies.
For example, the shell script named "banana.x.do" is expected to produce the content for the file named "banana.x".
When you say
# redo banana.x
redo invokes banana.x.do with the command:
sh -x banana.x.do banana.x banana XXX > ZZZ
so banana.x.do is invoked with three arguments and its output is redirected to a file.
$1 denotes the target file
$2 denotes the target file without its extension
$3 is a temp file: XXX, in this case.
banana.x.do is expected to either produce output in $3 or write to stdout, but not both. If there are no failures redo will chose the correct one, rename the output to banana.x and update the dependency database.
If banana.x depends on grape.y, you add the line
redo-ifchange grape.y
to the banana.x.do, creating a dependency.
redo will rebuild grape.y (recursively) when necessary.
The only other commands I haven't mentioned are init and redo-ifcreate, which are obvious and rarely used, respectively.
> redo might be simpler and more reliable, but shell isn't.
Not quite sure what you mean here. The scripts don't do anything complicated and redo catches errors that could occur.
As for readability, etc, I suppose it's relative.
Simple makefiles do read very nicely. Unfortunately, they aren't always simple and hairy makefiles are just horrible to write, read and maintain. I've had no such problems with do scripts.
In general, I found CMake quite useable for my needs, and quite clean. It also required less build system code than redo. CMake fits quite nicely into a (C or C++) project which consists of many binaries and libraries which can depend on each other.
One big advantage of vanilla Make is the community. There are some very nice tools that work well with make (such as https://github.com/mbostock/smash).
I've recently been playing with ninja, which does a good job of not being 'just another make' http://martine.github.io/ninja/. To quote their website, "Where other build systems are high-level languages Ninja aims to be an assembler.". It's used as a backend for GYP (Google Chromium) and is supported by CMake as well. I've had good success generating the files manually using something like ninja_syntax.py: https://github.com/martine/ninja/blob/master/misc/ninja_synt....
Heh. The one and only time that I ever wrote a parser in my professional career was for a build tool. In my defence, at the time I didn't know much about command line tools, and had only really programmed in IDEs. So when the new project was to be compiled on the command line, I quickly discovered that maintaining dependencies, changing targets and doing all the other things that a build system generally does by hand quickly gets old. Not knowing that autotools, cmake, ant, and about a bajillion other tools already existed to do just this, I wrote my own language, with a parser in ruby, no less :D
I have since repented. I find autotools (with occasionally a script of [ruby|python|perl] to handle something that would otherwise be tricky to do in make or m4, which is then called by the makefile) works a treat. Just don't try to do anything tricky in the auto tool files - as I said, boot anything exotic out to a separate tool.
Also, any discussion of build tools without also discussing package management is but half a discussion.
At least we are moving the direction of Grunt/Gulp rather than a maven sort of direction. Many lives lost to maven, somewhat of a Vietnam of build tools. You might think you are a Java developer with it but truly you are a maven servant.
the last 10 years in build tools has felt like 1 step forward, two steps back. i like being able to write tasks in any language other than Makefile. however, it seems like many of the new popular options (cake, grunt, etc.) don't do what, to me, is Make's real purpose: resolve dependencies and only rebuild what's necessary. new task runners have either eliminated or pigeonholed the (typically one-to-one in makeland) correspondence between tasks and files, meaning the build system can't be intelligent about what tasks to run and which to not.
computers are fast enough that this doesn't often bother me anymore, but i've run across some huge Rakefiles that could benefit from a rewrite in Make.
> however, it seems like many of the new popular options (cake, grunt, etc.) don't do what, to me, is Make's real purpose: resolve dependencies and only rebuild what's necessary.
You might like tup[1]. Its killer feature is that it automatically determines file-based dependencies by tracking reads and writes (using a FUSE filesystem). It has an extreme emphasis on correct, repeatable builds, and is very fast. Other stuff:
- does work in parallel, and will let you know if your build isn't parallel safe. (note it is NOT relying on your specification of dependencies: even if you manually specify dependencies, it will tell you if something's wrong based on what it actually observes your dependencies to be)
- tracks changes to the build script and reruns if the commands change.
- cleans up old output files automatically if build rules are removed.
- lets you maintain multiple build variants (say for different architectures, configurations, etc)
- autogenerates .gitignore files for your build output
- very easy to get started, and "Just Works".
- for advanced usage, it is scriptable in Lua.
I've tried every build system out there. For Unix-y file-based build tasks, tup is, by far, the best. I don't know why it isn't more well known.
+1 for the link. I hadn't seen tup before, and I really like how it feels like make in its simplicity, but is more explicit about the graph inputs and outputs, cares more about the output (e.g. deleting old files), and watches the file system for changes.
I was already sold on tup after reading the first paragraph comparing it to make[1]:
"This page compares make to tup. This page is a little biased because tup is so fast. How fast? This one time a beam of light was flying through the vacuum of space at the speed of light and then tup went by and was like "Yo beam of light, you need a lift?" cuz tup was going so fast it thought the beam of light had a flat tire and was stuck. True story. Anyway, feel free to run your own comparisons if you don't believe me and my (true) story."
I think MSBuild does something like this - there is a filetracker that tracks what files were read/written while running a tool, and writes that information in a file. I think you can even install your own file-notification-changes .dll to track changes your way (maybe file system that is not supported, or something else).
I agree completely, and I think the blame rests with Ant and Java. Java's dependency management was painful enough to deal with in 'make' that Ant was built to support building Java projects. But in doing so the authors threw away the explicit file dependencies that made 'make' so powerful in the first place. Instead it got people to think in terms of a graph of 'tasks', each of which could either figure out its own dependency management, or more commonly ignore them completely. Most tools that followed seem to have gone down the 'graph of tasks' avenue, with 'graph of file dependencies' as an additional mechanism if you're lucky.
The huge Rakefiles you've seen could possibly have simply benefited from a rewrite in Rake. Rake has 'file' tasks which implement the file dependencies of 'make' but for some reason most users of Rake seem to ignore them completely.
It can be worse than that, one build system I looked at built everything every time. Why? Because "Computers are fast enough that trying to figure out exactly what needs to be rebuilt is an anachronism, this can rebuild everything in the time it took that crufty old system to figure out what it actually had to build."
I've given up trying to educate folks, I just make a note to check in with them, 6 months to a year later, to see if they are still building everything.
I think everyone goes through a phase where they try to find the perfect build tool, and then at least entertain the idea of writing one themselves.
Eventually, you grow out of it. There's a lot of build tools, each are better at some things than others. It's not that much grunt work to convert things from one to another (even very large projects). If your build tool is working for you, leave it alone. If it's getting in your way or slowing things down, try another one. Move on.
I find these posts somewhat amusing. We've got people who (rightfully) question the tools they use and look for alternatives. They then discover Make and have some kind of zen Unix moment that they want to share with the world.
If what you are doing in your flavor-of-the-month build tool translates to a roughly equivalent number of lines in Make, then yes, you should probably look at using Make. But the thing is, Make is stupid, it doesn't know a lot. Sometimes that is a good thing, sometimes it is not.
I've written about this before on HN: I mostly program in C++ and when I build my stuff I want a build tool that understands things like header dependencies, optimization levels, shared libraries etc. It's a bonus if my build files are portable.
My point is that these alternative tools often strive to raise the abstraction level and the reason people use them isn't necessarily because they haven't discovered Make.
It reminds me of the jQuery cycle: use jQuery for everything -> decide that depending on frameworks is lame -> use "vanilla JS" for everything -> realize this requires polyfills and various other inconvenient, inelegant things -> either go back to using jQuery, or gain a much deeper understanding as to why everyone uses it.
Make is not an amazing (abit slightly bloated) meta tool that solves all your problems on all platforms (abit slowly).
Make is vanilla javascript, along with all the bumps and hassles of not working correctly on multiple platforms, having odd obscure syntactic oddities and only kind of supporting various operations in newer versions (which may or may not be available on various platforms).
The newer build tools are trying to do exactly what jquery does, and abstracting away those rough edges for a consistent build behavior with better syntax.
Going back to make is the 'use "vanilla JS" for everything' step in your list above, not the final step.
Taking the analogy a bit further, a vanilla proponent might put Make in the second category as well. Funny that modern web development requires a build process to change a line of CSS.
Amusing indeed. (Functional) Reactive Programming [1] [2] anyone. That's the same thing we've learned during development to be profitable and of real use. And it seems that build systems also converge towards the same lesson learned, but slowly.
For me, the only justification for using a language-specific build tool (e.g. grunt, rake, paver, ...) is when you actually want to exchange data with a library / program written in that language. On the other hand, you could probably accomplish the same effect using environment variables, with the upside of having a cleaner interface.
For those that are curious which build tools exist for Python, here's an (incomplete) list:
* pyinvoke (https://github.com/pyinvoke) - claims to be the successor of fabric, pretty solid and well-maintained
Documentation can be challenging to find, and it isn't the most actively developed project in the world, but what it does, it does pretty well (including supporting more than python dependencies).
Thanks for posting the link! I hesitated including configuration management tools since the use case is not the same. There's a lot of interesting stuff going on there though: With Saltstack and Ansible we have two serious "chef" contenders for Python now.
I hate how far spread apart the JS community is but I'm excited for the day it starts to all come together and we don't have to worry about Javascript stacks becoming outdated within 6 months
Make is the "assembly" language for build systems. Qt's qmake/cmake target it, and the output produced is horrible, but then using "make -j8" or qt's own "jom.exe -j8" as replacement for the underperforming nmake.exe and you are all set.
Nothing against make, but I've found that it feels really nice when the majority of your toolset uses the same language. This is what I liked about Rails. Rails is ruby. Bundler is ruby. Rake is ruby. It's all ruby, which allows for a certain synergy, streamlined feel, and less cognitive overhead. I don't blame the js folks for attempting something similar.
Agreed. Mixing languages is fine when necessary, but a single language is usually preferable to me. My dev team has been using Grunt in projects thus far and have been pleased with Gulp in small experiments.
Might go a bit off topic but i have to bring this up since 9 out of 10 make tutorials on the internet do the same horrific mistake as you just did, 11 out of 10 code bases out in the wild as well.
In your make file example the .o files are just depending on the .cpp files, not the header files they include, the header files those included header files include and the files they include etc etc. This means nothing will be recompiled/relinked if a constant in a header file changes for example! Changed function signatures will give you cryptic linker errors with the standard solution "just try make clean first".
To solve this you can either manually update the make file every time any file changes the files it includes, which almost defeats the purpose of having an automatic build system. Or you can use automatic dependency generation by invoking your compiler with a special flag (-MMD for GCC), and suddenly make isn't as simple anymore as you laid it out to be. In conclusion your build tool must be aware of ALL inclusion rules as your compiler(preprocessor) has, or be given the information somehow. Maybe it's better to just use something designed for your particular toolchain that can come bundled with this knowledge?
Right. Make is mostly a kludge around the nonexistent module system in C and C++.
It's so bad (specifically due to the way file preprocessing works), that you need to have large parts of a compiler to accurately determine what the dependencies of a source file are.
This is why a decent module system should be the top priority for C++17, though it doesn't look likely so far.
Yes, it is. automake will generate makefiles that track dependencies as a side-effect of compilation ( http://www.gnu.org/software/automake/manual/automake.html#De... ). What this means is that whenever a source file is compiled it will update the dependencies for the .o file. It has to be done then because different platforms could have different header dependencies (if you're having fun with #define).
In fact I agree. I would be really glad if somebody explained me persuasively which one build-tools is the best one ever, so I could use it always and for everything, even when a shell-script would be enough. Yeah, it would be nice. But then, don't we have the same thing with about every class of software? Tens of text-editors and no perfect one. Many OSes and nothing sane. So many programming languages with overlapping functionality! And we won't even talk about such thing as linux distributions (and their packaging tools), pepsi&coca-cola… oh, it's not even software.
So, yeah, there are too many build-tools. Whatever.
142 comments
[ 5.4 ms ] story [ 183 ms ] threadnix tried to solve much of this, but I agree it can't compete with the bazillion other options.
Such a tool should be: - Zero (or few) dependencies. Likely written in plain C (or C++, D, Rust) and compiled to distribute in binary form. - Cross-platform - Support any mix of project languages and build tasks. - Recognizes standard folder hierarchies for popular projects. - Easy enough to learn. Not overly verbose (looking at you, XML). Similar to Make if possible.
Examples of the auto-discovery: It can find "src", "inc", and "lib" directories then look inside and see .h files then make some educated guesses to build the dependency tree of header and source files (even with mix of C and C++). Or it could see a Rails app and figure out to invoke the right Rake commands, perhaps checking for the presence of an asset pipeline etc. Or a Node.js project. It could check for GIT or SVN and make sure any sub-modules have been checked out.
Almost every new project I download starts with a sad, manual, and demoralizing installation of a bunch of third-party stuff that you have to google to find out what's missing. And it's not educational at all, because in a few years all these tools will now be obsolete.
(The best project I ever encountered was the Stripe CTF, which almost always used just one command to install a complete working copy of everything you needed and didn't have. I'm still impressed with that.)
For instance, redux [https://github.com/gyepisam/redux] is written in Go (not compiled for binary distribution, but I could add that), is cross platform, supports any mix of languages and tasks, is very easy to learn.
It uses shell scripts to create targets so everything is scriptable. Stuff like recognizing standard folder hierarchies and auto-discovery can be added with small scripts or tools. It can be as simple as you want or as complex as you need.
You can do implicit rules which requires a very verbose makefile, which is what automake and other make generation tools do. God help you figure out what went wrong.
If you make people go to a directory approach you've now imposed a new structure on their code. One reason for the multitude of packages is each one matches their target community better.
If both the .o and the .a are created from another file, wouldn't it be safe to just rely on either one of them? (Obviously, you will need to be consistent in choice.)
That is, if every time a .o is created, so is the .a, then where is the difficulty? Just rely on one (the .o). I could conceive of a scenario where the .a updated but the .o didn't, but I don't know of any tools that really work that way right now. I thought the norm was to at least touch all output files.
Further, if that is happening, seems you are safest having two rules, anyway.
Also, having two rules means duplicating a command:
That's bad from a maintenance perspective.The solution to the multiple target problem is using the built in magic .INTERMEDIATE rule which isn't entirely obvious how it works.
I'm curious how .INTERMEDIATE helps in this case. I did find this link[1], which was a rather fun read down how one might go about solving this, along with all of the inherent problems.
[1] http://www.gnu.org/software/automake/manual/html_node/Multip...
(and for the really curious, a09 is a 6809 assembler; disasm.a is a 6809 diassembler, written in 6809; binary is a 2K relocatable library)
This is gnu make 4.0.
Try this next time (only the pertinent lines are included):
I've been using a makefile about 40 lines long and I've never needed to update the makefile as i've added source files. Same makefile (with minor tweaks) works across Erlang, C++, ErlyDTL and other compile-time templates and what have you. Also does automagic dependencies very nicely.> Generating all the targets to get around this is a nightmare that results in unreadable debug messages and horribly unpredictable call paths.
If you think of Makefiles as a series of call paths, you're going to have a bad time. It's a dependency graph. You define rules for going from one node to the next and let Make figure out how to walk the graph.
1) I have cookies enabled. 2) The Eurpoean law is daft, but since you feel you must comply do it in a more user friendly way.
Note that this doesn't exist. Only local law could force sites to implement this, and afaik only the UK had a defunct specification of something like that they did'nt even follow for government pages. So, just the usual law craze.
> Is there a nice JS lib to detect and put up an unobtrusive hover-over popup in the bottom corner or something?
Well, I don't know if this is nice enough as I didn't use it, but it looks ok: http://sitebeam.net/cookieconsent/
I've always liked how rockpapershotgun.com does it...it also uses a sticky-slim footer, but the text reads: "Rock Paper Shotgun uses cookies. For some reason we are now obliged to notify you of this fact. Not that you care"
Thanks.
Nice to see it mentioned in a context other than "oh god what a mess"... even though, in fairness, many aspects of it are a complete dog's dinner.
That is horrifying.
It is bloated, difficult to read, tends towards duplication. It also doesn't do dependency management all that well, doesn't cache build results (so it does a complete rebuild every time), and is difficult to extend.
Not an Ant fan. I have used both Rake and Gradle successfully, and have been much happier with each. Their scripts tend to be (much) more compact, easier to read, and less prone to duplication.
Wonder what Gradle's caching, then?
Let's hope it's catching another scripting language or two in its upcoming version 2 because having Groovy as the only option does it no favors.
Sadly it's not ready for prime-time yet (early designing stage), so I won't link my highly unfinished project.
The problem with adding flow control is that establishing the dependencies without actually executing the whole process becomes next to impossible.
https://code.google.com/p/fabricate/
He's making the wrong assumption that you don't need to setup a build environment when building with make, but to have gcc you will still also need to install g++ and build-tools. Also, he refers to building on Windows using yet another specialist tool, but have you recently tried building anything C++ on Windows when you don't have visual studio, or even worse, CYGWIN installed?
When you make such a statement, then please show me a makefile that's not 10.000 lines long, that will do the same as this, but without NPM and 'downloading half of the internet'.
I submit that's impossible, simply because this stuff took 2 years to evolve (for the Javascript toolchain, that is) and a lot of people went through hours of frustration trying out alternative methods.Not that I agree with him. Being able to do some printf debugging (or even use a real debugger!) to troubleshoot issues is a big plus that he doesn't mention.
PS. I'll happily agree that collecting the list of script files to operate on -- and the list of script files after the transformation (eg $(UGLY)) -- is slightly annoying in make.
Also, what your example looks like voodoo. Could you explain what the parameters do?
On top of that, tools like uglifyjs still run on NPM/Node afaik.
"-j4" means to run up to 4 jobs in parallel. It's orthogonal to the issues at hand.
The manual is actually pretty good. I've only started to dig my teeth into make.
The %.foo : %.bar things are pattern rules. [1]
The $<, $@, and $^ things are called automatic variables. [2] They correspond to the first input file for a rule, the target file for a rule, and the complete list of input files for a rule, respectively. Cryptic at first but really handy.
There's a standalone script for UglifyJS. [3]
[1] https://www.gnu.org/software/make/manual/html_node/Pattern-M...
[2] https://www.gnu.org/software/make/manual/html_node/Automatic...
[3] https://github.com/mishoo/UglifyJS2/blob/master/bin/uglifyjs
GMake was first released in 1977: http://en.wikipedia.org/wiki/Make_(software)
They've worked on this thing for decades
There are 42,000 issues filed for make. if you could resolve each of these issues in 10 minutes, you'd spend 291 days of frustration. http://savannah.gnu.org/bugs/?group=make
kids these days.
In my experience, make is coupled to Unix. Make is not coupled to C.
On the other hand, make can't cleanly handle #include dependency detection. I doubt that there is any major C project where "make extraclean" (or its equivalent) isn't occasionally necessary.
So yeah it's really not very well suited for C.
Given it would be nice if make had a builtin macro to do this, but it is not too bad to type out.
depend: .depend
.depend: ${SRC} ${CC} -MM -I. ${SRC} > .depend.X && mv .depend.X .depend
include .depend
Makefile: .depend
I believe make parses the entire Makefile before running it.
Or, you could use the C preprocessor option "-M" and its variants[0] to get it to generate make rules with C #include resolution for you.
See also Recursive Make Considered Harmful[1] for a good description on how to set up this in combination with GNU make's "include" facility to autogenerate your per-source #include resolution fragments.
[0] http://gcc.gnu.org/onlinedocs/gcc/Preprocessor-Options.html
[1] http://aegis.sourceforge.net/auug97.pdf
However, implementations typically include magic that is heavily biased toward building C and C++ code. As someone who works on a lot of projects, some using those languages and some not, I tend to think it's rather too magical at times. Personally, I'd prefer to have that kind of magic explicitly stated in some standard file that comes with the tool, so that file can be included with a one-liner for those projects that want it but there is nothing implicit going on by default.
The rule applicable to C pertains to building foo.o from foo.c. It amounts to two lines in a Makefile:
where COMPILE.c and the other variable is predefined to trivial values.There are other rules useful for C++, and for yacc, etc., but they are just as simple.
You can cancel all the implicit rules with "-r", or cancel selected implicit rules by naming them in the Makefile like this:
All this is in the make manual, which is quite good: https://www.gnu.org/software/make/manual/html_node/Catalogue...The single most valuable thing Make has going for it is that the primitive is a Unix pipeline. Anything that can be built with tools you can invoke from your shell can be built with Make, and the language of action is about as universal as it gets for Unix-like systems.
Yes, the dependency syntax is somewhat confusing, and it's not obvious how to understand and debug Makefiles at first glance, but the GNU make documentation is decent, and time spent learning the language and the tools (e.g., "make -d") is much better than trying to reinvent the system, especially without understanding it. Every reinvention I've had the displeasure of working with missed some important (if not widespread) use case.
You mean like this?
Have you ever wished there was one-character symbol for the word "pipe", like maybe '|'? Such a symbol would even make all those .'s and ()'s redundant. While we're at it, if our build script is going to be in its own specially-named file, wouldn't it be nice if instead of namespacing under 'gulp', within the special build script file there was a DSL where you could specify the task name and its dependencies with a single character, like ':'? And instead of 'function() { return ...; }', your instructions were delimited with just indentation, like a Tab character?Your example proves the opposite of the point you're trying to make. Starting from your example and trying to compress it with a DSL, you literally couldn't do better than Make syntax: the "gulp.task('" part is implicit, the "', function() { return gulp.src(" part is a single character (':'), every ").pipe(" is a single character ('|'), and ").pipe(gulp.dest('...'))" is a single character ('>').
I submit that's impossible, simply because this stuff took 2 years to evolve...
Before and during the entirety of those 2 years Make has been a better tool, for those of us JS coders who didn't dismiss it offhand as being always thousands of lines and only for dinosaur C coders.
Make has many problems, but taking thousands of lines to simply pipe together commands has never been one of them. Having to write .pipe() where in shell you could just do '|' has never been one of them, either.
I need cookies to read a blog post? Don't think so. Probably not worth the read
These are just shell scripts and can be extended as much as necesary. For instance, one can create a dependency on the compiler flags with these changes:
sed calls could be combined; separated here for readablility.[1] https://github.com/gyepisam/redux
And compared to that Makefile, the redo scripts you list don't seem simpler at all. I've seen reasonably compelling arguments for redo, but that wasn't one.
You're right, of course.
> Also, I think you wanted .o, not o.
I would, yes, but I copied the Makefile ;)
Should have been clearer; I meant that redo is simpler (and more reliable) than make.
For simple projects, redo scripts are a bit longer. However, as the projects grow, the redo scripts reach an asymptote whereas Makefiles don't. The only way to reduce the growth in make is to add functions and implicit rules which get ugly real fast.
With make it was easier for me to grasp the idea (or maybe I was simply 20 years younger then).
"as you can see in default.o.do, you can declare a dependency after building the program. In C, you get your best dependency information by trying to actually build, since that's how you find out which headers you need. redo is based on the following simple insight: you don't actually care what the dependencies are before you build the target; if the target doesn't exist, you obviously need to build it. Then, the build script itself can provide the dependency information however it wants; unlike in make, you don't need a special dependency syntax at all. You can even declare some of your dependencies after building, which makes C-style autodependencies much simpler."
https://github.com/apenwarr/redo
For example, the shell script named "banana.x.do" is expected to produce the content for the file named "banana.x".
When you say
redo invokes banana.x.do with the command: so banana.x.do is invoked with three arguments and its output is redirected to a file. banana.x.do is expected to either produce output in $3 or write to stdout, but not both. If there are no failures redo will chose the correct one, rename the output to banana.x and update the dependency database.If banana.x depends on grape.y, you add the line
to the banana.x.do, creating a dependency. redo will rebuild grape.y (recursively) when necessary.The only other commands I haven't mentioned are init and redo-ifcreate, which are obvious and rarely used, respectively.
That's it.
Not quite sure what you mean here. The scripts don't do anything complicated and redo catches errors that could occur.
As for readability, etc, I suppose it's relative. Simple makefiles do read very nicely. Unfortunately, they aren't always simple and hairy makefiles are just horrible to write, read and maintain. I've had no such problems with do scripts.
In general, I found CMake quite useable for my needs, and quite clean. It also required less build system code than redo. CMake fits quite nicely into a (C or C++) project which consists of many binaries and libraries which can depend on each other.
What's special about the Make community as opposed to the Grunt or Gulp communities?
Hindered by the fact I can't add an arbitrary github repo through Gradle? Yes. That seems like it should be solvable though...
I also note that google are working on a successor to GYP, GN which targets Ninja http://code.google.com/p/chromium/wiki/gn.
I have since repented. I find autotools (with occasionally a script of [ruby|python|perl] to handle something that would otherwise be tricky to do in make or m4, which is then called by the makefile) works a treat. Just don't try to do anything tricky in the auto tool files - as I said, boot anything exotic out to a separate tool.
Also, any discussion of build tools without also discussing package management is but half a discussion.
computers are fast enough that this doesn't often bother me anymore, but i've run across some huge Rakefiles that could benefit from a rewrite in Make.
You might like tup[1]. Its killer feature is that it automatically determines file-based dependencies by tracking reads and writes (using a FUSE filesystem). It has an extreme emphasis on correct, repeatable builds, and is very fast. Other stuff:
- does work in parallel, and will let you know if your build isn't parallel safe. (note it is NOT relying on your specification of dependencies: even if you manually specify dependencies, it will tell you if something's wrong based on what it actually observes your dependencies to be)
- tracks changes to the build script and reruns if the commands change.
- cleans up old output files automatically if build rules are removed.
- lets you maintain multiple build variants (say for different architectures, configurations, etc)
- autogenerates .gitignore files for your build output
- very easy to get started, and "Just Works".
- for advanced usage, it is scriptable in Lua.
I've tried every build system out there. For Unix-y file-based build tasks, tup is, by far, the best. I don't know why it isn't more well known.
[1]: http://gittup.org/tup/index.html
http://gittup.org/tup/ex_a_first_tupfile.html
"This page compares make to tup. This page is a little biased because tup is so fast. How fast? This one time a beam of light was flying through the vacuum of space at the speed of light and then tup went by and was like "Yo beam of light, you need a lift?" cuz tup was going so fast it thought the beam of light had a flat tire and was stuck. True story. Anyway, feel free to run your own comparisons if you don't believe me and my (true) story."
[1]: http://gittup.org/tup/make_vs_tup.html
Similar to what lsof, procmon (windows) do.
The huge Rakefiles you've seen could possibly have simply benefited from a rewrite in Rake. Rake has 'file' tasks which implement the file dependencies of 'make' but for some reason most users of Rake seem to ignore them completely.
I've given up trying to educate folks, I just make a note to check in with them, 6 months to a year later, to see if they are still building everything.
Eventually, you grow out of it. There's a lot of build tools, each are better at some things than others. It's not that much grunt work to convert things from one to another (even very large projects). If your build tool is working for you, leave it alone. If it's getting in your way or slowing things down, try another one. Move on.
If what you are doing in your flavor-of-the-month build tool translates to a roughly equivalent number of lines in Make, then yes, you should probably look at using Make. But the thing is, Make is stupid, it doesn't know a lot. Sometimes that is a good thing, sometimes it is not.
I've written about this before on HN: I mostly program in C++ and when I build my stuff I want a build tool that understands things like header dependencies, optimization levels, shared libraries etc. It's a bonus if my build files are portable.
My point is that these alternative tools often strive to raise the abstraction level and the reason people use them isn't necessarily because they haven't discovered Make.
It reminds me of the jQuery cycle: use jQuery for everything -> decide that depending on frameworks is lame -> use "vanilla JS" for everything -> realize this requires polyfills and various other inconvenient, inelegant things -> either go back to using jQuery, or gain a much deeper understanding as to why everyone uses it.
Make is not an amazing (abit slightly bloated) meta tool that solves all your problems on all platforms (abit slowly).
Make is vanilla javascript, along with all the bumps and hassles of not working correctly on multiple platforms, having odd obscure syntactic oddities and only kind of supporting various operations in newer versions (which may or may not be available on various platforms).
The newer build tools are trying to do exactly what jquery does, and abstracting away those rough edges for a consistent build behavior with better syntax.
Going back to make is the 'use "vanilla JS" for everything' step in your list above, not the final step.
Make -> VanillaJS
Grunt/Rake/whatever -> jQuery
––
[1] http://en.wikipedia.org/wiki/Reactive_programming
[2] http://en.wikipedia.org/wiki/Functional_reactive_programming
For those that are curious which build tools exist for Python, here's an (incomplete) list:
* pyinvoke (https://github.com/pyinvoke) - claims to be the successor of fabric, pretty solid and well-maintained
* fabric (http://www.fabfile.org/) - not actually a build tool but often used as one
* paver (http://paver.github.io/paver/) - no longer actively maintained
* doit (http://pydoit.org/) - one of the few tools that actually support monitoring file states (like the original make)
* disttools (https://docs.python.org/2/distutils/) - not actually a "universal" build tool but intended to distribute Python packages
1. http://www.buildout.org/
Documentation can be challenging to find, and it isn't the most actively developed project in the world, but what it does, it does pretty well (including supporting more than python dependencies).
https://algorithms.rdio.com/post/make/
In your make file example the .o files are just depending on the .cpp files, not the header files they include, the header files those included header files include and the files they include etc etc. This means nothing will be recompiled/relinked if a constant in a header file changes for example! Changed function signatures will give you cryptic linker errors with the standard solution "just try make clean first".
To solve this you can either manually update the make file every time any file changes the files it includes, which almost defeats the purpose of having an automatic build system. Or you can use automatic dependency generation by invoking your compiler with a special flag (-MMD for GCC), and suddenly make isn't as simple anymore as you laid it out to be. In conclusion your build tool must be aware of ALL inclusion rules as your compiler(preprocessor) has, or be given the information somehow. Maybe it's better to just use something designed for your particular toolchain that can come bundled with this knowledge?
It's so bad (specifically due to the way file preprocessing works), that you need to have large parts of a compiler to accurately determine what the dependencies of a source file are.
This is why a decent module system should be the top priority for C++17, though it doesn't look likely so far.
[1] http://clang.llvm.org/docs/Modules.html
.depend :
then at the bottom:source .depend
Just a nitpick: did you mean `include .depend`?
You may need to do a little finagling to bootstrap .depend when adding to an existing make file: echo "garp" > .depend; make .depend : may suffice.
So, yeah, there are too many build-tools. Whatever.