Lol, I just made the argument that ‘a simple makefile can do most of what people use grunt for’ on Twitter like 6 days ago. Worth nothing however everything in our makefile is a PHONY which makes things a ton simpler, though arguably we’re not using make to it’s full potential.
You may want to consider using 'stamp files'. These are just empty files created via 'touch' to serve as a timestamp that Make can use to determine if a target needs to be rebuilt if the target doesn't correspond to an actual file in the build directory. This is useful for build targets such as running docker-compose build.
Sometimes it is necessary though. I have had to use this pattern in 2 scenarios:
Building a static website where the targets are wildly different from the source files. I could probably make it explicit but it will be overly verbose and brittle.
Check for and remove trailing whitespace. Because the dependency is actually the target, a stamp file is necessary.
Specifically in the second case: How do you get the timestamp on these stamp files to update, or do you just do it manually? And if you are doing it manually, why not just run a shell script instead?
The timestamp on the source files is updated by sed, when it strips the whitespace. The stamp file is then updated with touch. The advantage of using Make is that only modified files will be fed to sed and, consequently, only they will be caught by the rsync command, that I use to push the code out to the server(s).
When seeing "simple" as an absolute measure, the author surely has a point. As for a relative measure - I think the argument is unjust, as you can easily come across much more complex makefiles. So my take on this is this: explain the width of the scale you're using when you measure in terms relative to it.
The Makefile shown in the article is simultaneously not complicated and also over-complicated. I feel like most Make users don't understand Make very well (especially the ones that complain really loudly about it).
Here's a couple examples:
1. The "%.o: %.c" line is a huge code smell for Makefiles. You shouldn't ever write that line—it's built in!
2. Mentioning .h files in the Makefile. Don't. Go look up the `-MMD` flag in your C compiler. Clang, gcc and many other C compilers support it. Add that and a "-include *.d" line in the Makefile and you'll have great dependency support.
Everything else in that makefile is just setting variables. It's made overly complicated by the directory junk, but I'm failing to see the problem.
Because "-include" is a non-standard[1,2] extension. "include" is standard POSIX behavior[1] but (1) does not seem to support wildcards and (2) errors if the file does not exist, which it never will on initial build. From a glance at the man pages, OpenBSD make does support -include[2], but FreeBSD make does not[3].
Frankly at this point in time, when people say "Make" they mean GNU Make. Unix distributions that wish to remain stuck in the 1980s with their original BSD Make are welcome to do so, but no sane developer is going to use BSD Make for a build system written any time in the last 25 years unless they have to.
The first two lines tell us which compiler we're using and set up some extra compiler warnings. Then we specify which libraries we use (it's a matter of taste whether this is a variable or just spelled out later) and the names of our object files.
The first target, hellomake, is our executable. This is the default target, so it's the one that'll be built if we run make with no arguments. It runs the C compiler with the flags we specified earlier. Its dependencies are the object files, which means they'll be built automatically and rebuilt when their dependencies change. As noted, there is a built-in rule to build the object files from the C source using our CC and CFLAGS. We don't need to tell make how to do that.
The next target, clean, is a "phony" target—it doesn't refer to a file. Normally a make target tells make how to build a file. A phony target, in contrast, is really just a shell script that doesn't produce any file output. This one deletes all the build products.
The remaining lines set up the dependency tracking for headers. This makes it so that a change in a header file will trigger a rebuild of any source that includes it. We begin by making a list of files by copying the list of object files and swapping in ".d" for the ".o" file extension. Next is a rule that tells how to build a .d file from a .c file. This rule runs the C compiler and tells it to output a dependency tracking file. The last line says to include each of the .d files in the Makefile. (The .d files contain Makefile syntax to specify the header dependencies.)
I omitted the IDIR and ODIR and such because in a project of this size I wouldn't bother putting includes and object files in separate directories from the source.
I hope that helps. I know make can seem outdated and weird but I think it really can be simple once you learn a few things about how it works. Hopefully this will get you started in that direction!
> I feel like most Make users don't understand Make very well (especially the ones that complain really loudly about it).
But the question is whether this is Make's fault for being incomprehensible, or the User's fault for being lazy.
The thing is virtually any system or tool can be understood with enough practice, but just because there is someone out there who can use it well does not mean it's not poorly designed.
Both C and Make are decades-old technologies, but most people can read a C file without much experience and intuitively understand what is going on. The same is not true for Makefiles.
Sure but if all things were kept intuitive, well-specified and simple, no one would have paid us six-figs to maintain it. It's part of the guild protection.
You can also create a point and click gui mechanism and brag about how wonderful it is, too, but it wouldn't be 'make' or be as capable, well-rounded, flexible, encompassing, etc.
'make' exists for what it's capable of doing and it must be so as it's used in every piece of software and operating sytem everywhere.
You are then suggesting that make is not popular and widespread because it's "good". I am saying that, when given the choice, most operating systems and software use 'make' as evidence.
> So it's extremely popular/widespread, therefore it must be good, and therefore you should use it (thereby making it even more popular/widespread)?
My experience has been that that people that truly understand and appreciate technology for its own sake are also likely to underestimate the value of a tool being popular and widespread. What 'popular' really means is that the tool is more likely to be where you need it to be and the people you need to use it will already know how. Those things outweigh a 'perfectly' designed tool that you don't have or know how to use.
> What 'popular' really means is that the tool is more likely to be where you need it to be and the people you need to use it will already know how.
There's a balance point, for sure. For instance, the pace of "innovation" in front-end javascript libraries throws out a lot of babies with the bathwater, and we might be well-served by accepting a paradigm for 10 minutes and building on it rather than always chasing after the next hot thing.
But at the other extreme, there was a time when PHP was ubiquitous for back-end development, and it stuck around largely because of inertia: the obvious flaws were accepted just because it was so accepted as the tool for that job. But I don't think many people are lamenting the fact that we've moved on to better tools for back-end development. In that case inertia was a bad thing because it delayed innovation.
For me an example of a "good enough" tool is GIT. It's not perfect, it's got a non-trivial learning curve, and surely some innovations could be made to improve version control. Some would already argue that Mercurial is the Betamax to GIT's VHS - but it does the job well enough, and everybody uses it, so it's good enough.
> we might be well-served by accepting a paradigm for 10 minutes and building on it rather than always chasing after the next hot thing.
Exactly this.
> I'd argue Make is more like PHP than it is GIT.
What would you want a make replacement to look like?
Make is pretty 'roll your own' in places, but I've found it to have a nice balance between expressiveness and terseness. For simple projects, there isn't much to do in make, and for the more complex projects I've been involved in (embedded work, a self-hosting compiler, etc.) make can still generally express the build process within the confines of Makefile syntax. (In contrast to more modern tools that might require plugins in a different language, etc.)
For the compiler project I mention, the build process includes building several executables in C, running them against a pre-built VM image to compile another VM image, translating that image into C code that is then compiled and linked into the VM to produce another executable, after which there's a unit test suite. Root rebuild command is 'make clean && make tested', and there are only a couple hundred lines of Makefile code to get there. (There's also cross-compile support, etc.)
I'll admit that there was a certain amount of effort involved in writing those Makefiles, but I've also implemented the same build process using a modern GUI tool and found it to be much more difficult.
> What would you want a make replacement to look like?
Admittedly I haven't given a ton of thought to this specific problem, but here goes:
I actually like Make from a conceptual standpoint. I think it's a good idea to codify build orchestration in code rather than non-human-readable output from a GUI application. My main gripe is that it seems to be very much a 1.0 tool: it was build to solve a specific problem, and then features were added as needed. I would love to have an iteration on that concept which benefits from what we've learned from Make while rounding off the rough edges of what feels like the result of a largely ad-hoc design process.
As far as what that would look like: I would like to see a concise, low-level API for doing the kinds of things Makefiles do (i.e. defining a dependency graph, setting up build parameters etc), and better tools for composition.
Also I would sacrifice a bit of terseness for the benefit of readability. For instance an arbitrary keyword like `Phony` could be replaced by a more extensible name-spacing syntax which lets you know exactly what's going on.
Regarding make, the thing to realize there is that it's just a DSL for specifying a dependency graph (and the way for satisfying dependencies). It's not a hard concept, but unless you come to the field with a bit of computer science background, it's not likely something you've been directly exposed to at the level required to understand make. But I don't know that this means make's choice of abstraction is necessarily a bad thing for its intended use cases. I also think this tends to put the lie to the notion that the difficulty is mainly around tabs... tabs may obscure the details, but switching to a clearer syntax won't solve the more fundamental gaps in understanding the mechanics of make.
I'll also argue that this is a potential problem with DSL's in general... because DSL's often don't use traditional imperative language semantics, there's always a bit of a learning curve that needs to be ascended. However, for make at least, there are a useful debugging tools to help understand what it's actually doing.
For me, at least, 'make -n' is great in that it instructs make to resolve the dependency graph based on the current state, and then just print the commands it'll run without executing them. If you know what the commands do, and know what needs to run, this can be a huge help understanding what's going on. (And there are other more powerful debugging capabilities as well for dumping variables, etc.)
DSLs are most suited to constrained execution spaces - where you know that there are relatively hard limits on what it is that the resultant program will have to do. The problem is that build systems are not really a constrained execution space - there's too often a lot more required than just specifying a dependency tree and just saying 'make it happen'.
As with many 'simplified' DSLs that are often unsuited to the task at hand (e.g. C++ templating), make (or versions of it, anyway), evolved to accommodate these use cases and then morphed into something that accidentally became turing complete: https://stackoverflow.com/questions/3480950/are-makefiles-tu...
...Nor does it mean that just because it has a steep learning curve it’s badly designed.
Make is not a needlessly complex tool (unlike most garbage tools these days); it’s complex because it needs to be flexible enough to build any programmer’s software. Since ways to program and build software are infinite, the tool itself cannot be simple, otherwise trade-offs have to be made.
> I feel like most Make users don't understand Make very well (especially the ones that complain really loudly about it).
This is the very same narrative that vim users proclaim: "It's totally not complex if you took some years to learn your tools" — while simply not realizing that the whole Make syntax is a pile of clunks and hacks and stopgaps and bandaids that has a super steep learning curve and looks like incomprehensive gibberish to the uninitated eye.
And most of that needlessly so IMHO. It shouldn't be hard to find a better, well designed syntax with sane defaults, stricter rules with typing and linting, less surprises (tabs, I'm looking at you) that's overall much less scary for newcomers — as well as less taxing for "expert casual" users like me. (I still go for my coworkers and Stack Overflow regularly if I have to fix one and I'm a fucking CTO with 10 years of dev experience)
Make's only reason for survival is its ubiquity. No matter which terminal you boot up, it's just there, that's why we use it too, and I hate it every day.
I think none of the alternatives (cmake, Jam, Gradle, Maven, grunt etc. etc.) has really succeeded as a cross-platform, cross-ecosystem solution that's both an approachable, simple and usable version of a DAG execution engine that's usable for lots of situation.
So the situation will probably stay like with vim and the QWERTY keyboard, the status quo pretty bad, but everyone's too used to it to go for something better.
> This is the very same narrative that vim users proclaim: "It's totally not complex if you took some years to learn your tools"
Something interesting I see more often than not when someone I know decides to try out vim:
They know I'm a vim user so they ask me for good tutorials and I tell them there are lots out there but they should start by running vimtutor (included with vim). Takes around 15-20 minutes to work through the interactive lessons and gives you all the basics of movement and editing.
A day or so later I ask how it's going and the answer is usually that they're struggling with vim, so I ask if they ran vimtutor. Usually nope, or they started it but quit after a minute or two.
Why would one switch to a tool with a different paradigm, ask for and get a tutorial recommendation, and then ignore the advice of spending less than 1/2 hour on a tutorial that ships with the tool? sigh
Because in the windows world- every ui-interface holds its contract, everything explains itself and does what it supossed to do.
There is no disk-icon that doesent save.
There is no set of shortkeys ignoring controll s.
Design is meeting the users expectations without additional instructions with functionality. VIM is not designed very well.
Just because Vim was not designed for the mouse clicking UI environment does not mean that it's not designed well. If it weren't designed well, Vim would not be in widespread use 40 years after its inception.
Sure, but if you're trying to carry over Windows expectations over to Linux you're sure to be disappointed.
In Vim there are modes that describe sets of keybindings and there are commands that are bound to them. The commands themselves are are straightforward for anyone who's fluent in Linux vocabulary.
'Saving' is a Windows word. We :write to files or :w for short.
> They know I'm a vim user so they ask me for good tutorials and I tell them there are lots out there but they should start by running vimtutor (included with vim). Takes around 15-20 minutes to work through the interactive lessons and gives you all the basics of movement and editing.
I think that I am rather good at memorizing lots of obscure raw data - but in my experience it takes a lot longer to remember all the "obscure" commands of vim. In this sense I would not say it takes 15-20 minutes, but at least a day if at the end you really want to have the commands in your brain.
In this sense the mistake in your argumentation is the timescale that you are talking about.
I never meant to imply that 15-20 minutes in vimtutor would make you a vim wizard, or that you'd remember everything that vimtutor went through. There's no quick and easy answer to switching to something that does things in a fundamentally different way.
Also, I hold some fairly controversial opinions: you don't need dozens of vim plugins; you don't need to compete with vim golf champions; you don't need an extensive configuration.
What you do need is to fully grok the idea of normal mode as an editing language with verbs, objects, and modifiers. Once you "get it" then you have a working knowledge of how to construct "sentences", and learning a new text object allows you to apply pretty much all verbs to it. Same for learning a new verb; you can apply it to lots of things.
Over the years I've learned a lot of interesting vim things and I've forgotten most of them. Because I don't need them every day or even every week. That's ok. The things I remember are things I use often, because they're useful in everyday editing.
"The "%.o: %.c" line is a huge code smell for Makefiles. You shouldn't ever write that line—it's built in!"
Does it hurt to write it though? It makes the makefile more readable for people who don't know it's built-in. Besides, it's only built-in for ".c". I prefer to keep it in order to have a generic Makefile I can adapt to other programming languages.
* Change tracking entirely depends on modified timestamps instead of something more robust like checksums.
* Makefiles are always platform specific. Make executes commands based on a dependency graph, but it has no commands of its own. You either write multiple makefiles or you build everything in a bash environment.
Do you have a good link/example, why timestamps are bad?
I think in the context of Make it is ok. If you rely on checksums, you must implicitly assume that all commands you execute are idempotent. That assumption easily breaks.
I don't have a link, but one reason to prefer checksums is if you have intermediary steps that produce artifacts, possibly binary input to another step, that may or may not need to actually be built.
When I worked with Clearcase (which is terrible in many ways, this is just one), one of its nice features was that it changed file timestamp to the timestamp of commit, not the timestamp it was changed on developer's machine. The commit timestamp may be after previous commit but also before the compiled object file. Because of this, the only way to do a reliable build was to start from scratch. Which took 45 minutes.
Re -MMD: without automated dependency discovery, make was half a tool: if a dependency was omitted, it might do a less-than minimal job of rebuilding, introducing regressions into the build output that did not exist in the source. For this reason, make was quite often used in an unconditional manner, and if you are doing that, you can get the same result more simply and clearly (and without surprises) with a procedural shell script. It is surprising how long it took for that gap in capabilities to be closed.
Something more along those lines, yes, although I'm really not that convinced about some of their design decisions and though the documentation is at least very detailed, it's rather obtuse.
There are any number of technologies that I've seen described as "simple" that I'm inexperienced in...and they look overly complex to me. When I learn to use them, they don't.
I've used make for a long time. I've never considered it difficult to learn or use. It has a limited number of things that it does for you, and leaves the rest to you. I like that paradigm a lot more than systems that seem to operate magically.
the only bad sin I've seen with makefiles is when organizations distribute complex ones that every project must use.
make should be simple, custom, and predictable for each piece of a project. I sometimes have a file for each dir. having a central invisible one is awful.
Compiling projects is inherently complicated, so the solution must also be complicated. You can solve the problem on a spectrum of easy but lack of configurability, or complex/verbose but very flexible. Make is the latter.
The author has shown that he does not understand the Make language very well since half of his final bullet points are wrong.
Makefiles are simple in that you can see every directive and very little is hidden from you. There is no magic - just dependency ordering using mtime as a very simple heuristic.
Except that there is a lot of magic [1] and it does cause bugs. For example the implicit `%: %.o` rule has to be cancelled if you have a rule like `$(SPECIFIC_OBJ_FILES): %.c.o: %.c`
I like to put `MAKEFLAGS += --no-builtin-rules` near the top of my makefiles. This way, all predefined rules are cancelled and I won't be caught off guard by a rule that I didn't specify explicitly being a match.
I also usually put `.SUFFIXES:`, but I guess that's redundant: "The `-r` or `--no-builtin-rules` flag causes the default list of suffixes to be empty." [1]
I use makefile to build my javascript and css. It works brilliantly. I initially used Gulp, but started running into problems when there were build errors that would crash Gulp so I had to use a plugin called Gulp-plumber just to keep the watcher going. Then when I started having a somewhat more complex chain of dependencies I didn't enjoy having to manually specify the build order. The final nail in the coffin for Gulp is when I wanted to dynamically alter flags based on certain conditions, whether it was a debug or prod build for example, it just started getting messier and messier.
I am completely satisfied with Make as a replacement. Took me a couple days of hacking to figure out how to set it up and it's been problem-free ever since.
By far, my favourite aspect is that I no longer need to depend on flimsy gulp-specific wrappers. I just call my build tools natively.
I used Make for my javascript for a while, but having to update the makefile everytime dependencies changed got annoying. Now I just `webpack --watch`; the startup time isn't great, but it's a one-time cost and the recompile happens faster than I can switch to the browser.
I write my makefile once at the beginning of a project and never touch it again. If you need to update yours with every new dependency, you might have your makefile set up sub-optimally. I make sure to have something like the following to include all my js/css dependencies.
I use a small program called watch https://github.com/tj/watch written by TJ Holowaychuck (the inventor of the Express framework) which automatically re-runs my makefile every second. As make is smart enough look for modified files and rebuild only what is necessary, it works quite well.
I'm writing for the browser so imports have to be bundled into the final product; I'm not seeing how that will rebuild a.js (depends on b.js) when b.js changes.
You could do something as follows assuming all source files are in src/, your entry point is src/a.js and your output is build/scripts.js. I use browserify for my compilation but this would work similarly with webpack.
Meh. The author uses a silly example of Makefiles. People get the wrong idea about make. They think it's a program for compiling things; it's not, it's a very early, slim, clean version of chef or ansible. Personally I only got to know how cool make really is when I started to use it for stuff other than compilation, like file transfers or service run control (at one point my toy distro's rc system was "/bin/make -f /etc/rc.mk", where /bin/make was a static version.)
Even for the example he uses, since he seems to be on GNU Make, it's a pretty perfect case to use Guile integration to handle that metaprogramming he's trying to do by hand.
I certainly think whatever incarnation of make that you would summon to do some sort of multi-machine configuration management (per your chef/ansible comparison) is something I would do everything in my power to avoid.
Many a time I see unslim and unclean Makefiles where the problem is mostly that someone started to use make then tried hard to use make as a single unifying tool for too many things, some clearly not belonging to make.
There is nothing wrong with writing wrapper and helper scripts or programs alongside a Makefile, so that make is only bothered with its dependency graph building.
Yes but personally, I blame the tool for being neither opinionated enough, or enforcing more "structure" so that those weaknesses don't come to bear.
History has shown that having sensible defaults for common use cases and workflows goes a long way to reducing complexity and ammunition for self-afflicted foot shooting should be isolated and explicit.
> So if you invoke a tool that expects some arguments wrapped in parentheses, you need to do stuff like
...like $(patsubst)?
$(patsubst %,(%),$(PARAMS))
Though the part about quoting still stands. And I couldn't be bothered with
checking whether the $(patsubst) is a function defined in SUS or is GNU specific.
How about that? make with no Makefile. The unicorn doesn't even exist. Which is as clean as it can get. make only resolves dependencies. Which is as slim as it can get. Early we agree on.
I've programmed in the early heyday when Make was my ONLY option and I'm primarily a C++ developer. These days? No thanks. I don't want to return to that hell. Any technology is "slim" if you cherry-pick just one feature. By any reasonable definition, make is filled with warts and all sorts of tricks that are REQUIRED for any non-trivial project that supports multiple platforms to work. It is old crufty technology and I'll take my chances with other options thanks.
You're free to love make all you want, but I've been there, was effectively held hostage by it for lack of better options, and hate it. Automake and asociated tools was the first attempt at getting us out of that hell, and it didn't succeed but cmake/premake/bazel are doing a whole lot better.
The long standing problem with make is that it tries to be two fairly orthogonal things that are needed for a build system: 1) a dependency resolution engine 2) a scripting language to poke at and act in the environment.
As a dependency resolution engine, make is ok-ish.
As a scripting language, make is an abomination.
Can't upvote this enough; I to create a Makefile for my app (https://zenaud.io) and came to the exact same conclusion. Once you're using the -MD clang option to generate makefile directives for include dependencies, your Makefile is no longer simple -- in fact it starts looking horrible. It's best to avoid IMHO.
>> You don't get to claim simplicity if you use rebar.
Yea, thats true. But good to see that I am not the only one who creates simplified makefiles to utilize other build tools.
My critique on erlang.mk is, that its far simpler to remember the 10-20ish commands of the rebar toolchain than thousands (or hundreds, but that it what it feels like) of erlang.mk options for every erlang tool that has ever been invented.
The reverse is also true: it's far simpler to remember the 10-20ish options of
the erlang.mk plugins you use than hundreds options of various rebar plugins.
There is no significant difference between the two.
Arguably the simplest Makefile is not having any Makefile and just doing
make <executable name>
The next step up would be a Makefile that is essentially just a set of sh scripts, like your example. But I think it's supposed to be implied that the blog is specifically talking about medium-sized C projects.
i find it always starts simple, and usually one maintainer; after a year, it’s a nightmare of hack on hack on hack with three+ contributors each knowing a slice. makefiles on projects that are active for two+ years, a year+ later might as well be chinese. (if it was chinese i could at least ask someone fluent to translate.)
I agree with the author - for C/C++ projects. The module system in C and C++ is pushed out of the compiler and into the build tool. This means make (which is simple) is insufficient to easily correctly implement the logic to build C/C++ projects of more than a few files.
However, a lot of other commenters in this thread have mentioned using make to manage their JS builds. JS build tools tend to work on whole directories of files together, and as such its fairly easy to compose these tools without a lot of logic in the build tool, and here make excells because its ubiquitous and simple. We use a makefile for building crystal and its simple and works great.
Make isnt useless, its just useless in the world it was created in.
That’s right, I don’t want to change something powerful which works really well because someone finds it hard to understand (lacking the prerequisite knowledge) or even worse, wants instant gratification and isn’t willing to master it. That’s like wanting to design a CPU accelerator without wanting to learn how to design electrical circuits. Of course I’m going to say no!, I don’t want to change the tool! If learning this stuff is too hard or not interesting to one, then one ought to consider doing something else for a living.
We have tools like Maven and Ant and Scons and CMake because someone didn’t get Make and shell but they thought they did. Now I’m stuck with toy build tools which use unnecessary dependencies (like Java and .NET and Python) and only do as much as their programmer could imagine, limited by that programmer’s imagination (and experience). You bet your patootie I’m going to say NO! I don’t want to be stuck with a toy, someone’s misunderstanding of Make.
Yeah, let's better create another abstraction layer and release several frameworks each year to make things simpler. /s
Make has its use. It's a great tool for building small/medium projects that don't have many dependencies. You can setup a new project in minutes with nothing more than a text editor and compiler. If you know *sh scripting, it's not really complicated. For bigger projects I prefer Bitbake to avoid "dependency hell".
Isn't the main problem with build tools that no one spends much time learning them? You set them up, they work, then forget about them. Skip forward six months and waste hours on SO figuring out how your build works.
I think that means, the cognitive complexity is not in scale with the business requirements. If I had a service I needed to set up once and maybe update a couple of times a year, but took multiple days to sucessfully update, I'd spend some effort on making iteasier to make changes (either by better understanding, simplifying the system, or rewriting it to remove overhead).
People in this thread might be able to offer me advice on a project I have open.
I know C but am not experienced working on large C codebases. Below I will describe its rough build structure. I hope for strong tool or re-framing advice.
I have an existing python codebase. I want to make it easy to create native-code plugins. The method I have worked out: I compile shared-libraries. Python then calls into these via python-ctypes.
Each plugin is a directory project/ext/plugin,
* Python code: __init__.py and ext.py
* C code, api.c
* A rough build script, build.py. This accesses global config (e.g. path information) held outside this directory. It compiles api.c into a .so or .dll. It does this by generating a shell script or batch file, and then shelling out to it.
A python script in the root directory walks through this structure and forks a process for each plugin build.py. It is multiplatform (currently linux gcc and windows mingw32). So I have if statements saying "if windows, generate this big string, otherwise generate that big string."
The shared-object is compiled to a separate directory, res/ext/plugin. I did this to create separation between my source and build artifacts separate. (Is this sensible?)
This naive build structure was easy to create, and allowed me to spend my focus elsewhere while I was bootstrapping. Now that is done, I am interested in how I could improve the build so that it could scale to a large number of plugins.
Would I be well-served to use Make or Cmake or something else? Can they access configuration from outside of their own directory? Maybe I should communicate configuration via environment variables?
> how I could improve the build so that it could scale to a large number of plugins
Is your existing parallel builder too slow? What's the problem with it? Make can do parallel builds as well, but it won't walk a directory so you'd have to write a million targets.
I think you should post this on the SO site for programming design/advice though.
It sounds like your build process is not well designed. Why would a build script need to generate a shell script to call? And why would a python script fork a process for each plugin?
I think waf, which I've plugged before, would be a good choice for your project. Without knowing the details, your build script would look something like:
def configure(ctx):
# Load C compiler support
ctx.load('compiler_c compiler_cxx')
if ctx.env.DEST_OS == 'win32':
# Special thing for Windows
def build(ctx):
for plugin in ['a', 'b', ...]:
ctx.shlib(source=ctx.path.ant_glob('ext/'+plugin+'/*.c'), target=plugin)
You get cross-platform support for free. The code will create an 'a.dll' on Windows, 'a.so' on Linux and something else on OS X. Note that while the build function above looks like imperative code, it actually is not. Instead it is Python code to create a dependency graph which waf rebuilds for you when it is needed.
134 comments
[ 4.1 ms ] story [ 233 ms ] threadSee https://www.technovelty.org/tips/the-stamp-idiom-with-make.h...
Stamp files are an often an indication that someone is trying to turn a Makefile into a shell script.
Rules should be generated that generate a target from its dependencies. The intermediate steps should be encapsulated as the rule commands.
Building a static website where the targets are wildly different from the source files. I could probably make it explicit but it will be overly verbose and brittle.
Check for and remove trailing whitespace. Because the dependency is actually the target, a stamp file is necessary.
In a simple C project packaged for Linux, FreeBSD anc Mac OS I seriously think a simple Makefile is possible.
And will continue to think that until I see an alternative that is easy to get going with on those systems.
I mainly use them to replace scripts to run tests/linters/formatters/containers/etc
Here's a couple examples:
1. The "%.o: %.c" line is a huge code smell for Makefiles. You shouldn't ever write that line—it's built in!
2. Mentioning .h files in the Makefile. Don't. Go look up the `-MMD` flag in your C compiler. Clang, gcc and many other C compilers support it. Add that and a "-include *.d" line in the Makefile and you'll have great dependency support.
Everything else in that makefile is just setting variables. It's made overly complicated by the directory junk, but I'm failing to see the problem.
Doesn't that sacrifice portability, though? Is this method portable between GNU make and FreeBSD make and OpenBSD make?
The special handling is in the compilers: they emit make file parts!
[1] cf. http://pubs.opengroup.org/onlinepubs/9699919799/utilities/ma...
[2] https://man.openbsd.org/make
[3] https://www.freebsd.org/cgi/man.cgi?make
The first target, hellomake, is our executable. This is the default target, so it's the one that'll be built if we run make with no arguments. It runs the C compiler with the flags we specified earlier. Its dependencies are the object files, which means they'll be built automatically and rebuilt when their dependencies change. As noted, there is a built-in rule to build the object files from the C source using our CC and CFLAGS. We don't need to tell make how to do that.
The next target, clean, is a "phony" target—it doesn't refer to a file. Normally a make target tells make how to build a file. A phony target, in contrast, is really just a shell script that doesn't produce any file output. This one deletes all the build products.
The remaining lines set up the dependency tracking for headers. This makes it so that a change in a header file will trigger a rebuild of any source that includes it. We begin by making a list of files by copying the list of object files and swapping in ".d" for the ".o" file extension. Next is a rule that tells how to build a .d file from a .c file. This rule runs the C compiler and tells it to output a dependency tracking file. The last line says to include each of the .d files in the Makefile. (The .d files contain Makefile syntax to specify the header dependencies.)
I omitted the IDIR and ODIR and such because in a project of this size I wouldn't bother putting includes and object files in separate directories from the source.
I hope that helps. I know make can seem outdated and weird but I think it really can be simple once you learn a few things about how it works. Hopefully this will get you started in that direction!
Please honour LDFLAGS and LDLIBS here.
> $(CC) $(CFLAGS) -MM -MF $@ $<
Please honour CPPFLAGS here.
Don't use wildcards in clean target. Use $(RM) $(OBJS) $(DEPS) instead.
But the question is whether this is Make's fault for being incomprehensible, or the User's fault for being lazy.
The thing is virtually any system or tool can be understood with enough practice, but just because there is someone out there who can use it well does not mean it's not poorly designed.
Both C and Make are decades-old technologies, but most people can read a C file without much experience and intuitively understand what is going on. The same is not true for Makefiles.
'make' exists for what it's capable of doing and it must be so as it's used in every piece of software and operating sytem everywhere.
So it's extremely popular/widespread, therefore it must be good, and therefore you should use it (thereby making it even more popular/widespread)?
I really hope that you poorly worded that/I'm grossly misunderstanding you because otherwise, your logic is batshit insane.
My experience has been that that people that truly understand and appreciate technology for its own sake are also likely to underestimate the value of a tool being popular and widespread. What 'popular' really means is that the tool is more likely to be where you need it to be and the people you need to use it will already know how. Those things outweigh a 'perfectly' designed tool that you don't have or know how to use.
There's a balance point, for sure. For instance, the pace of "innovation" in front-end javascript libraries throws out a lot of babies with the bathwater, and we might be well-served by accepting a paradigm for 10 minutes and building on it rather than always chasing after the next hot thing.
But at the other extreme, there was a time when PHP was ubiquitous for back-end development, and it stuck around largely because of inertia: the obvious flaws were accepted just because it was so accepted as the tool for that job. But I don't think many people are lamenting the fact that we've moved on to better tools for back-end development. In that case inertia was a bad thing because it delayed innovation.
For me an example of a "good enough" tool is GIT. It's not perfect, it's got a non-trivial learning curve, and surely some innovations could be made to improve version control. Some would already argue that Mercurial is the Betamax to GIT's VHS - but it does the job well enough, and everybody uses it, so it's good enough.
I'd argue Make is more like PHP than it is GIT.
Exactly this.
> I'd argue Make is more like PHP than it is GIT.
What would you want a make replacement to look like?
Make is pretty 'roll your own' in places, but I've found it to have a nice balance between expressiveness and terseness. For simple projects, there isn't much to do in make, and for the more complex projects I've been involved in (embedded work, a self-hosting compiler, etc.) make can still generally express the build process within the confines of Makefile syntax. (In contrast to more modern tools that might require plugins in a different language, etc.)
For the compiler project I mention, the build process includes building several executables in C, running them against a pre-built VM image to compile another VM image, translating that image into C code that is then compiled and linked into the VM to produce another executable, after which there's a unit test suite. Root rebuild command is 'make clean && make tested', and there are only a couple hundred lines of Makefile code to get there. (There's also cross-compile support, etc.)
I'll admit that there was a certain amount of effort involved in writing those Makefiles, but I've also implemented the same build process using a modern GUI tool and found it to be much more difficult.
Admittedly I haven't given a ton of thought to this specific problem, but here goes:
I actually like Make from a conceptual standpoint. I think it's a good idea to codify build orchestration in code rather than non-human-readable output from a GUI application. My main gripe is that it seems to be very much a 1.0 tool: it was build to solve a specific problem, and then features were added as needed. I would love to have an iteration on that concept which benefits from what we've learned from Make while rounding off the rough edges of what feels like the result of a largely ad-hoc design process.
As far as what that would look like: I would like to see a concise, low-level API for doing the kinds of things Makefiles do (i.e. defining a dependency graph, setting up build parameters etc), and better tools for composition.
Also I would sacrifice a bit of terseness for the benefit of readability. For instance an arbitrary keyword like `Phony` could be replaced by a more extensible name-spacing syntax which lets you know exactly what's going on.
While they might think they know what's going on, I'm not convinced they actually do. C can be an incredibly subtle language to get right...
http://blog.llvm.org/2011/05/what-every-c-programmer-should-...
Regarding make, the thing to realize there is that it's just a DSL for specifying a dependency graph (and the way for satisfying dependencies). It's not a hard concept, but unless you come to the field with a bit of computer science background, it's not likely something you've been directly exposed to at the level required to understand make. But I don't know that this means make's choice of abstraction is necessarily a bad thing for its intended use cases. I also think this tends to put the lie to the notion that the difficulty is mainly around tabs... tabs may obscure the details, but switching to a clearer syntax won't solve the more fundamental gaps in understanding the mechanics of make.
I'll also argue that this is a potential problem with DSL's in general... because DSL's often don't use traditional imperative language semantics, there's always a bit of a learning curve that needs to be ascended. However, for make at least, there are a useful debugging tools to help understand what it's actually doing.
For me, at least, 'make -n' is great in that it instructs make to resolve the dependency graph based on the current state, and then just print the commands it'll run without executing them. If you know what the commands do, and know what needs to run, this can be a huge help understanding what's going on. (And there are other more powerful debugging capabilities as well for dumping variables, etc.)
As with many 'simplified' DSLs that are often unsuited to the task at hand (e.g. C++ templating), make (or versions of it, anyway), evolved to accommodate these use cases and then morphed into something that accidentally became turing complete: https://stackoverflow.com/questions/3480950/are-makefiles-tu...
Isn't a C++ template the exact opposite of 'Domain Specific'?
Make is not a needlessly complex tool (unlike most garbage tools these days); it’s complex because it needs to be flexible enough to build any programmer’s software. Since ways to program and build software are infinite, the tool itself cannot be simple, otherwise trade-offs have to be made.
This is the very same narrative that vim users proclaim: "It's totally not complex if you took some years to learn your tools" — while simply not realizing that the whole Make syntax is a pile of clunks and hacks and stopgaps and bandaids that has a super steep learning curve and looks like incomprehensive gibberish to the uninitated eye.
And most of that needlessly so IMHO. It shouldn't be hard to find a better, well designed syntax with sane defaults, stricter rules with typing and linting, less surprises (tabs, I'm looking at you) that's overall much less scary for newcomers — as well as less taxing for "expert casual" users like me. (I still go for my coworkers and Stack Overflow regularly if I have to fix one and I'm a fucking CTO with 10 years of dev experience)
Make's only reason for survival is its ubiquity. No matter which terminal you boot up, it's just there, that's why we use it too, and I hate it every day.
I think none of the alternatives (cmake, Jam, Gradle, Maven, grunt etc. etc.) has really succeeded as a cross-platform, cross-ecosystem solution that's both an approachable, simple and usable version of a DAG execution engine that's usable for lots of situation.
So the situation will probably stay like with vim and the QWERTY keyboard, the status quo pretty bad, but everyone's too used to it to go for something better.
Sounds like a solid reason why makefiles are great to me then.
Vim is so good it finally is letting emacs-OS have a real editor.
Something interesting I see more often than not when someone I know decides to try out vim:
They know I'm a vim user so they ask me for good tutorials and I tell them there are lots out there but they should start by running vimtutor (included with vim). Takes around 15-20 minutes to work through the interactive lessons and gives you all the basics of movement and editing.
A day or so later I ask how it's going and the answer is usually that they're struggling with vim, so I ask if they ran vimtutor. Usually nope, or they started it but quit after a minute or two.
Why would one switch to a tool with a different paradigm, ask for and get a tutorial recommendation, and then ignore the advice of spending less than 1/2 hour on a tutorial that ships with the tool? sigh
In Vim there are modes that describe sets of keybindings and there are commands that are bound to them. The commands themselves are are straightforward for anyone who's fluent in Linux vocabulary.
'Saving' is a Windows word. We :write to files or :w for short.
I think that I am rather good at memorizing lots of obscure raw data - but in my experience it takes a lot longer to remember all the "obscure" commands of vim. In this sense I would not say it takes 15-20 minutes, but at least a day if at the end you really want to have the commands in your brain.
In this sense the mistake in your argumentation is the timescale that you are talking about.
Also, I hold some fairly controversial opinions: you don't need dozens of vim plugins; you don't need to compete with vim golf champions; you don't need an extensive configuration.
What you do need is to fully grok the idea of normal mode as an editing language with verbs, objects, and modifiers. Once you "get it" then you have a working knowledge of how to construct "sentences", and learning a new text object allows you to apply pretty much all verbs to it. Same for learning a new verb; you can apply it to lots of things.
Over the years I've learned a lot of interesting vim things and I've forgotten most of them. Because I don't need them every day or even every week. That's ok. The things I remember are things I use often, because they're useful in everyday editing.
Does it hurt to write it though? It makes the makefile more readable for people who don't know it's built-in. Besides, it's only built-in for ".c". I prefer to keep it in order to have a generic Makefile I can adapt to other programming languages.
Make has lots of built-in rules for lots of languages, C, C++, Fortran, Lex, Yacc, etc.
* Recursive Make Considered Harmful
* Changes to the Makefile do not implicitly trigger rebuilds
* Dependencies on directories is not really possible
* Lots of output by default
* Rules with multiple output files are not really possible
* can't handle spaces in pathnames
[0] https://github.com/qznc/annoying-build-systems#make-alone
* Change tracking entirely depends on modified timestamps instead of something more robust like checksums.
* Makefiles are always platform specific. Make executes commands based on a dependency graph, but it has no commands of its own. You either write multiple makefiles or you build everything in a bash environment.
I think in the context of Make it is ok. If you rely on checksums, you must implicitly assume that all commands you execute are idempotent. That assumption easily breaks.
Ceterum censeo, Clearcase must be destroyed.
This indicates that make is hard to understand. I've been using it for a while and I still struggle to grok concepts like order-only prerequisites.
* https://news.ycombinator.com/item?id=15060146
* Using declarative code as much as possible but still written in a good general purpose turing complete language (maybe python - not make).
* Built upon a foundation of well designed libraries, installable from a package manager that abstract common build workflows.
Make meets neither of these standards.
Of course, if you only care about POSIX, a lot of the make issues go away. Not all, but many.
I think it's possible to do better.
https://github.com/evoldoers/biomake
It uses the declarative programming language Prolog to write flexible rules.
A logic programming language like Prolog is very well suited for expressing rule-based dependencies.
I've used make for a long time. I've never considered it difficult to learn or use. It has a limited number of things that it does for you, and leaves the rest to you. I like that paradigm a lot more than systems that seem to operate magically.
make should be simple, custom, and predictable for each piece of a project. I sometimes have a file for each dir. having a central invisible one is awful.
The author has shown that he does not understand the Make language very well since half of his final bullet points are wrong.
Makefiles are simple in that you can see every directive and very little is hidden from you. There is no magic - just dependency ordering using mtime as a very simple heuristic.
[1]: https://www.gnu.org/software/make/manual/html_node/Catalogue...
I also usually put `.SUFFIXES:`, but I guess that's redundant: "The `-r` or `--no-builtin-rules` flag causes the default list of suffixes to be empty." [1]
[1]: https://www.gnu.org/software/make/manual/make.html#Suffix-Ru...
I am completely satisfied with Make as a replacement. Took me a couple days of hacking to figure out how to set it up and it's been problem-free ever since.
By far, my favourite aspect is that I no longer need to depend on flimsy gulp-specific wrappers. I just call my build tools natively.
Check it out: https://gist.github.com/molszanski/56de03b54f66885ef415af84e...
npm scrips are also awesome. But editing and maintaining them inside a package.json is not cool
Even for the example he uses, since he seems to be on GNU Make, it's a pretty perfect case to use Guile integration to handle that metaprogramming he's trying to do by hand.
I certainly think whatever incarnation of make that you would summon to do some sort of multi-machine configuration management (per your chef/ansible comparison) is something I would do everything in my power to avoid.
Many a time I see unslim and unclean Makefiles where the problem is mostly that someone started to use make then tried hard to use make as a single unifying tool for too many things, some clearly not belonging to make.
There is nothing wrong with writing wrapper and helper scripts or programs alongside a Makefile, so that make is only bothered with its dependency graph building.
History has shown that having sensible defaults for common use cases and workflows goes a long way to reducing complexity and ammunition for self-afflicted foot shooting should be isolated and explicit.
So if you invoke a tool that expects some arguments wrapped in parentheses, you need to do stuff like
(RIGHTPAREN isn't strictly needed, but I introduced it to have the symmetry with LEFTPAREN, which is absolutely necessary)...like $(patsubst)?
Though the part about quoting still stands. And I couldn't be bothered with checking whether the $(patsubst) is a function defined in SUS or is GNU specific.You're free to love make all you want, but I've been there, was effectively held hostage by it for lack of better options, and hate it. Automake and asociated tools was the first attempt at getting us out of that hell, and it didn't succeed but cmake/premake/bazel are doing a whole lot better.
As a dependency resolution engine, make is ok-ish. As a scripting language, make is an abomination.
You don't get to claim simplicity if you use rebar.
Yea, thats true. But good to see that I am not the only one who creates simplified makefiles to utilize other build tools.
My critique on erlang.mk is, that its far simpler to remember the 10-20ish commands of the rebar toolchain than thousands (or hundreds, but that it what it feels like) of erlang.mk options for every erlang tool that has ever been invented.
There is no significant difference between the two.
https://github.com/apex/apex/blob/master/Makefile
However, a lot of other commenters in this thread have mentioned using make to manage their JS builds. JS build tools tend to work on whole directories of files together, and as such its fairly easy to compose these tools without a lot of logic in the build tool, and here make excells because its ubiquitous and simple. We use a makefile for building crystal and its simple and works great.
Make isnt useless, its just useless in the world it was created in.
Because many old programmers don't want to change their habit.
> including every piece of software installed on it?
Look up ninja.
We have tools like Maven and Ant and Scons and CMake because someone didn’t get Make and shell but they thought they did. Now I’m stuck with toy build tools which use unnecessary dependencies (like Java and .NET and Python) and only do as much as their programmer could imagine, limited by that programmer’s imagination (and experience). You bet your patootie I’m going to say NO! I don’t want to be stuck with a toy, someone’s misunderstanding of Make.
Make has its use. It's a great tool for building small/medium projects that don't have many dependencies. You can setup a new project in minutes with nothing more than a text editor and compiler. If you know *sh scripting, it's not really complicated. For bigger projects I prefer Bitbake to avoid "dependency hell".
I know C but am not experienced working on large C codebases. Below I will describe its rough build structure. I hope for strong tool or re-framing advice.
I have an existing python codebase. I want to make it easy to create native-code plugins. The method I have worked out: I compile shared-libraries. Python then calls into these via python-ctypes.
Each plugin is a directory project/ext/plugin,
* Python code: __init__.py and ext.py
* C code, api.c
* A rough build script, build.py. This accesses global config (e.g. path information) held outside this directory. It compiles api.c into a .so or .dll. It does this by generating a shell script or batch file, and then shelling out to it.
A python script in the root directory walks through this structure and forks a process for each plugin build.py. It is multiplatform (currently linux gcc and windows mingw32). So I have if statements saying "if windows, generate this big string, otherwise generate that big string."
The shared-object is compiled to a separate directory, res/ext/plugin. I did this to create separation between my source and build artifacts separate. (Is this sensible?)
This naive build structure was easy to create, and allowed me to spend my focus elsewhere while I was bootstrapping. Now that is done, I am interested in how I could improve the build so that it could scale to a large number of plugins.
Would I be well-served to use Make or Cmake or something else? Can they access configuration from outside of their own directory? Maybe I should communicate configuration via environment variables?
Is your existing parallel builder too slow? What's the problem with it? Make can do parallel builds as well, but it won't walk a directory so you'd have to write a million targets.
I think you should post this on the SO site for programming design/advice though.
I think waf, which I've plugged before, would be a good choice for your project. Without knowing the details, your build script would look something like:
You get cross-platform support for free. The code will create an 'a.dll' on Windows, 'a.so' on Linux and something else on OS X. Note that while the build function above looks like imperative code, it actually is not. Instead it is Python code to create a dependency graph which waf rebuilds for you when it is needed.