Legacy methods in large projects need a big motivation to change. Every couple of years, someone proposes a new build system. The conversation usually dies within a week or two.
Why would you have spaces in source code filenames? It's just a bad idea in general and makes things more complicated for no reason, regardless of what build system you're using.
Spaces in filenames aren't inherently bad, they become bad because tools that don't support them are bad. Saying "why would you have spaces?" as a counterargument to "make doesn't support spaces" doesn't work, because the problem with spaces is exactly that make doesn't support them.
I'm not entirely sure why "make, but it handles filenames properly and allows spaces for indentation as well as tab" isn't the most popular fork of make.
You believe this because you probably know well how to use cmake+ninja, but you do not know how to use make (more precisely GNU make, which has valuable extensions over the traditional UNIX make).
Almost all the makefiles that I have ever seen used in open source software projects are in my opinion horrible garbage, and they do not use properly the GNU make features and they are extremely difficult to understand and modify.
That does not mean that makefiles must be written in this way. It is possible to write makefiles in such a way, so that the per project makefiles become extremely simple and extremely easy to maintain.
All the cmake+ninja projects that I have seen were also exceedingly complex and hard to maintain, much worse than when using good makefiles. Maybe there also is a better way to organize the cmake+ninja projects, but I have never bothered to study in detail cmake or ninja, to see how that could be done, because I already knew how to write proper makefiles and I have not seen yet any additional feature that I would want when building a software project.
Regarding the claim that cmake+ninja is faster, I have not seen any realistic benchmark proving this in the right way, i.e. not by comparing some good cmake+ninja project with a bad makefile.
I do not see why any of the 2 methods for building a project could be faster for the other. For a properly written makefile, the time spent in executing make is completely negligible in comparison with the compilation/linking steps.
I assume that the same must be true for cmake+ninja, so the speed must be the same when using the same speed-up methods for both, e.g. when providing the same number of source files per compiler invocation, to minimize the compiler start-up times.
Because the same work must be done, there is no way to for one tool to be faster than the other. Any trick that you might do to accelerate the compilation with one tool can also be done when using the other tool.
The only difference between tools is in the maintenance effort that must be done by the programmer when making changes in the project, e.g. when adding source files, deleting source files, renaming source files, moving source files and so on.
With properly written makefiles, nothing must be done for any of the operations mentioned above.
I do not know if this is also possible with cmake, because I have never seen such a well-written cmake project, but even if it is possible, the maximum that cmake could do would be to match what could already be done with make many years before cmake existed.
So I have never seen any reason to switch to cmake or cmake+ninja. All the arguments in their favor that I have ever seen were given by people who have never read the GNU make manual, so they compared their favorite tools with bad makefiles, not with good makefiles, so those arguments had nothing to do with make itself, but only with good ways or bad ways to organize the building of a software project.
> Almost all the makefiles that I have ever seen used in open source software projects are in my opinion horrible garbage, and they do not use properly the GNU make features and they are extremely difficult to understand and modify.
if tomorrow a pen maker released a new pen where the majority of artists using it only managed to produce shit artworks, everyone would rightly call this pen shit. Even if it's theoretically possible to use the pen in such a way as to not produce shit artworks. Likewise for, say, a tool used for making musical instruments, etc.
How well a tool lends itself to be used by the average person is an important property when choosing to use a technology over another.
Can you provide e.g. a project with your good makefiles, that I could use to build debug and release builds, with any of MSVC, GCC, and Clang ?
> I do not see why any of the 2 methods for building a project could be faster for the other. For a properly written makefile, the time spent in executing make is completely negligible in comparison with the compilation/linking steps.
can you provide one such makefile ? in my experience make always takes a few seconds for projects with >1k targets when changing a single file. Ninja is consitently instantaneous.
> How well a tool lends itself to be used by the average person is an important property when choosing to use a technology over another.
The reason why most makefiles in open source projects are horrible has nothing to do with the qualities of make, because typically those makefiles are not written by some human who knows about make, but they are generated by some other tools, e.g. autoconf/automake.
Moreover, the generation method typically used is flawed, because it generates the entire makefiles, in an incomprehensible form. The maintenance of those makefiles would have been much easier if makefiles carefully written by a human would have been used, which would have included a file generated by the autoconf tools, and the included file should have contained only definitions, and neither rules nor targets.
Unfortunately there is an ancient tradition established decades ago to generate the makefiles in an overly complicated way and nobody has the courage to change that in any project, because probably nobody understands any more what would happen if changes would be made.
> Can you provide e.g. a project with your good makefiles, that I could use to build debug and release builds, with any of MSVC, GCC, and Clang ?
I never had any need to compile something for MSVC, GCC and Clang.
Nevertheless, many of my projects had to be cross-compiled for various embedded CPU targets.
The way I do that is that I have one makefile that contains only definitions for each project building target, so I could have e.g. 3 makefiles, 1 for a MSVC/Windows target, 1 for a GCC/Linux target and 1 for a Clang/Mac OS target.
Each such makefile will have definitions for the names of all executables that may be needed for bulding a software project, e.g. compilers, assemblers, linkers, librarians, object file converters, copy commands, move commands, rename commands and so on.
It will also have definitions for all the command-line option flags that must be provided to each executable in order to perform whatever tasks are needed for building a software project.
Writing such a makefile for a compilation is a one-time effort, I might write it in an hour or more, while searching through the documentation of various tools, but then I might use it unchanged, during many years, for all projects that I intend to build for that target.
I need to write such makefiles only infrequently, when I begin to use a new CPU, or a new operating system, or a new compiler.
These makefiles with definitions dependent on the compilation target are included with an include directive in the complete makefiles that build the software project.
I build each final file, e.g. executable file, dynamic library or static library in a separate build directory. When I want to build multiple files with a single command, then their build directories are subdirectories of a directory where there is a makefile which will invoke the makefiles in the leaf subdirectories and maybe move or copy the built files somewhere else, if the end result needs them to be in certain relative positions in a directory hierarchy.
The makefile in the build directory of some file has only a few lines. It includes the makefile with the definitions dependent on the compilation target and a makefile that is the same for all my projects, with general definitions, rules and targets.
Besides the include directive, there are a few definitions, the type of file that must be built, the name of the built file (when absent, the current directory name is assumed, with an appropriate extension) a list of directories that must be searched to find the source files (if absent the current directory is assumed) and an optional prefix for the list of directories.
Because make searches itself for source files and automatically generates their dependencies, I do not have to do anything when I add/delete/move/rename source files.
For debug and release I obviously have these 2 make targets in the included makefile with general make rules and targets, wh...
> I never had any need to compile something for MSVC, GCC and Clang.
I am lucky that this is your case ; many of the software I've worked on had this requirement.
> The general rules from my makefiles could be modified to generate ninja input files instead of invoking the build commands,
that's basically reimplementing cmake in make. I don't understand why I would do this work myself when cmake exists and is constantly updated to adapt to newer compilers, etc. without me having to lift a finger. + the generation of targets for the IDE - just in the last two weeks I had users come and ask me to have a build that can be worked on in both Visual Studio and Xcode. Good luck with doing that in make.
> When a large number of files must be compiled, there is no chance to see any difference due to the speed of the project build tool.
1k isn't very large though. There are projects out there with the source code without any asset sized in gigabytes.
> On the other hand for cmake I am not aware of any useful application, because all the examples that I have seen of cmake projects were not simpler than if those projects would have used make.
project(bla)
add_executable(my_app main.cpp)
is the minimal CMake project and likely supports as-is more toggles and switches than your homegrown makefile system - toggles and switches that are necessary for pretty much any non-toy project.
After reading your reply, I have browsed the cmake reference manual.
It is likely that for many people cmake may be enough, but my homegrown makefile system can do much more than cmake.
The cmake equivalent for the makefiles with compilation-target-dependent definitions is called a toolchain file.
However, unlike for make, where I have complete freedom, cmake allows in the toolchain file only a relatively small set of options from a closed set.
Because I cross-compile for many unusual combinations of embedded CPUs and operating systems, with cmake I would certainly need to write many custom toolchain files, but cmake would not allow me to specify everything that I need.
Besides C & C++, I use many less popular programming languages. Even if cmake supports a decent number of programming languages, when you want to use any language that is not on the currently supported list, you are out of luck.
With my make system, if I decide to use a new programming language, I would need only a few minutes to update the makefile with generic rules and the toolchain makefiles in order to be able to build a project with the new language.
Why would I need to use a rigid tool like cmake, when the same and more can be done with make ?
I see in the cmake documentation the following recommendation, which appears to have been followed in all the cmake projects that I have seen.
"We do not recommend using GLOB to collect a list of source files from your source tree. If no CMakeLists.txt file changes when a source is added or removed then the generated build system cannot know when to ask CMake to regenerate."
This recommendation is very wrong.
Even if cmake has the limitation that it is triggered only by an updated CMakeLists.txt file, the workaround is not to avoid using GLOB.
The correct workaround is to use GLOB, but to also execute the command "touch CMakeLists.txt" whenever there are changes in the set of source files.
Executing a "touch" is much simpler than editing CMakeLists.txt.
The fact that the bad recommendation from the cmake documentation has been followed blindly in all the cmake projects that I have seen is one reason why I thought that cmake sucks.
However, like I have said above, even when cmake is used correctly, it is too rigid for those whose applications are not restricted to the targets understood by cmake.
"make" does not have any such limitations, so I still do not see any reason for the existence of "cmake", because to do what cmake does you do not need a new executable, you need just a set of generic makefiles, like those that I use, or like those that are used in some projects like FreeBSD (those, unlike cmake are not oriented towards multiple compilation targets, but they could be adapted for such a purpose with much less effort than writing a program like cmake).
> Almost all the makefiles that I have ever seen used in open source software projects are in my opinion horrible garbage
I mostly agree with your post, and I use Make for my own projects. But, I think there's a reasonable argument that if a tool is so hard to use that no one uses it correctly, it's not a very good tool.
Can you link to some examples? I've dug around on Github and tried to find some examples of projects using different kinds of build systems, but I don't think they are great examples of what you're talking about. For others who may want to see examples, here they are:
The makefiles indicated by you are relatively simple makefiles, they do not belong to the class "horrible".
The main problem even with these simpler makefiles is that they are too complicated, because they are written for a single project.
This is something that is difficult to change in the case of most open-source projects, which are intended to be independent from other software projects.
The greatest simplifications in makefiles are done by extracting the common parts of the makefiles from all projects in generic makefiles that should be included in the specific makefiles of each project.
Doing this, it is possible to reduce the effort of writing makefiles for 100 or 1000 software projects to little more than the effort of writing a single makefile.
Besides the reuse of the same make rules, of the same make targets and of many make definitions in all projects, other important simplifications are to never write manually the file dependencies when they can be detected automatically, which is normally true for all compilation tasks, and to never write manually the lists of source files, because they should also be detected automatically.
Agree that ninja+something can be much better than make. ninja is such a clean minimal design.
But cmake? Really? cmake —-help-full is like 2.5MB, and looks like it refers to any know library and tool in the universe.
Mahbe I am missing by a mile, but did we not get enough magic exposure with autoconf already?
Probably not. You'd need a lot more than a petty switch to grammar regarding indentation to convince people to switch to an unproven alternative, though.
My snarky response is that unproven is better than disproven, but my personal feelings aside, make undeniably has marketshare and people aren't likely to switch to something else unless it offers them a more significant improvement than naming/indentation.
I think, maybe, both of these responses are missing the fact that make is not a program, but rather, a number of programs with the same invocation name.
A fork of one of these makes, which did not have the two glaring flaws I refer to, but was otherwise make, would still be, well, make.
For starters, ffmpeg does not use raw Makefile; it has to be generated by running configure. Historically configure (and autoconf etc.) was required due to the variety of platforms to support, which is increasingly less relevant today. The real "raw" Makefile---where you are supposed to run just `make` to build---is not as cumbersome as you may think; even the dependency among source files can be automatically generated.
Makefiles are extremely powerful. I used to write them myself by hand, for a lot more than program generation. Makefiles can be used to recognize data events, and trigger responses - which I used for years in scientific visualization. In grad school, in the inevitable group projects where people lag and do the work at the last possible minute, I used makefiles to recognize when their contribution landed in the share folder so I could review and incorporate. Makefiles are the IFTTT of the pre-Internet era.
Makefiles are excellent, but I think they're hampered a bit by their extremely terse syntax (see also: perl). I wonder if a slightly different dialect would help modern users who are used to more verbose languages work with them.
Because if something works, and the actual project developers are comfortable with it, why change it?
For new projects, people may prefer to use something like Zig as the build system. But a build system is not what will make the project faster or magically add new features and fix bugs. What makes that happen is developers being comfortable with the source code, including its build system.
They both look like they're in sync, but at the top of git.videolan.org shows a notice saying they're moving their projects over to GitLab. The list of projects being moved over doesn't include FFmpeg, so I guess ultimate ownership is at ffmpeg.org.
The biggest consequence would be that this opens the door for all software built on top of ffmpeg to add support for JPEG XL. Some might even support it automatically if they're linked against an ffmpeg which supports it.
I've been waiting to use JPEG XL (.jxl) in my Video Hub App for over a year now! I even have a PR that shows that it works, but was waiting for FFmpeg to catch up.
It's not designed for still images, so it probably doesn't have all the features ImageMagick has for that. But it can convert between video files and loose numbered image files, and it can convert cover art. And there is animated JPEG XL.
Nice but we still need at least cameras, ideally also OSes' default picture viewers and major web browsers to support it OOTB or it will never become a thing.
Browsers are the least important part. People use ordinary JPEG because that's what their cameras produce without asking and that also is what they can just click and open on all their PCs/whatever using their default picture viewer app. E.g. WebP still opens a browser when you click it and that's weird.
Cameras are moving to support HEIF. Sony, Canon, and Nikon cameras support HEIF, as do iPhones. Android cameras may support HEIF depending on the vendor.
Sounds like this particular piece of news is underwhelming, but from my own casual experimentation, JPEG XL delivers much better quality vs compression ratio than Google's WebP (and the lossless compression is really good too).
I look forward to the day when JPEG XL is widely supported by web browsers.
Thanks, I didn't know AVIF was well supported alreadyA I remember I couldn't decide if I liked AVIF or JPEG XL better quality-wise, but the support would make AVIF the clear winner.
I've been using AVIF on my blog for about a year. It's worth noting that AVIF seems to have better quality at lower sizes (~0.5 bits per pixel, ~120KB at 1080p) than JPEG XL (at least for right now), so even if browsers supported JPEG XL, you still might get better optimization with AVIF.
On the average AVIFs are 1.5 BPP and JPEGs about 2.2 BPP in the internet. If you average at 0.5 BPP, you will differentiate by having blurrier images than competing blogs.
1. The vast majority of images on my blog are game screenshots. Often, a good portion of them are solid colors/gradients[0] or have blurry textures[1], which helps. When I played around with AVIF, 0.6 bpp was enough to keep the prominent parts, then tried less. For photos, unless it's a picture of grass, gravel, or trees[2], the same holds[3].
2. I encode only high resolution (x2) images as AVIFs, so blurriness gets covered up by downscaling or high PPI screens almost all of the time. On a desktop 1080p screen, you're looking at up to 2 bpp effectively.
3. I keep individual image sizes below 160KB (0.61 bpp@1080p) to keep page load sizes at 1MB or less.
Of course, I keep an archive of everything in PNG/RAW/etc format to redo if needed.
Which reminds me, what the heck happened to old-school progressive JPEGs? I remember seeing them a lot in the early days of images on the internet, but it seems like you don't see them around anymore. Why is that?
They're still being used. A newer, optimized JPEG encoder, mozJPEG[0], seems to use progressive encoding by default. I suspect with faster internet speeds, most JPEGs download and decode so fast that the cool 'enhance' animation is rarely seen anymore.
For it to be worth the effort to implement/maintain your users need to be bandwidth-constrained, which is not really a reality anymore, jpegs don't take upwards of a minute to load for most people.
It has applications still, such as in medical imaging.
If you treat progressive jpeg and sequential jpeg as separate image formats, then progressive jpeg is currently the fastest growing image format on the web. If you look at the statistics, the opposite is true of what your feeling is.
One thing that did change is that often the network is fast enough to make it unnecessary to show progressive passes, or if they're still shown, it happens quickly enough to be not really noticeable.
> This is likely to put too much pressure on the CPU and memory of the decoding client as each refresh event involves a full pass through the decompression system.
> On average, decoding a progressive JPEG image is equivalent to decoding the baseline one 3 times.
Interesting, although it will depend on how bandwidth constrained your users are vs. their CPU/memory profile. If downloading images takes longer than decoding, it swings in favor of progressive decode.
Progressive JPEG has ~10 scans by default in libjpeg/mozjpeg.
Lossy JPEG XL always allows minimal progression (2 scans, 1:8 followed by 1:1), and uses maybe 3-5 scans when encoding with more progression.
In any case, I think the "costs too much cpu" argument is not really fair, since there is nothing forcing an implementation
to do explicit flush/repaint at every scan. In fact, browsers will only do a refresh when cpu is _not_ the bottleneck, but network is. When cpu _is_ the bottleneck (e.g. the file is loaded from local cache), you can just decode the whole file non-progressively and have only a single refresh (just like with inherently non-progressive formats).
If network is the bottleneck, then the choice is between cpu time (avoiding progressive previews to save some cpu effort) and human time (show previews so something can be shown earlier). I think human time is more valuable, by a few orders of magnitude.
Another feature that is very interesting to me is bit-exact lossless conversation between jpeg and jpeg-xl, in both directions that is. This essentially allows to take a jpeg, run the vastly improved lossless compression stages of jpeg-xl over the the original lossy compressed jpeg data (jpeg "coefficients"), preserve metadata, and get a jpeg-xl result that the jpeg-xl implementations understand, and which you can losslessly convert back to an ordinary jpeg that is a bit-exact reproduction of the original jpeg, if need be.
The compression ratio will not be as good as full jpeg-xl (having skipped the lossy stages), but you still get something in the range of 20% smaller file sizes without any additional quality loss whatsoever.
This is something WEBP/AVIF/HEIC does not offer. When using those codecs, you will either have to accept additional lossy compression artifacts on top of the artifacts of the original jpeg codec, or convert in lossless mode with worse compression ratio.
Comparing compression ratios is tricky. What you usually want is "file size at same visual quality" or "visual quality as same file size" for comparisons. The problem here is that "visual quality" is in the eye of the beholder, Different base images will have different results.
My overall impression is that it's roughly on par or a little behind with AVIF, somewhat ahead of HEIC and very ahead of WEBP (and jpeg of course) in quality-per-byte.
As for encode and decode speed, it's a bit slower than WEBP but beats AVIF and HEIC by a margin (unless these codecs are back by hardware encoders). However, all three codecs are relatively new and there may be a lot of future speed optimizations possible. Especially jpeg-xl, which isn't "backed" by general AV1 (AVIF), vpx (WEBP) or HEVC (HEIC) research, may have a particularly large yet to be tapped potential for (hardware-specific) optimizations. As far as I know, jpeg-xl is also "simpler" than those other codecs (which basically used the iframes of their complex sibling video codecs as a base), so it might be more feasible to have more competing implementations.
But time will tell, and it will of course also largely depend on the adoption and thus network effects of these formats who much "love" each format receives from open source and proprietary software vendors.
Generally speaking the results I've seen so far indicate that at low bitrates (e.g. below 0.5 bpp), the video codecs (AVIF, HEIC, VVC) perform best, while at medium to high bitrates, JPEG XL performs best. This makes sense considering that video codecs aim at lower bitrates than still image codecs. Basically video codecs have more coding tools to "hide artifacts" at low bitrates (e.g. strong smoothing filters), while JPEG XL has more coding tools to "preserve detail" at high bitrates (e.g. strong entropy coding, more subtle filters).
This is of intense interest to me - I had no idea you could do this with an existing jpeg. 20% would be a world-changing difference for an application I am working on right now, assuming the added encoding/decoding stages do not add substantial latency to the process.
Is there some variant of libjpeg-turbo that works for XL?
Unfortunately as far as I can find there is no optimized encoder or decoder for JXL yet. Everything uses the spec encoder, which works fine on modern hardware, but for an example on my 20 year old laptop it takes several seconds to decode an image. Encoding an image with "tortoise" mode takes 10+ minutes. One nice feature though is that JPEG->JPEG XL lossless encoding was a priority, and it's as easy as doing "cjxl my.jpg my.jxl". The result is a lossless encode with EXIF data kept for a full bit perfect JPEG if you do "djxl my.jxl my.jpg." It's also pretty fast even on my old laptop.
The 10+ minutes that OP observed where in tortoise (=9) mode tho. That's the slowest mode, comparable to "placebo" mode in x264, and meant as a "test" mode, not as a daily driver.
The libjxl default is "squirrel" (=7), and you can go as low as lightning (=1), so you have a choice to trade compression efficiency for compression speed.
Additionally, there are different decoder modes (in the encoder), allowing you to make tradeoffs between efficiency/density vs speed.
But yeah, libjxl will remain slower than libjpeg-turbo, especially on encodes.
well every cinema probably uses jpeg2000 (dcp) and the new interoperable master format is using it as well (though it also supports more mainstream codecs)
dont see hollywood moving away from it pretty soon.
I managed to write a script to build a xstack command, that was great.
I recently tried to build a command to automatically add a silent audio track on inputs, while also adding pause and concatenate 300 small videos, all of this in a single command, but I still have a lot of trouble understanding how stream selector work.
Really wish https://github.com/phoboslab/jsmpeg-vnc would support this. The video compression from JPEG XL would be a huge boon. Wish audio was supported too so I can play Civ 5 on my tablet when I'm away from the PC
95 comments
[ 2.0 ms ] story [ 169 ms ] threadFFmpeg relies on libjxl for decoding and encoding. Support added in
https://git.videolan.org/?p=ffmpeg.git;a=commit;h=458cc7e314...
https://git.videolan.org/?p=ffmpeg.git;a=commit;h=5f0b4e9c70...
The Windows static builds have binaries with this support. The linux and MacOS builds don't, as yet.
If it works, it works. Haven't encountered any major issues with writing them in my years.
Which your uni's group policy would make it so that you can only write there (thankfully mine was sane, but I've seen it aha).
And even then, a build system that does not support arbitrary file names is just terminally broken and should not be used, even less encouraged.
Does it even exist?
Almost all the makefiles that I have ever seen used in open source software projects are in my opinion horrible garbage, and they do not use properly the GNU make features and they are extremely difficult to understand and modify.
That does not mean that makefiles must be written in this way. It is possible to write makefiles in such a way, so that the per project makefiles become extremely simple and extremely easy to maintain.
All the cmake+ninja projects that I have seen were also exceedingly complex and hard to maintain, much worse than when using good makefiles. Maybe there also is a better way to organize the cmake+ninja projects, but I have never bothered to study in detail cmake or ninja, to see how that could be done, because I already knew how to write proper makefiles and I have not seen yet any additional feature that I would want when building a software project.
Regarding the claim that cmake+ninja is faster, I have not seen any realistic benchmark proving this in the right way, i.e. not by comparing some good cmake+ninja project with a bad makefile.
I do not see why any of the 2 methods for building a project could be faster for the other. For a properly written makefile, the time spent in executing make is completely negligible in comparison with the compilation/linking steps.
I assume that the same must be true for cmake+ninja, so the speed must be the same when using the same speed-up methods for both, e.g. when providing the same number of source files per compiler invocation, to minimize the compiler start-up times.
Because the same work must be done, there is no way to for one tool to be faster than the other. Any trick that you might do to accelerate the compilation with one tool can also be done when using the other tool.
The only difference between tools is in the maintenance effort that must be done by the programmer when making changes in the project, e.g. when adding source files, deleting source files, renaming source files, moving source files and so on.
With properly written makefiles, nothing must be done for any of the operations mentioned above.
I do not know if this is also possible with cmake, because I have never seen such a well-written cmake project, but even if it is possible, the maximum that cmake could do would be to match what could already be done with make many years before cmake existed.
So I have never seen any reason to switch to cmake or cmake+ninja. All the arguments in their favor that I have ever seen were given by people who have never read the GNU make manual, so they compared their favorite tools with bad makefiles, not with good makefiles, so those arguments had nothing to do with make itself, but only with good ways or bad ways to organize the building of a software project.
if tomorrow a pen maker released a new pen where the majority of artists using it only managed to produce shit artworks, everyone would rightly call this pen shit. Even if it's theoretically possible to use the pen in such a way as to not produce shit artworks. Likewise for, say, a tool used for making musical instruments, etc.
How well a tool lends itself to be used by the average person is an important property when choosing to use a technology over another.
Can you provide e.g. a project with your good makefiles, that I could use to build debug and release builds, with any of MSVC, GCC, and Clang ?
> I do not see why any of the 2 methods for building a project could be faster for the other. For a properly written makefile, the time spent in executing make is completely negligible in comparison with the compilation/linking steps.
can you provide one such makefile ? in my experience make always takes a few seconds for projects with >1k targets when changing a single file. Ninja is consitently instantaneous.
The reason why most makefiles in open source projects are horrible has nothing to do with the qualities of make, because typically those makefiles are not written by some human who knows about make, but they are generated by some other tools, e.g. autoconf/automake.
Moreover, the generation method typically used is flawed, because it generates the entire makefiles, in an incomprehensible form. The maintenance of those makefiles would have been much easier if makefiles carefully written by a human would have been used, which would have included a file generated by the autoconf tools, and the included file should have contained only definitions, and neither rules nor targets.
Unfortunately there is an ancient tradition established decades ago to generate the makefiles in an overly complicated way and nobody has the courage to change that in any project, because probably nobody understands any more what would happen if changes would be made.
> Can you provide e.g. a project with your good makefiles, that I could use to build debug and release builds, with any of MSVC, GCC, and Clang ?
I never had any need to compile something for MSVC, GCC and Clang.
Nevertheless, many of my projects had to be cross-compiled for various embedded CPU targets.
The way I do that is that I have one makefile that contains only definitions for each project building target, so I could have e.g. 3 makefiles, 1 for a MSVC/Windows target, 1 for a GCC/Linux target and 1 for a Clang/Mac OS target.
Each such makefile will have definitions for the names of all executables that may be needed for bulding a software project, e.g. compilers, assemblers, linkers, librarians, object file converters, copy commands, move commands, rename commands and so on.
It will also have definitions for all the command-line option flags that must be provided to each executable in order to perform whatever tasks are needed for building a software project.
Writing such a makefile for a compilation is a one-time effort, I might write it in an hour or more, while searching through the documentation of various tools, but then I might use it unchanged, during many years, for all projects that I intend to build for that target.
I need to write such makefiles only infrequently, when I begin to use a new CPU, or a new operating system, or a new compiler.
These makefiles with definitions dependent on the compilation target are included with an include directive in the complete makefiles that build the software project.
I build each final file, e.g. executable file, dynamic library or static library in a separate build directory. When I want to build multiple files with a single command, then their build directories are subdirectories of a directory where there is a makefile which will invoke the makefiles in the leaf subdirectories and maybe move or copy the built files somewhere else, if the end result needs them to be in certain relative positions in a directory hierarchy.
The makefile in the build directory of some file has only a few lines. It includes the makefile with the definitions dependent on the compilation target and a makefile that is the same for all my projects, with general definitions, rules and targets.
Besides the include directive, there are a few definitions, the type of file that must be built, the name of the built file (when absent, the current directory name is assumed, with an appropriate extension) a list of directories that must be searched to find the source files (if absent the current directory is assumed) and an optional prefix for the list of directories.
Because make searches itself for source files and automatically generates their dependencies, I do not have to do anything when I add/delete/move/rename source files.
For debug and release I obviously have these 2 make targets in the included makefile with general make rules and targets, wh...
I am lucky that this is your case ; many of the software I've worked on had this requirement.
> The general rules from my makefiles could be modified to generate ninja input files instead of invoking the build commands,
that's basically reimplementing cmake in make. I don't understand why I would do this work myself when cmake exists and is constantly updated to adapt to newer compilers, etc. without me having to lift a finger. + the generation of targets for the IDE - just in the last two weeks I had users come and ask me to have a build that can be worked on in both Visual Studio and Xcode. Good luck with doing that in make.
> When a large number of files must be compiled, there is no chance to see any difference due to the speed of the project build tool.
1k isn't very large though. There are projects out there with the source code without any asset sized in gigabytes.
> On the other hand for cmake I am not aware of any useful application, because all the examples that I have seen of cmake projects were not simpler than if those projects would have used make.
is the minimal CMake project and likely supports as-is more toggles and switches than your homegrown makefile system - toggles and switches that are necessary for pretty much any non-toy project.It is likely that for many people cmake may be enough, but my homegrown makefile system can do much more than cmake.
The cmake equivalent for the makefiles with compilation-target-dependent definitions is called a toolchain file.
However, unlike for make, where I have complete freedom, cmake allows in the toolchain file only a relatively small set of options from a closed set.
Because I cross-compile for many unusual combinations of embedded CPUs and operating systems, with cmake I would certainly need to write many custom toolchain files, but cmake would not allow me to specify everything that I need.
Besides C & C++, I use many less popular programming languages. Even if cmake supports a decent number of programming languages, when you want to use any language that is not on the currently supported list, you are out of luck.
With my make system, if I decide to use a new programming language, I would need only a few minutes to update the makefile with generic rules and the toolchain makefiles in order to be able to build a project with the new language.
Why would I need to use a rigid tool like cmake, when the same and more can be done with make ?
I see in the cmake documentation the following recommendation, which appears to have been followed in all the cmake projects that I have seen.
"We do not recommend using GLOB to collect a list of source files from your source tree. If no CMakeLists.txt file changes when a source is added or removed then the generated build system cannot know when to ask CMake to regenerate."
This recommendation is very wrong. Even if cmake has the limitation that it is triggered only by an updated CMakeLists.txt file, the workaround is not to avoid using GLOB.
The correct workaround is to use GLOB, but to also execute the command "touch CMakeLists.txt" whenever there are changes in the set of source files.
Executing a "touch" is much simpler than editing CMakeLists.txt.
The fact that the bad recommendation from the cmake documentation has been followed blindly in all the cmake projects that I have seen is one reason why I thought that cmake sucks.
However, like I have said above, even when cmake is used correctly, it is too rigid for those whose applications are not restricted to the targets understood by cmake.
"make" does not have any such limitations, so I still do not see any reason for the existence of "cmake", because to do what cmake does you do not need a new executable, you need just a set of generic makefiles, like those that I use, or like those that are used in some projects like FreeBSD (those, unlike cmake are not oriented towards multiple compilation targets, but they could be adapted for such a purpose with much less effort than writing a program like cmake).
I mostly agree with your post, and I use Make for my own projects. But, I think there's a reasonable argument that if a tool is so hard to use that no one uses it correctly, it's not a very good tool.
- A Makefile that runs cmake which generates ninja build files that build the project. Neovim: https://github.com/neovim/neovim/blob/master/Makefile
- Cmake that generates ninja build files which build the application. OBS (Open Broadcaster Software) : https://github.com/obsproject/obs-studio/wiki/build-instruct...
- A makefile that builds the application. Localstack : https://github.com/localstack/localstack/blob/master/Makefil...
- Cmake that generates Makefiles that build the application. Krita : https://docs.krita.org/en/untranslatable_pages/building_krit...
The main problem even with these simpler makefiles is that they are too complicated, because they are written for a single project.
This is something that is difficult to change in the case of most open-source projects, which are intended to be independent from other software projects.
The greatest simplifications in makefiles are done by extracting the common parts of the makefiles from all projects in generic makefiles that should be included in the specific makefiles of each project.
Doing this, it is possible to reduce the effort of writing makefiles for 100 or 1000 software projects to little more than the effort of writing a single makefile.
Besides the reuse of the same make rules, of the same make targets and of many make definitions in all projects, other important simplifications are to never write manually the file dependencies when they can be detected automatically, which is normally true for all compilation tasks, and to never write manually the lists of source files, because they should also be detected automatically.
A fork of one of these makes, which did not have the two glaring flaws I refer to, but was otherwise make, would still be, well, make.
Off the top of my head, take this make example:
and spell it out a bit more: Obviously this needs a lot more thought than that, but I think newbies would find the 2nd format much more friendly than the first.For new projects, people may prefer to use something like Zig as the build system. But a build system is not what will make the project faster or magically add new features and fix bugs. What makes that happen is developers being comfortable with the source code, including its build system.
https://github.com/whyboris/Video-Hub-App/pull/702
My software extracts screenshots from videos and shows them in a nice browsable searchable gallery.
As I use mpv both as a video player and as an image viewer, this is great news for me.
Perhaps this means it can be shoehorned in anyways?
I look forward to the day when JPEG XL is widely supported by web browsers.
https://caniuse.com/jpegxl
1. The vast majority of images on my blog are game screenshots. Often, a good portion of them are solid colors/gradients[0] or have blurry textures[1], which helps. When I played around with AVIF, 0.6 bpp was enough to keep the prominent parts, then tried less. For photos, unless it's a picture of grass, gravel, or trees[2], the same holds[3].
2. I encode only high resolution (x2) images as AVIFs, so blurriness gets covered up by downscaling or high PPI screens almost all of the time. On a desktop 1080p screen, you're looking at up to 2 bpp effectively.
3. I keep individual image sizes below 160KB (0.61 bpp@1080p) to keep page load sizes at 1MB or less.
Of course, I keep an archive of everything in PNG/RAW/etc format to redo if needed.
Blog link: https://theandrewbailey.com/
[0] https://theandrewbailey.com/fileImmutable/AXW_xuoI/post/2018...
[1] https://theandrewbailey.com/fileImmutable/AX3-7eYX/post/2021...
[2] https://theandrewbailey.com/coroner/404
[3] https://theandrewbailey.com/fileImmutable/AXW_xuox/post/2020...
https://youtu.be/UphN1_7nP8U
Which reminds me, what the heck happened to old-school progressive JPEGs? I remember seeing them a lot in the early days of images on the internet, but it seems like you don't see them around anymore. Why is that?
[0] https://github.com/mozilla/mozjpeg
It has applications still, such as in medical imaging.
One thing that did change is that often the network is fast enough to make it unnecessary to show progressive passes, or if they're still shown, it happens quickly enough to be not really noticeable.
> On average, decoding a progressive JPEG image is equivalent to decoding the baseline one 3 times.
This feature wasn't added for a reason[0].
[0]: https://developers.google.com/speed/webp/faq#does_webp_suppo...
In .jxl, it is an optional feature.
In any case, I think the "costs too much cpu" argument is not really fair, since there is nothing forcing an implementation to do explicit flush/repaint at every scan. In fact, browsers will only do a refresh when cpu is _not_ the bottleneck, but network is. When cpu _is_ the bottleneck (e.g. the file is loaded from local cache), you can just decode the whole file non-progressively and have only a single refresh (just like with inherently non-progressive formats).
If network is the bottleneck, then the choice is between cpu time (avoiding progressive previews to save some cpu effort) and human time (show previews so something can be shown earlier). I think human time is more valuable, by a few orders of magnitude.
The compression ratio will not be as good as full jpeg-xl (having skipped the lossy stages), but you still get something in the range of 20% smaller file sizes without any additional quality loss whatsoever.
This is something WEBP/AVIF/HEIC does not offer. When using those codecs, you will either have to accept additional lossy compression artifacts on top of the artifacts of the original jpeg codec, or convert in lossless mode with worse compression ratio.
There are some attempts at such comparisons, e.g. https://cloudinary.com/blog/how_jpeg_xl_compares_to_other_im... (cloudinary might be biased, as they are a major driver behind jpeg-xl)
My overall impression is that it's roughly on par or a little behind with AVIF, somewhat ahead of HEIC and very ahead of WEBP (and jpeg of course) in quality-per-byte.
As for encode and decode speed, it's a bit slower than WEBP but beats AVIF and HEIC by a margin (unless these codecs are back by hardware encoders). However, all three codecs are relatively new and there may be a lot of future speed optimizations possible. Especially jpeg-xl, which isn't "backed" by general AV1 (AVIF), vpx (WEBP) or HEVC (HEIC) research, may have a particularly large yet to be tapped potential for (hardware-specific) optimizations. As far as I know, jpeg-xl is also "simpler" than those other codecs (which basically used the iframes of their complex sibling video codecs as a base), so it might be more feasible to have more competing implementations.
But time will tell, and it will of course also largely depend on the adoption and thus network effects of these formats who much "love" each format receives from open source and proprietary software vendors.
Is there some variant of libjpeg-turbo that works for XL?
Ouch.
I'll stick to regular jpeg for now... I can crank out a 1080p encode in under 10 milliseconds using libjpeg-turbo on my target platform.
The libjxl default is "squirrel" (=7), and you can go as low as lightning (=1), so you have a choice to trade compression efficiency for compression speed.
Additionally, there are different decoder modes (in the encoder), allowing you to make tradeoffs between efficiency/density vs speed.
But yeah, libjxl will remain slower than libjpeg-turbo, especially on encodes.
dont see hollywood moving away from it pretty soon.
problem is jpeg2k just is too heavy to compute.
I recently tried to build a command to automatically add a silent audio track on inputs, while also adding pause and concatenate 300 small videos, all of this in a single command, but I still have a lot of trouble understanding how stream selector work.