Alas this is a huge foot gun that ensnares many orgs. Because engineers seem drawn like moths to the flame to Alpine container images. Yes they are small, but the ramifications of Alpine & using musl are significant.
Optimizing for size & stdlib code simplicity is probably not the best fit for your application server! Container size has always struck me as such a Goodhart's Law issue (and worse, already a bad measure as it measures only a very brief part of the software lifecycle). Goodhart's Law:
> When a measure becomes a target, it ceases to be a good measure
This particular musl/Alpine footgun can be worked around. It's not particularly hard to install and use another allocator on Alpine or anywhere really. Ruby folks in particular seem to have a lot of lore around jemalloc, with various versions preferences and MALLOC_CONFIGs on top of that. But in general I continue to feel like Alpine base images bring in quite an X factor, even if you knowingly adjust the allocator: the prevalence of Alpine in container images feels unfortunate & eccentric.
Going distorless is always an option. A little too radical for my tastes though usually. I think of musl+busybox+ipkg as the distinguishing aspects of Alpine, so on that basis I'm excited to see the recent huge strides by uutil, the rust rewrite of gnu coreutils focused on compatibility. While offering a BusyBox-like all-in-one binary convenience! It should make a nice compact coreutils for containers! The recent 0.2 has competitive performance which is awesome to see. https://www.phoronix.com/news/Rust-Coreutils-0.2
The root cause of the issue, is that musl malloc uses a single head, and relies on locking to support multiple heaps. This means each allocation/free must acquire this lock. Imo it's good for single threaded programs (which might've been musls main usecase), but Rust programs nowadays mostly use multiple threads.
In contrast mimalloc, a similarly minimalistic allocator has a per-thread heap, which each thread owning the memory it allocates, and cross-thread free's are handled in a deferred manner.
This works very well with Rust's ownership system, where objects rarely move between threads.
Internally, both allocators use size-class based allocation, into predefined chunks, with the key difference being that musl uses bitmaps and mimalloc uses free lists to keep track of memory.
Musl could be fixed, it they switch from a single thread model, to a per-thread heap as well.
For docker images, cgr.dev/chainguard/wolfi-base (https://images.chainguard.dev/directory/image/wolfi-base/ver...) is a great replacement for Alpine. Wolfi is glibc based. It's easy to switch from Alpine since Wolfi uses apk for package management with similar package names and also contains busybox like Alpine.
My simple rule of thumb: if the general purpose allocator shows up in performance profiles, then there's too much allocation going on in the hot path (e.g. depending on the 'system allocator' being fast in all situations is a convenient but sloppy attitude for code that's supposed to be portable since neither the C standard nor POSIX say anything performance).
> Corollary: hats off to Red Hat for supporting their distro releases for such a lengthy period of time.
This has been my bane at various open source projects, because at some point somebody will say that all currently supported Linux distributions should be supported by a project. This works as a rule of thumb, except for RHEL, which has some truly ancient GCC versions provided in the "extended support" OS versions.
* The oldest supported versions in "production" is RHEL 8, and in "extended support" is RHEL 7.
* RHEL 8 (released 2019) provides gcc 8 (released May 2018). RHEL 7 (released 2014) provides gcc 4.8 (released March 2013).
* gcc 8 supports C++17, but not C++20. gcc 4.8 supports most of C++11 (some C++ stdlib implementations weren't added until later), but doesn't support C++14.
So the well-meaning cutoff of "support the compiler provided by supported major OS versions" becomes a royal pain, since it would mean avoiding useful functionality in C++17 until mid-2024 (when RHEL 7 went from "production" to "extended support") or mid-2028 (when RHEL 7 "extended support" will end). It's not as bad at the moment, since C++20 and C++23 were relatively minor changes, but C++26 is shaping up to be a pretty useful change, and that wouldn't be usable until around 2035 when RHEL 10 leaves "production".
I wouldn't mind it as much if RHEL named the support something sensible. By the end of a "production" window, the OS is still absolutely suitable as a deployment platform for existing software. Unlike other "production" OS versions, though, it is no longer reasonable as a target for new development at that point.
> The mallocng allocator was designed to favor very low memory overhead, low worst-case fragmentation cost, and strong hardening over performance. This is because it's much easier and safer to opt in to using a performance-oriented allocator for the few applications that are doing ridiculous things with malloc to make it a performance bottleneck than to opt out of trading safety for performance in every basic system utility that doesn't hammer malloc.
We also hit this scalability issue at the default memory allocator for musl. Switching back to glibc allowed increasing the performance in production by 5x on a machine with 96 CPU cores [1].
17 comments
[ 2.2 ms ] story [ 58.7 ms ] threadHarmful should be reserved for things that affect security or privacy e.g. accidentally encourage bugs like goto does.
The new one was drafted here: https://github.com/richfelker/mallocng-draft
Optimizing for size & stdlib code simplicity is probably not the best fit for your application server! Container size has always struck me as such a Goodhart's Law issue (and worse, already a bad measure as it measures only a very brief part of the software lifecycle). Goodhart's Law:
> When a measure becomes a target, it ceases to be a good measure
This particular musl/Alpine footgun can be worked around. It's not particularly hard to install and use another allocator on Alpine or anywhere really. Ruby folks in particular seem to have a lot of lore around jemalloc, with various versions preferences and MALLOC_CONFIGs on top of that. But in general I continue to feel like Alpine base images bring in quite an X factor, even if you knowingly adjust the allocator: the prevalence of Alpine in container images feels unfortunate & eccentric.
Going distorless is always an option. A little too radical for my tastes though usually. I think of musl+busybox+ipkg as the distinguishing aspects of Alpine, so on that basis I'm excited to see the recent huge strides by uutil, the rust rewrite of gnu coreutils focused on compatibility. While offering a BusyBox-like all-in-one binary convenience! It should make a nice compact coreutils for containers! The recent 0.2 has competitive performance which is awesome to see. https://www.phoronix.com/news/Rust-Coreutils-0.2
Its not so long ago that the GNU libc had a very similar allocator too, and thats why you'd pop Hoard in your LD_PRELOAD or whatever.
Not every program is multi-threaded, and so not every program would experience thread contention.
In contrast mimalloc, a similarly minimalistic allocator has a per-thread heap, which each thread owning the memory it allocates, and cross-thread free's are handled in a deferred manner.
This works very well with Rust's ownership system, where objects rarely move between threads.
Internally, both allocators use size-class based allocation, into predefined chunks, with the key difference being that musl uses bitmaps and mimalloc uses free lists to keep track of memory.
Musl could be fixed, it they switch from a single thread model, to a per-thread heap as well.
EDIT: Ah, they were mentioned, of course.
On some malloc replacements, telescope -a gopher/gemini client- used to be a bit crashy until I used jemalloc on some platforms (with LD_PRELOAD).
Also, the performance rendering pages with tons of links improved a lot.
This has been my bane at various open source projects, because at some point somebody will say that all currently supported Linux distributions should be supported by a project. This works as a rule of thumb, except for RHEL, which has some truly ancient GCC versions provided in the "extended support" OS versions.
* The oldest supported versions in "production" is RHEL 8, and in "extended support" is RHEL 7. * RHEL 8 (released 2019) provides gcc 8 (released May 2018). RHEL 7 (released 2014) provides gcc 4.8 (released March 2013). * gcc 8 supports C++17, but not C++20. gcc 4.8 supports most of C++11 (some C++ stdlib implementations weren't added until later), but doesn't support C++14.
So the well-meaning cutoff of "support the compiler provided by supported major OS versions" becomes a royal pain, since it would mean avoiding useful functionality in C++17 until mid-2024 (when RHEL 7 went from "production" to "extended support") or mid-2028 (when RHEL 7 "extended support" will end). It's not as bad at the moment, since C++20 and C++23 were relatively minor changes, but C++26 is shaping up to be a pretty useful change, and that wouldn't be usable until around 2035 when RHEL 10 leaves "production".
I wouldn't mind it as much if RHEL named the support something sensible. By the end of a "production" window, the OS is still absolutely suitable as a deployment platform for existing software. Unlike other "production" OS versions, though, it is no longer reasonable as a target for new development at that point.
> The mallocng allocator was designed to favor very low memory overhead, low worst-case fragmentation cost, and strong hardening over performance. This is because it's much easier and safer to opt in to using a performance-oriented allocator for the few applications that are doing ridiculous things with malloc to make it a performance bottleneck than to opt out of trading safety for performance in every basic system utility that doesn't hammer malloc.
[1]https://www.openwall.com/lists/musl/2025/09/05/3
[1] https://github.com/VictoriaMetrics/VictoriaLogs/issues/517