89 comments

[ 0.21 ms ] story [ 4.1 ms ] thread
> backed by its own ELF loader (x86-64 and aarch64) and a glibc ABI bridge

Yacks

If you want to load the OpenGL/Vulkan vendor driver then unfortunately you don’t have much of a choice: those are linked against glibc, and I believe generally also against libwayland (I might be wrong about the latter). If you instead want to load plugins or whatnot into your statically linked executable, then personally I’d argue that you shouldn’t be emulating Linux dynamic linking semantics at all, because the whole late-bound global namespace thing is silly and wrong. (Solaris, which is where Glibc took this model from, moved away from it[1] as much as compatibility allowed, and so did Darwin[2], and Windows never had the issue to begin with, but Glibc persisted and Musl copied it.)

[1] https://www.linker-aliens.org/blogs/rie/entry/direct_binding...

[2] https://web.archive.org/web/20011004090044/http://developer....

[delayed]
The binary itself is completely static; the link even provides commands on how to check this!
Ah yeah my mistake static != no external dependencies. I was thinking of golang static binaries (built without CGO) which do not have a dependency on glibc at all.
I don't know much about musl.

> GPU: Vulkan and OpenGL drivers are supplied by the host as shared objects, usually built against glibc, and a fully static musl binary cannot normally dlopen() them.

Why? Have people managed to break the ancient concept of shared libraries, and this is a fix for that?

musl has no problem building and using shared libraries.

What you can't do is build something statically with musl and then reliably dlopen shared libraries built with glibc.

> Why? Have people managed to break the ancient concept of shared libraries

If you break or remove a shared lib here, you may no longer be able to compile something from source. I had that happen in the past before I started to use more statically compiled programs (and busybox too).

Assuming everything works as-is via shared libraries at all times, makes little sense for ALL linux systems. For instance, some people upgrade glibc manually. Then you need a working base system to resume compilation. I do that for my customized gobolinux system, so I can use any program version as well as any glibc version (assuming I can still compile the program; many older programs no longer compile).

Because many folks don't understand UNIX systems introduced dynamic linking for several reasons, and they actually only had static linking for almost 20 years, since UNIX was known outside Bell Labs.

Additionally many other OSes have had both approaches since their early days, Xerox PARC ones.

For some strange reason they assume to know better than all those researchers.

These decisions, and these studies, were made a VERY long time ago. It's completely unclear why the decisions made then are relevant now, and why they can't be challenged.
It is like advocating that we should drop cars and go back to chariots, because wooden wheels don't get flat, while forgetting why they are mostly used for tourists nowadays.
How are we supposed to take this stuff seriously if the author (sic) isn't even willing to write the readme? Claude exists! If I want some slop I can push the button myself.
Tell me, are there any substantive comments on the text, on what has been done, and on the technical implementation, and not on the form?
Why would I ask you about this work? The root of my question is why is this type of output shared on github, rather than the inputs?
Disclaimer: This is a user’s perspective rather than a programmer’s perspective.

valid point. I am usually okay with LLM generated code since even if it might not be architecturally sound It is usually well commented and has tests and documentation for helping another agent/human debug any issues.

But, just the painful experience of debugging any dlopen related crashes and/or intermittent bugs; and the sheer amount of tokens burnt by an LLM chasing tangents when shown a stack trace; I wouldn’t touch this at least as a packager/consumer of certain apps for personal usage on older distros. So far, AnyLinux-Appimages seem to be a mature solution with great support from the developers, in case anyone lands here for packaging applications to run on older distros.

Modern models, when properly managed with a human in the loop, write higher-quality code than humans and introduce significantly fewer bugs. Therefore, it's quite the opposite - you should expect fewer "dlopen-related crashes and/or intermittent bugs."
> Modern models, when properly managed with a human in the loop, write higher-quality code than humans and introduce significantly fewer bugs.

I don't think anybody believes this, and interjecting it into every thread is not really convincing anyone.

[delayed]
They require admin privileges. A consumer on an enterprise servers might not have them, which is where I would guess glibc might not be latest one. If you are an admin then sure those solutions can be feasible.
It is a testament to the complete failure of the GNU/Linux userland that something like this seems at all attractive to spend time on (or, it seems, LLM tokens).

Actually, scratch that, because Windows and macOS have historically struggled with ABI compatibility as well (macOS less so, due to not caring about backward compatibility in the first place).

How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?

I'm mostly taken aback all the solutions devised to go around the issue, especially the container-based ones. I really disliked it when I grabbed the flatpak version of Blender only to find out that it can't have HIP support. (they might have fixed it by now but you get the point)
Linux loves to leave papercuts unfixed or undocumented for decades. The solution is to build against an older version of glibc but no one tells you that or how to do it.
It's a pretty poor solution, to be honest.

1) Why should I limit myself to the available APIs?

2) Not just glibc. For example, if I build against the latest libstdc++, it will automatically support the more recent glibc. And pinning the old libstdc++ -well, that's just not a good idea.

Luckily Zig makes this quite easy to do. In Mach[0] we are able to just `zig build -Dtarget=x86_64-linux-gnu.2.28` to build GUI apps against an ~8 year old glibc version for maximum compatibility.

This is possible because Zig allows for targeting most glibc versions out of the box with its cross-compilation support.

[0] https://machengine.org

Glibc is ridiculously large. Below are runtime size of "sleep" tool on different systems, measured like so:

  $ sleep 1 & pmap `pidof sleep` | grep total
Ubuntu noble (glibc): 5692K

Alpine v3.24 (musl): 2700K

Manually compiled (dietlibc): 388K

Complete failure is strong words when lots and lots of Linux boxes are running just fine.

I think the disconnect is mostly people that can't decide if they want a stable distro or a rolling release distro. Most everyone uses a stable distro because it's stable, but then the want some up-to-date software that isn't ore-built for their (crusty old) stable distro and they get annoyed. My solution was to finally give in and embrace a rolling release distro (I use arch, btw). If there isn't a package for something I want, it's not hard to build something myself because all my build tools, kernel, and libs are up to date.

Other reasons to want static linking is to distribute proprietary software with no source code available. Linux certainly does not cater to that scenario and I suppose some might call that a complete failure ¯ \ _ ( ツ ) _ / ¯

> How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?

Most Linux distros have been built around the ability to compile their software together in a large repository, from source, so this was rarely ever an issue. Proprietary distribution or executing binaries from the internet like on Windows just wasn't really a common issue.

The problem arises when you start combining distros (glibc and MUSL for instance) or if you try to do the Windows model of sharing software. Historically, projects just compiled different versions for different distros.

When doing static compilation, just targetting an old version of glibc (which is generally forward compatible) also works.

You can hack your way into using software like this (or rather, have an LLM hack its way in) but I don't think any real distro actually cares. This issue exists in a quite small space where people are trying to use proprietary software built for glibc in MUSL environments for whatever reason, and the usual compatibility tricks don't work.

It's a niche use case for most Linux distros. It's not a "complete failure" of the GNU/Linux userland, it's the result of a couple of proprietary components not having MUSL builds available, or MUSL-based distributions not including libraries people want.

I'd like glibc to change so that these hacks aren't necessary for these use cases anymore, but it's not really a problem in practice for the vast majority of Linux use cases.

I disagree that this is a niche problem.

Yes, on the one hand, each specific distribution doesn't have this problem because it can pick up everything it needs.

But for us, independent developers of small programs, the problem is truly stark: we can't afford to build our programs for every distribution. And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access).

And we can't count on someone else packaging our work until we become incredibly popular.

> But for us, independent developers of small programs, the problem is truly stark: we can't afford to build our programs for every distribution.

Most people use the major distributions like Ubuntu, Debian, Fedora, Arch, alpine linux, void linux,… A build machine with VMs can easily take care of those.

> And we can't afford to waste time navigating all the idiosyncrasies of various package repositories, both technical and political (not all repositories allow easy access).

Most (if not all) package managers allow for custom repositories. No need to get access to the official ones.

> How did we get to the point where people feel they need to go to the length of embedding an ELF loader in their binary (!!) rather than just linking with glibc?

Due to FUD, mostly.

Sane people do just link with an old enough version of glibc.

If you dynamically sold an SO are you still static even if you did it "custom" ? At that point it's a dynamic loader in another name?
Technically, you're right, it's a dynamic loader. Technically, it's pure dynamic loading.

If we look at the issue at its core, we're still a statically linked program in a hostile environment, forced to dynamically load device drivers from the system.

It's similar to Golang; on MacOS, it has to use libSystem, even though otherwise, these are the statically linked Go binaries we're used to and love.

Let me add a little more detail: if I use vdso with gettimeofday in a statically linked program on Linux, am I still a statically linked program, or not? :)

Do people say "so", "ess-oh" or "dot-ess-oh"? The title "a .so" is clunky to the "ess-oh" gang.
I don’t think I’ve said it out loud more than a couple times in my life. But in general I think I spell out / pronounce the “dot” in file extensions unless it’s completely obvious from context.
Well .so is short for shared object so I think "a .so" is correct.
That's not generally how the English language works. We say "an SoC" but "a system-on-chip".
If you can figure out your own ELF loader, you can figure out how to build a partially static executable that doesn't need this. You can mix static and dynamic linking. Build tooling around that is just shit.
In this scenario, I'll have to choose which libc I want to run. These won't be portable Linux binaries in the true sense of the word; I'll have to leave Alpine out, and possibly Android, which I don't want.
You don't strictly need to; you can write freestanding C, and use the Linux kernel directly.
Fair point - while it's certainly possible to make a decision per individual library elsewhere, libc is the one thing that really will need to be linked dynamically for glibc, and it'll freeze a minimum version into the binary (i.e. it should be built against an old version of glibc).

I don't think Android is a target in most cases of this, so this would boil down to shipping 2* binaries, one glibc and one musl.

I'd need to check how musl behaves for compatibility, I'm assuming it'll be a minimum like glibc.

All of this said - you'll need to ship multiple binaries anyway, these days: x86_64 + aarch64. Possibly more…

Can you really mix it? When I break or remove a shared lib, some binaries no longer work. With static libs or even better, e. g. statically compiled busybox, I don't have that issue, so I disagree on the claim that mixing solves everything as such. I keep the basic toolchain I use as statically compiled variant. The whole system works better if I can break it less easily.
I'm not sure that's particularly relevant; if I null out bits of the statically linked binary to remove some code paths, they break too.
This is a big forwards-compatibility risk. Suppose glibc adds a new symbol, and then a GPU driver adds a dependency on that symbol. The user wants to run an old executable with the updated GPU driver (maybe the old GPU driver doesn’t support their GPU). Normally, this would work fine: the user has to use a new copy of glibc, which will be compatible with both the new GPU driver and the old executable. But with your approach, the GPU driver is forced to use the glibc reimplementation which has been statically linked into the executable. Which, since the executable is old, can’t possibly implement the new symbol.

The same issue would occur if glibc adds a new version of an existing symbol and then the GPU driver is recompiled. (Or, for that matter, if a GPU driver adds a dependency on a symbol which glibc has always supported but which isn’t in the subset that you reimplemented, though in theory that could be solved if you reimplemented 100% of the symbols.)

Yes this is just asking for trouble - and all it does is solve a problem that doesn't really exist. Just dynamically link against the oldest glibc you want to support. Its annoying that Linux toolchains don't have built in easy mode support for that but its much easier to deal with than this thing will be when it breaks.

It's also not just new symbols, the loader semantics also aren't static and new enough libraries may not support older semantics - e.g. the loader used to use DT_HASH entries for symbol resolution but now they are no longer present on all distributions.

This is not a desirable solution on musl based systems. Whatever you do in that situation ends up horrible, so it is about finding the least bad solution. Which this seems like a workable variant of.
> Just dynamically link against the oldest glibc you want to support

I wish it would that simple for practical use cases.

I ship professional software for colorists for Hollywood studios and they absolutely love to never upgrade. We have to ship for RockyLinux 8. Sad.

RockyLinux doesn't sound so bad. It's not even EOL.
We still have RHEL7 hosts in production. Every day it gets harder to find packages with a glibc 2.17 floor.
(I'm certain you know this already but I'm posting this for people who might want more context)

Part of the problem that people don't really realize is that it's not practical to just 'link against old glibc'.

In order to ensure compatibility with RHEL7 we need to link against RHEL7 libraries, which means we need to build on RHEL7. That means old, unpatched versions of glibc, libpcap, libcurl, openssl, or who knows what else, in case there's some backwards-incompatible change in newer RHEL versions.

Alternately, we can build/patch the libraries we use and compile them statically into the binary, which is vastly more maintenance work for us for very little benefit.

Meanwhile, compiling against older versions of libraries like glibc means we don't get the benefit of updates; not just new features in glibc, but things like more/better SIMD support in glibc algorithms, more/better optimizations in GCC, and so on.

Alternately, we can stop supporting RHEL7, like Redhat did, and build against RHEL8... Or we can build a separate version against each version of RHEL we want to support.

Funnily enough, we've been shipping RHEL7 RPMs of our product for years, and only recently realized that they won't actually run on RHEL7 because at some point the toolchain updated and now we're compiling against too new of a glibc version and then packaging it into a RHEL7 RPM. It wouldn't have worked on RHEL <8 for the past... few years? But no one uses RHEL7 so it went completely untested for ages and we didn't get any customer complaints.

Now we're re-labelling our RPMs as EL8, but it's just a cosmetic change so that we're claiming the version that we actually require.

static link everything? ship a docker image? ship your own userspace (like Oracle DB used (?) to)?
Yeah, to make things spicy, what we ship is a plugin, so there's limit of how static we can link. glibc stays dynamic and that's a huge limiting factor.

I've got plans to try to build on RockyLinux9 and package its glibc along with the app. Simple helloworld works, so I have a glimpse of hope that it would be possible to ship like that.

Neither of those fixes issues with system-calls not existing because the linux kernel was so old.
> Just dynamically link against the oldest glibc you want to support.

Just the other day I tried running an older binary and it failed with a glibc error, despite it being linked to a glibc version that's barely 5 releases behind the one on my system. So maybe glibc isn't backwards compatible after all...

Or maybe you misunderstood the error message.
Or may be not.
This means the user will update the binary with my program and continue using it happily.

I'm not offering a silver bullet, but the approach I've implemented is much better than what the industry currently offers.

Exactly. I think you're doing fine. It's weird that you're using C++ here, since that brings in a lot of crap, but then your programs are probably C++ anyway, so it's not a problem.

Honestly I'm impressed by this work. It's a thankless thing to have built, it's definitely feasible to build but more work than almost anyone would have been willing to do. What would be even neater is if Musl had this builtin.

So we are re-inventing patched a.out files, back when UNIX systems started to introduce dynamic loading, before ELF was invented?

Advocates of static linking keep forgetting once upon a time UNIX only had static linking, then we had overlays, and eventually dynamic linking came to be.

Of course, I remember those times very well.

And I also remember very well that dynamic linking appeared ONLY because we were catastrophically short on memory; everything else was added much later.

Now we have plenty of memory, and we can very well return to our blessed roots!

Not at all, the pain of doing plugins with UNIX IPC was another one.

Ah, you have lots of memory, we're very wealthy. /s

I've implemented the same thing for micron (more or less). One advice I'd give you is to _really_ take care regarding SysV/ELF ABI conventions, there's tons of undocumented stuff in there and it's really easy to mess something up or cause a security defect (see AT_SECURE). That being said the way you're doing is also tricky(ish) because if I understood your implementation correctly you're hooking this into an already running musl which could cause backwards compatibility issues if musl changes under you. Doing this is safer if you control the entire runtime.
> One advice I'd give you is to _really_ take care regarding SysV/ELF ABI conventions, there's tons of undocumented stuff in there and it's really easy to mess something up or cause a security defect (see AT_SECURE)

Yes, it's not trivial, but I hope that over time everything will settle down.

> if I understood your implementation correctly you're hooking this into an already running musl which could cause backwards compatibility issues if musl changes under you

No, I hook this to musl, which is statically linked into my binary, and I have complete control over it.

Every couple of years, I revisit my PL dev hobby and this time I decided to create a language/runtime with pre-emptive scheduling using instruction fuel. While I always do freestanding builds, this time I decided that I also wanted to support native FFI.

That is when I realized the true horror of (g)libc. It wants to inject itself at the root of the library/program and everything from threading to dlopen/dlsym is impacted. I tried a lot of workarounds including trying to implement a loader myself, but the complexity (and fragility) grew so much that I felt it was not worth it.

Finally, I retreated into the safe world of a freestanding runtime + syscalls. FFI, if it has to happen, will occur via IPC of some kind. A second process linked against glibc that will manage calls on behalf of the clean first one.

I have faced a similar issue in the past, and I don't understand how static binaries from the host are supposed to solve this.

From what I remember, GPU access on Linux 'works' by accessing specific FDs under /dev, which are vendor specific - this is what these libs do under the hood.

The libraries don't have any magic powers - if the FD is inaccessible, you won't be able to do anything.

So there's some vendor specific access needed in containers anyway (or a blanket allow, which is a BAD idea).

Also not sure why dynamic linking isn't good enough for this - the issue lies with the permissions, not how you load/link libraries.

You normally do this sort of thing with plan old `dlopen` and `dlsym` and make your own "plugin" system.
Is the 'hello.jpg' reference intentional?
i'm not sure how much people realize that the modern "graphics driver" is actually just "the kernel multiplexes userspace messages to/from the GPU and we've taught mesa to understand each family of GPU you'd ever care to support."

i helped somebody get Doom running on an old embedded system running some 4.x kernel. we just built everything, including mesa, statically and deployed that. imagine building everything static but loading the system's libgl.so instead (which was probably just a symlink or abstraction over mesa's own implementation). what's the benefit: we'd get older, less optimized graphics routines from 5 years ago?

if you're statically linking, then just bring your own graphics "driver". the kernel interfaces are stable enough. it's not conceptually different than embedding `syscall`s directly into your application the same way you do when statically linking libc.

if you really want a single elf that could link to system libraries with a foreign libc (you shouldn't), the somehow correct-ish way to do it is map your preferred ldso manually , setup stack, call DT_ENTRY etc etc like how kernel does it, and just yank the libdl symbols.

see cosmo_dlopen for an example implementation.

The README.md link describes cosmopolitan's approach, among other things, and it's worse than mine.