Given the movement to static linking in compiled languages like LLVM, Go, Rust for deploying self-contained servers and cross-platform single binary executables, I think this guy has clearly lost the argument.
I don't agree with you.
I think that we are on the top of a tide of the usual 'old is new again' hype, but at some point people will realize again all the issues and limitations of that and go back to dynamic linking.
In the same way as containers per application is trendy now.
So far it we are encountering all the drawbacks and limitations that used to be known long time ago and that once leaded to dynamic loading...
> ...but at some point people will realize again all the issues and limitations of that and go back to dynamic linking.
Except that for using third-party libraries and distributing apps or games, there is no reason to use dynamically linked libraries here and also in software distribution.
I guess if you talked to iOS / Android app developers who complained about dynamically linked third-party libraries that slowed app execution startup times, desktop app developers who wanted a single statically-linked executable that prevents their app being hijacked with a malicious dynamically linked library via LD_PRELOAD or DYLD_INSERT_LIBRARIES or in general developers or users don't want to have to upgrade other dependencies using dynamically linked libraries to install your software, then I guess one can say it would be better to use static binaries here.
Who's to say that a hacker can simply modify a .dll, .dylib or .so in your game and enable developer cheats to ruin it. The authors point on LD_PRELOAD alongside with 'security benefits' is nothing more than outdated and has now been intellectually bankrupted.
I am more interested in security for users than for developers. Sure, it's convenient and might be slightly more work for users to mess with your app, but I don't trust every proprietary app developer to be monitoring the CVEs for every library they use.
I'm sure there's a ton of multiplayer games out there that statically linked things like libpng and networking libraries, and by now are an open door into the systems of anyone playing online.
> The authors point on LD_PRELOAD alongside with 'security benefits' is nothing more than outdated and has now been intellectually bankrupted.
I LD_PRELOAD (and DYLD_INSERT_LIBRARIES) code into every process I spawn from the command line, except setuid executables (and sometimes even then–I jump through the hoops necessary to make this work), because it lets me extend compiled applications in arbitrary ways to better suit whatever I want them to do. Saying that my usecase is "outdated and…intellectually bankrupted" just because you're not aware of the very real and very convenient benefits of function interposing is the real ignorance here.
Security is a wrong argument. You can have dynamic loading and locked source folders and disabled LD_PRELOAD.
At the opposite, remember that each statically loaded exe would have to constantly be updated/recompiled for all the libraries updates that they use or could let a lot of security holes opened for a long time.
On the flip side, with libraries that use version symbols, you need to update all of the versions, so dynamic linking might have some of the same problem.
I don't understand people who like containerization. So now instead of just taking on the onus of maintaining a software library, now I'm going to have to fuck around with maintaining how I expect it to be deployed, and force other people to adhere to that model, thereby forcing my paradigms and alternate system architecture/exotic filesystem and concomitant weirdness onto the user?
I don't know, it just never clicked for me. Seems like a solution that appeals to people who like spending most of their time in config/build files than actually using a system, but maybe I'm just a curmudgeon/haven't used it right.
Docker is for applications. Using it in an actual library for a programming language should be limited to setting up the dev environment needed to run tests [for the benefit of consistency]. It makes sense for someone like Discourse to only support running via docker since they want to make sure you're running with the exact same dependencies they're running with which makes issue-reporting and debugging easier.
Static Linking vs Dynamic Linking is a choice that has tradeoffs. If you only consider what's good about dynamic linking and ignore its shortcomings, sure: dynamic linking is The Right Choice!
But I see no mention of those tradeoffs. Examples: what if the system-provided library is old or has bugs? What if you're delivering to multiple distributions (or even OSes) and you need a consistent behaviour? What if you need different versions of a library?
Yeah, this article seems a bit disingenuous... it says "I will prove there are no benefits to static linking!", but then goes on just to mention positives about dynamic linking.
It never even mentions the reasons why people choose static linking, let alone attempt to refute them.
What if the 1.0.1 version of the library isn't 100% compatible with the 1.0.0 and the app now crashes?
Semantic versioning is nice in theory, but you can't rely on everybody getting in 100% correct and not introducing obscure bugs due to slightly different memory layout of structs.
It also makes testing apps more difficult as you now doesn't know exactly what code the clients are running as there can be an incredible high number of permutations of library versions.
Dynamic linking also makes it impossible to do link-time optimizations (e.g. inlining, dead-strip, locating code near the function that calls it)
- With Dynamic Linking, it's the job of the system admin (typically a distribution maintainer) to ensure that the system-provided library and application are compatible — and add patches if needed;
- With Static Linking, the package developer takes control of the whole experience; they deem more important to deliver a working application to the end user than to ensure compatibility with an OS.
This makes sense for isolated applications, e.g. video games.
For a system's core stack (coretools, X11, etc.), the end user expects to receive security fixes in a timely manner. From the distribution's point of view, it seems simpler to have a single library to patch, recompile and push, than to have the maintainer of each package statically linked against said library release an updated version...
Dynamic linking provides some benefits: You save disk space and some memory, especially for things that are used by most of the system like libc. You gain the ability to fix some problems in libraries without updating every app that uses the library.
Dynamic linking as implemented by most UNIX-Like OSes has a huge drawback: you only have one system copy of any given library, so all your applications are stuck using the same copy. There's no easy way to have multiple versions (say, with different features) of the same library without giving them different names. This isn't really a failing of dynamic linking, it's a failing of the OS library management model.
Dynamic linking in general has some other drawbacks: you lose Link-Time Optimization, there's more overhead at load time (and on the first call with lazy linking), and a few other performance-related issues can crop up on occasion.
There's also the issue that replacing a dynamically linked library is a lot easier than replacing a statically linked one for an attacker, so dynamic libraries can allow for vulnerabilities to be introduced (though this also is an OS issue, and there are often good mitigations for it).
I'm more interested in the Nix/Guix approach, where you get the benefit of exact version linking, but also an "immutable" version controlled OS, so things like Puppet aren't necessary to manage state. (And likewise, deploy small VMs/containers with just the package manager tools without needing Docker as the entire OS state can be rebuilt using a definition file/version hash.)
Almost any ELF based dynamic linker has support not only for versioned library dependencies but also for versioned symbols in these libraries. The issue is mostly that some widely used libraries implement this wrong and to larger extent that people who try to ship linux binaries just link to whatever latest version they have on their build system.
> you only have one system copy of any given library
This is clearly not true. Libraries are versioned and one can have unlimited versions installed. Applications can link to the most specific or most generic version desired.
The "X Considered Harmful" trope has become exhausting ("harmful", even?).
Every single technical decision has trade-offs. You might make a case that you (usually) shouldn't do X in cases like Y, or maybe even that you (usually) shouldn't do X in general, but it's a very big statement to say that something is flatly harmful. Very, very few things are so simple. If you aren't Edsger Dijkstra, there's a good chance you're outing yourself as either inexperienced, or willfully ignorant, by claiming them to be so. Yet this meme has encouraged lots of people to do just that, by making dogmatism easy and catchy.
I wanted to quote a few parts of the article that I considered incorrect, but then realized I would be quoting most of the article. Short version, almost anyone who's ever tried to ship a nontrivial game on Linux should realize this article is wrong.
I particularly disagree with the idea that doing any dynamic linking in a program at all means that instantly it's just as non-portable as a fully dynamically linked program. These things are a continuum, there are degrees of fragility -- you can make a program that breaks less often when ported to new systems. See some of Ethan Lee's writing on this topic.[0]
On a meta-level, you should be instantly suspicious of any programming advice that includes quotes like:
> Conclusion: Never use [thing]!
> This has never been the case and never will be the case.
In reality, most software techniques have benefits and downsides. If a debate has survived for over a decade, that means that the answer probably isn't so completely universal that you can completely ignore one side.
I'm not going to say that there are no black-and-white correct answers in software development, but people who regularly claim to have found them should be regarded with suspicion. On average, they'll often just be oversimplifying things.
> I'm not going to say that there are no black-and-white correct answers in software development, but people who regularly claim to have found them should be regarded with suspicion.
I'm going to go a step further and say that the whole reason that software exists as a large profession is specifically because these trade-offs and domain and situation specific.
If we really had one-size-fits-all software as a whole would largely be a solved domain.
Games on Linux are a beast, aren’t they! I think doing this kind of thing successfully requires making good decisions about which dependencies you consider to be part of the system and which dependencies you want to include with your application.
For example, it is fairly reasonable to require LibSDL2 to be installed. However, if you bundle LibSDL2 with your app, you’re now directly dependent on things like X11 and ALSA.
> I think doing this kind of thing successfully requires making good decisions about which dependencies you consider to be part of the system and which dependencies you want to include with your application.
Herein lies the problem, because Linux has absolutely no distinction between "system" and "application". Everything is part of the system.
Controversial opinion but making sure that your game runs well under Wine is not the worst way to ship games for Linux.
This is especially true once the game gets older and less supported. The native Linux libraries might move on over time and the version you require might become unavailable or broken in some way but once a game works on Wine it will likely continue working forever unless you make some change to break it.
These days when it comes to my Steam library I am less worried about the Proton[1] games than the native ones and there have been several cases where I opted to run the Proton version because it worked better than native.
> For example, it is fairly reasonable to require LibSDL2 to be installed.
You mean, "the game's package has libsdl2 listed as a requirement"? Or "if you install the game without libsdl2 present, it will install, but will crash on launch"?
> However, if you bundle LibSDL2 with your app, you’re now directly dependent on things like X11 and ALSA.
Well, either those are present and the game will work, or you've tried to install the game on a headless server and surely you don't expect to miraculously just work?
> You mean, "the game's package has libsdl2 listed as a requirement"? Or "if you install the game without libsdl2 present, it will install, but will crash on launch"?
Are you some kind of malevolent genie? Trying to figure out the worst possible user experience that corresponds to the words I wrote?
> Well, either those are present and the game will work, or you've tried to install the game on a headless server and surely you don't expect to miraculously just work?
The assumption here is that the only two options are either “X11+ALSA” or “headless”, and I don’t agree with that assumption. SDL2 on Linux has two video drivers and five audio drivers. Fact is, I can run horribly old Linux software that uses SDL for audio and it works fine, but old software that uses OSS or /dev/dsp is not going to work. With Wayland, I think it's not unreasonable to prefer a dependency on SDL over a direct dependency on X11.
Oh, I think I understand your point now: there are old versions of SDL2 for old Linuxes, and more recent versions of SDL2 for modern Linuxes, and they use different Linux subsystems, but since they have the same API, any old software that used SDL2 still functions properly to modern Linux even though the low levels of the stack have changed dramatically.
Yes, that's a valid point actually. But ultimately it's a bet on what API will end up more durable: X11 or SDL2, isn't it?
Yeah, it’s a bet on which API is more durable. Some other minor factors are that you might want a newer SDL anyway, because the library comes with a database of different models of gamepads (https://github.com/gabomdq/SDL_GameControllerDB) and if you link against SDL dynamically, you get support for newer gamepad models for “free”.
One of the other factors here is that the ability to play old games is in some sense about the preservation of culture. If you have an x64 Linux binary that uses SDL + OpenGL, I have a strong feeling that you’ll be able to play it 20 years from now, although it might inconvenient to get working.
>almost anyone who's ever tried to ship a nontrivial game on Linux should realize this article is wrong.
How? I'm honestly confused what you mean here, can you explain how to build a nontrivial game on Linux while statically linking GL/Mesa/X11/Wayland/ALSA/Pulseaudio/libpng/etc, and guaranteeing that it will work on all systems and won't have any issues or security problems? Because as far as I can tell this is close to impossible. Or do you consider use of dlopen to be a different case than what the article is discussing?
You don't need to statically link core standard libraries (you don't want to ship an entire OS), but neither should you dynamically link everything in your program.
If you go into shipping a game using only dynamic linking (as this article advocates), then you're going to run into a bunch of portability problems. I'm hard pressed to think of a reason you would ever want to dynamically link to something like vorbis or libpng for a game.
The goal is not ideological purity in either direction, it's to minimize compatibility issues. It's to look at each library and decide whether the upsides of using the system's version (assuming it has that library) override the downsides of needing to deal with all of the different systems and library versions people might be running.
If your nontrivial game has an online component with voice chat or custom avatars or something like that, then you now have an attack vector. If your game is closed source, I hope you are following updates from these libraries and either updating your vendored copy or backporting patches and then immediately pushing out a new release whenever an upstream update is published. And you will likely want to be committed to doing this for however many years you plan to keep your game online.
And if you're going through all the trouble of doing that, I don't see what benefit you're getting from static linking these libraries anymore. (Unless the argument is that you're going to patch it faster than the distros)
And if your nontrivial game doesn't have an online component, then you don't have an attack vector. For a lot of games, you're never going to deal with user-submitted audio files.
If you do have an online component or mod-loader that's affected by a vulnerability, you'll still find that the amount of work required to recompile a game and ship an update when a security release is issued for a dependency is less than the amount of work required to deal with accidental breaking changes in minor version updates, or with systems that don't install the latest package versions, or with systems that have renamed their libraries, or with systems that have stuck custom incompatible patches on top of their libraries.
If you're monitoring CVE releases for your statically linked dependencies, then when something breaks you push one patch and that's it, you're done. The alternative is that the problem never, ever goes away because people don't keep their systems up to date, and they have weird setups, and there's no patch you can ship to address that problem. You really don't want to be in a position where you're supporting multiple versions of the same library. When a security patch gets issued, you're going to do the same amount of testing regardless, so you might as well ship one version of the library at the end of that testing process that you know will work.
Aside from the support issue, there's also the fact that for security reasons you'd really like to fix most security bugs in your app yourself anyway, because, again, people don't keep their systems up to date, so it doesn't matter how quickly distros patch their libraries. If you don't ship an update that fixes the issue for everyone, then it won't be fixed for everyone. You can't trust users to update their own system libraries or not to forcibly maintain older versions, and for any app that needs to maintain a reputation of security (ie, most encrypted chat apps and password managers), you don't want your users to get hacked just because they were using a shared library ended up being vulnerable.
And even that's ignoring the fact that there are tradeoffs even for security fixes. If we're talking about a 10 year old game that's been abandoned, we'd like to have the ability to easily run it, even if we acknowledge that it might have some security holes.
But ultimately, this is really not worth fighting about. Prove Ethan wrong, try shipping dozens of commercial games with the strategy you propose and see whether or not your support requests go up or down. There's a reason game devs overall trend towards statically linking libraries. We're not doing it just to be difficult.
Well, the trick is to ship your own libpng/libvorbis linked dynamically. So you only update a tiny library than half a gigabyte executable when you need a security fix. You also don't depend on the distro to provide the library.
In fact a lot of indie games that support linux do it this way, setting LD_LIBRARY_PATH with a shell script and it works well.
It seems like this is coming down to a wager that you can get security updates out faster than a distro. Short-term this is probably true, long-term I doubt it. The incentives are not there for you.
If a 10 year old game has been abandoned, I don't mind testing or recompiling it for some security updates. I've done this for lots of open source games. I personally would never work on a closed source game, so most of what you/Ethan are talking about doesn't concern me. The situation changes when you stop trying to sidestep distro maintainers, and start committing to working with them. I get that it's inconvenient to do that and you're not just being difficult, but surely you know the trade-offs.
This is how it is done on Windows, which is a far more successful game platform than Linux. Obviously you need to dynamically link to OS interfaces, but you either statically link or ship with your dependencies.
On Linux, relying on system provided dependencies, especially C++ ones, is guaranteed to break over time and over different distributions. So you would have to maintain your Linux version either way, shipping with your dependencies is just less insane.
Maintainers aren't going to keep outdated packages around forever, even if you "work with them". In effect, there is some ongoing maintenance there either way. If I just maintain my own dependencies, then I don't have to work with maintainers from a dozen distributions.
>Maintainers aren't going to keep outdated packages around forever, even if you "work with them".
In general, yes they will. You don't have to do much besides communicate to them that it's safe to bump a dependency and won't cause breakage. And you don't have to track down a dozen maintainers either, simply putting it in your release notes where they can find it is enough. The main reason why packages are dropped is because of lack of maintenance, possibly caused by absent or unresponsive upstream. That's what outdated means. Do you think packagers are going through all the trouble of dropping unmaintained versions of Python 2 on a whim?
There are still too many people out there who think (or even insist) that you should ship your game on Linux. This has never been the case and never will be the case.
The problem is that linux is pretty much a source-level operating system. It's designed that way philosophically and practically... or is that impractically?
Dynamic linking is probably your path if you're developing an app that is packaged and shipped with some distribution of linux and you use standard interfaces.
But yeah, a game would make this all wrong. With dynamic linking your game would last one release cycle. As soon as ubuntu 20.04 comes out, your 18.04 game would stop working. You'd get some weird shared library mismatch in your 100 dependencies and that would be that.
I was trying to figure out how old this page was, but I'm still not certain. The headers show a "Last-Modified" of 2017, but it's definitely much earlier than that.
Dynamic linking has a major versioning problem. We shipped a product that was an .so file. We once spent months chasing a bug in the field where some of the clients would load a previous version of libstdc++ , calls to our .so would crash the application because we were linked against a higher version. Most ABi is backward compatible but not forward compatible. Plus there is the benefit of being more portable.
Didn't see one of the IMO biggest downsides of static linking in that list, which is that there is no sane way to prevent symbol clashes if different components of your application need different versions of the same dependency.
When using shared linking dynamic libraries can link some of their dependencies statically into the .so, using strictly local symbols, allowing the application to load it even if it already links some other version of the same dependency.
We had this problem at my workplace where we developed a component that needed to embed a relatively modern version of the Lua interpreter. When the team that developed the application that used our component tried to link it, they found out that some ancient component they also had been linking in for some completely different and unrelated purpose had a transitive dependency on Lua 5.0, released almost 20 years ago. For backwards compatibility reasons it was not an option to modify the old component to use the newer Lua version, and linking both Lua versions statically into the same binary is impossible. This would never have been a problem if we could have shipped our component as a shared library (which we already build and use for other internal applications), because there the Lua interpreter could have been linked into the shared library privately.
--redefine-sym old=new
Change the name of a symbol old, to new. This can be useful when one is trying link two things together for which you have no source, and there are name collisions.
--redefine-syms=filename
Apply --redefine-sym to each symbol pair "old new" listed in the file filename. filename is simply a flat file, with one symbol pair per line. Line comments may be introduced by the hash character. This option may be given more than once.
I’m not sure I understand how this would help? When linking the main application there will be two component .a files linked into it, that both contain references to symbols that should be resolved to something in a liblua.a, so that needs to be linked along as well, but one component needs to resolve them to liblua.a from Lua 5.2, the other to liblua.a from 5.0. Most (but not all) of the Lua symbols from both versions have the same names. How can objcopy resolve this? Multi-pass linking or something like that?
You rename things in liblua.a from version 5.0. to something different (append __5.0 to names or something), then in the .a file that should link to version 5.0, you rename all missing Lua symbols too. Then you link all this together:
main.o
component_for_lua5.0.a (modified: referenced Lua symbols renamed)
component_for_lua5.2.a (with symbols left intact)
liblua.a (from version 5.0, modified: symbols renamed)
liblua.a (from version 5.2, its symbols left intact)
Aha got it, thanks! Like you said it’s not pretty but it works, and somewhat better than the abomination they came up with instead, which involved building only the minimal part that depended on the new Lua 5.2 as shared library, then making fake shared libraries for all remaining unsatisfied symbols referenced from it to trick the linker, and compiling them in statically instead like they used to do before. Hard to explain, don’t ask me why they did it... :-/
Edit: there is one pretty glaring downside to this approach though, which is that both the component we need to patch up with renamed symbols and the liblua.a are also used in other places, so they would either have to be built differently depending on the application, or we would have to maintain multiple versions of the same 5.0 liblua.a and change the build config/detect logic, for even more confusion and ugliness >_<
Introduce the project- nay, the organization-wide mangling scheme! Every symbol is now required to have the library's version name in its name, may as well throw in some namespaces there too. Then maintain some sort of metainfo of what part is requiring what versions, and what project is providing what versions, and... and well, at this stage you're kinda reinventing the package management, but at least you don't need a custom linker, which is a good thing, right?
The not-so-funny part is that it would probably be about the same effort to do that compared to convincing everybody to use shared linking for these kinds of things >_<
> Didn't see one of the IMO biggest downsides of static linking in that list, which is that there is no sane way to prevent symbol clashes if different components of your application need different versions of the same dependency.
wouldn't symbol versioning allow to do something like this ? it needs a bit of modification of the Lua sources but that can likely be quickly automated with some bash script
> fixes (either security or only bug) have to be applied to only one place: the new DSO(s).
Yes, and all of your bugs can be introduced in a new single place as well!
Seriously though. People love static linking because it solves more problems than it introduces, despite its downsides.
I will argue that it’s easier to keep a system secure with static linking. You can upgrade the service that is insecure immediately. Or else you need to wait for a Big Bang deployment, and it can be held up by some random thing that isn’t even in the critical path.
Although I get the point of dynamic linking (security, size...), I still prefer CLI to be written in a language that enable you to easilly build a static binary such as Go, Rust. IMO, the benefit of static binary is that I can literally using wget, mv, scp or other tools to move them to other machine and expect them to just work. To install a binary (not provided by my distro) locally, I only need to copy the result binary into my `~/.local/bin` and it simply works.
That is a terrible article.
I appreciate statically linked programs that load faster
and force the code to behave as a unit vs. a collection
of libraries that require the housekeeping of ensuring the
correct version the program and the library or else...
This article is from 2004 - a time when memory and disk were comparatively expensive to the price they are today. Nowadays those cost factors aren't an issue. Package managers are regularly utilized to keep systems up-to-date and developers now regularly build, test, and distribute their software. The changes between now and 2004 are so stark that now we'd consider dynamic linking to be harmful! How times have changed!
I don't know about you, but I always static link when possible because I am so tired of the DLL (SO) hell.
Quick real-life example: I had a program that heavily relied upon curl library and fork(), compiled (dynamically linked) and ran in debian 8 would work flawlessly, when ran over debian 9 would have a lot of memory leaks and sometimes SEGVs, when statically linked the program would ran the same way (good) in debian 8 and 9. It took me days to track down and find out that unlike in debian 8 in debian 9 the curl library uses a threaded dns resolver which of course is not a very smart thing to fork() after, so the new curl library had a new feature that messed up the program logic completely. So, no, I would say static linking is way better than dynamic because is CONSISTENT, the library functions you call are guaranteed to behave the way you think they will. Also the ltrace, LD_PRELOAD, etc doesn't legitimize the use of shared libs, why would you debug a release binary?
Quick real-life example: I had a program that heavily relied upon curl library and fork(), compiled (dynamically linked) and ran in debian 8 would work flawlessly, when ran over debian 9 would have a lot of memory leaks and sometimes SEGVs, when statically linked the program would ran the same way (good) in debian 8 and 9. It took me days to track down and find out that unlike in debian 8 in debian 9 the curl library uses a threaded dns resolver which of course is not a very smart thing to fork() after, so the new curl library had a new feature that messed up the program logic completely. So, no, I would say static linking is way better than dynamic because is CONSISTENT, the library functions you call are guaranteed to behave the way you think they will. Also the ltrace, LD_PRELOAD, etc doesn't legitimize the use of shared libs, why would you debug a release binary?
- A bugfix to a library requires every program that uses it to be relinked. It's true for dynamically linked programs, too, they are relinked over and over. The solution to a forgetful admin is a freaking makefile.
- Security by obscurity, don't care. But if you care, addresses can be randomized during static linking, which should happen on the target system anyway (see previous point).
- Nobody cares anymore. Hard disks are enormous, memory is enormous, a tiny fraction of either is taken up by binaries. Unless you're on a microcontroller, where you don't have enough ram to even consider dynamic linking.
- No! Bad Ulrich! Terrible, horrible, awful design! Bad Ulrich! The "features" of glibc that require dynamic linking should be taken out and shot. The stuff that is too complicated to be implemented without dynamically loading code should be moved into a separate process. Just use dbus, it's pretty much needed for everything anyway. Or embed a scripting language. No, not the one implemented by ld.so!!
- Wrong! You can distribute objects and link on the target system. Which you have to do anyway, see above.
- I'm not sure how I feel about those 'hacks'. At any rate, if we want these hacks, we also get the complexity of the dynamic linker, some form of DLL hell and "Warning: gethostbyname() requires the exact version of glibc, etc..." I don't like the tradeoff, maybe because I don'd debug using LD_PRELOAD shenanigans.
- Bonus, from the clarification: What, fewer DSOs make startup faster? Didn't you just say, that with prelinking, dynamic linking was as fast as static linking? Did you just make that up?
I get the distinct feeling, Ulrich Drepper is on a crusade to eradicate static linking, and these "arguments" taste a lot like motivated reasoning. I don't know what's driving this guy, but things like the terrible implementation of the NSS cause actual problems in practice. Has he just found a shiny toy and refuses to let go of it? Does he get off on knowing that every program in existence runs a script on startup and he's in control of the interpreter (ld.so)? I don't know, but rationally, his arguments make no sense.
73 comments
[ 2.0 ms ] story [ 139 ms ] threadBoth him and the authors of glibc.
In the same way as containers per application is trendy now. So far it we are encountering all the drawbacks and limitations that used to be known long time ago and that once leaded to dynamic loading...
Except that for using third-party libraries and distributing apps or games, there is no reason to use dynamically linked libraries here and also in software distribution.
I guess if you talked to iOS / Android app developers who complained about dynamically linked third-party libraries that slowed app execution startup times, desktop app developers who wanted a single statically-linked executable that prevents their app being hijacked with a malicious dynamically linked library via LD_PRELOAD or DYLD_INSERT_LIBRARIES or in general developers or users don't want to have to upgrade other dependencies using dynamically linked libraries to install your software, then I guess one can say it would be better to use static binaries here.
Who's to say that a hacker can simply modify a .dll, .dylib or .so in your game and enable developer cheats to ruin it. The authors point on LD_PRELOAD alongside with 'security benefits' is nothing more than outdated and has now been intellectually bankrupted.
I'm sure there's a ton of multiplayer games out there that statically linked things like libpng and networking libraries, and by now are an open door into the systems of anyone playing online.
you know that even when they are dynamically linked, the developer just ships the dlls next to the app right ?
I LD_PRELOAD (and DYLD_INSERT_LIBRARIES) code into every process I spawn from the command line, except setuid executables (and sometimes even then–I jump through the hoops necessary to make this work), because it lets me extend compiled applications in arbitrary ways to better suit whatever I want them to do. Saying that my usecase is "outdated and…intellectually bankrupted" just because you're not aware of the very real and very convenient benefits of function interposing is the real ignorance here.
- LTO - small libraries/huge pages can link multiple libraries to one page - lower overhead on first call assuming you lazily link
Surely a big reason containers became popular is because you can avoid all the massive headaches you get when deploying dynamically linked software.
I don't know, it just never clicked for me. Seems like a solution that appeals to people who like spending most of their time in config/build files than actually using a system, but maybe I'm just a curmudgeon/haven't used it right.
But I see no mention of those tradeoffs. Examples: what if the system-provided library is old or has bugs? What if you're delivering to multiple distributions (or even OSes) and you need a consistent behaviour? What if you need different versions of a library?
It never even mentions the reasons why people choose static linking, let alone attempt to refute them.
What if the 1.0.1 version of the library isn't 100% compatible with the 1.0.0 and the app now crashes?
Semantic versioning is nice in theory, but you can't rely on everybody getting in 100% correct and not introducing obscure bugs due to slightly different memory layout of structs.
It also makes testing apps more difficult as you now doesn't know exactly what code the clients are running as there can be an incredible high number of permutations of library versions.
Dynamic linking also makes it impossible to do link-time optimizations (e.g. inlining, dead-strip, locating code near the function that calls it)
- With Dynamic Linking, it's the job of the system admin (typically a distribution maintainer) to ensure that the system-provided library and application are compatible — and add patches if needed;
- With Static Linking, the package developer takes control of the whole experience; they deem more important to deliver a working application to the end user than to ensure compatibility with an OS.
This makes sense for isolated applications, e.g. video games. For a system's core stack (coretools, X11, etc.), the end user expects to receive security fixes in a timely manner. From the distribution's point of view, it seems simpler to have a single library to patch, recompile and push, than to have the maintainer of each package statically linked against said library release an updated version...
Dynamic linking as implemented by most UNIX-Like OSes has a huge drawback: you only have one system copy of any given library, so all your applications are stuck using the same copy. There's no easy way to have multiple versions (say, with different features) of the same library without giving them different names. This isn't really a failing of dynamic linking, it's a failing of the OS library management model.
Dynamic linking in general has some other drawbacks: you lose Link-Time Optimization, there's more overhead at load time (and on the first call with lazy linking), and a few other performance-related issues can crop up on occasion.
There's also the issue that replacing a dynamically linked library is a lot easier than replacing a statically linked one for an attacker, so dynamic libraries can allow for vulnerabilities to be introduced (though this also is an OS issue, and there are often good mitigations for it).
This is clearly not true. Libraries are versioned and one can have unlimited versions installed. Applications can link to the most specific or most generic version desired.
Every single technical decision has trade-offs. You might make a case that you (usually) shouldn't do X in cases like Y, or maybe even that you (usually) shouldn't do X in general, but it's a very big statement to say that something is flatly harmful. Very, very few things are so simple. If you aren't Edsger Dijkstra, there's a good chance you're outing yourself as either inexperienced, or willfully ignorant, by claiming them to be so. Yet this meme has encouraged lots of people to do just that, by making dogmatism easy and catchy.
https://news.ycombinator.com/item?id=9744916
I particularly disagree with the idea that doing any dynamic linking in a program at all means that instantly it's just as non-portable as a fully dynamically linked program. These things are a continuum, there are degrees of fragility -- you can make a program that breaks less often when ported to new systems. See some of Ethan Lee's writing on this topic.[0]
On a meta-level, you should be instantly suspicious of any programming advice that includes quotes like:
> Conclusion: Never use [thing]!
> This has never been the case and never will be the case.
In reality, most software techniques have benefits and downsides. If a debate has survived for over a decade, that means that the answer probably isn't so completely universal that you can completely ignore one side.
I'm not going to say that there are no black-and-white correct answers in software development, but people who regularly claim to have found them should be regarded with suspicion. On average, they'll often just be oversimplifying things.
[0]: https://gist.github.com/flibitijibibo/b67910842ab95bb3decdf8...
I'm going to go a step further and say that the whole reason that software exists as a large profession is specifically because these trade-offs and domain and situation specific.
If we really had one-size-fits-all software as a whole would largely be a solved domain.
For example, it is fairly reasonable to require LibSDL2 to be installed. However, if you bundle LibSDL2 with your app, you’re now directly dependent on things like X11 and ALSA.
Herein lies the problem, because Linux has absolutely no distinction between "system" and "application". Everything is part of the system.
This is especially true once the game gets older and less supported. The native Linux libraries might move on over time and the version you require might become unavailable or broken in some way but once a game works on Wine it will likely continue working forever unless you make some change to break it.
These days when it comes to my Steam library I am less worried about the Proton[1] games than the native ones and there have been several cases where I opted to run the Proton version because it worked better than native.
[1] Proton is the version of Wine Steam uses
You mean, "the game's package has libsdl2 listed as a requirement"? Or "if you install the game without libsdl2 present, it will install, but will crash on launch"?
> However, if you bundle LibSDL2 with your app, you’re now directly dependent on things like X11 and ALSA.
Well, either those are present and the game will work, or you've tried to install the game on a headless server and surely you don't expect to miraculously just work?
Are you some kind of malevolent genie? Trying to figure out the worst possible user experience that corresponds to the words I wrote?
> Well, either those are present and the game will work, or you've tried to install the game on a headless server and surely you don't expect to miraculously just work?
The assumption here is that the only two options are either “X11+ALSA” or “headless”, and I don’t agree with that assumption. SDL2 on Linux has two video drivers and five audio drivers. Fact is, I can run horribly old Linux software that uses SDL for audio and it works fine, but old software that uses OSS or /dev/dsp is not going to work. With Wayland, I think it's not unreasonable to prefer a dependency on SDL over a direct dependency on X11.
Yes, that's a valid point actually. But ultimately it's a bet on what API will end up more durable: X11 or SDL2, isn't it?
One of the other factors here is that the ability to play old games is in some sense about the preservation of culture. If you have an x64 Linux binary that uses SDL + OpenGL, I have a strong feeling that you’ll be able to play it 20 years from now, although it might inconvenient to get working.
How? I'm honestly confused what you mean here, can you explain how to build a nontrivial game on Linux while statically linking GL/Mesa/X11/Wayland/ALSA/Pulseaudio/libpng/etc, and guaranteeing that it will work on all systems and won't have any issues or security problems? Because as far as I can tell this is close to impossible. Or do you consider use of dlopen to be a different case than what the article is discussing?
If you go into shipping a game using only dynamic linking (as this article advocates), then you're going to run into a bunch of portability problems. I'm hard pressed to think of a reason you would ever want to dynamically link to something like vorbis or libpng for a game.
The goal is not ideological purity in either direction, it's to minimize compatibility issues. It's to look at each library and decide whether the upsides of using the system's version (assuming it has that library) override the downsides of needing to deal with all of the different systems and library versions people might be running.
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-5146
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-7317
If your nontrivial game has an online component with voice chat or custom avatars or something like that, then you now have an attack vector. If your game is closed source, I hope you are following updates from these libraries and either updating your vendored copy or backporting patches and then immediately pushing out a new release whenever an upstream update is published. And you will likely want to be committed to doing this for however many years you plan to keep your game online.
And if you're going through all the trouble of doing that, I don't see what benefit you're getting from static linking these libraries anymore. (Unless the argument is that you're going to patch it faster than the distros)
If you do have an online component or mod-loader that's affected by a vulnerability, you'll still find that the amount of work required to recompile a game and ship an update when a security release is issued for a dependency is less than the amount of work required to deal with accidental breaking changes in minor version updates, or with systems that don't install the latest package versions, or with systems that have renamed their libraries, or with systems that have stuck custom incompatible patches on top of their libraries.
If you're monitoring CVE releases for your statically linked dependencies, then when something breaks you push one patch and that's it, you're done. The alternative is that the problem never, ever goes away because people don't keep their systems up to date, and they have weird setups, and there's no patch you can ship to address that problem. You really don't want to be in a position where you're supporting multiple versions of the same library. When a security patch gets issued, you're going to do the same amount of testing regardless, so you might as well ship one version of the library at the end of that testing process that you know will work.
Aside from the support issue, there's also the fact that for security reasons you'd really like to fix most security bugs in your app yourself anyway, because, again, people don't keep their systems up to date, so it doesn't matter how quickly distros patch their libraries. If you don't ship an update that fixes the issue for everyone, then it won't be fixed for everyone. You can't trust users to update their own system libraries or not to forcibly maintain older versions, and for any app that needs to maintain a reputation of security (ie, most encrypted chat apps and password managers), you don't want your users to get hacked just because they were using a shared library ended up being vulnerable.
And even that's ignoring the fact that there are tradeoffs even for security fixes. If we're talking about a 10 year old game that's been abandoned, we'd like to have the ability to easily run it, even if we acknowledge that it might have some security holes.
But ultimately, this is really not worth fighting about. Prove Ethan wrong, try shipping dozens of commercial games with the strategy you propose and see whether or not your support requests go up or down. There's a reason game devs overall trend towards statically linking libraries. We're not doing it just to be difficult.
In fact a lot of indie games that support linux do it this way, setting LD_LIBRARY_PATH with a shell script and it works well.
If a 10 year old game has been abandoned, I don't mind testing or recompiling it for some security updates. I've done this for lots of open source games. I personally would never work on a closed source game, so most of what you/Ethan are talking about doesn't concern me. The situation changes when you stop trying to sidestep distro maintainers, and start committing to working with them. I get that it's inconvenient to do that and you're not just being difficult, but surely you know the trade-offs.
On Linux, relying on system provided dependencies, especially C++ ones, is guaranteed to break over time and over different distributions. So you would have to maintain your Linux version either way, shipping with your dependencies is just less insane.
In general, yes they will. You don't have to do much besides communicate to them that it's safe to bump a dependency and won't cause breakage. And you don't have to track down a dozen maintainers either, simply putting it in your release notes where they can find it is enough. The main reason why packages are dropped is because of lack of maintenance, possibly caused by absent or unresponsive upstream. That's what outdated means. Do you think packagers are going through all the trouble of dropping unmaintained versions of Python 2 on a whim?
Dynamic linking is probably your path if you're developing an app that is packaged and shipped with some distribution of linux and you use standard interfaces.
But yeah, a game would make this all wrong. With dynamic linking your game would last one release cycle. As soon as ubuntu 20.04 comes out, your 18.04 game would stop working. You'd get some weird shared library mismatch in your 100 dependencies and that would be that.
try this for fun:
I think the main content dates from at least 2004 from when it was on Redhat's site: https://web.archive.org/web/20041221091001/http://people.red...
The Clarification looks to have been added in late 2006: https://web.archive.org/web/20061119024251/http://people.red...
When using shared linking dynamic libraries can link some of their dependencies statically into the .so, using strictly local symbols, allowing the application to load it even if it already links some other version of the same dependency.
We had this problem at my workplace where we developed a component that needed to embed a relatively modern version of the Lua interpreter. When the team that developed the application that used our component tried to link it, they found out that some ancient component they also had been linking in for some completely different and unrelated purpose had a transitive dependency on Lua 5.0, released almost 20 years ago. For backwards compatibility reasons it was not an option to modify the old component to use the newer Lua version, and linking both Lua versions statically into the same binary is impossible. This would never have been a problem if we could have shipped our component as a shared library (which we already build and use for other internal applications), because there the Lua interpreter could have been linked into the shared library privately.
Edit: there is one pretty glaring downside to this approach though, which is that both the component we need to patch up with renamed symbols and the liblua.a are also used in other places, so they would either have to be built differently depending on the application, or we would have to maintain multiple versions of the same 5.0 liblua.a and change the build config/detect logic, for even more confusion and ugliness >_<
wouldn't symbol versioning allow to do something like this ? it needs a bit of modification of the Lua sources but that can likely be quickly automated with some bash script
annotate lua 5 functions definitions to look like
annotate lua 5.1 functions likewise and then in the files which uses Lua 5 put at top level : and conversely VERS_51 for lua 5.1 - now that .o file should resolve to lua_func_v50 or lua_func_v51 when lua_func is calledYes, and all of your bugs can be introduced in a new single place as well!
Seriously though. People love static linking because it solves more problems than it introduces, despite its downsides.
I will argue that it’s easier to keep a system secure with static linking. You can upgrade the service that is insecure immediately. Or else you need to wait for a Big Bang deployment, and it can be held up by some random thing that isn’t even in the critical path.
Quick real-life example: I had a program that heavily relied upon curl library and fork(), compiled (dynamically linked) and ran in debian 8 would work flawlessly, when ran over debian 9 would have a lot of memory leaks and sometimes SEGVs, when statically linked the program would ran the same way (good) in debian 8 and 9. It took me days to track down and find out that unlike in debian 8 in debian 9 the curl library uses a threaded dns resolver which of course is not a very smart thing to fork() after, so the new curl library had a new feature that messed up the program logic completely. So, no, I would say static linking is way better than dynamic because is CONSISTENT, the library functions you call are guaranteed to behave the way you think they will. Also the ltrace, LD_PRELOAD, etc doesn't legitimize the use of shared libs, why would you debug a release binary?
- A bugfix to a library requires every program that uses it to be relinked. It's true for dynamically linked programs, too, they are relinked over and over. The solution to a forgetful admin is a freaking makefile.
- Security by obscurity, don't care. But if you care, addresses can be randomized during static linking, which should happen on the target system anyway (see previous point).
- Nobody cares anymore. Hard disks are enormous, memory is enormous, a tiny fraction of either is taken up by binaries. Unless you're on a microcontroller, where you don't have enough ram to even consider dynamic linking.
- No! Bad Ulrich! Terrible, horrible, awful design! Bad Ulrich! The "features" of glibc that require dynamic linking should be taken out and shot. The stuff that is too complicated to be implemented without dynamically loading code should be moved into a separate process. Just use dbus, it's pretty much needed for everything anyway. Or embed a scripting language. No, not the one implemented by ld.so!!
- Related, stop sabotaging statically linked binaries already, will you, Ulrich?
- Wrong! You can distribute objects and link on the target system. Which you have to do anyway, see above.
- I'm not sure how I feel about those 'hacks'. At any rate, if we want these hacks, we also get the complexity of the dynamic linker, some form of DLL hell and "Warning: gethostbyname() requires the exact version of glibc, etc..." I don't like the tradeoff, maybe because I don'd debug using LD_PRELOAD shenanigans.
- Bonus, from the clarification: What, fewer DSOs make startup faster? Didn't you just say, that with prelinking, dynamic linking was as fast as static linking? Did you just make that up?
I get the distinct feeling, Ulrich Drepper is on a crusade to eradicate static linking, and these "arguments" taste a lot like motivated reasoning. I don't know what's driving this guy, but things like the terrible implementation of the NSS cause actual problems in practice. Has he just found a shiny toy and refuses to let go of it? Does he get off on knowing that every program in existence runs a script on startup and he's in control of the interpreter (ld.so)? I don't know, but rationally, his arguments make no sense.
MUSL is great software, btw.