What would we do about all of the processes that fork but don't ever exec? And isn't posix_spawn implemented in terms of vfork, which a lot of people like even less?
You could hang and hope someone else free()'s enough for you to continue, but the system will lock up if too many processes try that without giving up after a timeout
There are lots of non-fatal ways to handle an out of memory condition (including doing your own paging) depending on the use-case of the allocation. Programmers suffer from learned helplessness because they're writing for systems that lie to them about memory and languages that make handling errors a pain.
That sounds obvious until you realize that you can’t get away with just fixing the obvious parts of allocating large optional caches, you have to defend for every single allocation you do to fail. In every library you use, for every string you copy, for every structure you create. That’s a lot harder than handwaving about how someone else might be able to fix some cases.
Yes, and? The difficulty of this is overstated. Except maybe that library thing, but my general guiding principle is to use as few dependencies as possible already anyway so that mitigates it somewhat.
If there are some situations where there is no logical course of action? Then yeah, crash I guess. Assuming your software isn't responsible for people's lives. It's still better than the OS randomly killing you because some other program fucked up.
I mean this is just crazy. The OS lies to you about memory, you ignore it anyway since it always lies to you, and then we randomly kill programs when we're out of memory. This is not a reasonable way to do things.
The problem, as I see it, is that by now programmers are so used to not paying attention to memory (or any other resource in most cases) that there's a ton of inertia behind the "nothing can be done" sentiment.
Leave that aside for a sec. Are you honestly saying that because it isn't easy to do this, the best course of action is to make it impossible by lying to the program?
If it bothers you, you can just turn off overcommit. It’s not hard:
$ sysctl vm.overcommit_memory=2 # policy number 2
$ sysctl vm.overcommit_ratio=0 # ratio = 0%
So here’s your golden opportunity to show us how easy it is to program against that!
I didn’t claim that it is not easy. I claimed that it is practically impossible. You claimed it was easy. So please, prove me wrong and just show us how easy it is.
That's for the entire system though. If this behavior is default because programs suck at dealing with memory then that will definitely cause problems, right? That's why we're in this shitpickle to begin with. Obvious solution is obvious: opt in to no-overcommit behavior on a per-program basis and exempt programs that opt-in from being OOM killed.
> I claimed that it is practically impossible.
Suddenly it makes sense why modern software is such unreliable slow garbage. You don't by chance work for Microsoft on the Windows Terminal team do you? If this is so impossible, why would so many other OSs actually correctly not-lie to the program about memory allocation failure?
> So here’s your golden opportunity to show us how easy it is to program against that!
Alright, what would you like me to program for you for free and how long are you going to give me[0], because the vast majority of code I write doesn't even have to allocate at all. Even the stuff that does just does the obvious thing when it can't allocate what it needs: tell the user it can't allocate what it needs and exit as gracefully as possible. Even that is impossible on an OS that fucking lies to you.
[0] No, I'm not going to actually do it. This whole tactic is predicated on the idea that it is more work than someone is willing to go through to prove a point on the internet, you know that, that's why you went there.
So, to the surprise of really no-one, you claim programmers have ‘learned helplessness’ because they can’t apply your easy solutions, but back down on the ground when it’s time to demonstrate your superior abilities, it turns out to be all talk.
You don’t need to go and write software for me, if your arguments are worth anything you would be able to demonstrate them using the software you already wrote as you gained the experience to back your handwaving claims. Clearly you didn’t do that (if you did you would have known your original claims were a bit shallow), you just show up on your high horse to talk down to others who do. All talk and no action.
Or to put it another way, the only way you are able to defend the ludicrous idea that an OS should be lying to programs about their allocations is to claim it is impossible and get all demanding and pissy to the first person who comes by to contradict you.
Which is ridiculous, because plenty of programs will stop and tell you they have run out of memory for an operation. It's the easy way to handle allocation failure you're so hell bent on me showing you.
You can do that kind of thing when the OS doesn't lie to you. Depending on circumstances you can do other things.
But hell, even if you think that's stupid and I'm stupid for saying that this is better than crashing, I still can't fathom why you think the status quo on Linux is the best possible idea. Hey, I know, let's pretend we can't run out of disk space either and just randomly delete files when the disk starts to fill up.
I never said any of these things, I said it is practically impossible to deal with not getting memory when you need it, which you claim but can’t show is easy. And then there’s a whole lot of rambling about operating systems which has nothing to do with it.
If you do one huge allocation and it fails, yes it’s easy to handle that. But programs do tons of small allocations all the time and it’s impossible to handle if that fails. Do you think you can even show that fancy pop-up when you can’t allocate memory?
I didn't say it was easy, I said the difficulty (nearly impossible, according to you) was overstated. I also said there are circumstances where there isn't really anything the program can reasonably do other than terminate. However, there are other things it can do in other circumstances.
And the OS behavior has everything to do with it, that's the whole context of this discussion: Linux lying to programs about memory and having to deal with the consequences in a really dumb way.
> But programs do tons of small allocations all the time
Mine don't, at least they don't allocate a bunch of small shit from the OS all the time[0], that's slow. Again I suspect learned helplessness here: people have been doing it a certain way for so long they don't even consider alternatives.
[0] Actually, come to think of it I don't think most malloc implementations do it differently.
I think you are missing the point of an OOM kill. The intention is to keep the SYSTEM healthy, not any one application. Let me assure you, system-wide, this policy is sane. Do you have any idea what happens when the kernel is not able allocate when it needs to? Bad things.
This is different from whether applications handle this reality well (by and large, they do not). You are 100% correct with this. As is the poster who says fixing it is next to impossible (given the wide amount of deployed, critical software). I think if you actually dive into this, you will find Windows does not handle this as gracefully as you want to believe it does.
Everyone says "turn of overcommit" as if it is a general solution. This works only when you have tight control of what runs on the system, and it is tailored for a specific application or deployment that behaves well. And guess what? If this is you, then you already know how to disable overcommit in the kernel build for the custom OS you are building for your product. I will leave it as an exercise for the reader to determine if the policy of crashing the current process is a better strategy than what the OOM killer does (it favors to reap shorter-lived and larger-RSS processes).
Linux is not Windows. People get upset when they first realize this. They want different defaults (nevermind that they can customize it however they want to, and it isn't as though this is some esoteric kernel topic buried in the lkml that should surprise anyone).
If anything, I'd say that Linux needs easier ways to securely load a different set of code for running at OOM time.
> The intention is to keep the SYSTEM healthy, not any one application
Right, but it is only necessary because Linux lies to the application. If Linux instead reserved memory for itself to remain responsive and properly allowed allocations to fail when they would exceed the available resources, the applications that ask for memory that is unavailable would be the ones having to deal with the consequences rather than the OS having to kill processes based on some heuristic.
> I think if you actually dive into this, you will find Windows does not handle this as gracefully as you want to believe it does.
Personally, when I've seen modern Windows systems grind to a halt it is because of disk IO saturation, not memory saturation. Maybe I'm just not in the right contexts to see it.
> You are 100% correct with this. As is the poster who says fixing it is next to impossible (given the wide amount of deployed, critical software).
Ok, so that may be speaking at cross purposes then. I mean on an individual application level this is a not-too-hard solvable problem (usually). If they mean "given the current state of the ecosystem, it is globally nearly impossible" then I can see where they are coming from, but again I have to wonder what the point of having all this software be FOSS is if we can't work towards a goal like this.
> Linux is not Windows.
Yes, and in many ways I find this unfortunate. Good ideas are good ideas regardless of which OS they come from, fanboism is stupid.
> They want different defaults
Because defaults matter. As mentioned, because of the default applications have been written a certain way over years and years and now it is not a simple matter of disabling overcommit because of what happens to that software when you do that. That's why I proposed a solution based on opt-in.
It is in fact not merely necessary because the OS will overcommit. Even without overcommit, you hit the same problem, except you have no way to decide how to handle it---the application requesting the allocation suffers.
I guess my entire point with respect to defaults is that for anyone who knows what they're doing and the defaults do not work for them, they already have a huge pile of systems design and architecture work to do, and the mechanics of changing this OOM policy for that is a trivial change.
For people who do not know what they're doing and want the OS to do the hard stuff for them, there is no sane default. There are tradeoffs that will make large numbers of people unhappy. It is a no-win situation.
I don't understand why you believe that forcing the process that is requesting the allocation to deal with it is the best thing generally. From a systems perspective, it is actually worse because there's a stronger chance you're killing a nice, stable process that might be critical simply because it lost the lottery.
The OOM killer strategy is to try to reap short lived processes that allocated a lot. From a systems perspective, this is clearly better... though as I said it might be better if you could more easily modify the OOM heuristic.
> I don't understand why you believe that forcing the process that is requesting the allocation to deal with it is the best thing generally.
Because it is in the best position to understand the consequences of failure and the options for dealing with it.
> From a systems perspective, it is actually worse because there's a stronger chance you're killing a nice, stable process that might be critical simply because it lost the lottery.
If the process is truly critical, it should be designed to stay stable in the event it can't allocate memory.
That depends on what you mean by ‘crashing,’ doesn’t it?
One could at least fail gracefully: at start-up, request enough memory for a buffer reading ‘request to allocate memory failed; exiting’ and then print it to standard output, then exit.
One might initiate a garbage collection cycle.
One might even allocate enough buffers at start-up to cleanly process outstanding work, write results and then terminate.
It requires a little thought and engineering, but there are approaches which don’t just dump core.
For very large values of ‘a little’. Just like that other poster with the ‘easy’ solutions, I challenge you to actually implement a demonstration of those easy ideas.
You can actually take some ideas from the Linux kernel if you like because it has what you propose, a way to allocate memory that never fails and never blocks. And there you can tell how easy that is.
> What would we do about all of the processes that fork but don't ever exec?
I speculate it isn't common, or at least it isn't required for ordinary desktop apps. Windows survives without fork().
> And isn't posix_spawn implemented in terms of vfork, which a lot of people like even less?
That sounds like a glibc implementation detail, not something required by the standard. I'm not an expert however. Simply, again I come back to: Windows can spawn processes without using it as an excuse to overcommit.
Windows is a terrible OS, especially for servers, so pointing out that it doesn't have something isn't really a great argument for that thing not being very important.
Windows is proprietary and until recently, not very scriptable. None of that is relevant. We're talking about desktop users having their session killed.
> I speculate [fork] isn't common, or at least it isn't required for ordinary desktop apps.
From a science perspective, the ability to fork is really convenient for parallel processing. Load some data and then fork a pool of worker processes, and they can all read almost free copies of that data. This is much easier than setting up shared memory. It's a bigger deal on an HPC machine with 128 cores than a laptop with 8 cores, but even on a laptop it's a significant point in favour of Linux.
Getting rid of it in Linux would also be going against Linus' rule that the kernel never breaks userspace.
Discussions about overcommit are red herring for OOM situations. You could be fully within commit limit for allocated unreclaimable memory, and still be in effective OOM situation, if working set is larger than RAM.
And even if we ignore issues with too large working sets and only focus on memory allocations, then if some badly written program eats all memory, then running some OOM killer (who hopefully kills the offender) is better than keeping it running and just failing all consequent allocations, as such system is unusable anyways.
what? reading your comments here you seem to have a deep and abiding irrational hatred for all things systemd.
It's not systemd-oomd that is the main problem here, it's Fedora's implementation/application of it to all the user@.service. systemd-oomd does suck at logging, but that seems to me like a silly deal-breaker to throw out the baby with the bathwater. It can (and will) be fixed.
> When are we going to stop replacing battle-tested components of our important computers with new and broken systemd functionality?
We aren't. Our time's up. The new generation is on now, and we didn't teach them hardly anything we learned because we were so invested in making fun of them for enjoying ubuntu.
Best way we can move past it is to take the next curmudgeonly step and move to OpenBSD and Plan9, as they're the only things sufficiently dense and opaque to be as yet out of reach of the current generation.
I’m a certified curmudgeon then. If a workload can run on OpenBSD then it does when I’m working. If I’m ever speaking the programmers language and cursing a computer then it is rarely an OpenBSD box. They just work.
What are you talking about? Systemd-oomd is literally part of systemd. Its source code is maintained in the same Git repo, bugs are tracked in the same project, etc. Are you getting it mixed up with Facebook's oomd?
Maybe the problem is with the bloated apps that don't free memory? You can't blame the kernel for that. I routinely see Firefox consuming 10GB of RAM on my Linux desktop. It's nuts.
Chrome and Edge not functioning without overcommit is not the fault of the kernel though.
The browsers will allocate more memory than is physically available in the system but often not "touch" it. Without over-commit this causes an OOM situation and the kernel kills the browser, but that's what it's supposed to do when over-commit is disabled!
I believe some of these massive allocations are used for security purposes or somehow related to the chromium sandbox but I can't remember exactly what's going on. Some sanitizers like ASAN do something similar for detecting invalid memory accesses (and hence can't be used without over-commit either)!
If you read the article, you will see that the issue is that Fedora configures it really poorly for a single user system not that systemd-oomd doesn't work. They enroll all the user processes into it.
Can I ship a badly configured Linux kernel in my distro? Yes.
Is that the fault of the kernel developers? No.
I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it.
Replacing system components is hard and the early days are going to suck. Just because there is a lot of pain and suffering with Wayland does not mean that it was a mistake and we should have just stuck with X11.
Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn.
> Can I ship a badly configured Linux kernel in my distro? Yes. Is that the fault of the kernel developers? No.
If the default configuration were good and you changed it to be bad, that's entirely your fault. But if the default configuration were bad and you just didn't fix it, that's mostly if not entirely upstream's fault.
> I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it.
Yes, we can do that. The point is that we shouldn't have to.
> Replacing system components is hard and the early days are going to suck.
Nobody's complaining that things are getting replaced. We're complaining that the replacement is worse than the old thing.
> Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn.
Systemd didn't exist until 2010, so any changes from 1970 to 2010 clearly can't have made it necessary. And the BSDs are all still good, functional operating systems, despite not being able to run it, so it's clearly not necessary even for what's changed since 2010.
On the other hand, not having a userspace OOM killer means (meant?) that if your system runs out of memory, it will become so slow as to be totally unresponsive and the only thing you can do is to reboot. Which is even worse. Is that still true today or did the kernel OOM killer improved?
I'll just say that earlyoom is much more bearable out-of-the-box than what this sounds like and much better than waiting for Linux to trigger the internal OOM killer (or trying to remember the SysRq key sequence to just trigger it. I always forget and don't have it close by when I need it :) ). I had to play around with the configuration a little to get it to match the conditions I could potentially get an OOM, but when it did reach it on my desktop, it would kill something like Slack, definitely not the whole X session. (FWIW, this happened when building a large Java project, that would burst with memory usage at certain stages of a build, on my computer that just has 16gb. Outside of managing OOM, it's something to play around with before giving up and just upping the RAM. :) )
Ah yeah, that :D But that's a full terminate process/filesystem sync/shutdown + restart ...
You got me to look it up: Alt-SysRq-F just for the kernel oom_killer ...
On ThinkPads, that's: Press Fn-Alt-S, then release Fn-S, then press the desired letter. Result from dmesg: "[154256.637037] sysrq: This sysrq operation is disabled." Bleh. Didn't work when you needed it :) Of course these can be enabled, but earlyoom is soo much easier than remembering the OOM dance or the REISUB dance (particularly if it is just an OOM :D)
Can't confirm. I have more than 32GiB of memory and very fast storage and I see freezes more often than something being killed. The worst I saw was LUKS/dm-crypt being unable to allocate memory and slightly corrupting one of the mounted filesystems.
Linux Desktop in OOM situations is absolutely trash.
anecdotally, i've found that on a low-ram laptop having zram (i.e. compressed swap in ram) helps a lot, smoothing out the memory exhaustion. things start to get slower very gradually (as now swap churn means constant (de)compression rather than disk access), allowing you or the oom killer time to act.
I can also share this sentiment, though I have to add that unfortunately the kernel can still get into a state where it constantly churns pages to-from disk swap as well, freezing so hard that not even alt+printscr+f (oom killer) can react. Reisub works though. It seems to get fixed/reintroduced by version to version..
I found (no hard evidence though) that zswap behaves even more nicely, at least on a Raspberry Pi (and on top of that is easier to configure, it's just an on/off setting that can be done via procfs or boot argument, zswap.enabled=1).
> By having the user@.service template be enrolled in systemd-oomd, Fedora made the cgroup that systemd-oomd would select to be killed be all of your processes
This is so obviously poorly thought-out as to border malicious negligence.
I've been running nohang for quite a while now on all my desktop linux devices. In the rare cases where I mismanaged my memory or had a program hang it has always managed to kill the culprit and only the culprit.
It also gives me a nice desktop notification about what it killed, though I'm not sure where/if a more permanent log is written.
Systemd-oomd, may be great for non-desktop server's needs for some, but definitely not great for single-user desktops, much less multi-user desktops.
THE Pain point remains in systemd-oomd's poor (and often, no) crash dump or traceback support.
Nevertheless, systemd-oomd remains an excellent OOM mass-killer (via cgroupv2), just not the discriminating assassin ... as (mis-)configured ... by Redhat/Fedora.
This is non-init related systemd from IBM->Red Hat->Fedora in a nutshell. It's good for fleets of mega-corp servers. It's bad for human people running linux desktops. But since IBM is the primary entity paying people to write non-init systemd is everywhere and only niche distros avoid it.
Have you ever met someone running fleets of mega-corp servers who was a fan of systemd?
When I've seen them complain the systemd people tell them that they're just stuck in their ways and that systemd is a revelation on desktops where users don't have the historical unix experience holding them back...
I've just heard from people claiming the absurd way NIC are named in systemd is good for server fleet types and the like. Because it's obviously terrible for human people with desktop computers. I guess I just assumed the bad design decisions had some purpose.
I miss the old NIC names, but for systems with multiple NICs the old way would name them differently based on the winner of the init race. It often felt deterministic, but was not.
I've had critical systems fail on reboot because the NICs swapped names so WAN port got configured as LAN and vice versa. That can also create a terrifying security problem too
Yes. When we got a shipment of 1000+ servers that are all supposed to be cabled the same way, both internally and externally, the device naming made it easy to see when something got messed up. This allowed us to hold the system builder accountable, simplify our configuration management, and reduce the amount of configuration discovery a engineer needed to do during an incident.
While sure, I would like shorter device names, the fact is... the only time I need to use them is when ambiguity is something I/we can't really afford.
So, I'm thankful for the cryptic-yet-specific names.
Everyone else (read: desktop people) can use whatever DHCP'd.
o/
I'm not a systemd fan but I'm not a hater either. I think it does its job and, although it was a bit immature when initially released (especially by Fedora) it has matured into a very good init system that does it job.
But yes, Fedora keeps pushing for the bleeding edge, but its users should know this by now.
Yeah. That's also mostly my opinion too. Systemd itself is okay. Just another init system, does some things better, a few things are kind of stirred around for no gain. What I've less been a fan of is the rider tech, networkd, their ntp support, oomd, all of which have more issues and don't seem to be converging. But if they do become stable I expect some new broken tentacle arm to form. :)
okayness meter decreased another notch today: system was ignoring fstab changes. turns out that systemd now handles fstab and you have to trigger the service to re-read the file if you edit it.
I and multiple of my coworkers love systemd. We work in a "hyperscale" environment and we moved from sysvinit through Upstart to systemd at scale, predominantly on RHEL and a little Debian. We were all skeptical at first. But sysvinit and Upstart both sucked for various reasons, and we had tried to work around the warts. systemd has mostly solved all of those problems for us. Some of the things that systemd improved:
Init scripts suck to test and debug. When manually-called, the caller's environment would get used, rather than a clean env, which would cause problems with the poorer-quality scripts. These showed up multiple times in production where an engineer would bounce a process manually and it got some configuration from their environment. We took some steps to alleviate this, like encouraging use of `service`, but using the scripts directly was reflexive for most because of years of indoctrination and it's portability to some other operating systems. systemd eliminated this entire class of issues, because unit files are just config files, not scripts.
Dependencies actually work. Usually, this came up when needing to wait on the network. With systemd, this is just a single dependency in the unit file. It also meant we stopped having to manage S## and K## which became something of a time-consuming art on machines with lots of services.
Not exactly init system, but systemd's triggered units absolutely destroy cron. Manually running a job (e.g. when testing or when a run failed) guarantees the same environment, whereas cron has no similar capabilities. It also actually keeping status means we can more easily see if the last run succeeded. This may ease our monitoring burden, too, but we haven't gotten to give it attention yet.
Certainly no one loves whatever hellhole preceded systemd. It turns out that (some) complexity is non-reducible; if a declarative, properly solved service manager doesn’t cut it, then neither does that shitty ad-hoc bash script system abomination with race conditions.
Dumb question: why is this a thing at all if the kernel oom killer already exists? Doesn't this effectively do the same as the existing oom killer, but worse?
Yes it does. Basically everything under the systemd umbrella does the same thing as something that already existed (remember that we had perfectly good Linux boxes before its release in 2010), but worse than the old thing.
Their post seemed pretty tame. No evocative emotions or deity-assisted emphasis. They just said that between the available free, libre, open-source projects they prefer what they had before -- akin to saying they like to install qutebrowser rather than firefox.
Iirc, systemd-oomd comes from oomd which comes from Facebook who had issues with the kernel oom killer. (The kernel oom killer, especially with slow swap, can also take ages to activate, which means you're just down the entire time, and it's better to kill stuff more often than to sometimes freeze.)
I've had real-world downtime as a result of exactly this. The kernel OOM often takes forever to kick in. Like 10 to 15 minutes is not uncommon.
The most facepalm one was somebody uploaded their `.vimrc` which had a bug that infinitely recursed the parent directory loading ctags until it gobbled up all available memory and crashed. So every time he opened vim, it started about a 40 second timer of doom unless vim was exited before memory exhausted. Took forever to figure it out too, because he often only opened files for 10 seconds or less, short enough not to trigger the OOM.
I believe the kernel OOM killer only fires when you're completely out of memory. But it's not unusual for swap thrashing to make the system unusable before that, forcing the user to either try to kill a likely culprit by hand, or hold down the power button.
systemd-oomd (and earlyoom, which Fedora used for a couple of releases before switching) try to detect that you're running out of RAM before the system becomes unusable, and kill a big memory user. It periodically killed Firefox before I bought some more RAM, which was a pain, but less of a pain than hard-rebooting the laptop.
I confirm that the state of systemd-oomd is ridicolous right now and will easily kill important background processes that aren't even taking up that much RAM. It killed my Thunar file transfer, for example
Systemd seems great for servers and sysadmins. Does it's size, scope, and complexity actually do something for the grandmas of the world who use Linux? For that basic type of user, was the millions of lines of code and probably hundreds of thousands of hours of programmer's time worth it?
It makes changes that break things (like suddenly kill -9'ing all of user's nohup and screen processes when they exited their SSH sessions, completely defeating the purpose of the tools) and replaces old simple ways of doing things with new complicated ways, without benefits of the new ways to justify the changes.
> like suddenly kill -9'ing all of user's nohup and screen processes when they exited their SSH sessions
This was surprising, I was unable to replicate it though. Did you set systemd to automatically clean up sessions?
From the man page of logind.conf,
KillUserProcesses= Takes a boolean argument. Configures whether the processes of a user should be killed when the user logs out. If true, the scope unit corresponding to the session and all processes inside that scope will be terminated.
That is what was doing it, but I didn't set it to do that. It was on by default when it was first released. Since then, the world realized how insane that was and major distros now turn it off for you.
They gave both a compile time and runtime configuration option to set the default behavior, package maintainers didn’t read the docs and shipped it incorrectly. I doubt systemd people are at fault here.
Plus, on a theoretical level that is the correct behavior, you don’t want to leave running processes gobbling up resources under a logged out user without explicitly allowing it.
The point of a default is that it's what a setting should be if you don't go out of your way to change it. The default was on, and you're faulting the distro maintainers for not changing it to off, when it should have just defaulted to off all along. (Or are you saying that it did default to off, and that the distro maintainers originally went out of their way to turn it back on? If so, that would surprise me, but I'd concede the point.)
Correct behavior is that a demonized process doesn't ever get killed just because another process in its original family tree dies.
> Correct behavior is that a demonized process doesn't ever get killed just because another process in its original family tree dies
Who decides what is a demonized process, vs one that just froze and won’t respond to some random UNIX convention, that can’t even reliably signal its state (what is tmux when it is frozen?)
So the solution is to explicitly specify that a given process wishes to live outside of the seat’s lifetime (either by specifying it as a service, which your earlier example ssh is on basically any distro, or by executing it as with systemd-run)
Speaking as one of many who has lost work to the systemd-kills-your-processes issue: No, it's 100% systemd's fault. They shipped a change that broke nohup, screen, tmux, and a dozen other things. All of the feedback they've gotten about the process-killing feature has been negative, but their response has been to shut their ears and blame the problem on distros for not overriding their insane design decision.
Very little locality of behavior. A service can appear in something like 8 different folders, can depend on stuff that is who knows where. It is just a not very elegant system, it's baroque and confusing where it doesn't need to be.
The solution of course is more systemd. Instead of using generic Linux tools like grep, learn and remember a bunch of weird systemd-journald specific commands. Unfortunately I'm also a programmer, and only really have room in my head for one grep command, so it introduces frission and I have to look up the commands every time.
Also just a lot of random breakage over the years.
It’s almost like service management is a hard problem, and that shitty bash one-liner won’t solve anything correctly. It is basically a dependency manager, yet you are not replacing apt/pacman whatever with unix tools piped together, do you?
It's a bit of a false choice to say the options are systemd or write everything in bash. Personally I do like OpenRC.
Service management is a hard problem when you start adding things like socket activation, home directory management, DNS resolving, container management, log management, and whatever else they want to throw in.
Now you might say "well DNS resolving isn't actually part of systemd, it's a seperate daemon", well they've been more and more tightly coupling all these different components together. Yes you can disable systemd's DNS resolver, but probably something else will break. They're all pretty tightly coupled together and they mostly seem to do that in a dumb way.
OOMD seems to have a bug that if you have >64 GB of ram (or maybe >128GB?) it will randomly kill very large processes even though you're nowhere near running out of memory (much less swap).
Cost me god knows how much money or wasted power from lost computations until I figured that out and disabled it.
Same here; been disabling it since it was introduced to Fedora... a release or two ago.
My desktops have 64GB+ of memory typically, they're virtualization workhorses. I don't know what it is in oomd, but any time I make actual real use of it, things started getting killed.
Zero memory pressure. Talking sections of 60+GB free, so it's not fragmentation either.
The grouping of the entire session the link suggests seems likely
Maybe Off-topic, not sure if systemd is to blame or Apache or the package maintainers , but on a LAMP stack i was creating some images in the temp folder then do some work on them.
So for debugging I wanted to opent he temp folder and look at the images. The files were not there, the system is now full of magic, it pretends it gives you a temp file like /tmp/file1.png but the file is not there. I fcking Have o idea why they think is a good idea to lie to me what the file path is, I had luck that some other person hit this issue and somehow google managed to put it as a result to my searches.
It's because Apache httpd or other services are probably using their own "version"/"instance" of /tmp for better isolation from the rest of the system and extra security.
From the point of view of the program the files are written under /tmp, but in reality they're stored somewhere else. Try this to see the files:
ls /tmp/systemd-private-$(systemd-id128 boot-id)-httpd.service-*/tmp
Run
systemctl show --all httpd.service
and look out for the PrivateTmp directive [1]. Use
Thanks. But this is such a antiuser feature, like the incident was like
1 I use std function to get a temp file path
2 I write content in that file
3 I do all the checks to ensure the write worked, I use std functions to check file exists
4 I use std function to print the file path
And the file path is a lie, It all would be fine if it would use the real path, and if a newb would hardocde "/tmp/file1" then I would be happy with a error message that I can't write there .
I assume they had some reasons for the lie, I am not convinced it was worth it, I do not like magic hidden stuff.
I've been bitten by this, so I know the feeling. Once you know what's going on, it's not magic anymore :-)
The "antifeature" can be disabled if you don't like it, but keep in mind that it increases security. There have been vulnerabilities related to temporary files [1][2].
Usually there is no need for a web server running in production to make its temporary files accessible to the rest of the world.
The "lie" is similar to what happens when you're using virtual machines. The virtual machine thinks it's using some hardware, but in reality...
On a side note, Flatpak also does this by default sometimes. It can run applications in a sandbox in which even the home directory is partially inaccessible.
> Achievement unlocked: I just had systemd-oomd kill my entire X session for unclear reasons.
> Does it log details about the session it killed so I could see, say, which program was using too much memory?
> Nope.
>
> Systemd-oomd has now been voted off my island.
To be totally fair, that also happens with the kernel's OOM killer.
Sometimes you wish X had been killed, instead of a total freeze or for example LUKS/dm-crypt being unable to allocate memory (and causing way way way worse issues than X being killed).
It's also problematic that he doesn't seem to have reported the issue to either his distributor or upstream systemd. If he did, it's not mentioned in his rant. There aren't any links to any bug reports where the developers demonstrably failed to care or anything rant-worthy.
Considering systemd-oomd is a relatively new addition to the project, contributed by FB staff, it wouldn't surprise me if there were desktop integration issues yet to be sorted out by the various distros.
The post doesn't strike me as made in good faith, seems written by someone with an axe to grind with systemd.
Systemd is wonderful. But could they please stop trying to invent lightweight replacements?
You're not suckless. Nobody wants you to be suckless. I like you specifically because you aren't anything like suckless.
Don't replace anything unless you have close to full feature parity. Nobody wants your minimum viable product components. The people who do don't want systemd.
Init system? Absolutely. Timers? You've done a wonderful job. Time sync? Please no, crony does it better.
So many things wrong with this article. You can absolutely see which programs triggered systemd-oomd into action in journalctl logs. Even better, I now seem to get notifications with the culprit whenever it kicks in for my desktop setup.
I can also see the design decision made to kill the graphical session when the system becomes unresponsive. The alternative to that situation pre-systemd-oomd was a force reboot of the system.
137 comments
[ 2.8 ms ] story [ 165 ms ] threadI think fork() tends to be implemented using COW memory. Would it help to deprecate fork+exec in favour of posix_spawn()?
Alternatively, just let programs opt-out of overcommit and also OOM killer. Then only badly behaving programs will be shot in the head randomly.
If there are some situations where there is no logical course of action? Then yeah, crash I guess. Assuming your software isn't responsible for people's lives. It's still better than the OS randomly killing you because some other program fucked up.
I mean this is just crazy. The OS lies to you about memory, you ignore it anyway since it always lies to you, and then we randomly kill programs when we're out of memory. This is not a reasonable way to do things.
The problem, as I see it, is that by now programmers are so used to not paying attention to memory (or any other resource in most cases) that there's a ton of inertia behind the "nothing can be done" sentiment.
I didn’t claim that it is not easy. I claimed that it is practically impossible. You claimed it was easy. So please, prove me wrong and just show us how easy it is.
> I claimed that it is practically impossible.
Suddenly it makes sense why modern software is such unreliable slow garbage. You don't by chance work for Microsoft on the Windows Terminal team do you? If this is so impossible, why would so many other OSs actually correctly not-lie to the program about memory allocation failure?
> So here’s your golden opportunity to show us how easy it is to program against that!
Alright, what would you like me to program for you for free and how long are you going to give me[0], because the vast majority of code I write doesn't even have to allocate at all. Even the stuff that does just does the obvious thing when it can't allocate what it needs: tell the user it can't allocate what it needs and exit as gracefully as possible. Even that is impossible on an OS that fucking lies to you.
[0] No, I'm not going to actually do it. This whole tactic is predicated on the idea that it is more work than someone is willing to go through to prove a point on the internet, you know that, that's why you went there.
You don’t need to go and write software for me, if your arguments are worth anything you would be able to demonstrate them using the software you already wrote as you gained the experience to back your handwaving claims. Clearly you didn’t do that (if you did you would have known your original claims were a bit shallow), you just show up on your high horse to talk down to others who do. All talk and no action.
Which is ridiculous, because plenty of programs will stop and tell you they have run out of memory for an operation. It's the easy way to handle allocation failure you're so hell bent on me showing you.
Here's one: https://knowledge.ni.com/servlet/rtaImage?eid=ka03q0000012W2...
You can do that kind of thing when the OS doesn't lie to you. Depending on circumstances you can do other things.
But hell, even if you think that's stupid and I'm stupid for saying that this is better than crashing, I still can't fathom why you think the status quo on Linux is the best possible idea. Hey, I know, let's pretend we can't run out of disk space either and just randomly delete files when the disk starts to fill up.
If you do one huge allocation and it fails, yes it’s easy to handle that. But programs do tons of small allocations all the time and it’s impossible to handle if that fails. Do you think you can even show that fancy pop-up when you can’t allocate memory?
And the OS behavior has everything to do with it, that's the whole context of this discussion: Linux lying to programs about memory and having to deal with the consequences in a really dumb way.
> But programs do tons of small allocations all the time
Mine don't, at least they don't allocate a bunch of small shit from the OS all the time[0], that's slow. Again I suspect learned helplessness here: people have been doing it a certain way for so long they don't even consider alternatives.
[0] Actually, come to think of it I don't think most malloc implementations do it differently.
This is different from whether applications handle this reality well (by and large, they do not). You are 100% correct with this. As is the poster who says fixing it is next to impossible (given the wide amount of deployed, critical software). I think if you actually dive into this, you will find Windows does not handle this as gracefully as you want to believe it does.
Everyone says "turn of overcommit" as if it is a general solution. This works only when you have tight control of what runs on the system, and it is tailored for a specific application or deployment that behaves well. And guess what? If this is you, then you already know how to disable overcommit in the kernel build for the custom OS you are building for your product. I will leave it as an exercise for the reader to determine if the policy of crashing the current process is a better strategy than what the OOM killer does (it favors to reap shorter-lived and larger-RSS processes).
Linux is not Windows. People get upset when they first realize this. They want different defaults (nevermind that they can customize it however they want to, and it isn't as though this is some esoteric kernel topic buried in the lkml that should surprise anyone).
If anything, I'd say that Linux needs easier ways to securely load a different set of code for running at OOM time.
Right, but it is only necessary because Linux lies to the application. If Linux instead reserved memory for itself to remain responsive and properly allowed allocations to fail when they would exceed the available resources, the applications that ask for memory that is unavailable would be the ones having to deal with the consequences rather than the OS having to kill processes based on some heuristic.
> I think if you actually dive into this, you will find Windows does not handle this as gracefully as you want to believe it does.
Personally, when I've seen modern Windows systems grind to a halt it is because of disk IO saturation, not memory saturation. Maybe I'm just not in the right contexts to see it.
> You are 100% correct with this. As is the poster who says fixing it is next to impossible (given the wide amount of deployed, critical software).
Ok, so that may be speaking at cross purposes then. I mean on an individual application level this is a not-too-hard solvable problem (usually). If they mean "given the current state of the ecosystem, it is globally nearly impossible" then I can see where they are coming from, but again I have to wonder what the point of having all this software be FOSS is if we can't work towards a goal like this.
> Linux is not Windows.
Yes, and in many ways I find this unfortunate. Good ideas are good ideas regardless of which OS they come from, fanboism is stupid.
> They want different defaults
Because defaults matter. As mentioned, because of the default applications have been written a certain way over years and years and now it is not a simple matter of disabling overcommit because of what happens to that software when you do that. That's why I proposed a solution based on opt-in.
I guess my entire point with respect to defaults is that for anyone who knows what they're doing and the defaults do not work for them, they already have a huge pile of systems design and architecture work to do, and the mechanics of changing this OOM policy for that is a trivial change.
For people who do not know what they're doing and want the OS to do the hard stuff for them, there is no sane default. There are tradeoffs that will make large numbers of people unhappy. It is a no-win situation.
I don't understand why you believe that forcing the process that is requesting the allocation to deal with it is the best thing generally. From a systems perspective, it is actually worse because there's a stronger chance you're killing a nice, stable process that might be critical simply because it lost the lottery.
The OOM killer strategy is to try to reap short lived processes that allocated a lot. From a systems perspective, this is clearly better... though as I said it might be better if you could more easily modify the OOM heuristic.
Because it is in the best position to understand the consequences of failure and the options for dealing with it.
> From a systems perspective, it is actually worse because there's a stronger chance you're killing a nice, stable process that might be critical simply because it lost the lottery.
If the process is truly critical, it should be designed to stay stable in the event it can't allocate memory.
One could at least fail gracefully: at start-up, request enough memory for a buffer reading ‘request to allocate memory failed; exiting’ and then print it to standard output, then exit.
One might initiate a garbage collection cycle.
One might even allocate enough buffers at start-up to cleanly process outstanding work, write results and then terminate.
It requires a little thought and engineering, but there are approaches which don’t just dump core.
You can actually take some ideas from the Linux kernel if you like because it has what you propose, a way to allocate memory that never fails and never blocks. And there you can tell how easy that is.
So, all of them then?
I speculate it isn't common, or at least it isn't required for ordinary desktop apps. Windows survives without fork().
> And isn't posix_spawn implemented in terms of vfork, which a lot of people like even less?
That sounds like a glibc implementation detail, not something required by the standard. I'm not an expert however. Simply, again I come back to: Windows can spawn processes without using it as an excuse to overcommit.
From a science perspective, the ability to fork is really convenient for parallel processing. Load some data and then fork a pool of worker processes, and they can all read almost free copies of that data. This is much easier than setting up shared memory. It's a bigger deal on an HPC machine with 128 cores than a laptop with 8 cores, but even on a laptop it's a significant point in favour of Linux.
Getting rid of it in Linux would also be going against Linus' rule that the kernel never breaks userspace.
https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
But an app which thinks it has successfully been allocated memory, being chosen at random to crash? Not ideal, just like in OP's example.
Doing this wont give you the fix you think it will.
It's not systemd-oomd that is the main problem here, it's Fedora's implementation/application of it to all the user@.service. systemd-oomd does suck at logging, but that seems to me like a silly deal-breaker to throw out the baby with the bathwater. It can (and will) be fixed.
We aren't. Our time's up. The new generation is on now, and we didn't teach them hardly anything we learned because we were so invested in making fun of them for enjoying ubuntu.
Best way we can move past it is to take the next curmudgeonly step and move to OpenBSD and Plan9, as they're the only things sufficiently dense and opaque to be as yet out of reach of the current generation.
OOMD is a workaround for this behaviour.
And it can't be disabled. I disabled the kernel memory overcommit and then neither Chrome nor Edge were running.
I can't use my computer without my browser of choice. Therefore, overcommit can't be disabled.
The browsers will allocate more memory than is physically available in the system but often not "touch" it. Without over-commit this causes an OOM situation and the kernel kills the browser, but that's what it's supposed to do when over-commit is disabled!
I believe some of these massive allocations are used for security purposes or somehow related to the chromium sandbox but I can't remember exactly what's going on. Some sanitizers like ASAN do something similar for detecting invalid memory accesses (and hence can't be used without over-commit either)!
I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it.
Replacing system components is hard and the early days are going to suck. Just because there is a lot of pain and suffering with Wayland does not mean that it was a mistake and we should have just stuck with X11.
Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn.
If the default configuration were good and you changed it to be bad, that's entirely your fault. But if the default configuration were bad and you just didn't fix it, that's mostly if not entirely upstream's fault.
> I thought systemd is open source and if you don't like something about it, you can just submit a patch or even fork it or just not use it.
Yes, we can do that. The point is that we shouldn't have to.
> Replacing system components is hard and the early days are going to suck.
Nobody's complaining that things are getting replaced. We're complaining that the replacement is worse than the old thing.
> Computing has changed since 1970, some things in our unix-like systems will also need to change to accommodate the current world and some of it is going to be painful, that's just how it is, we made fun of Vista, now it's our turn.
Systemd didn't exist until 2010, so any changes from 1970 to 2010 clearly can't have made it necessary. And the BSDs are all still good, functional operating systems, despite not being able to run it, so it's clearly not necessary even for what's changed since 2010.
On the other hand, not having a userspace OOM killer means (meant?) that if your system runs out of memory, it will become so slow as to be totally unresponsive and the only thing you can do is to reboot. Which is even worse. Is that still true today or did the kernel OOM killer improved?
How does earlyoom compare against systemd-oomd?
Or, as a mnemonic: Reboot Even If System Utterly Borked*
*Being Irish, I prefer "Banjaxed" for the last word, personally.
You got me to look it up: Alt-SysRq-F just for the kernel oom_killer ...
On ThinkPads, that's: Press Fn-Alt-S, then release Fn-S, then press the desired letter. Result from dmesg: "[154256.637037] sysrq: This sysrq operation is disabled." Bleh. Didn't work when you needed it :) Of course these can be enabled, but earlyoom is soo much easier than remembering the OOM dance or the REISUB dance (particularly if it is just an OOM :D)
It's been well over a decade since I've seen that kind of behavior. Storage devices became much lower latency and the kernel swapper got much smarter.
Linux Desktop in OOM situations is absolutely trash.
This is so obviously poorly thought-out as to border malicious negligence.
It also gives me a nice desktop notification about what it killed, though I'm not sure where/if a more permanent log is written.
THE Pain point remains in systemd-oomd's poor (and often, no) crash dump or traceback support.
Nevertheless, systemd-oomd remains an excellent OOM mass-killer (via cgroupv2), just not the discriminating assassin ... as (mis-)configured ... by Redhat/Fedora.
When I've seen them complain the systemd people tell them that they're just stuck in their ways and that systemd is a revelation on desktops where users don't have the historical unix experience holding them back...
I've had critical systems fail on reboot because the NICs swapped names so WAN port got configured as LAN and vice versa. That can also create a terrifying security problem too
Which also was wrong when switching hardware around, but that's a different issue.
eth0 tells you the first ethernet class interface that was activated that boot. Not particularly helpful.
I feel like if you need to care about these names as a user... the distribution has failed to provide effective wrappers. This is window dressing.
While sure, I would like shorter device names, the fact is... the only time I need to use them is when ambiguity is something I/we can't really afford.
So, I'm thankful for the cryptic-yet-specific names.
Everyone else (read: desktop people) can use whatever DHCP'd.
At least it doesn't require a reboot. Yet.
Other times, like oomd, I just want it to get out of the way.
Service dependencies are an area I love. With systemd making a service robust around dependencies like mounts and other services is a cake walk.
If you have manual hand-holding processes on services running under systemd, I suggest looking into supplying these kinds of directives.
I'm torn for networkd. The configuration of interfaces is nice and simple, I've grown to like the link, netdev, network concepts/files.However, forwarding is significantly different. I'd think twice about using it on a system doing NAT/masq
Init scripts suck to test and debug. When manually-called, the caller's environment would get used, rather than a clean env, which would cause problems with the poorer-quality scripts. These showed up multiple times in production where an engineer would bounce a process manually and it got some configuration from their environment. We took some steps to alleviate this, like encouraging use of `service`, but using the scripts directly was reflexive for most because of years of indoctrination and it's portability to some other operating systems. systemd eliminated this entire class of issues, because unit files are just config files, not scripts.
Dependencies actually work. Usually, this came up when needing to wait on the network. With systemd, this is just a single dependency in the unit file. It also meant we stopped having to manage S## and K## which became something of a time-consuming art on machines with lots of services.
Not exactly init system, but systemd's triggered units absolutely destroy cron. Manually running a job (e.g. when testing or when a run failed) guarantees the same environment, whereas cron has no similar capabilities. It also actually keeping status means we can more easily see if the last run succeeded. This may ease our monitoring burden, too, but we haven't gotten to give it attention yet.
It’s only useful on desktop because of its similarities with launchd, which is particularly good on desktop.
The most facepalm one was somebody uploaded their `.vimrc` which had a bug that infinitely recursed the parent directory loading ctags until it gobbled up all available memory and crashed. So every time he opened vim, it started about a 40 second timer of doom unless vim was exited before memory exhausted. Took forever to figure it out too, because he often only opened files for 10 seconds or less, short enough not to trigger the OOM.
systemd-oomd (and earlyoom, which Fedora used for a couple of releases before switching) try to detect that you're running out of RAM before the system becomes unusable, and kill a big memory user. It periodically killed Firefox before I bought some more RAM, which was a pain, but less of a pain than hard-rebooting the laptop.
The plans for Fedora to do this are public pages:
https://fedoraproject.org/wiki/Changes/EnableEarlyoom https://fedoraproject.org/wiki/Changes/EnableSystemdOomd
But it's such a misfit on desktop distros that the criticism there is valid and much louder. It never should've been a one-daemon-fits-all solution.
This was surprising, I was unable to replicate it though. Did you set systemd to automatically clean up sessions?
From the man page of logind.conf,
Plus, on a theoretical level that is the correct behavior, you don’t want to leave running processes gobbling up resources under a logged out user without explicitly allowing it.
Correct behavior is that a demonized process doesn't ever get killed just because another process in its original family tree dies.
Who decides what is a demonized process, vs one that just froze and won’t respond to some random UNIX convention, that can’t even reliably signal its state (what is tmux when it is frozen?)
So the solution is to explicitly specify that a given process wishes to live outside of the seat’s lifetime (either by specifying it as a service, which your earlier example ssh is on basically any distro, or by executing it as with systemd-run)
Okay, so it does that now. But now how do you tell if it's frozen?
The point is that the OS does actually know “statically” that a given process is a daemon or not.
The solution of course is more systemd. Instead of using generic Linux tools like grep, learn and remember a bunch of weird systemd-journald specific commands. Unfortunately I'm also a programmer, and only really have room in my head for one grep command, so it introduces frission and I have to look up the commands every time.
Also just a lot of random breakage over the years.
Service management is a hard problem when you start adding things like socket activation, home directory management, DNS resolving, container management, log management, and whatever else they want to throw in.
Now you might say "well DNS resolving isn't actually part of systemd, it's a seperate daemon", well they've been more and more tightly coupling all these different components together. Yes you can disable systemd's DNS resolver, but probably something else will break. They're all pretty tightly coupled together and they mostly seem to do that in a dumb way.
Ubuntu 22.04 was also affected: https://news.ycombinator.com/item?id=31530739
And flipped a config flag to make it less aggressive: https://www.phoronix.com/news/Ubuntu-Drops-Swap-Kill
Cost me god knows how much money or wasted power from lost computations until I figured that out and disabled it.
This might have made some IaaS firms happy?
My desktops have 64GB+ of memory typically, they're virtualization workhorses. I don't know what it is in oomd, but any time I make actual real use of it, things started getting killed.
Zero memory pressure. Talking sections of 60+GB free, so it's not fragmentation either.
The grouping of the entire session the link suggests seems likely
From the point of view of the program the files are written under /tmp, but in reality they're stored somewhere else. Try this to see the files:
Run and look out for the PrivateTmp directive [1]. Use to override it.[1]: https://www.freedesktop.org/software/systemd/man/systemd.exe...
1 I use std function to get a temp file path
2 I write content in that file
3 I do all the checks to ensure the write worked, I use std functions to check file exists
4 I use std function to print the file path
And the file path is a lie, It all would be fine if it would use the real path, and if a newb would hardocde "/tmp/file1" then I would be happy with a error message that I can't write there .
I assume they had some reasons for the lie, I am not convinced it was worth it, I do not like magic hidden stuff.
The "antifeature" can be disabled if you don't like it, but keep in mind that it increases security. There have been vulnerabilities related to temporary files [1][2]. Usually there is no need for a web server running in production to make its temporary files accessible to the rest of the world.
The "lie" is similar to what happens when you're using virtual machines. The virtual machine thinks it's using some hardware, but in reality...
On a side note, Flatpak also does this by default sometimes. It can run applications in a sandbox in which even the home directory is partially inaccessible.
[1]: https://cwe.mitre.org/data/definitions/377.html
[2]: https://nvd.nist.gov/vuln/search/results?form_type=Advanced&...
https://github.com/systemd/systemd/blob/v252/src/oom/oomd-ma...
Chris's wiki has now been voted out of my sight.
Sometimes you wish X had been killed, instead of a total freeze or for example LUKS/dm-crypt being unable to allocate memory (and causing way way way worse issues than X being killed).
Yeah, looks like logging was added 1.5 years ago so hard to take his perspective seriously.
Considering systemd-oomd is a relatively new addition to the project, contributed by FB staff, it wouldn't surprise me if there were desktop integration issues yet to be sorted out by the various distros.
The post doesn't strike me as made in good faith, seems written by someone with an axe to grind with systemd.
You're not suckless. Nobody wants you to be suckless. I like you specifically because you aren't anything like suckless.
Don't replace anything unless you have close to full feature parity. Nobody wants your minimum viable product components. The people who do don't want systemd.
Init system? Absolutely. Timers? You've done a wonderful job. Time sync? Please no, crony does it better.
I can also see the design decision made to kill the graphical session when the system becomes unresponsive. The alternative to that situation pre-systemd-oomd was a force reboot of the system.