> A concept of system extension images is introduced. Such images may be used to extend the /usr/ and /opt/ directory hierarchies at runtime with additional files (even if the file system is read-only). When a system extension image is activated, its /usr/ and /opt/ hierarchies and os-release information are combined via overlayfs with the file system hierarchy of the host OS.
To be fair overlay fs is a good way to manage network boot / nfs root systems, live cds... All sorts of things where I want an underlying fs that clients can't mess with but they still need to think they can write to it
Overlays are fine. I’ve used them for all sorts of stuff.
Tying them to pid 1, and forcing them to be used all the time for some hacked up thing where /usr is different depending on which process is looking at it? Not so much.
It's basically the ostree idea on a smaller scale. Did you ever want to install some proprietary software with access to system libraries/config, but without actually polluting the global namespace? It's this. Like running a docker service with host as your base layer. (Minus other namespaces)
Honestly, it seems great and not very tricky for a system which already manages namespaces with private mounts.
Munging - messing with. Per the (somewhat abandoned)standard for filesystem layout /usr (except /usr/local) is the domainof the system/distribution. It's not very nice for a package to stomp on that. Although systemd already claim some authority over logs via journalctl of course.
Ghosting: by bind-mounting over the system fs -you might introduce a bug/condition that doesn't show up in a recovery boot scenario, or forensic analyzis of the filesystem/disk - you could introduce "ghost" files/artefacts.
> It's not very nice for a package to stomp on that.
Yes, sure. That doesn't stop hundreds of vendors with custom install scripts who will use /usr and /opt which is the current state of things. Systemd actually provides a workaround here which stops that package from changing those hierarchies globally, which... seems like something you'd want?
> you might introduce a bug/condition that doesn't show up in a recovery boot scenario
Why would you try to run anything but recovery tools in a recovery boot scenario?
> or forensic analyzis of the filesystem/disk
You can reproduce the same environment using the same overlay configuration - that setup is not hidden from you. It's not any more confusing in forensic than a docker service and we deal with them (including bind-mounts) just fine.
Just so we’re clear, unless I’m mistaken this overlay only affects the target software. It’s not allowing said extensions to trample the “root” file system. The altered fs only applies to a limited namespace.
There's actually a directly analogous case in the gaming world, of all things: the app Mod Organizer 2 uses a runtime virtual file system There's actually a directly analogous case in the gaming world, of all things: the app Mod Organizer 2 uses a runtime-only virtual file system to layer together the files from different mod packages, making it easy to adjust for and dynamically disable/enable overlapping mod resource files (of which there can be a huge number in games like Skyrim), as well as redirecting output files from mods to a sane holding area instead of letting them pollute your game installs.
It's a fine concept. Good for replacing stuff like appimage besides the packaging part?
Mixing this with OS package management will require some consideration.
So they are using overlayfs, which means that "upper" directories have visibility precedence on file name conflict. Which means that some auditing facility should be required that alerts if an upper sys image masks a lower one, or if an OS package some time later installs a file to the "lower" directory, perhaps the original rootfs, but a sysfs package still takes precedence…
Actually, I don't think something like busybox included in systemd is necessarily bad. You could have a system recoverable even if everything else breaks. Of course, it would really enable systemd/linux.
Including busybox in the initrd or whatever it's called now is probably a good idea. Having systemd developers write a new interpretation of that, rehashing old bugs is not great.
Most of what you would need to cobble together a busybox like binary is there in the systemd/src/basic directory. stat-util, procfs-util, sort-util, path-util, and so on.
which uses LLVM for building Linux and Systemd, replaces all core utils with non GNU versions.
It use a non GNU libc and so on...
But that aside people often confuse/mixup the systemd service manager and the systemd project which bundles a wide variety of programs which can be used with systemd.
The reasons for this is because systemd itself confuses this. The change log is not split by sub-projects and the sub-projects have sometimes
no clean boundary between them and the systemd
service manager.
I think it would be nice if systemd changes in three ways:
1. Have more clean boundaries between the core service manager and it's sub-project.
2. Have a recurring periods where new features are blocked from being merged and development focused on cleaning up (e.g. fixing bugs). (I can recommend that
to any projects, very few do so).
3. Get "outside" feedback for certain mainly UX aspects, some parts of systemd are out right counter intuitive unless you are not very familiar with it. A thinks which is very easy to happen to any project, but still could be improved on (and maybe already is improved on).
Oh and there where a very few security WTF are you doing moments, I hope there will be no more such moments in the future as systemd on itself it quite
useful for Linux, especially desktop Linux.
I actually seriously foresee a split occurring in the Linux community that will also affect users of other Unix-like operating systems. On one side you have modern Linux software projects like systemd and Wayland. These projects address deficiencies in traditional solutions, but certain design choices in these projects have not been well-received by more traditional users of Linux and the BSDs.
The increase in the amount of user-level software that depends on systemd and Wayland is a forcing factor. For example, my understanding is that official distributions of GNOME depend on systemd, and many Linux desktops are being actively developed for Wayland (though I don't think any major desktop environment or toolkit has abandoned X11). I expect this trend to continue, given that systemd is part of the most popular Linux distributions, Wayland will be default in Ubuntu (https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-2...), and GNOME runs on Wayland (with KDE's port to Wayland being actively developed: see https://www.phoronix.com/scan.php?page=news_item&px=KDE-2021...).
So, yes, I do see a split occurring between "modern Linux" (with systemd and Wayland) and "traditional Linux," with the BSDs fitting in closer to the "traditional Linux" camp.
There will be Linux for desktops to run a web browser (aka poorly emulate a chromebook) and for docker and docker only.
If you want a non-containerized server or some kind of non-desktop embedded thingy, or anything that can be computed other than emulating a chromebook and doing containers, you'll have to run FreeBSD or some other OS.
The days of "just install Debian to do anything anywhere" are gone, which is sad, because there's really no technical reason or need for the destruction.
> The days of "just install Debian to do anything anywhere" are gone, which is sad, because there's really no technical reason or need for the destruction.
What are you missing from Debian these days that was there 5/10/20 years ago? What has been "destroyed"?
Every time I see all the stuff packed into it I feel extreme urge to switch to freebsd again.
Oomd, oh yea, good idea, but WHY it has to be in systemd? The same applies to about 3/4 of things in there. Because of this i've started to call it LennartOS. And also I don't feel like linux is about freedom anymore but more like some sort of weird power display games.
This is a common mistake. systemd-oomd is not "in" systemd the init system. Similar to systemd-resolved, systemd-networkd, systemd-boot, and others it just has the systemd name because it is developed by the systemd team. A minimal system using sytemd for the init system and other software for everything else only needs sytemd, dbus, and journald.
I use several of these projects because I like the systemd design philosophy (ex the way the CLI interface and config files are) and trust the developers of their software. However, there are popular altenatives available for all of these components, should you choose to use them:
systemd-resolved: bind9, dnsmasq, powerdns
systemd-networkd: NetworkManager, ConnMan
systemd-boot: GRUB2, Refind, EFISTUB
systemd-oomd: earlyoom, nohang, oomd (the facebook one, not systemd)
This is exactly my problem. It is made by systemd team. "Everything" seems like it is getting replaced by stuff made by systemd team as if they had patented wisdom or what. Including other perfectly functioning software. I suddenly have to use systemd-* over them.
Like, hell, what is systemd-boot and when and why it replaced grub2?
I don't call this evolution. This is bad form of centralization and seizing power.
You know I feel that way about the Linux kernel. Everything is done by the Linux kernel developers and it’s shipped as one monolithic thing. Wouldn’t it be better to get drivers for networks cards from other places? Or have to pick from a list of memory management packages that get downloaded from GitHub on every reboot? And why do they include things like spin lock implementations? Should that be an NPM package? /s
You're joking but having drivers separate from the rest of the Linux kernel and with a stable ABI would certainly help other OSes as well (e.g. ReactOS, Haiku).
> Like, hell, what is systemd-boot and when and why it replaced grub2?
I've been really happy with systemd-boot; in my limited (used it on probably a dozen machines across several distros at this point) experience, it's better at autodetecting nonstandard images/locations than grub, and is vastly friendlier to weird partitioning schemes. It also has a much more minimal config surface than grub, which I like (since, across distros, I no longer have to learn a distro-specific set of nested/included grub config files and conventions, nor do I have to worry about what will be auto-regenerated when a distro does a kernel update versus what configs I need to edit by hand). However, that limited config surface may be a drawback to people who need to customize very esoteric boot cases.
I also aesthetically prefer systemd-boot's minimalism and focus on newer (UEFI) boot strategies compared to grub; the latter is huge in terms of functionality. A lot of that size is needed for rare cases and older boot configurations (and if you fall into either group, grub is what you should use), but it complicates the "happy path" on post-2010 computers.
> I agree but I wouldn't call systemd a init system.
Systemd is the name of the init system. Systemd is also the name of the project developing multiple systemd-* services. Maybe they could help the situation a bit by renaming the service itself systemd-init, but we've had enough time for people to learn to deal with a tiny bit of ambiguity...
FreeBSD includes a kernel, drivers, and userland utilities under one project. If your complaint is that systemd is "bloated", it doesn't make any sense why you would want to switch to a system that's even more "bloated".
The issue here is branding, not engineering or design.
Systemd is an umbrella under which many key userspace tools are being developed. The actual init system itself is a lot bigger than the old SysV Init system was, but it is separate than all these other things that also say Systemd on them.
The best analogy is GNU. You don't complain that GNU contains a compiler, and also a text editor, and also C System Library do you? GNU is a project that develops a load of components that play well with each other. Systemd is the same.
It's kind of a funny analogy in the context of systemd-oomd, because the latter is sort of like saying "hmm, when the plane is out of fuel, sometimes there's not enough power to actually eject anyone from the plane, so we're going to start ejecting people while there's still a little bit of fuel left!"
For reference, [1] describes the SSD near-OOM thrash problem, where the speed of refaulting file pages is so high that the kernel detects enough activity and doesn't trigger OOM.
It's a heck of a problem to deal with, and a OOM killer will go part of the way to fixing it, but I still have two complaints about the thrash problem.
1. In similar situations on other operating systems, I don't usually see the GUI freeze to an unusable state. In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted. I suppose it's possible that I just haven't used other OSes enough in a long time and they have this too, but this seems like a solvable problem. I assume it's not a CPU issue, since the scheduler should handle that just fine. So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition?
2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it (even if the OOM killer blames something else). Now, I understand that in theory unused memory is wasted memory. So a browser using a ton of memory can be a good thing. But this is only true when memory the browser is holding but not using can be reclaimed to be used by another program, and this seems to basically never happen. If I close and reopen the browser (with the tabs automatically restored) the memory usage drops by 80% or more; the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
I suspect solving these two issues would largely fix thrashing for most desktop users, without the need to ever kill anything, which is obviously undesirable.
Feel free to tell me if one or both of these is incorrect or impossible. These are just my observations, I'm not a kernel dev or anything close to one.
> So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition?
This! Always wanted a /proc/$PID flag to flag process' pages never to go to swap and reserve a number of cache pages specifically for the process. Something similar to mlock.
> So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition?
Because it's not the memory shortage that's nailing the UI. It's disk contention. When pathological swap activity causes disk thrashing, every process that needs to use the disk is now waiting in a very long queue with all the swap page in/out requests and every other process that is now backed up.
If you have a metrics agent that emits a handful of metrics on a per second granularity, you can see a bunch of things happen:
* Memory rises
* Some pages get swapped out
* Even if the system has no other processes writing to disk, you'll see the swap-out correlating with disk writes
* Memory rises some more
* You see more pages get swapped out
* (...and see more disk writes)
* Memory rises some more
* You see pages get swapped out and in, because now we're shuffling out pages that are imminently required by another running process.
* Disk queue lengths rise
* One by one, all processes are waiting (driving up the CPU "wa" number, and causing load averages to shoot up)
This is why, if you're oncall and you're trying to remotely troubleshoot "high load", you likely can't log in because the system needs to access /etc/passwd, /etc/nsswitch.conf, /etc/ldap/ldap.conf, /bin/bash... Your login gets handed a ticket with "∞" written on it, and is told to stand in the line for the disk.
But aren't these files already in the disk cache? Possibly. But another thing that happens when the system runs out of memory is that disk caches get purged. So you have to go back to the disk again. Also, when disk caches get dropped, your disk read performance is dogshit.
So what can you do? Try zswap? Use a different disk devices for swap? Try to pin important processes so the can't be swapped out?
NOTE: I might not be right about this stuff - I've never worked on the Linux kernel, and I rarely even compile it these days. But I've worked as SRE / *Ops for years and saw this problem a lot in badly configured systems. Same patterns all the time.
Thanks for the comment, that's very interesting. I hadn't considered the possibility of the desktop environment needing to do disk reads, I was thinking everything was in memory already.
I suppose the solution is to figure out how to reduce or eliminate the number of disk reads after the DE initially loads, or cache everything needed in memory with a "do not evict" flag on it. Especially for the TTY, that's basically a crucial system function and keeping everything needed for it always available in memory wouldn't take much space relative to a typical desktop system.
I think at least some of the problem is the program code pages getting evicted to disk, so they have to be read back to memory too, which goes to the back of the disk queue. When I'm experiencing thrashing, even basic stuff like alt-tab to switch windows doesn't work. And I assume the window manager doesn't need to do disk reads to switch windows. Even the damn X cursor freezes!
Desktops could be improved quite a bit by tweaking default nice and ionice priorities for apps, instead of relying on one scheduling algorithm to be able to just "figure it out" on desktops, servers, phones, etc. A memory eviction priority, if niceness isn't treated as memory priority, could also help, but the existing tools can already go a long way and they just aren't being used.
I was able to get glitch-free audio by tweaking software and soft-irq realtime and non-realtime priorities, when the default Ububtu (with low-latency kernel) was nearly unusable for audio.
Which problem? Yes, there's always disk contention. But if you don't have swap enabled then you won't have pathological swap activity dominating the disk.
There are several bad things that happen when you run out of memory. One of them is pathological swapping. Another is loss of disk caches. Another is (eventual) oom. The one that has the most detrimental effect, and typically results in the system having to be reset, is swap activity saturating disk io. Even when you've got fast SSDs, because they're still orders of magnitude slower than memory.
You still have pathological disk io even with swap disabled because Linux will evict file-backed pages. More aggressive OOM killing is currently the only solution, which is not ideal, but losing one process beats losing all of them. I use earlyoom, and I assume systemd-oomd is something similar.
For 2) last time I looked I couldn't find a way to have a memory contract with the kernel that allowed for a given allocation to be evicted at will by the kernel while also telling the application that has happened. This would be ideal for apps that can grow their cache unbounded (e.g., photo processing apps with each photo, browsers with each tab) but can also regenerate that cache at will. You end up having tunables for how much cache to hold in RAM which is never correct as depending on what else the user is doing there may be a lot or very little free RAM.
You can use PSI for that. And it actually makes more sense when all applications listen on the global PSI indicators so that if one application would start thrashing another can free up memory cooperatively. But if needed it can also be done per cgroup.
Something like madvise(MADV_FREE) sounds closer than MADV_DONTNEED -- the app can check for zero where it wrote nonzero -- but doesn't guarantee the freeing will be delayed until there is memory pressure.
>In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted.
You can recover system from this state by manually triggering OOM killer using Magic SysRq keys. (Alt+SysRq+F, provided you have SysRq enabled in /proc/sys/kernel/sysrq)
Is there a version of this that works on typical laptop keyboards that don't have SysRq? I could map an unused key to it with xmodmap of course, but if my X server isn't responding I don't think that would help very much!
> 2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it
> Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
I went the ballistic route: most of my browsers run inside local Docker containers, on which I put CPU and RAM quota.
It's my machine: I decide how "citizens" behave : )
Theoretically this does not prevent Firefox from using more than 4 GBs of memory (you need MemoryMax for that), but practically it does.
MemoryHigh=bytes
Specify the throttling limit on memory usage of the executed processes in this unit. Memory usage may go above the limit if unavoidable, but the processes are heavily slowed down and memory is taken away aggressively in such cases. This is the main mechanism to control memory usage of a unit.
MemoryMax=bytes
Specify the absolute limit on memory usage of the executed processes in this unit. If memory usage cannot be contained under the limit, out-of-memory killer is invoked inside the unit. It is recommended to use MemoryHigh= as the main control mechanism and use MemoryMax= as the last line of defense.
I use a similar script to prevent my torrent client from pushing more useful data out of the page cache.
You can put resource limits on basically anything (see systemd.resource-control(5)).
Speaking of which, Firefox behaviour has improved on OpenBSD significantly recently. It used to either crash or act weird when it hit the memory limit in login.conf (which it would eventually do). Now it just works. No idea what changed.
Control groups are your friends. Run your browser in a separate control group with a constrained memory. It took me discovering memory leaks from qemu running X over virgl machines to actually spend a weekend figuring it out, but it is doable.
* I run separate cgroups for admin SSH with (1G min, 4G max ) - worst case scenario I can ssh into the desktop and kill offending processes.
* Separate cgroup for two browsers ( 8G each max ).
* Separate cgroup per qemu VM ( VM max ram + 512M each max ).
After implementing that I went from periodic lock ups to a quick browser crash and a restart maybe once a week.
How much memory is enough for 1? Back on Fedora 32, on a system with close to a year of uptime (it was attached to my TV), gnome-shell consumed nearly 20GB of RAM on a 32GB system! And I rarely interacted with it (outside of watching the occasional concert on it). Most of the time was spent on the lockscreen, with the TV on a different input.
Perhaps this falls outside the category of "any memory condition"... :)
Though, it was responsive. :D
Needless to say, I think 1 is harder than expected. Fedora has made some recent changes on the OOM killer front though, perhaps those are closer to what you're interested in.
> the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
The "Auto Tab Discard" extension[1] for Firefox unloads tab contents, keeping only the title and favicon. Tabs can be put in an allowlist to prevent discarding. I don't know if Chrome or Safari have something similar.
In that regard Linux is much worse than Windows. In Windows you basically never go out of memory in a way to lock the system, sice the OS reserves some memory for its functions. It will block applications, but the operating system remain responsive.
In Linux I was never been able to generate a similar behaviour. No matter how aggressive I set the swappiness, the system didn't began to swap until it was practically locked up with the mouse pointer lagging.
Windows reserves some memory for its functions, for example the UI (maybe the fact that the GUI is in the kernel helps), and thus doesn't have these kind of problem, especially with an SSD (and I know that page file on SSDs is not ideal... but had a disk for 5 years swapping on it and never had a problem).
What do you mean “didn’t begin to swap”? It should always be using swap. It starts locking up when you run out of both swap and RAM (because it’s run out of things to swap out that aren’t critical to what’s currently running). What it fails to do in time is outright kill processes.
I have some strong memories of paging death on Windows XP, on spinning rust. I recall taking some minutes to recover: Ctrl-Alt-Del and wait for secure authentication screen; hit shortcut for Task Manager and wait for it to start and come to the foreground; navigate to offending process and kill it, with blind keystrokes.
The mouse cursor did move smoothly throughout, though, as I recall.
When a cgroup hits its memory.max the kernel OOM killer is invoked. systemd-oomd enables finer grained policy than the kernel OOM killer. systemd-oomd can also prevent livelocks (which memory.max does not).
I think the distinction is that MemoryMax= is just an interface to the cgroupv2 setting, i.e., that rule is implemented inside the kernel and invokes the kernel's OOM killer within a cgroup. The manpage for systemd-oomd says, "systemd-oomd is a system service that uses cgroups-v2 and pressure stall information (PSI) to monitor and take action on processes before an OOM occurs in kernel space."
It looks like systemd-oomd is related to (based on? from the same people as?) Facebook's oomd https://github.com/facebookincubator/oomd , whose documentation gives a bunch of reasons as to why you would prefer a userspace oomd that takes in PSI data and can be configured to proactively kill misbehaving processes instead of just letting the kernel OOM killer handle it. The major reason is time to recovery: a misbehaving process can cause a system to be so far under pressure that the kernel OOM killer will take a long time to flush things out, but a userspace component can respond in advance with more configurable rules (and more flexibility, since the kernel doesn't believe you're at capacity yet).
> It looks like systemd-oomd is related to (based on? from the same people as?) Facebook's oomd
Yep, I authored the original standalone oomd (along with many others' help) and Anita has been working on systemd-oomd. There's plans to converge and simplify everything that's been learned from oomd into systemd-oomd so the community can benefit from all our experiments.
Could they maybe start handling the 1400+ bugs that has been a part of systemd for such a long time (some unresolved since 2015) rather than keep adding and adding and adding and adding code!
OK, but they don't always label bugs that way (example: systemd/#5616) and the bugs that are labeled can get pretty old (systemd/#3700). Furthermore a lot of RFEs are really just disguised bug reports (systemd/#5917 is just an RFE rephrasing of system/#3700).
In my experience, unless you can get Red Hat to care about patching a specific bug, you can rely on opened bugs to stay open.
> don't always label bugs that way (example: systemd/#5616)
Nobody figured out if that issue is in logind or X yet. They did not close it so it's not like they say it can't possibly be on systemd side. (typical Lennart comments aside)
People add features they need, and fix bugs that impact them. Scratching your own itch is the fundamental principle in open source. I recommend you to do the same.
The first line of the tl;dr is: "Having swap is a reasonably important part of a well functioning system. Without it, sane memory management becomes harder to achieve."
Now, afaik linux as a kernel is pretty much designed to have swap memory and really not meant to be ran without it. Except disabling swap is required to run kubernetes, so much so that the kubelet will plain refuse to start if it detects swap memory enabled.
The historic reason Kubernetes didn't let you run with swap is that it's complicated from a resource management perspective - it's not clear whether each pod should be requesting some amount of swap the same way it requests some amount of RAM, and also the container runtimes didn't really have support for that (largely because you need cgroupv2 to have a hope of that sort of per-cgroup swap resource control working at all). But it's not an insurmountable problem, and it sounds from that ticket like they recognize that letting systems have swap enabled for the sake of a userspace oomd is an important use case.
Why is Linux not meant to be ran without swap? In 2021? I've never heard of this. I've operated massive clusters of thousands of hosts with no swap enabled with zero hiccups.
Swap comes with a lot of caveats -- when a performance-critical process begins swapping it can on occasion be more useful to just let it OOM instead.
My personal desktop has 64GB of memory, with no swap enabled. Why would I want swap it I don't need more than that?
Linux works fine without swap. It also works fine with swap. It does not work fine under memory contention regardless of swap, but it is much, much worse if you have many GB of swap on a spinning disk.
Note that managing resources is much harder when swap is in the picture, because you can get the resident memory of a process, but not the swap usage of a process.
It used to be that the recommendation for swap was 2x ram. This recommendation was from the days were single-digit mega bytes of ram was common, and also from when linux core-dumps were both expected, and stored to swap.
Fast forward to 2010ish. You might have a 64GB or 128GB SSD, but you probably also have a spinning drive. you have 8 or 16GB of ram, so your 16-32GB swap would be 1/4 the SSD, plus you heard that putting swap on an SSD wears it out faster, so you put a 16-32GB swap file on the spinning drive.
Lets say you are developing software and you accidentally allocate in a tight loop. At some point the majority of the pages in ram not from the malloc loop end up in swap, so you try to do something to kill it, but Xorg and xterm both have their pages scattered across spinning drives, and your program is allocating memory faster than your machine saps in. If you are patient enough to not hard reset, an hour later, swap fills up and the OOM killer starts killing programs, and probably (but not definitely) kills the buggy malloc loop before it kills something more vital.
You are annoyed, so disable swap and do the same thing: the OOM killer kills probably just your web browser (which is virtually indistinguishable to a malloc loop with Web 2.0), and the malloc loop. The system recovers in minutes. You now swear off ever running with swap again.
Today: If you have a spinning disk 1GB or swap on it is fine, but zram is another good alternative. If you have slow and/or fragile SSD storage (e.g. sdcard or eMMC), zram is a good option. If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself.
> If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself
Not sure about Linux, but under macOS I had a bug that was leaking uncompressed 12 MP photos in a loop. I was browsing the web while waiting for my process to run and I got up to something like 150 GB of swap used (macOS uses dynamically growing swap files) before I even noticed that there was something going on.
> Note that managing resources is much harder when swap is in the picture, because you can get the resident memory of a process, but not the swap usage of a process.
Why can you not get swap usage? It’s in smaps, right?
Oh smaps is newer than my knowledge. I did since more research and your still can't limit over RSS+swap with ulimit though. There's a chance you could with cgroups though as there's lots of features there.
> I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself.
We are running a distributed in-memory olap database. All nodes have 500gb ram used basically at capacity. One node has 1.5T of nvme swap (initially just for testing but it works well) so it hosts a shard and the manager/reduce node. We've tested forcing the nodes onto swap and saw a x10 performance decrease, but our implementation is pretty shit when it comes to memory access patterns so it's not too bad. The collocated node is slightly slower in terms of performance but doesn't impact total response by a meaningful margin.
> It used to be that the recommendation for swap was 2x ram.
It always boiled down to never allocate more swap than you're willing to wait, and memory capacity multiplied over the years far faster than real world drive thruput.
So I was willing to slow everything down to use 8 megs of swap in 1994 in order to have enough virtual memory to compile a kernel, and in 2021 I'm still willing to slow everything down to use 8 megs of swap, in fact drive thruput has maybe quintupled since then, maybe, so I'd be willing to slow my system down into using 32 megs of swap in 2021.
The problem is 8 megs of swap in 1994 dramatically increased my horizons and I could use it to compile the kernel and stuff like that. In 2021, 32 megs of swap just makes my web browser OOM kill about 100 milliseconds later so why bother?
As I understand it, swap is used even when the system isn’t under direct memory pressure.
If the kernel sees that some memory is barely, if ever used, it will swap it out preemptively. This gives you more memory available for both programs and temporary caches.
As such it’s still advantageous to add much more swap than this, since it effectively gives you extra free space in RAM at essentially no cost.
If you want to understand why a oomd daemon is necessary and oom alone is not enough, you might be interested in this talk from FB "Linux memory management at scale" https://youtube.com/watch?v=cSJFLBJusVY
That change log is HUGE and it’s the main reason we don’t use systemd. Creeping featurism isn’t good in your OS when you need things to be consistent and work the same every time. Systemd has a lot of state.
Our organization. It doesn’t make sense to rely on software which grows to consume all available niches. Systemd is a perfect example of this kind of software. I don’t wish to have to ascertain what new features it’s reimplemented, sometimes in ways which are not POSIX compliant, every time there’s an update.
Do I guess correctly that your organization doesn't value clean services initialization, solid services dependencies management and reliable log handling?
Well, there must be room for such organizations too. It's a colorful world after all.
No, we get all of that with time-worn traditional methods which work well, are documented properly, and conform to standards. I’m not interested in beta testing Red Hat’s replacement for Linux.
I'm very happily using systemd since a bit less than a decade now. It's fantastic to have a common standardized way to manage your services and jobs that doesn't rely on a fragile set of scripts, and unified logs that can be easily queried and exported to a machine readable format.
I've never had an issue with the shell scripts that run my systems, and I consider it a strength that they can be easily modified. Why do people always seem to make 'no more fragile scripts' their first point in favor of systemd?
Because sh and bash are horror shows. Maintaining and creating new service init scripts is a real pain and difficult to do with consistency. Bash scripts can do anything, meaning they have to be audited one by one, every single time they are updated. Every single distribution has its own way to run services via scripts. And that can change between upgrades.
In the systemd world, I can audit a unit file via a simple `systemctl cat <name>`, and edit it directly via `systemctl edit <name>`. I can easily see services dependencies, I can easily change the user running the service, the workdir, etc. There is a standard interface to start, stop, restart, enable at startup. All of this from one system that is consistent and the same between Linux systems.
sh isn't a horror show, it's a POSIX standard and it's pretty easy to learn. Does systemd conform to any known standard? Of course not, they're making it up as they go. When's it going to be finished? Nobody knows. You simply can't rely on it.
The core primitives of systemd units haven't changed in a long while. You can always rely on it being able to start your service and stop it. What might not work depending on what version of systemd you ship is additional parameters (such as requiring a path to exist outside mountpoints or setting resource limits).
Well, thanks for telling me how you see it. I can't say I agree with almost anything you've said, but it's definitely informative.
My perspective is that on the one hand, systemd is very 'cerimonial' in how it goes about things, while shell-based systems have the potential to be extremely well done. My go-to example is the OpenBSD init script - I once tried to boot OBSD over the network, and mount it on a RAM filesytem, and was able to simply open that boot script and change the mount commands. Ever since, I've been impressed by the clarity and simplicity of that shell script (in its original form, not after I was done with it (: ).
> Because sh and bash are horror shows.
I disagree. The syntax is obtuse, but it is at this point very well understood, and the rest of the system runs on it. Better to use fewer languages throughout, than more. Are you referring to something other than the syntax?
> Maintaining and creating new service init scripts is a real pain and difficult to do with consistency.
I'm not sure, but it sounds like you're talking about systemd services. If you are, I agree; multiple files with layers of cerimony are hard to compare to an extra line of shell core in an rc.local. If you're talking about non-systemd, I've never encountered this, and certainly found the process far less painful than the systemd way.
> Bash scripts can do anything, meaning they have to be audited one by one, every single time they are updated.
Just diff the scripts...not that hard. Just have to check the changes. That's no harder than any other program you care about.
> Every single distribution has its own way to run services via scripts.
vive la différence! No point to the stagnation caused by homogeniety; I don't see why I should expect debian services to run the same as FreeBSD.
> And that can change between upgrades.
Systemd is notorious for changing things. One great example is when they tried to change how processes can persist after the user logs out. I don't think you have a leg to stand on here.
In the systemd world, I can audit a unit file via a simple `systemctl cat <name>`, and edit it directly via `systemctl edit <name>`.
I can too: `cat /etc/rc.local` and `vim /etc/rc.local`. Added benefit, I'm using the same tools and interfaces that I do for the rest of my system administration.
> I can easily see services dependencies, I can easily change the user running the service, the workdir, etc.
Service dependencies are the strong point of systemd, I will give you that. However, `su` can change the user running the command, and `cd` is this awesome little tool I found for changing the directory I'm in. No extra benefit there, just new commands for doing the same old things.
> There is a standard interface to start, stop, restart, enable at startup.
That standard interface is only useful because the underlying complexity has increased so much that `systemctl start` has become a necessary abstraction to cover it up. Your standard interface is medicine for an ailment I do not have.
> All of this from one system that is consistent and the same between Linux systems.
Why is that beneficial? It just means that you've put all the innovation in the hands of the systemd devs. I'm over on systems like void, where my startup times are faster, my base system load is a third, and I can get my hands dirty in the system internals without googling for strange systemd incantations.
> But really the best features IMHO is journald.
You're going to have to spell that out, because I really like being able to use a proliferation of text-oriented tools to manage my logs. I'm not a fan of binary, no matter the supposed speed improvements.
How will systemd-oomd communicate to the user?
I was using https://github.com/hakavlad/nohang
And it sent notification signaling to save your work before the application X will be killed. (I always wanted a way to click somewhere on the notification to prevent nohand from killing it when unwanted)
Also nohang use PSI data does oomd does it too?
129 comments
[ 2.6 ms ] story [ 190 ms ] threadOh, what fresh hell is this?
Tying them to pid 1, and forcing them to be used all the time for some hacked up thing where /usr is different depending on which process is looking at it? Not so much.
It’s no more privilege than pid 1 had before because of .mount units.
You know this doesn't do anything unless explicitly configured in the system and/or unit configuration?
Honestly, it seems great and not very tricky for a system which already manages namespaces with private mounts.
Ghosting: by bind-mounting over the system fs -you might introduce a bug/condition that doesn't show up in a recovery boot scenario, or forensic analyzis of the filesystem/disk - you could introduce "ghost" files/artefacts.
Yes, sure. That doesn't stop hundreds of vendors with custom install scripts who will use /usr and /opt which is the current state of things. Systemd actually provides a workaround here which stops that package from changing those hierarchies globally, which... seems like something you'd want?
> you might introduce a bug/condition that doesn't show up in a recovery boot scenario
Why would you try to run anything but recovery tools in a recovery boot scenario?
> or forensic analyzis of the filesystem/disk
You can reproduce the same environment using the same overlay configuration - that setup is not hidden from you. It's not any more confusing in forensic than a docker service and we deal with them (including bind-mounts) just fine.
Mixing this with OS package management will require some consideration.
So they are using overlayfs, which means that "upper" directories have visibility precedence on file name conflict. Which means that some auditing facility should be required that alerts if an upper sys image masks a lower one, or if an OS package some time later installs a file to the "lower" directory, perhaps the original rootfs, but a sysfs package still takes precedence…
Just wait until ls has to have hooks into dbus to do its job.
Including busybox in the initrd or whatever it's called now is probably a good idea. Having systemd developers write a new interpretation of that, rehashing old bugs is not great.
which uses LLVM for building Linux and Systemd, replaces all core utils with non GNU versions. It use a non GNU libc and so on...
But that aside people often confuse/mixup the systemd service manager and the systemd project which bundles a wide variety of programs which can be used with systemd.
The reasons for this is because systemd itself confuses this. The change log is not split by sub-projects and the sub-projects have sometimes no clean boundary between them and the systemd service manager.
I think it would be nice if systemd changes in three ways:
1. Have more clean boundaries between the core service manager and it's sub-project.
2. Have a recurring periods where new features are blocked from being merged and development focused on cleaning up (e.g. fixing bugs). (I can recommend that to any projects, very few do so).
3. Get "outside" feedback for certain mainly UX aspects, some parts of systemd are out right counter intuitive unless you are not very familiar with it. A thinks which is very easy to happen to any project, but still could be improved on (and maybe already is improved on).
Oh and there where a very few security WTF are you doing moments, I hope there will be no more such moments in the future as systemd on itself it quite useful for Linux, especially desktop Linux.
The increase in the amount of user-level software that depends on systemd and Wayland is a forcing factor. For example, my understanding is that official distributions of GNOME depend on systemd, and many Linux desktops are being actively developed for Wayland (though I don't think any major desktop environment or toolkit has abandoned X11). I expect this trend to continue, given that systemd is part of the most popular Linux distributions, Wayland will be default in Ubuntu (https://www.phoronix.com/scan.php?page=news_item&px=Ubuntu-2...), and GNOME runs on Wayland (with KDE's port to Wayland being actively developed: see https://www.phoronix.com/scan.php?page=news_item&px=KDE-2021...).
So, yes, I do see a split occurring between "modern Linux" (with systemd and Wayland) and "traditional Linux," with the BSDs fitting in closer to the "traditional Linux" camp.
There will be Linux for desktops to run a web browser (aka poorly emulate a chromebook) and for docker and docker only.
If you want a non-containerized server or some kind of non-desktop embedded thingy, or anything that can be computed other than emulating a chromebook and doing containers, you'll have to run FreeBSD or some other OS.
The days of "just install Debian to do anything anywhere" are gone, which is sad, because there's really no technical reason or need for the destruction.
What are you missing from Debian these days that was there 5/10/20 years ago? What has been "destroyed"?
Oomd, oh yea, good idea, but WHY it has to be in systemd? The same applies to about 3/4 of things in there. Because of this i've started to call it LennartOS. And also I don't feel like linux is about freedom anymore but more like some sort of weird power display games.
I use several of these projects because I like the systemd design philosophy (ex the way the CLI interface and config files are) and trust the developers of their software. However, there are popular altenatives available for all of these components, should you choose to use them:
systemd-resolved: bind9, dnsmasq, powerdns
systemd-networkd: NetworkManager, ConnMan
systemd-boot: GRUB2, Refind, EFISTUB
systemd-oomd: earlyoom, nohang, oomd (the facebook one, not systemd)
Like, hell, what is systemd-boot and when and why it replaced grub2?
I don't call this evolution. This is bad form of centralization and seizing power.
You really don't. You have the choice to go with a distro which doesn't use them, or to mix&match yourself.
I've been really happy with systemd-boot; in my limited (used it on probably a dozen machines across several distros at this point) experience, it's better at autodetecting nonstandard images/locations than grub, and is vastly friendlier to weird partitioning schemes. It also has a much more minimal config surface than grub, which I like (since, across distros, I no longer have to learn a distro-specific set of nested/included grub config files and conventions, nor do I have to worry about what will be auto-regenerated when a distro does a kernel update versus what configs I need to edit by hand). However, that limited config surface may be a drawback to people who need to customize very esoteric boot cases.
I also aesthetically prefer systemd-boot's minimalism and focus on newer (UEFI) boot strategies compared to grub; the latter is huge in terms of functionality. A lot of that size is needed for rare cases and older boot configurations (and if you fall into either group, grub is what you should use), but it complicates the "happy path" on post-2010 computers.
https://en.wikipedia.org/wiki/Gummiboot_(software)
It's a service manager for your Linux OS which manage all services down to initial startup related services.
Just viewing it as a init system is misleading.
As a side note I think a service manager directly above the kernel is a must have for any modern server or desktop operating system ;=)
Systemd is the name of the init system. Systemd is also the name of the project developing multiple systemd-* services. Maybe they could help the situation a bit by renaming the service itself systemd-init, but we've had enough time for people to learn to deal with a tiny bit of ambiguity...
«It (systemd) provides a system and service manager» (from https://www.freedesktop.org/wiki/Software/systemd/).
So it manages services and manages the system, as a whole.
> Every time I see all the stuff packed into it I feel extreme urge to switch to freebsd again.
Meh, a lot of people bring this up, but they never switch.
Lennax
Try understanding a bit more than just the name's prefix before getting so bent out of shape.
However, you should probably switch to FreeBSD because you think it’s great too, not because you hate systemd.
https://papers.freebsd.org/2018/bsdcan/rice-the_tragedy_of_s...
Joke's on you.
Systemd is an umbrella under which many key userspace tools are being developed. The actual init system itself is a lot bigger than the old SysV Init system was, but it is separate than all these other things that also say Systemd on them.
The best analogy is GNU. You don't complain that GNU contains a compiler, and also a text editor, and also C System Library do you? GNU is a project that develops a load of components that play well with each other. Systemd is the same.
[0] https://lwn.net/Articles/104185/
[1] https://lkml.org/lkml/2019/8/4/15
It's a heck of a problem to deal with, and a OOM killer will go part of the way to fixing it, but I still have two complaints about the thrash problem.
1. In similar situations on other operating systems, I don't usually see the GUI freeze to an unusable state. In Linux hitting a thrash situation will freeze the computer to the point where Ctrl-Alt-F2 will not switch the vterm even when waiting several minutes, and it has to be rebooted. I suppose it's possible that I just haven't used other OSes enough in a long time and they have this too, but this seems like a solvable problem. I assume it's not a CPU issue, since the scheduler should handle that just fine. So why not have a mechanism by which the user's desktop environment and init / login system can reserve itself enough memory to always be responsive under any memory condition?
2. Browsers. The state of browser memory management is atrocious. Just about every time I've run out of memory (other than compiling some complex software with -O3) it's been because my browser is hogging a huge chunk of it (even if the OOM killer blames something else). Now, I understand that in theory unused memory is wasted memory. So a browser using a ton of memory can be a good thing. But this is only true when memory the browser is holding but not using can be reclaimed to be used by another program, and this seems to basically never happen. If I close and reopen the browser (with the tabs automatically restored) the memory usage drops by 80% or more; the issue is that the browser simply never suspends tabs I haven't used in days or weeks. Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
I suspect solving these two issues would largely fix thrashing for most desktop users, without the need to ever kill anything, which is obviously undesirable.
Feel free to tell me if one or both of these is incorrect or impossible. These are just my observations, I'm not a kernel dev or anything close to one.
This! Always wanted a /proc/$PID flag to flag process' pages never to go to swap and reserve a number of cache pages specifically for the process. Something similar to mlock.
Because it's not the memory shortage that's nailing the UI. It's disk contention. When pathological swap activity causes disk thrashing, every process that needs to use the disk is now waiting in a very long queue with all the swap page in/out requests and every other process that is now backed up.
If you have a metrics agent that emits a handful of metrics on a per second granularity, you can see a bunch of things happen:
* Memory rises
* Some pages get swapped out
* Even if the system has no other processes writing to disk, you'll see the swap-out correlating with disk writes
* Memory rises some more
* You see more pages get swapped out
* (...and see more disk writes)
* Memory rises some more
* You see pages get swapped out and in, because now we're shuffling out pages that are imminently required by another running process.
* Disk queue lengths rise
* One by one, all processes are waiting (driving up the CPU "wa" number, and causing load averages to shoot up)
This is why, if you're oncall and you're trying to remotely troubleshoot "high load", you likely can't log in because the system needs to access /etc/passwd, /etc/nsswitch.conf, /etc/ldap/ldap.conf, /bin/bash... Your login gets handed a ticket with "∞" written on it, and is told to stand in the line for the disk.
But aren't these files already in the disk cache? Possibly. But another thing that happens when the system runs out of memory is that disk caches get purged. So you have to go back to the disk again. Also, when disk caches get dropped, your disk read performance is dogshit.
So what can you do? Try zswap? Use a different disk devices for swap? Try to pin important processes so the can't be swapped out?
NOTE: I might not be right about this stuff - I've never worked on the Linux kernel, and I rarely even compile it these days. But I've worked as SRE / *Ops for years and saw this problem a lot in badly configured systems. Same patterns all the time.
I suppose the solution is to figure out how to reduce or eliminate the number of disk reads after the DE initially loads, or cache everything needed in memory with a "do not evict" flag on it. Especially for the TTY, that's basically a crucial system function and keeping everything needed for it always available in memory wouldn't take much space relative to a typical desktop system.
I think at least some of the problem is the program code pages getting evicted to disk, so they have to be read back to memory too, which goes to the back of the disk queue. When I'm experiencing thrashing, even basic stuff like alt-tab to switch windows doesn't work. And I assume the window manager doesn't need to do disk reads to switch windows. Even the damn X cursor freezes!
I was able to get glitch-free audio by tweaking software and soft-irq realtime and non-realtime priorities, when the default Ububtu (with low-latency kernel) was nearly unusable for audio.
Which problem? Yes, there's always disk contention. But if you don't have swap enabled then you won't have pathological swap activity dominating the disk.
There are several bad things that happen when you run out of memory. One of them is pathological swapping. Another is loss of disk caches. Another is (eventual) oom. The one that has the most detrimental effect, and typically results in the system having to be reset, is swap activity saturating disk io. Even when you've got fast SSDs, because they're still orders of magnitude slower than memory.
Thrashing and becoming unusable for a long time.
You can recover system from this state by manually triggering OOM killer using Magic SysRq keys. (Alt+SysRq+F, provided you have SysRq enabled in /proc/sys/kernel/sysrq)
> Modern browsers are not good "desktop citizens", if you will. They hog memory to the point where opening anything new will create a pressure stall.
I went the ballistic route: most of my browsers run inside local Docker containers, on which I put CPU and RAM quota.
It's my machine: I decide how "citizens" behave : )
You can put resource limits on basically anything (see systemd.resource-control(5)).
https://man.openbsd.org/login.conf.5
* I run separate cgroups for admin SSH with (1G min, 4G max ) - worst case scenario I can ssh into the desktop and kill offending processes.
* Separate cgroup for two browsers ( 8G each max ).
* Separate cgroup per qemu VM ( VM max ram + 512M each max ).
After implementing that I went from periodic lock ups to a quick browser crash and a restart maybe once a week.
Perhaps this falls outside the category of "any memory condition"... :)
Though, it was responsive. :D
Needless to say, I think 1 is harder than expected. Fedora has made some recent changes on the OOM killer front though, perhaps those are closer to what you're interested in.
The "Auto Tab Discard" extension[1] for Firefox unloads tab contents, keeping only the title and favicon. Tabs can be put in an allowlist to prevent discarding. I don't know if Chrome or Safari have something similar.
[1] https://addons.mozilla.org/en-US/firefox/addon/auto-tab-disc...
In Linux I was never been able to generate a similar behaviour. No matter how aggressive I set the swappiness, the system didn't began to swap until it was practically locked up with the mouse pointer lagging.
Windows reserves some memory for its functions, for example the UI (maybe the fact that the GUI is in the kernel helps), and thus doesn't have these kind of problem, especially with an SSD (and I know that page file on SSDs is not ideal... but had a disk for 5 years swapping on it and never had a problem).
The mouse cursor did move smoothly throughout, though, as I recall.
It looks like systemd-oomd is related to (based on? from the same people as?) Facebook's oomd https://github.com/facebookincubator/oomd , whose documentation gives a bunch of reasons as to why you would prefer a userspace oomd that takes in PSI data and can be configured to proactively kill misbehaving processes instead of just letting the kernel OOM killer handle it. The major reason is time to recovery: a misbehaving process can cause a system to be so far under pressure that the kernel OOM killer will take a long time to flush things out, but a userspace component can respond in advance with more configurable rules (and more flexibility, since the kernel doesn't believe you're at capacity yet).
> It looks like systemd-oomd is related to (based on? from the same people as?) Facebook's oomd
Yep, I authored the original standalone oomd (along with many others' help) and Anita has been working on systemd-oomd. There's plans to converge and simplify everything that's been learned from oomd into systemd-oomd so the community can benefit from all our experiments.
If you filter out the RFEs (Requests For Enhancement) you find 617 issues. If you filter for only issues labeled as a bug you get 97 results.
In my experience, unless you can get Red Hat to care about patching a specific bug, you can rely on opened bugs to stay open.
> don't always label bugs that way (example: systemd/#5616)
Nobody figured out if that issue is in logind or X yet. They did not close it so it's not like they say it can't possibly be on systemd side. (typical Lennart comments aside)
I started reading about systemd-oomd, then followed the link to its manpage (at https://www.man7.org/linux/man-pages/man8/systemd-oomd.8.htm...) then followed a link from there to this page: "In defence of swap: common misconceptions" at https://chrisdown.name/2018/01/02/in-defence-of-swap.html).
The first line of the tl;dr is: "Having swap is a reasonably important part of a well functioning system. Without it, sane memory management becomes harder to achieve."
Now, afaik linux as a kernel is pretty much designed to have swap memory and really not meant to be ran without it. Except disabling swap is required to run kubernetes, so much so that the kubelet will plain refuse to start if it detects swap memory enabled.
Why?
The historic reason Kubernetes didn't let you run with swap is that it's complicated from a resource management perspective - it's not clear whether each pod should be requesting some amount of swap the same way it requests some amount of RAM, and also the container runtimes didn't really have support for that (largely because you need cgroupv2 to have a hope of that sort of per-cgroup swap resource control working at all). But it's not an insurmountable problem, and it sounds from that ticket like they recognize that letting systems have swap enabled for the sake of a userspace oomd is an important use case.
Swap comes with a lot of caveats -- when a performance-critical process begins swapping it can on occasion be more useful to just let it OOM instead.
My personal desktop has 64GB of memory, with no swap enabled. Why would I want swap it I don't need more than that?
In my experience, even with ample swap, when Linux gets memory starved it goes haywire anyways.
I'm an ex linux sysadmin, and haven't run swap on my personal systems in years.
Note that managing resources is much harder when swap is in the picture, because you can get the resident memory of a process, but not the swap usage of a process.
It used to be that the recommendation for swap was 2x ram. This recommendation was from the days were single-digit mega bytes of ram was common, and also from when linux core-dumps were both expected, and stored to swap.
Fast forward to 2010ish. You might have a 64GB or 128GB SSD, but you probably also have a spinning drive. you have 8 or 16GB of ram, so your 16-32GB swap would be 1/4 the SSD, plus you heard that putting swap on an SSD wears it out faster, so you put a 16-32GB swap file on the spinning drive.
Lets say you are developing software and you accidentally allocate in a tight loop. At some point the majority of the pages in ram not from the malloc loop end up in swap, so you try to do something to kill it, but Xorg and xterm both have their pages scattered across spinning drives, and your program is allocating memory faster than your machine saps in. If you are patient enough to not hard reset, an hour later, swap fills up and the OOM killer starts killing programs, and probably (but not definitely) kills the buggy malloc loop before it kills something more vital.
You are annoyed, so disable swap and do the same thing: the OOM killer kills probably just your web browser (which is virtually indistinguishable to a malloc loop with Web 2.0), and the malloc loop. The system recovers in minutes. You now swear off ever running with swap again.
Today: If you have a spinning disk 1GB or swap on it is fine, but zram is another good alternative. If you have slow and/or fragile SSD storage (e.g. sdcard or eMMC), zram is a good option. If you have fast nvme, I hear that you can go fairly crazy with swap sizes and have a system that works well under memory pressure, but I have not tried it myself.
Not sure about Linux, but under macOS I had a bug that was leaking uncompressed 12 MP photos in a loop. I was browsing the web while waiting for my process to run and I got up to something like 150 GB of swap used (macOS uses dynamically growing swap files) before I even noticed that there was something going on.
Why can you not get swap usage? It’s in smaps, right?
We are running a distributed in-memory olap database. All nodes have 500gb ram used basically at capacity. One node has 1.5T of nvme swap (initially just for testing but it works well) so it hosts a shard and the manager/reduce node. We've tested forcing the nodes onto swap and saw a x10 performance decrease, but our implementation is pretty shit when it comes to memory access patterns so it's not too bad. The collocated node is slightly slower in terms of performance but doesn't impact total response by a meaningful margin.
It always boiled down to never allocate more swap than you're willing to wait, and memory capacity multiplied over the years far faster than real world drive thruput.
So I was willing to slow everything down to use 8 megs of swap in 1994 in order to have enough virtual memory to compile a kernel, and in 2021 I'm still willing to slow everything down to use 8 megs of swap, in fact drive thruput has maybe quintupled since then, maybe, so I'd be willing to slow my system down into using 32 megs of swap in 2021.
The problem is 8 megs of swap in 1994 dramatically increased my horizons and I could use it to compile the kernel and stuff like that. In 2021, 32 megs of swap just makes my web browser OOM kill about 100 milliseconds later so why bother?
If the kernel sees that some memory is barely, if ever used, it will swap it out preemptively. This gives you more memory available for both programs and temporary caches.
As such it’s still advantageous to add much more swap than this, since it effectively gives you extra free space in RAM at essentially no cost.
https://media.ccc.de/v/arch-conf-online-2020-6390-linux-memo...
Q&A Questions: https://gitlab.archlinux.org/archlinux/conf-files/-/blob/mas...
Well, there must be room for such organizations too. It's a colorful world after all.
In the systemd world, I can audit a unit file via a simple `systemctl cat <name>`, and edit it directly via `systemctl edit <name>`. I can easily see services dependencies, I can easily change the user running the service, the workdir, etc. There is a standard interface to start, stop, restart, enable at startup. All of this from one system that is consistent and the same between Linux systems.
But really the best features IMHO is journald.
My perspective is that on the one hand, systemd is very 'cerimonial' in how it goes about things, while shell-based systems have the potential to be extremely well done. My go-to example is the OpenBSD init script - I once tried to boot OBSD over the network, and mount it on a RAM filesytem, and was able to simply open that boot script and change the mount commands. Ever since, I've been impressed by the clarity and simplicity of that shell script (in its original form, not after I was done with it (: ).
> Because sh and bash are horror shows.
I disagree. The syntax is obtuse, but it is at this point very well understood, and the rest of the system runs on it. Better to use fewer languages throughout, than more. Are you referring to something other than the syntax?
> Maintaining and creating new service init scripts is a real pain and difficult to do with consistency.
I'm not sure, but it sounds like you're talking about systemd services. If you are, I agree; multiple files with layers of cerimony are hard to compare to an extra line of shell core in an rc.local. If you're talking about non-systemd, I've never encountered this, and certainly found the process far less painful than the systemd way.
> Bash scripts can do anything, meaning they have to be audited one by one, every single time they are updated.
Just diff the scripts...not that hard. Just have to check the changes. That's no harder than any other program you care about.
> Every single distribution has its own way to run services via scripts.
vive la différence! No point to the stagnation caused by homogeniety; I don't see why I should expect debian services to run the same as FreeBSD.
> And that can change between upgrades.
Systemd is notorious for changing things. One great example is when they tried to change how processes can persist after the user logs out. I don't think you have a leg to stand on here.
In the systemd world, I can audit a unit file via a simple `systemctl cat <name>`, and edit it directly via `systemctl edit <name>`.
I can too: `cat /etc/rc.local` and `vim /etc/rc.local`. Added benefit, I'm using the same tools and interfaces that I do for the rest of my system administration.
> I can easily see services dependencies, I can easily change the user running the service, the workdir, etc.
Service dependencies are the strong point of systemd, I will give you that. However, `su` can change the user running the command, and `cd` is this awesome little tool I found for changing the directory I'm in. No extra benefit there, just new commands for doing the same old things.
> There is a standard interface to start, stop, restart, enable at startup.
That standard interface is only useful because the underlying complexity has increased so much that `systemctl start` has become a necessary abstraction to cover it up. Your standard interface is medicine for an ailment I do not have.
> All of this from one system that is consistent and the same between Linux systems.
Why is that beneficial? It just means that you've put all the innovation in the hands of the systemd devs. I'm over on systems like void, where my startup times are faster, my base system load is a third, and I can get my hands dirty in the system internals without googling for strange systemd incantations.
> But really the best features IMHO is journald.
You're going to have to spell that out, because I really like being able to use a proliferation of text-oriented tools to manage my logs. I'm not a fan of binary, no matter the supposed speed improvements.
Finally the author of nohang is working on arguably superior solutions: https://github.com/hakavlad/prelockd https://github.com/hakavlad/memavaild https://github.com/hakavlad/le9-patch It's unfortunate that systemd has chosen to integrate an inferior solution