I disagree with the position in this post -- in my experience swap has universally been a contributor to system instability and performance issues. I run all my Linux servers without swap, and for workstations try to restrict its use to suspend/hibernate only.
1. Allowing a process's mapped pages to be flushed to disk will cause performance to become unpredictable. This applies to both anonymous pages written to swap, and file-backed pages. The first thing every server process should do is mlockall(MCL_FUTURE) so the kernel can't decide to swap out parts of your RPC handlers. More sophisticated implementations (e.g. databases) can selectively mlock() specific pages.
2. Using swap to mitigate memory overcommit isn't useful because the process that overcommitted its memory should just be killed instead. This is where cgroups are useful: you can load test to understand the allocation curve, then tell the kernel to limit your service's process to 64 GiB or whatever. If it tries to go wild and take up more than its share, it gets a SIGKILL instead of taking the entire machine out of service.
3. Swap will destroy SSDs. You thought the write load from logging is bad? Try putting a consumer-grade SSD into a machine with 512 GiB RAM and let the kernel swap to it -- that SSD will be dead in a year.
My reading of your comments above is that you are speaking about about cases where swap is being used as a replacement for memory. I take that from, among others, "Swap will destroy SSDs". Swap used only for paging unused data out is an infrequent occurrence, no way it's going to kill an SSD.
My reading of this article is that he is specifically advocating NOT using it as a replacement for having adequate RAM.
My experiences, around 5 years ago, of running ~100 VMs without swap, basically confirmed the assertions in the article. Mostly the systems would run ok, but when they got into memory pressure they would live-lock. I had hoped for just OOM killing, which happened sometimes, but also sometimes the machine was just locked.
I switched over to having some small amount of swap, maybe 0.5-1GB on these boxes and haven't had a livelock since.
Plus, it's better to get unused pages out of RAM, which can be written to swap.
The one thing that HAS killed an SSD, in this case an Intel (320 series sticks in my head) is ZFS slog+L2ARC. Swap hasn't been a problem in my experience, but I also am not using swap to account for not having enough RAM.
I don't recall exactly, but probably 1-4GB. Not particularly low memory, but when something would come along that needed more memory than the system had, say a load burst, I was quite surprised to find the systems without swap just stop responding. I was expecting OOM killer to kill the offending process, the load balancer to remove it from the load, monitoring to say what process had gotten killed, but instead, even after waiting half an hour, the system was just wedged.
That seems pretty low, especially if this was 1GB of RAM. I'd use swap on such systems. I don't think there are any reasons today to buy physical machines with that little RAM, but I can see how you might have such VMs,
That said, I agree with other commenters that said this sounds like a kernel bug. I am wondering what was going on to wedge the system that bad. Would putting executables on tmpfs (so they cannot be paged out) helped?
I think it's usually code page thrashing, since executables and shared libraries can always be swapped back out to disk, only to be swapped back in when the process gets the next quanta of time to execute. If people say instruction cache misses are costly, wait till you need to read the next instruction from a 5400 RPM HDD...
Thanks, understood. However file-backed pages such as executables and shared libs are not written to swap space as the pages are just dropped and read back in from the regular(non-swap) filesystem no? Not that causes any less disk I/O of course but they are not anonymous pages which is generally what is written to swap. When I think swap I think of "written to swap" but maybe you mean swap in the general sense of paged out?
Yes, I meant "swapped out" in the general sense of "paged out". On the other hand, if you have actual swap space, hot code pages will be much less likely to be paged out than colder anonymous pages.
> Try putting a consumer-grade SSD into a machine with 512 GiB RAM and let the kernel swap to it -- that SSD will be dead in a year.
Are you sure? I’ve heard the argument before that swap is one of the better cases for SSD longevity, because it’s few big writes instead of many tiny ones, and found that convincing. Also, on which workloads would a machine with that much RAM touch swap at all? My workstation has “only” 64 GiB, has swap configured (to enable suspend), even during heavy use with some databases running never touches it.
I know you're not the GP, but please give some specific examples of such workloads, I honestly want to know.
To be clear, yes, it's well-known that the kernel will page out rarely used stuff, but those are (almost by definition) pages that are basically never written to. The claim was that writes to swap occur all the time, to the point where they wear out the SSD, that seems to be something different.
At least I've never seen anything like that with any of my (workstation) workloads, like large C++ builds, TensorFlow experiments, or running lots of virtual machines. And that was only with 64 GiB RAM, not even the 512 GiB that the post I replied to mentioned.
The kernel write the pages into swap long before it decides they are rarely used. Otherwise rewriting them would be way too slow to be of any use. Any normal use will fill some swap space, and if the memory pages keep getting dirty (and you have spare disk IO), your computer will keep writing them to swap.
So I run servers used to process images, 99% of the time it is typical ~5 megapixel photos, but occasionally it will be ~150 megapixels. We use swap because that 99% of the time it will fit in 4GB of ram. The occasional ones take an understandably longer time to process.
I had good success using an Intel Optane drive for swap. Of course, that was because the task I was doing required hundreds of gigs of memory and my poor little machine only had 64; not exactly common in a business setting.
Performance was great. Normally when swapping the performance of the machine is terrible, as you say. But those Optanes are so fast that it just didn’t matter. I was doing several hundred thousand operations per second, and they were all taking <4µs. It was a wonderful upgrade.
> Swap will destroy SSDs. You thought the write load from logging is bad? Try putting a consumer-grade SSD into a machine with 512 GiB RAM and let the kernel swap to it -- that SSD will be dead in a year.
I have an OCZ Vertex LE in a 32-bit laptop that was my daily driver for somewhat-serious development for the better part of a decade and sees somewhat regular use as a bedside media consumption machine. It ran (and still runs) Gentoo Linux, so -while it was my daily driver- its drive saw _frequent_ writes due to weekly system-updating-build activity.
This machine has 4GiB of RAM installed, of which only 3.2 GiB is available due to weird BIOS limitations. It has _always_ had swap enabled, and has _often_ made heavy use of it.
This SSD has 93,560 power-on hours, and has written 48,384 GiB. As far as I (and SMART) can tell, it's as good as the day I flashed the v1.1 firmware on it.
I don't believe your claim is generally true. Other folks who have intentionally run drives _way_ past their advertised wear-out points have similar stories to mine.
Running a light workload on a consumer SSD is fine. That's what they're designed for. Your ~48 TiB of writes over a decade is well within its design parameters.
If you used it as swap in a server, you could expect something closer to its max write rate, which is on the order of 10-20 TiB per day[1]. Run that for a year and you're at 3+ PiB total writes.
[1] Contemporary reviews say the OCZ Vertex LE had a maximum write throughput of 250 MB/s. Times 3600 (per hour), times 24, converted to IEC notation is 20,116 GiB per day.
> Swap will destroy SSDs. You thought the write load from logging is bad? Try putting a consumer-grade SSD into a machine with 512 GiB RAM and let the kernel swap to it -- that SSD will be dead in a year.
You can run with a low swappiness value and the kernel will not proactively swap out process memory unless required. Even light use of swap is probably fine as it likely involves rare process pages that will never be written to after being swapped out so there's no threshing on the SSD, only a few large writes.
Ahh now I understand why people disable swap on Linux boxes.
I came from a SunOS/Solaris & HP-UX background (late 90s) where we routinely configured swap for the machines.
Later on I encountered Linux admins who refused to configure even a small amount of swap and they couldn't explain why. It seemed very odd to me. Nobody ever said the simple "swap sucks in the Linux kernel" to me.
Indeed. At a past job we had this million dollar cluster of HP machines (T520s), and one of the managers wanted to do some end of year spending, so they upgraded the memory in the cluster from 2GB to 4GB (back when that was an insane amount of memory, ~1995).
But, we had not storage to back that new memory. They purchased 2GB per machine for god knows what, but didn't add 2GB discs. So we literally couldn't use it (because of how storage was allocated on that cluster, we couldn't just re-carve it to get an extra 2GB swap, it had dedicated 2GB drives for the swap).
The HP-UX kernel required at least as much swap as RAM, due to the way it handled memory and swap. I used to know the details of those algorithms, but have freed those portions of my memory for other uses. It was something like: The algorithms HP-UX uses for virtual memory consider that memory based on swap, and the RAM is kind of like a cache layer in front of the swap. This makes the handling of allocation and swapping easier because you know where everything would go before you start trying to swap.
Disk are still much bigger than memory on the majority systems.
I can buy >16TB disks: I don't think I've deployed a machine with more than 1TB of RAM. (Maybe 2-4 (out of hundreds) in an HPC environment at previous employer had 1.5T?)
This video is one of the most amazing, useful, gem-after-gem explanations of Linux you could possibly ever encounter. There are so so few who have the in depth knowledge, across so many broad experiences, at such scale, to understand Linux's memory management.
Side topic #1: this video's advice on cgroups just makes me sad, because I love Kubernetes, but Kubernetes building it's own scheduler, in contest with not in tandem with the kernel's cgroup system, is super super super f-ing sad. Kubernetes running all it's shit in a single cgroup is some decrepit sad sad shit. I still love & user Kubernetes, but god damn, for what basically amounts to a cloud os, it's using the underlying host-OS extremely extremely poorly. This video was key in making me think & consider how much we've lost, how sad it is we don't take advantage of what the kernel is trying to do for us with trading off workloads. Leaving the kubernetes scheduler to try to fit all the workloads feels like we're letting a bunch of ignorant elementary schoolers run the show. It's embarrassingly superficial.
Side topic #2: it would be really interesting to try to go a bit deeper on the topic. I'd love a world where we can easily ask- what are these anonymous rarely used memory pages we get to swap out? That's the main advantage, under low & medium pressures: the system has some flexibility to move out things not really used, where-as it has to keep everything resident without swap. But answering: what are these things not being used: that probably requires bptrace foo to figure out what regions are getting swapped, plus deep skills to figure out what those addresses correspond to. I doubt Linux will just tell us, oh, I evicted this text block from the binary. (It likely doesn't know much about what blocks it's evicting.)
Keep in mind that positive memory pressure is something that if done by more than a single application at a time, will completely break your system, and if done only one application at a time will severely reduce your performance predictability (but can indeed increase throughput).
So, what he talks about on that video is a very specialized performance optimization that may work on some (certainly not all) systems that you have complete control of. It isn't general advice in any way.
If you have a build farm / CI machines, don't use swap. With swap, if a user schedules too many compiles at once, machine will slow to a halt and become kinda-dead, not quite tripping dead timer, but not making any progress either. Instead, set up the OOM priority on the users processes so they are killed first. If OOM hits, clang is killed, build process fails, and we can go on.
If you have a latency-critical server, don't use swap. With swap, some data will get swapped out, and you will have sudden latency trying to access it. Without swap, it may still swap out the code, but the code is usually much smaller than the data, plus it is all pretty hot.
If occasionally you have to process a very large datasets, and you are willing to wait a minute or twenty, you may enable swap. But be patient, and make sure your ssh has no timeout which can be hit.
If you have a small amount of RAM, enable swap. It would be slow, but at least you would be able to do something without crashing.
> If you have a build farm / CI machines, don't use swap. With swap, if a user schedules too many compiles at once, machine will slow to a halt and become kinda-dead, not quite tripping dead timer, but not making any progress either. Instead, set up the OOM priority on the users processes so they are killed first. If OOM hits, clang is killed, build process fails, and we can go on.
This doesn't really work that well. It's true that if you enable swap and have significant memory pressure for any extended period your machine will grind to a halt, but this will _also_ happen if you don't use swap and rely on the Linux OOM killer.
Indeed, despite the lack of swap, as part of trying to avoid OOM killing applications, Linux will grind the hell out of your disk - because it will drop executable pages out of RAM to free up space, then read them back in again on demand. As memory pressure increases, the period of time between dropping the page and reading it back in again becomes very short, and all your applications run super slowly.
An easy solution to this is a userspace OOM-kill daemon like https://facebookmicrosites.github.io/oomd/ . This works on pressure stall information, so it knows when your system is genuinely struggling to free up memory.
On the historical fleets I've worked on pre-OOMD/PSI, a reasonable solution was to enable swap (along with appropriate cgroups), but target only allowing brief periods of swapin/out. This gives you two advantages:
* allows you to ride out brief periods of memory overconsumption
* allows genuinely rarely accessed memory to be swapped out, giving you more working space compared to having no swap
Eh, I’ve never seen a machine actually use any notable amount of swap and not be functionally death spiraling.
I’m sure someone somewhere is able to use swap and not have the machine death spiral, but from desktop to servers? It’s never been me.
I always disable swap for this reason, and it’s always been the better choice. Not killing something off when you get to that point ASAP is a losing bargain.
Don't mistake every machine you have seen death spiraling using swap, with every machine using swap as death spiraling. Notably, how many machines did you not have to look at, because the swap was doing just fine?
That I’ve administered? None under any significant load!
I even finally disabled it on the lab raspberry pi’s eventually, and a SBC I use to rclone 20+ TB NVR archives due to performance problems it was causing.
It’s a pretty consistent signal actually - if I look at a machine and it’s using any swap, it’s probably gotten wonky in the recent past.
If you pushed something to swap, you didn’t have enough RAM to run everything at once. Or you have some serious memory leaks or the like.
If you can take the latency hit to load what was swapped out back in, and don’t care that it wasn’t ready when you did the batch process, then hey, that’s cool.
What I’ve had happen way too many times is something like the ‘colder’ data paths on a database server get pushed out under memory pressure, but the memory pressure doesn’t abate (and rarely will it push those pages back out of swap for no reason) before those cold paths get called again, leading to slowness, leading to bigger queues of work and more memory pressure, leading to doom loops of maxed out I/O, super high latency, and ‘it would have been better dead’.
These death spirals are particularly problematic because since they’re not ‘dead yet’ and may never be so dead they won’t, for instance, accept TCP connections, they defacto kill services in ways that are harder to detect and repair, and take way longer to do so, than if they’d just flat out died.
Certainly won’t happen every time, and if your machine never gets so loaded and always has time to recover before having to do something else, then hey maybe it never doom spirals.
I do a lot of ci/CD where we just have weird load and it would be a waste of money/resources to just shelf out the max memory.
Other example would be something like Prometheus: when it crashes and reads the wal, memory spikes.
Also it's probably a unsolved issue to tell applications how much memory they actually are allowed to consume. Java has some direct buffer and heap etc.
I have plenty of workloads were I prefer to get an alert warning and acting on that instead of handling broken builds etc.
> Eh, I’ve never seen a machine actually use any notable amount of swap and not be functionally death spiraling.
For my low-end notebook with solid-state storage I set the kernel's swappiness setting to 100 percent and this problem got magically fixed. It's rock-solid now.
FreeBSD isn't Linux, but I've had FreeBSD machines fill their swap and work just fine for months. I had one machine that had a ram issue and started up with a comically small amount of ram (maybe 4 mb instead of 256 mb... It was a while ago) and just ran a little slow, but it was lightly loaded.
I've also had plenty of machines that fill the swap and then processes either crash when malloc fails or the kernel kills some stuff (sometimes the wrong thing) or something things just hang. Measuring memory pressure is tricky, a small swap partition (I like 512 MB, but limit to 2x ram if you're running vintage/exotic hardware that's got less than 256MB) gives you some room to monitor and react to memory usage spikes without instantly falling over, but without thrashing for long.
You should monitor (or at least look at) both swap used % and also pages/second. If the pages/second is low, you're probably fine even with a high % use, you can take your time to figure out the issue; if pages/second is high, you better find it quick.
When I reenabled swap on my desktop (after running without swap for years assuming it would avoid the death spiral, only to find out it was almost always worse because there was no spiral: it just froze the whole system almost immediately), it would frequently hold about 25% of my RAM capacity with the system working perfectly fine (this is probably an indication of the amount of memory many desktop apps hold onto without actually using more than anything else, but it was useful). In my experience if you want a quick kill in low memory you need to run something like earlyoom to kill the offending process before the kernel desperately tries to keep things running by swapping out code pages and slowing the system to a crawl.
It's only one datapoint, but at this very moment a server at work is using a notable amount of swap, 1.5 GiB to be more precise, while functioning perfectly normally.
$ free -h
total used free shared buff/cache available
Mem: 3.9Gi 1.7Gi 573Mi 180Mi 1.6Gi 1.7Gi
Swap: 4.0Gi 1.5Gi 2.5Gi
I wish you luck! Only time that’s happened before was memory leaks for me, and it didn’t go very long before death spiraling. But if you’re comfortable with it, enjoy.
I think the key here is what you mean by using swap. Having a lot of data swapped out is not bad in and of itself - if the machine genuinely isn't using those pages much, then now you have more space available for everything else.
What's bad is a high frequency of moving pages in and out of swap. This is something that can cause your machine to be functionally unavailable. But it is important to note that you can easily trigger somewhat-similar behaviour even with swap disabled, per my previous comment. I've seen machines without swap go functionally unavailable for > 10 minutes when they get low on RAM - with the primary issue being that they were grinding on disk reloading dropped executable pages.
I agree that in low memory situations killing off something ASAP is often the best approach, my main point here is that relying on the Linux OOM killer is not a good way to kill something off ASAP. It kills things off as a last resort after trashing your machine's performance - userspace OOM killers in concert with swap typically give a much better availability profile.
In a situation where a bunch of memory is being used by something that is literally not needed and won’t be needed in a hurry, then it’s not a big deal.
In my experience though, it’s just a landmine waiting to explode, and someone will touch it and bam useless and often difficult to fix machine, usually at the most inconvenient time. But I also don’t keep things running that aren’t necessary.
If someone puts swap on something with sufficiently high performance, then obviously this is less of a concern too. Have a handful of extra NVMe or fast SSD lying around? Then ok.
I tend to be using those already though for other things (and sometimes maxing those out, and if I am, almost always when I have max memory pressure), so meh.
I’ve had better experience having it fail early and often so I can fix the underlying issue.
I think "Linux drops will drop executable pages without swap" is a symptom of machines with small amount of memory, say 4G or less. So it is pretty outdated for regular servers, and probably only relevant when you are saving money by buying tiny VMS.
Those build servers had at least 64GB of RAM, while executables were less than 1GB (our entire SDK install was ~2.5GB and it had much more stuff than just clang). So a machine would need to finely balance on memory pressure: high enough to cause clang to be evicted, but low enough to avoid OOM killer wraith.
I don't think this is very likely in machines with decent amount of memory.
Fair enough - I've seen it more commonly in smaller machines, but they're also more common in the fleets I've observed (and the ones that are more likely to run close to the edge memory-wise). I have also seen it in systems up to 32GB RAM, so it's by no means a non-issue in systems that are at least somewhat larger. The general observation that oomd/earlyoom + swap is a better solution than no swap still generally applies.
Because once swap activates, build now takes hours instead of tens of minutes. So it would timeout anyway, but only after wasting lots of resources. And even if you increase the timeout a lot instead, your machine how has a bunch of things swapped out, so now your tests timeout, which is even worse.
Yes, killing that part of the build did destroy the work of hours. It was still better to disable the swap than try to "ride it out".
I don't understand your paragraph on Kubernetes. Say you have 1000 machines with 4 TiB each, and you want to schedule processes that will cumulatively allocate 4 EiB. How would you do this without letting Kubernetes handle the scheduling?
Once Kubernetes decides which processes should run on each machine, it allocates separate cgroups to each process group, which it uses for enforcing CPU and RAM resource limits. You can see this by examining the cgroupfs control filesystem. How would you propose improving this?
Although the article claims "Disabling swap doesn't prevent pathological behaviour at near-OOM," I've definitely found this to be the case. My desktop Linux machine has no swap, and when I've run into memory pressure the OOM killer just swiftly terminated the program that was using all my RAM (I later bought more).
When I've used machines with swap, my experience has been the OOM killer was much less aggressive. As a result you're left with an unresponsive system blocked on disk I/O.
That's interesting, I've have just the opposite experience.
I used to run with no swap; when compiling with many parallel processes, my system would sometimes live-lock. The OOM killer never triggered, and as far as I could tell, the issue actually was I/O saturation caused by thrashing on the few pages that the kernel _could_ evict. After enabling swap, I've never encountered this issue again. The machine remains interactive, if slow, under high load.
Anecdotally, SIGSTOP-ing some processes when memory pressure is high, even without swap, seems to have some benefit. I'd resume those batch processes later.
And I had the exact opposite experience as you just last week on multiple machines with swap enabled: compiling and going over the available memory made them entirely inoperable and required hard resets. Had to use another computer to be able to browse internet at the same time, until earlyoom can be installed there.
Sounds like the cause is orthogonal to the presence of swap. What kind of persistent storage do you have on those computers? For me it's been an M.2 in both cases.
I think the key is near OOM. In some cases (e.g. runtimes with configurable heap sizes that trigger garbage collection when the max size is reached) it's easy to setup a system that operates near OOM
I have no swap on my laptop, and due to an accidental bug in a program I was developing my machine ran out of memory a few months ago (twice... my first attempt didn't fix it, oops!) In both cases the system locked up for well over a minute before the process got killed by OOM. It was the "classic Linux experience" regarding OOM.
Back in the day I used a lot of FreeBSD on fairly cheap/constraint machines (even for the time) and somewhat frequently-ish ran out of memory, and it was a much "better" experience. I mean, OOM killing stuff is never a "good" experience, but it didn't lock for over a minute.
With swap: We can choose to swap out rarely-used anonymous memory that may only be used during a small part of the process lifecycle, allowing us to use this memory to improve cache hit rate, or do other optimisations."
How do you improve cache hit rate when there's no memory contention? And what other optimizations?
"Without swap: We cannot swap out rarely-used anonymous memory, as it's locked in memory. While this may not immediately present as a problem, on some workloads this may represent a non-trivial drop in performance due to stale, anonymous pages taking space away from more important use."
When there's no memory contention, what's the "more important use"?
These are honest questions - I'm not an expert on the topic.
I think the point is that, the more free RAM you have, the more data the OS can speculatively read from disk when there's otherwise unutilized bandwidth. If a program happens to need the data, then it's already in RAM and the access latency is almost zero, vs. the non-zero latency of waiting for a hard disk platter to spin around.
The claim being made is that there are certain situations where processes allocate a lot of physical memory but then very rarely accesses it. If it gets paged out to swap, then there's a latency hit on the next access. If the OS can avoid more latency hits total using the same physical RAM for caching, then the computer could do more overall calculations per second.
Yep! The idea of "free memory" largely doesn't exist in modern operating systems. We haven't gotten to the point of power gating DRAMs (have we?), so we're always paying the cost of the DRAM all the time. As such, not using it means you're just throwing energy away. Keeping lots of purgeable data in these unallocated space like (as you said) disk cache not only speeds up access time but also saves a ton of power since a hit to DRAM is leagues lower than having to access external storage. While this is a non-trivial engineering problem, it's almost always a good trade off since it boost performance and saves power without any extra cost other than the effort of speculative loads (which, when linear, is much less expensive than a random one off load!) and managing it
By cache, I assume the writer means disk cache. Disks are slow! (Yes, even your PCIe 4.0 x4 NVMe SSD is.) So the kernel (usually) uses RAM to cache reads and writes. (This can be bypassed, but let's ignore that.) Under heavy disk IO, using fast RAM for this cache is more beneficial than reserving it on behalf of a program that may use it very rarely, or maybe even not at all (in the case of memory leaks)!
If on a hypothetical 8GiB RAM system, programs require 6 but 2 of those are rarely used, there is no memory contention but the extra space for disk cache is still a helpful boost.
I do think program memory is more valuable on most desktop systems, since commonly memory access is expected to be fast while disk access is expected to be slow. This means that interactive software tends to mitigate for slow IO, while it can't do much against being swapped out.
On servers however I'd probably rather let the kernel do its job (at least if I'm focusing on throughput), unexpected slowness may worsen a worst case but the average case is helped enough.
Luckily, the Linux kernel, for all of its memory management flaws (don't get me started on overcommit), allows configuring this using the vm.swappiness parameter that tells it what to prioritize.
Swap has always been so slow for me that I just disable it on all of my machines. I would rather the OOM reaper just SIGKILL whatever is using all my RAM than deal with slowness (which often persists after the OOM situation is gone).
Yes, but that's not quite how it works. As mentioned in this article, without swap, the system can live-lock before the OOM killer can take care of things. This has been my experience as well.
I had hoped that getting rid of swap would prevent thrashing, but instead the system would live-lock.
I feel like the kernel live locking in low memory environments should be treated as a bug and not something we try to solve using swap. Like...if the OOM killer can't free memory under low memory constraints, something has gone seriously wrong
I've tried twice to update Lakka (a custom Linux image that runs RetroArch and emulators) on a RPi 3B with 1 gigabyte of memory (and I think no swap). Updating Lakka from the Pi itself invokes a script which downloads a file from the network to SD card storage. This should not take up unbounded amounts of memory, but seems to trigger OOM livelock or something anyway (I can't tell what it is, since the GUI and SSH session hang, and Lakka doesn't enable Alt+SysRq or TTYs for some godforsaken reason). Enabling SSH and running `watch -n1 free -h` shows a concerningly low free memory amount, and IIRC the available memory number crept downwards before the GUI and SSH session both hung simultaneously.
Terminating the GUI and performing an offline update from one SSH session while running `watch -n1 sync` avoids the hang. I haven't tried sync while leaving the GUI running.
The problem is the OOM killer only runs if no pages can be freed. But, on any system, there is some amount of memory that can be swapped out: code pages can be swapped out to their executable/library files on disk. On a system only running a few processes, this is unlikely to matter.
But if you have a lot of processes, you may end up in a situation where you can free enough pages by swapping them all out to disk. Now you have no OOMKill, but the next time a process gets to run, you will stall until it is swapped in from disk. Then, the next process will be scheduled, and it too will need to be swapped in, causing another stall, and so on. The machine will probably end up executing, at best, a few instructions per millisecond for hours...
I can't for the life of me find where the article talks about live-lock and the OOM killer. I searched for "lock" and "OOM" and none of them related to that.
He doesn't use the term live-lock, but see "6" at the top of the article where he talks about "pathological behavior at near-OOM". Also "3" "Disabling swap does not prevent disk I/O from becoming a problem". Those, in my experience, have been the situations that I was in that I called live-lock.
This is an area where everyone's anecdotal evidence is different, because people are running all sorts of different workloads on all sorts of different RAM/swap/disk configurations.
Naturally, some people will have had consistently positive experiences with swap, while others will have come out with consistently negative impressions. 32GB RAM + 512MB swap backed by a fast SSD is very different from 2GB RAM + 512MB swap backed by spinning rust. JVM is different from MySQL, which again uses memory differently than PostgreSQL.
And yet we are so eager to hurl sweeping generalizations at each other, holding religiously onto our own subjective cargo cults. :(
For Linux on desktop, my experience with swap used to be extremely bad before systemd-oomd. I got those system lockups easily in various workloads that for the longest time I went without swap, and a crashing program was a better experience.
On the server, sprinkling in a bit of swap was standard practice in some orgs. As under load I could still log into a slow server and do something about it.
My current problem with swap in Linux is that they make bad default choices. While a swapfile isn't optimal, allocating a slice of the disk for swap complicates my life when I do ram upgrades. Hibernate being tied to configured swap on a slice of disk (ugh), I just didn't even bother with it since upgrading ram.
Rails, Django, Elixir/Phoenix web development, sometimes with docker. No swap on my 32 GB laptop both when I used it with a 750 GB HDD in its early years and with two 1 TB SSDs now. I'm usually using about 20 GB RAM because I'm leaving every single project open in its own virtual desktop, to get there quickly if a customer needs something done.
No problem whatsoever that I remember. I'm never close to filling up RAM, that could be the reason. I'd buy extra RAM instead of enabling swap. The price would be extra battery depletion during the night.
kill java, and keep on killing java with fire. Then move on to the printer daemon, the 200 vtys you forked on console I never use, the mouse daemon, and maybe in the end, sendmail.
4GB swap on a 4GB RAM VPS allows me to do an occasional yarn build, which would otherwise crash the server and who doesn’t need 8GB RAM the rest of the time.
I find swap and memory usage in general confusing. On my Windows machine, Chrome is listed as using about 1GB of RAM and I have maybe 50 tabs open. Seems like a lot. I think it’s possible that Chrome creates 50 processes and that the memory of DLLs loaded get over counted 50x. So 20MB of actual RAM DLL usage could be listed as 1GB. However maybe the DLLs are say 10MB and then Chrome would be using 500MB of data of 500MB of code. This might help with switching tabs and instantly seeing a page. For me though none of the tabs data need caching. I would prefer they just reload when needed. So I have this confusion. How much RAM is Chrome using? How much is truly needed vs hopefully fortuitous caching. Can some data be marked more important or is it all just LRU? Should Chrome free all memory on non-visible tabs?
How do I know how much RAM to buy when I don’t know what anything really needs.
Process Manager shows 8GB used of 16GB, but I currently have very little running. Chrome, Outlook, Ring Central. How would one know if this could run with 4GB or 8GB? Do you just have to try it and see if it seems sluggish.
How does one distinguish memory used by code vs data?
To answer your Chrome question, it’s open source and the tab discarding code is pretty straightforward.
GetSortedLifecycleUnits sorts using GetSortKey, which for tabs is their last used time. So it’s just LRU. But some comments hint that maybe something smarter will happen eventually.
As for your overall question, there are no easy answers. Deciding if memory is being used well, or could be repurposed to do something more useful is a difficult question to answer.
Edit: also, it’s really hard to reason about how much memory you need unless it’s very well defined (you know the size of data, exactly what processes to run), once you get to desktop operating systems, it’s more run and find out than reasoning from first principals.
Part of the problem, and perhaps the main justification for swapping today, are all the programs which run infrequently but don't use the OS features to get themselves started on demand or periodically. On Linux, do "ps axl"; on Windows, use Task Manager. Look at all that stuff.
There has to be some way to evict those memory hogs.
Articles like this further cement my belief that Linux's low memory handling is deeply flawed. (macOS's too, btw.)
My experience with Linux machines that have HDDs and swap is that once they start swapping, they slow down not only until memory is freed (ctrl-C gets through, OOM killer runs, whatever) but actually until reboot. My theory is that processes' VM space is now a minefield. Both anonymous memory and file-backed mmaps now have many gaps which get paged in on demand with inadequate readahead. With each seek taking ~10 ms, it doesn't take too much of that before you should give up and just reboot.
Once memory is available, I'd like the system to proactively page back in both swap and file-backed pages to prevent this. If it does need to happen reactively, it should be in big blocks (each 10 ms of seeking should be follow by at least 10 ms of reading => ~1 MiB, not ~4 KiB or whatever it's doing now). By default.
I'm not convinced the decision to use swap is ever a good one. E.g., from the article:
> With swap: We can choose to swap out rarely-used anonymous memory that may only be used during a small part of the process lifecycle, allowing us to use this memory to improve cache hit rate, or do other optimisations.
This is short-sighted. Programs can be written to tolerate their filesystem I/O being slow. They often take care to do I/O on a particular thread which is not handling network or GUI operations. They can't realistically be written to tolerate arbitrary memory accesses (including program text, stack, and heap) being that slow. It's a stupid idea to potentially slow down the latter to (try to) speed up the former. And programs shouldn't all have to mlockall, which iirc has other bad consequences. (maybe it allocates backing pages for all mappings immediately? even stuff like stack guard pages that's pure waste?) I have written programs to mlock (just) their program text; I'd prefer that weren't necessary either.
I prefer that under memory pressure, a task dies promptly. Ideally the "right" task, but really it just has to be better than making the system unusable for a long time. Even if it kills the display server or pid 1 inappropriately, I was probably going to have to reboot anyway, so...
> I prefer that under memory pressure, a task dies promptly.
This is exactly what memory pressure indicators can be used for. Linux also supports volatile page ranges, that get wiped out under memory pressure even with no swap support.
I agree. Another issue with Linux is that it provides no way to choose which tasks to kill when you start heavily swapping, and the OOM killer won't do anything because it still thinks everything is fine. You can't just use the normal UI to kill tasks at that point because it will have slowed to a crawl due to all the swapping.
Back when I used Linux with spinning disks it was much better to disable swap and hope the OOM killer did something sane than to wait for 10 minutes for a terminal to open and eventually give up and hard restart.
Windows handles it much better. You just press ctrl-alt-delete and because the GUI is properly integrated into the kernel it can always react and pause other tasks even if the system is heavily loaded.
The os will drop program text if it encounters memory pressure even if swap is disabled. Anything that isn’t anonymous memory is fair game and it does it for even well behaved programs on the system.
I mentioned that, yes. It's also problematic, particularly on HDD systems. (SSDs are fast enough that it's mostly okay.) This is why I said "Linux's low memory handling is deeply flawed" rather than just "turning off swap fixes everything".
Without swap, even with SSD's, the system gets into a very bad state when a cgroup is close to running out of memory.
The behavior ends up being that the kernel ends up constantly swapping out everything but anonymous memory until it can't anymore and then it finally terminates the process when it can't page anything else out. That can unfortunately impact other process on the system during that time.
It seems like there are enough failure modes that everyone has a different story about bad behavior, and unfortunately I don't think there's one solution a sysadmin can adopt (e.g., swap or not, sysctl settings, cgroup settings) that reliably prevents them all. I've been guilty in the past of telling people turning off swap dramatically improves things, but in your scenario it sounds like the opposite is true. /shruggie
The PSI-based oom daemon seems like an improvement, but I imagine there are still gaps there if it doesn't react quickly enough. Adding more parts can often paper over problems but rarely completely solve them...I think what's needed is a rethink: don't page these vital things out. Instead kill a process (chosen by userspace if possible, anything if not). Reserve enough for the killing rather than doing more reclamation to make it happen.
Swap is exceptionally evil with any managed language (e.g. Java/C#) - running the GC over swap is effectively indistinguishable from a hard crash... except it takes forever.
I have not used swap for over 20y, and never "missed" it.
You do need swap if you're running Linux on a laptop!
If you want Hibernate to work on your Linux laptop, you'll need to have a swap partition with space equal to (or bigger than) your RAM. Apparently, while there is a way to avoid setting aside a partition by using a swap file instead, it doesn't seem to work as well as using a partition.
Hibernate seems to work OK on my current laptop running Linux * and using a swap partition. I've had poorer experiences with previous laptops though. Now that I'm sure it's possible, having functioning "Hibernate" is a requirement for me when buying Linux laptops.
* - A LaptopWithLinux laptop running up-to-date Manjaro.
Hibernate is not a requirement for everyone. First time I set up Arch Linux, I just never got round to setting up swap, and never felt any need for it either. Over the course of three years, I ran into OOM trouble four times (8GB of RAM was a lot in those days…).
On my current laptop, I have a swap file and hibernation works fine (because this time I wanted to be able to switch over to Windows occasionally briefly without losing my session). It’s definitely more fiddly to set up in the Arch DIY style, but it’s still just following a set of instructions, so a distro that didn’t have the DIY philosophy could set it up easily enough.
I think this is a little overblown. The old kernel ntfs driver which nobody uses and which is read only anyway ignores the fact that a drive is mounted by windows, so you could read corrupted data from the ntfs drive mounted by windows if windows hasn't fully completed writes that are hibernated, but you can't fuck up the partition because Linux won't write anything.
The user space ntfs-3g, which is the one everyone actually uses, will refuse to mount a hibernated windows drive. The same appears to be true for the new kernel ntfs3 driver, since it's mount options include an option to force mount a dirty drive.
If you use the options to force mount a dirty drive read/write, then yes, all bets are off, but you have to have dug pretty deep into your config to find out how to do that and at that point you have to know what you're doing.
Okay, so I'm aware of the conundrums surrounding mounting the ntfs volume of a hibernated windows installation detailed on the wiki page.
However, the same section clearly states that booting another OS even while a Linux system is hibernating can result in a broken EFI System Partition.
This clearly does not relate to NTFS. I'm not knowlegdeable enough to understand the means by which this occurs, but I felt like I needed to give the person a heads-up before they brick their install.
Most people don't really need hibernate. If you use your computer every couple days sleep is good enough. And if you make sure the apps you use save their state often enough that you won't lose things, it's just as fast to boot normally.
Hibernate is just a workaround for having too long power-on or power-off times.
I have been using Linux on all my laptops for the last 20 years. I have always taken care to configure them such that the boot time does not exceed some tens of seconds at worst.
The result is that I have never felt any need to use either hibernate or a swap.
It's also related to the amount of volatile state that exists in applications: it doesn't matter in the slightest to me that I can boot my laptop to the desktop in 10 seconds, it'll take 10 minutes to actually open all my applications and restore them to the state they were in when I turned it off.
All the applications where the volatile state matters, e.g. browsers like Chrome or Firefox, source code editors, office applications, etc. already restore their tabs/working documents on restart, in a second or so on my computers.
If there are such applications which do not restore their previously open documents upon restart, it is they who should be improved, instead of using hibernate.
More frequently than wanting to close the entire computer, I just happen to stop using a certain application. In that case I want to be able to close it and restart from the same state at some later time.
So this is a feature that I want in any application. If the feature already exists at the lower granularity of each application, it also removes the need for a hibernate feature of the entire computer.
The biggest issue is restoring terminal states. I think there are some tmux plugins for this (but never looked too much in to it), and maybe some terminals support it natively too, but getting it properly restored is not so easy (including environment, shell variables, the shell's "local" history, and that kind of thing). Maybe there's some zsh plugin for it though.
Now, if you don't use terminals much then that's not much of an issue, but I tend to do most things there, with Firefox being the only actual GUI I'm running in Xorg.
> Swap is a storage area for these seemingly "unreclaimable" [anonymous] pages that allows us to page them out to a storage device on demand. This means that they can now be considered as equally eligible for reclaim as their more trivially reclaimable friends, like clean file pages, allowing more efficient use of available physical memory.
I don't see that argument. Why should anonymous pages be treated equally?
And executable pages / program text, maybe those also never should be swapped out? Is this configurable? This is usually more the behavior I would want on my system. I want that the OOM killer just kills sth directly.
I remember that Google also has some custom handling for OOM on Linux. Although I don't find much information now on it, except: https://lwn.net/Articles/432223/
> And executable pages / program text, maybe those also never should be swapped out? Is this configurable? This is usually more the behavior I would want on my system.
Aren't they? It's always been my believe that when such pages are reclaimed, they are not swapped out because they can be read back from disk anyway. Don't these pages fall under what the author means by "clean file pages"?
Swap implementations have gotten a lot more sophisticated and complicated over time. But fundamentally it's an undesirable thing to be dependent on it as it is just a lot slower than normal memory.
Part of the problem is that there is a lot of magical thinking rooted in decades of half truths about swapping on various operating systems with completely different implementations. Most of those per-conceptions if they ever were true at all, are now obsolete.
For example, Windows XP used to have swapping. It commonly sucked the life out of any PC it was on. For no good reason whatsoever. Symptoms: the system performance would degrade over time and people would routinely reboot to fix that. Even with plenty of memory on a system, it would spend lots of time swapping things in and out of memory. Very infuriating if you just spent a lot of money on RAM. It was optimized for people with cheap hardware. Computers with enough memory were an afterthought. So it pretended you were running out of memory even when you weren't.
Easy solution: disable swap completely; especially on machines with more memory than the total addressable memory per process on 32 bit windows XP (2 GB). I remember having a particularly slow windows laptop with 4GB of RAM. My life improved massively after I disabled swap on that thing. Alt tab was instantaneous; always. I could get individual processes to crash with an out of memory error but only if I deliberately opened way more processes than fit in memory (so, don't do that). Exactly the kind of thing that would make the laptop completely unusable with swap on in any case. I actually preferred the quick crash. Trying to close a non responsive application to free up some memory while the system is 100% engaged with grinding your disk to fine dust is not fun.
Linux is way more complicated and it depends on the use case. A swapping server is a dead server. So, you should be conservative with any kind of swap there. Basically, if you have enough memory and no memory leaks there should be no need for it to degrade response times (aka swap), ever. If you do have memory leaks, swapping is actively harmful because it degrades response times. Better reboot the server. Or rely on the docker oom killer. Or assign more memory. Spin up more servers. Etc. And of course you monitor memory usage on servers to preempt any kind of issues. Swapping is the punishment that happens on a poorly, configured un-monitored server. You should have no need for it.
For a laptop with not enough memory because the buyer tried to save some cash and bought something with a pitiful amount of memory, definitely have swap space available. Performance will suck but at least it will be able to run some software. And Linux isn't as bad at this as Windows XP used to be. People actually care.
Otherwise, you probably want it to be able to hibernate and you need swap space for that. But perhaps with a very low swappiness value so that it won't actually swap. Or in case you don't need hibernate, turn it off completely and save some disk space.
The windows XP era was quite long. I used it between 2001 and 2008. The laptop I mentioned, had more memory than the 32 bit version could technically allocate for a single process and most processes used far less than the maximum. So, I could run one big process like that and have all the usual stuff in the background without much issues.
The 64 bit version was released later and of course allowed for more memory. They never really addressed the swapping issues. So it continued to make sense to upgrade the memory and switch off swapping as opposed to replacing the entire laptop because it was "slow". Most instance of people complaining about their laptop performance basically were running into this and never realized the fix was a 100$ upgrade of their memory + simple configuration change. A lot of those laptops are still fine for running Linux today.
I agree with this post, though there's another factor to consider: SSD endurance. A system that swaps a lot will write more, decreasing endurance. You might need to buy a new SSD every 5 years.
I have 16GB of RAM in my linux laptop running Ubuntu and it isn't quite enough: Chrome, Virtual box, gopls and thunderbird regularly push it over the edge into swap death needing a reset to recover.
My first install was the earloom killer a user space oom killer. This has a nice desktop notification and when things got tight I'd get notifications about it killing chrome processes.
That made things liveable with but not perfect. I couldn't run Zoom without earlyoom killing it.
I finally added swap to zram and that has helped enormously meaning I have enough ram to run everything I need without the earlyoom kicking in regularly.
Anyway I'm just about to upgrade to a new laptop I can put more than 16GB of ram in, but I think I'll leave both the earlyoom and the compressed swap in place.
The one thing that used to drive me crazy, especially about Windows, was "why the hell are you using swap and why are you freezing to do stuff with the hard drive on each click when you have a third of the physical RAM free right here according to your own task manager". I still don't understand that. I still feel like modern desktop OSes are terrible with memory management when you run out of physical memory (which I did way too often on all my previous machines).
Not sure if the above case has been solved in M1 Mac
On x86 Mac, if the programs exceed the memory available, it would start swapping to disk too, though that is not as how Windows do swapping. On Windows, if I don't enable pagefile swap, some programs will crash even if there are so much memory available
Anyway, I do monitoring on memory usage on any OS that I use. Out of memory situation by people mostly is due to negligence, it's what it is. If people are unhappy because doing memory monitoring would waste their focus energy, just buy more computers
With the concerns stated in the article, I can't see any benefits to adding swap vs adding an equivalent amount of ram. The author mentions that not needing swap due to having more than enough ram is a common misconception, but I cannot see how that can be. For any given workload, you could replace the useful amount of swap for an equivalent amount of additional ram. Sure some pages cannot be reclaimed, but the amount of anonymous pages that can be reclaimed is limited by swap size. In an alternate system with more ram instead, the pages that would be in swap instead are locked in memory, but this does not give more useful memory. More ram would be more flexible as it allows more cache than the swap scenario when you dont have swap full of pages. Thus we can conclude that there will always some amount of ram that is sufficient, and therefore if your system has enough memory that you dont face memory contention, you don't need swap.
If theres a better explanation for why swap is useful, I'd like to hear it, but the article is unconvincing.
I have stopped using swap on all my servers, desktops and laptops about 20 years ago.
I have never encountered any situation when swap could have been useful, but I have always installed generous quantities of RAM on all computers.
As long as the quantity of used memory does not exceed the physical memory, swapping pages cannot improve the performance in any way.
The only exception, which is mentioned in the parent article, is when some very seldom used memory pages would be swapped and the freed memory would be used as a disk cache, hopefully increasing the performance for a I/O bound application.
Whether this strategy of hoping that a larger disk cache, filling all the available physical memory, can improve the performance more than what can be lost if the prediction that the swapped pages will not be needed soon is wrong, is extremely application dependent.
This hypothesis, that a significant performance improvement can be enabled by a slightly larger disk cache, might happen to be true in certain cases on computers which still use slow HDDs.
Except for backup/archiving, where the disk cache matters very little, I have not used HDDs for more than a decade.
On computers with SSDs, the chances to find an application where a small increase in disk cache size can cause a noticeable performance increase, larger that what is lost by swapping the seldom used memory pages, are negligible, in my opinion.
172 comments
[ 2.9 ms ] story [ 206 ms ] thread1. Allowing a process's mapped pages to be flushed to disk will cause performance to become unpredictable. This applies to both anonymous pages written to swap, and file-backed pages. The first thing every server process should do is mlockall(MCL_FUTURE) so the kernel can't decide to swap out parts of your RPC handlers. More sophisticated implementations (e.g. databases) can selectively mlock() specific pages.
2. Using swap to mitigate memory overcommit isn't useful because the process that overcommitted its memory should just be killed instead. This is where cgroups are useful: you can load test to understand the allocation curve, then tell the kernel to limit your service's process to 64 GiB or whatever. If it tries to go wild and take up more than its share, it gets a SIGKILL instead of taking the entire machine out of service.
3. Swap will destroy SSDs. You thought the write load from logging is bad? Try putting a consumer-grade SSD into a machine with 512 GiB RAM and let the kernel swap to it -- that SSD will be dead in a year.
My reading of this article is that he is specifically advocating NOT using it as a replacement for having adequate RAM.
My experiences, around 5 years ago, of running ~100 VMs without swap, basically confirmed the assertions in the article. Mostly the systems would run ok, but when they got into memory pressure they would live-lock. I had hoped for just OOM killing, which happened sometimes, but also sometimes the machine was just locked.
I switched over to having some small amount of swap, maybe 0.5-1GB on these boxes and haven't had a livelock since.
Plus, it's better to get unused pages out of RAM, which can be written to swap.
The one thing that HAS killed an SSD, in this case an Intel (320 series sticks in my head) is ZFS slog+L2ARC. Swap hasn't been a problem in my experience, but I also am not using swap to account for not having enough RAM.
That said, I agree with other commenters that said this sounds like a kernel bug. I am wondering what was going on to wedge the system that bad. Would putting executables on tmpfs (so they cannot be paged out) helped?
Are you sure? I’ve heard the argument before that swap is one of the better cases for SSD longevity, because it’s few big writes instead of many tiny ones, and found that convincing. Also, on which workloads would a machine with that much RAM touch swap at all? My workstation has “only” 64 GiB, has swap configured (to enable suspend), even during heavy use with some databases running never touches it.
Also:
> on which workloads would a machine with that much RAM touch swap at all
On most of them? You seem to be using a flawed model of the Linux swapping algorithm. Writes aren't caused by lack of memory.
To be clear, yes, it's well-known that the kernel will page out rarely used stuff, but those are (almost by definition) pages that are basically never written to. The claim was that writes to swap occur all the time, to the point where they wear out the SSD, that seems to be something different.
At least I've never seen anything like that with any of my (workstation) workloads, like large C++ builds, TensorFlow experiments, or running lots of virtual machines. And that was only with 64 GiB RAM, not even the 512 GiB that the post I replied to mentioned.
How much RAM you have is not really relevant.
Performance was great. Normally when swapping the performance of the machine is terrible, as you say. But those Optanes are so fast that it just didn’t matter. I was doing several hundred thousand operations per second, and they were all taking <4µs. It was a wonderful upgrade.
I have an OCZ Vertex LE in a 32-bit laptop that was my daily driver for somewhat-serious development for the better part of a decade and sees somewhat regular use as a bedside media consumption machine. It ran (and still runs) Gentoo Linux, so -while it was my daily driver- its drive saw _frequent_ writes due to weekly system-updating-build activity.
This machine has 4GiB of RAM installed, of which only 3.2 GiB is available due to weird BIOS limitations. It has _always_ had swap enabled, and has _often_ made heavy use of it.
This SSD has 93,560 power-on hours, and has written 48,384 GiB. As far as I (and SMART) can tell, it's as good as the day I flashed the v1.1 firmware on it.
I don't believe your claim is generally true. Other folks who have intentionally run drives _way_ past their advertised wear-out points have similar stories to mine.
If you used it as swap in a server, you could expect something closer to its max write rate, which is on the order of 10-20 TiB per day[1]. Run that for a year and you're at 3+ PiB total writes.
[1] Contemporary reviews say the OCZ Vertex LE had a maximum write throughput of 250 MB/s. Times 3600 (per hour), times 24, converted to IEC notation is 20,116 GiB per day.
Which you should avoid by having proper monitoring.
You can run with a low swappiness value and the kernel will not proactively swap out process memory unless required. Even light use of swap is probably fine as it likely involves rare process pages that will never be written to after being swapped out so there's no threshing on the SSD, only a few large writes.
I came from a SunOS/Solaris & HP-UX background (late 90s) where we routinely configured swap for the machines.
Later on I encountered Linux admins who refused to configure even a small amount of swap and they couldn't explain why. It seemed very odd to me. Nobody ever said the simple "swap sucks in the Linux kernel" to me.
Most non-Linux Unixes also do/did not allow memory (RAM+swap) over-commit by default, which may have coloured behaviours.
But, we had not storage to back that new memory. They purchased 2GB per machine for god knows what, but didn't add 2GB discs. So we literally couldn't use it (because of how storage was allocated on that cluster, we couldn't just re-carve it to get an extra 2GB swap, it had dedicated 2GB drives for the swap).
There's no way you'd enable swap equal to memory size these days, for large memory machines.
I can buy >16TB disks: I don't think I've deployed a machine with more than 1TB of RAM. (Maybe 2-4 (out of hundreds) in an HPC environment at previous employer had 1.5T?)
You should have swap. You should have swap!!
This video is one of the most amazing, useful, gem-after-gem explanations of Linux you could possibly ever encounter. There are so so few who have the in depth knowledge, across so many broad experiences, at such scale, to understand Linux's memory management.
Side topic #1: this video's advice on cgroups just makes me sad, because I love Kubernetes, but Kubernetes building it's own scheduler, in contest with not in tandem with the kernel's cgroup system, is super super super f-ing sad. Kubernetes running all it's shit in a single cgroup is some decrepit sad sad shit. I still love & user Kubernetes, but god damn, for what basically amounts to a cloud os, it's using the underlying host-OS extremely extremely poorly. This video was key in making me think & consider how much we've lost, how sad it is we don't take advantage of what the kernel is trying to do for us with trading off workloads. Leaving the kubernetes scheduler to try to fit all the workloads feels like we're letting a bunch of ignorant elementary schoolers run the show. It's embarrassingly superficial.
Side topic #2: it would be really interesting to try to go a bit deeper on the topic. I'd love a world where we can easily ask- what are these anonymous rarely used memory pages we get to swap out? That's the main advantage, under low & medium pressures: the system has some flexibility to move out things not really used, where-as it has to keep everything resident without swap. But answering: what are these things not being used: that probably requires bptrace foo to figure out what regions are getting swapped, plus deep skills to figure out what those addresses correspond to. I doubt Linux will just tell us, oh, I evicted this text block from the binary. (It likely doesn't know much about what blocks it's evicting.)
So, what he talks about on that video is a very specialized performance optimization that may work on some (certainly not all) systems that you have complete control of. It isn't general advice in any way.
If you have a build farm / CI machines, don't use swap. With swap, if a user schedules too many compiles at once, machine will slow to a halt and become kinda-dead, not quite tripping dead timer, but not making any progress either. Instead, set up the OOM priority on the users processes so they are killed first. If OOM hits, clang is killed, build process fails, and we can go on.
If you have a latency-critical server, don't use swap. With swap, some data will get swapped out, and you will have sudden latency trying to access it. Without swap, it may still swap out the code, but the code is usually much smaller than the data, plus it is all pretty hot.
If occasionally you have to process a very large datasets, and you are willing to wait a minute or twenty, you may enable swap. But be patient, and make sure your ssh has no timeout which can be hit.
If you have a small amount of RAM, enable swap. It would be slow, but at least you would be able to do something without crashing.
This doesn't really work that well. It's true that if you enable swap and have significant memory pressure for any extended period your machine will grind to a halt, but this will _also_ happen if you don't use swap and rely on the Linux OOM killer.
Indeed, despite the lack of swap, as part of trying to avoid OOM killing applications, Linux will grind the hell out of your disk - because it will drop executable pages out of RAM to free up space, then read them back in again on demand. As memory pressure increases, the period of time between dropping the page and reading it back in again becomes very short, and all your applications run super slowly.
An easy solution to this is a userspace OOM-kill daemon like https://facebookmicrosites.github.io/oomd/ . This works on pressure stall information, so it knows when your system is genuinely struggling to free up memory.
On the historical fleets I've worked on pre-OOMD/PSI, a reasonable solution was to enable swap (along with appropriate cgroups), but target only allowing brief periods of swapin/out. This gives you two advantages:
* allows you to ride out brief periods of memory overconsumption
* allows genuinely rarely accessed memory to be swapped out, giving you more working space compared to having no swap
I’m sure someone somewhere is able to use swap and not have the machine death spiral, but from desktop to servers? It’s never been me.
I always disable swap for this reason, and it’s always been the better choice. Not killing something off when you get to that point ASAP is a losing bargain.
I even finally disabled it on the lab raspberry pi’s eventually, and a SBC I use to rclone 20+ TB NVR archives due to performance problems it was causing.
It’s a pretty consistent signal actually - if I look at a machine and it’s using any swap, it’s probably gotten wonky in the recent past.
Batch process for example.
With proper monitoring you can actually act on it yourself instead of just restarting which just leads to a oom loop.
If you can take the latency hit to load what was swapped out back in, and don’t care that it wasn’t ready when you did the batch process, then hey, that’s cool.
What I’ve had happen way too many times is something like the ‘colder’ data paths on a database server get pushed out under memory pressure, but the memory pressure doesn’t abate (and rarely will it push those pages back out of swap for no reason) before those cold paths get called again, leading to slowness, leading to bigger queues of work and more memory pressure, leading to doom loops of maxed out I/O, super high latency, and ‘it would have been better dead’.
These death spirals are particularly problematic because since they’re not ‘dead yet’ and may never be so dead they won’t, for instance, accept TCP connections, they defacto kill services in ways that are harder to detect and repair, and take way longer to do so, than if they’d just flat out died.
Certainly won’t happen every time, and if your machine never gets so loaded and always has time to recover before having to do something else, then hey maybe it never doom spirals.
I do a lot of ci/CD where we just have weird load and it would be a waste of money/resources to just shelf out the max memory.
Other example would be something like Prometheus: when it crashes and reads the wal, memory spikes.
Also it's probably a unsolved issue to tell applications how much memory they actually are allowed to consume. Java has some direct buffer and heap etc.
I have plenty of workloads were I prefer to get an alert warning and acting on that instead of handling broken builds etc.
I am a little surprised that every machine you admin has had issues related to swap. Feels high.
For the ones that are now using swap and likely went wonky before, how many would have that crashed due to said wonkiness?
For my low-end notebook with solid-state storage I set the kernel's swappiness setting to 100 percent and this problem got magically fixed. It's rock-solid now.
I don't know how it works but it does.
I've also had plenty of machines that fill the swap and then processes either crash when malloc fails or the kernel kills some stuff (sometimes the wrong thing) or something things just hang. Measuring memory pressure is tricky, a small swap partition (I like 512 MB, but limit to 2x ram if you're running vintage/exotic hardware that's got less than 256MB) gives you some room to monitor and react to memory usage spikes without instantly falling over, but without thrashing for long.
You should monitor (or at least look at) both swap used % and also pages/second. If the pages/second is low, you're probably fine even with a high % use, you can take your time to figure out the issue; if pages/second is high, you better find it quick.
What's bad is a high frequency of moving pages in and out of swap. This is something that can cause your machine to be functionally unavailable. But it is important to note that you can easily trigger somewhat-similar behaviour even with swap disabled, per my previous comment. I've seen machines without swap go functionally unavailable for > 10 minutes when they get low on RAM - with the primary issue being that they were grinding on disk reloading dropped executable pages.
I agree that in low memory situations killing off something ASAP is often the best approach, my main point here is that relying on the Linux OOM killer is not a good way to kill something off ASAP. It kills things off as a last resort after trashing your machine's performance - userspace OOM killers in concert with swap typically give a much better availability profile.
In a situation where a bunch of memory is being used by something that is literally not needed and won’t be needed in a hurry, then it’s not a big deal.
In my experience though, it’s just a landmine waiting to explode, and someone will touch it and bam useless and often difficult to fix machine, usually at the most inconvenient time. But I also don’t keep things running that aren’t necessary.
If someone puts swap on something with sufficiently high performance, then obviously this is less of a concern too. Have a handful of extra NVMe or fast SSD lying around? Then ok.
I tend to be using those already though for other things (and sometimes maxing those out, and if I am, almost always when I have max memory pressure), so meh.
I’ve had better experience having it fail early and often so I can fix the underlying issue.
Those build servers had at least 64GB of RAM, while executables were less than 1GB (our entire SDK install was ~2.5GB and it had much more stuff than just clang). So a machine would need to finely balance on memory pressure: high enough to cause clang to be evicted, but low enough to avoid OOM killer wraith.
I don't think this is very likely in machines with decent amount of memory.
Not sure why you wouldn't want swap for it?
It will allow you to fine-tune the build later and give that build a realistic chance to finish
Yes, killing that part of the build did destroy the work of hours. It was still better to disable the swap than try to "ride it out".
And it also totally depends on how much memory is missing.
I still prefer to have something taking 20 minutes longer instead of failing and fine-tuning the resources after it.
Once Kubernetes decides which processes should run on each machine, it allocates separate cgroups to each process group, which it uses for enforcing CPU and RAM resource limits. You can see this by examining the cgroupfs control filesystem. How would you propose improving this?
Note that Kubernetes now has code to work with swap, although some work remains. Tracking issue: https://github.com/kubernetes/enhancements/issues/2400
There is a lengthy conversation in this issue, with some deep insights (and a lot of misconceptions).
https://github.com/kubernetes/kubernetes/issues/53533
When I've used machines with swap, my experience has been the OOM killer was much less aggressive. As a result you're left with an unresponsive system blocked on disk I/O.
I used to run with no swap; when compiling with many parallel processes, my system would sometimes live-lock. The OOM killer never triggered, and as far as I could tell, the issue actually was I/O saturation caused by thrashing on the few pages that the kernel _could_ evict. After enabling swap, I've never encountered this issue again. The machine remains interactive, if slow, under high load.
Anecdotally, SIGSTOP-ing some processes when memory pressure is high, even without swap, seems to have some benefit. I'd resume those batch processes later.
Back in the day I used a lot of FreeBSD on fairly cheap/constraint machines (even for the time) and somewhat frequently-ish ran out of memory, and it was a much "better" experience. I mean, OOM killing stuff is never a "good" experience, but it didn't lock for over a minute.
With swap: We can choose to swap out rarely-used anonymous memory that may only be used during a small part of the process lifecycle, allowing us to use this memory to improve cache hit rate, or do other optimisations."
How do you improve cache hit rate when there's no memory contention? And what other optimizations?
"Without swap: We cannot swap out rarely-used anonymous memory, as it's locked in memory. While this may not immediately present as a problem, on some workloads this may represent a non-trivial drop in performance due to stale, anonymous pages taking space away from more important use."
When there's no memory contention, what's the "more important use"?
These are honest questions - I'm not an expert on the topic.
The claim being made is that there are certain situations where processes allocate a lot of physical memory but then very rarely accesses it. If it gets paged out to swap, then there's a latency hit on the next access. If the OS can avoid more latency hits total using the same physical RAM for caching, then the computer could do more overall calculations per second.
If on a hypothetical 8GiB RAM system, programs require 6 but 2 of those are rarely used, there is no memory contention but the extra space for disk cache is still a helpful boost.
I do think program memory is more valuable on most desktop systems, since commonly memory access is expected to be fast while disk access is expected to be slow. This means that interactive software tends to mitigate for slow IO, while it can't do much against being swapped out.
On servers however I'd probably rather let the kernel do its job (at least if I'm focusing on throughput), unexpected slowness may worsen a worst case but the average case is helped enough.
Luckily, the Linux kernel, for all of its memory management flaws (don't get me started on overcommit), allows configuring this using the vm.swappiness parameter that tells it what to prioritize.
I had hoped that getting rid of swap would prevent thrashing, but instead the system would live-lock.
Terminating the GUI and performing an offline update from one SSH session while running `watch -n1 sync` avoids the hang. I haven't tried sync while leaving the GUI running.
Will Linux OOM handling ever be fixed?
But if you have a lot of processes, you may end up in a situation where you can free enough pages by swapping them all out to disk. Now you have no OOMKill, but the next time a process gets to run, you will stall until it is swapped in from disk. Then, the next process will be scheduled, and it too will need to be swapped in, causing another stall, and so on. The machine will probably end up executing, at best, a few instructions per millisecond for hours...
* https://news.ycombinator.com/item?id=13715249
Naturally, some people will have had consistently positive experiences with swap, while others will have come out with consistently negative impressions. 32GB RAM + 512MB swap backed by a fast SSD is very different from 2GB RAM + 512MB swap backed by spinning rust. JVM is different from MySQL, which again uses memory differently than PostgreSQL.
And yet we are so eager to hurl sweeping generalizations at each other, holding religiously onto our own subjective cargo cults. :(
On the server, sprinkling in a bit of swap was standard practice in some orgs. As under load I could still log into a slow server and do something about it.
My current problem with swap in Linux is that they make bad default choices. While a swapfile isn't optimal, allocating a slice of the disk for swap complicates my life when I do ram upgrades. Hibernate being tied to configured swap on a slice of disk (ugh), I just didn't even bother with it since upgrading ram.
Rails, Django, Elixir/Phoenix web development, sometimes with docker. No swap on my 32 GB laptop both when I used it with a 750 GB HDD in its early years and with two 1 TB SSDs now. I'm usually using about 20 GB RAM because I'm leaving every single project open in its own virtual desktop, to get there quickly if a customer needs something done.
No problem whatsoever that I remember. I'm never close to filling up RAM, that could be the reason. I'd buy extra RAM instead of enabling swap. The price would be extra battery depletion during the night.
What it decided to kill, was usually something I didn't want it to kill, like X
Ideally I'd want to tell it: first, kill java. Next, python. I think this is possible but I never was able to understand how to do it.
https://news.ycombinator.com/item?id=27218036
GetSortedLifecycleUnits sorts using GetSortKey, which for tabs is their last used time. So it’s just LRU. But some comments hint that maybe something smarter will happen eventually.
As for your overall question, there are no easy answers. Deciding if memory is being used well, or could be repurposed to do something more useful is a difficult question to answer.
https://source.chromium.org/chromium/chromium/src/+/main:chr...
Edit: also, it’s really hard to reason about how much memory you need unless it’s very well defined (you know the size of data, exactly what processes to run), once you get to desktop operating systems, it’s more run and find out than reasoning from first principals.
Disclosure: I work on ChromeOS.
There has to be some way to evict those memory hogs.
My experience with Linux machines that have HDDs and swap is that once they start swapping, they slow down not only until memory is freed (ctrl-C gets through, OOM killer runs, whatever) but actually until reboot. My theory is that processes' VM space is now a minefield. Both anonymous memory and file-backed mmaps now have many gaps which get paged in on demand with inadequate readahead. With each seek taking ~10 ms, it doesn't take too much of that before you should give up and just reboot.
Once memory is available, I'd like the system to proactively page back in both swap and file-backed pages to prevent this. If it does need to happen reactively, it should be in big blocks (each 10 ms of seeking should be follow by at least 10 ms of reading => ~1 MiB, not ~4 KiB or whatever it's doing now). By default.
I'm not convinced the decision to use swap is ever a good one. E.g., from the article:
> With swap: We can choose to swap out rarely-used anonymous memory that may only be used during a small part of the process lifecycle, allowing us to use this memory to improve cache hit rate, or do other optimisations.
This is short-sighted. Programs can be written to tolerate their filesystem I/O being slow. They often take care to do I/O on a particular thread which is not handling network or GUI operations. They can't realistically be written to tolerate arbitrary memory accesses (including program text, stack, and heap) being that slow. It's a stupid idea to potentially slow down the latter to (try to) speed up the former. And programs shouldn't all have to mlockall, which iirc has other bad consequences. (maybe it allocates backing pages for all mappings immediately? even stuff like stack guard pages that's pure waste?) I have written programs to mlock (just) their program text; I'd prefer that weren't necessary either.
I prefer that under memory pressure, a task dies promptly. Ideally the "right" task, but really it just has to be better than making the system unusable for a long time. Even if it kills the display server or pid 1 inappropriately, I was probably going to have to reboot anyway, so...
This is exactly what memory pressure indicators can be used for. Linux also supports volatile page ranges, that get wiped out under memory pressure even with no swap support.
Back when I used Linux with spinning disks it was much better to disable swap and hope the OOM killer did something sane than to wait for 10 minutes for a terminal to open and eventually give up and hard restart.
Windows handles it much better. You just press ctrl-alt-delete and because the GUI is properly integrated into the kernel it can always react and pause other tasks even if the system is heavily loaded.
This is false, of course. oom_score_adj, or just read on oom_killer.
Anyway, the "real solution" is oomd + PSI (pressure stall information), which can detect when memory is running low and the system is affected.
Without swap, even with SSD's, the system gets into a very bad state when a cgroup is close to running out of memory.
The behavior ends up being that the kernel ends up constantly swapping out everything but anonymous memory until it can't anymore and then it finally terminates the process when it can't page anything else out. That can unfortunately impact other process on the system during that time.
The PSI-based oom daemon seems like an improvement, but I imagine there are still gaps there if it doesn't react quickly enough. Adding more parts can often paper over problems but rarely completely solve them...I think what's needed is a rethink: don't page these vital things out. Instead kill a process (chosen by userspace if possible, anything if not). Reserve enough for the killing rather than doing more reclamation to make it happen.
I have not used swap for over 20y, and never "missed" it.
If you want Hibernate to work on your Linux laptop, you'll need to have a swap partition with space equal to (or bigger than) your RAM. Apparently, while there is a way to avoid setting aside a partition by using a swap file instead, it doesn't seem to work as well as using a partition.
Hibernate seems to work OK on my current laptop running Linux * and using a swap partition. I've had poorer experiences with previous laptops though. Now that I'm sure it's possible, having functioning "Hibernate" is a requirement for me when buying Linux laptops.
* - A LaptopWithLinux laptop running up-to-date Manjaro.
On my current laptop, I have a swap file and hibernation works fine (because this time I wanted to be able to switch over to Windows occasionally briefly without losing my session). It’s definitely more fiddly to set up in the Arch DIY style, but it’s still just following a set of instructions, so a distro that didn’t have the DIY philosophy could set it up easily enough.
The user space ntfs-3g, which is the one everyone actually uses, will refuse to mount a hibernated windows drive. The same appears to be true for the new kernel ntfs3 driver, since it's mount options include an option to force mount a dirty drive.
If you use the options to force mount a dirty drive read/write, then yes, all bets are off, but you have to have dug pretty deep into your config to find out how to do that and at that point you have to know what you're doing.
Okay, so I'm aware of the conundrums surrounding mounting the ntfs volume of a hibernated windows installation detailed on the wiki page. However, the same section clearly states that booting another OS even while a Linux system is hibernating can result in a broken EFI System Partition.
This clearly does not relate to NTFS. I'm not knowlegdeable enough to understand the means by which this occurs, but I felt like I needed to give the person a heads-up before they brick their install.
https://wiki.archlinux.org/title/Power_management/Suspend_an...
If you need swap enabled to have it, why would anyone want this? Swap makes everything worse.
I have been using Linux on all my laptops for the last 20 years. I have always taken care to configure them such that the boot time does not exceed some tens of seconds at worst.
The result is that I have never felt any need to use either hibernate or a swap.
If there are such applications which do not restore their previously open documents upon restart, it is they who should be improved, instead of using hibernate.
More frequently than wanting to close the entire computer, I just happen to stop using a certain application. In that case I want to be able to close it and restart from the same state at some later time.
So this is a feature that I want in any application. If the feature already exists at the lower granularity of each application, it also removes the need for a hibernate feature of the entire computer.
Now, if you don't use terminals much then that's not much of an issue, but I tend to do most things there, with Firefox being the only actual GUI I'm running in Xorg.
I don't see that argument. Why should anonymous pages be treated equally?
And executable pages / program text, maybe those also never should be swapped out? Is this configurable? This is usually more the behavior I would want on my system. I want that the OOM killer just kills sth directly.
I remember that Google also has some custom handling for OOM on Linux. Although I don't find much information now on it, except: https://lwn.net/Articles/432223/
Aren't they? It's always been my believe that when such pages are reclaimed, they are not swapped out because they can be read back from disk anyway. Don't these pages fall under what the author means by "clean file pages"?
Part of the problem is that there is a lot of magical thinking rooted in decades of half truths about swapping on various operating systems with completely different implementations. Most of those per-conceptions if they ever were true at all, are now obsolete.
For example, Windows XP used to have swapping. It commonly sucked the life out of any PC it was on. For no good reason whatsoever. Symptoms: the system performance would degrade over time and people would routinely reboot to fix that. Even with plenty of memory on a system, it would spend lots of time swapping things in and out of memory. Very infuriating if you just spent a lot of money on RAM. It was optimized for people with cheap hardware. Computers with enough memory were an afterthought. So it pretended you were running out of memory even when you weren't.
Easy solution: disable swap completely; especially on machines with more memory than the total addressable memory per process on 32 bit windows XP (2 GB). I remember having a particularly slow windows laptop with 4GB of RAM. My life improved massively after I disabled swap on that thing. Alt tab was instantaneous; always. I could get individual processes to crash with an out of memory error but only if I deliberately opened way more processes than fit in memory (so, don't do that). Exactly the kind of thing that would make the laptop completely unusable with swap on in any case. I actually preferred the quick crash. Trying to close a non responsive application to free up some memory while the system is 100% engaged with grinding your disk to fine dust is not fun.
Linux is way more complicated and it depends on the use case. A swapping server is a dead server. So, you should be conservative with any kind of swap there. Basically, if you have enough memory and no memory leaks there should be no need for it to degrade response times (aka swap), ever. If you do have memory leaks, swapping is actively harmful because it degrades response times. Better reboot the server. Or rely on the docker oom killer. Or assign more memory. Spin up more servers. Etc. And of course you monitor memory usage on servers to preempt any kind of issues. Swapping is the punishment that happens on a poorly, configured un-monitored server. You should have no need for it.
For a laptop with not enough memory because the buyer tried to save some cash and bought something with a pitiful amount of memory, definitely have swap space available. Performance will suck but at least it will be able to run some software. And Linux isn't as bad at this as Windows XP used to be. People actually care.
Otherwise, you probably want it to be able to hibernate and you need swap space for that. But perhaps with a very low swappiness value so that it won't actually swap. Or in case you don't need hibernate, turn it off completely and save some disk space.
Win XP-era machines had tiny amounts of RAM, so it makes total sense that swap would be heavily relied on.
The 64 bit version was released later and of course allowed for more memory. They never really addressed the swapping issues. So it continued to make sense to upgrade the memory and switch off swapping as opposed to replacing the entire laptop because it was "slow". Most instance of people complaining about their laptop performance basically were running into this and never realized the fix was a 100$ upgrade of their memory + simple configuration change. A lot of those laptops are still fine for running Linux today.
I have 16GB of RAM in my linux laptop running Ubuntu and it isn't quite enough: Chrome, Virtual box, gopls and thunderbird regularly push it over the edge into swap death needing a reset to recover.
My first install was the earloom killer a user space oom killer. This has a nice desktop notification and when things got tight I'd get notifications about it killing chrome processes.
That made things liveable with but not perfect. I couldn't run Zoom without earlyoom killing it.
I finally added swap to zram and that has helped enormously meaning I have enough ram to run everything I need without the earlyoom kicking in regularly.
Anyway I'm just about to upgrade to a new laptop I can put more than 16GB of ram in, but I think I'll leave both the earlyoom and the compressed swap in place.
Not sure if the above case has been solved in M1 Mac
On x86 Mac, if the programs exceed the memory available, it would start swapping to disk too, though that is not as how Windows do swapping. On Windows, if I don't enable pagefile swap, some programs will crash even if there are so much memory available
Anyway, I do monitoring on memory usage on any OS that I use. Out of memory situation by people mostly is due to negligence, it's what it is. If people are unhappy because doing memory monitoring would waste their focus energy, just buy more computers
If theres a better explanation for why swap is useful, I'd like to hear it, but the article is unconvincing.
I have stopped using swap on all my servers, desktops and laptops about 20 years ago.
I have never encountered any situation when swap could have been useful, but I have always installed generous quantities of RAM on all computers.
As long as the quantity of used memory does not exceed the physical memory, swapping pages cannot improve the performance in any way.
The only exception, which is mentioned in the parent article, is when some very seldom used memory pages would be swapped and the freed memory would be used as a disk cache, hopefully increasing the performance for a I/O bound application.
Whether this strategy of hoping that a larger disk cache, filling all the available physical memory, can improve the performance more than what can be lost if the prediction that the swapped pages will not be needed soon is wrong, is extremely application dependent.
This hypothesis, that a significant performance improvement can be enabled by a slightly larger disk cache, might happen to be true in certain cases on computers which still use slow HDDs.
Except for backup/archiving, where the disk cache matters very little, I have not used HDDs for more than a decade.
On computers with SSDs, the chances to find an application where a small increase in disk cache size can cause a noticeable performance increase, larger that what is lost by swapping the seldom used memory pages, are negligible, in my opinion.