This is a nice blog and it makes sense to me now. As a gamer and linux user myself, I've previously had to do tweaks and go-arounds without really understanding what was going on behind the scenes. :)
Great writeup, gpuvis looks particularly interesting and glad the kernel is providing tracepoints for performance events.
> Not only does the display hardware like scanned-out images to be in VRAM, it also completely skips past the GPU’s virtual memory architecture and works with physical addresses exclusively.
Well there's your problem. Only so smart your memory management can be when you have to pay the cost of doing it manually. Although presumably this only applies to a small fraction of the VRAM?
> Although presumably this only applies to a small fraction of the VRAM?
They did mention they saw 4GiB of eviction for a single 32MiB scan out image.
So while I would call the image allocation small, it seems to cause an avalanche of evictions. Amplified by the fact that each frame has one of these images, though I expect subsequent frames might have a better chance of already fitting into evicted space.
What I don't exactly understand is: doesn't it make sense to always reserve the contiguous physical memory for this case and not allow anything else to be put in it?
Or alternatively, instead of evicting it all, can you move the data around in physical memory while updating the page table, so that you can clear a large enough continuous block of physical memory?
I doubt it makes much of a difference, and you can always manually manage what data lives in the GPU when if you 100% have to overcommit. Games have a much larger and more diverse set of objects in the VRAM, and their usage is less predictable, so manual scheduling of the memory is infeasible typically.
If you have model layers/experts that reside in CPU RAM, it's generally better to do that part of inference on the CPU than pay the cost of shipping them over to the GPU. LLM inference is generally bottlenecked by data-movement, not compute. There may be a limited exception for prefill or perhaps decode of very wide batches, where shuffling the data around may be justified.
Great article. I find that I learn something every time I read a post about linux kernel work.
I guess an LRU with priority would handle VRAM for games pretty decently without going getting too application specific.
What about VRAM to Disk specifically NVME, would direct to disk be feasible for large workloads, I know it is used for streaming in assets directly via. PCIE, but i wonder how the performance would be on compute workloads running with NVME as a swap for GPU VRAM.
Gosh, 7.2 literally just dropped with a bunch of awesome of performance/gaming related improvements (large folios, cache-aware scheduling, improved MGLRU reclaiming, Fair GPU Scheduler etc)... and I already can't wait for 7.3 to come out.
Meanwhile in the Windows world, users hate updates... Like I genuinely can't think of a single instance that made users exclaim, "oh boy I just can't wait for the next Patch Tuesday!".
I might be in the minority here, but some of the latest Windows updates that dropped in the Insider channel are really cool, and I'm excited to see those coming to main.
I don't use windows too often, but I remember I was happy with an update a year or two ago which allows changing the volume by scrolling on the tray icon.
Now, of course, it still doesn't work with a touchpad, but baby steps, right?
This feature has been was available on GNOME since forever. I'm still waiting for Microsoft to steal another nice thing: MRU (Most Recently Used) keyboard layout switching. If you have two often used keyboard layouts A and B, and less used layouts C, D, E, then pressing the switch shortcut once will switch between A and B without the need to cycle through C, D, E.
I never really cared for Gnome so I've never actually used it, but I can't say I'm surprised.
But this has worked in KDE since version 3, like, 20 years ago? And also with pasystray running under i3, since before win11 was a thing. They both support scrolling with the touchpad, too.
I drove me up a wall not being able to do this when I had to use Windows at work.
As for your MRU layout thing, I'm not sure I'd like it. While I like MRU for alt-tab, i like my not-so-often-used functions to be predictable. For this specific case, I always use us-macintosh, so I'd like to be sure that if I press the shortcut X times I'll get Y layout, without having to remember the order in which I used them. Now, to be fair, in my case I only have US-Mac available since it does everything I need.
Not to start a flame war, but people who move their taskbar to the left edge of the screen are extremely sinister and should not be trusted, because they probably use tabs instead of spaces, and vi instead of emacs. Putting your taskbar on the right edge is fine though, as is using spaces and emacs. Putting the taskbar on the top or bottom just means you're gay, not that there's anything wrong with that. ;)
Any improvements with fractional scaling? Can anyone please link to any material reg the work being done for improving fractional scaling in Linux; it would be great if it could match the rendering quality of macOS.
In my laptop 200% fractional scaling of the built-in QHD+ OLED display on KDE Plasma is great, but when I connect my portable 1080p displays or e-paper monitor (UXGA) the fractional scaling of it is not as good as when compared to when the same monitors are used with macOS (M4) in HiDPI mode under lower resolutions.
As the other comment says, I suspect it to do with GPU as the built-in displays are usually driven by intel and external displays are always driven by my Nvidia 4090.
But surprisingly, when I drive my e-paper monitor using original Jetson Nano (Ubuntu 20.04 ESM / Nvidia Maxwell); the fractional scaling is quite good perhaps even better than macOS.
Also, I wouldn't hold the rendering quality of macOS as a high standard at all, since all it does is render at a multiple and then downsample, wasting energy and introducing blurriness.
Perhaps it's subjective, I have multiple portable displays including a e-paper monitor and I find that for low resolutions the HiDPI in the macOS produces more legible text in those monitors.
`gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer', 'xwayland-native-scaling']"
`
That stops being necessary as some point around 49/50 I think.
Phoronix.com is my go to place for good news. It always gives me a sense of progress and makes me feel humble since so many smart people share their work
I use arch based linux. This year updates have broken video decoding twice. Font rendering once (still not fixed without manually deleting files.). And broken the whole system due to there not being a mechanism that makes sure all QT libraries upate at the same time once or twice. That is all one system.
Meanwhile 4 separate windows pcs have not had problems with updates for at least 6 years
It's not fair comparing Arch and Windows as they are two very different types of OSes. Arch is bleeding edge and it is completely normal and expected to be prone to breakages. It's not meant for the average PC user, it's meant for experienced users who're comfortable with troubleshooting technical issues, reporting bugs, and know and are happy to downgrade packages/rollback to previous snapshots etc. On the upside, Arch also gets fixes before anyone else.
If you want a stable Linux experience though, pick something like Fedora, or even better, pick an immutable distro like Aurora or Bazzite, if you really, really care about stability.
Arch is rolling release distro. That story would be different even if it was rolling release linux vs fixed release linux.
That said, even some fixed release linux distros break things almost every release. That's because Windows updates is just windows base + kernel + drivers, while most linux updates are everything all at once with no distinction between base system and user applications. Plus windows and macOS present certain things more or less as a blackbox to user - I never had broken DE and keyboard layout switcher during windows update, while every ubuntu update used to break everything for me.
I use Arch exclusively and haven't had any of the issues you describe.
That said, I do occasionally deal with quirks due to the rolling distro nature of Arch. The benefits far outweigh the occasional upgrade pain IMO though.
How would you detect broken video decoding or font rendering? I don't know what those are. I don't think i've had any issues this year (Maybe i had do do some pacman-wrangling once this year, but i'm slow to update, so that was probably my fault...)), that I noticed anyway, but maybe everything's broken and i just can't tell.
The best thing about Win10 reaching EOL is not being at the mercy of Windows Update any more. It was by far the biggest risk to having a functioning system.
Most of the CVEs I've seen for that stack are in DNS, and Firefox's DNS-over-HTTPS implementation bypasses the OS DNS almost completely in sensible configurations.
MS should either fix the RCEs in Windows 10 (it is unsupported but the defects were already there when it was still supported, just undiscovered), or they should refund their customers (since they never managed to release a defect-free version).
Seriously my spare PC became so much less annoying overnight. It used to not only auto-reboot but also add some fresh non-dismissed nags with every update. Now it's just the nag to install Win11, which, I'd rather eat my entire PC.
Oh god that "OOBE" horror. I almost got locked out when it demanded a Microsoft Account login, which I never use for anything. I'd only just grudgingly acquired a portable telephone at that point; without it, I'd have had severe difficulty getting at my mementos.
You can (could?) install Win10 with no MS account by disconnecting it from the internet. But that was the only way. And you can't disconnect it halfway through the install, it has to be disconnected at boot.
I don't even mind making an MS account, I have one, just don't want Windows using that account and thus constantly freaking out about whatever bs like "you haven't enabled 8-factor auth."
I mean, Windows has managed to fix a lot of gaming/GPU stuff like this lately as well. They properly support HDR, VRR and even VRR with bornderless mode on older games - all of which has been very buggy and broken on Linux for a long time. Not to mention actually working eGPU support - trying to use DaVinci on Linux with eGPU still ends up with "Not enough GPU" errors and constant crashes.
This goes especially when used with more powerful GPUs from nVidia - so perhaps arrogance here isn't warranted?
Of course. These are the people that the industry needs who understands proper performance-level engineering.
The commits they authored in [0] match and all of what they have mentioned in the article and this is how you know the author knows exactly what they are talking about and have explained it well.
Puts all the vibe-coders, and the so-called "AI Engineers" to shame.
I imagine "we" "owe" a lot more to young men or just men in general for the same, but I guess there's no virtue signaling points to be harvested from making that observation.
It seems my praise was hurting you. Let me praise some people from an unrelated project: we owe a lot to the greybeards keeping Scribus alive. And you are right, we owe a lot to all kinds of people.
To quote myself from another comment: «I am inclined to rather point out remarkable things than unremarkable things. There seems to be relatively many trans people in this niche.» That's all. I didn't know this required a trigger warning.
> virtue signaling points
I didn't know about them but you do. Can I sell them to you?
does niche coding cause men to transition? does autism cause men to become niche coders and transition? does porn addiction cause autistic male niche coders to transition? does niche coding cause autism, and that combined with porn addiction cause a gender crisis? nobody knows. this would be extremely interesting research.
> As a side note, it strikes me how much we owe to young trans people for low level performance engineering.
The issue of gender identity is clearly irrelevant and off-topic here and your other responses make it obvious that you intend to provoke by introducing it; people asked you entirely fair questions about your motivations and you responded by questioning their psychological stability and making an ironic swipe about "trigger warnings" (this exact phrasing makes it clear you're accustomed to this discourse).
> Please don't use Hacker News for political or ideological battle. It tramples curiosity.
Indeed, I made a small friendly side note to provoke you. Sir/madam, I sincerely think your reasoning is a bit off here, but if you feel you have been done wrong by me, please inform the admins. For the rest, please go back on topic.
Can someone please explain to me why it is apparently off-topic to point out that someone is bringing up irrelevant issues of gender identity, but not off-topic to do so in the first place?
I worked on a PC/PS4/Xbox One game once. I wish I remembered more details, but at some point when looking at memory usage I found a 2048x2048 texture for something on a character. This texture was never anywhere near occupying the full screen. It got downsized once I told one of the artists, but I wouldn't be surprised if some games are inadvertently shipping with that kind of waste.
The release version of City Skylines 2 had enormous issues due to this problem. Some tiny details, like a pile of wood behind a shed, had ridiculous amount of pixels. I do believe that should be pretty much fixed in that title nowadays, however.
I don't know about that specific optimisation, but I remember Helldivers 2 did drop support for loading from HDDs, which allowed them to deduplicate assets bringing the game down from what I remember like 120GB to somewhere round 30GB.
I hope there will be an update where when my RAM gets full my PC doesn't freeze and becomes unusable... I remember that Linux and Windows do this in different ways and Windows doesn't have the problem.
Me too, but you can improve the behavior yourself too. It's been an issue with desktop linux and the default settings for a long time. You could maybe tune OOM killer settings, but another option is to install a package like earlyoom: https://man.archlinux.org/man/earlyoom.1.en
When my Intel Mac ran out of disk space once, it became impossible to clear the space without rebooting (due to APFS's annoying journaling crap or whatever -- it would literally say "rm: can not remove [whatever]: no space left on device" or something), which caused Google Chrome to wipe all my local storage and cookies because it loves to do that when you run out of disk space, and there's no way to get that data back even if you do somehow clear space with the browser still running. I've been bitten by this many times on at least four different platforms. I wish it could just keep everything on disk instead of deleting it all? Like anything that deletes my data automatically can just go die in my opinion. This is one of the reasons I don't use Google Chrome anymore, another is how it deletes "old" history too (after 3 months) even if you have terabytes free, which is just so offensive I can't even explain it. Ungoogled Chromium lets you turn some of this stuff off, but when I moved to Mac I just started using Safari.
The only way to save my file system was to boot into recovery mode and run a verification with Disk Utility, then I was able to remove some files and get my space back.
Both MacOS and Windows start having issues if you have less than 10% of the disk free. The problem with the Mac is your ~/Library/Application support/ folder can get huge and take up a ton of space. Mine is currently at 55GB
I've also found that the fastest way to totally lock up a Mac kernel is to have some external hard drive fail while connected. You will be unable to open, quit, force quit, or reboot, and it won't boot up again unless that drive is disconnected.
Its not enabled by default, but enabling MGLRU setting /sys/kernel/mm/lru_gen/min_ttl_ms to 70 has entirely resolved the issue for me. It _does_ usually pick firefox or vscode to kill, but my system remains interactive, with no need for earlyoom.
I kept my floppy drive installed until I got tired of explorer freezing every time I accidentally selected it (quite often as I have a habit of navigating with the keyboard)
Are these directly comparable, though? Doesn't Windows default to using free space on the system drive to grow the swap file in case of memory pressure? Linux has you set up swap files/partitions explicitly.
From my experience, Windows does freeze and become completely unusable too, just not usually from RAM usage hitting 100%.
I don't use Windows much, nor do I care much about that environment, but from what I've seen it seems to keep RAM usage below 100% most of the time. What I do see pretty often is the drive getting stuck at 100% usage instead, which makes the whole system ridiculously unusable anyway.
This has so many variables that it's practically useless as a data point. We have a few Ubuntu VMs running complex business web applications and their Postgres database on only 4 GB or RAM and 20-50 GB of storage, no swap space. Ocassional high load, but Zero OOM problems for many years.
We could upgrade the RAM, but we were stingy when we created them, and it has worked well so far, so why waste resources?
True. It's just a tiny Lenovo that I've repurposed as a homelab and that is punching above it's weight with to many applications I suppose. Either way the logs point to RAM issues but it could be my incompetence.
Windows uses eager memory allocation, and applications will die when they allocate (and think image editor loads a beefy image) or fail to start when it runs out of memory.
Linux uses lazy allocation and overcommits, so what dies isn’t necessarily what you would expect thanks to the OOM killer.
With a fixed swap size windows doesn’t necessarily slow down as it runs out of memory.
YMMV if you don’t have 64 gigs of RAM and a 64 gig swap.
macOS includes a GUI user land whereas Linux is just a kernel.
So Apple can build a whole user-controlled GUI workflow around memory exceptions, which Linux cannot.
Maybe what needs to happen is new Linux syscalls + signals for DEs to utilise for building memory exception UIs? But the problem then with that is you’re impacting the portability of those DEs. So some maintainers might still refuse to work with Linux to provide this.
That all said, I’m pretty sure KDE does provide GUI tools for when this kind of scenario arises.
I mentioned they aren't the only UNIX where this isn't a problem.
Also we cannot use the reasoning Linux means distributions, instead of using GNU/Linux for it, and other times Linux is just the kernel, depending on convenience.
> I mentioned they aren't the only UNIX where this isn't a problem.
You did. But without examples, it’s a meaningless throwaway.
Personally, I can’t say I’ve pushed many other UNIX desktops that far that they’ve OOMed. Except maybe OpenSolaris and FreeBSD, neither of which faired much better. But they’d also suffer from the same kind of DE development issues I described for Linux.
> Also we cannot use the reasoning Linux means distributions, instead of using GNU/Linux for it, and other times Linux is just the kernel, depending on convenience.
I’m not saying it for convenience. I’m saying it because it matters in this specific situation.
Surely you must understand that the reason Windows and macOS have arguably better GUI support for OOM is because the same company that owns the kernel also writes the frontend too? It’s hardly rocket science as a concept.
As I said in my previous comment, there are ways Linux (the kernel) could work with DEs to bridge that gap. But at present, they’re entirely separate concerns.
> Who says it needs to be desktop for proper memory management, servers and embedded can crash willy-nilly with OOM?
The context of the thread is about the UX.
And the OS crashing isn’t a better option than an OOM killer. Frankly, there’s no good automated options. That’s why modern desktop OSs present users with a warning. And hence why we are talking about GUIs.
> In any case, Xenix, DG/UX, SunOS/Solaris, Aix, HP-UX, z/OS UNIX, NeXTSTEP.
Listing OSs doesn’t explain how they handle things better.
For example you’ve cited SunOS and I already said previously, I’ve seen SunOS completely die on its arse when OOM.
Windows will aggressively page out as you approach RAM usage, MacOS does the same but based on activity iirc. On Linux if you just set up paging (forgot how i did this on arch) you won't have any issues.
Windows does freeze and become unstable, but it's usually because of some ring 0 errors not getting trapped appropriately.
But Windows has always handled both OOM and out of disk space very well. The system will be extremely sluggish, but it typically continues operating.
The Linux design is to keep using memory, then push to swap, and then when you OOM you hard lock. The built in kernel OOM module can miss when RAM usage spikes rapidly. You can enable OOM monitors like systemd-oomd or earlyoom, but they do not run by default, and their behavior is to term the offending process.
The thing about Windows is that when you request memory, you're only granted memory if it can be guaranteed in the first place. The application will get a "not enough memory" error. On Linux, you're permitted to request more memory than the system actually has, with the idea that you won't actually use that much memory. It's optimistic that way. But if you do use it, then you're screwed.
Note that the fork()/execve() semantics of Linux pretty much depend on memory overcommit. You may get weird OOM crashes when there is seemingly no memory pressure, if you turn overcommit off.
In the short time between fork() and execve(), the new process duplicates the entire virtual memory of the old one. None of this is physically allocated due to copy-on-write, but still all counts as virtual memory.
Has anybody tried a compromise? Like, the program forks and overcommit is ignored for it for a 1 second grace period? I know it would be more complicated than that, but something along those general lines.
Doesn't Windows, like macOS, automatically expand swap? You can't run out of memory nearly as quickly as you can on Linux. And at least on Windows I always figured this is why it felt so sluggish under load--it ends up paging alot. Not sure why macOS has always felt more consistent; perhaps the OS and major apps are less gratuitous memory hogs?
If you're referring to Apple Silicon, my intuition is that the SoC design allows for higher bandwidth between SSD and RAM which reduces the performance impact of paging. I might be totally wrong though
In Windows 9x (95/98/ME) the swap file could expand indefinitely, with no mechanism to limit its size.
Under Windows NT 4.0, Win2k, or WinXP and on, the page file has a built-in size limit that you can set. You can indirectly force that minimum size to increase if you enable full memory dump capture in the event of a crash (it must be larger than main memory to do that) and that was really easy to do so I think a lot of people made that mistake. Even on the modern Windows, if you have a crash then the system will automatically enable full memory dumps for the next 4 weeks hoping to be able to debug the second crash. That can also result in the minimum size increasing beyond the maximum you set, but in neither case does it grow indefinitely.
Eventually the OS will start returning an error that it's out of memory or out of virtual memory. But if the application you're running can handle that gracefully, then it may not even crash.
I think Windows is sluggish now in part because it's about 5 or 6 layers of abstractions piled on top of each other between the application and the hardware, and also because Microsoft keeps jamming services and side projects onto you. It's been a long time since Microsoft has adopted the "shut up and get out of the user's way" approach to OS design. Now it loves to distract you from the program you're working in.
Solaris family kernels move this problem to run/allocation time, instead of waiting for the system to become unusable before trying to do something about it. Kinda nice
Anyways, you've unlocked an old memory of mine, where Windows would crap itself due to low RAM and the fonts and interface elements would render in a "compatible" way, like using "fixedsys" font everywhere.
It happens sometimes, even in the present day: almost certainly it means something leaked and hit 9999 GDI objects (or user objects?) -- you can add those columns in task manager. Killing the offending process should fix it
Have you tried setting up systemd-oomd [1]? It works by killing the offending process (hopefully) before a true OOM.
When I run in a frozen system because of RAM, I'm a happy user of the Magic SysRq 'f' [2], it bypasses most kernel scheduling so it works even on frozen systems
Windows does freeze when a Chromium-based browser decides to hug the pc.
Chrome/Vivaldi are the only browsers where, when i have a long HN page open, when i resize it, my laptop freezes for like a minute. Not a full freeze, the mouse still moves, but like it's the year 1999 with a refresh rate of 4 seconds.
I stopped using swap almost a quarter of century ago, and since then I have not seen a PC with Linux freezing because of full RAM.
Before the recent increase in RAM prices, it was not difficult to have a generous amount of RAM in a PC, that would make very unlikely the out-of-memory situations.
When OOM really happened, it was much better to become aware of this immediately, and reduce the number of concurrent threads for a running job, or whatever caused that, instead of having reduced performance or even freezing.
Unfortunately Linux makes it difficult to fully get away from swapping because any file-backed page is eligible to be evicted, and even if you have "disabled swap" the kernel will cheerfully page-out your program, even if it needs to be immediately paged in again to continue. The only way to stop it from doing so is to remap all executable mappings into anonymous memory, a thing which is possible but which the loader and linker do not make easy.
This type of thrashing was the most common OOM issue I saw on my servers, the system would effectively deadlock until I pulled the plug. In my setup this seemed to be caused by slow-ish disks, and using a small amount of zram made things more CPU-bound / less IO-bound. I haven’t seen the problem since, the OOM killer does what I expect.
My PC doesn't have swap and it freezes for about 20 s when I run out of memory until the OOM killer finishes its job, usually by killing Code or more rarely Firefox. I'll take this over the machine swapping so slowly that I need to hard reset anyway, but I'd appreciate it if the OS could stay more responsive when it happens.
Mannnnn.... I've done similar and have NOT had the same experience. The problems I've had are usually because I'm loading stuff in a weird disk-memory-cpu trifecta and they all really want to push the remaining 30GB stuff into a database. Then it's off to ctrl-alt-f5 to wait for gnome to crash so I can auth in and kill the ravaging process.
It depends on the memory conditions you end up in an OOM with. In some cases Linux starts to just trash swap and never kills anything because it thinks it can free a few pages at a sufficient tempo. This is where turning off swap helps by just forcing the OOM killer to act sooner. On other devices it can make things worse. In the end it's still all terribly handled by Linux.
That's because windows always uses swap by default, and your system will freeze if you run out of swap which is usually 33 gigabytes, or ir your drive and ram are full.
Configure overcommit_memory, overcommit_ratio, admin|user_reserve_kbytes, min_free_kbytes, oom_kill_allocating_task, oom_score_adj; if you're on a dev box make it such that even if you approach memory limits to just instantly start killing processes rather than swapping. It's not a solution for servers, but for dev environments it's basically perfect.
So normally if you overcommit it might be undefined behavior if you not have enough memory. However if everything is fine pods can preempt based on prio to new nodes.
With the right settings only the faulty app might fail. Without it it can be a real pain to find the rouges
I had this symptom due to having an SSD without a RAM cache, which caused the latency of disk access to rise to seconds under heavy paging, effectively freezing the computer. Enabling the kyber IO scheduler, tuned for responsiveness over throughput, solved it for me.
A lot of cargo-culted sysctl tuning blogs will get you there... Linux doesn't have that problem, your chosen distro/config does. E.g., CachyOS is tuned for absolute minimum latency and assumes you won't be heavily multitasking. Using that as a workstation requires adjustments.
This seems to always be explained in a confusing and sometimes wrong manner, so I may as well also try to explain it in a confusing and sometimes wrong manner too.
Both Windows and Linux allow applications to map memory in an uncommitted state. However, in Linux, uncommitted memory can magically become committed by simply trying to use it, whereas on Windows you have to first actually commit it with a separate API call.
You might wonder why you would even bother allocating uncommitted memory on Windows if you have to explicitly commit it later. Simple: just to reserve a contiguous slice of address space for later.
The consequence of this is that on Windows an allocation failure usually occurs at an API call where it can return a failure, and indeed does. On Linux though, an allocation can fail during a page fault that is entirely transparent to the application. So instead, the default behavior is to allow overcommit, where applications are allowed to commit more memory than the system actually has, under the assumption that in many cases it won't all be in use at once. Instead of an application hitting an error or crashing on a failed allocation, if there is no memory left, a process simply hangs.
Here's my take on why this is the way it is:
- Because programs are written without any knowledge of what memory pages are committed, applications will allocate physical memory pages transparently even if they didn't malloc or mmap anything. Because of that, when an allocation fails under memory pressure, it's pretty likely the first page to stall on overcommit isn't really related to the actual memory pressure.
- To try to mitigate this, during high memory pressure situations, mechanisms like the OOM killer have to score tasks based on several factors to try to guess which ones are causing the problem.
- When you have something like make -j running, it is pretty easy for it to choose Firefox instead of the 200 instances of Clang, since individually none of those Clang instances are really using much memory. This, of course, doesn't fix the bleeding, it just kills your browser. There are ways to mitigate this problem, but they are not often implemented and nothing will ever be perfect.
One may wonder if it is worth the trouble... Probably not, but it does have its advantages, particularly when it comes to databases and caches, which can aggressively reserve memory and mmap files and heavily lean on demand paging for memory management. It's just that in this case, managing memory pressure becomes somewhat heuristics-based, and tools like cgroups are often desired to isolate failure domains and control memory allocations in production systems.
This is a tough problem to solve. Many improvements to the Linux OOM situation have been made, from simply fixing problems that made the wedging even worse, to tweaking the OOM scoring, to usermode daemons like systemd-oomd that try to catch memory pressure earlier. However, so far this is a solidly unsolved problem - the same exact wedging is still possible today.
As hopeless as it seems, I am still open-minded here. The Linux desktop moves slow, but nonetheless it seems pretty good at overcoming challenging obstacles.
Just last night I was debugging some batch job on a Linux box making the machine go unresponsive even though there's no swap. Python process using all the RAM and the OOM killer doesn't stop it, instead the killer itself gets live-locked, I'm guessing on minor page faults.
When your OOM killer isn't aggressive enough, not having swap makes you go fully unresponsive sooner. You still get into a situation where the OS keeps having to purge pages and read them again from disk. But without swap one hand is tied behind its back and it can only purge code pages, not data pages.
This gripe (that I literally crashed out on recently after maining Fedora for the first time) is top of my list for annoyance/frustration with Linux as a desktop.
I now have btop permanently running in a Terminal and I habitually Alt-Tab to it to watch my swap. As soon as my free memory gets less than 1 Gb I kill any Firefox tabs (the memory black hole).
I know this won't be popular, but Linux people have some kind of Stockholm Syndrome on this. They tell you all of the obscure configuration files you have to edit, the additional programs you install. They insist that just having the OS kill a random program to save the DE is a good plan. It's the typical "your holding it wrong" mantra.
I run both Windows and MacOS and have done so for decades. I cannot remember the last time I had a hard freeze like I get on Fedora. On the exact same box running Windows I was able to have VS Code, Chrome with 15+ tabs, terminals, docker, etc. all running with zero issues. On Fedora I have a handful of Terminals and Firefox with less than ten tabs and nothing else running and I have to watch btop like a hawk.
The excuses people make are mind bending, and the shifting of blame from the system onto the user is shameful.
The Linux Desktop people have been singing the song of "works for me" for decades, which is as long as they've been unable to crack double digit users.
I mean, I'm to blame at the moment as well. I'm putting up with it because I don't want to use Windows, mostly because I'm doing a lot of agent dev and I saw them struggle in a Windows environment. And there is a little spite driving me since Windows 11 is full of ads and I felt like it was pushed on me when I wanted to stay on Windows 10.
The vast majority of my computer use has been on MacOS for the last 15+ years, since a Macbook Pro is the standard issue dev environment at every company I have worked at. I had a Windows desktop entirely for gaming that I've repurposed to be a Linux dev box. I genuinely thought all of the hype about Linux desktop that I have seen lately was an indication that it was finally ready for general use.
Now that I am vigilant watching RAM I haven't had the freezes. And I refuse to let the OS roll the dice on what program it wants to kill in its last ditch attempt to save itself from a problem it shouldn't have in the first place. And since Firefox seems to demand 800MB-1GB per tab process, it is a reliable scheme to just close heavy tabs when I see memory dip into the danger zone.
I just wish RAM wasn't so expensive right now, since I'd buy as much as I could just to avoid this pain.
If you have an extra NVME slot you could look at picking up an Optane 800P drive. Extremely low latency reads and writes, which makes it ideal for swap. They're available for ~$100 on eBay.
Who should do it for you? If anyone wants they can submit a patch for it.
The Linux world is a different kind of beast. I too went through the same motions as many newbies but at some point I just stopped complaining and started fixing what I didn't like.
If you start screaming for corporate money which should fix things that's another can of worms, corporate doesn't really do good will, they are usually buying control whenever they pour money into a project. Which can become tricky for Linux.
"Works for me and if it doesn't work for you, just fix it." Great, I should learn to become a kernel and device driver developer just to have a working desktop environment with non-tearing windowed video (still somehow an issue)?
What a fucking joke lmao. I'll just keep buying Macs.
I honestly don't really want you to use it. You want to use it, not me. I am not selling anything, Linus is not selling anything. The kernel is free my dude.
Stay on macos. Or windows. Whatever. Have fun.
edit: Just to be clear, my point is about entitled people demanding things like they paid for it. You do that in a restaurant, where you pay for the service, you expect certain things. I understand if something isn't working it isn't great, but spitting on it while it's free because you feel entitled is the wrong attitude. That's what a lot of people getting into Linux get wrong, they feel entitled to it, while it's free. You need to adjust your expectations.
Our servers run Ubuntu. In practice it doesn't make that much of a difference I assume, but why diverge at all if I don't have to. I have separate laptops for work and personal anyway.
I'm ashamed to daily run Ubuntu at home with Chrome (a dozen tabs), Firefox (literally hundreds of tabs, but only a few active), Thunderbird, RSSGuard, whatever I'm currently working with, and sometimes Steam and a AAA game on top of that. Never felt the need to close anything, but I must confess I have 64 Gb of RAM.
At work I ran the same Firefox profile in Windows, with 16 GB of RAM, it used to eat RAM more eagerly, and restarting only helped for a few hours. I never understood why, but GMail and WhatsApp tabs were usually the biggest memory drains.
I'm just happy I don't have to deal with that anymore, since we were moved to iMacs at work, and the same Firefox profile gives no trouble with 24 Gb or RAM. WhatsApp is still the hungriest tab, but it stabilizes. Maybe there's a sweet spot between which tabs are opened and how much RAM do you have?
I’ve found a few idle YouTube tabs is enough to bring macOS to its knees. Though I do wonder if uBlock Origin is the hidden cause rather than Firefox itself.
YouTube is absolutely atrocious. And it never stops trying to load AD content, even if the user is Premium. So uBlock Origin can sit at 1500 blocked requests in less than an hour.
> I now have btop permanently running in a Terminal and I habitually Alt-Tab to it to watch my swap. As soon as my free memory gets less than 1 Gb I kill any Firefox tabs (the memory black hole).
This is what nohang [0] does. There is no reason to do that by hand. Obscure configuration files to edit?
Yeah it's kind of annoying that it's not built in to most desktop distros, but... literally `apt install nohang; systemctl enable --now nohang.service`. Hardly that difficult.
So your saying the problem I am reporting is not a problem because it is obviously my responsibility to research the dozen available workarounds listed on internet forums, figure out which ones are not brand new footguns or genuine security risks, install sudo level programs from github that as the Warnings section on that repo confess "the daemon runs with super-user privileges and has full access to all private memory of all processes and sensitive user data;"
And then I scroll down and see the install instructions for my distro:
> To install on Fedora:
>
> Orphaned for 6+ weeks, not available.
It's funny I hear people slag npm all the time for developers just YOLO installing any dependency for the webservers they run.
But hey, just sudo install this package! What could go wrong? You only use your desktop to access your online banking, etc. Forget the trust mechanism of your distro and just live a little!
Which OS are you using? systemd-oomd (or similar) afaik is enabled in many modern releases and fixes it. The default settings (on Arch) make it kill stuff when your CPU is spending 60% of time on reclaiming memory over some time period. So you'll get 20 seconds of lag, then it is fixed.
By killing a program, yes, but how else do you expect it to work? If your RAM is not enough you have to kill stuff. And I really doubt you mean to say that Windows is _more RAM efficient_ than Linux, because that's kind of absurd. Maybe Firefox is less RAM efficient than Chrome.
Edit: MacOS I wouldn't doubt being better in RAM management than the others because they have been selling their devices with ridiculously low amounts of RAM for decades.
The ideal would be to go into swap without becoming unresponsive, so that the user can quit a program of their choice to free memory. If you really want to make an issue out of it, do something like pre-X macOS, where it would provide a warning when you are getting close to running out of memory, so that a user might have the chance to do something before the current process starts getting swapped out.
> The ideal would be to go into swap without becoming unresponsive, so that the user can quit a program of their choice to free memory.
Linux already does this. The issue is when your swap is full too, or you’ve exhausted your RAM at such a rapid pace that the disk IO cannot keep up with the read and writes to swap.
I’ve seen Windows choke under the same conditions too.
> do something like pre-X macOS, where it would provide a warning when you are getting close to running out of memory, so that a user might have the chance to do something before the current process starts getting swapped out.
This is where the desktop and the kernel being entirely separate organisations (not even projects, but literal organisations) makes things a touch harder. And those DEs aren’t even Linux specific either. So any tooling would have to portable (which, in fairness, should be possible with POSIX in this particular scenario).
I think KDE does provide tools here for gracefully handling memory constraints. But that’s just one desktop environment of many.
>> do something like pre-X macOS, where it would provide a warning when you are getting close to running out of memory
And I've run out of memory while compiling packages (I'm looking at you, composable-kernel) on Gentoo. It's not like it can notify me and I close things in the milliseconds it takes for a few more GCC processes to spawn and allocate and fault all the remaining memory.
> [...] or you’ve exhausted your RAM at such a rapid pace that the disk IO cannot keep up with the read and writes to swap.
I don't know what that means. When your system is in such extreme memory pressure, it comes to a crawl, exactly because your memory use can't outpace your IO. There's no 'cannot keep up' state.
(1) While there's plenty of headroom, you might still want to swap out 'cold' memory, so you can keep more files cached in RAM.
(2) When you are (nearly) out of memory, you need to evict to swap so that you don't run out-of-memory.
Swap mostly helps you, when you have plenty of cold memory that are pretty much never accessed again. Or when you have applications that you switch, but that you don't use concurrently.
When your active working set is bigger than your RAM, then, yes, swap is slow.
macOS shows a modal with a list of offending applications and how much RAM they're using. Some of those applications are "paused", marked in red, some are "not responding", and some are responding normally.
You can pick applications from that list and choose between "Resume" (for "paused" applications) or "Force Quit".
Linux can, and does, prevent a program to claim any new memory. Like literally malloc() returns null. How program reacts to that is up to the program.
Also I know linux has `cgroup_enable=memory` setting in /etc/default/grub, that's for Docker/K8S to be able to watch over containers RAM. It's not enabled by default as it slows down OS a little. Don't know if it's related to my first point.
Possibly an unpopular opinion on this forum, but virtual memory has always been one of the strengths of the NT kernel. (And integration between the cache manager, virtual memory manager, file system (NTFS), overlapped I/O, completion ports, and threading. It’s a very robust and performant foundation when leveraged correctly.)
Same problem I've had. On a 24GB MacBook Air, I can have a VMWare VM, Lima and an IDE and a bunch of Chrome tabs loaded and the system is fluid.
Meanwhile if I try something similar on a 32GB Linux system, it will sputter or outright freeze. Suspend the system at 16GB used and on wakeup it takes 15 seconds to get back to full responsiveness.
Yes, some of this can be addressed with swappiness settings, however that behavior isn't partically clear and doesn't take into account that depending on your workflow on a given day (e.g. coding then video editing) different swap behaviors may be more optimal. Nor does it help for people who are trying to get good productivity out of a 16GB or lower system.
The one thing that has helped is moving to Brave Origin. Whether that's loading less JS ad SDKs or general optimization, it does seem to use a lot less ram than Edge/Chrome.
Same. What I find pretty incredible is that we've had single core scheduling, then multi-core CPUs, page files, etc., all that stuff as computers progressed through the decades, just for a single app freezing the entire system in 2026.
My work PC, at least with all of the corporate fuckware installed...
I tend to see two behaviors when moving between work and home.
Either:
A - It won't go out of sleep, it basically forces a reboot.
Or,
B - It leaks memory that I can't free without a hard reboot, to where even copy-pasting becomes absolutely jank (my favorite nasty behavior; it double-processes paste operations, not fun when dealing with credentials...)
I do have 'local admin' and while I only checked a couple of times, both times it looked like even then I couldn't do a force kill/restart of whatever was jacking the thing up.
Meanwhile my wife's current 'my old' PC with the same amount of RAM as work laptop and probably lower/older CPU spec could last between forced windows updates without a reboot and a lot more general churn.
-----
If there's one reason I'm bringing this up, it's because I know my work PC has a lot more 'kernel' type management stuff installed...
And yeah, workplace people in charge of that stuff gives that same weird 'blame the user' focus as what you describe.
Possibly not a solution for your use case, but I use earlyoom[0]. When memory crosses a threshold value, it will kill the worst offender. Has some knobs to tweak: what threshold level you want, which programs to keep safe, which to prioritize, etc. Never had to touch the configuration, out of the box has been sufficient for me.
I use it when I am doing exploratory work and might materialize something too big. It has always killed the Python process or terminal program which did something silly.
> I run both Windows and MacOS and have done so for decades. I cannot remember the last time I had a hard freeze like I get on Fedora.
I've gotten plenty of freezes and even more slowdowns on my MacBook Air M3 when using lots of RAM. But I'm using a lot more than 15 tabs in Firefox, and quiet a few VS Code instances, too.
I agree that we should be improving Linux instead of making excuses.
Linux might freeze from occasional OOMs, but Windows is sluggish and unresponsive even when RAM is far from full with all its unnecessary background processes. That's the difference for me.
Wait, how does literally no one in this thread mention swap?
IME, many default installations in recent years don't configure swap by default. Dedicate a few tens-of-gigabytes on your hard drive and memory overflows there instead of borking.
It only postpones the inevitable. It doean't actually fix it. I went for a 64 GB swapfile to hopefully not have to worry about it any more, only to find my swapfile full after a few days anyway, even when configuring swappiness to a minimum. Shit just piles up for no apparent reason. I'm sure someone smarter than me can diagnose the problem, but I just decided to reduce the swapfile size to a more sensible value and reboot every now and then. I should probably try Zram or whatever to see if that mitigates the problem.
Most applications these days are very good about that; you likely have just a single app that you use regularly that leaks. If you kill the process the memory will be recovered, save for some allocations the kernel maintains.
"top" or maybe "slabtop" will find the offending, very very broken, program that's leaking memory like crazy, making your system doomed no matter what you do or how much physical memory you install.
Sometimes I use swap, but >half the time I'm intentionally disabling it so the system doesn't go into livelock. I'd rather have an OOM than an unresponsive system. Granted this is usually on some headless Linux machine running batch jobs, not my PC.
it depends on what you want to do. in my case I only want disk swap to be used for hibernation. I calibrated the system to make do with the very generous zram compression ratio of 3 and if it doesn't work out a userspace OOM killer will fix it. zswap will be polluting your SSD all the time.
Agreed. I've run into this way too many times, and now with a super aggressive oom killer config it still slows down until vscode or firefox just get closed. I wish it was similar to how it seems to be on MacOS from what I see on friends' screens, where theres some kind of hypervisor running that pops up a nice responsive GUI window saying "hey, xyz is using too much memory, close it or something else" while not locking up. But maybe that's also just the grass being greener on the other side?
Defaults for swapping and paging unfortunately aren't great. But it's easy to adjust. There's lots of information and fwiw, talking with an LLM I found that they ended up suggesting similar things to what I set for my system (pre LLMs). Of course, don't blindly trust, but it was one thing I did to test them (and expected to be relatively good considering the amount of available information, though hard to sort through, especially for non experts). On the other side, many modern distros are getting better defaults. The difficulty here is that the right values are wrong for most setups and Linux is targeting a much broader range of hardware than windows. Though that's where distros come in
I'm on Ubuntu 26.04 and I feel it's made substantial improvements in that regard compared to 24.04. It does kill the offending process (well, its best guess at it) before it freezes for good.
This post focuses on performance, but what about crashing due to apps seeing no RAM available to allocate once VRAM is filled up?
I remember having this problem when I used an NVIDIA 750 Ti with 2 GB of memory. Just opening a few Firefox or Ghostty terminals (both are GPU accelerated) would result in those apps crashing or not being able to render their contents. Sometimes the compositor would crash or become unstable too (kwin, niri, etc.). I had to reboot every few hours. Complete system instability.
This problem only happened with Wayland, when I switched to X windows these problems went away.
Since then I switched to an AMD RX 480 (8 GB) GPU and never had an issue here with Wayland, if GPU memory gets full, system memory gets seamlessly used instead. It felt like with the NVIDIA card, it never allocated back to system memory if no VRAM was available. This was only a problem on Linux since that same NVIDIA card was fine in Windows for 10+ years with zero issues related to instability around VRAM allocation.
Tons of people reported the same issue on Linux on NVIDIA's forums for years.
I gave myself 32GB of swap since SSDs are quite fast and the pages sent to disk seem to intelligently be rarely referenced ones. Feels amazing to run 4 VMs and 12 Claude Code instances on my mid tier hardware
Your last sentence sums it all: AMD is the only serious option on Linux. Not perfect, but far better than Nvidia, thanks to their open drivers that anyone (and especially Valve) can improve.
It's got to be an edge case for Wayland to be used with a 2GB VRAM card? Targeting recent cards and letting people using older hardware install x-windows, or whatever, seems sensible.
You say (in your blog post) "Linux" didn't work well so you reverted to "Windows", but as is so often the case you buried the distro/version info. You don't seem to have been using Win11 (later you mention Win7 & Win10Pro) ... So it's cutting edge software didn't work on my 2014 system, but older software did. The Linux/Windows aspects aren't particularly relevant AFAICT.
Nvidia refused for many years to properly support Linux, whilst AMD have supported it. So no surprise if a Nvidia card didn't work well/stably but an AMD one did.
> You say (in your blog post) "Linux" didn't work well so you reverted to "Windows", but as is so often the case you buried the distro/version info.
Not sure what you mean. The post directly mentions I'm using Arch and was using the official drivers from NVIDIA (the latest at the time). I also didn't switch back to Windows. Although now the experience is much better with an AMD card.
> It's got to be an edge case for Wayland to be used with a 2GB VRAM card?
I had thought that one of the biggest "selling" points of Linux is the ability to extend the useful lifetime of older hardware. And if the distros are going to turn away from X11, well.
Yes, it has been a selling point that Linux can revive old hardware. It still is. But when a whole new desktop rendering system is implemented, just hitting the mainstream in the last year or so (for me on [K]Ubuntu) then I think you should expect backwards support of a 12 year old system to be limited. That's kinda my bugbear, they say (paraphrasing) 'linux didn't work' but they're comparing older Windows systems with newer Linux systems. Yes, ultimately the story was positive for Linux, but there's an inherent bias IMO where Linux is put down if it doesn't do everything better than every version of Windows for every piece of hardware. If you're using the very latest hardware, or older hardware, you're going to have to get your hand dirty choosing a distro or configuring an install (or possibly compiling drivers).
Win11 wouldn't work on 2 year old hardware (no compatible TPM was the complaint, but I worked around it with hacks that MS didn't publicise). I should probably have used that as a chance to move that family member to a friendly Linux distro ...
Just search KWin Nvidia memory leak or Wayland Nvidia memory leak or kwin wayland Nvidia memory leak. It’s constantly issues and I remain convinced some intermixing of the two + monitors is at fault.
Such great work!
Happens quiet regularly I hit a game just realizing seconds later I haven't shut down my local LLM yet. At least punishment will be less harsh. Lets see, maybe some games in window mode will just work fine.
I don't know, I guess some rule about the capitalization of acronyms. But I have seen it written as VRAM since the early '90s when it meant dual-ported video RAM. I think this is the first time I see it written as vRAM with a lowercase v.
Generally in our IT jargon writing "vRAM" would mean some sort of virtualized RAM thing, while VRAM shows the V is part of the initialism and thus VIDEO RANDOM ACCESS MEMORY. Nothing formal, just informal usage.
Honestly, swap thrashing on low vRAM setups has been painful forever. Glad to see Linux kernel optimizations targeting memory pressure where it actually hurts.
A curious thing happens in my Ubuntu 24 system, maybe that's just the normal OoM killer algo but no matter what process is hogging, Firefox is killed. Chrome eating too much memory? Firefox dies. Too many apps open? Firefox dies. Its funny.
The halting problem asks you to conclusively answer if a arbitrary program will stop running in the future. This asks to stop running a given program when stats seem bad currently. I don't really see what the relation is supposed to be.
I see all the comments about how Windows and Linux handle OOM, but I suppose I'll also mention macOS.
On my M4 Max Mac Studio if I try to load too big of an AI model with protections off, the desktop starts glitching back and forth between the past few hundred frames. It looks bad when it happens but CtrlC still works to kill llama.cpp and if you were using LM Studio, SSH also works. Once you kill the offending process, or if macOS does it for you, the desktop comes back immediately.
I have a Z1 Extreme that has 16GB shared between the CPU and GPU (what AMD calls the APU). I've noticed that mangohud, the SteamOS utility that shows performance diagnostics, often reports RAM + VRAM numbers that exceed 16GB. I've been curious as to why. I presume it has to do with compression or something.
This seems like an impressive improvement, and I'm looking forward to it eventually being upstreamed. Though unfortunately I'm on Nvidia right now, and have been struggling with vram. They don't seem to support any kind of paging at all.
I am curious about the bit on virtual memory fragmentation. Would it make sense for the kernel to occasionally defragment that memory in place? I assume that would create a noticeable hitch, but might improve performance otherwise and allow for some applications to just fit in.
260 comments
[ 0.26 ms ] story [ 8.1 ms ] thread> Not only does the display hardware like scanned-out images to be in VRAM, it also completely skips past the GPU’s virtual memory architecture and works with physical addresses exclusively.
Well there's your problem. Only so smart your memory management can be when you have to pay the cost of doing it manually. Although presumably this only applies to a small fraction of the VRAM?
They did mention they saw 4GiB of eviction for a single 32MiB scan out image.
So while I would call the image allocation small, it seems to cause an avalanche of evictions. Amplified by the fact that each frame has one of these images, though I expect subsequent frames might have a better chance of already fitting into evicted space.
What I don't exactly understand is: doesn't it make sense to always reserve the contiguous physical memory for this case and not allow anything else to be put in it?
What does this mean for compute workloads? Specifically, LLM inference.
Does it mean anything at all, or is this purely a games-thing?
I guess an LRU with priority would handle VRAM for games pretty decently without going getting too application specific.
What about VRAM to Disk specifically NVME, would direct to disk be feasible for large workloads, I know it is used for streaming in assets directly via. PCIE, but i wonder how the performance would be on compute workloads running with NVME as a swap for GPU VRAM.
4x as slow in absolute best case, NVMe drives have 4 PCIe lanes usually
Meanwhile in the Windows world, users hate updates... Like I genuinely can't think of a single instance that made users exclaim, "oh boy I just can't wait for the next Patch Tuesday!".
I cant wait for the number one gripe of all my users to finally be catered for.
Now, of course, it still doesn't work with a touchpad, but baby steps, right?
But this has worked in KDE since version 3, like, 20 years ago? And also with pasystray running under i3, since before win11 was a thing. They both support scrolling with the touchpad, too.
I drove me up a wall not being able to do this when I had to use Windows at work.
As for your MRU layout thing, I'm not sure I'd like it. While I like MRU for alt-tab, i like my not-so-often-used functions to be predictable. For this specific case, I always use us-macintosh, so I'd like to be sure that if I press the shortcut X times I'll get Y layout, without having to remember the order in which I used them. Now, to be fair, in my case I only have US-Mac available since it does everything I need.
I haven't touched MS in this decade, but I swear that used to be a thing at least through 7...
As the other comment says, I suspect it to do with GPU as the built-in displays are usually driven by intel and external displays are always driven by my Nvidia 4090.
But surprisingly, when I drive my e-paper monitor using original Jetson Nano (Ubuntu 20.04 ESM / Nvidia Maxwell); the fractional scaling is quite good perhaps even better than macOS.
https://wayland.app/protocols/fractional-scale-v1
Also, I wouldn't hold the rendering quality of macOS as a high standard at all, since all it does is render at a multiple and then downsample, wasting energy and introducing blurriness.
Gnome >= 47 is fine, you will need to do:
`gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer', 'xwayland-native-scaling']" ` That stops being necessary as some point around 49/50 I think.
Meanwhile 4 separate windows pcs have not had problems with updates for at least 6 years
If you want a stable Linux experience though, pick something like Fedora, or even better, pick an immutable distro like Aurora or Bazzite, if you really, really care about stability.
That said, even some fixed release linux distros break things almost every release. That's because Windows updates is just windows base + kernel + drivers, while most linux updates are everything all at once with no distinction between base system and user applications. Plus windows and macOS present certain things more or less as a blackbox to user - I never had broken DE and keyboard layout switcher during windows update, while every ubuntu update used to break everything for me.
That said, I do occasionally deal with quirks due to the rolling distro nature of Arch. The benefits far outweigh the occasional upgrade pain IMO though.
I don't even mind making an MS account, I have one, just don't want Windows using that account and thus constantly freaking out about whatever bs like "you haven't enabled 8-factor auth."
Majority of Windows users are not programming enthusiasts and just don't care.
OS works. Programs continue to work. Good. Keep it that way.
This goes especially when used with more powerful GPUs from nVidia - so perhaps arrogance here isn't warranted?
The commits they authored in [0] match and all of what they have mentioned in the article and this is how you know the author knows exactly what they are talking about and have explained it well.
Puts all the vibe-coders, and the so-called "AI Engineers" to shame.
[0] https://gitlab.freedesktop.org/pixelcluster/kernel/-/commits...
To quote myself from another comment: «I am inclined to rather point out remarkable things than unremarkable things. There seems to be relatively many trans people in this niche.» That's all. I didn't know this required a trigger warning.
I didn't know about them but you do. Can I sell them to you?The issue of gender identity is clearly irrelevant and off-topic here and your other responses make it obvious that you intend to provoke by introducing it; people asked you entirely fair questions about your motivations and you responded by questioning their psychological stability and making an ironic swipe about "trigger warnings" (this exact phrasing makes it clear you're accustomed to this discourse).
> Please don't use Hacker News for political or ideological battle. It tramples curiosity.
Do you understand that the nature of the group is actually completely irrelevant to my criticism?
Do you see why you would be perceived as being the one who went off topic in the first place?
Game developers often aren't super careful with this stuff, and there could be literal gigabytes of data which isn't used at all.
all these electron apps and invisible virtual machines (looking at you Claude) really don't help.
What’s a decent amount? No idea. Don’t test it :)
The only way to save my file system was to boot into recovery mode and run a verification with Disk Utility, then I was able to remove some files and get my space back.
Docs https://docs.kernel.org/admin-guide/mm/multigen_lru.html#thr...
I don't use Windows much, nor do I care much about that environment, but from what I've seen it seems to keep RAM usage below 100% most of the time. What I do see pretty often is the drive getting stuck at 100% usage instead, which makes the whole system ridiculously unusable anyway.
Thing is that I never know this could happen. As a Windows user I'm used to a system that does stuff in the background without knowing how it works.
My Linux VM's can be broken because I'm in charge and I'm a noob. I kind of love it. I've learned so much from troubleshooting.
We could upgrade the RAM, but we were stingy when we created them, and it has worked well so far, so why waste resources?
Linux uses lazy allocation and overcommits, so what dies isn’t necessarily what you would expect thanks to the OOM killer.
With a fixed swap size windows doesn’t necessarily slow down as it runs out of memory.
YMMV if you don’t have 64 gigs of RAM and a 64 gig swap.
So Apple can build a whole user-controlled GUI workflow around memory exceptions, which Linux cannot.
Maybe what needs to happen is new Linux syscalls + signals for DEs to utilise for building memory exception UIs? But the problem then with that is you’re impacting the portability of those DEs. So some maintainers might still refuse to work with Linux to provide this.
That all said, I’m pretty sure KDE does provide GUI tools for when this kind of scenario arises.
Also we cannot use the reasoning Linux means distributions, instead of using GNU/Linux for it, and other times Linux is just the kernel, depending on convenience.
You did. But without examples, it’s a meaningless throwaway.
Personally, I can’t say I’ve pushed many other UNIX desktops that far that they’ve OOMed. Except maybe OpenSolaris and FreeBSD, neither of which faired much better. But they’d also suffer from the same kind of DE development issues I described for Linux.
> Also we cannot use the reasoning Linux means distributions, instead of using GNU/Linux for it, and other times Linux is just the kernel, depending on convenience.
I’m not saying it for convenience. I’m saying it because it matters in this specific situation.
Surely you must understand that the reason Windows and macOS have arguably better GUI support for OOM is because the same company that owns the kernel also writes the frontend too? It’s hardly rocket science as a concept.
As I said in my previous comment, there are ways Linux (the kernel) could work with DEs to bridge that gap. But at present, they’re entirely separate concerns.
In any case, Xenix, DG/UX, SunOS/Solaris, Aix, HP-UX, z/OS UNIX, NeXTSTEP.
The context of the thread is about the UX.
And the OS crashing isn’t a better option than an OOM killer. Frankly, there’s no good automated options. That’s why modern desktop OSs present users with a warning. And hence why we are talking about GUIs.
> In any case, Xenix, DG/UX, SunOS/Solaris, Aix, HP-UX, z/OS UNIX, NeXTSTEP.
Listing OSs doesn’t explain how they handle things better.
For example you’ve cited SunOS and I already said previously, I’ve seen SunOS completely die on its arse when OOM.
Alas, not one that was made back in 1970, so here we are.
To be clear: I’m not trying to defend Linux here. Just calling out the GPs BS when he said Linux is uniquely bad in POSIX.
But Windows has always handled both OOM and out of disk space very well. The system will be extremely sluggish, but it typically continues operating.
The Linux design is to keep using memory, then push to swap, and then when you OOM you hard lock. The built in kernel OOM module can miss when RAM usage spikes rapidly. You can enable OOM monitors like systemd-oomd or earlyoom, but they do not run by default, and their behavior is to term the offending process.
The thing about Windows is that when you request memory, you're only granted memory if it can be guaranteed in the first place. The application will get a "not enough memory" error. On Linux, you're permitted to request more memory than the system actually has, with the idea that you won't actually use that much memory. It's optimistic that way. But if you do use it, then you're screwed.
sysctl -w vm.overcommit_memory=2 sysctl -w vm.overcommit_ratio=50
It is probably a bad idea to do that though as it will limit the amount of virtual address space an application can use
In the short time between fork() and execve(), the new process duplicates the entire virtual memory of the old one. None of this is physically allocated due to copy-on-write, but still all counts as virtual memory.
Under Windows NT 4.0, Win2k, or WinXP and on, the page file has a built-in size limit that you can set. You can indirectly force that minimum size to increase if you enable full memory dump capture in the event of a crash (it must be larger than main memory to do that) and that was really easy to do so I think a lot of people made that mistake. Even on the modern Windows, if you have a crash then the system will automatically enable full memory dumps for the next 4 weeks hoping to be able to debug the second crash. That can also result in the minimum size increasing beyond the maximum you set, but in neither case does it grow indefinitely.
Eventually the OS will start returning an error that it's out of memory or out of virtual memory. But if the application you're running can handle that gracefully, then it may not even crash.
I think Windows is sluggish now in part because it's about 5 or 6 layers of abstractions piled on top of each other between the application and the hardware, and also because Microsoft keeps jamming services and side projects onto you. It's been a long time since Microsoft has adopted the "shut up and get out of the user's way" approach to OS design. Now it loves to distract you from the program you're working in.
Linux OOM handling is just atrocious.
https://www.kernel.org/doc/html/latest/mm/overcommit-account...
Maybe not in the same way then.
Anyways, you've unlocked an old memory of mine, where Windows would crap itself due to low RAM and the fonts and interface elements would render in a "compatible" way, like using "fixedsys" font everywhere.
But only in veeery old Windows, think Win95B or the like.
When I run in a frozen system because of RAM, I'm a happy user of the Magic SysRq 'f' [2], it bypasses most kernel scheduling so it works even on frozen systems
[1]: https://man.archlinux.org/man/systemd-oomd.8
[2]: https://en.wikipedia.org/wiki/Magic_SysRq_key
Chrome/Vivaldi are the only browsers where, when i have a long HN page open, when i resize it, my laptop freezes for like a minute. Not a full freeze, the mouse still moves, but like it's the year 1999 with a refresh rate of 4 seconds.
But yeah it’s completely ridiculous and frustrating that the default is to just lock up your DE
Before the recent increase in RAM prices, it was not difficult to have a generous amount of RAM in a PC, that would make very unlikely the out-of-memory situations.
When OOM really happened, it was much better to become aware of this immediately, and reduce the number of concurrent threads for a running job, or whatever caused that, instead of having reduced performance or even freezing.
https://kubernetes.io/docs/concepts/scheduling-eviction/pod-...
https://kubernetes.io/docs/concepts/configuration/manage-res...
https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/#... (New feature)
So normally if you overcommit it might be undefined behavior if you not have enough memory. However if everything is fine pods can preempt based on prio to new nodes.
With the right settings only the faulty app might fail. Without it it can be a real pain to find the rouges
Both Windows and Linux allow applications to map memory in an uncommitted state. However, in Linux, uncommitted memory can magically become committed by simply trying to use it, whereas on Windows you have to first actually commit it with a separate API call.
You might wonder why you would even bother allocating uncommitted memory on Windows if you have to explicitly commit it later. Simple: just to reserve a contiguous slice of address space for later.
The consequence of this is that on Windows an allocation failure usually occurs at an API call where it can return a failure, and indeed does. On Linux though, an allocation can fail during a page fault that is entirely transparent to the application. So instead, the default behavior is to allow overcommit, where applications are allowed to commit more memory than the system actually has, under the assumption that in many cases it won't all be in use at once. Instead of an application hitting an error or crashing on a failed allocation, if there is no memory left, a process simply hangs.
Here's my take on why this is the way it is:
- Because programs are written without any knowledge of what memory pages are committed, applications will allocate physical memory pages transparently even if they didn't malloc or mmap anything. Because of that, when an allocation fails under memory pressure, it's pretty likely the first page to stall on overcommit isn't really related to the actual memory pressure.
- To try to mitigate this, during high memory pressure situations, mechanisms like the OOM killer have to score tasks based on several factors to try to guess which ones are causing the problem.
- When you have something like make -j running, it is pretty easy for it to choose Firefox instead of the 200 instances of Clang, since individually none of those Clang instances are really using much memory. This, of course, doesn't fix the bleeding, it just kills your browser. There are ways to mitigate this problem, but they are not often implemented and nothing will ever be perfect.
One may wonder if it is worth the trouble... Probably not, but it does have its advantages, particularly when it comes to databases and caches, which can aggressively reserve memory and mmap files and heavily lean on demand paging for memory management. It's just that in this case, managing memory pressure becomes somewhat heuristics-based, and tools like cgroups are often desired to isolate failure domains and control memory allocations in production systems.
This is a tough problem to solve. Many improvements to the Linux OOM situation have been made, from simply fixing problems that made the wedging even worse, to tweaking the OOM scoring, to usermode daemons like systemd-oomd that try to catch memory pressure earlier. However, so far this is a solidly unsolved problem - the same exact wedging is still possible today.
As hopeless as it seems, I am still open-minded here. The Linux desktop moves slow, but nonetheless it seems pretty good at overcoming challenging obstacles.
When your OOM killer isn't aggressive enough, not having swap makes you go fully unresponsive sooner. You still get into a situation where the OS keeps having to purge pages and read them again from disk. But without swap one hand is tied behind its back and it can only purge code pages, not data pages.
I now have btop permanently running in a Terminal and I habitually Alt-Tab to it to watch my swap. As soon as my free memory gets less than 1 Gb I kill any Firefox tabs (the memory black hole).
I know this won't be popular, but Linux people have some kind of Stockholm Syndrome on this. They tell you all of the obscure configuration files you have to edit, the additional programs you install. They insist that just having the OS kill a random program to save the DE is a good plan. It's the typical "your holding it wrong" mantra.
I run both Windows and MacOS and have done so for decades. I cannot remember the last time I had a hard freeze like I get on Fedora. On the exact same box running Windows I was able to have VS Code, Chrome with 15+ tabs, terminals, docker, etc. all running with zero issues. On Fedora I have a handful of Terminals and Firefox with less than ten tabs and nothing else running and I have to watch btop like a hawk.
The excuses people make are mind bending, and the shifting of blame from the system onto the user is shameful.
The vast majority of my computer use has been on MacOS for the last 15+ years, since a Macbook Pro is the standard issue dev environment at every company I have worked at. I had a Windows desktop entirely for gaming that I've repurposed to be a Linux dev box. I genuinely thought all of the hype about Linux desktop that I have seen lately was an indication that it was finally ready for general use.
Now that I am vigilant watching RAM I haven't had the freezes. And I refuse to let the OS roll the dice on what program it wants to kill in its last ditch attempt to save itself from a problem it shouldn't have in the first place. And since Firefox seems to demand 800MB-1GB per tab process, it is a reliable scheme to just close heavy tabs when I see memory dip into the danger zone.
I just wish RAM wasn't so expensive right now, since I'd buy as much as I could just to avoid this pain.
The Linux world is a different kind of beast. I too went through the same motions as many newbies but at some point I just stopped complaining and started fixing what I didn't like.
If you start screaming for corporate money which should fix things that's another can of worms, corporate doesn't really do good will, they are usually buying control whenever they pour money into a project. Which can become tricky for Linux.
What a fucking joke lmao. I'll just keep buying Macs.
> Who should do it for you?
People who want me to use it.
Stay on macos. Or windows. Whatever. Have fun.
edit: Just to be clear, my point is about entitled people demanding things like they paid for it. You do that in a restaurant, where you pay for the service, you expect certain things. I understand if something isn't working it isn't great, but spitting on it while it's free because you feel entitled is the wrong attitude. That's what a lot of people getting into Linux get wrong, they feel entitled to it, while it's free. You need to adjust your expectations.
> decades, which is as long as they've been unable to crack double digit users
and you say
> I honestly don't really want you to use it.
then you kinda make their point entirely for them.
I'm not even going to bother addressing the rest of the projection in your post. Whew.
Is it acceptable? No. Did it solve my endless crashes that I used to get every day? Yes.
I don't love Ubuntu, I just hate modern Windows and find my ubuntu for work popos for gaming setup to be better for me. Ymmv.
At work I ran the same Firefox profile in Windows, with 16 GB of RAM, it used to eat RAM more eagerly, and restarting only helped for a few hours. I never understood why, but GMail and WhatsApp tabs were usually the biggest memory drains.
I'm just happy I don't have to deal with that anymore, since we were moved to iMacs at work, and the same Firefox profile gives no trouble with 24 Gb or RAM. WhatsApp is still the hungriest tab, but it stabilizes. Maybe there's a sweet spot between which tabs are opened and how much RAM do you have?
I’ve found a few idle YouTube tabs is enough to bring macOS to its knees. Though I do wonder if uBlock Origin is the hidden cause rather than Firefox itself.
This is what nohang [0] does. There is no reason to do that by hand. Obscure configuration files to edit?
Yeah it's kind of annoying that it's not built in to most desktop distros, but... literally `apt install nohang; systemctl enable --now nohang.service`. Hardly that difficult.
[0]: https://github.com/hakavlad/nohang
And then I scroll down and see the install instructions for my distro:
It's funny I hear people slag npm all the time for developers just YOLO installing any dependency for the webservers they run.But hey, just sudo install this package! What could go wrong? You only use your desktop to access your online banking, etc. Forget the trust mechanism of your distro and just live a little!
By killing a program, yes, but how else do you expect it to work? If your RAM is not enough you have to kill stuff. And I really doubt you mean to say that Windows is _more RAM efficient_ than Linux, because that's kind of absurd. Maybe Firefox is less RAM efficient than Chrome.
Edit: MacOS I wouldn't doubt being better in RAM management than the others because they have been selling their devices with ridiculously low amounts of RAM for decades.
Linux already does this. The issue is when your swap is full too, or you’ve exhausted your RAM at such a rapid pace that the disk IO cannot keep up with the read and writes to swap.
I’ve seen Windows choke under the same conditions too.
> do something like pre-X macOS, where it would provide a warning when you are getting close to running out of memory, so that a user might have the chance to do something before the current process starts getting swapped out.
This is where the desktop and the kernel being entirely separate organisations (not even projects, but literal organisations) makes things a touch harder. And those DEs aren’t even Linux specific either. So any tooling would have to portable (which, in fairness, should be possible with POSIX in this particular scenario).
I think KDE does provide tools here for gracefully handling memory constraints. But that’s just one desktop environment of many.
And I've run out of memory while compiling packages (I'm looking at you, composable-kernel) on Gentoo. It's not like it can notify me and I close things in the milliseconds it takes for a few more GCC processes to spawn and allocate and fault all the remaining memory.
I don't know what that means. When your system is in such extreme memory pressure, it comes to a crawl, exactly because your memory use can't outpace your IO. There's no 'cannot keep up' state.
Btw, there's (at least) two sides to swap:
(1) While there's plenty of headroom, you might still want to swap out 'cold' memory, so you can keep more files cached in RAM.
(2) When you are (nearly) out of memory, you need to evict to swap so that you don't run out-of-memory.
Swap mostly helps you, when you have plenty of cold memory that are pretty much never accessed again. Or when you have applications that you switch, but that you don't use concurrently.
When your active working set is bigger than your RAM, then, yes, swap is slow.
You can pick applications from that list and choose between "Resume" (for "paused" applications) or "Force Quit".
At no point the system becomes unresponsive.
That is, if you are lucky enough for the dialog to pop up.
Also I know linux has `cgroup_enable=memory` setting in /etc/default/grub, that's for Docker/K8S to be able to watch over containers RAM. It's not enabled by default as it slows down OS a little. Don't know if it's related to my first point.
Meanwhile if I try something similar on a 32GB Linux system, it will sputter or outright freeze. Suspend the system at 16GB used and on wakeup it takes 15 seconds to get back to full responsiveness.
Yes, some of this can be addressed with swappiness settings, however that behavior isn't partically clear and doesn't take into account that depending on your workflow on a given day (e.g. coding then video editing) different swap behaviors may be more optimal. Nor does it help for people who are trying to get good productivity out of a 16GB or lower system.
The one thing that has helped is moving to Brave Origin. Whether that's loading less JS ad SDKs or general optimization, it does seem to use a lot less ram than Edge/Chrome.
Where did the technology go?
Well, As a bizarre counter thought...
My work PC, at least with all of the corporate fuckware installed...
I tend to see two behaviors when moving between work and home.
Either:
A - It won't go out of sleep, it basically forces a reboot.
Or,
B - It leaks memory that I can't free without a hard reboot, to where even copy-pasting becomes absolutely jank (my favorite nasty behavior; it double-processes paste operations, not fun when dealing with credentials...)
I do have 'local admin' and while I only checked a couple of times, both times it looked like even then I couldn't do a force kill/restart of whatever was jacking the thing up.
Meanwhile my wife's current 'my old' PC with the same amount of RAM as work laptop and probably lower/older CPU spec could last between forced windows updates without a reboot and a lot more general churn.
-----
If there's one reason I'm bringing this up, it's because I know my work PC has a lot more 'kernel' type management stuff installed...
And yeah, workplace people in charge of that stuff gives that same weird 'blame the user' focus as what you describe.
I use it when I am doing exploratory work and might materialize something too big. It has always killed the Python process or terminal program which did something silly.
[0] https://github.com/rfjakob/earlyoom
I've gotten plenty of freezes and even more slowdowns on my MacBook Air M3 when using lots of RAM. But I'm using a lot more than 15 tabs in Firefox, and quiet a few VS Code instances, too.
I agree that we should be improving Linux instead of making excuses.
IME, many default installations in recent years don't configure swap by default. Dedicate a few tens-of-gigabytes on your hard drive and memory overflows there instead of borking.
Most applications these days are very good about that; you likely have just a single app that you use regularly that leaks. If you kill the process the memory will be recovered, save for some allocations the kernel maintains.
Of course I did this when RAM was cheap but one of the best decisions I have ever made in life.
I remember having this problem when I used an NVIDIA 750 Ti with 2 GB of memory. Just opening a few Firefox or Ghostty terminals (both are GPU accelerated) would result in those apps crashing or not being able to render their contents. Sometimes the compositor would crash or become unstable too (kwin, niri, etc.). I had to reboot every few hours. Complete system instability.
This problem only happened with Wayland, when I switched to X windows these problems went away.
Since then I switched to an AMD RX 480 (8 GB) GPU and never had an issue here with Wayland, if GPU memory gets full, system memory gets seamlessly used instead. It felt like with the NVIDIA card, it never allocated back to system memory if no VRAM was available. This was only a problem on Linux since that same NVIDIA card was fine in Windows for 10+ years with zero issues related to instability around VRAM allocation.
Tons of people reported the same issue on Linux on NVIDIA's forums for years.
I wrote a lot more details and recorded demo videos around 9 months ago here https://nickjanetakis.com/blog/gpu-memory-allocation-bugs-wi..., but since then a lot of these things have been resolved from switching over to AMD.
You say (in your blog post) "Linux" didn't work well so you reverted to "Windows", but as is so often the case you buried the distro/version info. You don't seem to have been using Win11 (later you mention Win7 & Win10Pro) ... So it's cutting edge software didn't work on my 2014 system, but older software did. The Linux/Windows aspects aren't particularly relevant AFAICT.
Nvidia refused for many years to properly support Linux, whilst AMD have supported it. So no surprise if a Nvidia card didn't work well/stably but an AMD one did.
Not sure what you mean. The post directly mentions I'm using Arch and was using the official drivers from NVIDIA (the latest at the time). I also didn't switch back to Windows. Although now the experience is much better with an AMD card.
I had thought that one of the biggest "selling" points of Linux is the ability to extend the useful lifetime of older hardware. And if the distros are going to turn away from X11, well.
Win11 wouldn't work on 2 year old hardware (no compatible TPM was the complaint, but I worked around it with hacks that MS didn't publicise). I should probably have used that as a chance to move that family member to a friendly Linux distro ...
Here’s one for example: https://forums.developer.nvidia.com/t/kwin-6-7-x-causes-mass...
Just search KWin Nvidia memory leak or Wayland Nvidia memory leak or kwin wayland Nvidia memory leak. It’s constantly issues and I remain convinced some intermixing of the two + monitors is at fault.
Just be careful where you click
[1]: https://en.wikipedia.org/wiki/Xkill
On my M4 Max Mac Studio if I try to load too big of an AI model with protections off, the desktop starts glitching back and forth between the past few hundred frames. It looks bad when it happens but CtrlC still works to kill llama.cpp and if you were using LM Studio, SSH also works. Once you kill the offending process, or if macOS does it for you, the desktop comes back immediately.
I am curious about the bit on virtual memory fragmentation. Would it make sense for the kernel to occasionally defragment that memory in place? I assume that would create a noticeable hitch, but might improve performance otherwise and allow for some applications to just fit in.