They allude to this in the article but I would emphasize caution when using mode 2 especially if one has already adjusted overcommit ratios as one can prevent forks. Test this in a QA/Perf environment first, also testing the restart of all applications. Load test and do full QA tests before deploying to Production and even then when deploying to production I would just dynamically change the setting via app deployment scripts until confidence is high instead of putting it in the sysctl config files.
I've gone through this exercise in the past on much older kernels which they cover as well and just me personally I ran into less issues by leaving overcommit to 0 and just dropping the overcommit ratio to 0 and setting the oom_score_adj for programs as high as 1000 if I wanted vmscan to leave them alone and of course using the Redhat formulas for setting vm.min_free_kbytes, vm.admin_reserve_kbytes, vm.user_reserve_kbytes. And of course be vigilant in disallowing app owners from using every last bit of memory.
This has bitten me multiple times. The problem I have is that at work we deploy the application (written in Go) and PostgreSQL on the same machine. The backend app allocates a lot of virtual memory, and initially we had overcommit to 0 (heuristic). This caused crashes on big queries in PostgreSQL and we set it to 2. The whole system became a bit unstable because the backend would still allocate a lot of virtual memory and at some point we ran into errors when allocating.
For now, we have overcommit_ratio set to a value that is stable from experience, but there really seems to be no silver lining. Go is very happy to allocate a lot of virtual memory, but so are most managed languages. The best solution would probably be to host the backend and the database on separate servers.
I agree with the blog post's technical contents, but I feel we came across too strong in the title. For Ubicloud as a managed Postgres provider, we use strict memory overcommit. Our experience with operating Postgres at scale taught us that it's better to enable this than going with the defaults.
However, I can see many other scenarios, where using strict memory overcommit would have unanticipated side-effects. That's why Linux doesn't go with strict memory commit as its default.
Nothing worse than memory management on Hyperscaler VMs which do not use Swap :|
Took k8s ages to get Swap support.
We lost something when we accepted that Hyperscalers just tell you to use more moemory. It was shitty 5 years ago and today especially after the ram price increases
Yeah I noticed and was surprised after using ec2 that it didnt really have a sizable swap so it would lock up if I used enough memory importing a db export. Was genuinely surprised.
> Both Windows and macOS do so much better out of the box for essentially any workload.
We test FreeBSD, Linux, macOS, NetBSD, OpenBSD, and Windows in Zig's CI fleet. Of these, Windows is the only OS that we've had to configure with swap double the size of physical RAM to not hit completely unjustifiable OOMs.
By "unjustifiable", I mean that we're not even close to actually running out of physical memory (let alone swap), but the MM seems to be doing a horrible job of making unused memory actually available to processes.
It's possible there's a relevant configuration knob here that we're just not aware of... but the point is, the default behavior does in fact suck.
Mode 0 (Heuristic) is described incorrectly. All this complex heuristic was removed almost a decade ago. Currently, the kernel refuses a single allocation that exceeds the physical memory. That is all.
The article ignores the proper modern solution to prevent OOM killing of critical processes - OOM Score Adjust.
Tuning CommitLimit manually is an archaic, imprecise, and error-prone way to handle memory limits, only suitable for single-process workloads that can handle ENOMEM properly. It completely ignores dynamic file page cache memory allocation. You still can get OOM if you get unusually high file activity. On the other hand, under low file activity, it wastes memory on the same page cache, because it can't be reclaimed without memory pressure, and memory pressure can't be created because workload hits ENOMEM earlier. Don't use strict overcommit.
First, Linux's default memory management strategy is bonkers. OOM killing rarely actually works in my experience, at least on desktop. It takes ages to kick in and usually the system just freezes and you have to hard reboot. I've experienced this on every Linux system I've used, even my current one with 128GB of RAM and 64GB of swap, so don't say "it works for me". Windows and Mac do not have this issue at all, so clearly it's possible to do it better.
Has anyone tried using strict overcommit on desktop Linux?
Second, this bug is a great counterpoint to those annoying people who naysay Rust with "but not all bugs are memory safety bugs, what about logic bugs? huh?". Rust code would not have had this bug.
I'd be interested to see a Linux distribution whose entire shtick is to run well-behaved under a kernel with overcommit disabled. But it would be a huge undertaking. Besides the obvious issue with fork(), there are a lot of programs and libraries out there that implicitly rely on overcommit due to not checking malloc() for failure.
Sigh. Malloc failure should have had to be trapped with a signal or something, not just a return status. I know, I know, threads and nesting handlers make that hard and historical precedent makes it impossible to retrofit, but I can still dream.
The problem with disabling the memory overcommit is that then the RAM is wasted. That can be worked around with setting up swap but then the disk space is wasted.
The proper way to handle OOM is to do what mature databases do: implement your own memory accounting, use only your own allocators integrated with the accounting system, and ensure that every allocation path can recover from OOM. Easier said than done.
I have a couple of points, not really sure if they should be in one post or not, but whatever.
Firstly, if you take what's written at face value, it seems there's a serious logic error in the OOM killer. If it genuinely counts shared memory against a process, then its logic is wrong because killing that process wouldn't release that much memory, it'd need to kill all the processes sharing that memory to release it. So, maybe it should ignore shared memory in its calculations, or weight them by number of processes sharing it, or whatever.
The other issue is kind of true, but shows a stubbornness from the developers to change how they approach the problem. It's true that if any task could be partially through updating shared memory when it is killed, then all bets are off as to the state of that memory. However, if that's the case, then there should already be some kind of locking mechanisms in place to prevent multiple processes updating the same pages anyway.
Probably the current solution is: something gets locked when modified; every other process would need to spinlock until it's released; locking process is killed; everything else is stuck; another PG thread notices the child has died and kills everything else; on restart the DB has to be recovered.
A different solution could be: give every process its own private part of the shared memory for when it starts a transaction; have an indirection table from page number to shared memory page; for every page that needs to be modified, a new page is allocated from the freed pages list; that allocation is recorded in the private part of the shared memory along with the page number it's replacing; the old page is left untouched and copied to the new page along with any changes; the change list is terminated; then as the last step we update the indirection table for every page that was modified. If the process was killed at any point, we can either roll back the entirety of the transaction (freeing every allocation it made for replacement pages), or if the list was marked as terminated, finish off updating the indirection table with the changes required and marking the original pages as unused. At that point, the lock can be released knowing that shared memory is still entirely consistent.
Some of that is probably happening anyway if postgres supports reading from tables concurrently with an active write transaction on the same table, in which case the logic on how to mark those now freed pages as still in use is required. In that case, each process can also maintain a list of pages it has marked as still being used in case a reading process is killed off.
23 comments
[ 6.0 ms ] story [ 128 ms ] threadI've gone through this exercise in the past on much older kernels which they cover as well and just me personally I ran into less issues by leaving overcommit to 0 and just dropping the overcommit ratio to 0 and setting the oom_score_adj for programs as high as 1000 if I wanted vmscan to leave them alone and of course using the Redhat formulas for setting vm.min_free_kbytes, vm.admin_reserve_kbytes, vm.user_reserve_kbytes. And of course be vigilant in disallowing app owners from using every last bit of memory.
Unfortunately, many programs commit 2x memory than they actually use. Often I see ~32GB committed and ~16GB resident.
For now, we have overcommit_ratio set to a value that is stable from experience, but there really seems to be no silver lining. Go is very happy to allocate a lot of virtual memory, but so are most managed languages. The best solution would probably be to host the backend and the database on separate servers.
I agree with the blog post's technical contents, but I feel we came across too strong in the title. For Ubicloud as a managed Postgres provider, we use strict memory overcommit. Our experience with operating Postgres at scale taught us that it's better to enable this than going with the defaults.
However, I can see many other scenarios, where using strict memory overcommit would have unanticipated side-effects. That's why Linux doesn't go with strict memory commit as its default.
Took k8s ages to get Swap support.
We lost something when we accepted that Hyperscalers just tell you to use more moemory. It was shitty 5 years ago and today especially after the ram price increases
- system dies under memory pressure (regardless of swapping, actually not having swap makes it worse which should be common knowledge by now)
- system dies under disk pressure even if there are tons of free memory (this one is fun to diagnose)
- system can technically not die, but render itself useless (or worse) under memory pressure by the oom killer
- memory compression of any sort is not enabled
- ...
Both Windows and macOS do so much better out of the box for essentially any workload.
We test FreeBSD, Linux, macOS, NetBSD, OpenBSD, and Windows in Zig's CI fleet. Of these, Windows is the only OS that we've had to configure with swap double the size of physical RAM to not hit completely unjustifiable OOMs.
By "unjustifiable", I mean that we're not even close to actually running out of physical memory (let alone swap), but the MM seems to be doing a horrible job of making unused memory actually available to processes.
It's possible there's a relevant configuration knob here that we're just not aware of... but the point is, the default behavior does in fact suck.
The article ignores the proper modern solution to prevent OOM killing of critical processes - OOM Score Adjust.
Tuning CommitLimit manually is an archaic, imprecise, and error-prone way to handle memory limits, only suitable for single-process workloads that can handle ENOMEM properly. It completely ignores dynamic file page cache memory allocation. You still can get OOM if you get unusually high file activity. On the other hand, under low file activity, it wastes memory on the same page cache, because it can't be reclaimed without memory pressure, and memory pressure can't be created because workload hits ENOMEM earlier. Don't use strict overcommit.
First, Linux's default memory management strategy is bonkers. OOM killing rarely actually works in my experience, at least on desktop. It takes ages to kick in and usually the system just freezes and you have to hard reboot. I've experienced this on every Linux system I've used, even my current one with 128GB of RAM and 64GB of swap, so don't say "it works for me". Windows and Mac do not have this issue at all, so clearly it's possible to do it better.
Has anyone tried using strict overcommit on desktop Linux?
Second, this bug is a great counterpoint to those annoying people who naysay Rust with "but not all bugs are memory safety bugs, what about logic bugs? huh?". Rust code would not have had this bug.
Firstly, if you take what's written at face value, it seems there's a serious logic error in the OOM killer. If it genuinely counts shared memory against a process, then its logic is wrong because killing that process wouldn't release that much memory, it'd need to kill all the processes sharing that memory to release it. So, maybe it should ignore shared memory in its calculations, or weight them by number of processes sharing it, or whatever.
The other issue is kind of true, but shows a stubbornness from the developers to change how they approach the problem. It's true that if any task could be partially through updating shared memory when it is killed, then all bets are off as to the state of that memory. However, if that's the case, then there should already be some kind of locking mechanisms in place to prevent multiple processes updating the same pages anyway.
Probably the current solution is: something gets locked when modified; every other process would need to spinlock until it's released; locking process is killed; everything else is stuck; another PG thread notices the child has died and kills everything else; on restart the DB has to be recovered.
A different solution could be: give every process its own private part of the shared memory for when it starts a transaction; have an indirection table from page number to shared memory page; for every page that needs to be modified, a new page is allocated from the freed pages list; that allocation is recorded in the private part of the shared memory along with the page number it's replacing; the old page is left untouched and copied to the new page along with any changes; the change list is terminated; then as the last step we update the indirection table for every page that was modified. If the process was killed at any point, we can either roll back the entirety of the transaction (freeing every allocation it made for replacement pages), or if the list was marked as terminated, finish off updating the indirection table with the changes required and marking the original pages as unused. At that point, the lock can be released knowing that shared memory is still entirely consistent.
Some of that is probably happening anyway if postgres supports reading from tables concurrently with an active write transaction on the same table, in which case the logic on how to mark those now freed pages as still in use is required. In that case, each process can also maintain a list of pages it has marked as still being used in case a reading process is killed off.