NUMA can cause really crappy performance. We deployed a Go based LLM gateway in Kubernetes deployed on a server with hundreds of CPU cores. We didn't explicitly set GOMAXPROCS so Go runtime scheduled goroutines over different CPUs and it constantly used 200% CPU and GC was causing latency spikes. Then we set GOMAXPROCS 8 and all performance issues went away. Until recently Kubernetes didn't work well with NUMA.
Heck, we saw crazy performance degradation with redis when its memory usage exceeded a single NUMA block. Not much to be done about that at the k8s level when redis is single-threaded. Have to be super conscious of the underlying hardware at that point.
I don't see how GOMAXPROCS alone can help here though. You would have to use Topology Manager (single-node policy) to avoid cross-NUMA allocations. This is in addition to other managers - Memory and CPU Manager.
CPU Pinning (via CPU Manager's Static Policy) will also be required to ensure your processes don't just get a CFS quota/share but are actually pinned onto specific CPU cores.
Some lessons are never lost it seems, back when Windows NT was recent, we had to lock threads/processes to specific CPUs on SMP machines (affinity), exactly for similar reasons.
Something I didn’t see mentioned was that this unequal memory access time also affects pcie I/O. If your thread on CPU A needs to get data in or out of a nic on CPU B, your throughput/latency will be impacted.
We have to explain this to customers of our software all the time, it’s something that’s easy to miss.
Was going to mention this too, as it burned me once. Not cause I didn't know about it but because I was accidentally running stuff on the wrong node, and it wasn't obvious which slot was which node.
NUMA is one of those amazing things that trip you up in all sorts of ways at unexpected times. The amazing "invisible" performance killler (invisible because unless you're already aware of NUMA, or remember to check, you won't know it's there potentially crippling you.)
It has been a source of routine conversations with customers and engineers of all kinds, and often one of those things you don't know about until too late.
I don't know if the kernel has improved this behaviour in the several years since last tested, but a coworker realised that the linux page-cache wasn't fully split by NUMA node. They were benchmarking mysql running it in each NUMA node, and noticed the second NUMA node was noticeably slower. Then discover after a reboot the second node was fast, and the first was slower. After a bit of thinking and tinkering they discovered that libmysql was ending up in the page cache in the same NUMA node as the benchmark client was run in first, so even though they were pinning the benchmark tool and mysql process to the NUMA node, the benchmark client was causing the OS to reach across the NUMA node to get at the page cached library.
Yeah, when you have tall servers this can be a really surprising factor. In some sense you could view this as an extension of processor caching behaviors, which also causes some memory accesses to be lower - just due to cache behaviors, not physical location. But in many cases, the same tools can be used to fight both "far" memory accesses and cache trashing, by using a thread-isolated architecture.
I have been dealing with the topic for a few years now and it was surprisingly hard to track down the bottlenecks to actual numbers. Some time ago I managed to find a good example to demonstrate the effect in a tangible way and wrote up an article about it. If the topic sounds interesting, you might enjoy https://sander.saares.eu/2025/03/31/structural-changes-for-4... (Structural changes for +48-89% throughput in a Rust web service).
I'm baffled by the fact that NUMA is still an issue in 2026. My impression is that this was all solved back in dotcom era already on those big SUNs. At least in HPC we solved this already in mid 2000s. Why is supposedly modern world still wasting time on this? Kernel these days exposes just about everything you would ever want to know about a system topology and every runtime should be making use of that information. If it does not, I cannot consider it ready for this century.
Sun was one of the few RISC design houses who stubbornly resisted NUMA in favour of SMP, which they have perfected in the hardware and in the software (Solaris). The «Solaris internals» book discusses the subject of SMP vs NUMA vs ccNUMA in the «Parallel systems architectures» section (3.2).
You might be thinking SGI who went in big on NUMA. IBM and HP have also built ccNUMA systems (e.g. HP Superdome and Superdome 2).
My question: why do mainstream users tolerate NUMA? 99% of you don't need to. Single-socket servers exist and they are not only tolerable but better in most ways. Dealing with NUMA in software consists of trying to logically partition the machine, but you can instead physically partition the machine. It's so much simpler!
Amazon gets this. Except for the 4th generation their Graviton systems are not NUMA.
NUMA latencies across machines are way worse than across sockets or across core complexes. :p
Single socket doesn't necessarily get you away from NUMA anyway, AMD server sockets are 4 way NUMA (you can set it for interleaving, but you could do better with NUMA-aware software), and I think Intel is doing NUMA on server socket as well.
A lot of people like to take one big machine and partition it into several smaller virtual machines. In that case, it shouldn't be too hard to partition vms into NUMA zones? Only vms that are two big to fit in one zone have to worry about it (or that need to be repacked into a different zone)
Complete slop from start to finish. Some of the distinctions manufactured by the LLM author are nonsensical: “a thread may run on node 0 but access data on node 1. Conversely, it could also run on node 1 but access data on node 0.” These cases are different how? Or “there are 3 possible cases. A thread might run on the right node but access data on the wrong node. Or it might access data on the right node but run on the wrong node. Or both.” WTF could this possibly mean? This hallucinated nonsense logic wastes the time of the reader and whoever posted it, and whoever submitted it anyway, should be ashamed of themselves. The prevalence of this garbage just makes it harder to find accurate sources on topics of interest. They are certainly out there, but it’s getting harder and harder to find them.
Why use NUMA if most of the time to get max performance you need locality and thus end up pinning things to a specific node to avoid the cost of crossing nodes. Moreover, you almost always must set this manually.
Wouldn’t it just be easier to skip it all together and just use single node architectures for much less cost and time?
20 comments
[ 5.0 ms ] story [ 54.8 ms ] threadWas that a Power9 or some sort of IBM machine?
Not all NUMA is the same, ccNUMA from the Intel is a different beast from the PPC version of the same.
CPU Pinning (via CPU Manager's Static Policy) will also be required to ensure your processes don't just get a CFS quota/share but are actually pinned onto specific CPU cores.
We have to explain this to customers of our software all the time, it’s something that’s easy to miss.
It has been a source of routine conversations with customers and engineers of all kinds, and often one of those things you don't know about until too late.
I don't know if the kernel has improved this behaviour in the several years since last tested, but a coworker realised that the linux page-cache wasn't fully split by NUMA node. They were benchmarking mysql running it in each NUMA node, and noticed the second NUMA node was noticeably slower. Then discover after a reboot the second node was fast, and the first was slower. After a bit of thinking and tinkering they discovered that libmysql was ending up in the page cache in the same NUMA node as the benchmark client was run in first, so even though they were pinning the benchmark tool and mysql process to the NUMA node, the benchmark client was causing the OS to reach across the NUMA node to get at the page cached library.
I have been dealing with the topic for a few years now and it was surprisingly hard to track down the bottlenecks to actual numbers. Some time ago I managed to find a good example to demonstrate the effect in a tangible way and wrote up an article about it. If the topic sounds interesting, you might enjoy https://sander.saares.eu/2025/03/31/structural-changes-for-4... (Structural changes for +48-89% throughput in a Rust web service).
I'm pretty sure all the big Sun boxes were SMP, not NUMA, at least during the dot-com era. Not sure about later UltraSPARC T or M Series systems.
You might be thinking SGI who went in big on NUMA. IBM and HP have also built ccNUMA systems (e.g. HP Superdome and Superdome 2).
Amazon gets this. Except for the 4th generation their Graviton systems are not NUMA.
Single socket doesn't necessarily get you away from NUMA anyway, AMD server sockets are 4 way NUMA (you can set it for interleaving, but you could do better with NUMA-aware software), and I think Intel is doing NUMA on server socket as well.
A lot of people like to take one big machine and partition it into several smaller virtual machines. In that case, it shouldn't be too hard to partition vms into NUMA zones? Only vms that are two big to fit in one zone have to worry about it (or that need to be repacked into a different zone)
Wouldn’t it just be easier to skip it all together and just use single node architectures for much less cost and time?