Depends, given that operating systems like Solaris, macOS, Windows, also have userspace thread like APIs (tasks, dispach queues, concurrent runtime), that do the kernel/userspace M:N mapping in a language agnostic way.
The post literally mentions that a few paragraphs down in further detail.
> The less that doing things with threads crosses the CPU's security boundary into (and out of) the kernel, the faster the threads go but the less we can really describe them as 'OS threads' and the harder it is to get things like forced thread preemption. And this applies not just for the 'OS threads' themselves but also to their activities. If you want 'OS threads' that perform 'synchronous IO through simple system calls', those IO operations are also transitioning into and out of the kernel. If you work to get around this purely through software, I suspect that what you wind up with is something that looks a lot like 'green' (user-space) threads with asynchronous IO once you peer behind the scenes of the abstractions that programs are seeing.
We took a shot at doing ultra-fast kernel threads on FreeBSD a few decades ago. For various reasons, it was reverted and removed a few major versions later.
If you look a the old KSE work, the general gist was that if you were about to block in a syscall then you'd effectively get a signal-style longjmp back to your userland thread scheduler. You'd pick another thread and continue running all in the same process/task context.
There were many problems with what we did and how we did it, but the unavoidable fundamental problem at the time was that it inverted assumptions about costs of low level primitives. Important(TM) software was optimized for the world where threads and blocking were expensive and things like pthread mutex operations were cheap. Our changes made threads and blocking trivially cheap but added non-trivial overhead to pthread mutex etc operations. Applications that made extensive use of pthread mutexes to coordinate work dispatching on a precious small pool of expensive threads were hit with devastating performance losses. Most critically, MySQL. We'd optimized for hundreds of thousands of threads rather than the case of multiplexing work over a few threads.
It became apparent that this was going to be an eternal uphill battle and we eventually pulled the plug to do it the same way as Linux. We made a lot of mistakes with all of this.
Since they mention "a few decades ago" I'm guessing it's a very different mutex than the one you'd use today.
Today I believe all of these operating systems (except MacOS, LOL) have either futex or an equivalent technology (the Windows thing works on bytes rather than aligned 32-bit values, so that's cool) and so it seems like only contention could be more expensive as the whole point of a futex is that uncontended acquisition is a single CPU store and that's userspace, the kernel is nowhere near it.
I was not aware. I wonder if anybody tried asking Apple? That's what happened for SRWLOCK in Windows. Mara asked Microsoft if they can promise it actually does what it seems like it does, formally, and they did document that promise so then Rust used SRWLOCK [today your Rust does not use SRWLOCK, this year it was replaced for performance reasons, which is timely because it turns out SRWLOCK is faulty and so "Don't use SRWLOCK" is the most practical fix in the medium term, awkward for poor C++ though as they used SRWLOCK too].
Apple are even less talkative than Microsoft, but it is possible that the reason libc++ uses this is that it's actually guaranteed and if properly asked Apple would say so. It would be nice for (newer) macOS to get the same promises as the mainstream platforms with respect to these primitives.
This argument is basically hand waving, and it's factually wrong.
Yes, switching from user to kernel mode and back is expensive. But let's count the syscalls.
Model 1: Threads.
1 blocking read(), 1 blocking write().
Plus cost for the OS scheduler.
Model 2:
1 "I want to read", 1 "can I read now?", 1 actual read, 1 "I want to write now", 1 "can I write now?", 1 actual write.
Multiply as necessary if the read or write are only partial.
Add IPC cost as necessary if you have more than one thread handling async events.
Measuring by the syscall overhead, blocking I/O is much more efficient.
There are other costs to it, though, so it usually consumes more resources per handled socket and turns out to be more expensive than async I/O. But it's not the syscall cost causing it.
This ignores the existence of batching capabilities like epoll and io uring. Async IO may need to do more syscalls than blocking IO for a single operation, but the story looks different if you consider a large number of operations that are awaited as a group.
It seems epoll could be slower than poll for async/await environments. It requires epoll_ctl (syscall) on each fd state transition. `poll` could wait in a single syscall for a small set of current blocked fds.
io_uring actually changes this story because you can communicate events without crossing the syscall barrier, but only if you get enough incoming events to saturate your processing capabilities.
epoll batching only works in respect to the event notifications from kernel to user space. The only other thing that could be considered batched is that you can switch from "I want to read" to "I want to write" without telling the kernel first that you don't want to read anymore. But that's hardly worth mentioning.
That is an assumption and generalization you can't make. In fact it is usually wrong. I invite you to do actual measurements before claiming things as fact in the future.
I, for example, implement an async web server and your case basically never happens. You get a connection, you call read, there is not going to be data yet. In fact calling read immediately is basically wasted effort, so I'm not doing it. I save one syscall over your proposal right away. Optimizations like TCP fast open might change this over time, but definitely won't when you do SSL. You need to wait for a request and then for replies. SSL involves several additional back and fros over plain TCP.
Writing is worse unless you tell the OS to provide ungodly buffer space per connection, which you shouldn't, because then your server won't scale.
I don’t think we should parrot the syscalls-are-expensive too much. The overhead of thread syscalls is smaller than you’d think, and for say TCP and file IO our buffers are so big that we can mitigate the amounts in practice to saturate hardware (like NICs and SSDs) limitations, no problem.
The issue has more to do with thread lifecycle management, ie creating, maintaining a stack and destroying the thread. When you have very lightweight tasks, those add up and have no direct mitigations. That’s a big reason we moved runtimes into userspace.
I wish I could see the perf chart of these. If anyone finds them please link. It’s important to have objective facts to base these discussions in.
For thread creation, the author of the original post said "somewhere between 10 and 20 microseconds". This matches my own (very simple) benchmarks on my Ubuntu desktop, which was able to create and destroy just over 100k threads/second.
Depending on your use case, using a thread pool could mitigate this cost entirely.
The problem with this analysis is that single system calls aren't the only way to invoke the operating system. You can also use an io_uring-style mechanism to batch your system calls. Or BPF to run mini-programs in the context of the OS, which is also a sort of batching. These don't have the same problematic overhead as system calls.
This is really just archaic *nix world problems; the NT kernel has had actual asynchronous I/O since inception -- even 30 years later, io_uring is mere batching to the kernel upper-half (and is a carbon-copy, down to the particular terminology, of RIO)
24 comments
[ 3.1 ms ] story [ 67.8 ms ] thread> The less that doing things with threads crosses the CPU's security boundary into (and out of) the kernel, the faster the threads go but the less we can really describe them as 'OS threads' and the harder it is to get things like forced thread preemption. And this applies not just for the 'OS threads' themselves but also to their activities. If you want 'OS threads' that perform 'synchronous IO through simple system calls', those IO operations are also transitioning into and out of the kernel. If you work to get around this purely through software, I suspect that what you wind up with is something that looks a lot like 'green' (user-space) threads with asynchronous IO once you peer behind the scenes of the abstractions that programs are seeing.
If you look a the old KSE work, the general gist was that if you were about to block in a syscall then you'd effectively get a signal-style longjmp back to your userland thread scheduler. You'd pick another thread and continue running all in the same process/task context.
There were many problems with what we did and how we did it, but the unavoidable fundamental problem at the time was that it inverted assumptions about costs of low level primitives. Important(TM) software was optimized for the world where threads and blocking were expensive and things like pthread mutex operations were cheap. Our changes made threads and blocking trivially cheap but added non-trivial overhead to pthread mutex etc operations. Applications that made extensive use of pthread mutexes to coordinate work dispatching on a precious small pool of expensive threads were hit with devastating performance losses. Most critically, MySQL. We'd optimized for hundreds of thousands of threads rather than the case of multiplexing work over a few threads.
It became apparent that this was going to be an eternal uphill battle and we eventually pulled the plug to do it the same way as Linux. We made a lot of mistakes with all of this.
[1] https://en.wikipedia.org/wiki/Scheduler_activations
Today I believe all of these operating systems (except MacOS, LOL) have either futex or an equivalent technology (the Windows thing works on bytes rather than aligned 32-bit values, so that's cool) and so it seems like only contention could be more expensive as the whole point of a futex is that uncontended acquisition is a single CPU store and that's userspace, the kernel is nowhere near it.
Apple are even less talkative than Microsoft, but it is possible that the reason libc++ uses this is that it's actually guaranteed and if properly asked Apple would say so. It would be nice for (newer) macOS to get the same promises as the mainstream platforms with respect to these primitives.
Yes, switching from user to kernel mode and back is expensive. But let's count the syscalls.
Model 1: Threads. 1 blocking read(), 1 blocking write(). Plus cost for the OS scheduler.
Model 2: 1 "I want to read", 1 "can I read now?", 1 actual read, 1 "I want to write now", 1 "can I write now?", 1 actual write. Multiply as necessary if the read or write are only partial. Add IPC cost as necessary if you have more than one thread handling async events.
Measuring by the syscall overhead, blocking I/O is much more efficient.
There are other costs to it, though, so it usually consumes more resources per handled socket and turns out to be more expensive than async I/O. But it's not the syscall cost causing it.
epoll batching only works in respect to the event notifications from kernel to user space. The only other thing that could be considered batched is that you can switch from "I want to read" to "I want to write" without telling the kernel first that you don't want to read anymore. But that's hardly worth mentioning.
Nonblock read(), if EAGAIN: append fd into event waiting queue, wait event and read() again
Nonblock write(), if EAGAIN: append fd into event waiting queue, wait event and write() again
For loaded systems EAGAIN case is quite rare and syscall count is same.
I, for example, implement an async web server and your case basically never happens. You get a connection, you call read, there is not going to be data yet. In fact calling read immediately is basically wasted effort, so I'm not doing it. I save one syscall over your proposal right away. Optimizations like TCP fast open might change this over time, but definitely won't when you do SSL. You need to wait for a request and then for replies. SSL involves several additional back and fros over plain TCP.
Writing is worse unless you tell the OS to provide ungodly buffer space per connection, which you shouldn't, because then your server won't scale.
The issue has more to do with thread lifecycle management, ie creating, maintaining a stack and destroying the thread. When you have very lightweight tasks, those add up and have no direct mitigations. That’s a big reason we moved runtimes into userspace.
I wish I could see the perf chart of these. If anyone finds them please link. It’s important to have objective facts to base these discussions in.
For thread creation, the author of the original post said "somewhere between 10 and 20 microseconds". This matches my own (very simple) benchmarks on my Ubuntu desktop, which was able to create and destroy just over 100k threads/second.
Depending on your use case, using a thread pool could mitigate this cost entirely.
Asynchronous IO: the next billion-dollar mistake?
https://news.ycombinator.com/item?id=41471707
Your phrasing here suggests the NT kernel can/does have API’s that reach further down, is that correct, or am I reading too deeply? Haha