It's important to specify which OS is being used because spawning goes a lot faster on Linux which has vfork(). If 40 spawns per second is the Linux speed, then I don't even want to know what that looks like on Darwin and OpenBSD. With Cosmopolitan Libc, 20 spawns per second is about the speed I get when simulating fork() on Windows. What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens. See the IEEE POSIX.1 notes surrounding pthread_atfork(). So yeah, that means everything is de facto blocked until the spawn completes, regardless of the thread that's doing it. Using the separate process to spawn is a smart idea. Especially if you can talk to it via a process shared condition variable.
> What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens
Are there any circumstances where you'd want to fork a multi-threaded process?
forking is useful if you need to call other code when you can't or don't want to link that code into your own process. Pretty useful for things like ffmpeg where you wouldn't want that in your process space (and you may want to put more than just a single fork/exec between your code and theirs in that case), but also handy for "simple" os integrations where it's easier to run an existing utility than find a library or interface more directly yourself.
Downsides in forking from a large multi-threaded process make it pretty common to add something in early init that forks a process (or several) that remain small and can be commanded to fork at runtime. On modern systems, it's a very small cost if you don't actually fork, and the small added cost of intermediating fork/exec pays off the first time you don't have to wait for hundreds or thousands or millions of fds to be closed in the child process during the process. Of course, if your parent process remains small with few opened filehandles, maybe it is a measurable negative, but it's still likely to be very small.
posix_spawn is not a syscall, last I checked, on Linux or FreeBSD. It's a library function of libcs, implemented in terms of fork, vfork, or clone and execve and has the same underlying performance issues in a large process as doing it yourself.
It would be a different thing if it was a syscall, because then you could presumably use it in a large, multi threaded process without any of the hazards that arise from using fork, vfork, and clone.
Are you asking if there's a good reason to have a fork without exec in a multithreaded program? I would think probably not, it would be pretty confusing to have all the memory and filedescriptors in the child process but only one thread --- you'd still have the other thread's stacks, but no thread running on them; that's asking for massive confusion and I think you'd need to do some cleanup for locks and whatever else. There might well be some applications where it makes senss to fork, and then spawn threads in the child processes, although I can't think of one off the top of my head. I've definitely had experience with use cases where it made more sense to fork rather than spawn threads and vice versa, but I haven't seen a reason to do both.
Yeah, I meant from a libc user's point of view, using posix_spawn() directly seems safer (presumably they've managed to ensure they don't do anything dumb; I don't trust myself to be as good). A random copy of its man page actually says it's implemented with clone() with CLONE_VFORK plus exec().
IMHO, using posix_spawn is OK if you know what libc your application is going to be using.
If you don't, you may have inconsistent results, and you'll end up having to sample the various libcs, and probably take the best posix_spawn so your application has consistent behavior. Or you'll have to build in detailed specifications of which libcs and which versions of those libcs you'll work with.
* In single-threaded programs (potentially without calling exec). This was widely used to get parallelism within a single program prior to 1995 or so, and is still a valid approach (except on OS X, which forbids single-threaded programs). In this case `pthread_atfork` is a reasonable thing to use, and the child process is not limited to async-signal-safe functions.
* In (potentially-multi-threaded) programs calling exec. In this case, `pthread_atfork` is not meaningful (there's a recent `_Fork` that avoids it entirely), and the child is limited to async-signal-safe functions - basically, you should do all the setup beforehand, then after `fork` just call various syscalls. `posix_spawn` and `vfork` may replace this, but both are bug-prone.
No, you can fork, what you can’t do is “fork and then try to keep running after the fork”. That’s never really been a supported model on Darwin, and in recent years with more threading occurring it fails faster than it did in the past.
On Darwin there is very little you can safely do in the child after fork(), essentially the only thing guaranteed to work at that point is exec() and a few other basic operations.
Again, this has always been the case for fork under Darwin, the only difference is that you are more likely to fail sooner now than 20 years ago. This is also why all documentation says to use posix_spawn: fork +exec is necessarily slower, and can’t really do anything that posix spawn doesn’t.
I sounds to me that while fork() was perhaps initially a simple idea, it has now (with multithreading etc) become a monstrosity that is better replaced by a number of syscalls that are less general and have more intent behind them.
Perhaps fork() will go away naturally when other options turn out to be faster or people start porting stuff to Darwin and other OSes where fork() is not a good idea anyway.
How else would you expect a green-threaded runtime with multiple scheduler threads to spawn a managed subprocess (think: Erlang port-program) that has access to the runtime's file descriptors, unix domain sockets, etc?
Do you have a sense for why things were so much faster in Go and Rust if they are both using a single process? I have also been experimenting with napi-rs which uses https://nodejs.org/api/n-api.html and allows you to run Rust in threads alongside the Node process. Performance with that approach has been very good, so I have wondered about the within-process limitations.
If you want to spawn a new process on Darwin you really want to reach for posix_spawn, not any variant of fork. Using this API should also do whatever is optimal on Linux, which still doesn't seem to be merely calling vfork, as you instead can use clone--with CLONE_VFORK, sure--to limit the work to CLONE_VM (per the man page for glibc's posix_spawn). (Of course, this API would also be the perfect fit for use on Windows, but I don't think MinGW or Cygwin implement it, for reasons which are beyond me, given how horrible simulating fork is in that OS; but, I believe gnulib provides it, hopefully using CreateProcess on Win32, for all of these platforms than are missing out... and, they really are "missing out", as I feel most developers--including me--end up incorrectly dealing with the issues surrounding multi-threaded fork.)
> If you want to spawn a new process on Darwin you really want to reach for posix_spawn
Assuming you can avoid the bugs and version-sensitivities. There are a lot of them on various platforms, and my gut says they're more prominent on Darwin since alternatives are unrecommended.
The dup2/cloexec mess is the one that sticks in my mind but it's far from the only one.
nodejs (via libuv) transparently switches from fork to pthread_spawn on Darwin when the options passed support it. The dup2/cloexec is a mess, but there are some Darwin-specific enhancements to work around it. There is work done to make it also do the same on Linux, but seems it hasn’t been merged yet.
The reason glibc uses clone is that it can specify an extra flag to avoid some undefined behavior inherent in the way vfork was originally designed, so it can get all of the equivalent speed with none of the pointy edges
To be fair for Windows, you shouldn’t emulate fork() but rather spawn new threads, which Windows can do in less than a millisecond. This should give you much quicker results, but comes with other complexities, especially for UN*X-ported apps. Given you emulated fork(), you probably know this.
But those aren't really the same? I mean yes they are both ways to acquire additional units of execution but if you're reaching for subprocesses it's likely specifically because it's something you can't do in threads*.
* Or can do but would be horrifying, like trying to run a compiled binary by mmaping it into your address space.
I would agree, if you’re wanting to run your server software on windows, your software should be designed to work well with windows. That goes for any OS though - if you’re running a high volume server then you shouldn’t be throwing away perf by designing your software in a way that works badly on the host OS.
That said multiple threads are not multiple processes:
Using multiple threads means you get a single set of globals, and if any of them are contended you get a pile of sadness.
Similarly, tearing down threads is not necessarily sufficient to release all resources a library may have consumed because again: globals.
Related to globals: I’m not sure how heavily TLS is restricted on windows, but I recall various arbitrary limits being annoying on windows back when I did dev work, and I recall some annoying stuff with TLS — but that was back in the days of needing to support XP, so maybe it’s less of an issue these days?
You also get security advantages by pushing things into multiple processes that you don’t get from multiple threads, though I don’t really think that that’s an issue in this particular context.
For .NET Framework, AppDomains were a solution to isolation. .NET [Core] deprecated this feature, moving towards assembly loading contexts (and the, IMO, unfortunate move towards microservices -- like everyone needs them /s).
Not sure what your reference to TLS is about; any more specifics?
100%. I was of course speaking in the context of this thread, but yes you probably wouldn't take a Windows-developed app and simply shift it to UN*X without some form of re-thought on the implementation.
> What makes fork() slow for large programs like Node is that fork() needs to lock and hold every single mutex that exists in a process while it happens
It really doesn't; if it does that's a bug to fix. It only requires that the child process does not call any function that is not async-signal-safe if threads exist, and end with a call to exec or _exit. In C, this is usually pretty easy to arrange; at most the difficulty is if you want to support arbitrary file-descriptor mappings. However, it is impossible from languages like Javascript (or Python, etc.).
pthread_atfork() is really only useful to call if you know you're using multiple processes and not using threads at all. _Fork() is supposed to be a thing now (to avoid any historical atfork handlers) but can't be relied on to exist.
posix_spawn() has the unfortunate problem of being buggy and version-dependent when various options are used. It's probably fine if you don't need the options (for example, if you exec a shim first), or if you know how recent a platform you can use.
vfork() on Linux is mostly an optimization (for scheduling or for when overcommit is disabled), but compiler optimizations can make it buggy (though given that /bin/sh etc. use it, compiler probably won't break it too much if you call it in the same way). For best results, it's advisable to either call clone (with a separate stack directly), or use architecture-specific assembly to call `vfork` in a way the compiler can't mess up.
vfork() on many other platforms is basically just an alias for fork() (or maybe _Fork() ?), but I haven't investigated the details much.
fork() is fine if your parent process isn't big enough that you're worried about memory accounting though, and you haven't added braindead atfork handlers.
You absolutely need to lock all the locks, because fork() kills all the other threads, so if any one of them was in a critical section while fork was called, then the new process will deadlock if it tries to acquire that lock afterwards. Yes, I know the POSIX standard has turned against fork() and considers vfork() beyond the pale. I don't know how they can forbid UNIX behavior that's essential to the workings of /bin/sh. No it's not realistic for the OS to define its most important function to be as useless as a signal handler, but here we are. That advice will be never be accepted. People on the best OSes like Linux and FreeBSD will always use vfork() regardless of what POSIX says. The de facto standard is that only vfork() must guarantee a swift execve() call. You learn to take the official standard with a grain of salt when you realize it's bankrolled by regulated enterprises who hire cheap engineers to work on life or death machinery and therefore can't be trusted with powerful footguns.
> then the new process will deadlock if it tries to acquire that lock afterwards
which is not async-signal safe and thus illegal if other threads ever existed (or if we're called from a signal handler). All semi-new programs (which might use threads) are responsible for not introducing such bugs.
And if we're the only thread (and not in a signal handler), which all sufficiently-old programs (including shells) will be, it's the caller's responsibility to not call fork() while holding a mutex.
Thus, one way or another, there's really no need to do this. From an application perspective, it's unnecessary, and from a libc perspective it is at most a QoI feature (one which should be limited to a very small number of locks at most, and frankly we've strongly learned should just detect the error (which is a simple matter of 2 thread-local counters - one for "number of mutexes held" and one for "number of async-signal contexts entered" - as well as knowledge of whether other threads exist) rather than trying to continue in the face of UB).
Illegal it may be, but as jart (who has probably seen some shit, to put it mildly) points out, that doesn’t stop a very wide variety of programs from depending on this behavior, intentionally or inadvertently.
Heck, runtimes like Python (and Node, if jart is correct in GGP) ship with atfork handlers specifically for the purpose of massaging runtime-created mutexes/condvars/etc. I don’t think those are there because the language maintainers like breaking the rules; they’re probably there because users kept depending on this behavior in spite of cautionary statements.
Say more? Is this just get-gud anti scripting language elitism, or a reference to some specific history that the Python/Node/others(?) maintainers have regarding decisions made in this area?
> Using the separate process to spawn is a smart idea. Especially if you can talk to it via a process shared condition variable.
You can even use prctl(2) to set the parent as a subreaper for the spawner’s children so the process hierarchy continues to behave as expected and you don’t end up with orphans. Or just run such that the parent is PID 1 in a container anyway.
With either approach, watch out if you’re using a thicker runtime/language whose child process accounting can get freaked out if an unexpectedly-present child indicates ready-for-reap. Using pidfds on Linux (optionally combined with fdpass via UNIX sockets to signal that the spawner did its thing) can help with this.
> With Cosmopolitan Libc, 20 spawns per second is about the speed I get when simulating fork() on Windows.
Egads, this reminds me of when I used to muck with Cygwin. fork() is quite a complex bit of kernel magic, and that’s on kernels designed for it. Simulating fork() on Windows well enough to be usable has always struck me as quite impressive.
But it’s also ridiculous. Forking to spawn a process is an almost comically poor design, and this is one of maybe two things IMO that Windows got right! CreateProcess() may be ugly, its arguments may involve nasty data structures, the command line string may be a disaster that continues to cause problems to this day, but it’s a far superior design for spawning processes. I hope that no one who cares even slightly about process spawning performance on Windows tries to do it using fork.
Not that Windows has ever been particularly fast at spawning processes…
Both Go and Rust had CPU near 100% on all cores. Most of the Deno and Bun workloads were at around 80%, some of the slower Node benchmarks were around 40-60% and that increased over time, I think most of the fastest benchmarks were seeing very good CPU utilization across all runs.
Sorry, this is all from memory of just having `btop` open for some of the benchmarks. Maybe in a future post I will dig into CPU utlization more.
I'm curious if anyone has any insights into the answer to the titular question. The article, while certainly interesting, mostly discusses workarounds, but doesn't really dive into a root cause analysis.
One addendum I would add to the section on node:cluster is that Deno hasn't actually implemented the cluster module yet (it's just a stub), so using it with Deno is pure overhead [1]. Also, there's ongoing work to add node:cluster to Bun [2].
> Turns out, that if you use a Unix socket and the filename starts with a null byte, like \0foo the socket will not exist on the filesystem and it’ll be automatically removed when no longer used. Weird! Cool!
Is that…intentional on Unix’s part? Seems kind of a weird thing to implement.
files on unix are referenced counted. and unix sockets are just a type of file. every link adds to the count, a link can be a filesystem name but open file handles are also links. once there are no more links to a file the file can be deleted.
so this is opening a unix socket with a no name(the initial null terminates the C style string) thus leaving the only link to it as the file discripter.
One trick I used to do was to sneak another link to the file to keep it around before the process ended , The serious real use case for this was if you had accidentally deleted a file but some process was keeping it open somewhere, but I mainly used it back when youtube was flash based it would try and keep the video cache file hidden this way. and you could make a copy of the video by snaging the correct file descriptor.
I have fond memories of modifying the memory of flash-based games when they first launched on Facebook.
I don't remember the FB account credentials, but I received thousands of messages from people either telling me I was a terrible person, or trying to bribe me so they could beat their sibling... all for a simple helicopter game.
Nice article! I remember three years ago, I needed to spawn a few processes in Node, and I ended up using child process spawn to spawn an external binary. It was fast and instantaneous (no benchmarks). It was on a Linux OS, ARM-based.
Have you looked into profiling with strace? You should be able to see the actual system calls each of your tests are making. This will probably give you some good insights into what accounts for the speed differences.
58 comments
[ 2.6 ms ] story [ 71.8 ms ] threadAs a heads up, the author confuses its and it's. Makes the article look unprofessional. https://youryoure.com/?its here's how to differentiate
It's easy to forget what a big deal being semi fluent in multiple languages is.
Are there any circumstances where you'd want to fork a multi-threaded process?
Downsides in forking from a large multi-threaded process make it pretty common to add something in early init that forks a process (or several) that remain small and can be commanded to fork at runtime. On modern systems, it's a very small cost if you don't actually fork, and the small added cost of intermediating fork/exec pays off the first time you don't have to wait for hundreds or thousands or millions of fds to be closed in the child process during the process. Of course, if your parent process remains small with few opened filehandles, maybe it is a measurable negative, but it's still likely to be very small.
It would be a different thing if it was a syscall, because then you could presumably use it in a large, multi threaded process without any of the hazards that arise from using fork, vfork, and clone.
Are you asking if there's a good reason to have a fork without exec in a multithreaded program? I would think probably not, it would be pretty confusing to have all the memory and filedescriptors in the child process but only one thread --- you'd still have the other thread's stacks, but no thread running on them; that's asking for massive confusion and I think you'd need to do some cleanup for locks and whatever else. There might well be some applications where it makes senss to fork, and then spawn threads in the child processes, although I can't think of one off the top of my head. I've definitely had experience with use cases where it made more sense to fork rather than spawn threads and vice versa, but I haven't seen a reason to do both.
If you don't, you may have inconsistent results, and you'll end up having to sample the various libcs, and probably take the best posix_spawn so your application has consistent behavior. Or you'll have to build in detailed specifications of which libcs and which versions of those libcs you'll work with.
* In single-threaded programs (potentially without calling exec). This was widely used to get parallelism within a single program prior to 1995 or so, and is still a valid approach (except on OS X, which forbids single-threaded programs). In this case `pthread_atfork` is a reasonable thing to use, and the child process is not limited to async-signal-safe functions.
* In (potentially-multi-threaded) programs calling exec. In this case, `pthread_atfork` is not meaningful (there's a recent `_Fork` that avoids it entirely), and the child is limited to async-signal-safe functions - basically, you should do all the setup beforehand, then after `fork` just call various syscalls. `posix_spawn` and `vfork` may replace this, but both are bug-prone.
…uh, what?
Thus, calling fork() on OS X means you can only use async-signal-safe functions, unlike other platforms where abstinence is a choice.
On Darwin there is very little you can safely do in the child after fork(), essentially the only thing guaranteed to work at that point is exec() and a few other basic operations.
Again, this has always been the case for fork under Darwin, the only difference is that you are more likely to fail sooner now than 20 years ago. This is also why all documentation says to use posix_spawn: fork +exec is necessarily slower, and can’t really do anything that posix spawn doesn’t.
This is blatantly untrue. Probably the most common example is `chdir`, but there's a whole list in signal-safety(7).
Using a shim is always a possibility but that's annoying.
Perhaps fork() will go away naturally when other options turn out to be faster or people start porting stuff to Darwin and other OSes where fork() is not a good idea anyway.
Assuming you can avoid the bugs and version-sensitivities. There are a lot of them on various platforms, and my gut says they're more prominent on Darwin since alternatives are unrecommended.
The dup2/cloexec mess is the one that sticks in my mind but it's far from the only one.
The reason glibc uses clone is that it can specify an extra flag to avoid some undefined behavior inherent in the way vfork was originally designed, so it can get all of the equivalent speed with none of the pointy edges
* Or can do but would be horrifying, like trying to run a compiled binary by mmaping it into your address space.
It's the "Windows way". Windows process creation is slow. No, they're not identical, they're different approaches to the problem.
> subprocesses it's likely specifically because it's something you can't do in threads
I suppose the question would be, "what can't you do, if you did it the Windows way?"
That said multiple threads are not multiple processes:
Using multiple threads means you get a single set of globals, and if any of them are contended you get a pile of sadness.
Similarly, tearing down threads is not necessarily sufficient to release all resources a library may have consumed because again: globals.
Related to globals: I’m not sure how heavily TLS is restricted on windows, but I recall various arbitrary limits being annoying on windows back when I did dev work, and I recall some annoying stuff with TLS — but that was back in the days of needing to support XP, so maybe it’s less of an issue these days?
You also get security advantages by pushing things into multiple processes that you don’t get from multiple threads, though I don’t really think that that’s an issue in this particular context.
Not sure what your reference to TLS is about; any more specifics?
EDIT: Ugh was thinking SSL! Duh.
https://learn.microsoft.com/en-us/windows/win32/procthread/t...
> That goes for any OS though
100%. I was of course speaking in the context of this thread, but yes you probably wouldn't take a Windows-developed app and simply shift it to UN*X without some form of re-thought on the implementation.
...Unless it was a .NET app :-)
It really doesn't; if it does that's a bug to fix. It only requires that the child process does not call any function that is not async-signal-safe if threads exist, and end with a call to exec or _exit. In C, this is usually pretty easy to arrange; at most the difficulty is if you want to support arbitrary file-descriptor mappings. However, it is impossible from languages like Javascript (or Python, etc.).
pthread_atfork() is really only useful to call if you know you're using multiple processes and not using threads at all. _Fork() is supposed to be a thing now (to avoid any historical atfork handlers) but can't be relied on to exist.
posix_spawn() has the unfortunate problem of being buggy and version-dependent when various options are used. It's probably fine if you don't need the options (for example, if you exec a shim first), or if you know how recent a platform you can use.
vfork() on Linux is mostly an optimization (for scheduling or for when overcommit is disabled), but compiler optimizations can make it buggy (though given that /bin/sh etc. use it, compiler probably won't break it too much if you call it in the same way). For best results, it's advisable to either call clone (with a separate stack directly), or use architecture-specific assembly to call `vfork` in a way the compiler can't mess up.
vfork() on many other platforms is basically just an alias for fork() (or maybe _Fork() ?), but I haven't investigated the details much.
fork() is fine if your parent process isn't big enough that you're worried about memory accounting though, and you haven't added braindead atfork handlers.
which is not async-signal safe and thus illegal if other threads ever existed (or if we're called from a signal handler). All semi-new programs (which might use threads) are responsible for not introducing such bugs.
And if we're the only thread (and not in a signal handler), which all sufficiently-old programs (including shells) will be, it's the caller's responsibility to not call fork() while holding a mutex.
Thus, one way or another, there's really no need to do this. From an application perspective, it's unnecessary, and from a libc perspective it is at most a QoI feature (one which should be limited to a very small number of locks at most, and frankly we've strongly learned should just detect the error (which is a simple matter of 2 thread-local counters - one for "number of mutexes held" and one for "number of async-signal contexts entered" - as well as knowledge of whether other threads exist) rather than trying to continue in the face of UB).
Heck, runtimes like Python (and Node, if jart is correct in GGP) ship with atfork handlers specifically for the purpose of massaging runtime-created mutexes/condvars/etc. I don’t think those are there because the language maintainers like breaking the rules; they’re probably there because users kept depending on this behavior in spite of cautionary statements.
You think wrong.
You can even use prctl(2) to set the parent as a subreaper for the spawner’s children so the process hierarchy continues to behave as expected and you don’t end up with orphans. Or just run such that the parent is PID 1 in a container anyway.
With either approach, watch out if you’re using a thicker runtime/language whose child process accounting can get freaked out if an unexpectedly-present child indicates ready-for-reap. Using pidfds on Linux (optionally combined with fdpass via UNIX sockets to signal that the spawner did its thing) can help with this.
Egads, this reminds me of when I used to muck with Cygwin. fork() is quite a complex bit of kernel magic, and that’s on kernels designed for it. Simulating fork() on Windows well enough to be usable has always struck me as quite impressive.
But it’s also ridiculous. Forking to spawn a process is an almost comically poor design, and this is one of maybe two things IMO that Windows got right! CreateProcess() may be ugly, its arguments may involve nasty data structures, the command line string may be a disaster that continues to cause problems to this day, but it’s a far superior design for spawning processes. I hope that no one who cares even slightly about process spawning performance on Windows tries to do it using fork.
Not that Windows has ever been particularly fast at spawning processes…
Did you happen to look at how the load on the 8 cores looked at any given time?
Sorry, this is all from memory of just having `btop` open for some of the benchmarks. Maybe in a future post I will dig into CPU utlization more.
In the meantime the discussion on lobste.rs includes some lower-level speculation: https://lobste.rs/s/tr8ozm/why_is_spawning_new_process_node_...
[1] https://docs.deno.com/runtime/manual/node/compatibility/
[2] https://github.com/oven-sh/bun/pull/11492
Is that…intentional on Unix’s part? Seems kind of a weird thing to implement.
so this is opening a unix socket with a no name(the initial null terminates the C style string) thus leaving the only link to it as the file discripter.
One trick I used to do was to sneak another link to the file to keep it around before the process ended , The serious real use case for this was if you had accidentally deleted a file but some process was keeping it open somewhere, but I mainly used it back when youtube was flash based it would try and keep the video cache file hidden this way. and you could make a copy of the video by snaging the correct file descriptor.
I don't remember the FB account credentials, but I received thousands of messages from people either telling me I was a terrible person, or trying to bribe me so they could beat their sibling... all for a simple helicopter game.
Fun times.