He misses the main problem with fork which is poor interaction with threads. Amongst the things you cannot do between the fork and the exec is malloc: another thread in the parent might be in the middle of malloc, might have grabbed a lock needed by malloc, then that thread disappears after fork but the lock is still held by the new child process which deadlocks in malloc.
This and other things are a massive cause of trouble for libraries that need to run subprocesses but need to also link to main programs that might use threads (such as libvirt).
For most more complex programs and libraries, you have no control over when they malloc(). If you spawn a thread and run any library function in it, it may malloc().
Doesn't this mean that any program written in any programming language that supports multiple threads and spawns subprocesses is broken (it is racy for deadlocks, if the spawned process also wants to malloc)?
(EDIT: I misunderstood what the parent said, so it's not so bad. See my comment below.)
Is this problem discussed anywhere? Are there known ways how people handle this problem?
> Doesn't this mean that any program written in any programming language that supports multiple threads and spawns subprocesses is broken (it is racy for deadlocks, if the spawned process also wants to malloc)?
Yes.
> Are there known ways how people handle this problem?
The usual way is to ensure that after forking you only execute async-signal-safe code. Which includes exec(). So if you have more complicated logic which cannot be made async-signal-safe, you exec() an executable which performs that logic.
This is a massive pain though. For example setenv() is not async-signal-safe so if I want to exec() something with specific env vars then I have to go through /usr/bin/env. Sigh.
> The usual way is to ensure that after forking you only execute async-signal-safe code.
Reading your comment I realised I misunderstood what the parent said. This misunderstanding is what made it a suprise for me.
Critically, I missed "... between the fork and the exec ..." and only considered the statement "the lock is still held by the new child process which deadlocks in malloc", assuming this was somehow the case even after the exec(), which would be a huge problem if even the exec()ed child wouldn't be allowed to malloc.
Of course on second thought this wrong assumption is nonsensical: When you exec(), you get a new process space, so any lock potentially thought up by libc or anything else, will be gone.
So all is fine:
For the purpose of starting another executable / implementing a popen(), one just must assure to not malloc (and as you say, in general write signal-handler-safe code) between fork and exec. This is feasible, and I suppose most programming languages do ensure that.
Right, however even this is harder than it looks. For example, what happens if exec fails? How do you print an error? Answer: very carefully - certainly not with printf!
What if the exec succeeds? The pipe is never closed.
If you mark the pipe as O_CLOEXEC and rely on the parent treating EOF as success: what if the child crashes prematurely before exec? E.g. killed by sysadmin, or by OOM killer.
Unless you are trying to implement systemd's "forking" readiness protocol for dæmons, in which case you have to adopt all of these problematic program design items.
the "forking" 'readiness protocol' exists for backwards compatibility and interop, because it represents the behavior of many daemons designed for sysvinit style daemon management, or at least the behaviour they should have in order to avoid races between services in that model. It was not designed by systemd nor is recommended by them.
The "forking" readiness protocol precisely mimics daemon interaction from an old init script, and grants you all the certainty of readiness and information of state that daemonization is capable of giving you: None.
If you try to implement the forking readiness protocol (you mentioned that doing so would be problematic), you are doing something wrong. It only exists for compatibility with existing legacy services, which implement behavior that has been a bad idea and problematic since the 80's.
I'm not personally a systemd fan, but no "forking" readiness problems can be attributed to systemd.
No, it does not. I've pointed you at an explanation of how it does not twice now already, as well as laid it out explicitly right at the start of this thread by explaining what behaviours that programs avoid one has to actually instead adopt wholesale. I've also pointed out pretty clearly that next to nothing speaks this protocol, which would clearly not be the case if it were the behaviour of things that do. Please read properly.
If you think "forking" is a broken readiness protocol, than you have misunderstood what the forking type is.
If you instead of stopping at the title read the actual post you linked, you'd notice that it explains that the issue lies not in the forking protocol, but in that legacy applications cannot signal readiness. See the third and fourth paragraph in the "No-one speaks the forking protocol" chapter. It also explains how daemonizing is a bad idea altogether, but that's separate.
The essence here is that init systems predating upstart/systemd/... did not have a readiness protocol. Instead, they generally assumed that a daemonized service was immediately available after the parent exited, as that was when the script continued. Forking is meant to pug
The "forking" protocol is only meant to plug such an application into systemd, nothing more. It grants you zero readiness information (just like when it was started by a sysvinit script), as such information does not exist. It would be stupid to try to implement this protocol. It is nothing but a compatibility layer.
If you have an actual counter-argument (so far you have only posted a link, and then restated that you posted that link several times), please present it. I don't mind being proven wrong.
Why would you do that? If you're designing for systemd, go for "simple" (that is, don't fork), or if you need to signal when your service is truly ready rather than just monitor its lifetime, go for "notify" (or "dbus" if you're into that sort of thing).
Using "forking" as readiness protocol doesn't make sense. Unlike "notify", it doesn't signal readiness. It's only there to try to understand old services that always daemonize, which was only done because the old init systems required it. Daemonizing is pointless under systemd or other init systems that handle this themselves.
Maybe a tangent, but I never really understood how the memory pointed to by envp is supposed to be managed. Does fork() copy the allocator state such that you can just do malloc(), fork(), then free() in the parent process, then execve() blows away the whole memory map in the child?
Yes, the flow you describe is pretty much how it goes. The OS will "overlay" the fork'ed child process if exec succeeds in loading the program specified. If not, errno is set and typically the child process reports the problem (using something like strerror_r[0]) then exits.
In either case, the parent process can free the memory allocated for the execve call once it returns.
Nothing special is needed here. execve() does blow away the address space of the process and replaces it. As to fork(), it copies the address space. Of course, you can't putenv() on the child-side of fork() if the parent was multi-threaded, and you may not even be able to getenv() in that case either (check the standards and your C library's fork() man page).
It sounds to me like fork() is unable to handle threads, unlike clone(). I would therefore say that fork() is to blame. Of course, it is a bit of nitpicking, as the problem is that they do not work well when combined.
That's the point of clone(), but the point is that the problems of fork() are with its semantics. Even with clone(), if you want to spawn a new process then your best bet is to go with vfork() semantics -- in principle you could create a thread-like process sharing the parent's address space that can then exec() without affecting the parent, but in practice you can only do this inside a C library because the protocol for setting up TLS (for errno) is underdocumented and requires knowledge of and access to your C library's internals.
That's relatively non-problematic: setup everything you need, fork(), exec() or _exit().
But there's lots of other issues. Things like PID number reuse, libraries in single-threaded programs that may need to reinitialize on fork() (via pthread_atfork()? no because fork() might have been called directly via syscall() or by trapping from asm code, bypassing the C library altogether), and so on.
Just use posix_spawn().
EDIT: Also, ultimately, the worst thing about fork() is that it can't possibly be high-performance. Either fork() copies the writable parts of the address-space, or sets up COW, or copies the RSS and sets up COW for the rest. Well, on modern CPUs COW is really expensive. And copying a JVM is no fun either.
I have always wondered about Linux's overcommit feature, which BTW I dislike -- Doesn't ISO C require that if malloc returns a nonzero pointer, it has to backed physically (need not be RAM, could be swap on disk or some other thing). I have been told that the OS is allowed to overcommit but space obtained through malloc needs to be backed, if it has to comply with ISO C. Apparently AIX and ISO committee got into some argument about this when AIX's malloc implementation allowed for overcommit, anyone remembers this ?
You can always disable it using 'sysctl -w vm.overcommit_memory=2' or have more control with vm.overcommit_* various sysctls.
In the general case, I found that overcommit is usually more useful than strict accounting, but I agree that for more industrial cases it can be the opposite (but then you can spend time tuning it).
Sort of a chicken and egg thing. If the API and behavior makes it easy to abuse, people will abuse it. It might have helped if there were two different library functions (i) malloc that gave you physically backed memory and (ii) another that just reserved some address space.
Unless you have vm.overcommit_memory set to 1, then AIUI you can implement malloc s.t. a read/write to the returned memory won't generate a SIGSEGV, e.g. using mmap sans MAP_NORESERVE. Even if vm.overcommit_memory=1, you can still (I assume) do it by simply having your malloc implementation populate each page.
That said, as long as "too small to fail" (https://lwn.net/Articles/627419/) is around (which is presumably "forever"), the oom-killer won't be going away. Fork obviously compounds this massively due to copy-on-write. So ensuring your malloc'd memory is actually writable may just make you a more tempting target for the oom-killer.
> Doesn't ISO C require that if malloc returns a nonzero pointer, it has to backed physically (need not be RAM, could be swap on disk or some other thing)
I don't see anything like that, though admittedly I don't have a "real" copy of the standard. I'd also be at least a little surprised if it were true (though my being surprised by the C standard would...actually not be surprising). While the "spirit of the law" would probably make sense, the "letter of the law" would be difficult. Consider e.g. an otherwise side-effect free function with a statically computable result assuming a successful malloc (and corresponding free), could the compiler optimize away the malloc call?
If it doesn't exist already, there should be a "standard spawn stub" that, in absence of spawn it creates a new spawner process at the very beginning of an application start up, so when you do spawn, the spawner process is the one that forks with none of the copied memory and open file descriptors.
> That this nearly 50 year old crappy design choice has come to this astonishes me.
That makes me feel old. For me Unix was always invented 30-something years ago...
The concept of Fork was not designed to "spawn new programs", it was created to allow parallelism on a single program with several processes collaborating on the same task. It's even in the name (i.e. you fork the program in two, not spawn a new process).
For this purpose, the choice to start with the same data structures and the same execution point is quite sensible - it's essentially what happens when you create a thread nowadays. So, the hypothetical discussion where someone decided to copy the address space in order to launch new programs, would never have happened.
The day where someone decided to reuse the existing fork code to launch processes for new programs, that one must have been worth to watch.
> it's essentially what happens when you create a thread nowadays
I'm not familiar with the guts of threading libs, but AFAIK the syscall that spawns threads is clone(), which takes a function pointer and a stack pointer, so it very much does not start "with the same data structures and the same execution point".
You're talking about the C library function clone(). The actual system call "clone" on Linux doesn't care about functions, or stack pointers, it just sets a flag like the system call fork did. Everything else is implemented in the library by checking the return code.
This is all fine, there are two scenarios, neither of which is the imagined "race" described above:
1. "Fork-and-exec" Our new execution context will actually run a different program. We immediately do so, the new program doesn't care about most of the state of the old program, it throws that state away, including locks that were held by the old program's allocator.
2. "New Thread" Our new context will share all the state, if another thread has a process-wide lock for some reason, it can and should apply to us too. If it took a thread-specific lock, we're not that thread, so we already don't care.
> You're talking about the C library function clone().
Huh? I saw that the manpage for clone() is in section 2, which AFAIK means it's kernel API, whereas C library functions are usually in section 3. Am I mistaken?
Even syscalls(2) refers to clone(2) in its list of syscalls.
You are correct that the manual section 2 is "for" system calls, but in practice today these are just the lowest level of the C library, and not the actual system calls.
My copy of the clone(2) manual page hints at this by commenting
/* For the prototype of the raw system call, see NOTES */
and more explicitly in its body text by saying "The main text describes the wrapper function; the differences for the raw system call are described toward the end of this page."
Other modern stuff is the same way, the section 2 documentation is describing what an ordinary C programmer needs to know, but if you're say, implementing the C standard library or writing a compiler for a language that runs on the bare metal you need to read the kernel documentation which is not provided as manual pages.
> The concept of Fork was not designed to "spawn new programs"
I thought the fork syscall was the only way to spawn new programs in earlier versions of Unix, and that the spawn syscall only came later. Is that not right? If there was no other syscall, fork must have been designed for this purpose.
That's false teleology. Not everything we end up with was intentionally designed to end there. Maybe it did not occur to the designers that you would want to create new processes for anything but parallelism; or that if someone did, they would invent a new mechanism for that, instead of deciding it was convenient to re-use and extend the fork mechanism.
> Maybe it did not occur to the designers that you would want to create new processes for anything but parallelism
Parallelism? In the original unix? You're talking absolute nonsense. These were single socket, single core systems. How on earth do you think it was designed for parallelism?
But we don't even need to debate this - read the 1971 manual.
The OP's argument that fork doesn't make a lot of sense for what it gets used for seems persuasive, I'm definitely curious how it ended up that way. This was relatively early days for multi-tasking (ie "time-sharing") at all, maybe the only explanation is that it was early days and it wasn't clear how it was going to end up being used and 'fork' was convenient to implement. Or that's how the previous time-sharing OSs did it, for equally mysterious reasons, and it was copied? I dunno.
The suggestion that it was done this way because the expected main use case was a fork without an exec at least provides a rational explanation (I don't think 'threads' as distinct from processes existed yet in unix? a fork without an exec was the closest thing you had), but I have no idea if it's historically accurate. Would be interesting to find people who were around and get some background.
Ah, this is interesting:
> Then came Unix, in the early 1970s. The Unix notion of a “process” became a sequential thread of control plus a virtual address space (incidentally, the Unix notion of a process derived directly from the Multics process design [Saltzer, 66]). So “processes”, in the Unix sense, are quite heavyweight machines. Since they cannot share memory (each has its own address space), they interact through pipes, signals, etc). Shared memory (also a rather ponderous mechanism) was added much later.
> After some time, Unix users started to miss the old processes that could share memory. This led to the “invention” of threads: old-style processes that shared the address space of a single Unix process. They also were called “lightweight”, by way of contrast with “heavyweight” Unix processes. This distinction dates back to the very late 1970s or early 1980s, i.e. to the first “microkernels” (Thoth (precursor of the V-kernel and QNX), Amoeba, Chorus, the RIG-Accent-Mach family, etc).
There is no way parallelism was the reason for fork. It was a PDP-7, so there was no parallelism in the first place. And even worse, I believe only one user-mode process was in memory at a time; any process switch meant swapping out the current process and swapping in a new one.
There was also no form of interprocess communication other than the filesystem, nor any synchronization other than waiting for a specific process to exit. So having multiple processes work on the same problem would mostly have been very awkward.
> The concept of Fork was not designed to "spawn new programs", it was created to allow parallelism on a single program with several processes collaborating on the same task. It's even in the name (i.e. you fork the program in two, not spawn a new process).
I really doubt this. Is there an instance of a program with 1st edition lineage that use processes to achieve concurrency? There were pipes, sure, but that could be achieved with a spawn-like call.
I don’t see processes being used for concurrency in UNIX history until BSD sockets and forking servers emerged.
Well, yes, fork()/exec() and spawn-like things are capable of doing the same things, more or less.
On the other hand, shell pipelines were used for concurrency before BSD added networking code. And I have doubts that shell pipelines would have developed without the fork()/exec() model---see the author's Windows NT (and earlier) with really, seriously broken shell models.
Whatever the original purpose, the purposes it actually served were: a) spawning new processes (fork() then exec()) with the spawn code implemented in user-land, thus enabling agile shell development, and b) parallelism.
I would argue that (a) was easily the most important aspect of fork(). One can always get (b) by spawning more instances of the same program and passing them whatever state is to be shared. But (a) needs a decent API and to be kernel-side if one has no fork(). Yes, posix_spawn() is better, but fork() enabled Unix to kick the can of standardizing a spawn API that doesn't suck until enough experience was had with it to make such a thing.
Really POSIX should just acquire CreateProcess and have done with the problem. It's the API you generally want when starting a new, unrelated program. Probably for POSIX you'd want to allow specifying which handles become the new process stdin/stdout/stderr, since Windows handles consoles in a stranger, less convenient way.
What happens if you want the child to inherit other fds?
Or set other attributes such as the session, euid, egid etc?
The elegance of fork() is in avoiding the complexity of requiring every attribute of the process to be explicitly stated in order to create a new process. We simply inherit our parent's attributes and if that turns out not be desirable it is fixed in the child with the usual syscalls.
I'm not positive that it's elegant. It may also be considered an ugly hack (not sure either), and I'm pretty certain it's been responsible for many security holes.
It would be interesting to see in what ways Unix (especially shell scripts) would be different if processes would not inherit all the baggage by default.
Attributes is just another word for "global state". When we speak of elegance in computing, it often means functional programming languages where eradicating global state is the primary motivation.
So really fork is the opposite of elegance. It makes state implicit, not explicit.
Your "avoiding the complexity" in fact turns into its mirror image: in order to ensure sane behaviour, you have to have a preamble to avoid inheriting things you don't want. This results in people writing things like http://www.linuxprogrammingblog.com/threads-and-fork-think-t... : the list of things you have to remember not to use in order to use the "elegant" API.
That may have been fine in 1970, when there was very little to inherit (no thread handles, no named pipes, no semaphores, IIRC) and security wasn’t a big concern.
Nowadays, I think ‘whitelisting’ what the new process can do is the choice to make, not forking and then (hopefully) ‘blacklisting’ what you don’t want (that’s especially important if you eventually will be running code you didn’t write or, maybe, don’t even have source for)
That also is easier to test for. If you forget to specify a capability before forking, the bugs you see will be better reproducible than when you forget a thing you don’t need.
I can’t find it now, but try googling an article on how to properly fork a process nowadays. It is insanely difficult to do right.
It doesn't matter which thing you do here. Almost all of NT is like that, if there's a feature you're sure NT has that Linux doesn't have, or vice versa, usually either you're wrong or the relevant team is working frantically to correct this mistake.
fork()/clone() obliges you to track which things the child doesn't want from the parent, and have a way to override them
CreateProcess obliges you to track which things the child DOES want from the parent, and have a way to override them.
There's a convergent evolution at work, the same way bats and birds have both got wings without a common ancestor having wings, wings work, not quite the _same_ wings, but the idea works, so it's inevitable.
futex() and WaitOnAddress have the same funhouse mirror effect. They're doing the same exact thing (very, very cheap synchronisation primitive, with just barely enough features to be useful), the names and descriptions are really different, but if you use both you're like "Oh, these are basically the same, huh". Both names are equally useless, futex() is a clever name if you already know what it does, but until you do you've no idea what to look for. On the other hand WaitOnAddress doesn't actually er, wait on an address, that would be expensive, like futex waking it up when something changes is _your_ job as the programmer.
Yes, but the distinction between "must enumerate things to child" and "child inherits everything" isn't symmetrical. I believe that it's better for security and reliability if the list is made explicit, but evidently not everyone agrees.
WIN32 has a number of far, far superior features though, such as WaitForMultipleObjects(), Security Descriptors and Access Tokens, and SIDs. And its event system is like *BSD kqueue and Illumos event ports (i.e., it doesn't suck), while Linux's epoll() is... well, a disaster.
"What if we copied the entire address space of the program into a new process running from the same spot"
This was actually the cause of 1-2 second blocks in our Node event loop. We were spawning a clamd scan asynchronously, from a process with several gigabytes of RSS. I never would have imagined that fork would blindly copy gigabytes of RSS every invocation. I should have known. It took a few months to track down.
The kicker is that Node calls fork from the main event loop thread and not from the thread pool, in order to hack around the GC, if I understand correctly.
fork() doesn't copy the blocks -- they're CoW -- but rather the time it takes is in setting up the CoW table. You can mitigate this if you tell clone(2) that the only thing you are going to do right after creating a new process is execve(2) so don't bother actually creating that. I see that mentioned at the end, but with the question of whether the setup is also combinable (it probably is).
I'm not quite sure, but I think I remember reading that CoW (Copy on Write) was not there in the earliest Unixes. Might have been implemented at Berkeley some time later. Seem to remember reading this some years ago when reading about fork, because without CoW, fork's copying was expensive, particularly if you were going to exec another program in the child just after forking.
fork has two uses. One is to run a bunch of child processes that the parent can control and hand work to. Think web workers in apache2 with the parent accepting connections and the children processing the requests.
The other is to start unrelated programs. Think bash which can start vim.
I think for the first use case, it is stupid to have to reap the child process or else it gets inherited by init. Instead you should be able to mark that process as a worker from the get go. It doesn't make sense for apache2 workers to keep running if the parent process died.
In the latter use case, I agree with the OP. I'd rather set up the structure of what I want to happen, then start that process as a completely independent thing. spawn seems like the way to go for this.
The good news is that we can totally have all those options as long as we keep fork as is at least for the next 50 years :)
"Fork is the direct cause of the stupidest component I’ve ever heard of in an operating system: the out-of-memory (aka OOM) killer."
Well, technically, no.
Originally (1st Edition?), IIUC, fork() would fail if enough memory to copy the process wasn't free. Then, when fork() was changed (4th Edition?) to read-only share the address space, that wouldn't happen, but the exec() might fail if there wasn't enough space to set up the child. But those were all doing segmented memory so they're completely irrelevant to modern systems.
The OOM-killer comes with (IIRC) BSD's paged virtual memory. Some bright spark (yes, I have hated this design since I first tripped over it in the early- to mid-'90s) had the brilliant idea of supporting sparse matrices by not requiring allocated memory to be backed by valid pages, either RAM or disk.
If you don't require allocated memory to be backed, you can allocate a giant block of memory; if you never touch most of it, everything will be copacetic. It's good, I was told ca. 1992, for scientific programs in, like, FORTRAN. Yay. Unfortunately, lots of programs started using the feature, including the X server.
(Why do I hate with the fiery passion of a thousand suns the perverted genius who came up with this? AIX 3.2.5/4.1. Running the machine out of memory (remember, allocations don't fail, everything chugs along until some proggie touches an address and the OS can't realize the page) would kick off the OOM-killer, which would immediately hit inetd in the head with its little silver hammer, resulting in a machine that would still be working, -ish, sort-of, but unless you were logged in somewhere already to reboot it, you would have to go visit the physical console.)
Turning off the OOM-killer and overallocation policy was possible, but not advised, since many programs, including the X server, would suddenly use a truly atrocious amount of memory.
TL;DR: Memory overcommitment doesn't have anything to do with fork()/exec(), but rather is a deliberate design decision made with the addition of paged virtual memory, to support a limited class of applications, that probably didn't work very well even for that class.
My favorite part of fork+exec is the _temporary_ exhaustion of virtual address space on a heavily loaded system, where the parent process is using a shitton of memory. 50G forked is instantly 100G of virtual address space, even if it is followed by an exec. Couple this with sysadmins not configuring swap files means that virtual address space == physical address space. On a heavily loaded system, high memory using processes can't even launch simple utilities w/o incurring the wrath of the OOMK. Configure swap, even if it will never be used, it still affects the size of your virtual address space.
That really depends on your overcommit settings. It seems that you describe an overcommit = 2 case, when the total VIRT for all processes will not exceed swap + memory * overcommit_ratio%. But that's not default mode (default is overcommit = 0). And you can mitigate this issue in other ways (as enabling swap can lead into some very bad situations when _physical_ memory gets exhausted):
- set overcommit = 1, OOM now comes only when sum(RSS) exhausts swap + memory
- just increase overcommit_ratio if you want to stay under overcommit =2
I see a few people complaining about oom killing. Allow me to make a case for it. I think oom killing is a very pragmatic solution to deal with low memory. Without overcommit you don't need the oom killer. But just simply disabling overcommit will make your system unusable during low memory situation just as much. Imagine that one job ate all your memory. How do you restore your system? You cannot allocate more memory so you can no longer start up a new shell to kill the bad process. So you are just as stuck with an unusable system. I wrote about this here: https://superuser.com/a/984715
What I really want is to have an always responsive system. You cannot get this easily on linux because it doesn't allow to disable all paging (executable pages are still paged out even with swap disabled). This means you can easily lock up your machine if you use a lot of memory (even without swap -- I don't use swap). The above post contains a 20 line C program example that will break your system that you can only cure by applying the oom killer. I mean what other options do you have you have 0 free memory? Could a kernel hacker reading this add such an option to linux to a poor sysadmin like myself?
Imho what the author is either missing on voluntarily ignoring is the context in which the original Unix was developed: as an unofficial, simpler version of Multics, after the Multics project had failed.
So my guess is that Thompson and co basically went with a "worse is better" approach and delivered a model that performed worse but was a lot simpler to implement and reason about.
Also... It's easy to criticize designs 40 years after. That time was a time of proposals an research. Despite this, the original Unix design, no matter how flawed, stood the test of time.
91 comments
[ 4.0 ms ] story [ 278 ms ] threadThis and other things are a massive cause of trouble for libraries that need to run subprocesses but need to also link to main programs that might use threads (such as libvirt).
For most more complex programs and libraries, you have no control over when they malloc(). If you spawn a thread and run any library function in it, it may malloc().
Doesn't this mean that any program written in any programming language that supports multiple threads and spawns subprocesses is broken (it is racy for deadlocks, if the spawned process also wants to malloc)?
(EDIT: I misunderstood what the parent said, so it's not so bad. See my comment below.)
Is this problem discussed anywhere? Are there known ways how people handle this problem?
Yes.
> Are there known ways how people handle this problem?
The usual way is to ensure that after forking you only execute async-signal-safe code. Which includes exec(). So if you have more complicated logic which cannot be made async-signal-safe, you exec() an executable which performs that logic.
This is a massive pain though. For example setenv() is not async-signal-safe so if I want to exec() something with specific env vars then I have to go through /usr/bin/env. Sigh.
Reading your comment I realised I misunderstood what the parent said. This misunderstanding is what made it a suprise for me.
Critically, I missed "... between the fork and the exec ..." and only considered the statement "the lock is still held by the new child process which deadlocks in malloc", assuming this was somehow the case even after the exec(), which would be a huge problem if even the exec()ed child wouldn't be allowed to malloc.
Of course on second thought this wrong assumption is nonsensical: When you exec(), you get a new process space, so any lock potentially thought up by libc or anything else, will be gone.
So all is fine:
For the purpose of starting another executable / implementing a popen(), one just must assure to not malloc (and as you say, in general write signal-handler-safe code) between fork and exec. This is feasible, and I suppose most programming languages do ensure that.
Instead what I'd recommend is that you have a pipe() for passing an error message (one byte will do) back to the parent, and then _exit().
What if the exec succeeds? The pipe is never closed.
If you mark the pipe as O_CLOEXEC and rely on the parent treating EOF as success: what if the child crashes prematurely before exec? E.g. killed by sysadmin, or by OOM killer.
What if a write() to the pipe fails?
- Don't thread before you fork.
- Don't lock before you fork.
- Don't open before you fork.
- Don't do anything that is not meant to be a shared resource before you fork.
Locks and threads is the worst. Locks can end up in weird states.
* http://jdebp.eu./FGA/unix-daemon-readiness-protocol-problems...
It is a compatibility mode, and it is broken beyond repair because the technique itself is broken beyond repair.
If you try to implement the forking readiness protocol (you mentioned that doing so would be problematic), you are doing something wrong. It only exists for compatibility with existing legacy services, which implement behavior that has been a bad idea and problematic since the 80's.
I'm not personally a systemd fan, but no "forking" readiness problems can be attributed to systemd.
If you instead of stopping at the title read the actual post you linked, you'd notice that it explains that the issue lies not in the forking protocol, but in that legacy applications cannot signal readiness. See the third and fourth paragraph in the "No-one speaks the forking protocol" chapter. It also explains how daemonizing is a bad idea altogether, but that's separate.
The essence here is that init systems predating upstart/systemd/... did not have a readiness protocol. Instead, they generally assumed that a daemonized service was immediately available after the parent exited, as that was when the script continued. Forking is meant to pug
The "forking" protocol is only meant to plug such an application into systemd, nothing more. It grants you zero readiness information (just like when it was started by a sysvinit script), as such information does not exist. It would be stupid to try to implement this protocol. It is nothing but a compatibility layer.
If you have an actual counter-argument (so far you have only posted a link, and then restated that you posted that link several times), please present it. I don't mind being proven wrong.
Using "forking" as readiness protocol doesn't make sense. Unlike "notify", it doesn't signal readiness. It's only there to try to understand old services that always daemonize, which was only done because the old init systems required it. Daemonizing is pointless under systemd or other init systems that handle this themselves.
0 - https://www.freebsd.org/cgi/man.cgi?query=execve&apropos=0&s...
In either case, the parent process can free the memory allocated for the execve call once it returns.
0 - https://www.freebsd.org/cgi/man.cgi?query=strerror_r&apropos...
* https://cr.yp.to/lib/pathexec.html
That's a problem with threads, not fork(). It generalizes to all sorts of things needing to know about internals of threads and locks.
But there's lots of other issues. Things like PID number reuse, libraries in single-threaded programs that may need to reinitialize on fork() (via pthread_atfork()? no because fork() might have been called directly via syscall() or by trapping from asm code, bypassing the C library altogether), and so on.
Just use posix_spawn().
EDIT: Also, ultimately, the worst thing about fork() is that it can't possibly be high-performance. Either fork() copies the writable parts of the address-space, or sets up COW, or copies the RSS and sets up COW for the rest. Well, on modern CPUs COW is really expensive. And copying a JVM is no fun either.
I understand why this behavior is default, if you _have_ to support fork() this perhaps the more pragmatic default.
That is a strong claim.
That said, as long as "too small to fail" (https://lwn.net/Articles/627419/) is around (which is presumably "forever"), the oom-killer won't be going away. Fork obviously compounds this massively due to copy-on-write. So ensuring your malloc'd memory is actually writable may just make you a more tempting target for the oom-killer.
> Doesn't ISO C require that if malloc returns a nonzero pointer, it has to backed physically (need not be RAM, could be swap on disk or some other thing)
I don't see anything like that, though admittedly I don't have a "real" copy of the standard. I'd also be at least a little surprised if it were true (though my being surprised by the C standard would...actually not be surprising). While the "spirit of the law" would probably make sense, the "letter of the law" would be difficult. Consider e.g. an otherwise side-effect free function with a statically computable result assuming a successful malloc (and corresponding free), could the compiler optimize away the malloc call?
> That this nearly 50 year old crappy design choice has come to this astonishes me.
That makes me feel old. For me Unix was always invented 30-something years ago...
For this purpose, the choice to start with the same data structures and the same execution point is quite sensible - it's essentially what happens when you create a thread nowadays. So, the hypothetical discussion where someone decided to copy the address space in order to launch new programs, would never have happened.
The day where someone decided to reuse the existing fork code to launch processes for new programs, that one must have been worth to watch.
I'm not familiar with the guts of threading libs, but AFAIK the syscall that spawns threads is clone(), which takes a function pointer and a stack pointer, so it very much does not start "with the same data structures and the same execution point".
This is all fine, there are two scenarios, neither of which is the imagined "race" described above:
1. "Fork-and-exec" Our new execution context will actually run a different program. We immediately do so, the new program doesn't care about most of the state of the old program, it throws that state away, including locks that were held by the old program's allocator.
2. "New Thread" Our new context will share all the state, if another thread has a process-wide lock for some reason, it can and should apply to us too. If it took a thread-specific lock, we're not that thread, so we already don't care.
Huh? I saw that the manpage for clone() is in section 2, which AFAIK means it's kernel API, whereas C library functions are usually in section 3. Am I mistaken?
Even syscalls(2) refers to clone(2) in its list of syscalls.
You are correct that the manual section 2 is "for" system calls, but in practice today these are just the lowest level of the C library, and not the actual system calls.
My copy of the clone(2) manual page hints at this by commenting /* For the prototype of the raw system call, see NOTES */
and more explicitly in its body text by saying "The main text describes the wrapper function; the differences for the raw system call are described toward the end of this page."
Other modern stuff is the same way, the section 2 documentation is describing what an ordinary C programmer needs to know, but if you're say, implementing the C standard library or writing a compiler for a language that runs on the bare metal you need to read the kernel documentation which is not provided as manual pages.
I thought the fork syscall was the only way to spawn new programs in earlier versions of Unix, and that the spawn syscall only came later. Is that not right? If there was no other syscall, fork must have been designed for this purpose.
Parallelism? In the original unix? You're talking absolute nonsense. These were single socket, single core systems. How on earth do you think it was designed for parallelism?
But we don't even need to debate this - read the 1971 manual.
https://www.bell-labs.com/usr/dmr/www/pdfs/man21.pdf
"fork is the only way new processes are created"
exec is also in there, and refers to fork and describes how to use them to run programs.
The suggestion that it was done this way because the expected main use case was a fork without an exec at least provides a rational explanation (I don't think 'threads' as distinct from processes existed yet in unix? a fork without an exec was the closest thing you had), but I have no idea if it's historically accurate. Would be interesting to find people who were around and get some background.
Ah, this is interesting:
> Then came Unix, in the early 1970s. The Unix notion of a “process” became a sequential thread of control plus a virtual address space (incidentally, the Unix notion of a process derived directly from the Multics process design [Saltzer, 66]). So “processes”, in the Unix sense, are quite heavyweight machines. Since they cannot share memory (each has its own address space), they interact through pipes, signals, etc). Shared memory (also a rather ponderous mechanism) was added much later.
> After some time, Unix users started to miss the old processes that could share memory. This led to the “invention” of threads: old-style processes that shared the address space of a single Unix process. They also were called “lightweight”, by way of contrast with “heavyweight” Unix processes. This distinction dates back to the very late 1970s or early 1980s, i.e. to the first “microkernels” (Thoth (precursor of the V-kernel and QNX), Amoeba, Chorus, the RIG-Accent-Mach family, etc).
http://www.serpentine.com/blog/threads-faq/the-history-of-th...
Not totally sure what light that sheds on the question, if any, heh.
There was also no form of interprocess communication other than the filesystem, nor any synchronization other than waiting for a specific process to exit. So having multiple processes work on the same problem would mostly have been very awkward.
Until Doug McIlroy invented pipes. And the modern Unix shell pipeline style of creating complex programs.
I really doubt this. Is there an instance of a program with 1st edition lineage that use processes to achieve concurrency? There were pipes, sure, but that could be achieved with a spawn-like call.
I don’t see processes being used for concurrency in UNIX history until BSD sockets and forking servers emerged.
On the other hand, shell pipelines were used for concurrency before BSD added networking code. And I have doubts that shell pipelines would have developed without the fork()/exec() model---see the author's Windows NT (and earlier) with really, seriously broken shell models.
I would argue that (a) was easily the most important aspect of fork(). One can always get (b) by spawning more instances of the same program and passing them whatever state is to be shared. But (a) needs a decent API and to be kernel-side if one has no fork(). Yes, posix_spawn() is better, but fork() enabled Unix to kick the can of standardizing a spawn API that doesn't suck until enough experience was had with it to make such a thing.
https://msdn.microsoft.com/en-us/library/windows/desktop/ms6...
The elegance of fork() is in avoiding the complexity of requiring every attribute of the process to be explicitly stated in order to create a new process. We simply inherit our parent's attributes and if that turns out not be desirable it is fixed in the child with the usual syscalls.
EDIT: s/And/Or set/
It would be interesting to see in what ways Unix (especially shell scripts) would be different if processes would not inherit all the baggage by default.
So really fork is the opposite of elegance. It makes state implicit, not explicit.
Nowadays, I think ‘whitelisting’ what the new process can do is the choice to make, not forking and then (hopefully) ‘blacklisting’ what you don’t want (that’s especially important if you eventually will be running code you didn’t write or, maybe, don’t even have source for)
That also is easier to test for. If you forget to specify a capability before forking, the bugs you see will be better reproducible than when you forget a thing you don’t need.
I can’t find it now, but try googling an article on how to properly fork a process nowadays. It is insanely difficult to do right.
[0] http://pubs.opengroup.org/onlinepubs/009695399/functions/pos...
fork()/clone() obliges you to track which things the child doesn't want from the parent, and have a way to override them
CreateProcess obliges you to track which things the child DOES want from the parent, and have a way to override them.
There's a convergent evolution at work, the same way bats and birds have both got wings without a common ancestor having wings, wings work, not quite the _same_ wings, but the idea works, so it's inevitable.
futex() and WaitOnAddress have the same funhouse mirror effect. They're doing the same exact thing (very, very cheap synchronisation primitive, with just barely enough features to be useful), the names and descriptions are really different, but if you use both you're like "Oh, these are basically the same, huh". Both names are equally useless, futex() is a clever name if you already know what it does, but until you do you've no idea what to look for. On the other hand WaitOnAddress doesn't actually er, wait on an address, that would be expensive, like futex waking it up when something changes is _your_ job as the programmer.
This was actually the cause of 1-2 second blocks in our Node event loop. We were spawning a clamd scan asynchronously, from a process with several gigabytes of RSS. I never would have imagined that fork would blindly copy gigabytes of RSS every invocation. I should have known. It took a few months to track down.
The kicker is that Node calls fork from the main event loop thread and not from the thread pool, in order to hack around the GC, if I understand correctly.
See the relevant Node issue here:
https://github.com/nodejs/node/issues/14917
* https://www.cloudatomiclab.com/antisyscall/ (https://news.ycombinator.com/item?id=15998828)
* https://www.cloudatomiclab.com/prosyscall/ (https://news.ycombinator.com/item?id=15998344)
The other is to start unrelated programs. Think bash which can start vim.
I think for the first use case, it is stupid to have to reap the child process or else it gets inherited by init. Instead you should be able to mark that process as a worker from the get go. It doesn't make sense for apache2 workers to keep running if the parent process died.
In the latter use case, I agree with the OP. I'd rather set up the structure of what I want to happen, then start that process as a completely independent thing. spawn seems like the way to go for this.
The good news is that we can totally have all those options as long as we keep fork as is at least for the next 50 years :)
Well, technically, no.
Originally (1st Edition?), IIUC, fork() would fail if enough memory to copy the process wasn't free. Then, when fork() was changed (4th Edition?) to read-only share the address space, that wouldn't happen, but the exec() might fail if there wasn't enough space to set up the child. But those were all doing segmented memory so they're completely irrelevant to modern systems.
The OOM-killer comes with (IIRC) BSD's paged virtual memory. Some bright spark (yes, I have hated this design since I first tripped over it in the early- to mid-'90s) had the brilliant idea of supporting sparse matrices by not requiring allocated memory to be backed by valid pages, either RAM or disk.
If you don't require allocated memory to be backed, you can allocate a giant block of memory; if you never touch most of it, everything will be copacetic. It's good, I was told ca. 1992, for scientific programs in, like, FORTRAN. Yay. Unfortunately, lots of programs started using the feature, including the X server.
(Why do I hate with the fiery passion of a thousand suns the perverted genius who came up with this? AIX 3.2.5/4.1. Running the machine out of memory (remember, allocations don't fail, everything chugs along until some proggie touches an address and the OS can't realize the page) would kick off the OOM-killer, which would immediately hit inetd in the head with its little silver hammer, resulting in a machine that would still be working, -ish, sort-of, but unless you were logged in somewhere already to reboot it, you would have to go visit the physical console.)
Turning off the OOM-killer and overallocation policy was possible, but not advised, since many programs, including the X server, would suddenly use a truly atrocious amount of memory.
TL;DR: Memory overcommitment doesn't have anything to do with fork()/exec(), but rather is a deliberate design decision made with the addition of paged virtual memory, to support a limited class of applications, that probably didn't work very well even for that class.
https://linux.die.net/man/2/mlock
- set overcommit = 1, OOM now comes only when sum(RSS) exhausts swap + memory
- just increase overcommit_ratio if you want to stay under overcommit =2
What I really want is to have an always responsive system. You cannot get this easily on linux because it doesn't allow to disable all paging (executable pages are still paged out even with swap disabled). This means you can easily lock up your machine if you use a lot of memory (even without swap -- I don't use swap). The above post contains a 20 line C program example that will break your system that you can only cure by applying the oom killer. I mean what other options do you have you have 0 free memory? Could a kernel hacker reading this add such an option to linux to a poor sysadmin like myself?
So my guess is that Thompson and co basically went with a "worse is better" approach and delivered a model that performed worse but was a lot simpler to implement and reason about.
Also... It's easy to criticize designs 40 years after. That time was a time of proposals an research. Despite this, the original Unix design, no matter how flawed, stood the test of time.