> But despite the apparent strangeness, fork() and exec() are actually pretty convenient once you want to do anything slightly more complicated than just launch a new process.
What I don't understand is why there isn't something for the simple case where we do just want to run another process and not set anything else. There is system() which I believe is just a wrapper for fork/exec, and is tempting to use, but the issue is it just takes a string and passes it to sh which opens you up to shell expansion exploits if you have untrusted/improperly sanitized inputs. I wish there was a version of system() that took an array or args like exec instead and you could specify the command to run directly as well, instead of sh.
What you're looking for is essentially posix_spawn, if you set the file_actions and attributes parameters to NULL. Then it becomes a function where you specify the executable, arguments, and environment for the new process, with a single error to process.
As you have yourself said, system() is not a wrapper for fork and exec, but it executes a shell process and it gives to the shell a command to execute, which may be a binary executable or a script.
Fork() and exec() are POSIX functions, while system() is a Standard C function that is available in C regardless of the operating system and regardless whether fork() and exec() exist.
On linux at least, and probably most other unixes, it is implemented with fork/exec. In fact, there are two execs, one to start the shell, and one for the shell to start the actual program. And depending on your command, and the shell implementation you might have another fork as well.
The Standard C function system() is defined to execute a shell process and its semantics do not depend on how a process is executed by the host operating system.
Obviously in any UNIX-derived OS a shell process must be executed via fork and exec, and the shell itself executes children processes also via fork and exec. Under other operating systems no fork or exec are used.
What matters is that system() is not defined as a combination of fork with exec, but it does a more complex operation, with a greater overhead, and which works more or less in the same way under any OS.
As others have mentioned, POSIX does have a wrapper for fork and exec, which is posix_spawn().
> its semantics do not depend on how a process is executed by the host operating system
Implementation details can leak though. For example on unix, if your process uses a lot of memory, a fork can result in an out of memory error, if there isn't enough memory left on the system to copy the parent process memory. And afaict things like how file descriptors are inherited are implementation defined.
> which works more or less in the same way under any OS
Even ignoring subtle differences in how subprocesses are spawned, there is the little issue that shell escaping and argument splitting works differently on different OSes, or even different shells on the same OS.
Because the differences mentioned by you between shells and OSes are unavoidable, the C standard specifies that the interpretation of the character string passed as an argument of the system() function is implementation-defined and it must be documented by any implementation of the standard C library.
This does not change the fact that whether a system() function implementation uses or does not use fork and exec is neither specified by the function definition nor relevant for the user, and in any case system() is not the function that should be used when the intention is to simplify some code that otherwise would invoke fork and exec.
> This does not change the fact that whether a system() function implementation uses or does not use fork and exec is neither specified by the function definition nor relevant for the user
I agree that it isn't specified, but it can be relevant for the user, depending on what the user is doing.
> in any case system() is not the function that should be used when the intention is to simplify some code that otherwise would invoke fork and exec
Which was my original point. system doesn't save you from the problems of fork/exec, because depending on the system, it might be using fork/exec anyway, and adds additional limitations and caveats on top.
posix_spawn() is specified such that it is possible (but does not required!) it to be implemented in terms of [v]fork() and exec().
On macOS it is a syscall and is generally faster than using vfork() and exec(). It also has a number of extensions like POSIX_SPAWN_CLOEXEC_DEFAULT, POSIX_SPAWN_SETEXEC, and `posix_spawn_file_actions_addchdir_np()` that allow it to actually be used in many cases where other systems need to resort to fork() and exec().
That doesn't even begin to cover the issues. Eg, PIDs may be quickly reused, so having a PID is no guarantee that you're going to send a signal to the right process. Signals suck. Handles get passed to children, which can cause weird issues. Information about processes is racy -- if you want to know something about a process you have to look in /proc, and you could have that data disappear if the process exits, or get replaced with a different one mid-execution.
On a higher level, things aren't much better. Bash is a constant annoyance if you want to actually accurately pass arguments to commands, and not have things break from an extra space.
What surprises me the most is that until systemd there was very little effort made to improve the situation. You'd think that after decades and thousands of init scripts dealing with the same PID file nonsense and other things that go in init scripts often in buggy ways somebody would think that a fix is needed, but no, it took until 2010.
It's not ideal, but it's the best solution I know of.
systemd actually went for as complete of a redesign as it could. Rather than just patch one issue up it tried to improve almost everything at once.
Which is a fresh approach compared to what we've had before -- every distro having their own flavor of SysV helper scripts, which only do part of the job and all slightly differently.
pidfd does a bit to help with various PID issues, but obviously it's nowhere near a complete solution. A better solution that would fix much more things is if the various syscalls that affected process attributes let you change those attributes on another process. Then you could have just a simple syscall that was "make new process with exe, argv, envp, and immediately put in stop phase before executing", and you could use the other syscalls to modify what you needed to before letting it start running via init. Basically like posix_spawn, but instead of having to enumerate all the stuff you want to do to the process before spawning it, you can do that at your leisure before telling it to start running.
> On a higher level, things aren't much better. Bash is a constant annoyance if you want to actually accurately pass arguments to commands, and not have things break from an extra space.
It seems to me that the Unix answer to this was "having spaces in arguments is clearly a user bug, what's wrong with you?"
There are two syscalls I really wish linux had (well maybe more, but these are the ones I've thought about the most, and are relevant here):
1. A spawn syscall that can replace fork/exec, that defaults to only inheriting the "std" file descriptors (0, 1, and 2), but allows passing others in an array. Has options for replacing the environment and the working directory, decreasing privileges (including capabilities and namespaces), etc. And has a way to return the pidfd for the child process.
2. Something like fork, but it only copies readonly program memory, and instead of returning in the child process, it calls a function, essentially the main function for the child process). This would be useful if you want to fork a child process using the same executable, but unlike fork can actually be thread safe.
23 comments
[ 6.6 ms ] story [ 62.3 ms ] threadI'll fix the link, thanks
https://www.youtube.com/watch?v=jrA-y1HG31E https://lpc.events/event/16/contributions/1213/attachments/1... https://lwn.net/Articles/908268/
You create a new task, which may or may not inherit your address space. This new task does not contain any threads.
You can then give it address space as needed (for example if you didn't set it to inherit any).
You can then create threads within the new task and start them as needed.
And yes, having a separate task without threads can be useful.
What I don't understand is why there isn't something for the simple case where we do just want to run another process and not set anything else. There is system() which I believe is just a wrapper for fork/exec, and is tempting to use, but the issue is it just takes a string and passes it to sh which opens you up to shell expansion exploits if you have untrusted/improperly sanitized inputs. I wish there was a version of system() that took an array or args like exec instead and you could specify the command to run directly as well, instead of sh.
Fork() and exec() are POSIX functions, while system() is a Standard C function that is available in C regardless of the operating system and regardless whether fork() and exec() exist.
The Standard C function system() is defined to execute a shell process and its semantics do not depend on how a process is executed by the host operating system.
Obviously in any UNIX-derived OS a shell process must be executed via fork and exec, and the shell itself executes children processes also via fork and exec. Under other operating systems no fork or exec are used.
What matters is that system() is not defined as a combination of fork with exec, but it does a more complex operation, with a greater overhead, and which works more or less in the same way under any OS.
As others have mentioned, POSIX does have a wrapper for fork and exec, which is posix_spawn().
Implementation details can leak though. For example on unix, if your process uses a lot of memory, a fork can result in an out of memory error, if there isn't enough memory left on the system to copy the parent process memory. And afaict things like how file descriptors are inherited are implementation defined.
> which works more or less in the same way under any OS
Even ignoring subtle differences in how subprocesses are spawned, there is the little issue that shell escaping and argument splitting works differently on different OSes, or even different shells on the same OS.
This does not change the fact that whether a system() function implementation uses or does not use fork and exec is neither specified by the function definition nor relevant for the user, and in any case system() is not the function that should be used when the intention is to simplify some code that otherwise would invoke fork and exec.
I agree that it isn't specified, but it can be relevant for the user, depending on what the user is doing.
> in any case system() is not the function that should be used when the intention is to simplify some code that otherwise would invoke fork and exec
Which was my original point. system doesn't save you from the problems of fork/exec, because depending on the system, it might be using fork/exec anyway, and adds additional limitations and caveats on top.
On macOS it is a syscall and is generally faster than using vfork() and exec(). It also has a number of extensions like POSIX_SPAWN_CLOEXEC_DEFAULT, POSIX_SPAWN_SETEXEC, and `posix_spawn_file_actions_addchdir_np()` that allow it to actually be used in many cases where other systems need to resort to fork() and exec().
That doesn't even begin to cover the issues. Eg, PIDs may be quickly reused, so having a PID is no guarantee that you're going to send a signal to the right process. Signals suck. Handles get passed to children, which can cause weird issues. Information about processes is racy -- if you want to know something about a process you have to look in /proc, and you could have that data disappear if the process exits, or get replaced with a different one mid-execution.
On a higher level, things aren't much better. Bash is a constant annoyance if you want to actually accurately pass arguments to commands, and not have things break from an extra space.
What surprises me the most is that until systemd there was very little effort made to improve the situation. You'd think that after decades and thousands of init scripts dealing with the same PID file nonsense and other things that go in init scripts often in buggy ways somebody would think that a fix is needed, but no, it took until 2010.
It just won.
And it's still not ideal. Not even close.
systemd actually went for as complete of a redesign as it could. Rather than just patch one issue up it tried to improve almost everything at once.
Which is a fresh approach compared to what we've had before -- every distro having their own flavor of SysV helper scripts, which only do part of the job and all slightly differently.
The unfortunate lesson of systemd is that playing politics will help you win over your competitors.
Which ones you mean?
> The unfortunate lesson of systemd is that playing politics will help you win over your competitors.
The way I see it, systemd knew who the "customers" were. Not so much normal end-users, as distro maintainers.
It heavily appealed to people sick of writing hundreds of init scripts
> On a higher level, things aren't much better. Bash is a constant annoyance if you want to actually accurately pass arguments to commands, and not have things break from an extra space.
It seems to me that the Unix answer to this was "having spaces in arguments is clearly a user bug, what's wrong with you?"
1. A spawn syscall that can replace fork/exec, that defaults to only inheriting the "std" file descriptors (0, 1, and 2), but allows passing others in an array. Has options for replacing the environment and the working directory, decreasing privileges (including capabilities and namespaces), etc. And has a way to return the pidfd for the child process.
2. Something like fork, but it only copies readonly program memory, and instead of returning in the child process, it calls a function, essentially the main function for the child process). This would be useful if you want to fork a child process using the same executable, but unlike fork can actually be thread safe.