Which is a problem of their own making. If they use two sockets, which would be needed if all three programs were on different machines, then it would work exactly the same as the pipe example.
The problem is that they saw TCP sockets are two-way and then thought to optimize the TCP connection/state overhead by using it for double-duty without applying appropriate abstractions to transparently use a two-way channel as two one-way channels.
Compare with what existed when NCP (RFC 33) was in use.
An early FTP protocol (452, 454, 542) originally used a server listen port of 3 (and hence a server "return port" of 2*), but 542 allowed temporary use of port 21 as the protocol was changed. A different pair of ports (even or odd depending upon TX or RX end) were used for the data channel, and the port picked was related to the Telnet protocol port in use.
RFC 640 then seems to have fixed the use of port 21 instead of 3. Then when TCP came in to use RFC 765 seems to have simplified things somewhat, sticking to port 21.
Also see RFC 165 for the/a Telnet Initial Connection Protocol showing the use of the two even/odd ports for forming a bidirectional connection.
* Or 4, but I believe the return channel was one port below, not above.
This is nonsense, pipes are the weird case, not the typical. The socket interface looks largely identical to ttys, which were in the first version of Unix over a decade before Berkeley sockets were introduced. Pipes didn't exist until Research Unix v3.
While historically you are correct, having a single file descriptor for reading and writing to distinct streams is confusing and can break code. Bidirectionally buffered wrappers over file descriptors (FILE, std::fstream for example) share the buffer between the read and write side, so you have to use distinct buffered wrappers with dup'd file descriptor anyway.
Also the tty is typically connected to stdin, stdout and stderr, that are separate file descriptor (again, a leaky abstraction when they refer to the same description).
Not a lot of application software interacts directly with ttys. I think the more relevant example here is stdio, which uses separate fds for reading and writing, like the pipe example.
The difference is that the callee working on the fd it received need to check if fd's type is pipe or socket and call close() or shutdown(fd, 1) accordingly.
This is not a pipe vs socket issue only but a penalty for having a vfs abstraction. Files, pipes, sockets, and more all have stream like behavior but they can vary widely and many cases demand leaking the vfs abstraction to check what's the fd type.
For a new OS, would it be good to circumvent this with a rudimentary runtime type system managed by the kernel? It would probably cost a bit of memory and performance overhead on syscalls, and eat up the syscall space. It could be reflected in the standard OS types in programming languages.
When I read beej’s guide to network programming I was confused when he said you have to call close() even after you call shutdown(). The way I understood it was that shutdown gave you granular control to shut down only the reading or writing part of the connection. Reading your comment I am even more lost lol
ah yes that's because close() closes the file descriptor but shutdown does not. You can have a socket that is shutdown on both ends that has a valid fd.
TCP was clearly designed by people who strove for elegance and understood the Unix philosophy, and you almost want to blame something else when all the wonky edge cases show up. Witness all the NODELAY issues, DDoS mitigations, and contortions (and eventual abandonment) that HTTP resorted to. TCP should have Just Worked, but alas, we live in an imperfect universe.
I am fairly certain that TCP and IP were designed and published before UNIX started being widely distributed (or known). And Berkeley sockets appeared in the early eighties, about ten years after TCP.
It was probably more of a co-evolution as the design of the Internet and Unix solidified, building on earlier established principles. But you can see the same thinking in both designs, the idea of plumbing together simple systems in a flexible way. A very successful approach since both TCP/IP and Unix-like computing are still in use today.
It sounds like the author forgot to dup the fd before giving it to consume-data.
If he would have done that it would have closed the socket only once both fd were closed and the fds would have separate lifetimes like in the pipe example.
duping the fd's as you describe would achieve nothing, as both fds will share the same file, and hence while either fd is open, the kernel can not detect a close.
The process-data task is on a remote machine, probably listening on a single port, and feeding its output back over the same TCP connection as its input.
Some process has then set up the series of connections, by connecting to the remote server, plumbing and doing fork/exec to run consume-data with the socket on stdin, while doing an exec of generate-data having the socket on stdout.
Now the processes being run are assumed to be general unix processes, i.e. pipeline processes ala how cat, head, grep etc operate. They generally read from input, write to output, and exit when the input is exhausted. That "exit signal" will then be fed through the pipeline as a series of "EOF" notifications, which FIN can achieve for TCP.
However because the output fd from generate-data, and the input fd to consume-data both share the same underlying socket, the mere close of the output fd from generate-data exiting will not result in a FIN being sent to process-data.
That is because the same "file" (the socket) is open by consume-data (in read/write mode), and so the kernel can not know that consume-data will not write to the socket.
If the UNIX abstraction for the sockets had the ability to split the socket across to "files", not just fds, then the issue would be removed; but it doesn't.
No, he wants a way for the close of the FD from generate-data to process-data to trigger a TCP FIN over the connection, such that process-data sees an input EOF.
However for that to work, the two FDs (generate-date to process-data, and process-data to consume-data) must not be associated with the same "open file", even though they could (with a change) represent the same TCP socket.
i.e. he wants generate-data to exit, closing its output FD, and have that drive EOF all the way through to consume-data. Such that consume-data does not determine it is done until it sees EOF on its input socket.
(One can observe a similar issue if one opens FIFO's in read/write mode, but since they can also be opened read-only or write-only, the issue does not arise.)
FD have been designed for file systems where you definitely wants to share the same data structure for read/write/seek and imply that by giving it the same ID. The fact that seek does not work on socket is just because socket is a sub specialisation of FD manipulation, just as pipes are.
Lots of the complexity of TCP at a protocol level comes from treating it as a single duplex connection instead of a pair of simplex connections. And the article here shows that this complexity leaks up to the API level, too.
Well, breaking the socket into two pieces would add complexity. It would create a new requirement for non-trivial programs to add bookkeeping to track the two bits together and make it easier for the two bits to become accidentally permanently estranged from each other. It would add new states to the already complicated TCP set of states.
Then again, in most of my "non-trivial" network code I already end up cracking the socket into two pieces anyhow, and while I take advantage of the pun that closing either of them closes the other it would be trivial for me to add this behavior to the separated sockets if that's what I wanted. Partially this is a result of my "non-trivial" network code being in Go so I end up running a goroutine each for send and receive, but still I imagine this would be a common structure one way or another in a lot of non-trivial network servers.
But the additional states on the TCP socket would definitely bite some people. Even in its current simpler state it still confuses people and there's a post about how some server turned out to be leaking sockets because it didn't properly close them shows up with some frequency on HN.
Nah - one would just add a new syscall, to split the fd.
int sock_fd, split_pair[2];
char c;
int n = socket_split(sock_fd, split_pair);
close(sock_fd);
n = write(split_pair[1], "a", 1);
close(split_pair[1]);
n = read(split_pair[0], &c, 1);
read(split_pair[0], &c, 1); /* returns 0 / EOF */
The effect being to split the fds, and give an output similar to pipe(2). Moreover it would also duplicate the underlying open file structure, such that one is read only, the other write only, and both referencing the same socket. Then the read fd would reference one "open file", the write-fd the other.
(One can observe a similar issue if one opens FIFO's in read/write mode, but since they can also be opened read-only or write-only, the issue does not arise.)
I wonder if perhaps you replied to the wrong comment? I never questioned that it could be done, I was discussing the results from the point of view of it already being done (and for my discussion's purpose, mandatory).
If the splitting is optional, as it would have to be now, and as I proposed it would not add complexity since it would be opt in.
Theoretically if it had split from the start, it may have added a minor amount of complexity to some programs, but not a significant amount. I don't get the need to add TCP states, as the TCP machine will not change, and the user space app shouldn't really be tracking/replicatting the TCP states.
Anyway, your assumed (and mandatory) point of view was not obvious to me, I interpreted it from the position of "if that was done now". Which would add a trivial amount of logic to the kernel, mainly reusing existing logic, structures, and counters.
> It would add new states to the already complicated TCP set of states.
Huh? Half-closed connections already exist. The complexity in the state machine is there, and more TCP users should be aware of it (the source of soo.... many... bugs)
While the current "half closed state" and "one of the read/write halves is closed and you don't realize it" may sound similar, they're two different "halves". Splitting the socket fully into two separate file handles would mean that users could close one of them without realizing it, and thus be able to write a new sort of bug where they would be able to ask stack overflow "I'm able to read from my socket but not write from it, what gives?"
This wouldn't necessarily manifest as a new TCP state in the kernel state table. You could do this to TCP today at the API level alone. Which is further evidence it's a different "half".
And if it sounds silly that anyone would make such a mistake, like many such hypothetical mistakes it's important to remember that in the wild it's not always next to each other. My least favorite networking experience ever involved being three levels deep in abstraction in some Perl libraries, every level of which felt like it can and should make all sorts of helpful little calls, the end result being all kinds of fun to debug as soon as any of them were wrong. The thing that closed one of the halves of the socket and the person confused by the result can be arbitrarily separated in the code.
Or, to put it another way, you can't grow the API's complexity without growing the API's complexity. If today closing closes both reading and writing but a new API severs that guarantee and creates at least two new states, that inevitably results in a new way for things to go wrong. You can't have one without the other.
> Splitting the socket fully into two separate file handles would mean that users could close one of them without realizing it, and thus be able to write a new sort of bug where they would be able to ask stack overflow "I'm able to read from my socket but not write from it, what gives?"
shutdown(sockfd, 1)
gives you "I'm able to read from my socket but not write from it, what gives?"
The real bugs with TCP come from "why am I not able to download the file because the server did close(sockfd) rather than shutdown(sockfd, 1), and my TCP stack ACK'd, causing the server to RST, causing my machine to give me an error before I read all of the data".
This is already waaaay harder than the counterfactual you're imagining, and the bugs are here, today.
As far as I'm aware, it was driven by the "piggybacking" of ACKs on packets travelling in the opposite direction, which turns out to have been a slight misfeature.
Yes. TCP connection establishment inherently forms a two-way channel between two entitys. There is little reason to duplicate the connection/routing (in the generic sense) establishment for each direction in the general case where you might want a two-way channel.
The problem in the presented example is that there are, fundamentally, 3 communicating entitys A->B->C, which logically demands two channels. A quirk of the example where A and C happen to be on the same system and coincidentally can use the channel disjointly (read-only and write-only) in conjunction with TCP being inherently two-way allows it to be kludged into a single channel, but frankly it is your fault for relying on bizarre implementation details [1].
If you want a proper abstraction, what you would actually do is multiplex multiple logical streams over a single duplex channel. The code would then operate purely on the logical streams (pipe-like) and everything would be clean. The implementation would then only need to establish connection once, removing the inherent overhead associated with establishing multiple network protocol connections, while still giving you a clean and consistent abstraction (barring the standard leakiness of the TCP abstraction).
> Lots of the complexity of TCP at a protocol level comes from treating it as a single duplex connection instead of a pair of simplex connections.
I've created a few communication protocols in my time (all people of a certain age have, because we pre-date TCP/IP), and I am reasonable familiar with TCP itself. I don't see the extra complexity you speak of. At most, it is de-multiplexing the ACK fields in the header from the rest. If you want to whinge about unnecessary complexity in the TCP protocol, I'd suggest starting with it being byte orientated rather than packet orientated.
Secondly Dan is out of touch. He's right in saying it's a complex solution for a shell pipeline, but the TCP is designed to operate in a less reliable world. In TCP's world you can no longer assume a message you sent was acted upon by the other end. The network could have died, or the system crashed before the data made it to disk. Thus the shell command:
echo Run for your life or you will die | mail friend@doomed.com
is perfectly reasonable. You know the message will make it or you will be told something went wrong. But if the "mail" command is running on a different machine, unsupervised, then you have no idea the message made it unless the mail programs sends a message back saying "email safely and durably received". That requires a back channel. You almost always require a back channel for that reason and what's worse, they are a pita to set up using unidirectional pipes. That makes a unidirectional pipe is a terrible API for sending messages to a remote location.
It looks like TCP's successor will be QUIC. In QUIC there isn't just two streams, you can create lots of sub-streams, using datagrams if you want. Now that is a lot of extra complexity. But from the applications point of view it's a very nice feature. It's a bit like the second channel TCP gives you, as it's surprising how often you need a more than one channel between two programs. In fact the very first use TCP was put to was FTP and guess what - FTP creates two TCP connections. And it's example of how painful it can be to set that up as NAT broke the original FTP for a while. We now finally have an off the shelf solution in QUIC. It only took 30+ years to arrive.
It looks to me like the world has voted. The outcome is a firm Nay to Dan's suggestion, and it's now moving in the opposite direction.
PASV works around some but not all NAT breakage. There are things FTP can do outside a NAT that it can't do with a NAT around. (Workarounds can be concocted, but NAT really did take away some of the natural abilities of FTP.)
The complexity I have in mind is roughly that the TCP state machine isn't quite the cartesian product of two opposed unidirectional state machines, but it could be. (No proof of this, only a TCP implementation based on the idea. I really must sort the question out rigorously one of these days.)
Re the reliability: the same is true of pipes. You can't assume `mail` will take the message, even with pipes, because, just as with TCP, there's no feedback channel. The disk could have filled up after `mail` took the final bytes from its input, for example.
Your point about application-level acknowledgements is 100% spot on, then, but pipes need application-level acks too, in much the same circumstances as TCP would, for just the same reasons. Even pipes make a distributed system.
Not all protocols need application-level acks, though. In the `echo | mail` example, you do get feedback that the process exited abnormally, but this is feedback to the supervisor (here, the shell), not to the preceding process in the pipeline. The same could (and should) be arranged for a TCP-based link between `echo` and `mail`. The failure-signalling takes on some of the responsibilities than an app-level ack would have discharged. Pipes vs sockets makes no difference.
Your point about the difficulty of constructing the necessary back-channels for app-level acks is also spot on. If one were to reformulate a transport layer in terms of simplex channels, one would want to retain an easy way to configure a pair of them as a single duplex channel. This is as true of pipes as of sockets. A nice design would allow capability-passing as well as regular message sending: then, you'd open an outbound stream to a server, and as your first message, you'd send a capability for the write-end of a matching inbound stream. Modular session establishment. Permits a bunch of other interaction patterns, too, such as session handoff, redirection, continuation-passing, load-balancing etc. Richer topological possibilities even than SCTP and QUIC. If capability-passing was in TCP, it'd have been trivially handled by the NAT, too, so FTP wouldn't have been broken. Ah well, never mind.
All that said, Dan isn't taking aim at TCP-the-protocol here. (He does, elsewhere.) Instead, the focus is the sockets API. The API could easily have involved two unidirectional fds instead of just one, without changing the protocol, or how easy/difficult it is to establish a connection. It'd just expose a read cap and a write cap instead of a single mixed capability. It'd be a small change that'd eliminate a bunch of special-casing. I think he has a good point; it's a shame we can't get there from here...
> The API could easily have involved two unidirectional fds instead of just one, without changing the protocol, or how easy/difficult it is to establish a connection.
I can't see how it's an improvement. The pipe code is invariably:
pipe(fds);
close(fd[0]); /* Close the one we don't need. */
If you don't want one direction of a socket the code is:
connect(soc, ...);
shutdown(soc, ...); /* Close the one we don't need. */
Dan's complaint is quote "The BSD socket designers introduced a shutdown(fd,1) system call to send FIN through a TCP connection. This is exactly the sort of device-specific garbage that UNIX fds were designed to avoid." However the code for pipes and sockets near identical bar the names. Just to re-enforce how similar they are, if you don't call "close()" for pipes, his core complaint that nothing exits because "the same fd is still open in the consume-data program" happens in the pipe case.
The problem is you can't statically know, in e.g. Dan's `generate-data`, which to call: close() or shutdown(). You have to reflect on stdout to discover what kind of resource you have.
So there's a downside to the inconsistency. Is there a corresponding upside I'm not seeing? Cause if not, perhaps all else equal, in which case I'd rather have the static simplification.
> The problem is you can't statically know, in e.g. Dan's `generate-data`, which to call: close() or shutdown().
True. But code like this is invariably a 3 step process - setup the invariants, loop (generate-data), then close down. The steps above happen in the open, which means `generate-data` doesn't need to know what to call.
The Dan's pipeline gave is a good example of this as any. It is the shell that does the open step, so it's the thing that does the open, so it creates the pipes and does the close() of the unwanted end. Thus Dan `generate-data` does not need to know whether it is sending over s pipe or TCP connection.
In fact bash accepts /dev/tcp/host/port syntax. I presume it already does the required shutdown as part of that of setting it up. Which demonstrates that in real life `generate-data` in indeed blissfully unaware of whether it's a pipe or socket.
Finally, notice the open step is always going to be different, so this additional work of setting it up different is not a big imposition. It's has to be different because it has to connect the two ends, and that is going to be different depending on whether they are local or remote.
Ran into a variant of this when working on an IO loop, ended up with a little "Side" abstraction to wrap bidirectional/unidirectional descriptors to behave identically for the rest of the library. It's an annoying wart but pretty easy to avoid in userspace.. https://github.com/mitogen-hq/mitogen/blob/798032b9/mitogen/...
Honestly, I think a better idea would be to write a program that can act as an intermediary, sending the stdin data through TCP, and writing out the data it receives back through stdout.
That ship sailed years ago. We could have had "pipes between hosts" with a distributed OS design like Plan9. But Linus doesn't care, so we've got what we got. Until somebody forks the kernel and goes their own way for fun, it will never change.
These remote pipes are a largely a pipe dream due to very agile crypto scene. It requires a bloated spec to describe policies and be interoperable standard nobody would use. `command1 | ssh host process | command2` works perfectly fine.
And what prevents to run stunnel on Linux? Is there a way to specify remote address or remote run to use TLS to wrap your application transparently in plan 9? Otherwise plan 9 on the same abstraction level.
stunnel is the tired version of inter-machines communication. It's a basic, raw socket between two processes, and all layers above must be reinvented again and again.
Plan 9 with its protocol 9p is the wired version of inter-machines communication. You don't just link two random processes, you have the possibility to interconnect two directory trees, and since most useful software works through a directory tree, this means two processes can be linked by importing another tree on another machine and doing everything locally. It's possible to write something like "echo UNSEEN | imap-search | imap-get-summary" and have actual IMAP stuff done on another machine, and you don't see any of it.
It is also possible to use `rcpu` to do what ssh does, and even more: the remote machine has access to the local machine. So for instance you can edit files locally, and have a rcpu connection to run builds on another machine that picks it from your local workstation. No clunky sshfs, no failing NFS, ...
Why should Linus care about plan 9? If the plan 9 design appeals to you then use it. It's waaaaaay smaller and simpler than Linux yet has much more flexibility out of the box. Give 9front a go, it's actively developed and actually works.
I have always had this sort of nagging feeling about the way BSD sockets works: it really is a bit strange that the reading and writing operations are effectively entirely separate. When you write to most FDs, the inverse operation is reading; but sockets don't work that way, reading and writing are effectively working on two different streams!
The insight that a TCP connection is basically equivalent to a pipe is a pretty good demonstration of how this could have been better.
As another related example, imagine if the same had been done for standard I/O. What if instead of having separate file descriptors, all of the standard I/O happened through one file descriptor? Imagine how much more painful everything would be.
Unfortunately, this is now pretty much how a lot of stuff works, but it did not really have to go this way.
An easy counterargument is that sockets support out of band read operations: for example, I can call getsockname() on a socket to obtain meaningful information about it after shutdown(). Pipes have no such analogue.
Also, it's a general interface. It's easy to imagine exotic protocols where shutdown() is not so cleanly equivalent to close() as it is for TCP.
TBH I find it odd to focus on a preference for one semantic over the other. The real thing we can all probably agree on is that in a perfect world, they ought to be the same.
These are great for duct-taping connections. I created a whole suite of utilities for another programs for uploading firmware updates. I was able to create one program for handling the uploads with several different comms protocols and devices, without having to handle details of tcp connections, serial connections, UDP, etc. socat is another great tool for this.
67 comments
[ 3.0 ms ] story [ 134 ms ] threadIn the first example, he is using two pipes; in the second example, he is using one socket. And he complains that that they don't work the same!?!
The problem is that they saw TCP sockets are two-way and then thought to optimize the TCP connection/state overhead by using it for double-duty without applying appropriate abstractions to transparently use a two-way channel as two one-way channels.
An early FTP protocol (452, 454, 542) originally used a server listen port of 3 (and hence a server "return port" of 2*), but 542 allowed temporary use of port 21 as the protocol was changed. A different pair of ports (even or odd depending upon TX or RX end) were used for the data channel, and the port picked was related to the Telnet protocol port in use.
RFC 640 then seems to have fixed the use of port 21 instead of 3. Then when TCP came in to use RFC 765 seems to have simplified things somewhat, sticking to port 21.
Also see RFC 165 for the/a Telnet Initial Connection Protocol showing the use of the two even/odd ports for forming a bidirectional connection.
* Or 4, but I believe the return channel was one port below, not above.
Even being receive, odd transmit. That then led to various higher level protocols using pairs of ports. Once TCP took over, that was relaxed.
The best mitigation is to not close `stdin` when you get EOF on it -- just stop reading from `stdin`.
Also the tty is typically connected to stdin, stdout and stderr, that are separate file descriptor (again, a leaky abstraction when they refer to the same description).
This is not a pipe vs socket issue only but a penalty for having a vfs abstraction. Files, pipes, sockets, and more all have stream like behavior but they can vary widely and many cases demand leaking the vfs abstraction to check what's the fd type.
If he would have done that it would have closed the socket only once both fd were closed and the fds would have separate lifetimes like in the pipe example.
duping the fd's as you describe would achieve nothing, as both fds will share the same file, and hence while either fd is open, the kernel can not detect a close.
The process-data task is on a remote machine, probably listening on a single port, and feeding its output back over the same TCP connection as its input.
Some process has then set up the series of connections, by connecting to the remote server, plumbing and doing fork/exec to run consume-data with the socket on stdin, while doing an exec of generate-data having the socket on stdout.
Now the processes being run are assumed to be general unix processes, i.e. pipeline processes ala how cat, head, grep etc operate. They generally read from input, write to output, and exit when the input is exhausted. That "exit signal" will then be fed through the pipeline as a series of "EOF" notifications, which FIN can achieve for TCP.
However because the output fd from generate-data, and the input fd to consume-data both share the same underlying socket, the mere close of the output fd from generate-data exiting will not result in a FIN being sent to process-data.
That is because the same "file" (the socket) is open by consume-data (in read/write mode), and so the kernel can not know that consume-data will not write to the socket.
If the UNIX abstraction for the sockets had the ability to split the socket across to "files", not just fds, then the issue would be removed; but it doesn't.
Which is what the author wants. He wants the socket to close once both generate-data and consume-data are done with the socket.
However for that to work, the two FDs (generate-date to process-data, and process-data to consume-data) must not be associated with the same "open file", even though they could (with a change) represent the same TCP socket.
i.e. he wants generate-data to exit, closing its output FD, and have that drive EOF all the way through to consume-data. Such that consume-data does not determine it is done until it sees EOF on its input socket.
(One can observe a similar issue if one opens FIFO's in read/write mode, but since they can also be opened read-only or write-only, the issue does not arise.)
Then again, in most of my "non-trivial" network code I already end up cracking the socket into two pieces anyhow, and while I take advantage of the pun that closing either of them closes the other it would be trivial for me to add this behavior to the separated sockets if that's what I wanted. Partially this is a result of my "non-trivial" network code being in Go so I end up running a goroutine each for send and receive, but still I imagine this would be a common structure one way or another in a lot of non-trivial network servers.
But the additional states on the TCP socket would definitely bite some people. Even in its current simpler state it still confuses people and there's a post about how some server turned out to be leaking sockets because it didn't properly close them shows up with some frequency on HN.
(One can observe a similar issue if one opens FIFO's in read/write mode, but since they can also be opened read-only or write-only, the issue does not arise.)
If the splitting is optional, as it would have to be now, and as I proposed it would not add complexity since it would be opt in.
Theoretically if it had split from the start, it may have added a minor amount of complexity to some programs, but not a significant amount. I don't get the need to add TCP states, as the TCP machine will not change, and the user space app shouldn't really be tracking/replicatting the TCP states.
Anyway, your assumed (and mandatory) point of view was not obvious to me, I interpreted it from the position of "if that was done now". Which would add a trivial amount of logic to the kernel, mainly reusing existing logic, structures, and counters.
So I guess we were slightly at cross purposes.
Huh? Half-closed connections already exist. The complexity in the state machine is there, and more TCP users should be aware of it (the source of soo.... many... bugs)
This wouldn't necessarily manifest as a new TCP state in the kernel state table. You could do this to TCP today at the API level alone. Which is further evidence it's a different "half".
And if it sounds silly that anyone would make such a mistake, like many such hypothetical mistakes it's important to remember that in the wild it's not always next to each other. My least favorite networking experience ever involved being three levels deep in abstraction in some Perl libraries, every level of which felt like it can and should make all sorts of helpful little calls, the end result being all kinds of fun to debug as soon as any of them were wrong. The thing that closed one of the halves of the socket and the person confused by the result can be arbitrarily separated in the code.
Or, to put it another way, you can't grow the API's complexity without growing the API's complexity. If today closing closes both reading and writing but a new API severs that guarantee and creates at least two new states, that inevitably results in a new way for things to go wrong. You can't have one without the other.
shutdown(sockfd, 1)
gives you "I'm able to read from my socket but not write from it, what gives?"
The real bugs with TCP come from "why am I not able to download the file because the server did close(sockfd) rather than shutdown(sockfd, 1), and my TCP stack ACK'd, causing the server to RST, causing my machine to give me an error before I read all of the data".
This is already waaaay harder than the counterfactual you're imagining, and the bugs are here, today.
The problem in the presented example is that there are, fundamentally, 3 communicating entitys A->B->C, which logically demands two channels. A quirk of the example where A and C happen to be on the same system and coincidentally can use the channel disjointly (read-only and write-only) in conjunction with TCP being inherently two-way allows it to be kludged into a single channel, but frankly it is your fault for relying on bizarre implementation details [1].
If you want a proper abstraction, what you would actually do is multiplex multiple logical streams over a single duplex channel. The code would then operate purely on the logical streams (pipe-like) and everything would be clean. The implementation would then only need to establish connection once, removing the inherent overhead associated with establishing multiple network protocol connections, while still giving you a clean and consistent abstraction (barring the standard leakiness of the TCP abstraction).
[1] https://xkcd.com/1172/
To clarify, would one be send and the other be receive
or
are you saying one would be for the application data and the 2nd would be for the protocol SYN/ACK and all that?
I've created a few communication protocols in my time (all people of a certain age have, because we pre-date TCP/IP), and I am reasonable familiar with TCP itself. I don't see the extra complexity you speak of. At most, it is de-multiplexing the ACK fields in the header from the rest. If you want to whinge about unnecessary complexity in the TCP protocol, I'd suggest starting with it being byte orientated rather than packet orientated.
Secondly Dan is out of touch. He's right in saying it's a complex solution for a shell pipeline, but the TCP is designed to operate in a less reliable world. In TCP's world you can no longer assume a message you sent was acted upon by the other end. The network could have died, or the system crashed before the data made it to disk. Thus the shell command:
is perfectly reasonable. You know the message will make it or you will be told something went wrong. But if the "mail" command is running on a different machine, unsupervised, then you have no idea the message made it unless the mail programs sends a message back saying "email safely and durably received". That requires a back channel. You almost always require a back channel for that reason and what's worse, they are a pita to set up using unidirectional pipes. That makes a unidirectional pipe is a terrible API for sending messages to a remote location.It looks like TCP's successor will be QUIC. In QUIC there isn't just two streams, you can create lots of sub-streams, using datagrams if you want. Now that is a lot of extra complexity. But from the applications point of view it's a very nice feature. It's a bit like the second channel TCP gives you, as it's surprising how often you need a more than one channel between two programs. In fact the very first use TCP was put to was FTP and guess what - FTP creates two TCP connections. And it's example of how painful it can be to set that up as NAT broke the original FTP for a while. We now finally have an off the shelf solution in QUIC. It only took 30+ years to arrive.
It looks to me like the world has voted. The outcome is a firm Nay to Dan's suggestion, and it's now moving in the opposite direction.
NAT broke FTP for about the 5 seconds it took for one to realise they needed to type in the PASV command - which existed right from the start.
It was in RFC 542 (1973) - pre TCP, and RFC 765 (1980) - first post TCP.
NAT dates from around 1994.
Re the reliability: the same is true of pipes. You can't assume `mail` will take the message, even with pipes, because, just as with TCP, there's no feedback channel. The disk could have filled up after `mail` took the final bytes from its input, for example.
Your point about application-level acknowledgements is 100% spot on, then, but pipes need application-level acks too, in much the same circumstances as TCP would, for just the same reasons. Even pipes make a distributed system.
Not all protocols need application-level acks, though. In the `echo | mail` example, you do get feedback that the process exited abnormally, but this is feedback to the supervisor (here, the shell), not to the preceding process in the pipeline. The same could (and should) be arranged for a TCP-based link between `echo` and `mail`. The failure-signalling takes on some of the responsibilities than an app-level ack would have discharged. Pipes vs sockets makes no difference.
Your point about the difficulty of constructing the necessary back-channels for app-level acks is also spot on. If one were to reformulate a transport layer in terms of simplex channels, one would want to retain an easy way to configure a pair of them as a single duplex channel. This is as true of pipes as of sockets. A nice design would allow capability-passing as well as regular message sending: then, you'd open an outbound stream to a server, and as your first message, you'd send a capability for the write-end of a matching inbound stream. Modular session establishment. Permits a bunch of other interaction patterns, too, such as session handoff, redirection, continuation-passing, load-balancing etc. Richer topological possibilities even than SCTP and QUIC. If capability-passing was in TCP, it'd have been trivially handled by the NAT, too, so FTP wouldn't have been broken. Ah well, never mind.
All that said, Dan isn't taking aim at TCP-the-protocol here. (He does, elsewhere.) Instead, the focus is the sockets API. The API could easily have involved two unidirectional fds instead of just one, without changing the protocol, or how easy/difficult it is to establish a connection. It'd just expose a read cap and a write cap instead of a single mixed capability. It'd be a small change that'd eliminate a bunch of special-casing. I think he has a good point; it's a shame we can't get there from here...
I can't see how it's an improvement. The pipe code is invariably:
If you don't want one direction of a socket the code is: Dan's complaint is quote "The BSD socket designers introduced a shutdown(fd,1) system call to send FIN through a TCP connection. This is exactly the sort of device-specific garbage that UNIX fds were designed to avoid." However the code for pipes and sockets near identical bar the names. Just to re-enforce how similar they are, if you don't call "close()" for pipes, his core complaint that nothing exits because "the same fd is still open in the consume-data program" happens in the pipe case.This makes me think of hobgoblins and minds.
So there's a downside to the inconsistency. Is there a corresponding upside I'm not seeing? Cause if not, perhaps all else equal, in which case I'd rather have the static simplification.
True. But code like this is invariably a 3 step process - setup the invariants, loop (generate-data), then close down. The steps above happen in the open, which means `generate-data` doesn't need to know what to call.
The Dan's pipeline gave is a good example of this as any. It is the shell that does the open step, so it's the thing that does the open, so it creates the pipes and does the close() of the unwanted end. Thus Dan `generate-data` does not need to know whether it is sending over s pipe or TCP connection.
In fact bash accepts /dev/tcp/host/port syntax. I presume it already does the required shutdown as part of that of setting it up. Which demonstrates that in real life `generate-data` in indeed blissfully unaware of whether it's a pipe or socket.
Finally, notice the open step is always going to be different, so this additional work of setting it up different is not a big imposition. It's has to be different because it has to connect the two ends, and that is going to be different depending on whether they are local or remote.
...can curl already do that?
http://cr.yp.to/ucspi-tcp/tcpclient.html
Plan 9 with its protocol 9p is the wired version of inter-machines communication. You don't just link two random processes, you have the possibility to interconnect two directory trees, and since most useful software works through a directory tree, this means two processes can be linked by importing another tree on another machine and doing everything locally. It's possible to write something like "echo UNSEEN | imap-search | imap-get-summary" and have actual IMAP stuff done on another machine, and you don't see any of it.
It is also possible to use `rcpu` to do what ssh does, and even more: the remote machine has access to the local machine. So for instance you can edit files locally, and have a rcpu connection to run builds on another machine that picks it from your local workstation. No clunky sshfs, no failing NFS, ...
The insight that a TCP connection is basically equivalent to a pipe is a pretty good demonstration of how this could have been better.
As another related example, imagine if the same had been done for standard I/O. What if instead of having separate file descriptors, all of the standard I/O happened through one file descriptor? Imagine how much more painful everything would be.
Unfortunately, this is now pretty much how a lot of stuff works, but it did not really have to go this way.
Also, it's a general interface. It's easy to imagine exotic protocols where shutdown() is not so cleanly equivalent to close() as it is for TCP.
TBH I find it odd to focus on a preference for one semantic over the other. The real thing we can all probably agree on is that in a perfect world, they ought to be the same.
https://cr.yp.to/ucspi-tcp.html
These are great for duct-taping connections. I created a whole suite of utilities for another programs for uploading firmware updates. I was able to create one program for handling the uploads with several different comms protocols and devices, without having to handle details of tcp connections, serial connections, UDP, etc. socat is another great tool for this.