Fundamental? I see it the other way -- this is an illustration of how neat, fundamental concept ("When you bind to a port, it puts a pointer to your socket in that table.") is actually full of random hacks and ad-hoc changes. The linux kernel is full of random logic like this, and this article could be continued (there are at least network namespaces and sys.net options to consider).
Don't get me wrong -- all of this functionality is useful in some context, and it is there for a reason. We cannot radically simplify the kernel without making it less useful. But it is still somewhat sad there is so much essential complexity even in simple things like "listen" call.
Well only sort of. As far as I understand, nothing about TCP means that ports can be reused, or that 0.0.0.0 has special meaning, or that processes can’t listen on the same port, or IPv4 to IPv6 delivery etc. These are all operating system design choices, some of which stem from the sockets API.
I’m pretty sure you could design a spec-compliant alternative which didn’t do any of these things, if you really wanted to.
TCP and UDP don't really dictate any of this behavior. Most of these hacks are the result of design choices in Linux and the socket API - abstractions on top of TCP/UDP that aren't the only option.
In my experience, the network layer is often way better-designed, more reliable, more interoperable, and less hacky than most OS abstractions built on top of it.
This article has taught me the concept of "discovery fiction" and, I'm sorry to say so, but I already hate it.
I get the idea of presenting concepts in a more natural flow and smuggling in some spaced repetition, but the whole story just felt extremely forced and artificial to me, sort of like a drawn-out sequence of expospeak.
There has to be a better way to include spaced repetition and discovery in teaching.
Trigonometry The Easy Way, was brilliant! I wouldn't categorize it as forced in the manner of the OP. The book creates a fantasy backdrop to connect trigonometrical concepts in a fun way. I really enjoyed the book and it made trigonometry really stick in my brain!
Haha entertaining. I rather like this style. I skipped till the first code and then the interludes were short. But I think I would have been annoyed if the first code block was not demarcated clearly so I could skip to it. The prelude was too long for me to read since I didn’t know if there was going to be payoff.
One thing I've learned about pedagogy is that it's a fool's errand to try to create material that works for everyone, because everyone learns differently. In fact, the meta story here is supposed to be about Liz and Tim having different styles of learning -- Liz learns by exploring off the beaten path, while Tim prefers to stick to the course content. Neither are wrong.
Ok, I didn't realize this aspect of the story, this is actually pretty cool, I agree.
I also like that the story encourages experimenting and testing out your mental models, like Liz did.
It's just the particular way they discover things that feels forced to me. The story is all about them coming to their own conclusions and thinking up their own experiments, but - it being a story - you know, it's actually all guided beforehand.
Maybe what irks me is that this concept of a character discovering "on their own" some ostensible deep truth (which is really just the author's personal opinion about something) has been used for a lot of worse reasons in other stories - even though, in this case, the "truth" is perfectly harmless and beneficial.
Anyway, it's always easier to criticize than to create, so I won't say I really have better ideas of how to do it.
This is saying that VARK method of describing learning styles is incorrect. This does not necessarily mean that there aren't differences in what are the best method's for different people learn. One issue with VARK is that it involves self-identification, as was the case in all studies mentioned in the video. There's no reason to assume that the way people prefer to learn is actually the best way for them to learn, it may be that people choose what seems right to them rather than what actually is. Another issue is, as is mentioned in the video, the domain you are studying also has an affect and multi-modal approaches are most useful. Whether learning styles exist or not, some information intrinsically is better presented as a diagram than a wall of text. And no matter if some people learn better in certain ways, it seem universally true that people learn better if information is presented to them in multiple forms instead of just one.
Which is all interesting, but mostly just disproves VARK and similar approaches to describing differences in how people learn. There are still so many different ways to teach someone something, they're just much more holistic ways of teaching than the simple VARK split. That some people learn better from X course of teaching and others learn better from Y course of teaching still seems likely to me(admittedly, just pulling from personal experience and the anecdotes of others on that ). That we don't have a neat way to categorize that might just mean it's too messy to do so, or could mean we just haven't figured out the right way to look at it yet.
>> I've seen somewhere that "everyone learns differently" might not actually be true.
Even if we assume that people all learn the same way, good learning integrates new facts or concepts into ones pre-existing mental model of the world. Not everyone has the same mental model - of this I am certain. Sometimes new information just hangs on the existing model, and sometimes the existing model needs to be updated. This can make the process seem like everyone learns differently, since different explanations can make more or less sense depending what's already in their head.
On top of that, I think some people have (maybe inherently) very different abilities in things like visualization, memorization, vocabulary, etc... So yeah, I think everyone learns differently even if at some neuronal level it's all the same.
I think “everyone learns differently” is an overblown claim and gets repeated ad-nauseum. It is an escape hatch to avoid deeper discussions of successful strategies, what it means to teach and learn. A sort of a deus-ex-machina of arguments. There are more universals about learning than differences. It effectively surrenders the failure of teaching to “Well, it must be because everyone learns differently”. This way, we can never learn to teach or even debate about it.
This is actually a failure of discourse in many areas of life – "X is hard to solve or haven't thought about it, so it must be that X is different for everyone". Nutrition, Product Reviews, etc. I've learned over years that anyone that claims "X is different for everyone" is most likely exhibiting a defeatist attitude.
The universals about learning, which I've taken away from the discussion, is that active learning and recall is generally effective, while passive learning (e.g. re-reading) is less effective. A good research paper that covers this (which I've mentioned in that discussion) is called "Improving Students’ Learning With Effective Learning Techniques: Promising Directions From Cognitive and Educational Psychology" [PDF]: https://pcl.sitehost.iu.edu/rgoldsto/courses/dunloskyimprovi...
I personally just hated the long-form-journalism style of "okay, at the start, you have to wade through a ton of irrelevant crap about the settings and the coil-bound notebooks they're using".
I'm convinced lots of sites do this just to push out the actual content below the "please login / buy subscription to continue reading" point, or if it is a free article at least make you scroll through as many inline ads as they can.
>I personally just hated the long-form-journalism style of "okay, at the start, you have to wade through a ton of irrelevant crap about the settings and the coil-bound notebooks they're using".
Yeah, I'm really glad this was the top comment. That style of journalism has become all pervasive today, and the grotesque self indulgence of it is honestly sickening. I refer to it as "college term paper journalism", because they all read like a sophomore English 102 essay.
It's way too long since I read it, so I can't really compare, but I remember that the whole topic was actively discussed and reflected in the novel. That's something different.
Like, Sophie being a character in a novel was an important plot point of just that novel, if I remember correctly.
You can see there's no dialogue; the idea is to frame an article as "how might you discover/invent this concept on your own", rather than just directly explaining how a thing works. You are given a fictional goal, and you "discover" the subject matter by building iterative solutions.
This example that explains kerberos is entirely dialog
https://web.mit.edu/kerberos/www/dialogue.html
Really helped me understand kerberos in a way other articles and technical documents had failed to.
agreed. it's not useful for me as a more "tech experienced" person and having taught many people just breaking into the industry i don't think it would help them much, either.
people are often helped by metaphor (e.g. "a port is like a mail slot in a huge mailroom...") but i feel like this "discovery fiction" thing is like an inverse version of that... it allows your brain to be creative when thinking through the topic but just focused on the wrong part of the explanation.
My first exposure to programming was a book my dad gave me called The Little Schemer, and I absolutely loved it. It's like discovery fiction without the fiction. It just asks a question and then immediately answers it; mostly composed of code with minimal explanation. It feels like a very natural progression of being exposed to something new, getting familiar with it, and starting to explore all the nuances.
I can imagine TFA re-written without the dumb story being really good. It makes use of spaced repetition, and it does a good job of introducing one piece of information at a time. But I agree; the story part of it is stupid.
I wholeheartedly concur about The Little Schemer, though well into adulthood when first exposed to it. It works so well due to combining a Q&A format with a respectful "tone" that never talks down to the student-reader. The authors' subtle sense of humor contributes too.
To be sure this article isn't in the same league at all. Great teachers are scarce no doubt because few are gifted with the requisite raw talent. It implies you're right, it's hard to produce top-quality written teaching material, after all excellent examples are uncommon, even vanishingly rare.
As others have pointed out (e.g.: https://news.ycombinator.com/item?id=30324567 ), the style in this article is not what I had in mind when I coined the phrase "discovery fiction". It also has nothing to do with spaced repetition or having a "more natural flow".
Sorry to drag you into this! I meant “inspired by” in the literal sense, not as a claim that it would fall under your definition. I mentioned so when I tweeted it, but not in the post itself https://twitter.com/paulgb/status/1492923557801869318?s=21
For myself - and I know others respond very differently - I usually want everything in an essay to serve the overarching point. So I find fictional asides pretty distracting. There are exceptions: Imre Lakatos did it well in "Proofs and Refutations", and Douglas Adams did it well too. But it's tough to pull off!
The style can work. In this case it largely doesn't, mostly because the writer insists in trying to be cute about "Tim". Tim is a sockpuppet for the author, like Socrates in Plato's works. After the first couple of paragraphs, there is no point pretending he is anything but that. Trying to flesh him out, over and over through the text, is a waste of everyone's time.
It isn't interesting to list the generic shortcomings of a post. We want curious conversation here.
The moderation comments I post are for sure repetitive and tedious. The justification for them is not that they're interesting in their own right (they aren't! and they're even more tedious to write than to read). It's that without them, HN would be globally worse off. They're an out-of-band feedback channel forming one component of the system by which the forum regulates itself.
You can compare them to medicine which is toxic but which one takes anyway because the alternative is worse.
That was fun, but more about Linux than ports in general. I was expecting something more along the lines of a computer checks the port of every incoming packet and if it matches something it is expecting it does something. Though this was much more interesting, and now I'm curious about even more specifics of how Linux handles this.
Had to read quite a bit before getting to the core insight:
> “So when you listen on a port, you’re really listening on a combination of a port, an IP, a protocol, and an IP version?”
> “Yeah, unless you listen on all local IPs. And if you listen on all IPv6 IPs, you also listen on all IPv4 IPs, unless you specifically ask not to before you call bind.”
> “Right. So the operating system must have, like, a hash map from a port and IP pair to a socket, for each combination of TCP or UDP, IPv4 or IPv6.”*
> “To a list of sockets”, Liz corrects. “Remember how I could listen on more than one?”
> “But it also has to handle listening on all ‘home’ IPs, and to be able to find a socket listening on IPv6 from an IPv4 IP.”
thanks for this summary (the lecture was pretty boring to me). As i understood it, the article explains what happens when you listen to same or similar combinations of a port-ip-protocol-whatever...
but i didn't learn what does it mean to "listen" on a port.
how does two programs communicate by this way?
The port number is one field in a TCP or UDP packet. When you listen, you register with the operating system (or whatever controls the network card, I guess) saying that you want a particular port. When a packet comes in (on the network card), it will be routed to you (on your socket) if it's addressed to you (by including a port number you're registered for).
You can think of the port number as the second half of your IP address. As far as the networking goes, an IP address and a port number are basically the same thing. The port number is just the lower bits of the "combined IP address".
"and an IP version" is more of a matter of interpretation though. I'm pretty sure you can model that behavior in terms of binding to an IPv4 address and binding to its associated IPv6 address. Rather than binding to the same IP with different IP versions.
And it's worth mentioning that IP doesn't have ports. You don't take a port and then divide it into TCP and UDP use. TCP and UDP each independently build their own version of ports.
I have to admit I'm not a fan of "discovery fiction" as presented in this post, but reading Michael Nielson's linked post and some of his examples (e.g https://michaelnielsen.org/ddi/why-bloom-filters-work-the-wa...) I think the idea is great if executed well.
The goal isn't to embed information in meaningless dialogue, but to walk the learner through an entire thought process rather than just presenting the finished result. Or to use one of tech's favorite expressions, to construct an idea from first principles.
Not a blog post, but https://howdns.works/ is a really brilliant (animated) webcomic explaining how DNS works through a story. Each panel has some title text as well.
- the OS maintains a hash map mapping port/IP/protocol/protocol-version to lists of sockets
- there's a scoring algorithm that determines which socket wins, based on specificity of how the socket(s) were bound
- if multiple sockets are bound with SO_REUSEPORT, things are routed based on a hash of the source/destination address/port, presumably so packets from a single client aren't split between different handlers - essentially, simple zero-backpressure load balancing
- read the article for nuances of when and how to use this, especially performance considerations about pre/post fork
- consider adding TL;DR segments like this, whether you're writing discovery fiction like the OP, or technical deep-dives like the link in this comment. Different readers will prefer different formats, but everyone benefits from having the option to consume an abstract before reading lengthy nonfiction prose!
How does ICMP routing work? I always wondered about that. There's no port in ICMP, how does OS understand which ping executable receives ICMP echo replies.
Thanks. It seems that's not exactly correct. ICMP has type, code and checksum fields. Rest of data is type-specific. But echo-request and echo-reply types do have identifier and sequence fields indeed.
I made extensive use of the SO_REUSEPORT flag a couple decades back to build agents that on the same machine would have their own UDP socket which received data from a local piece of software. Basically, by using that flag each socket received a copy of the buffer, so that each agent could act upon the contents ("is it for me? - if yes then read this field and act accordingly, otherwise ignore"). Very handy, despite all limitations of UDP. It allowed me to keep the code very tight and specific for every agent, although network buffer structures were all in common so upgrading the protocol to all agents was easy.
With some adaptations, it worked both on Windows's POSIX stack and Unix.
My recollection of SO_REUSEPORT behavior is that it will deliver a datagram to one listener, rather than all of them. The kernel will distribute datagrams fairly between listeners over time, so none are starved. This makes it super easy to share load over multiple servers, but doesn't get the same datagram to multiple listeners.
Yes, that was the case. Also I probably confused SO_REUSEADDR, which in Windows is implemented quite differently, with SO_REUSEPORT.
Anyway, I can confirm that on Windows (2000) I could open N programs whose receiving socket was bound on the same address:port and all of them would receive a copy of the transmitted datagram. If say I run 5 copies of the same agent, all of them would display the same data. Those were stand alone applications without any shared memory, so apparently the system made a copy of the buffer for each socket. I'm not sure however if I tested this particular behavior also under Unix, since I needed it only on the graphical interface, but on Windows it definitely worked.
The freebsd page doesn't claim that it delivers everything to every listener. It says that's possible if multicast or broadcast is configured. That won't happen by default.
> This option permits multiple instances of a program to each receive UDP/IP multicast or broadcast datagrams destined for the bound port.
Right!? This just shows by example the basics of calling the socket syscalls via Python. I still do not know how these syscalls work or what it really means to be listening to on a port.
It would be improved by changing the framing from a discussion to singular persons internal dialogue showing progression and train of thought. I’ve used this before in a work context to explain complex debugging issues.
“Hmm that doesn’t work, what about X”
<code>
“Oh but that didn’t do what I wanted. What if we added Y and Z”
> Anyone care to comment if the fall into either group?
I know a thing or two about networking and didn't really learn anything new from the article.
And I don't really like the style, but I don't feel like my expertise with networking plays much into it.
Why didn't I like it? Hmm, first of all, I skipped the italic blurb and jumped straight in. So at first I didn't know I was reading fiction. It sounded like story time, as in a story of something interesting that actually happened, and for which you may need to give a bit of context.. I don't have a great example in mind but think something like the 500-mile email (https://www.ibiblio.org/harris/500milemail.html). That's the kind of story I expected I was getting into. And then I was disappointed that there was no story.
Once I got over that and looked back.. well, one obvious thing that's missing for me is motivation. “Yeah, I know that, but how?” Liz says. Why does she care? Why should I care?
I immediately got the same vibe that I get when I come across someone with an X-Y problem and they're the stubborn kind who refuse to explain why or what they're actually trying to find out. Or when I come across someone who's curious but trying to pass the burden onto someone else (kinda like a help leech except that they don't even have any concrete problem they need help with). It wouldn't be the first time I've said on IRC something like "Sorry, I don't know. If you really want me to read the kernel code for you and tell you how it works, I might do that later tonight but I figured you could satisfy your own curiosity." Someone wants to know something but there's no real motivation for the next person to care. I guess I kinda feel how I imagine Tim feels: he can guess what the OS might be doing, but he doesn't have motivation to find out more.
And then there's the fiction-fluff that doesn't really do anything for me. Liz and Tim aren't interesting, there's not much personality, and even if they had personality, there's not much reason to care; they're just random nobodies. The setting isn't interesting. I don't care if there's a coffee shop because the coffee shop also isn't a key element of the story; arguably, there are no key elements. There isn't anything exciting going on, Liz just wants to know how Linux demultiplexes TCP and UDP. So all the fluff feels superfluous and forced.
It could work if there was an interesting story. A reason to build something, a reason to find something out, a reason to poke the kernel around a bit. And in that case, you wouldn't need so much "fluff"; the narrative could support and enrich the story (explain things and continue to add motivation) rather than just pad it. Writing that sort of educative story is really hard though.
Not a networking expert, but like classic literature (and some modern stuff). The writing, especially the dialogue sounds like a primary school first writing assignment.
It'd be ideal to have a very succinct and clear 3 sentence articulation. Followed by maybe a couple of examples.
If we're exploring the mysteries of the Universe, stories help, but our own arbitrary creations? They are just tools. They should be simple, robust, anti-complex and as mundane as we can make them. T
This is horrible to read. I gave up after the first screen page. When I'm trying to extract knowledge from a post or an article, I don't want it padded with useless prose that I need to skip when parsing the text. Like an invasion of this fluff style of "long read" that has become fashionable in what used to be journalism.
Interestingly, the life stories surrounding recipes may be a form of copyright protection.
> In the U.S. copyright law doesn't protect "a mere listing of ingredients," but "where a recipe or formula is accompanied by substantial literary expression in the form of an explanation or directions... there may be a basis for copyright protection."
This is from an article about a tool developed to extract recipes from their blogs:
Does anybody else dislike the socket interface abstraction? To me, an individual TCP port + IP is, effectively, giving me a 'virtual uart'. Instead of saying COM4 or whatever, you say tcp port 12345 IP address 6.7.8.9
Similar thing for ranges of IP address, except in that case, the 'listener' has to figure out which 'virtual uart' to act upon. In that case, the OS splits the 'virtual uart' assignment with the server listening to a range of address.
Sockets seem unnaturally messy to me. Maybe I'm ignorant.
The COM4 endpoint only exists inside that one computer. There is one, and only one COM4 in that one computer; anybody who wants to read or write from/to it knows how to identify it.
A socket is reachable from other computers, so it must have a network address of some kind associated with it. The process that owns the port may want to refuse connections from another computer, so there has be a step in which the receiver accepts an attempted connection.
COM4 can essentially never reject either a reader or a writer.
Yes. It is bad. And it’s warped programmer brains for decades now. If you want to have some fun and design a better interface, go grab a direct kernel-bypass layer 2 NIC interface and learn how much better things could have been. Unfortunately, the sockets api isn’t going away. But, on the positive side, it’s one of the most portable OS interfaces out there.
I switched from hardware engineering to software engineering, and the socket abstraction (as well as the TCP bytestream abstraction, which appears to have been part of the inspiration for how sockets work) seemed absolutely nuts to me. Kernel bypass style queues make a lot more sense from first principles.
I remember some of this on Windows when I was implementing a prototype DNS that could saturate a 10G link.
There were two components, Receive-Side Scaling and some special user-mode thing. RSS would load-balance 5-tuples onto different cores somehow, triggering interrupts on the right core for that stream. If you were careful with the driver stack you could have zero-copy all the way to user mode, where there was some magic that let you dispatch into user mode and read packet data from a ring buffer.
Gave me an appreciation for how a single machine is made of lots of pieces, and how frustrating it is to chase perf.
The socket API is slowly going away though. On iOS devices the socket API simply isn't good enough to deal with complicated things like transitioning from cellular radio to Wi-Fi. Apple has discouraged the traditional socket API and encouraged their own CFStream/NSStream API for a long time.
Since I read from left to right its major:minor to me. The major one is the IP, since it can have 65535 ports. Its unfortunately not the same with dates. I prefer ISO 8601. Major:minor:micro (or whatever you call it with 3).
UARTs have 7 vs 8 bit, parity negotiation, speed negotiation and all other manner of arcane config that has to be exactly perfect for anything to flow.
I'm not sure you quite understood my point? I'm not claiming that uarts are the end-all, be-all. What I am saying is that, even with the features of 'sockets' e.g. being able to be a 'server', ultimately, when data transmission happens, it's as if you have a virtual uart.
I don't care about the specific uart-level detail here.
write() and read() work on a connected socket fd just as they work on a uart. So I don't really understand what the problem here is?
Of course you have datagram protocols and recvmsg() & co, which are useful precisely when you're dealing with something that does not looks like a uart.
Go is very much a descendent of Plan 9. It’s written by Rob Pike of Bell Lab / Plan 9, is heavily inspired from languages such as Alef https://en.wikipedia.org/wiki/Alef_(programming_language) which ran on Plan 9 and, if I recall correctly, Go’s original compiler was basically a modified compiler for one of this Plan 9 systems languages (to enable Pike et al to speed up the release
of version 1.0).
Sockets are ridiculously complicated as APIs go. STREAMS [0] was a much nicer networking API but sockets won because in the late 90s sockets were faster (supposedly) and STREAMS became untouchable because they were part of the SCO IP fiasco. Now that SCO is history it might be worthwhile to revive STREAMS and explore it again.
> To me, an individual TCP port + IP is, effectively, giving me a 'virtual uart'.
It's not, though. A single destination can have an arbitrary number of sockets/uarts connected to it. You can server on port 80, and accept a connection from 127.0.0.1 and from 192.168.100.23 and from 10.98.54.4 and from anywhere else on the internet and have them all operating simultaneously. And in fact each destination port can have many connections to the same IP and port combination, discriminated by the source port which is a number the programmer generally never sees (unless you bind it specifically, it's automatically assigned).
It's exactly that kind of almost-right-but-not-quite mental misrepresentation that the article is trying to address.
I don't think you fully understood my point. What I'm saying is that when any comm happens between network endpoints, it's essentially identical to a uart stream.
The fact that a tuple (my IP, my socket, your ip, your socket, protocol) is used instead of COM4 is just a detail.
It talks about mapping a port (and ip address, and protocol) to a process, but on Linux (and I think other unixes as well) that isn't quite accurate. It maps to a file descriptor, and that file descriptor can potentially be share with other processes (for example by forking, or passing it over a unix domain socket). One common way this is used is to pass a socket file descriptor from the old process to the new process when restarting a service.
245 comments
[ 2.2 ms ] story [ 256 ms ] threadDon't get me wrong -- all of this functionality is useful in some context, and it is there for a reason. We cannot radically simplify the kernel without making it less useful. But it is still somewhat sad there is so much essential complexity even in simple things like "listen" call.
It's their stack of hacks.
I’m pretty sure you could design a spec-compliant alternative which didn’t do any of these things, if you really wanted to.
In my experience, the network layer is often way better-designed, more reliable, more interoperable, and less hacky than most OS abstractions built on top of it.
I get the idea of presenting concepts in a more natural flow and smuggling in some spaced repetition, but the whole story just felt extremely forced and artificial to me, sort of like a drawn-out sequence of expospeak.
There has to be a better way to include spaced repetition and discovery in teaching.
Edit: Note this correction by Michael Nielsen: https://news.ycombinator.com/item?id=30325048
Fictional kingdom discovers Trigonometry out of necessity.
I also like that the story encourages experimenting and testing out your mental models, like Liz did.
It's just the particular way they discover things that feels forced to me. The story is all about them coming to their own conclusions and thinking up their own experiments, but - it being a story - you know, it's actually all guided beforehand.
Maybe what irks me is that this concept of a character discovering "on their own" some ostensible deep truth (which is really just the author's personal opinion about something) has been used for a lot of worse reasons in other stories - even though, in this case, the "truth" is perfectly harmless and beneficial.
Anyway, it's always easier to criticize than to create, so I won't say I really have better ideas of how to do it.
I found this video from Veritasium that explains it: https://www.youtube.com/watch?v=rhgwIhB58PA
Which is all interesting, but mostly just disproves VARK and similar approaches to describing differences in how people learn. There are still so many different ways to teach someone something, they're just much more holistic ways of teaching than the simple VARK split. That some people learn better from X course of teaching and others learn better from Y course of teaching still seems likely to me(admittedly, just pulling from personal experience and the anecdotes of others on that ). That we don't have a neat way to categorize that might just mean it's too messy to do so, or could mean we just haven't figured out the right way to look at it yet.
Even if we assume that people all learn the same way, good learning integrates new facts or concepts into ones pre-existing mental model of the world. Not everyone has the same mental model - of this I am certain. Sometimes new information just hangs on the existing model, and sometimes the existing model needs to be updated. This can make the process seem like everyone learns differently, since different explanations can make more or less sense depending what's already in their head.
On top of that, I think some people have (maybe inherently) very different abilities in things like visualization, memorization, vocabulary, etc... So yeah, I think everyone learns differently even if at some neuronal level it's all the same.
This is actually a failure of discourse in many areas of life – "X is hard to solve or haven't thought about it, so it must be that X is different for everyone". Nutrition, Product Reviews, etc. I've learned over years that anyone that claims "X is different for everyone" is most likely exhibiting a defeatist attitude.
The universals about learning, which I've taken away from the discussion, is that active learning and recall is generally effective, while passive learning (e.g. re-reading) is less effective. A good research paper that covers this (which I've mentioned in that discussion) is called "Improving Students’ Learning With Effective Learning Techniques: Promising Directions From Cognitive and Educational Psychology" [PDF]: https://pcl.sitehost.iu.edu/rgoldsto/courses/dunloskyimprovi...
I liked it, for what it's worth.
Yeah, I'm really glad this was the top comment. That style of journalism has become all pervasive today, and the grotesque self indulgence of it is honestly sickening. I refer to it as "college term paper journalism", because they all read like a sophomore English 102 essay.
It's way too long since I read it, so I can't really compare, but I remember that the whole topic was actively discussed and reflected in the novel. That's something different.
Like, Sophie being a character in a novel was an important plot point of just that novel, if I remember correctly.
You can see there's no dialogue; the idea is to frame an article as "how might you discover/invent this concept on your own", rather than just directly explaining how a thing works. You are given a fictional goal, and you "discover" the subject matter by building iterative solutions.
people are often helped by metaphor (e.g. "a port is like a mail slot in a huge mailroom...") but i feel like this "discovery fiction" thing is like an inverse version of that... it allows your brain to be creative when thinking through the topic but just focused on the wrong part of the explanation.
I can imagine TFA re-written without the dumb story being really good. It makes use of spaced repetition, and it does a good job of introducing one piece of information at a time. But I agree; the story part of it is stupid.
To be sure this article isn't in the same league at all. Great teachers are scarce no doubt because few are gifted with the requisite raw talent. It implies you're right, it's hard to produce top-quality written teaching material, after all excellent examples are uncommon, even vanishingly rare.
For myself - and I know others respond very differently - I usually want everything in an essay to serve the overarching point. So I find fictional asides pretty distracting. There are exceptions: Imre Lakatos did it well in "Proofs and Refutations", and Douglas Adams did it well too. But it's tough to pull off!
Other great examples (though in a whole other league) are the phoenix project and the unicorn project, where the characters learn along the way.
Of course the learning and story is "guided" by the author. Every story ever written was.
Great article, enjoyed it.
Personally I want the gory technical details. This reads like those recipe sites that give you an unnecessary life backstory.
The gore may be less approachable, but I'd say the overall effort to understanding the material is less.
https://news.ycombinator.com/newsguidelines.html
The moderation comments I post are for sure repetitive and tedious. The justification for them is not that they're interesting in their own right (they aren't! and they're even more tedious to write than to read). It's that without them, HN would be globally worse off. They're an out-of-band feedback channel forming one component of the system by which the forum regulates itself.
You can compare them to medicine which is toxic but which one takes anyway because the alternative is worse.
https://hn.algolia.com/?dateRange=all&page=0&prefix=false&so...
> “So when you listen on a port, you’re really listening on a combination of a port, an IP, a protocol, and an IP version?”
> “Yeah, unless you listen on all local IPs. And if you listen on all IPv6 IPs, you also listen on all IPv4 IPs, unless you specifically ask not to before you call bind.”
> “Right. So the operating system must have, like, a hash map from a port and IP pair to a socket, for each combination of TCP or UDP, IPv4 or IPv6.”*
> “To a list of sockets”, Liz corrects. “Remember how I could listen on more than one?”
> “But it also has to handle listening on all ‘home’ IPs, and to be able to find a socket listening on IPv6 from an IPv4 IP.”
You can think of the port number as the second half of your IP address. As far as the networking goes, an IP address and a port number are basically the same thing. The port number is just the lower bits of the "combined IP address".
And it's worth mentioning that IP doesn't have ports. You don't take a port and then divide it into TCP and UDP use. TCP and UDP each independently build their own version of ports.
The goal isn't to embed information in meaningless dialogue, but to walk the learner through an entire thought process rather than just presenting the finished result. Or to use one of tech's favorite expressions, to construct an idea from first principles.
https://web.mit.edu/kerberos/www/dialogue.html
https://www.goodreads.com/book/show/17255186-the-phoenix-pro...
TL;DR:
- the OS maintains a hash map mapping port/IP/protocol/protocol-version to lists of sockets
- there's a scoring algorithm that determines which socket wins, based on specificity of how the socket(s) were bound
- if multiple sockets are bound with SO_REUSEPORT, things are routed based on a hash of the source/destination address/port, presumably so packets from a single client aren't split between different handlers - essentially, simple zero-backpressure load balancing
- read the article for nuances of when and how to use this, especially performance considerations about pre/post fork
- consider adding TL;DR segments like this, whether you're writing discovery fiction like the OP, or technical deep-dives like the link in this comment. Different readers will prefer different formats, but everyone benefits from having the option to consume an abstract before reading lengthy nonfiction prose!
[0] https://flylib.com/books/en/3.475.1.74/1/
This seems to confirm: https://lwn.net/Articles/542629/
Maybe the exact behavior varies by platform. I've never used it on Windows.
Yes, that was the case. Also I probably confused SO_REUSEADDR, which in Windows is implemented quite differently, with SO_REUSEPORT. Anyway, I can confirm that on Windows (2000) I could open N programs whose receiving socket was bound on the same address:port and all of them would receive a copy of the transmitted datagram. If say I run 5 copies of the same agent, all of them would display the same data. Those were stand alone applications without any shared memory, so apparently the system made a copy of the buffer for each socket. I'm not sure however if I tested this particular behavior also under Unix, since I needed it only on the graphical interface, but on Windows it definitely worked.
Ahhh, the joys of "portability"...
> This option permits multiple instances of a program to each receive UDP/IP multicast or broadcast datagrams destined for the bound port.
As a person who doesn’t do much networking, I appreciated the context.
I wonder if the people who didn’t like the post style are networking experts who are already steeped in this stuff?
Anyone care to comment if the fall into either group?
“Hmm that doesn’t work, what about X”
<code>
“Oh but that didn’t do what I wanted. What if we added Y and Z”
<code>
I know a thing or two about networking and didn't really learn anything new from the article.
And I don't really like the style, but I don't feel like my expertise with networking plays much into it.
Why didn't I like it? Hmm, first of all, I skipped the italic blurb and jumped straight in. So at first I didn't know I was reading fiction. It sounded like story time, as in a story of something interesting that actually happened, and for which you may need to give a bit of context.. I don't have a great example in mind but think something like the 500-mile email (https://www.ibiblio.org/harris/500milemail.html). That's the kind of story I expected I was getting into. And then I was disappointed that there was no story.
Once I got over that and looked back.. well, one obvious thing that's missing for me is motivation. “Yeah, I know that, but how?” Liz says. Why does she care? Why should I care?
I immediately got the same vibe that I get when I come across someone with an X-Y problem and they're the stubborn kind who refuse to explain why or what they're actually trying to find out. Or when I come across someone who's curious but trying to pass the burden onto someone else (kinda like a help leech except that they don't even have any concrete problem they need help with). It wouldn't be the first time I've said on IRC something like "Sorry, I don't know. If you really want me to read the kernel code for you and tell you how it works, I might do that later tonight but I figured you could satisfy your own curiosity." Someone wants to know something but there's no real motivation for the next person to care. I guess I kinda feel how I imagine Tim feels: he can guess what the OS might be doing, but he doesn't have motivation to find out more.
And then there's the fiction-fluff that doesn't really do anything for me. Liz and Tim aren't interesting, there's not much personality, and even if they had personality, there's not much reason to care; they're just random nobodies. The setting isn't interesting. I don't care if there's a coffee shop because the coffee shop also isn't a key element of the story; arguably, there are no key elements. There isn't anything exciting going on, Liz just wants to know how Linux demultiplexes TCP and UDP. So all the fluff feels superfluous and forced.
It could work if there was an interesting story. A reason to build something, a reason to find something out, a reason to poke the kernel around a bit. And in that case, you wouldn't need so much "fluff"; the narrative could support and enrich the story (explain things and continue to add motivation) rather than just pad it. Writing that sort of educative story is really hard though.
If we're exploring the mysteries of the Universe, stories help, but our own arbitrary creations? They are just tools. They should be simple, robust, anti-complex and as mundane as we can make them. T
Nothing is keeping you from writing it the way you suggested.
> In the U.S. copyright law doesn't protect "a mere listing of ingredients," but "where a recipe or formula is accompanied by substantial literary expression in the form of an explanation or directions... there may be a basis for copyright protection."
This is from an article about a tool developed to extract recipes from their blogs:
https://www.eater.com/22307633/why-are-people-mad-at-recipea...
Similar thing for ranges of IP address, except in that case, the 'listener' has to figure out which 'virtual uart' to act upon. In that case, the OS splits the 'virtual uart' assignment with the server listening to a range of address.
Sockets seem unnaturally messy to me. Maybe I'm ignorant.
A socket is reachable from other computers, so it must have a network address of some kind associated with it. The process that owns the port may want to refuse connections from another computer, so there has be a step in which the receiver accepts an attempted connection.
COM4 can essentially never reject either a reader or a writer.
There were two components, Receive-Side Scaling and some special user-mode thing. RSS would load-balance 5-tuples onto different cores somehow, triggering interrupts on the right core for that stream. If you were careful with the driver stack you could have zero-copy all the way to user mode, where there was some magic that let you dispatch into user mode and read packet data from a ring buffer.
Gave me an appreciation for how a single machine is made of lots of pieces, and how frustrating it is to chase perf.
Sockets don’t look too bad compared to that
I don't care about the specific uart-level detail here.
Of course you have datagram protocols and recvmsg() & co, which are useful precisely when you're dealing with something that does not looks like a uart.
[0] https://en.m.wikipedia.org/wiki/STREAMS
The merge work was driven by Bill Joy and Blaine Garst.
https://www.computerhistory.org/collections/catalog/10271717...
It's not, though. A single destination can have an arbitrary number of sockets/uarts connected to it. You can server on port 80, and accept a connection from 127.0.0.1 and from 192.168.100.23 and from 10.98.54.4 and from anywhere else on the internet and have them all operating simultaneously. And in fact each destination port can have many connections to the same IP and port combination, discriminated by the source port which is a number the programmer generally never sees (unless you bind it specifically, it's automatically assigned).
It's exactly that kind of almost-right-but-not-quite mental misrepresentation that the article is trying to address.
The fact that a tuple (my IP, my socket, your ip, your socket, protocol) is used instead of COM4 is just a detail.
Kit Marlowe was listening on a port down by his favorite tavern, planning on skipping out on bail, when he was knifed to death.
The storytelling style, however is very difficult to skim. It really requires reading dialogues and actions.
However, it's a Sunday afternoon, raining and there's thunder in the background. It's the perfect style to relax and enjoy.
Maybe in the next chapter Tim can confess his feelings for Liz while they learn together about Unix Domain sockets?
But this was such obtuse writing that was painful to read.
I recommend a less clunky narrative based format.