Good to see that people still have this spirit of rolling their sleeves up and doing something like this for fun. Underneath it there is a very real point about mono-cultures and the increased magnitude of the impact of a risk, although more eyes checking one implementation should reduce the probability.
Prediction: as agentic code generation gets better, more and more code will be written JIT for it to be executed for security reasons. The engineer of the future is not only the one who uses agents to code, he uses agents to code each time "the" program is ran.
What's there to gain security-wise? Nondeterministic generation & then execution of that code without proper verification seems to be the opposite of a good security posture.
Humanity is a Type I civilisation. It has energy constraints. AI coding runs at hundreds of watts, and more capable agents consume more. That is paid, today, from capex, provided by the biggest bubble in human history. I can deliver your fantasy by running a space heater while i download a docker image.
So do I. According to today’s prevalent tech/science culture, “everyone” should
- write their own network stack, compiler, text editor, operating system
- learn 3-5 foreign languages
- travel extensively
- learn how computers work from the silicon principles to the userspace
- have a complete understanding of algorithms and data structures before even attempting to build anything, because nothing else will do
- read obscure Russian poetry
- “deeply” engage in 5-10 different philosophical practices and meditation techniques
- work out regularly, but according to strictly scientific principles, which you first have to research by reading a dozen books
- appreciate fine art, homestead food, and family life
Sorry folks, y’all are just Instagrammers with MSc’s. Most people here probably have no idea how ridiculous this world appears once you step out for a year or two and then take a look back in.
The pursuit is performative and pretentious, in much the same way as body culture on Instagram. It’s about collecting achievements in an imaginary social game, and then bragging about it, while insisting that it’s not like that at all but some modern pathway to enlightenment.
To be clear, I’m not saying that people don’t genuinely achieve many of these things; they do. Many Instagram models also genuinely have spectacular bodies. It’s not about whether, but about why they are doing this, and the culture surrounding all of it.
Living my life is just everything I do? If I did all those things I’d live an extremely rich life, a lot better than what I’m currently doing, which is some mix of being addicted to the internet and watching all the interesting things get taken away by AI.
Everyone should: be computer illiterate, monolingual, never leave their home town, use left-pad, be regular illiterate, sedentary, uncultured, unlanded, and... alienated from their family?
Yes, everyone should explore the details of the technology they build on top of, and also should explore questions about what it means to be a person or how the world works.
My first job was at Westinghouse instrumenting a nuclear power plant. Our entire hardware-software package was a freebie given to the customer as part of a $200 million refueling contract (and that's in 1988 dollars). We, a team of 10, created a bespoke application stack which was effectively a web browser. The client machines were Sun 3 workstations. The server was some industrial mini computer that didn't have a TCP/IP stack - so we did in fact have to roll our own network stack.
Hah, the first engineering job I had was on Ovation DCS doing power plant controls. That system traces the lineage to the WDPF systems.
Was the server you mention here the actual DCS? With the Motorola CPU's? Running vxworks? Pretty cool recognizing something niche that is still, vaguely familiar. :)
The plant computer wasn't a DCS. I think it was a DEC. It gathered data from the control systems and instrumentation and had a bespoke in memory realtime-ish database. That database was then partially replicated down to the Sun workstations (code the 23 year old me wrote). Then the browser consumed that via a unix shared memory interface.
Was it perhaps DEC Baseway[1] system? It would arguably qualify as a form of DCS, with central VAX servers and running high level software, PDP-11 "shop floor" computers, which in turn talked with PLCs and related devices.
And additionally, TCP/IP was an optional layered product - either DEC's own UCX (Ultrix Communication eXtensions) or 3rd party
I guess they didn't buy the optional network software.
I remember working with this engineer writing in the lower level TCPIP stack on the deck - maybe in Fortran? Anyway, he didn't understand why he wasn't getting all the data transmitted and while I was no expert in TCP/IP, I was already familiar with packets and maximum segment size (MSS). I remember being impressed that he banged out a network stack in a couple weeks.
Baseway, and various other industrial control systems that also ran on VMS and related, tended to either use DEC-suggested (DECnet) protocols or implemented custom Ethernet-layer ones
Graduated and figured I should get a job. Honestly wasn't enthusiastic about the idea. Saw an ad in the paper. It may have mentioned C programming. I went and interviewed with two senior engineers. All they asked was do I know how to do TCP/IP. I said that I did and was working the next week. Honestly the coolest job I've had in my career. Colleagues were all engineers - mostly nuclear.
A while back I wrote an FPGA-based minimal network stack to work with the MRMAC hard IP of an AMD VC1902 with basically the same functionality: ARP, IPv4, ICMP Echo reply and UDP send/receive. I added the bare bare minimum to get UDP GRO working on the receiver side (Linux).
I also enjoyed the learning experience and works perfectly fine at up to 100G. But I definitely appreciate that I didn't have to go through the same pain under Linux where this stuff just works.
Everyone should build their own ** is the general advise I would give just about anything. Otherwise it is hard to appreciate how much effort it goes into things that we take for granted.
Everything looks simple from the outset until you try to build it yourself.
We do these things not because they are easy, but because we thought they would be easy.
I also think there is an incredible transference of skills from one domain to another. This is a risk (everything looks like nail when you have hammer), but there are many domains that are very mature where you can learn a trick and apply it to a new domain over and over again.
I recently saw this guy on HN who had written a web server for an Atmel AVR and also implemented a TCP/IP stack to boot. It obviously was far from feature complete, but it did work. He used SLIP to communicate with the "server."
well that's the trick. If you look at implementing a Web browser or a physics engine you instantly give up, and you are right about it, it's just too hard. If instead you write a partial HTTP client which can display some text from a URL or have a 3D scene and in it an object Y coordinates get smaller each millisecond then suddenly in terms of the essence of each, you are 80% there really. Now if you want to implement the "rest" you might need an entire lifetime, or more. Still IMHO it's definitely worth doing very compact and limited versions of all those in order to learn, even if done only once in your life.
I used to implement these things professionally and to write an IP + TCP + HTTP is certainly less than a year with experience. would I ship that and expect people to use it without a lot of burnin? no..but to say that its a lifetime of work is off by at least a factor of 10.
Maybe not an HTTP client but a modern browser with its JavaScript engine, encryption support, CSS engine, etc I think for most people really is a lot of work. If one does rely on the right libraries it's a weekend of fun (and pain) so I would argue between few hours and a lifetime, depending on how much you implement.
My point is that the essence, the core mechanism, is typically a 1-liner, but only with a pedagogical context, not a production (in the sense of anybody else using it reliably in realistic situations) context.
Back in the 8-bit days, we all wrote our own kernel process schedulers in order to make simple games. If you had an Apple II and wanted music, you'd need to make yours a hard real time one as well.
This is just an opinion and not a very sensible one at that. The Linux TCP/IP stack is one of the most scrutinized network stacks in the world and more than likely free of bugs that coud lead to a server takeover.
lols, what about the sendfile and other vulnerabilities that lasted decades. Linux just does not have a secure mentality, it has a hacker mentality, maybe acceptable for a lot of uses, but it is what it is.
Huh, I thought the thesis was going to be "... because it's a good learning experience everyone should do to understand how networking works" but the reason offered comes down to security. However, I'm going to go with what I had thought originally and actually make this an assignment for students. Anyone have experience witht that, could this be done in a semester?
I did this for a couple years at University of Washington. We did IP, TCP, RIP and a couple other things. they weren't compatible - they were kind of stripped down to the bare essence. I wrote the sample and the tests, and then I ripped out the guts and presented them with a shell, went over the protocol in lecture and provided a kind of spec for them to implement.
the parts I was most proud of was (a) your score was directly related to the number of tests passing, so you know in advance what your score would be and could decide how much effort. (b) every assignment came with a relevant visualization that showed things like the routing table as a directed graph, or the TCP sequence diagram. I was available in the lab for a good 2-3 hours a day on weekdays to answer questions.
I think some of the students got a pretty good picture of how things worked, but the overwhelming opinion was that it was just too much work. a lot of the students just copied off each other. we also had quarters, so ...
I would seriously recommend against building your own network stack for production use. Or writing your own web server for production use. Or database (whatever SQL or NoSQL) also for production use.
These sort of things are more complicated than you could expect and have plentiful pitfalls. Whatever you can do in reasonable amount of time is probably not that good compared to existing alternatives.
If you have actually pressing reason to build one and are ready and have means to spend time and effort on it go ahead. But carefully consider the effort needed...
holy shit. I was just tinkering with my production Laravel stack serving thousands in production trying to figure out how to improve worker concurrency (running out of RAM) and I encounter a random comment on HN which appears to solve my exact problem. This is so serendipitous
Wanting to incorporate an http server into a C project of mine, the only battle-tested or feature-complete options either allocated dynamic memory UTH without an option to override the allocators, or are too opinionated with the core API, like mandating callbacks for everything. Suprisingly there's no decent (single header) C library that don't have these 2 issues, at least that I could find.
This article might be a bit tongue-in-cheek but it's actually something I've been thinking seriously about for a while. There is a security advantage in not using popular technology stacks and rebuilding everything public-facing in-house. Because with Mythos-level AI capabilities, it is almost guaranteed that anyone with access to Mythos/Cyber can find a zero day in your stack. If your stack is proprietary and bespoke with no public access to source code it's much harder to find vulnerabilities, because they can only probe your public facing interfaces.
They won't be compatible with each other, guaranteed. Compatibility in reality comes from everyone using the same reference implementations that have been compatibility-tested, not from everyone implementing the same specifications.
I used to do this work in the IETF. take draft specs and write a C implementation and compare notes with the spec authors and other implementations. sometimes I would find a bug later on and see the same bug in multiple vendors implementations, and I would send them mail describing the issue and the fix.
its been very sad for me to see standardization get ignored, and the implementation become the point of reference. for a long time the only way to implement QUIC was to try to extract it from the Chromium code base - a pointless waste of time. but they spec for years was a couple of pages of meeting notes and a sample compression dictionary.
if we continue down this road we are impeding our own ability to evolve. if compatibility is defined by the implementation then were left with nasty balls of antiquated source that we carefully move around and try to plumb into our new environments, but no one really understands.
at the core, things like IP are very straightforward. we can and should specify the semantics of every bit, and have a wide variety of other implementations to test against. this provides a path for us to build new techniques and environments that are still useful in the world that exists today.
This is the same erroneous reasoning as historical claims that FOSS was less secure due to being public. That has been disproven many times over by now.
The only way it might hold here is if no one with access to the cutting edge models bothers to run your code through one while an adversary does. That seems highly unlikely, particularly for any well funded organization.
That said I agree that monoculture is bad. Vulnerabilities like heartbleed or spectre are so devastating precisely because they apply to approximately everyone.
> The only way it might hold here is if no one with access to the cutting edge models bothers to run your code through one while an adversary does. That seems highly unlikely, particularly for any well funded organization.
But since AI models are non-deterministic and the harness matters a lot, simply "running your code through one" doesn't guarantee you'll find all the zero days. An attacker can simply run a few more iterations or use a slightly more advanced harness, or increase the temperature level and they are likely to find zero days which the maintainers didn't.
> But since AI models are non-deterministic and the harness matters a lot, simply "running your code through one" doesn't guarantee you'll find all the zero days.
The exact same line of reasoning applies to human security researchers and pen testers as well. The only difference is that automated testing via frontier models is affordable even for individuals.
The VM comment is total nonsense. That's only going to be the case for an extremely short period here until the major projects all adapt their release procedures to include automated security audits. I imagine the kernel as a whole and KVM in particular will begin receiving a huge amount of such attention. I also expect increasing amounts of such load bearing code to be formally verified as the primary barrier to that has always been the amount of human time required.
The real nightmare scenario is not just finding a vulnerability in the Linux network stack but finding vulnerabilities at the protocol level in TCP/IP itself or HTTP. That’s a problem that cannot be fixed. Has there ever been a precedent for this in computing?
When these things come up, what usually happens is implementations stop following the spec as written.
Many issues can be handled with minimal coordination, but some things work much better with consensus and you see flag day campaigns. Ipv6 flag day had very limited impact on that day, but may restarted the process and actually got things moving. DNS flag days have been successful.
Yes, lots. Early spdy was vulnerable to CRIME attacks and so was TLS1.0 (and to bunch of other attacks). Kaminsky DNS cache poisoning is another one that comes to mind
Many: OS detection (ICMP, IGMP, ARP, TCP, etc.), topology disclosure (ICMP), DoS (many early protocols did not perceive resource exhaustion in their threat model; a prominent example being SYN floods / TCP half-open), etc.
This kind of project is so much fun and a great learning exercise. While I respect the tongue-in-cheek proposition, we really would benefit from a wider understanding of network protocols. Doing an exercise like this is eye opening.
75 comments
[ 1.6 ms ] story [ 7.6 ms ] thread- write their own network stack, compiler, text editor, operating system
- learn 3-5 foreign languages
- travel extensively
- learn how computers work from the silicon principles to the userspace
- have a complete understanding of algorithms and data structures before even attempting to build anything, because nothing else will do
- read obscure Russian poetry
- “deeply” engage in 5-10 different philosophical practices and meditation techniques
- work out regularly, but according to strictly scientific principles, which you first have to research by reading a dozen books
- appreciate fine art, homestead food, and family life
Sorry folks, y’all are just Instagrammers with MSc’s. Most people here probably have no idea how ridiculous this world appears once you step out for a year or two and then take a look back in.
To be clear, I’m not saying that people don’t genuinely achieve many of these things; they do. Many Instagram models also genuinely have spectacular bodies. It’s not about whether, but about why they are doing this, and the culture surrounding all of it.
Let's invert it.
Everyone should: be computer illiterate, monolingual, never leave their home town, use left-pad, be regular illiterate, sedentary, uncultured, unlanded, and... alienated from their family?
Well, I'm off to a great start!
Was the server you mention here the actual DCS? With the Motorola CPU's? Running vxworks? Pretty cool recognizing something niche that is still, vaguely familiar. :)
And additionally, TCP/IP was an optional layered product - either DEC's own UCX (Ultrix Communication eXtensions) or 3rd party
[1] https://bitsavers.org/pdf/dec/vax/vms/baseway/
I remember working with this engineer writing in the lower level TCPIP stack on the deck - maybe in Fortran? Anyway, he didn't understand why he wasn't getting all the data transmitted and while I was no expert in TCP/IP, I was already familiar with packets and maximum segment size (MSS). I remember being impressed that he banged out a network stack in a couple weeks.
I also enjoyed the learning experience and works perfectly fine at up to 100G. But I definitely appreciate that I didn't have to go through the same pain under Linux where this stuff just works.
Everything looks simple from the outset until you try to build it yourself.
I also think there is an incredible transference of skills from one domain to another. This is a risk (everything looks like nail when you have hammer), but there are many domains that are very mature where you can learn a trick and apply it to a new domain over and over again.
well that's the trick. If you look at implementing a Web browser or a physics engine you instantly give up, and you are right about it, it's just too hard. If instead you write a partial HTTP client which can display some text from a URL or have a 3D scene and in it an object Y coordinates get smaller each millisecond then suddenly in terms of the essence of each, you are 80% there really. Now if you want to implement the "rest" you might need an entire lifetime, or more. Still IMHO it's definitely worth doing very compact and limited versions of all those in order to learn, even if done only once in your life.
My point is that the essence, the core mechanism, is typically a 1-liner, but only with a pedagogical context, not a production (in the sense of anybody else using it reliably in realistic situations) context.
"Please don't"
All software is to be considered insecure, formally speaking.
the parts I was most proud of was (a) your score was directly related to the number of tests passing, so you know in advance what your score would be and could decide how much effort. (b) every assignment came with a relevant visualization that showed things like the routing table as a directed graph, or the TCP sequence diagram. I was available in the lab for a good 2-3 hours a day on weekdays to answer questions.
I think some of the students got a pretty good picture of how things worked, but the overwhelming opinion was that it was just too much work. a lot of the students just copied off each other. we also had quarters, so ...
These sort of things are more complicated than you could expect and have plentiful pitfalls. Whatever you can do in reasonable amount of time is probably not that good compared to existing alternatives.
If you have actually pressing reason to build one and are ready and have means to spend time and effort on it go ahead. But carefully consider the effort needed...
https://github.com/Qbix/webserver
I worked closely with Claude though
Engineers are an implementation detail.
its been very sad for me to see standardization get ignored, and the implementation become the point of reference. for a long time the only way to implement QUIC was to try to extract it from the Chromium code base - a pointless waste of time. but they spec for years was a couple of pages of meeting notes and a sample compression dictionary.
if we continue down this road we are impeding our own ability to evolve. if compatibility is defined by the implementation then were left with nasty balls of antiquated source that we carefully move around and try to plumb into our new environments, but no one really understands.
at the core, things like IP are very straightforward. we can and should specify the semantics of every bit, and have a wide variety of other implementations to test against. this provides a path for us to build new techniques and environments that are still useful in the world that exists today.
The only way it might hold here is if no one with access to the cutting edge models bothers to run your code through one while an adversary does. That seems highly unlikely, particularly for any well funded organization.
That said I agree that monoculture is bad. Vulnerabilities like heartbleed or spectre are so devastating precisely because they apply to approximately everyone.
> The only way it might hold here is if no one with access to the cutting edge models bothers to run your code through one while an adversary does. That seems highly unlikely, particularly for any well funded organization.
But since AI models are non-deterministic and the harness matters a lot, simply "running your code through one" doesn't guarantee you'll find all the zero days. An attacker can simply run a few more iterations or use a slightly more advanced harness, or increase the temperature level and they are likely to find zero days which the maintainers didn't.
The exact same line of reasoning applies to human security researchers and pen testers as well. The only difference is that automated testing via frontier models is affordable even for individuals.
The VM comment is total nonsense. That's only going to be the case for an extremely short period here until the major projects all adapt their release procedures to include automated security audits. I imagine the kernel as a whole and KVM in particular will begin receiving a huge amount of such attention. I also expect increasing amounts of such load bearing code to be formally verified as the primary barrier to that has always been the amount of human time required.
Many issues can be handled with minimal coordination, but some things work much better with consensus and you see flag day campaigns. Ipv6 flag day had very limited impact on that day, but may restarted the process and actually got things moving. DNS flag days have been successful.
This is a far cry from an actual network stack.