“Some of you might want to do things the Pure Windows Way. That’s very gutsy of you, and this is what you have to do: run out and get Unix immediately!”
He's just joking of course. At first glance this seems like a great read. And he's funny too! :)
I do wonder how much of this is applicable to Rust though.
Sadly the Windows asynchronous (or something, I never quite got it) networking API seems very powerful. But I never found a comprehensible manual/tutorial for it.
I remember chatting in the fringes when nodeJS was doing their proper win32 porting and they did have some issues getting things stable due to lack of documentation but iirc got some help from MS so the libuv code should be usable as a reference for behaviours (As well as being fairly battle tested by now).
I’ve been trying to figure out the entire windows networking stack recently, particularly some more obscure parts and the docs have been pretty bad unfortunately. A lot of the higher level stuff fees really superficial and the lower level stuff it’s it own mess (all sorts of broken links and links to outdated versions in the NDIS docs).
Socket programming never clicked for me before I learned about the protocols themselves first. My brain just rejected using an API for something I didn't understand. So if you're like me, I highly recommend reading Comer's "Internetworking with TCP/IP" first, and sitting down to socket programming later.
Volume 1 of TCP/IP Illustrated by W. Richard Stevens is a good one. If you're very pressed for time, start with the picture of the TCP state machine and work outwards.
Maybe, but in C land that high level socket API never materialized in the standard library. And you are right that it is awful. I've been doing network programming for over 20 years now and I still have to look up the manual when making a TCP/UDP connection in C.
I've made my own wrapper a few times but often don't get to use it for various reasons.
int tcp_connect(char* remote_host, short int port, options* options, static char** error_string);
int tcp_listen(short int port, options* options, static char** error_string);
int udp_listen(short int port, options* options, static char** error_string);
Does any necessary lookups for you and connects sets up the socket. The options structure can be NULL if you want the sane defaults, or you can zero it out and set only the options you care about like the connect timeout or flags like SO_REUSEADDR or O_NONBLOCK. Returns the socket number or -1 if an error occurs. In the event of an error the error_string is also set. The _listen variants spit out a socket ready for an accept() call.
If you need something weird then you're back to the low level BSD socket stuff, but this covers about 95% of my use cases. While it's only saving a handful of system calls, it ends up being quite a bit of boilerplate and error handling. Especially if you specify a connect timeout, since that is kind of awkward to do with the BSD socket API.
Can confirm this is a nice introduction to network programming; it's how I got started. Once you know the basics you can switch to the man pages for more details.
I began writing a book for them in 2010 and quit when the goalposts kept being moved and had a different point of contact every week. Absolutely terrible experience.
Similar experience. In my case, they wanted a video course on a library that wasn't finalized or published, and was still in an obvious beta with frequent breaking changes. On top of that, I was in a 12hr timezone difference, and only wanted to communicate with me regularly at 11/12 at night on week days, which didn't work well. Oh, and they only wanted to use skype, an app I never use normally. I was promised promo codes and release on multiple websites, and I got a single weekend promo code, and only published on two websites. And the icing on the cake is now 100% of the people I worked with don't work at Packt any more, so I don't even have a person I can contact any more to work on promotions or updates.
Back in 2010 I had my first embedded programming job. I needed to do some network programming in C (controlling a PLC over ethernet). My boss at the time told me "just read Beej's guide and you'll be fine". He was right. That system is running 24/7 since then.
Good to see that the guide is still being updated.
You may be able to get it from your local library via an app like Overdrive[0][1]. Some library programs even have a way for you to get free access to Safari Books Online[2] (I'd say most if not all, in the US at least)
My experience has been that there is rarely a backlog on technical materials like this.
Libraries are like the secret weapon of a learner. I don't know why our industry has such a lack of knowledge about this one! I only recently found out about being able to do this myself
(In some ways, we benefit, since there is little pressure for the large publishers not to offer these kinds of things for free via the library system)
Ancedota (and unrelated to the topic of this thread, but interesting none the less I think):
In California, if you go to a state university (but if I recall correctly this doesn't apply to the University of California system, but the CSU system, if I got my facts straight) they must provide a minimum number of copies of each textbook / reading material required for a class be available via the campus library.
I hardly paid for books once I found this out. I just coordinated with others to make sure it was available when needed. Worked well for me.
I could be hazy on the regulation on this though, it might have just been dumb luck that the two places I went to school had this as a policy, but I remember it being explained as I did above.
Heck, I passed my uni network programming courses thanks to Beej. It also helped tremendously that our professor was a seasoned UNIX programmer and during lectures he actually typed in his own code and ran it.
I had a few maths lectures who were highly regarded for turning up and proving things from memory in front of the blackboard. I hadn’t really considered that it might be done in CS.
The best lecturer I had in maths would screw up the proofs every time from memory, but the way he screwed them up taught me a lot more than if he'd gotten them right.
My experience is that they get about 45 minutes into the lecture, realize that there was an error in minute 10 and spend the rest of the period trying to fix it before telling you to check the book for the correct proof.
One professor provided scans of his written notes. For one of the theorems in his notes, he’d stated it, attempted to prove it, failed, crossed out the proof and wrote ‘QED’. At least he was honest enough to include it!
I remember a lecturer of mine having to state and prove a theorem due to Heawood[1]. I think he did it first because the statement was quite unpleasant and not easy to memorise. I found the point of proving things from memory is that to memorise a proof, one must distill it to key ideas with obvious steps in between. This was particularly useful if you might need to reproduce a proof in an exam.
Yes, I used the guide in 2006 I think or it at least looks similar. I had used it to write a little webserver in C to talk between the Flash UI and the internals of the device. Worked well :)
Haha, that was the exact class where I used it! Loved that class, though. By far my favorite in their MS program. It awakened a love for operating systems (and the general applicability of the concepts). AOS was also good and made me appreciate the point of cloud computing more.
Georgia Tech's Graduate Introduction to Operating systems OMSCS class rings a bell. Socket programming was fun in that class. And by fun I mean hours of hair pulling while debugging.
My latest immersion into Beej's guide was through the GIOS course too! It also opened my eyes to the amount of familiarity/expertise my older co-workers had with systems programming (they helped me when when they heard me whining about the socket programming assignment). It taught me not to dismiss someone just because they don't know the latest TensorFlow API/Neurips paper.
I had the most wonderful déjà vu the other day. I needed to do some serious-ish network programming for the first time in 20 years. I googled around a bit for a refresher. What do I find? Beej's guide – the same Beej's guide that was my first introduction to the topic 20 years prior!
It was wonderfully poetic to me that the same document that taught me how to make a wildly dangerous IRC bot in my teenage room for the lulz also re-taught me the stuff I needed to deliver a low-latency ML offloading demo at my serious(-ish) job 20 years later. Gave me a real warm fuzzy feeling :-)
I thought I'd add: I know that this is a very normal thing in some sense. Many textbooks remain relevant for decades and decades (I'm a mathematician, we know this perhaps better than most). But still, it's rarely the case for an online document – one about computer technology even more so!
> It was wonderfully poetic to me that the same document that taught me how to make a wildly dangerous IRC bot in my teenage room for the lulz also re-taught me the stuff I needed to deliver a low-latency ML offloading demo at my serious(-ish) job 20 years later. Gave me a real warm fuzzy feeling :-)
My experience with CTCP flooding large channels channels and causing netsplits in my youth provided a perfect base of understanding for when my production servers started flooding status updates to each other and couldn't stay connected.
PS sorry, undernet admins and users. Also, thanks to the admin on #quebec who asked me to flood his channel so he could tune his bot to block flooders.
Your question is insufficiently specific ... it could mean nearly anything. You need to be clearer about what equipment you have, and what you're trying to accomplish.
As it stands, I don't think anyone can answer except it a completely generic way:
Almost every system has libraries to help, but it depends on what you're running on, what you're running over, and what you're trying to do.
If you're serving webpages, look for a networking library that handles HTTP.
If you're sending arbitrary chunks of data (like for a game), use a socket library. I've used WebSockets in the past, which handles the low-level stuff under the hood. There are libraries that implement it for Java, Go, C++, JS, etc. Check it out at https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_...
It's not totally headache-free though - some issues I ran into were:
A couple of observations regarding select(): its use is not limited to sockets; and, quoting Wikipedia, "with the c10k problem, both select and poll have been superseded by the likes of kqueue, epoll, /dev/poll and I/O completion ports."
Strangely no mention of High Performance Browser Networking
https://hpbn.co
I also would mention Distributed systems for fun and profit
http://book.mixu.net/distsys/
while being not exactly about network programming, it is likely a next step for anyone who's building apps that is scattered across several backends/services connected by the network
Even though some of the API specifics change, the BSD sockets API has been widely copied. It's a pretty decent introduction to the concepts in general.
My first job out of university reqiured networking a small Linux device to an FPGA running a UDP stack, with nothing else. I'd never done anything with networking in C before, and this guide just made everything click
After reading Beej Guide I think I find "The Linux Programming Interface" a good book in general to learn about sockets and other linuxy thing etc. Sadly it hasn't been updated since 2010.
Beej has had great guides for many things on the internet for decades. I fondly remember learning C from Beej's guide to C [1] upon being recommended by WinterMute (from the Nintendo homebrewing community). Best introduction to programming I've seen by far!
As soon as I read the title, I felt joy! I had followed this guide when I had began programming early on and had a great experience. I recognize it by its name alone!
170 comments
[ 4.3 ms ] story [ 199 ms ] threadHe's just joking of course. At first glance this seems like a great read. And he's funny too! :)
I do wonder how much of this is applicable to Rust though.
https://www.drdobbs.com/open-source/io-multiplexing-scalable... (Also covers select and /dev/poll but lacks epoll,kqueue)
https://www.drdobbs.com/cpp/multithreaded-asynchronous-io-io...
I remember chatting in the fringes when nodeJS was doing their proper win32 porting and they did have some issues getting things stable due to lack of documentation but iirc got some help from MS so the libuv code should be usable as a reference for behaviours (As well as being fairly battle tested by now).
https://github.com/libuv/libuv/tree/v1.x/src/win
Well, some APIs are not meant to be used directly, but hidden behind a class or more application-specific functions.
I've made my own wrapper a few times but often don't get to use it for various reasons.
Does any necessary lookups for you and connects sets up the socket. The options structure can be NULL if you want the sane defaults, or you can zero it out and set only the options you care about like the connect timeout or flags like SO_REUSEADDR or O_NONBLOCK. Returns the socket number or -1 if an error occurs. In the event of an error the error_string is also set. The _listen variants spit out a socket ready for an accept() call.If you need something weird then you're back to the low level BSD socket stuff, but this covers about 95% of my use cases. While it's only saving a handful of system calls, it ends up being quite a bit of boilerplate and error handling. Especially if you specify a connect timeout, since that is kind of awkward to do with the BSD socket API.
If you want a cross platform network programming guide in C, Lewis Van Winkle has written an excellent reference book[1].
[1]https://www.amazon.com/Hands-Network-Programming-programming...
Good to see that the guide is still being updated.
edit: typo
Is there any free or low-cost version?
My experience has been that there is rarely a backlog on technical materials like this.
[0]: https://www.overdrive.com/
[1]: Example of for the LA public library of what I mean: https://lapl.overdrive.com/search?query=The+Linux+Programmin...
[2]: https://www.spl.org/books-and-media/books-and-ebooks/oreilly...
(In some ways, we benefit, since there is little pressure for the large publishers not to offer these kinds of things for free via the library system)
Ancedota (and unrelated to the topic of this thread, but interesting none the less I think):
In California, if you go to a state university (but if I recall correctly this doesn't apply to the University of California system, but the CSU system, if I got my facts straight) they must provide a minimum number of copies of each textbook / reading material required for a class be available via the campus library.
I hardly paid for books once I found this out. I just coordinated with others to make sure it was available when needed. Worked well for me.
I could be hazy on the regulation on this though, it might have just been dumb luck that the two places I went to school had this as a policy, but I remember it being explained as I did above.
[1] https://en.m.wikipedia.org/wiki/Heawood_number
I have fond memories of that guide and it's always a treat when it pops back up.
It was wonderfully poetic to me that the same document that taught me how to make a wildly dangerous IRC bot in my teenage room for the lulz also re-taught me the stuff I needed to deliver a low-latency ML offloading demo at my serious(-ish) job 20 years later. Gave me a real warm fuzzy feeling :-)
PS:
HN disucssion anno 2015: https://news.ycombinator.com/item?id=9445692
Anno 2017: https://news.ycombinator.com/item?id=13983212
My experience with CTCP flooding large channels channels and causing netsplits in my youth provided a perfect base of understanding for when my production servers started flooding status updates to each other and couldn't stay connected.
PS sorry, undernet admins and users. Also, thanks to the admin on #quebec who asked me to flood his channel so he could tune his bot to block flooders.
edit: it was the 90s
Can I use those low level API's or do you know some server/client libraries which simplify this for UDP and TCP channels?
As it stands, I don't think anyone can answer except it a completely generic way:
Almost every system has libraries to help, but it depends on what you're running on, what you're running over, and what you're trying to do.
There are also thousands of higher-level libraries that would let you avoid having to deal with these lower-level APIs directly.
[1] https://beej.us/guide/bgnet/html/#poll
If you're sending arbitrary chunks of data (like for a game), use a socket library. I've used WebSockets in the past, which handles the low-level stuff under the hood. There are libraries that implement it for Java, Go, C++, JS, etc. Check it out at https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_...
It's not totally headache-free though - some issues I ran into were:
- running locally vs over the internet
- data formatting
- recovering from timeouts/dropped connections
https://idea.popcount.org/2019-11-06-creating-sockets/
https://idea.popcount.org/2019-12-06-addressing/
It felt incredible that _I_ could do that as someone who had just started programming.
Thank you Beej.
> Your search - site:beej.us sctp - did not match any documents.
In contrast:
https://www.google.com/search?q=site%3Abeej.us+udp
> About 9 results (0.25 seconds)
So I'd guess that the guide doesn't cover sctp.
========
Then:
https://duckduckgo.com/?q=is+sctp+support+part+of+Kernel%2Fg...
Returns:
> https://www.kernel.org/doc/html/latest/security/SCTP.html
(and more ...)
I also would mention Distributed systems for fun and profit http://book.mixu.net/distsys/ while being not exactly about network programming, it is likely a next step for anyone who's building apps that is scattered across several backends/services connected by the network
- https://beej.us/guide/bgnet/html/#windows
My first job out of university reqiured networking a small Linux device to an FPGA running a UDP stack, with nothing else. I'd never done anything with networking in C before, and this guide just made everything click
[1]: https://beej.us/guide/bgc/