At lunch today I was just talking about Sendmail, which I can assure you is a rather rare occurrence. I was talking about the first time I set up sendmail, back in '91 or '92. I was using the bat book and nearly tore my hair out over a week getting that first setup working. I eventually came to understand and appreciate the m4 config, but I ended up moving to qmail and postfix in the mid '90s and never looked back.
Realistic. And, believe it or not, I know of at least one organization that plans to convert an entire literal skyscraper of office space from routed networks to a single, flat switched network for all the employees of all the subcompanies. In 2023.
Obviously everyone with a bit of braincells left tells them to not do that because it's utterly dumb, but hey, strategic decision by the holding company to save on costs...
At least they're not using hubs. (For the younger generation: a hub is an Ethernet device that takes any packet it ingests in one port and sends it out to all other ports, with no consideration at all if the device that the packet is destined for actually is on that port - something a switch does, by maintaining a mapping of MAC addresses to ports. Extremely dumb devices, but used to be way faster and especially cheaper than switches in the 90's/early '00s)
I still keep an old 4-port hub in my junk-box because that way I can diagnose/snoop on network traffic... Although so much of it is encrypted these days that it's harder to see what's going on.
P.S.: Yes, modern alternatives would be to to buy a switch and that can be configured to "mirror" packets onto a chosen port, or a smalls Ethernet network tap unit... But why buy more stuff if I don't really need to?
"In a "switched" network, when Device A wants to send data to Device B, the switch directly connects these two devices so they can chat. Think of it like a train switcher that directly links Track A to Track B for a specific train, instead of sending it through a maze of tracks where other trains are moving.
In contrast, a "hub-based" network is like a party line in old telephone systems. When Device A talks, EVERY device hears it, but only Device B cares and listens. This is less efficient and can be slower because all devices get the data, which clogs up the network.
Another option is a "routed" network, where a router decides the best path for the data. This is like GPS choosing the best route based on current traffic conditions. It's more flexible but can introduce more delays because the data might go through multiple steps to reach its destination.
It's called "switched" because the switch acts like a railroad switch operator, making a direct track connection from one device to another for each piece of data. It "switches" the pathway specifically for that data to make the communication as direct as possible."
That's on purpose—we want good articles to get multiple chances at attention, and we want the 'classics' to pop up periodically (just not too often), so newer cohorts of users get some exposure to them. So you did well :)
The purpose of linking to past threads is not to imply anything bad about the repost (if it were a bad repost, we would handle it differently) - rather, it's to point curious readers to other discussions on the same topic that may interest them.
I love this feature of HN. Between this, the onion salesman, C+- and a few other favorites I’m on a roughly 2 times a year cadence of getting to reread old favorites!
Every time I read this story the part that always surprises me again is the units command. Converting from 3 millilightseconds to miles is brilliant, and I am delighted every time that the units command can do this.
And Trey Harris's "500 mile email" story is what clued me on to GNU units and its capabilities.
Reminder: if you're on MacOS, or one of the BSDs, your default units is from BSD, not the GNU version, and is far less capable. GNU units can be installed on MacOS through Homebrew. The package is "gnu-units", the command is "gunits" once installed.
GNU Units is also available in FreeBSD ports, just type "pkg install units" (or go to /usr/ports/math/units and "make install"). It likewise gets the binary name "gunits"
He made the story up. When confronted with the fact that units didn't convert some of the units from his story his response was essentially "Oh yeah, well of course I always supplemented my units definitions on all my machines with obscure units.".
These potential anachronisms are easy to insert when retelling a story. It doesn't really mean he "made it up", but reverse engineering how the diagnostic was determined might lead one to use tools/definitions that didn't exist at the time.
It's also entirely plausible that a local definitions file was used to provide the unit that didn't come in the default install. Hence the anachronism is merely "potential" instead of "definite"
Anyone who likes the units command should plan an evening where they can sit in a comfortable chair with an appropriate beverage, and read all the comments in the data file in the source. It is like a novel about the history of measurement.
"You waited a few DAYS?" I interrupted, a tremor tinging my voice. "And
you couldn't send email this whole time?"
"We could send email. Just not more than--"
"--500 miles, yes," I finished for him, "I got that. But why didn't
you call earlier?"
"Well, we hadn't collected enough data to be sure of what was going on
until just now." Right. This is the chairman of *statistics*. "Anyway,
I asked one of the geostatisticians to look into it--"
"Geostatisticians..."
"--yes, and she's produced a map showing the radius within which we can
send email to be slightly more than 500 miles.
Pure gold. I love that the stats department put in such rigorous testing before submitting the ticket.
I know Trey. He's one of the most honest and forthright people I've ever met.
To a fault, probably.
And as the former comp.mail.sendmail FAQ maintainer, I can confirm all sorts of weirdness can and will happen if you mix versions of the binary versus the config file.
He certainly was honest and forthright about how much he embellished and... "took license" to spin this yarn. After reading the FAQ, I wondered if there was even a kernel of truth to be found in the story, because he successfully poked holes in practically every detail that made it interesting.
+1, After two years from joining HN I’m still learning about it. This is the first time I heard about highlights section! I couldn’t find it in lists nor on any other part of the site yet still interesting to read some comments there that do not show up in best comments section. How exactly this works?
We do in fact have down votes, you just need 500 (I think?) karma to be able to use it. When you see comments that are not "dead" but are greyed out, they've been downvoted more than they've been upvoted.
/bestcomments (or whatever name may become in the future) had a drawback, I believe. Technically speaking (and generally), the comments in this section are the ones that belong to stories from /best (or highest-upvoted). These stories obviously get more exposure and attention, so their top comments get more upvotes. For instance, a story with 2,000 points might have a comment with 200 points (which is tenth the score of its parent story) that will obviously hit the /bestcomments, while a comment of, say, 50 points on a 150-point story (which is a third) may never have the chance to hit /bestcomments.. Just some thoughts about my daily social network :)
There is probably some sweet spot of n between 0.5 and 2, but not near 1, such that the quotient of comment score^n and thread score^n gives good results.
I knew this existed, and I was looking for it a few weeks ago; it's an interesting page to browse through every once in a while. But I just couldn't remember the name until now.
Having a "highlight" people can't find doesn't seem much of a "highlight" to me?
Edit 2: I feel like adding an "I feel lucky" link on there that would give you a random sample from the list. It's in reverse chronological order and it would take quite a while to scroll back through all of the comments (there are over 400 at present).
I was wondering how I could have browsed to /lists and /highlights and eventually found the first one in the footer of the page. May I suggest to add /highlights there as well? Maybe just after FAQ and before Lists?
> I feel like adding an "I feel lucky" link on there that would give you a random sample from the list. It's in reverse chronological order and it would take quite a while to scroll back through all of the comments (there are over 400 at present).
How about a link with reverse chronology of random highlights, one per year of HN history?
This could inspire nominations for the highlights back catalog.
Yeah, at the time of writing I kinda defaulted to assuming it was "just" a tag for posts, which one can use the normal search on. But a while later realized that it very likely isnt.
But reading through the list, you made dang good choices in highlighted comments.
i think about this story often and i find that the person who figured out that it was 500 miles actually deserves more credit than they get in the story. have to really think out of the box to figure that one
Sometimes you just need to gather all the young'uns around the campfire and regale them with the tales of old, like The 500-Mile Email, or The Story of Mel.
I've wondered how feasible it would be to do something like this to have a website that could only be accessed when a client is within a certain physical proximity of the host. Could make for a fun CTF!
My quick hack would be to establish a websocket connection, and send a random stream of numbers to the client. If the client didn't return the number within a ping threshold, block their access.
Sure, but then you have the really interesting challenge of trying not to block your next door neighbour who’s accessing your service via tor/satellite.
Depends on the satellite; the ISS orbits at 211 miles, which means that it would be close enough to send a 500-mile email if it's flying directly overhead. :P
I’ve played CTF challenges where the latency to the host was a key factor in determining if you could get a flag or not. For those, I’ve often found it useful to spin up a cloud machine in a datacenter near the target (or, better yet, in the same datacenter if we can figure it out).
A very common case is when the challenge has a short timeout but requires a lot of interaction, e.g. you only get ten seconds but you have to perform 10000 queries for a heap spray or something.
The most insidious case I remember was a read() call that didn’t check the result, causing it to return short if the fragments of the input didn’t arrive fast enough.
So... if you're referring to a challenge that did that during one of the DDTEK years of DEFCON-CTF, that was one of mine.
The expectation wasn't to buy time in an adjacent cloud, but to use out of order ip fragmentation or tcp segments, having the servers network stack reassemble the packets such that the read was coherent in one go.
My goal was to teach competitors to model real world challenges of exploitation.
Cool! I don't believe I was referencing any specific challenge. We've had a number of challenges throughout the years that have had these issues. The most common culprits are challenges hosted by Asian organizers, as the ping times from America to some parts of Asia tend to be quite bad sometimes.
The challenge you're describing sounds cool - I wish I'd played it! By the time I started playing DEF CON CTF though, it was with LegitBS as organizers.
Didn't do that, but one of my earliest "dynamic" websites ca mid 90's would have a CGI try to ping the client with a short timeout, and if we got an answer that indicated a leased line or something rather than dialup and we'd serve up a heavy animated version of our logo instead of a static image... But could be used as a vague indicator of distance too.
Trickiest part of doing that today is so many fewer hosts are reachable via icmp, so you'd probably be better off serving up an initial response with some JS to measure more accurately.
(Another silly little thing we added was a link back to a users own ISP from the top ten or so of our competitors based on net block - got us a worried phone call from one of them who thought we'd been hacked and wanted to make sure we didn't think he was responsible)
Hey thanks for linking my site! I was wondering where the extra traffic was coming from :)
If anyone has any stories that I should add, please send them my way!
Also, I have been slacking a little with the newsletter because of life events. Since there have been several sign ups since this post, I'll be more active in getting the weekly newsletter out.
Hey I'm enjoying your site! It would be cool if there was a way to just get the stories. For example, Twitter is no longer capable of loading tweet replies on my machine for whatever hilarious reason, and so I could only read the teaser for the Twitter link.
Anyway, I understand that's probably not the kind of site you had in mind. Thanks for the cool site!
Thanks for the suggestion! Maybe I should add archive links from The Internet Archive for stories in case links die. In the case of tweets, I’d probably need to screenshot them or something.
That's terrific. I was worried that it was going to end at the terse OP, but the follow-up post has just the kind of play-by-play details that I can't get enough of in debugging stories.
Fun read! Along the way I was trying to guess the cause and my best guess was TTL-related. However I don’t quite understand the actual cause! If the connection timeout is 3ms in practice, shouldn’t that be for a packet round-trip? So ~250 miles? And wouldn’t we expect at least a small delay on the remote SMTP server to process the packet?
tangentially related: 15 or maybe 20 years ago i worked at a repair shop and someone brought in a TV that they said switched to spanish every night at 5pm.
they were watching over the air channels and there was only a setting in the tv for menu language. sure enough though, at 5pm that night we watched as the tv started speaking in spanish. we tried a few more channels and found that all but one or two were in spanish.
as it turns out, some stations broadcast audio in multiple languages and some tvs allow you to change the preference. sadly for this person, the used tv they bought came from a spanish speaking country and didn't have anyway to change that preference.
My Bluetooth speaker switched to Chinese after a few years of use. I have no idea of how it did it and no idea of how to revert it. There is no reference to it in the manuals.
A few days before, I brought a robot vacuum home. It was made and purchased in China. When I started it for the first time, it bumped into my server and unplugged it.
Therefore a state-sponsored cyberattack is not out of the question.
I have a pair of Bluetooth headphones that indexes through 3 languages (Mandarin, Korean, English) on startup when you also hold down some other button besides power (play/pause IIRC). I don't think it's documented. Maybe your speaker is similar.
Every September (in the 90s) a new cohort of University students would start and gain access to the internet, such as it was. Hence the reference to timing
As the Wikipedia article states, it's more about the specific period of late 1993–early 1994 when some large online service providers in the US started providing access to the greater internet, the USENET in particular, and masses of new (and usually clueless wrt established norms and netiquette) internet users flooded newsgroups and forever changed and disrupted their character and culture.
Yeah, but that was given the name 'Eternal September' as a reference to the literal Septembers of the preceding years, when there would be a regular influx of newbies.
I'm pretty new to this community and this was my first exposure to this story. It's definitely going into my "anecdotes from the internet" mental repository.
why would anybody want to do that? I thought karma points above a certain low number don't offer benefits and the number is not visible to anybody but you.
EDIT: I just realized it's visible in profiles :facepalm:
The story would be better, if they had kept the real numbers, and not added fake numbers afterwards (maybe they didn't keep notes, and forgot the real numbers).
At the end of the story, that 3 millilightseconds is the one-way distance, and that can't be correct.
Yes, that part doesn't add up. The time from sending SYN to receiving SYN+ACK would be six milliseconds assuming lightspeed between the source and a destination 500 miles away.
That said: I know the ending, and by now the details about SunOS and sendmail aren't too interesting, but the "This is the chairman of statistics" line always gets me laughing out loud.
206 comments
[ 3.2 ms ] story [ 423 ms ] threadAt lunch today I was just talking about Sendmail, which I can assure you is a rather rare occurrence. I was talking about the first time I set up sendmail, back in '91 or '92. I was using the bat book and nearly tore my hair out over a week getting that first setup working. I eventually came to understand and appreciate the m4 config, but I ended up moving to qmail and postfix in the mid '90s and never looked back.
is this realistic, or a writers license?
https://www.ibiblio.org/harris/500milemail-faq.html
Realistic. And, believe it or not, I know of at least one organization that plans to convert an entire literal skyscraper of office space from routed networks to a single, flat switched network for all the employees of all the subcompanies. In 2023.
Obviously everyone with a bit of braincells left tells them to not do that because it's utterly dumb, but hey, strategic decision by the holding company to save on costs...
At least they're not using hubs. (For the younger generation: a hub is an Ethernet device that takes any packet it ingests in one port and sends it out to all other ports, with no consideration at all if the device that the packet is destined for actually is on that port - something a switch does, by maintaining a mapping of MAC addresses to ports. Extremely dumb devices, but used to be way faster and especially cheaper than switches in the 90's/early '00s)
P.S.: Yes, modern alternatives would be to to buy a switch and that can be configured to "mirror" packets onto a chosen port, or a smalls Ethernet network tap unit... But why buy more stuff if I don't really need to?
Layer 2 switches were a rare animal indeed during 1994-1997. There sure wouldn't be any VLANs.
"In a "switched" network, when Device A wants to send data to Device B, the switch directly connects these two devices so they can chat. Think of it like a train switcher that directly links Track A to Track B for a specific train, instead of sending it through a maze of tracks where other trains are moving.
In contrast, a "hub-based" network is like a party line in old telephone systems. When Device A talks, EVERY device hears it, but only Device B cares and listens. This is less efficient and can be slower because all devices get the data, which clogs up the network.
Another option is a "routed" network, where a router decides the best path for the data. This is like GPS choosing the best route based on current traffic conditions. It's more flexible but can introduce more delays because the data might go through multiple steps to reach its destination.
It's called "switched" because the switch acts like a railroad switch operator, making a direct track connection from one device to another for each piece of data. It "switches" the pathway specifically for that data to make the communication as direct as possible."
The case of the 500-mile email (2002) - https://news.ycombinator.com/item?id=29213064 - Nov 2021 (93 comments)
We can't send email more than 500 miles (2002) - https://news.ycombinator.com/item?id=23775404 - July 2020 (135 comments)
500 miles (2002) - https://news.ycombinator.com/item?id=18675375 - Dec 2018 (32 comments)
The case of the 500-mile email (2002) - https://news.ycombinator.com/item?id=14676835 - July 2017 (56 comments)
The 500-mile email (2002) - https://news.ycombinator.com/item?id=9338708 - April 2015 (139 comments)
The case of the 500-mile email - https://news.ycombinator.com/item?id=2701063 - June 2011 (18 comments)
The case of the 500-mile email - https://news.ycombinator.com/item?id=1293652 - April 2010 (24 comments)
The case of the 500-mile email - https://news.ycombinator.com/item?id=385068 - Dec 2008 (28 comments)
The case of the 500-mile email - https://news.ycombinator.com/item?id=123489 - Feb 2008 (7 comments)
That's on purpose—we want good articles to get multiple chances at attention, and we want the 'classics' to pop up periodically (just not too often), so newer cohorts of users get some exposure to them. So you did well :)
The purpose of linking to past threads is not to imply anything bad about the repost (if it were a bad repost, we would handle it differently) - rather, it's to point curious readers to other discussions on the same topic that may interest them.
ty
Enjoy!
<https://news.ycombinator.com/item?id=36988917>
<https://news.ycombinator.com/item?id=36995046>
And Trey Harris's "500 mile email" story is what clued me on to GNU units and its capabilities.
Reminder: if you're on MacOS, or one of the BSDs, your default units is from BSD, not the GNU version, and is far less capable. GNU units can be installed on MacOS through Homebrew. The package is "gnu-units", the command is "gunits" once installed.
Edit: Corrected Homebrew package name.
brew install gnu-units
(I run the command far more often than I install the package.)
(I've not touched MS Windows in a couple of decades now.)
EDIT: #19 here https://www.ibiblio.org/harris/500milemail-faq.html
It's also entirely plausible that a local definitions file was used to provide the unit that didn't come in the default install. Hence the anachronism is merely "potential" instead of "definite"
edit: here it is https://bugs.launchpad.net/ubuntu/+source/cupsys/+bug/255161...
https://news.ycombinator.com/item?id=23775404
>Well, the consultant came in and patched our server and rebooted it. But I called him, and he said he didn't touch the mail system.
But the comment
>Since my preference to wipe and reload was unacceptable - too much downtime and too many billable hours - the obvious thing to do was update sendmail
Must be the part where
>The story is slightly altered in order to protect the guilty
To a fault, probably.
And as the former comp.mail.sendmail FAQ maintainer, I can confirm all sorts of weirdness can and will happen if you mix versions of the binary versus the config file.
We put it in /highlights which may be of interest to people: https://news.ycombinator.com/highlights
+1, After two years from joining HN I’m still learning about it. This is the first time I heard about highlights section! I couldn’t find it in lists nor on any other part of the site yet still interesting to read some comments there that do not show up in best comments section. How exactly this works?
It's always irritated me but I haven't changed it, out of deference to history and lack of a pithy alternate name. Anyone?
Maybe /up and /upcomments?
I don't have them, but I have seen comments in the negative, so there must be some truth to it
I knew this existed, and I was looking for it a few weeks ago; it's an interesting page to browse through every once in a while. But I just couldn't remember the name until now.
Having a "highlight" people can't find doesn't seem much of a "highlight" to me?
Edit: it's there now.
Edit 2: I feel like adding an "I feel lucky" link on there that would give you a random sample from the list. It's in reverse chronological order and it would take quite a while to scroll back through all of the comments (there are over 400 at present).
How about a link with reverse chronology of random highlights, one per year of HN history?
This could inspire nominations for the highlights back catalog.
No, I just added it to a list. But we can inch in that direction!
But reading through the list, you made dang good choices in highlighted comments.
It would be nice to have software to make this a community-driven process. Maybe we'll build some eventually.
The actual underlying transmission protocol of the relativistic universe shining through when trying to send an email.
https://news.ycombinator.com/item?id=29213472
0: https://grugbrain.dev/
To get a good server-client-server roundtrip with only HTTP/1.1, I'd personally try using a temporary redirect, maybe a 307.
A very common case is when the challenge has a short timeout but requires a lot of interaction, e.g. you only get ten seconds but you have to perform 10000 queries for a heap spray or something.
The most insidious case I remember was a read() call that didn’t check the result, causing it to return short if the fragments of the input didn’t arrive fast enough.
The expectation wasn't to buy time in an adjacent cloud, but to use out of order ip fragmentation or tcp segments, having the servers network stack reassemble the packets such that the read was coherent in one go.
My goal was to teach competitors to model real world challenges of exploitation.
The challenge you're describing sounds cool - I wish I'd played it! By the time I started playing DEF CON CTF though, it was with LegitBS as organizers.
Trickiest part of doing that today is so many fewer hosts are reachable via icmp, so you'd probably be better off serving up an initial response with some JS to measure more accurately.
(Another silly little thing we added was a link back to a users own ISP from the top ten or so of our competitors based on net block - got us a worried phone call from one of them who thought we'd been hacked and wanted to make sure we didn't think he was responsible)
This blog has also been discussed in a few other threads:
https://news.ycombinator.com/item?id=23908171
https://news.ycombinator.com/item?id=29215383
https://news.ycombinator.com/item?id=35708339
If anyone has any stories that I should add, please send them my way!
Also, I have been slacking a little with the newsletter because of life events. Since there have been several sign ups since this post, I'll be more active in getting the weekly newsletter out.
Thanks for looking!
Anyway, I understand that's probably not the kind of site you had in mind. Thanks for the cool site!
https://old.reddit.com/r/sysadmin/comments/9si6r9/postmortem...
Worthy, but no where near as circulated!
That sentence is a rather decent: double entendre.
they were watching over the air channels and there was only a setting in the tv for menu language. sure enough though, at 5pm that night we watched as the tv started speaking in spanish. we tried a few more channels and found that all but one or two were in spanish.
as it turns out, some stations broadcast audio in multiple languages and some tvs allow you to change the preference. sadly for this person, the used tv they bought came from a spanish speaking country and didn't have anyway to change that preference.
A few days before, I brought a robot vacuum home. It was made and purchased in China. When I started it for the first time, it bumped into my server and unplugged it.
Therefore a state-sponsored cyberattack is not out of the question.
What do you mean by that?
Every September (in the 90s) a new cohort of University students would start and gain access to the internet, such as it was. Hence the reference to timing
I await next week’s deployment.
Feelgood-karma-whoring-post-as-a-service
EDIT: I just realized it's visible in profiles :facepalm:
At the end of the story, that 3 millilightseconds is the one-way distance, and that can't be correct.
That said: I know the ending, and by now the details about SunOS and sendmail aren't too interesting, but the "This is the chairman of statistics" line always gets me laughing out loud.