Thank you for linking this - I need to save this locally because I reference this all the time. This is one of my favorite internet stories - it's just a great arc!
I think this is enough of a classic to be widely known even among younger people. I'm 23 (doing math msc) and I think all the CS people that I know would instantly recognize the 500 miles title.
Though I do somewhat envy the possibility of having read the article close to publication and feel in some sense part of the history when it crops up again like this.
Is there a library to re-introduce relevant delays into a CDN so that all users experience their own geographically-appropriate response times?
I mean, I want reliability. But I also want Europeans to be able to taste that authentic latency they'd expect from a fledgling startup running out of a garage in San Jose.
> There’s a lot to the story that’s obviously made up...
Obviously? I think I've had this phone call myself a few times, although in my experience it was never from a statistician and they didn't give me as much data, but I'm pretty sure the story is mostly accurate.
> I think this is nonsense... why would an invalid or incomplete sendmail configuration default to three milliseconds?
This is a wonderful question, and perhaps much more interesting than anything else in the page, but first, let's reproduce the timing;
My desktop, a 2017 Xeon E7-8880 (144 cores of 2.3ghz; 1tb ram) with a load of 2.26 at this moment:
$ time sleep 0.001
real 0m0.004s
user 0m0.001s
sys 0m0.003s
On my i9-10900k (3.7ghz) current load of 3,31:
$ time sleep 0.001
real 0m0,002s
user 0m0,000s
sys 0m0,001s
(In case you think I'm measuring exec; time /bin/echo returns 0's on both machine)
Now as to why this is? Well in order to understand that, you need to understand how connect() actually works, and how to create a timeout for connect(). Those skilled in the art know you've got a number of choices on how to do it, but they all involve multiple steps because connect() does not take a timeout as an argument. Here's one way (not too different than what sendmail does/did):
If you read this carefully, you only need to ask yourself how much time can pass between the top of connect() and the bottom of select(), and if you think it is zero like tedu does, you might probably have the same surprise: Computers are not abstract machines, but made out of matter and powered by energy and thus subject to the laws of physics, and so everything takes time.
For others, the surprise might be that it's still 3msec over twenty years later, and I think that is a much more interesting subject to explore than whether the speed of light exists.
> Obviously? I think I've had this phone call myself a few times, although in my experience it was never from a statistician and they didn't give me as much data, but I'm pretty sure the story is mostly accurate.
Yeah, the original retelling even states up-front:
> The story is slightly altered in order to protect
the guilty, elide over irrelevant and boring details, and generally make
the whole thing more entertaining.
It's pretty common to alter minor details of stories in order to make them easier to follow, not to mention that the entire account is also written several years after it happened, when details are presumably less likely to be completely accurate. Obviously the dialogue is reconstructive for narrative ease; no reader would look at that and assume it's intended to be a verbatim transcript.
Unless the author here can cite specific things that make it truly impossible for anything of that shape to have occurred, I'm not seeing anything that justifies the conclusion "there's a lot to the story that's obviously made up".
Well, first light does 500 miles in 3ms, but the connect signal needs to come back, right? So it should be 250 miles, at most? But this is just a detail.
More importantly, because it seems to assume that all other operations besides the signal actually reaching the destination are instantaneous. As you point out yourself, computers are not abstract machines, so the actual response time between the signal being received by the destination (even assuming it's just one straight line with zero electronics in between) and the destination replying is not zero. I imagine there can be a large variation between physical installations and different types of hardware, so much as to make it very hard to detect a clear 500 miles boundary.
> Well, first light does 500 miles in 3ms, but the connect signal needs to come back, right? So it should be 250 miles, at most?
I don't think this is terribly important (NB my examples have nothing to do with networking), but in the author's case it was probably the other way; maybe 10msec and a bit more: Copper gets up to ~0.6c but I think this detail makes the story less amusing, and is a distraction from wondering why does select() take so long...
> I imagine there can be a large variation between physical installations and different types of hardware
There is probably not as much as you think, and Sendmail retries, so with whatever variation that exists, only the bounds really matter.
> Or am I missing something?
Modern unixish systems have the same log-scale delay coming out of select() so this has almost nothing to do with the hardware being slower or variability in the network.
> The poll timeout is 3ms, as specified by the lore. I think this is nonsense, why would an invalid or incomplete sendmail configuration default to three milliseconds?
The answer is that per the original story, it was not defaulting to three milliseconds. It was defaulting to 0, and the 3ms was just how long it took the system to check for a response with a 0 timeout:
> Some experimentation established that on this particular machine with its typical load, a zero timeout would abort a connect call in slightly over three milliseconds.
This is a very different scenario, as it's not clear there should be a poll() there at all (or more likely select() given the age of the story) to match the original, but if there was, the select would have a timeout of 0, not 3ms, and would just happen to be unable to distinguish between 0 and up to 3ms.
I really wouldn't have predicted the extreme amount of centralization, and arguably unnecessary centralization, that we have today for things like university email and web servers. Even 20 years ago when I was in college, the servers I interacted with including email, were all in our school's /16. They did have software packages for LMS and stuff, but those were mostly deployed on-prem.
Today the websites are hosted on third party cloud servers (my school's main website is some company that hosts your Wordpress or Drupal site so you don't have to) and the email by Microsoft or Google. Same for every school it seems. I guess the IT department that used to run all the infra is now probably just a few people in charge of ordering new laptops for faculty/staff when they break, and replacing Wi-Fi access points every 5 years.
We have a program which the company who developed lost the ability to rebuild the app for some reason.
It has a 500ms timeout to load some settings from a server in the UK via TLS. If it goes more than that 500ms (or something, it's unclear the exact timeout cause) the app just vapourises.
This is fine in the UK, TLS needs about* 3 times RTT to complete though, so an RTT above about 160ms and it's screwed.
Almost all our users are in the UK, europe, mid-east or east coast USA, and in that 160ms RTT range.
We ran into issues when a dozen people tried to use it in Australia, so the principal still happens with some badly written code.
> there was a university president who couldn’t send an email more than 500 miles, and the wise sysadmin said that’s not possible, so the president said come to my office, and lo and behold, the emails stopped before going 500 miles.
NO. NO NO NO.
How can you get SO MANY facts wrong when the freaking story is googlable?
This annoys me because I know the original author and I remember when this happened (he told the story a few times).
Let's recap:
> there was a university president
NO! It was the chairman of the statistics department.
> who couldn’t send an email more than 500 miles,
True. Being in the statistics department he had the tools to make actual maps.
> and the wise sysadmin said that’s not possible, so the president said come to my office
Kind of true. There was an office involved.
> and lo and behold, the emails stopped before going 500 miles.
True.
> There’s a lot to the story that’s obviously made up,
NO! Zero of this story was made up.
ALL the people that were involved in the story are still alive. You can literally get them on the phone and talk to them. We're not debating whether or not Han Solo ever used a light saber. THIS SHIT REALLY HAPPENED.
23 comments
[ 19.1 ms ] story [ 229 ms ] thread(discussed previously on HN 5 years ago – https://news.ycombinator.com/item?id=23775404 – and 10 years ago – https://news.ycombinator.com/item?id=9338708)
Though I do somewhat envy the possibility of having read the article close to publication and feel in some sense part of the history when it crops up again like this.
I mean, I want reliability. But I also want Europeans to be able to taste that authentic latency they'd expect from a fledgling startup running out of a garage in San Jose.
Apparently not.
"10 years ago we couldn't send an email 500 miles, but these days we can't send it 500 miles because it just routes internally."
Too bad, I think that would have been more interesting to read.
Obviously? I think I've had this phone call myself a few times, although in my experience it was never from a statistician and they didn't give me as much data, but I'm pretty sure the story is mostly accurate.
> I think this is nonsense... why would an invalid or incomplete sendmail configuration default to three milliseconds?
This is a wonderful question, and perhaps much more interesting than anything else in the page, but first, let's reproduce the timing;
My desktop, a 2017 Xeon E7-8880 (144 cores of 2.3ghz; 1tb ram) with a load of 2.26 at this moment:
On my i9-10900k (3.7ghz) current load of 3,31: (In case you think I'm measuring exec; time /bin/echo returns 0's on both machine)Now as to why this is? Well in order to understand that, you need to understand how connect() actually works, and how to create a timeout for connect(). Those skilled in the art know you've got a number of choices on how to do it, but they all involve multiple steps because connect() does not take a timeout as an argument. Here's one way (not too different than what sendmail does/did):
If you read this carefully, you only need to ask yourself how much time can pass between the top of connect() and the bottom of select(), and if you think it is zero like tedu does, you might probably have the same surprise: Computers are not abstract machines, but made out of matter and powered by energy and thus subject to the laws of physics, and so everything takes time.For others, the surprise might be that it's still 3msec over twenty years later, and I think that is a much more interesting subject to explore than whether the speed of light exists.
I can't help but feel that's somewhat excessive for a desktop. Have you considered closing a few browser tabs?
Yeah, the original retelling even states up-front:
> The story is slightly altered in order to protect the guilty, elide over irrelevant and boring details, and generally make the whole thing more entertaining.
It's pretty common to alter minor details of stories in order to make them easier to follow, not to mention that the entire account is also written several years after it happened, when details are presumably less likely to be completely accurate. Obviously the dialogue is reconstructive for narrative ease; no reader would look at that and assume it's intended to be a verbatim transcript.
Unless the author here can cite specific things that make it truly impossible for anything of that shape to have occurred, I'm not seeing anything that justifies the conclusion "there's a lot to the story that's obviously made up".
Well, first light does 500 miles in 3ms, but the connect signal needs to come back, right? So it should be 250 miles, at most? But this is just a detail.
More importantly, because it seems to assume that all other operations besides the signal actually reaching the destination are instantaneous. As you point out yourself, computers are not abstract machines, so the actual response time between the signal being received by the destination (even assuming it's just one straight line with zero electronics in between) and the destination replying is not zero. I imagine there can be a large variation between physical installations and different types of hardware, so much as to make it very hard to detect a clear 500 miles boundary.
Or am I missing something?
I don't think this is terribly important (NB my examples have nothing to do with networking), but in the author's case it was probably the other way; maybe 10msec and a bit more: Copper gets up to ~0.6c but I think this detail makes the story less amusing, and is a distraction from wondering why does select() take so long...
> I imagine there can be a large variation between physical installations and different types of hardware
There is probably not as much as you think, and Sendmail retries, so with whatever variation that exists, only the bounds really matter.
> Or am I missing something?
Modern unixish systems have the same log-scale delay coming out of select() so this has almost nothing to do with the hardware being slower or variability in the network.
The answer is that per the original story, it was not defaulting to three milliseconds. It was defaulting to 0, and the 3ms was just how long it took the system to check for a response with a 0 timeout:
> Some experimentation established that on this particular machine with its typical load, a zero timeout would abort a connect call in slightly over three milliseconds.
This is a very different scenario, as it's not clear there should be a poll() there at all (or more likely select() given the age of the story) to match the original, but if there was, the select would have a timeout of 0, not 3ms, and would just happen to be unable to distinguish between 0 and up to 3ms.
Today the websites are hosted on third party cloud servers (my school's main website is some company that hosts your Wordpress or Drupal site so you don't have to) and the email by Microsoft or Google. Same for every school it seems. I guess the IT department that used to run all the infra is now probably just a few people in charge of ordering new laptops for faculty/staff when they break, and replacing Wi-Fi access points every 5 years.
It has a 500ms timeout to load some settings from a server in the UK via TLS. If it goes more than that 500ms (or something, it's unclear the exact timeout cause) the app just vapourises.
This is fine in the UK, TLS needs about* 3 times RTT to complete though, so an RTT above about 160ms and it's screwed.
Almost all our users are in the UK, europe, mid-east or east coast USA, and in that 160ms RTT range.
We ran into issues when a dozen people tried to use it in Australia, so the principal still happens with some badly written code.
NO. NO NO NO.
How can you get SO MANY facts wrong when the freaking story is googlable?
Here's the original email: https://web.mit.edu/jemorris/humor/500-miles
Here's the FAQ that covers the ambiguous parts: https://www.ibiblio.org/harris/500milemail-faq.html
This annoys me because I know the original author and I remember when this happened (he told the story a few times).
Let's recap:
> there was a university president
NO! It was the chairman of the statistics department.
> who couldn’t send an email more than 500 miles,
True. Being in the statistics department he had the tools to make actual maps.
> and the wise sysadmin said that’s not possible, so the president said come to my office
Kind of true. There was an office involved.
> and lo and behold, the emails stopped before going 500 miles.
True.
> There’s a lot to the story that’s obviously made up,
NO! Zero of this story was made up.
ALL the people that were involved in the story are still alive. You can literally get them on the phone and talk to them. We're not debating whether or not Han Solo ever used a light saber. THIS SHIT REALLY HAPPENED.
Sheesh.