> "But Juho, what does any of this have to do with S3?", you ask. Well, S3 is one of those rare services that disable timestamps.
Interesting. Every once in a while I have issues on my cable provider (UPC in Austria) with S3 and S3 only. Next time that happens I gotta have a closer look at it.
Actually I received a new one yesterday. I have to check again whether the issue persists. The new one is called "Connect Box" and is jointly manufactured by Compal und Arris.
Designed is very different from available in any form. If I couldn't buy a router that supports it with any amount of money, NAT was my only answer. This was the existing state of the world for a long time.
One can't control those that must deploy IPv6 first. That's the problem NAT solves. Developing NAT was likely not easy; consider how many implementations f it up anyway.
> I disagree. What problem does NAT solve, other than allowing people to cut corners and not deploy IPv6?
It by default ensures that machines inside a network are not reachable from the outside world. It gives you one address to the outside world which made IP blocks for abuse work quite well and enabled a lot of other users.
Sure, it's a bit of a pain for UDP communication but given the common NAT behaviours not completely terrible. If it was defined a bit clearer NAT would not have been that controversial.
I understand where you're coming from with that – but I can imagine a different Internet, where devices where by default globally routable but firewalled. That could even have affected the current trend towards 'stuff it all in my proprietary cloud system'.
It breaks the internet because it breaks IP - the internet protocol. Segregating devices onto separate networks that are not routable to each other is something that reduces the overall functionality of the internet, not enhances it.
It doesn't help that it's a shitty hack which causes endless lost time either spent wondering why something is broken, or through designing entire protocols just to get around it.
I think you've maybe been unfairly downvoted there!
Almost all of the issues I experience with Internet connectivity day-to-day are down to bad consumer networking equipment. APs start failing, or routers stop routing, seemingly at random.
For example: my router/AP/modem combo-box — which is the standard one that Comcast has been giving to all subscribers — when reset or power cycled, will forget the WiFi key. It just gets reset to the factory default. Every, single, time. And it's just they key: the SSID doesn't get reset. It's not the first modem I've had, either; its predecessor was the same make, had the exact same behavior.
I suspect it's because I'm using an actual key, instead of a passphrase, and nobody else does that. (Or, at least, not enough for Comcast to notice that the problem is them.)
The old one got returned b/c it was routinely dropping IPv6 packets. It took a while — multiple vists from multiple technicians — to convince Comcast that the symptom of "100% packet loss over IPv6, 0% packet loss over IPv4" was not a problem with the coax cable.
The predecessor of that modem got replaced when it had extreme difficulty forming IPv4 connections: almost all SYNs never got a SYN ACK, but in the rare case that they did, that connection proceeded with 0% packet loss. I highly suspect there was something going horridly wrong with its NAT processing, but that's a guess.
The worst is that the Linux kernel (and any BSD) would be able to make a perfect router, but manufacturers insist on using their own shitty implementation. If they would've just used used the default Linux/BSD they would've been fine.
We take network transmissions for granted sometimes, TCP/IP has been around so long we assume nothing strange can still happen. But implementations still matter, it only take one strange piece of software or hardware and you wind up with craziness like this.
Yep. Honestly I probably would have assumed it's not a protocol issue if both HTTP and HTTPS downloads from S3 behaved the same way. Both of those protocols are on TCP, though, so that would have thrown me off!
It's a bit unfortunate that there's no history of the title edits on HN, as it makes it hard to understand the context of any complaints about titles.
The post was originally submitted with the same title as the original post, "The mystery of the hanging S3 downloads". That was apparently judged to be clickbait, so the title was changed to something along the lines of "Hanging S3 downloads due to NAT router", and the story was also very heavily penalized at the same time. This title was indeed giving away most of the ending. But luckily it was a bit later edited again, to the current form of "Hanging S3 downloads".
Your cloud service does everything except for file backups, load balancing and can't handle huge file copies. And for reliability and responsiveness, requires a hybrid private-public cloud. Which begs the question as to what's the point of moving to the cloud in the first place.
I had the same reaction. If I need to support users with shitty routers should I stay away from amazon? My users will just blame me not their router, or switch to my competition who is not using Amazon since it works. My users aren't going to read a long explanation of TCP/IP. Some users have no choice like a school or work firewall/router. If my website doesn't work and the competitions does what do you think happens next if I don't stop using s3? The real question is how many users have a "shitty router".
Well S3 is popular enough (you don't realise it but a lot of sites use it under the hood for static assets, file uploads, etc) and this is the first time I've heard of this issue, so the number of routers so shitty that they break S3 (and only S3) would be pretty insignificant. There will be more routers that break the general web (no matter which site) than those only breaking S3.
38 comments
[ 4.7 ms ] story [ 78.0 ms ] threadInteresting. Every once in a while I have issues on my cable provider (UPC in Austria) with S3 and S3 only. Next time that happens I gotta have a closer look at it.
It by default ensures that machines inside a network are not reachable from the outside world. It gives you one address to the outside world which made IP blocks for abuse work quite well and enabled a lot of other users.
Sure, it's a bit of a pain for UDP communication but given the common NAT behaviours not completely terrible. If it was defined a bit clearer NAT would not have been that controversial.
For me that's a core feature of NAT, not something that breaks the internet.
It doesn't help that it's a shitty hack which causes endless lost time either spent wondering why something is broken, or through designing entire protocols just to get around it.
Almost all of the issues I experience with Internet connectivity day-to-day are down to bad consumer networking equipment. APs start failing, or routers stop routing, seemingly at random.
I suspect it's because I'm using an actual key, instead of a passphrase, and nobody else does that. (Or, at least, not enough for Comcast to notice that the problem is them.)
The old one got returned b/c it was routinely dropping IPv6 packets. It took a while — multiple vists from multiple technicians — to convince Comcast that the symptom of "100% packet loss over IPv6, 0% packet loss over IPv4" was not a problem with the coax cable.
The predecessor of that modem got replaced when it had extreme difficulty forming IPv4 connections: almost all SYNs never got a SYN ACK, but in the rare case that they did, that connection proceeded with 0% packet loss. I highly suspect there was something going horridly wrong with its NAT processing, but that's a guess.
The post was originally submitted with the same title as the original post, "The mystery of the hanging S3 downloads". That was apparently judged to be clickbait, so the title was changed to something along the lines of "Hanging S3 downloads due to NAT router", and the story was also very heavily penalized at the same time. This title was indeed giving away most of the ending. But luckily it was a bit later edited again, to the current form of "Hanging S3 downloads".
I think they had much the same solution, mix a random number into the packet. (They use UDP, so they didn't have the benefit of TCP timestamps.)
[1]: https://www.factorio.com/blog/post/fff-176
[2]: https://www.reddit.com/r/factorio/comments/5rvkyp/friday_fac...