Also, I've never heard of a system that sometimes uses PMTUD and sometimes doesn't. I can imagine that those SSL offload boxes may be doing weird stuff; maybe that's where the author saw this problem.
Thanks wmf I should have included the reference and have added it. There's a lot more that I don't cover e.g. PMTUD black hole routers, etc. But the article is targeted at an intermediate level audience, so leaving those out for now.
Hi axod, on request from a commenter on my blog I've described how I captured packets and analyzed. I'm interested that you saw the DF flag for non-HTTP traffic. Can you tell me which OS and client or browser you used? I used Ubuntu 9.04 and curl, but have seen the same df/non-df behavior with a variety of other clients on the same OS.
I retested on OSX and Linux and same result. (I'm running Snow Leopard)
I tested secure connection to https://help.ubuntu.com/ and the data packets have the DF flag on both OS's.
I tested insecure to www.ebay.com and the data packets have no DF flag.
Make sure you ignore the first few packets that are setting up the connection because those will probably have the DF flag set both ways. Only check the packets further down about halfway through the connection - you can tell the data packets by their size.
"""Please don't sign comments, especially with your url. They're already signed with your username. If other users want to learn more about you, they can click on it to see your profile."""
Because HTTPS clients and servers set the DF or Do Not Fragment IP flag on packets and regular HTTP clients and servers do not.
There is nothing anywhere in any standard that requires DF to be set for HTTPS traffic. There are certain links on the Internet that will set DF on your traffic in transit because it is more expensive to forward and even more expensive to do deep inspection on fragmented IP packets, but there is no reason this would be done for HTTPS and not for HTTP.
EDIT: For completeness: links to RFC 2246 for TLS 1.0 and one of the draft specs for SSL 3.0 (it was never an official standard). There is nothing about setting the DF bit in the IP header in either document.
This premise isn't taken from an RFC. It's based on my own real-world observations of what is crossing the wire. Grab wireshark, fire up tcpdump and form your own opinion - and please do share what you find with us.
On Linux (at least recent kernels), there are a number of variables that can determine whether or not DF is set on outgoing packets. I ran a few tests with (Ubuntu Hardy) firefox and it seems largely to vary based on the value of /proc/sys/net/ipv4/ip_no_pmtu_disc. If 1, no DF. If 0, DF set. But it has this behavior regardless of whether I'm using HTTPS or HTTP.
I understand your objection: standards do not often reflect the real world. But the title of your post is misleading. Saying that routers treat https differently than http is just incorrect. When I read it, I thought you were trying to imply that routers were doing some sort of deep inspection, or at least looking at the TCP port numbers.
What is true is that you can't expect PMTU discovery to work reliably over the Internet (another case of standards not reflecting the real world). I work at an ISP and it's a constant source of frustration. In general though, it's something that needs to be fixed on the client side (or at least on the client's ISP side), rather than on the server side.
So you're saying that with ip_no_pmtu_disc set to off (the default value), your HTTPS server does not set the DF flag on packets?
What I've seen is that all HTTPS servers (that I've observed) set DF on their data packets. No HTTP servers (that I've observed) set DF for their data packets.
What I would like to know is why SSL servers set DF on their data packets. As you've observed there's nothing in the FAQ and I've googled the socket options and couldn't find any recommendations. I'm curious if an underlying API like openssl is simply so widely used that it's a de facto standard.
Did you try tweaking ip_no_pmtu_disc and run your tests? I played a bit with the openssl test client/server and it seems to respect ip_no_pmtu_disc. Test goes something like this:
1) In one window run
openssl s_server
2) In another window run
openssl s_client -connect localhost:4433
3) In another window run
tcpdump -nvi lo port 4433
Try twiddling the ip_no_pmtu_disc sysctl and see what happens.
Also, when you say "HTTPS servers set DF," are you observing from the server or from the client? Over the Internet, it's likely that you'd see different values for DF on opposite sides of the connection. Any router can set or clear the DF bit on your packets for any reason.
17 comments
[ 2.9 ms ] story [ 31.6 ms ] threadAlso, I've never heard of a system that sometimes uses PMTUD and sometimes doesn't. I can imagine that those SSL offload boxes may be doing weird stuff; maybe that's where the author saw this problem.
Would have been nice if the OP had included some example logs showing the issue.
Mark.
Using firefox, on OSX, tcpdump.
HTTP request - flag set HTTP response - flag not set
HTTPS request - flag set HTTPS response - flag not set
I retested on OSX and Linux and same result. (I'm running Snow Leopard)
I tested secure connection to https://help.ubuntu.com/ and the data packets have the DF flag on both OS's.
I tested insecure to www.ebay.com and the data packets have no DF flag.
Make sure you ignore the first few packets that are setting up the connection because those will probably have the DF flag set both ways. Only check the packets further down about halfway through the connection - you can tell the data packets by their size.
Mark.
"""Please don't sign comments, especially with your url. They're already signed with your username. If other users want to learn more about you, they can click on it to see your profile."""
Because HTTPS clients and servers set the DF or Do Not Fragment IP flag on packets and regular HTTP clients and servers do not.
There is nothing anywhere in any standard that requires DF to be set for HTTPS traffic. There are certain links on the Internet that will set DF on your traffic in transit because it is more expensive to forward and even more expensive to do deep inspection on fragmented IP packets, but there is no reason this would be done for HTTPS and not for HTTP.
EDIT: For completeness: links to RFC 2246 for TLS 1.0 and one of the draft specs for SSL 3.0 (it was never an official standard). There is nothing about setting the DF bit in the IP header in either document.
TLS 1.0: http://www.ietf.org/rfc/rfc2246.txt
SSL 3.0: http://tools.ietf.org/html/draft-ietf-tls-ssl-version3-00
http://manpages.ubuntu.com/manpages/jaunty/man7/ip.7.html See: IP_MTU_DISCOVER
This premise isn't taken from an RFC. It's based on my own real-world observations of what is crossing the wire. Grab wireshark, fire up tcpdump and form your own opinion - and please do share what you find with us.
Mark.
I understand your objection: standards do not often reflect the real world. But the title of your post is misleading. Saying that routers treat https differently than http is just incorrect. When I read it, I thought you were trying to imply that routers were doing some sort of deep inspection, or at least looking at the TCP port numbers.
What is true is that you can't expect PMTU discovery to work reliably over the Internet (another case of standards not reflecting the real world). I work at an ISP and it's a constant source of frustration. In general though, it's something that needs to be fixed on the client side (or at least on the client's ISP side), rather than on the server side.
What I've seen is that all HTTPS servers (that I've observed) set DF on their data packets. No HTTP servers (that I've observed) set DF for their data packets.
What I would like to know is why SSL servers set DF on their data packets. As you've observed there's nothing in the FAQ and I've googled the socket options and couldn't find any recommendations. I'm curious if an underlying API like openssl is simply so widely used that it's a de facto standard.
1) In one window run
2) In another window run 3) In another window run Try twiddling the ip_no_pmtu_disc sysctl and see what happens.