if it's unfixable, why is 40% of our in-office internet traffic running over IPv6? Why is 100% of our LAN traffic over IPv6? Why is 30% of the traffic to our sites running over IPv6?
Works great for me on my laptop. I don't doubt for a second that there are problems (my phone doesn't get ipv6 at all and one of the ipv4 tests failed... but mobile networks are notorious when it comes to filtering traffic), but "unfixable"?
The comment I was responding to asserted that IPv6 as a whole is unfixable which is not true.
With regards to fragmentation, I would say "fragmentation in general is unfixable", no matter the protocol. Thankfully with IPv6, the guaranteed minimal packet size is relatively reasonable (1280 bytes), so maybe it's now time to finally go there.
v6 already changed the rules with fragmentation a bit in that routers won't have to fragment any more but are allowed to just drop packets smaller than the MTU and sending back an ICMPv6 packet announcing that they have done so.
Unfortunately, ICMP is still dropped at many firewalls even though I would argue that it's absolutely essential for IPv6 functionality (when it was more optional in v4).
As it stands now, hosts are still allowed to fragment packets themselves (which is what's happening in this article), but the solution is probably to not do that either in the future.
DNSSec over UDP is a problem in that regards, but people aren't convinced that DNSSec is the way to go anyways (just ask tptacek here on this site).
(edit: with regards to the fragment checking site you pointed me to: In our case, all is well because I do recognise the necessity of ICMPv6 and thus have configured the firewalls accordingly)
the firewall is probably blocking ICMP. For a long time, it was considered best-practice to blanket-block ICMP and for a long time you could mostly get away with it.
But over time some really useful stuff has been added to ICMPv4 (early congestion notification for example) and IPv6 mostly won't work at all without ICMP.
I would argue that it's time for firewall administrators to reconsider this decision. The time where ICMP packets could do harm are long past (remember the ping-of-death on Windows 98?) and I would argue that the time when a network could work at peak efficiency (or at all) without ICMP are also mostly past.
> For a long time, it was considered best-practice to blanket-block ICMP
I agree with your point, but think this needs some clarification about who considered it a best practice - among enterprise firewall admins and financial-industry compliance audits, perhaps.
Large scale network operators have seen the problems associated with entirely blocking ICMP for at least a decade. (v4 has blackhole problems too)
A lot of the hatred for ICMP came from simple things like easy endpoint discoverability. nmap blew that argument out of the water pretty quick.
The comment I was responding to asserted that IPv6 in general is unfixable whereas the article was only talking about IPv6 fragmentation where I'd say it's totally correct.
But in general, fragmentation is a bad idea and we should not have gone there in the first place.
DNSSec should only ever have been specified over TCP (or not at all if you ask some people like tptacek here on this site)
> IPv6 Extension Headers require any transport protocol-sensitive functions in network switches to unravel the packet header’s extension header chain. This takes a variable number of cycles for the device and furthermore requires that the switch should recognise all the extension headers encountered on the header chain. This is anathema to a switch, in so far as it entails a variable amount of time to process.
If it is so much anathema to their nature then maybe they shouldn't insist on violating network layers?
I agree that switches shouldn't care about TCP or UDP ports. That is a job for a firewall.
However, that does not mean it is a good idea to have variable length headers in IPv6. In fact it seems like a singularly bad idea to me.
Regarding firewalls, I imagine it still sucks a firewall cannot judge each IP-packet separately but needs to deal with possible fragmented packets. Then again, that is the cost of being a firewall. You need to function at the application level and yet decide on the IP-packet level.
Switches use L4 ports to implement flow-switching. When multiple paths for a packet are available, it is convenient (it leads to less re-ordering) if the related packets from the same flow all choose the same path.
It's not just convenience for anycast addresses, with multiple pathes and anycast if your round robin packets across different routes, you may end up receiving them in two different locations (ok for UDP, but TCP broken completely).
Anycast is widely deployed now and imho ECMP support is now considered "standard".
I'm not sure what are other alternative solution to ECMP that don't require looking into TCP headers.
I don't see an issue with extension headers in IPv6. The RFCs have rules on who should look at what header and what to do when an unrecognized one is encountered.
Most firewalls I've dealt with have some option to defragment the inbound packet at the firewall so that the rest of the rules can be processed properly and to protect nodes from malicious fragments.
Having a variable header complicates Cut-Through switching[0], which forwards the entire frame just after the switch gets the destination field of the header.
This sounds like it would force a return to Store-And-Forward[1], where the switch waits for the entire frame to load into memory, before forwarding to its next hop. Waiting these few milliseconds doesn't sound bad, until you consider that's added to each packet, reducing (e.g.) application performance.
The real problem is that you can no longer look at a fixed offset for your parameters, which complicates filtering silicon.
In other words, if you want to have a firewall rule of "drop tcp 135", in ipv4 you can just look at bytes 0x24-25 (for Ethernet). IPv6's extension-header mechanism means that the header has a non-fixed length, so you need to do work for the same effect. Given that firewalls are critical in a world without NAT, this can be a scary prospect.
Part of the blame falls on vendors who go "well, IPv6 is fringe so we'll just do all firewalling in software." The protocol is 20 years old, it's time to design the silicon.
Most enterprise firewalls these days do some level of layer 7 packet inspection which requires going beyond looking at fixed offsets. The silicon to support this already exists. Even the $50 Ubiquiti ER-X router I use at home can do deep packet inspection assisted by silicon.
But IPv4 headers could have a variable length too, it's just that we don't encounter that much in the wild. And what about the AH header, GRE and IPIP tunnel headers, 6rd/6in4 tunnel headers, etc. Filtering at fixed offsets sounds very brittle to me.
Isn't the real problem here that network providers and administrators appropriate the right to filter on OSI layers they shouldn't be touching? This problem sounds like the exact reason why Google insisted on having its QUIC headers ciphered: so network equipment can't pull of this kind of misbehaviour.
At least with IPv4 you can easily compute the start of layer 4, it's just ipv4[0]&0xf * 4 - trivially implemented with little real-estate in hardware.
IPv6 requires you to loop through all extension headers to reach layer 4, and you need to know about a handful of those extension headers as not all follow the same format.
I was confused at first, because switches were mentioned in the article. But the Fates have allowed my mistake to be correct this one time, and provided a trove of education about the state of the art of network switching.
You seem to be confusing IP headers with Ethernet headers. Cut-through switching only looks at the Ethernet header, which is not variable, and thus will still be perfectly viable.
The way cut-through is really implemented is that the first N bytes of the packet (usually 128 or 256) are treated as the "header" and are passed through the header pipeline regardless of how big the actual header is. So variable headers like VLAN tags, MPLS stacks, or IP extension headers shouldn't prevent cut-through.
Also, the difference between cut-through and store-and-forward is generally in single-digit microseconds, not milliseconds. A good rule of thumb is ~1 us/KB at 10G.
So there was a huge argument of CT vs SF switching way back when. IIRC there was Kalpana (CT) which Cisco acquired and Granite (Cat4K, SF) and Crescendo (Cat5/6K, SF). Cisco killed cut-through. The 1st real ASIC that did cut-through since then was the FM4000 (Fulcum) which was used in the Arista 7124, BNT, etc. The FM4000 latency was 600ns vs. 6 microseconds for the standard Wall Street switch at the time (Cat4900). 600ns to 6 microseconds is huge.
The FM4000 looked at 64 bytes before forwarding (IIRC). The 1st Broadcom chips that did cut-through were the Trident line. They looked at something like ~200 or ~500 bytes before making a forwarding decision (sorry exact number is not in cache anymore). They cut that down with the T+ chips.
The funny issue about cut-through is that it will forward a corrupted packet. Trying to pinpoint the source can be fun.
With static webpages, etc, old school internet the latency was not a big deal. The improvements in switching with cut-through and low latency were a game changer for storage, video, etc.
I've actually had this conversation before, and it get's a bit tricky.
I'd say it has a few main facets:
- ICMP is dangerous, and needs to be blocked.
This is partly true, as there is historically information leakage with ICMP under certain codes. So blocking all of it is easier than trying to figure out what should be allowed and what should not.
- It's not perceived to be a problem.
Most services just work, so TCP/IP with mss clamping will avoid the fragmentation issue for many services. So the services that have an issue, might be something like 2%, and the question that always lingers is if everything else works, and this one thing doesn't, it's not the filters problem.
- ICMP doesn't necessarily work with multiple layers of encapsulation anyways
I encountered this on mobile wireless networks, where doing multiple rounds of encapsulation caused multiple packet size reductions. Basically, if we encapsulate in tunnels used by the mobile network, then encapsulate again in say IPSec, the IPSec node will send it's icmp packet too big to the tunnel endpoint, not the originator of the traffic.
While I suppose it could be argued that the tunnel endpoint should maintain it's own PMTU, and pro-actively send it's own icmp packet too big based on this, it can be difficult to guarantee these components work.
I've seen lots of telco equipment struggle with handling this properly, and it's not just filtering, it's bugs.
So unfortunately, we're in a situation where many networks are likely miss-configured, but lots of other services for various reasons just work. So unless you can drive the networks with problems to actually change, realistically you just lose a portion of your own customer base if you don't handle the situation while your competitors just work.
I've fought this argument countless times and lost. Not being able to ping a server, check for "destination unreachable" messages, etc. leads to frustrating troubleshooting sessions.
> I'd say it has a few main facets: - ICMP is dangerous, and needs to be blocked. This is partly true, as there is historically information leakage with ICMP under certain codes. So blocking all of it is easier than trying to figure out what should be allowed and what should not.
It's actually quite easy to identify which should be allowed and which should not.
ICMP: 0,3,8,11 (echo reply, destination unreachable, echo, time exceeded)
ICMP6: 1,2,3,4,128,129,135,136 (unreachable, packet too big, time exceeded, parameter problem, echo request, echo reply, neighbor solicitation, neighbor advertisement)
See, that wasn't hard. These are the base requirements for you to have a reasonably functional IPv4 network, and the absolute minimum requirements for IPv6 to work properly.
It's just the argument that gets made for filtering, is we'd rather just filter it all instead of trying to understand what should and should not be filtered.
That's fine enough when that only directly affects those people or their networks. The article mentions Google public dns having this problem. If your customers are using them, and you're returning large DNS responses over ipv6, your customers will have trouble with resolving you names, and blame you, and you may not even be able to tell it's Google's fault.
The fix is for equipment to handle the fragmented packets properly, just like for IPv4. How is forward fragmentation, which involves creating multiple new packets + headers with recomputed checksums, easier than simply parsing a header, which is apparently "anathema to a switch"?
The problem is that the fragmentation header in IPv6 is not deterministic, so it can't be parsed in constant time, which is what makes it "anathema to a switch" which needs to be able to guarantee certain levels of packet throughput. IPv4-style fragmentation can be done in constant time.
Yes, the standard doesn't limit the number of extension headers, but looking at the fixed header, and at most 5-6 extenion headers is the maximum time required to process a packet. And that'd be enough for 100% of traffic.
The exact limit is the number of header types known to the processing node. Because encountering an unknown header should result in Parameter Error, and only one header can be used multiple times (the Destination Options), but that should be after the Frag one anyway.
44 comments
[ 3.1 ms ] story [ 98.4 ms ] threadSad to see even a NIC letting itself be MITMed by Cloudflare :(
http://icmpcheckv6.popcount.org
http://icmpcheck.popcount.org (IPv4 version)
via: https://blog.cloudflare.com/ip-fragmentation-is-broken/
With regards to fragmentation, I would say "fragmentation in general is unfixable", no matter the protocol. Thankfully with IPv6, the guaranteed minimal packet size is relatively reasonable (1280 bytes), so maybe it's now time to finally go there.
v6 already changed the rules with fragmentation a bit in that routers won't have to fragment any more but are allowed to just drop packets smaller than the MTU and sending back an ICMPv6 packet announcing that they have done so.
Unfortunately, ICMP is still dropped at many firewalls even though I would argue that it's absolutely essential for IPv6 functionality (when it was more optional in v4).
As it stands now, hosts are still allowed to fragment packets themselves (which is what's happening in this article), but the solution is probably to not do that either in the future.
DNSSec over UDP is a problem in that regards, but people aren't convinced that DNSSec is the way to go anyways (just ask tptacek here on this site).
(edit: with regards to the fragment checking site you pointed me to: In our case, all is well because I do recognise the necessity of ICMPv6 and thus have configured the firewalls accordingly)
It means my IPv6 vpn is better than IPv4 delivered by some wannabe corpo network masters.
But over time some really useful stuff has been added to ICMPv4 (early congestion notification for example) and IPv6 mostly won't work at all without ICMP.
I would argue that it's time for firewall administrators to reconsider this decision. The time where ICMP packets could do harm are long past (remember the ping-of-death on Windows 98?) and I would argue that the time when a network could work at peak efficiency (or at all) without ICMP are also mostly past.
When it's not blocked, you can debug and have healthy network, instead of a frankensteins monster.
I agree with your point, but think this needs some clarification about who considered it a best practice - among enterprise firewall admins and financial-industry compliance audits, perhaps.
Large scale network operators have seen the problems associated with entirely blocking ICMP for at least a decade. (v4 has blackhole problems too)
A lot of the hatred for ICMP came from simple things like easy endpoint discoverability. nmap blew that argument out of the water pretty quick.
But in general, fragmentation is a bad idea and we should not have gone there in the first place.
DNSSec should only ever have been specified over TCP (or not at all if you ask some people like tptacek here on this site)
If it is so much anathema to their nature then maybe they shouldn't insist on violating network layers?
Regarding firewalls, I imagine it still sucks a firewall cannot judge each IP-packet separately but needs to deal with possible fragmented packets. Then again, that is the cost of being a firewall. You need to function at the application level and yet decide on the IP-packet level.
Most firewalls I've dealt with have some option to defragment the inbound packet at the firewall so that the rest of the rules can be processed properly and to protect nodes from malicious fragments.
This sounds like it would force a return to Store-And-Forward[1], where the switch waits for the entire frame to load into memory, before forwarding to its next hop. Waiting these few milliseconds doesn't sound bad, until you consider that's added to each packet, reducing (e.g.) application performance.
[0] https://en.wikipedia.org/wiki/Cut-through_switching
[1] https://en.wikipedia.org/wiki/Store_and_forward
In other words, if you want to have a firewall rule of "drop tcp 135", in ipv4 you can just look at bytes 0x24-25 (for Ethernet). IPv6's extension-header mechanism means that the header has a non-fixed length, so you need to do work for the same effect. Given that firewalls are critical in a world without NAT, this can be a scary prospect.
Part of the blame falls on vendors who go "well, IPv6 is fringe so we'll just do all firewalling in software." The protocol is 20 years old, it's time to design the silicon.
IPv6 requires you to loop through all extension headers to reach layer 4, and you need to know about a handful of those extension headers as not all follow the same format.
Also, the difference between cut-through and store-and-forward is generally in single-digit microseconds, not milliseconds. A good rule of thumb is ~1 us/KB at 10G.
The FM4000 looked at 64 bytes before forwarding (IIRC). The 1st Broadcom chips that did cut-through were the Trident line. They looked at something like ~200 or ~500 bytes before making a forwarding decision (sorry exact number is not in cache anymore). They cut that down with the T+ chips.
The funny issue about cut-through is that it will forward a corrupted packet. Trying to pinpoint the source can be fun.
With static webpages, etc, old school internet the latency was not a big deal. The improvements in switching with cut-through and low latency were a game changer for storage, video, etc.
How about we just let the people who configure these filters solve their own problems?
I'd say it has a few main facets: - ICMP is dangerous, and needs to be blocked. This is partly true, as there is historically information leakage with ICMP under certain codes. So blocking all of it is easier than trying to figure out what should be allowed and what should not.
- It's not perceived to be a problem. Most services just work, so TCP/IP with mss clamping will avoid the fragmentation issue for many services. So the services that have an issue, might be something like 2%, and the question that always lingers is if everything else works, and this one thing doesn't, it's not the filters problem.
- ICMP doesn't necessarily work with multiple layers of encapsulation anyways I encountered this on mobile wireless networks, where doing multiple rounds of encapsulation caused multiple packet size reductions. Basically, if we encapsulate in tunnels used by the mobile network, then encapsulate again in say IPSec, the IPSec node will send it's icmp packet too big to the tunnel endpoint, not the originator of the traffic.
While I suppose it could be argued that the tunnel endpoint should maintain it's own PMTU, and pro-actively send it's own icmp packet too big based on this, it can be difficult to guarantee these components work.
I've seen lots of telco equipment struggle with handling this properly, and it's not just filtering, it's bugs.
So unfortunately, we're in a situation where many networks are likely miss-configured, but lots of other services for various reasons just work. So unless you can drive the networks with problems to actually change, realistically you just lose a portion of your own customer base if you don't handle the situation while your competitors just work.
I've fought this argument countless times and lost. Not being able to ping a server, check for "destination unreachable" messages, etc. leads to frustrating troubleshooting sessions.
It's actually quite easy to identify which should be allowed and which should not.
ICMP: 0,3,8,11 (echo reply, destination unreachable, echo, time exceeded)
ICMP6: 1,2,3,4,128,129,135,136 (unreachable, packet too big, time exceeded, parameter problem, echo request, echo reply, neighbor solicitation, neighbor advertisement)
See, that wasn't hard. These are the base requirements for you to have a reasonably functional IPv4 network, and the absolute minimum requirements for IPv6 to work properly.
It's just the argument that gets made for filtering, is we'd rather just filter it all instead of trying to understand what should and should not be filtered.
And it's not like fragmentation hasn't caused problems with firewalls in IPv4 either.
E.g. http://all.net/Analyst/netsec/1995-09.html
The fix is for equipment to handle the fragmented packets properly, just like for IPv4. How is forward fragmentation, which involves creating multiple new packets + headers with recomputed checksums, easier than simply parsing a header, which is apparently "anathema to a switch"?
Yes, the standard doesn't limit the number of extension headers, but looking at the fixed header, and at most 5-6 extenion headers is the maximum time required to process a packet. And that'd be enough for 100% of traffic.
The exact limit is the number of header types known to the processing node. Because encountering an unknown header should result in Parameter Error, and only one header can be used multiple times (the Destination Options), but that should be after the Frag one anyway.