I love this. There is too much conservative behaviour at the expense of innovation. Things break, let's be prepared, let's upgrade, let's not worry about 100% up time and start actually building something bigger.
Takeaways: a) 512K routes isn't necessarily a hardware limitation, it's the default TCAM allocation for IPv4 and B) most people most of the time don't need their routers to take a full BGP feeds worth of routes - and I hope those that do aren't running 6500's in Q3 2014 ;)
Hmm, I just realized, if enough peers with 6500s/7200s have flapping BGP sessions with their upstreams, that would be a problem even for upstreams with beefier routers and more available TCAM - too many flapping peers == too many dampened routes == lots of "fun" resetting BGP sessions and coordinating with peers. Yipes.
The backside of the reload-required TCAM reallocation is really brutal, as there are apparently a huge number of faulty cards out there in the wild which won't survive the reload:
Reminds me of when I worked at a large public university... the networking group planned on losing/replacing dozens of switches and power supplies during the summer when the power plant underwent maintenance. Losing power and reloading surfaces all sorts of defective components. And, when you have thousands of switches...
Ouch. In that case, woe be to anyone who tries to get Cisco to replace their grey-market way-beyond-deprecated almost-definitely-not-covered-by-SmartNet Sup720 linecards in 2014. Even if TAC went for that, I imagine the turnaround time would take forever...
Btw, thanks for the link! I suspect more than a few 6500 linecards that I've seen die in years past were due to something similar, as opposed to what we'd previously assumed to be the root cause - either damage in transit or "cosmic rays" ;)
It's generally a two-week turnaround for uncovered equipment, if they have remaining stock, which they always do. Given that SMARTnet-covered equipment turnaround can be 2-hour, 4-hour, or NBD and Cisco's massive global network of warehouses, it appears this constraint is artificially imposed to keep SMARTnet attach rates high.
It's very odd that "grey market" equipment is considered the standard way to refer to genuine Cisco equipment when sold by a company other than a Cisco partner. I understand keeping out counterfeits, but given the first-sale doctrine, how is a resold piece of equipment anything other than completely legitimate?
If you think about it, and get past the "it's just industry standard" mentality, it's generally insane the way that Cisco uses these pseudo-monopoly tactics. In the old days, say with maintenance on IBM Selectric typewriters, such schemes were called "bundling" and "tying," and the DOJ would pursue the companies for anti-trust violations.
Now, the DOJ arrests people based upon nebulous complaints from Cisco's general counsel. See e.g. http://abovethelaw.com/2011/07/sue-a-giant-corporation-get-r... wherein a British citizen was arrested in Canada for starting a company that competed with Cisco maintenance.
The Canadian court quashed the request for extradition after the DOJ's request trapped him in a foreign country for years. He remains under indictment here, despite the Canadian judge stating that the DOJ's case was a fairly transparent copy of Cisco's civil suit. The ruling was incendiary, stating that The extradition process to bring the applicant before United States Courts… involved innuendo, half truths and complete falsehoods.
The judge concluded:
The only reasonable inference I can draw from the facts is that the criminal process was used to pressure (unsuccessfully) the applicant into abandoning his antitrust suit against Cisco…. Any well-informed person acquainted with the truth would conclude that the collective result of the mistreatment of Mr. Adekeye offended fundamental notions of justice.
Doesn't this just make them want to go sidestep the issue and say you're not licensed to run the software, even if you can resell the hardware? Didn't Autodesk win on that, with respect to licensing software?
The "first-sale doctrine" is a legal concept that applies to combination hardware and software products. Essentially it holds that the copyright of any combination of a piece of intellectual property (e.g. Cisco IOS, or the information in a textbook) and a physical product (e.g. a router or the actual copy of a textbook book) is exhausted after the first sale. [0]
Now, there are court-recognized exceptions to this rule in various circuit courts (most significantly by the 9th Circuit in Vernor v. Autodesk, SCOTUS cert denied) for digital goods under the so-called "shrink-warp licensing" exception. [1]
So given that the highest court has allowed the ruling to stand in a large circuit, yet has consistently expanded the first-sale doctrine for four decades [2], the precedent is unclear.
Cisco does rely on Vernor and its progeny as it's justification for blacklisting resold items. But that argument is unlikely to hold water outside of the software-laden 9th Circuit.
The question, stated in layman's terms, is, "To which is a Cisco router or switch more similar, a copy of Windows or an iPod?" This is an open question from a legal standpoint, and thus we must unfortunately resort to that most unreliable of legal tools: reason.
Both options are almost equally unpalatable to the Supreme Court for policy reasons. Expand the shrink-wrap exception to cover Cisco and you risk encompassing all products that contain firmware, from TVs to microwaves, and thus strangling the half-trillion dollar secondary sale market for electronic goods. Keep the exception narrow and you choke off the single largest source of revenue (all told, including gains from new purchases, maintenance contracts, and paid software updates close to $20 Billion, or 40% of annual revenue) to the government's largest IT hardware provider (over 85% of DoD in particular), leaving the government stranded with vast networks of legacy Cisco hardware with little to no new development from the company.
The existing legal precedent creates a tough needle for the Court to thread. As a result, we have the current state of limbo, and thus discussion of "grey markets," which are neither clearly legal nor clearly illegal.
The lack of a legal secondary market can also mess with your accounting. If you don't have the right to resell the equipment because you are the sole owner of the license to the software, then its value is basically zero as soon as you buy it. Most accounting methods assume that goods retain value over time and depreciate in a set schedule. Otherwise every time you bought a Cisco router or Netapp filer the book value would be zero if it wasn't legal to sell, even during a bankruptcy auction etc.
>leaving the government stranded with vast networks of legacy Cisco hardware with little to no new development from the company.
Do we have reason to believe this is true? If resale was allowed, do we believe network hardware companies would just shrivel up? How do we square this with the facts of many platforms being intentionally crippled for marketing reasons? Or does that not happen on top-end platforms?
>If resale was allowed, do we believe network hardware companies would just shrivel up?
Network hardware companies in general wouldn't suffer, but Cisco's increasing reliance upon support, service, and firmware for revenue means that unrestricted resale would bite into Cisco's revenue hard, perhaps 25-50%.
Right now Cisco hardware powers the most critical government information systems, including military and financial communications (over 85% of the gear is Cisco.) Since many of these networks use Cisco proprietary functions, have scores of personnel trained on Cisco gear, and relationships with networks of Cisco suppliers, transition to another hardware manufacturer would be both expensive and difficult, at a time when the government has little funding for new initiatives.
Since the government needs Cisco to be a smoothly functioning company to meet critical information needs, it makes sense that they maintain tight relations.
>If resale was allowed, do we believe network hardware companies would just shrivel up?
Network hardware companies in general wouldn't suffer, but Cisco's increasing reliance upon support, service, and firmware for revenue means that unrestricted resale would bite into Cisco's revenue hard, perhaps 25-50%.
Right now Cisco hardware powers the most critical government information systems, including military and financial communications (over 85% of the gear is Cisco.) Since many of these networks use Cisco proprietary functions, have scores of personnel trained on Cisco gear, and relationships with networks of Cisco suppliers, transition to another hardware manufacturer would be both expensive and difficult, at a time when the government has little funding for new initiatives.
Since the government needs Cisco to be a smoothly functioning company to meet critical information needs, it makes sense that they maintain tight relations.
Sorry I'm not well versed in BGP (or bitcoin for that matter). I was hoping for someone with more domain knowledge to confirm or set me straight. Probably wasn't very clear by my terse initial comment.
How DOES downvoting work? All I see are upward triangles, if that has anything to do with it at all. How much more hidden functionality is there for “privileged members”? And how do you join the club?
Downvotes require a karma threshold (of 500 IIRC).
BTW, whatever you do don't overuse the 'flag' feature on stories as it seems when this is revoked it is permanent! I lost mine when trying to encourage news other than the passing of Steve Jobs.
Not really relevant to this current outage, no. But someone could potentially totally take advantage of all the noise in the current BGP route table to pull off a route hijack while NOC people have their eyes elsewhere...
I'm surprised this isn't getting more press. Shaw in Alberta has been having problems all day, not sure if it is because of this but I would hazard a guess that it is.
I find the economics of the routing table to be fascinating. When someone announces a route, it makes use of a constrained (and often expensive, TCAM-based) resource on routers all over the world. More discussion:
Wouldn't it be nice to allocate a small part of the IPv6 space to geographically encoded IP addresses? In other words, the address itself can be used to physically locate the destination and a route could be chosen partially according to geography. It seems like this should be less arbitrary than the routes you need to chose now. Or is it?
IPv6 addressing does sort of work that way: each RIR (regional Internet registry) has just a single prefix allocated out of the global unicast address space, instead of a whole bunch of prefixes all over the place as with IPv4. And the IPv6 address space is large enough that ISPs can be given a single huge prefix that they can grow into. Consequentially, the global IPv6 routing table should be a lot smaller than the IPv4 routing table is today.
There were some smaller chunks in the experimental days, but note that in 2006, each RIR was given a /12, and since then there have been zero allocations at the top level.
8 years of stability is hardly a "poorly planned mess".
In 2006 they also gave ARIN a /23 and APNIC a /20. Then, one month later they expanded some previous RIR allocations into /12's; separate from the previous two allocations I mentioned. They are also allocated in a seemingly random order throughout 2000::/3, which they've decided to constrain their allocations to for some reason.
To the point that the linux documentation project still recommends the following as an ipv6 default route example:
/sbin/ip -6 route add 2000::/3 via 2001:0db8:0:f101::1
Yea.. it's /workable/, but a lot of decisions with respect to IPv6 feel like wasted opportunities that have only slowed adoption and promoted general confusion.
Some gotchas emerge when you start thinking about actually implementing this. How can you aggregate a city under a single prefix if it has multiple ISPs?
Yes, at its core, this is the "overlapping hierarchy" problem. I believe in the case of the internet, this is almost a regulatory issue. Not to be dismissed, but failure to build a simple network structure is largely due to the number of players involved. Perhaps we could encode an ISP ID in the address as well, but that has its own issues.
Apart from the fact that "geographically close" does not always coincide with "network-wise close", that's essentially the principle behind route aggregation as used by routers today.
However, applying it successfully to IPv4 is severy compromised because, due to rampant address space fragmentation during the last years, increasingly longer prefixes (i.e., smaller address blocks) have had to be assigned. IPv6's much larger address space makes it unlikely that the routing table will ever become that fragmented in the foreseeable future.
The default partitioning of CAM space on Cisco gear is the obvious issue, but the root cause is the massive deaggregation of announced IPv4 routes on the Internet. You can see various statistics about this problem at http://www.cidr-report.org/, but the short of it is that if the top 30 networks (based on announced route savings) completely aggregated their announcements as much as possible, ~41,000 routes (~8% of the routing table) would be eliminated.
And that's just the top 30 networks — if every network cleaned up their announcements, it would eliminate ~232,000 routes (~45% of the table).
Adding to the deaggregation problem is the inability to easily filter out route announcements based on RIR minimum allocations without having to add tons of exceptions for CDNs that operate as islands of connectivity and carve out IP space for each island from a single address space allocation. (There's no covering route for the islands of connectivity since these CDNs have no "backbone" connecting the islands, so if you filter out those smaller announcements, you lose connectivity to those islands.)
There are many people who think this problem will just magically go away as IPv6 adoption increases, but all increased IPv6 adoption will do is make limited CAM space even more limited as network engineers have to balance dividing precious CAM space between a ballooning-quickly IPv4 route table and a ballooning-slightly-less-quickly IPv6 route table.
(To be clear: I think ubiquitous, functioning, end-to-end native IPv6 connectivity needs to happen sooner than later, but it's not a magic bullet for the Internet's technical problems.)
This is one of the reasons IPv6 addresses are being given out in such massive allotments. ICANN doesn't want anyone to have to get multiple "chunks" and blow up the routing table later.
So people multi-homing (which is the only reason I end up getting and announcing /24s) is not a significant amount of the route table? Cause IPv6 can't really fix that part (multihoming) can it?
You're assuming that just getting a massive opportunity for more de-aggregation won't cause de-aggregation out of shear laziness and convenience. It's much easier to de-aggregate than it is to aggregate.
The maximum IPv4 de-aggregation possible is 2^24 - 2 ^ 21. The maximum IPv6 de-aggregation with what's currently being handed out (mostly between /32 and /48's) is way, way, way more.
So IPv6 has more addresses, yes. Just so long as you don't actually use them. The problem of course is that the memory requirements for IPv4 assignments were going up linearly. If you bought gear with a good amount of memory, you could therefore expect it to last a few years. Clearly the network vendors that designed IPv6 saw this as a problem ... how can we make it explode ? Well IPv6 was the answer. Problem is that they went completely batshit insane overboard.
"If" IPv6 deaggregates we'll need routers with about 2^(48-24) TIMES more memory. Storing a deaggregated IPv6 routing table (which has to be in memory) requires 524288 terabytes of memory.
The rest of IPv6 isn't much better. It's some academics wish-list, with total disregard for real-world concerns. It is not possible to use 1/10th of the IPv6 features. Multicast ? Won't work (on any public or even just somewhat large network). Encryption ? Won't work (too many devices don't support it). Anycast ? Won't work. Site-local anycast ? Won't work. Larger address space ? Won't work (in half the world). NAT-avoidance ? Won't work (didn't get past security engineers). Autoconfiguration ? Actually kinda handy in some scenarios, but again, won't work on most networks, where you fall back to the IPv4 mechanism. Faster routing due to "smarter" headers ? Doesn't work according to my load stats. Better Qos ? Doesn't work in either of the major routing gear vendors. Mobility ? Let's not go there. Instead of implementing sane "live" re-addressing they went with ... Aargh. Let's not go there. Automatic network renumbering ... riiiight. Heh. I wonder if this was put in as a joke.
There are known solutions to all these problems (well, except the multicast and anycast ones), but of course, the IPv6 designers knew better.
Welcome to the "solution". On the other hand, solving the solution pays rather well.
Is CAM space a truly constrained resource, as-in, is Cisco really on the cutting edge to deliver whatever amounts of CAM they ship? Or is it similar to their "software" routers, where it's nearly entirely marketing/product differentiation driven? Is adding more CAM as hard as Intel shipping 64GB of L2 cache, for instance?
It is interesting that IP addresses are only 32 bits so you can provide a route for every individual IP address in a 16GB chunk of memory. You can build a 'next hop' appliance with an FPGA and a couple of DDR3 chips such that any 32 bit number input would return the next hop in 130ns after loading. It would be straight forward to create command port that took a network + netmask and filled in all the next hop info for every host on that network. And if you wanted to be cheesy you could re-use the 10/8 and the 192.168/16 spaces for various control changes.
But it is still a lot of network activity and IPV6 is just around the corner so probably not worth building today.
> It is interesting that IP addresses are only 32 bits so you can provide a route for every individual IP address in a 16GB chunk of memory ... 130ns.
1) you'd need at least 1 byte to identify a next-hop, right. So we're talking 8 bytes * 2^32 = 32 gigabyte, not 16
2) Luckily IPv6 fixed that. For IPv6 you'd need millions of terabytes to do this trick.
3) At, say, 64 10Gbit ports and 130ms lookups you'd need a mere 215 of these memories to actually be able to forward line rate.
4) Network devices are currently moving up to 40G and 100G ports in the top models. 8*100G forwarding cards exist today.
The way I was imagining how you would do this is that you would take the 32 bit IP address, shift it left by two, adding it to the base address of your 16GB of memory block and reading the 32 bits that were there. Those 32 bits being the IP address of the next hop.
I used a similar 'the address is an operand' trick to create a very fast 8x8 multiply on an 8 bit Z80, arg 0 was the upper 8 bits arg 1 was the lower 8 bits which was applied to A0-15 of a 64K x 16 EPROM, the contents of the eprom at any given location was the product of arg0 and arg1. That allowed me to save enough time to respond to packets coming in at 38400 baud over an FM side band at the time.
And if you had a tree based lookup that requires 4 lookups, but the first 2 levels are in L1 cache (suppose you do a byte trie, that's only 64k + 256 bytes), then have the next levels in small, "cheap", separate memory ?
If you had 100M of L2 cache, or 100M of SRAM, you could fit the third level in what amounts to L2 cache. You'd still have (at most) 1 memory lookup for any route (1 lookup in SRAM, 1 in RAM, worst case), but you'd need something like 256M of ram instead of 16 gigs. Vast majority of packets (going to < /24 routes) would be routed in < 10 ns, and the 130 ms becomes a 140ms worst-case. With average < 10ns routing suddenly having one of these processors for, let's say, 160 Gbit worth of ports becomes possible. And I'm sure your customers would immediately ask you to put 320G of ports on that processor.
(Principle of a trie is that IP address is a.b.c.d. So you build an array so that route_table[a][b][c][d] yields the next hop. route_table[a][b][c] happens entirely in cache + SRAM, since it only requires 100M)
Neat, I was up late last night and noticed several sites stop working at almost exactly the time of the spike on Cyrmu's graph that went away when I proxied through my Linode in the Dallas datacenter (normally I just go straight through Comcast), though I didn't really think much of it at the time.
52 comments
[ 3.2 ms ] story [ 117 ms ] threadhttp://www.cisco.com/c/en/us/support/docs/switches/catalyst-...
Takeaways: a) 512K routes isn't necessarily a hardware limitation, it's the default TCAM allocation for IPv4 and B) most people most of the time don't need their routers to take a full BGP feeds worth of routes - and I hope those that do aren't running 6500's in Q3 2014 ;)
Hmm, I just realized, if enough peers with 6500s/7200s have flapping BGP sessions with their upstreams, that would be a problem even for upstreams with beefier routers and more available TCAM - too many flapping peers == too many dampened routes == lots of "fun" resetting BGP sessions and coordinating with peers. Yipes.
http://www.cisco.com/web/about/doing_business/memory.html
Reminds me of when I worked at a large public university... the networking group planned on losing/replacing dozens of switches and power supplies during the summer when the power plant underwent maintenance. Losing power and reloading surfaces all sorts of defective components. And, when you have thousands of switches...
Btw, thanks for the link! I suspect more than a few 6500 linecards that I've seen die in years past were due to something similar, as opposed to what we'd previously assumed to be the root cause - either damage in transit or "cosmic rays" ;)
It's very odd that "grey market" equipment is considered the standard way to refer to genuine Cisco equipment when sold by a company other than a Cisco partner. I understand keeping out counterfeits, but given the first-sale doctrine, how is a resold piece of equipment anything other than completely legitimate?
If you think about it, and get past the "it's just industry standard" mentality, it's generally insane the way that Cisco uses these pseudo-monopoly tactics. In the old days, say with maintenance on IBM Selectric typewriters, such schemes were called "bundling" and "tying," and the DOJ would pursue the companies for anti-trust violations.
Now, the DOJ arrests people based upon nebulous complaints from Cisco's general counsel. See e.g. http://abovethelaw.com/2011/07/sue-a-giant-corporation-get-r... wherein a British citizen was arrested in Canada for starting a company that competed with Cisco maintenance.
The Canadian court quashed the request for extradition after the DOJ's request trapped him in a foreign country for years. He remains under indictment here, despite the Canadian judge stating that the DOJ's case was a fairly transparent copy of Cisco's civil suit. The ruling was incendiary, stating that The extradition process to bring the applicant before United States Courts… involved innuendo, half truths and complete falsehoods.
The judge concluded:
The only reasonable inference I can draw from the facts is that the criminal process was used to pressure (unsuccessfully) the applicant into abandoning his antitrust suit against Cisco…. Any well-informed person acquainted with the truth would conclude that the collective result of the mistreatment of Mr. Adekeye offended fundamental notions of justice.
Now, there are court-recognized exceptions to this rule in various circuit courts (most significantly by the 9th Circuit in Vernor v. Autodesk, SCOTUS cert denied) for digital goods under the so-called "shrink-warp licensing" exception. [1]
So given that the highest court has allowed the ruling to stand in a large circuit, yet has consistently expanded the first-sale doctrine for four decades [2], the precedent is unclear.
Cisco does rely on Vernor and its progeny as it's justification for blacklisting resold items. But that argument is unlikely to hold water outside of the software-laden 9th Circuit.
The question, stated in layman's terms, is, "To which is a Cisco router or switch more similar, a copy of Windows or an iPod?" This is an open question from a legal standpoint, and thus we must unfortunately resort to that most unreliable of legal tools: reason.
Both options are almost equally unpalatable to the Supreme Court for policy reasons. Expand the shrink-wrap exception to cover Cisco and you risk encompassing all products that contain firmware, from TVs to microwaves, and thus strangling the half-trillion dollar secondary sale market for electronic goods. Keep the exception narrow and you choke off the single largest source of revenue (all told, including gains from new purchases, maintenance contracts, and paid software updates close to $20 Billion, or 40% of annual revenue) to the government's largest IT hardware provider (over 85% of DoD in particular), leaving the government stranded with vast networks of legacy Cisco hardware with little to no new development from the company.
The existing legal precedent creates a tough needle for the Court to thread. As a result, we have the current state of limbo, and thus discussion of "grey markets," which are neither clearly legal nor clearly illegal.
[0] https://en.wikipedia.org/wiki/First-sale_doctrine [1] https://en.wikipedia.org/wiki/Vernor_v._Autodesk,_Inc. [2] Most recently in Kirtsaeng v. John Wiley & Sons, Inc., No. 11-697 (U.S. Mar. 19, 2013)
>leaving the government stranded with vast networks of legacy Cisco hardware with little to no new development from the company.
Do we have reason to believe this is true? If resale was allowed, do we believe network hardware companies would just shrivel up? How do we square this with the facts of many platforms being intentionally crippled for marketing reasons? Or does that not happen on top-end platforms?
Network hardware companies in general wouldn't suffer, but Cisco's increasing reliance upon support, service, and firmware for revenue means that unrestricted resale would bite into Cisco's revenue hard, perhaps 25-50%.
Right now Cisco hardware powers the most critical government information systems, including military and financial communications (over 85% of the gear is Cisco.) Since many of these networks use Cisco proprietary functions, have scores of personnel trained on Cisco gear, and relationships with networks of Cisco suppliers, transition to another hardware manufacturer would be both expensive and difficult, at a time when the government has little funding for new initiatives.
Since the government needs Cisco to be a smoothly functioning company to meet critical information needs, it makes sense that they maintain tight relations.
Network hardware companies in general wouldn't suffer, but Cisco's increasing reliance upon support, service, and firmware for revenue means that unrestricted resale would bite into Cisco's revenue hard, perhaps 25-50%.
Right now Cisco hardware powers the most critical government information systems, including military and financial communications (over 85% of the gear is Cisco.) Since many of these networks use Cisco proprietary functions, have scores of personnel trained on Cisco gear, and relationships with networks of Cisco suppliers, transition to another hardware manufacturer would be both expensive and difficult, at a time when the government has little funding for new initiatives.
Since the government needs Cisco to be a smoothly functioning company to meet critical information needs, it makes sense that they maintain tight relations.
edit: I'll take it by the downvotes without responses that's a "no"?
BTW, whatever you do don't overuse the 'flag' feature on stories as it seems when this is revoked it is permanent! I lost mine when trying to encourage news other than the passing of Steve Jobs.
http://markmail.org/message/n32fmeb2dmtnbsff
I find the economics of the routing table to be fascinating. When someone announces a route, it makes use of a constrained (and often expensive, TCAM-based) resource on routers all over the world. More discussion:
http://markmail.org/message/6sunzqtffav5jmfb
OH WAIT, WE DIDNT FKIN THINK OF THIS IN OUR RUSH TO PUSH A BROKEN INCOMPLETE SOLUTION.
http://www.iana.org/assignments/ipv6-unicast-address-assignm...
8 years of stability is hardly a "poorly planned mess".
To the point that the linux documentation project still recommends the following as an ipv6 default route example: /sbin/ip -6 route add 2000::/3 via 2001:0db8:0:f101::1
Yea.. it's /workable/, but a lot of decisions with respect to IPv6 feel like wasted opportunities that have only slowed adoption and promoted general confusion.
Some gotchas emerge when you start thinking about actually implementing this. How can you aggregate a city under a single prefix if it has multiple ISPs?
However, applying it successfully to IPv4 is severy compromised because, due to rampant address space fragmentation during the last years, increasingly longer prefixes (i.e., smaller address blocks) have had to be assigned. IPv6's much larger address space makes it unlikely that the routing table will ever become that fragmented in the foreseeable future.
And that's just the top 30 networks — if every network cleaned up their announcements, it would eliminate ~232,000 routes (~45% of the table).
Adding to the deaggregation problem is the inability to easily filter out route announcements based on RIR minimum allocations without having to add tons of exceptions for CDNs that operate as islands of connectivity and carve out IP space for each island from a single address space allocation. (There's no covering route for the islands of connectivity since these CDNs have no "backbone" connecting the islands, so if you filter out those smaller announcements, you lose connectivity to those islands.)
There are many people who think this problem will just magically go away as IPv6 adoption increases, but all increased IPv6 adoption will do is make limited CAM space even more limited as network engineers have to balance dividing precious CAM space between a ballooning-quickly IPv4 route table and a ballooning-slightly-less-quickly IPv6 route table.
(To be clear: I think ubiquitous, functioning, end-to-end native IPv6 connectivity needs to happen sooner than later, but it's not a magic bullet for the Internet's technical problems.)
The maximum IPv4 de-aggregation possible is 2^24 - 2 ^ 21. The maximum IPv6 de-aggregation with what's currently being handed out (mostly between /32 and /48's) is way, way, way more.
So IPv6 has more addresses, yes. Just so long as you don't actually use them. The problem of course is that the memory requirements for IPv4 assignments were going up linearly. If you bought gear with a good amount of memory, you could therefore expect it to last a few years. Clearly the network vendors that designed IPv6 saw this as a problem ... how can we make it explode ? Well IPv6 was the answer. Problem is that they went completely batshit insane overboard.
"If" IPv6 deaggregates we'll need routers with about 2^(48-24) TIMES more memory. Storing a deaggregated IPv6 routing table (which has to be in memory) requires 524288 terabytes of memory.
The rest of IPv6 isn't much better. It's some academics wish-list, with total disregard for real-world concerns. It is not possible to use 1/10th of the IPv6 features. Multicast ? Won't work (on any public or even just somewhat large network). Encryption ? Won't work (too many devices don't support it). Anycast ? Won't work. Site-local anycast ? Won't work. Larger address space ? Won't work (in half the world). NAT-avoidance ? Won't work (didn't get past security engineers). Autoconfiguration ? Actually kinda handy in some scenarios, but again, won't work on most networks, where you fall back to the IPv4 mechanism. Faster routing due to "smarter" headers ? Doesn't work according to my load stats. Better Qos ? Doesn't work in either of the major routing gear vendors. Mobility ? Let's not go there. Instead of implementing sane "live" re-addressing they went with ... Aargh. Let's not go there. Automatic network renumbering ... riiiight. Heh. I wonder if this was put in as a joke.
There are known solutions to all these problems (well, except the multicast and anycast ones), but of course, the IPv6 designers knew better.
Welcome to the "solution". On the other hand, solving the solution pays rather well.
But it is still a lot of network activity and IPV6 is just around the corner so probably not worth building today.
1) you'd need at least 1 byte to identify a next-hop, right. So we're talking 8 bytes * 2^32 = 32 gigabyte, not 16 2) Luckily IPv6 fixed that. For IPv6 you'd need millions of terabytes to do this trick. 3) At, say, 64 10Gbit ports and 130ms lookups you'd need a mere 215 of these memories to actually be able to forward line rate. 4) Network devices are currently moving up to 40G and 100G ports in the top models. 8*100G forwarding cards exist today.
I used a similar 'the address is an operand' trick to create a very fast 8x8 multiply on an 8 bit Z80, arg 0 was the upper 8 bits arg 1 was the lower 8 bits which was applied to A0-15 of a 64K x 16 EPROM, the contents of the eprom at any given location was the product of arg0 and arg1. That allowed me to save enough time to respond to packets coming in at 38400 baud over an FM side band at the time.
If you had 100M of L2 cache, or 100M of SRAM, you could fit the third level in what amounts to L2 cache. You'd still have (at most) 1 memory lookup for any route (1 lookup in SRAM, 1 in RAM, worst case), but you'd need something like 256M of ram instead of 16 gigs. Vast majority of packets (going to < /24 routes) would be routed in < 10 ns, and the 130 ms becomes a 140ms worst-case. With average < 10ns routing suddenly having one of these processors for, let's say, 160 Gbit worth of ports becomes possible. And I'm sure your customers would immediately ask you to put 320G of ports on that processor.
(Principle of a trie is that IP address is a.b.c.d. So you build an array so that route_table[a][b][c][d] yields the next hop. route_table[a][b][c] happens entirely in cache + SRAM, since it only requires 100M)