My ISP hasn't given me an IPv6 address so I can't play with it easily so it remains a mystery for me. I managed to get a tunnel up and running so I could test & configure the IPv6 address my server had available for years but I hadn't setup.
What needs to happen for IPv6 to take off? CloudFlare is doing a lot by allowing automatic IPv6 for many websites - it makes it very easy to get a foothold in IPv6.
1. Consumer grade routing hardware that always supports ipv6 out of the box.
2. ISPs transitioning into supplying both an ipv4 and ipv6 address to households.
3. Major network backbone services (google, amazon, etc) transition to both ipv4 and ipv6.
4. Once a majority of the critical network infrastructure is available on both, ISPs need to start dropping ipv4 addresses from customers, preferrably from new customers.
5. Slowly depreciate ipv4 by using a ISP provided ipv4 tunnel service for a while for those that still need to access the ipv4 web.
Basically, it requires most of the work be done on the backs of ISPs. In the US at least, the ISP business is an oligopoly of cable tv and Hollywood and will resist ipv6 because it enables more peer to peer trafficking and open communication over the Internet.
Actually, you'd be surprised -- Comcast is leading the charge on the residential IPv6 front in the US, and Time Warner Cable isn't too far behind. Verizon, the largest carrier in the US not owned by a content company (by my back-of-napkin math), has been slower in deployment of IPv6, by contrast. (OTOH, since IPv6 is critical to LTE deployment, VZW et al have been much quicker on the uptake.)
I actually think that the content hosting folks are behind the curve here. Amazon is one of the biggest hold-outs; their IPv6 rollout has thus far been a total joke. No IPv6 for Cloudfront? No IPv6 glue for Route53 authoritative nameservers? No IPv6 on VPC, barely on EC2 at all, etc., etc. From what I've heard, the management at Amazon doesn't really care about the network behind their infrastructure; instead of investing in building a backbone that could do a better job supporting this kind of stuff -- and all kinds of inter-region applications which would be useful -- they want pretend the network doesn't exist and that nothing needs to change. It's too bad. I think we'd see more IPv6 usage if Amazon would get their act together.
I have TWC residential service. Coincidentally, I just experimented with setting up IPv6 yesterday on my Airport Extreme. TWC is not issuing IPv6 addresses at my location. So I went with an automatic tunnel (192.88.99.1). That IP is 16 hops and 150 ms away for me (anycast by Congent in my case) and its performance was awful.
I switched to a manual tunnel from HE (tunnelbroker.net). I confirmed this worked, that my Apple devices behind the Airport Express all got IPv6 addresses and so forth, and that the performance was decent.
Experience in hand, I then tore it all down, since I couldn't think of a good reason why I really needed IPv6 at home. :-)
Good. IPv6 is a failed experiment. It's time to reconsider the IP service model. The future of the Internet belongs to startups, not to the IETF.
My hope is that by 2020, IP will have approximately the same importance to the global connectivity fabric that Ethernet has today; it'll be a particularly flexible next-hop technology, and we'll be routing something more clever than IP on top of it.
There is no candidate to replace IP for global routing connectivity in the next 20 years. I'll wager a long-bet at 10 to 1 odds up to $1,000 (you win $10,000) for anybody who thinks differently.
Thomas always says this whenever IPv6 comes up. He wants some sort of "overlay" network. He's ambiguous on what this would look like, why it would be better than IPv6, and why IPv4 will suffice as its underlying protocol.
My conclusion is that his application-level networking experience has fooled him into thinking he knows more than he does about actual network architecture, management, and performance issues. Or maybe he doesn't realize the issues even exist.
Um, you're aware that he wrote http://insecure.org/stf/secnet_ids/secnet_ids.html, worked for a networking (security) company (Arbor) for years, etc.? He may be wrong, and he may think that he knows more than he actually does, but he does know a thing or two. ;-)
Regarding overlay networks: note that HTTP, WebSocket, Skype, BitTorrent and such are already looking quite a bit like an overlay network, and that increasing amounts of NAT-ting are making the classic point-to-point model of the internet more and more irrelevant. I'm not sure I particularly like this vision of the future, and I agree that the performance will be worse, but it's far from insane.
(As to performance, note how everything is routed over HTTP these days, despite it being rather inefficient for lots of use cases.)
Forgive me, as I don't have time at the moment to review the paper in detail, but none of what I see skimming through it really contradicts what I already believed. Knowing how the protocols work is not the same thing as knowing how to architect and manage large networks.
The "overlay" networks you speak of are already hamstrung by the shrinking ability to directly address endpoints on the Internet. IPv4's fundamental flaws when applied to billions of nodes cannot be papered over forever.
> May I suggest you withdraw from this line of argument?
May I suggest that you look at the paper in question and realize it is one that requires careful and uninterrupted attention for an extended period of time to properly understand?
May I also suggest that you not make assumptions as to what another person is or is not doing, and how it may interfere with their ability to do some things but not others?
Or would you rather just be a judgmental, holier-than-thou asshole?
My conclusion is that his application-level networking experience has fooled him into thinking he knows more than he does about actual network architecture, management, and performance issues. Or maybe he doesn't realize the issues even exist.
My conclusion is that you made that comment having absolutely no idea what my experience was, and then decided to get pissy when the facts didn't line up with your bet.
I've made the same dumb bet on HN a bunch of times. "This is just some Ruby dork who knows fuck all about", I don't know, "tax law". It never works out for me, either.
The thread is now full of technical commentary about the point I made. Lots of smart people disagree with me; no surprise! You've contributed none of that commentary. Maybe now's a good time to start? You seem to feel passionately about the issue.
> you made that comment having absolutely no idea what my experience was
I made the comment based on the fact that the only thing you ever talk about actually doing is security-related. On everything else, you just pontificate context-free.
You implied earlier I should have looked at your "public linkedin profile". You should know that, for me, your profile contains nothing but your current and past employers, and your university. Nothing about what you did there. What was I supposed to conclude from that?
I've spent most of my career working for companies that do video-related work, but if you ask me a question about MPEG transport streams, I'll refer you to the guy who, y'know, actually deals with MPEG.
At this point I am pissy, yes, but that's not because I was wrong, it's because of the asshole who decided he knows better what my personal time commitments are than I do.
> The thread is now full of technical commentary about the point I made.
No, it isn't. The thread is full of cursory rejections of your equally cursory commentary. There is absolutely nothing interesting about any of it, because it's all surface bullshit that everybody already knows. There are no details. There's nothing concrete. There's nothing new. Boring, hand-wavy crap.
Propose something concrete, something that actually addresses the obvious criticisms you already know are going to be leveled, and I might get interested.
Right now, all I'm interested in is ignoring HN for a while.
It's a bad thing now, because the only way to get assured direct connectivity to the "Internet" is via IPv4 connectivity.
It would be less of a bad thing if IPv4 connectivity (specifically: what ASN was advertising the IPv4 prefixes your traffic was associated with when it traversed, say, Telecom Malaysia) wasn't the sine qua non of global connectivity, in the same sense that nobody cares what IP address you use when you connect to IRC.
You can connect to IRC because the IRC server is not behind a NAT. Try to run an IRC server on your host which runs behind the consumer NAT which runs behind the CGN. Yes, if the gear supports it, you can use uPNP for the first, and PCP for the second, to open up a port.
Unless someone is using it already.
Then you need to choose another port.
Which you need to communicate to the other clients behind the NATs around the world. Fine, we'll just update the DNS server with an SRV record. You probably will always have a spare public IPv4 address to map the ports 53 to something on the inside network, so this might work.
Until your CGN does not have enough IPv4 addresses/ports to permanently allocate one to you. Oh, by the way this is a great way to introduce a "premium" tier service with a public IP address - if you really want to run a server then you can buy an address. So doing the PBX-style circuit switching with NATs is not really in the economic interest of the ISPs. (Not saying that they need to do all the accounting/logging).
If the app is awesome enough that the users will pay extra to get the public address - great.
That will help to further exhaust the IPv4 addresses and drive the prices up. Rinse, repeat.
I am of course exaggerating.
But this presentation from NANOG may be a less biased view (though Lee's employer does deploy IPv6, so you can argue it is biased):
Full disclosure: I am into this IPv6 thing. Before that, I spent ~10 years of my life at a random 5-letter vendor to find the bugs in the code and misconfigurations causing outages in people's NATs. It's refreshing to be able to use access-lists again. (the latter statement is only partially correct, but this is a topic for another rant, IPv6 has tons of its warts as well :-)
With an overlay you don't need a public IP to run a server. Look at Tor hidden services for example. Only the overlay "backbone" servers themselves need public IPs.
(I am actually opposed to overlays uber alles, partly because I think it may actually happen.)
Of course. But it can't be turtles all the way down - at some level, the overlay traffic will be a TCP segment or UDP packet, sent from one node running overlay network to another node running overlay network. If both nodes in the overlay network are behind the two "cone" NAPTs like this:
A --- |> --- <| --- B
then getting from "A" to "B" is not easy. And the "cones" may be of varying degree of aggressiveness when it comes to filtering [1]. You can not guarantee that you can always establish connectivity between the two, even in a presence of the third party - with the addresses being more and more precious, the SPs will move from full-cone to more restricted NAPTs.
Besides these hurdles, moving large volumes of traffic atop overlays is not very optimal - I heard anecdotes that when Tor is abused in this manner, it is quite slow. Skype (and I guess probably Tor as well) uses supernodes that are not behind the NAPT to establish the communication. As there is less and less places for it.
So, the quality of communication over overlay degrades more and more.
Meanwhile, the bigger players will just place the caches close to eyeballs or directly peer with the eyeball providers. [2]
So, I think Lee's scenario is quite plausible.
Pretty much the only things keeping holding IPv4 in my home network are the client device bugs [3] and Skype. The web works quite well on DNS64.
I think the economics will take care of things. There are users on IPv6. A few clever interns last year made a website [4] that aggregates various IPv6-related stats. Take a look at the situation with users e.g. in Germany, Belgium, Switzerland. Yes, it is still not a whole lot but it is growing - and these three give the impression of how it might happen - sometimes steady, sometimes the switch will just turn on.
I am very curious to hear some VC chime in on this matter - what do they think about the prospects of business growth in the situation of the constrained address space and uneven competition. It is not the case today because ARIN still has numbers. Give it a couple of years.
I think it turns out not to be hard to resolve the A<->B rendezvous problem if you're willing to have an M that forwards packets.
I think a lot of crappy 1980s protocol architecture is based on a fear of forwarding packets that (to invoke a shorthand) Moore's Law has addressed for us.
Mostly because of the inefficiency. I'm concerned about who's going to pay for the servers. I'm concerned that a monopoly walled garden business model might win out and one company would take control of the servers as a strategic choke point. (I'm also concerned about who's paying for our IP routers, since we had a business model that worked fine but ISPs are starting to get greedy.)
Well, let's see. I ran all of technical operations for Chicago's most popular indie ISP, up through the point where we were default-free BGP4 peered, in the era where every customer connection every day was an OSPF recalculation.
Then I took a job that required me to write the clientside for every protocol exposed on a "modern network" circa 1996, including FTP, TFTP, NFS, SunRPC, DNS (for which I had to write 3 different recursors and an authority server to do serverside-serverside testing), RIP(!), and lord knows what else. Our product was a network security scanner, before there was Nessus, that worked by actually testing for bugs instead of doing banner grabs.
I moved to a research role in that job and built a testsuite for network intrusion detection systems based on a line-by-line audit the 4.4BSD IP stack, then used it to reverse engineer the behaviors of the Windows NT and Solaris (then closed-source) IP stacks, and used that to discover (with my then-partner Tim Newsham) 3 new classes of attacks/bugs on network observation tools. Incidentally: I did that research because I have an ethical problem with unchecked network surveillance. Half this site seems to think I'm a secret government agent.
Then I started, with two friends, a chat company that pivoted into multicast content delivery. My role on the dev team was the routing system, which implemented a multicast link-state routing protocol and a multicast service model comparable to PIM; I was also the project manager for our global traffic management system, which took full BGP4 feeds from the networks of friends of mine that ran backbones, and used them to compute anycast DNS results.
When that company failed because our VCs and the CEO they installed decided we should go head to head with Akamai and invest all our money in global colocation fees, I thought long and hard about where to go next and decided I didn't yet know enough about networking.
So I took at a job as the team lead for Arbor Networks Peakflow DOS, an anti-DOS product now deployed on the backbones of to a first approximation all the Internets. Peakflow worked by consuming NetFlow records from the backbone Cats and GSRs of companies like AT&T; ie, it binned and computed stats for a large percentage of all the packets traversing the--- what would you call it? Yeah, the Internet.
Your call whether my career experience as a network designer, a network operator, a provider of network management and performance monitoring tools, and a kernel network software developer is germane to the discussion.
Have we dispensed with the part of this thread where we wonder whether I have any business having ideas about how the Internet might work, and move on to the part where you actually try to shoot down those ideas? The former conversation is irritating, because I'm re-explaining things you could have found out simply by reading my public LinkedIn profile. But I would love to be proven wrong on my ideas; that would involve me actually learning something.
I think you now owe me your best shot at taking down the actual notion of a next-gen Internet built as an application-layer overlay.
Fine, you have a fair bit of experience, though I'm frankly still not seeing architectural experience in there. You're just wrong. I can live with that explanation.
And I'd be happy to shoot down your ideas, if you had any. All you have is your regular content-free rant against IPv6 that you constantly re-post.
> Half this site seems to think I'm a secret government agent.
No, we just think you're a strict authoritarian who gives the American left a bad name.
First of all, I've never even seen the second of those two threads. Shocking as it may be to your ego, not everyone follows each and every one of your myriad utterances on HN.
But neither thread is one of substance, only complaints and a vague "we need a general overlay network". Not one word about what it should look like. How it should work. Nothing.
If you want to argue about an idea, articulate one.
Edit: I see in the second thread you actually did say something about a directory service, so I guess that is one word. But as far as I can tell, you just described yet-another-DNS-replacement. If I were generous, maybe I would call it an Active Directory competitor.
Are you going to articulate an idea, or just be your usual self?
Edit: Ah, you've edited it to clarify. You're just going to be your usual self. How disappointing. Broadly speaking, you're an interesting, intelligent person I would love to know better and to learn from. But you can't stop yourself from being the get-off-my-lawn guy.
Since I'm randomly coining terms here, let me try to be clearer.
When I talk about next-hop transport, I mean, "the mechanism by which one node might connect to one or more other nodes in the topology". As soon as those hops form a connected graph --- a very easy problem to solve when your connection fabric is datagram delivery on IP --- routing over the topology is a Solved Problem. Think of what I mean by "next hop transport" as "the lowest common denominator of connectivity that allows any 2 hosts in the world, possibly by means of some number of intermediate hops, to talk to each other".
Then, to be clear about what I'm saying --- though this is probably not how you'd design such a system in the real word, imagine we did this:
* Stopped fussing about IPv6 (and IP multicast, although we've already stopped fussing about that).
* Took IRC
* Made it 8 bit clean and capable of delivering much larger messages.
* Gave it a real routing protocol so that it could express biconnected graphs.
* Built applications on top of that.
That system would provide a service model that would be compatible with a superset of all Internet applications today --- it would allow point-to-multipoint applications along with point-to-point applications, it would admit to client-server, it could express caching and CDN topologies, and it would admit to logical addressing that would enable you to attach to lots of different endpoints on the network with the same, static address, in the same way as I can talk to you now on HN as "tptacek" despite the fact that my IP address is temporarily owned by GoGo Inflight.
I did not intend for it to explode like that... but I'm on the Internet, so I'm not surprised.
I think I also see where you're going with this; we've got so many adhoc implementations of this already (I'll toss on XMPP as another example) that some sort of eventual agreement and standardization seems inevitable. Don't know where it'll come from or what it will look like, but Historical Imperative is calling for it.
Well, we have 12 millions+ nodes happily communicating every day via IPv6 that suggests IPv6 is a pretty successful "experiment."
I know you meant to say, "Failed experiment for consumer web connectivity" - but, I think it's important to remember that IPv6 has some very successful niches. Mesh networks, Smart Grids, anything with large scale flexible address and routing requirements has been using IPv6 very successfully for at least 5+ years now.
The only real competition IPv6 has over the next 20 years for global routing dominance though, is, ironically, IPv4.
The main reason is that all the Autonomous Systems have to have some common language to communicate, and share routing updates, and, for better or worse, IP is the "English" of network engineers when it comes to internetwork routing.
I agree: IPv4 is going to be a fierce competitor for the slot that IPv4 currently occupies and IPv6 hopes to occupy. The point of an overlay design is to moot the point of which one wins, by "pushing them down" the stack and building something new on top of them.
Frame Relay is a good example of what this design looks like: a "layer 2" technology with "layer 3" features (Frame had aspirations to be a layer 3 technology) that only network operators care about, because no Frame user ever expected global all-points connectivity to be provided by Frame.
IPvX should be demoted to Frame Relay status: a very flexible way of providing point-to-point connectivity between nodes that will provide global connectivity through routing done at a higher layer.
It looks like you went down the front page desperately looking for any non-NSA related articles to comment on, but rest assured the rest of us have been reading them and your boy Obama and his NSA have just blown the SaaS web startup industry out of the water. No web startup has a chance in hell of selling services to anybody outside the USA now (or anybody inside the USA with half a brain).
I'm a little confused as to what you're suggesting here. The best interpretation I can find is that you think IP will be replaced by a mish-mash of different network operators' ad hoc standards. But that hardly seems like something to hope for.
That could be the natural evolution of things, until at some point in the future one by one the CEOs of the world are breaking their touch screens in half out of frustration, "why is the network so fucking slow and unreliable?" Then the CTO casually mentions the darknet (ie. IPv6) that all the hardcore tech geeks are using for high performance network apps.
In other words, IPv6 adoption will accelerate by orders of magnitude once the real pain of IPv4 address exhaustion starts to be felt.
I think network operators should have about as much say in what service models Internet protocols should support as the telephone companies do about what apps run on Android and iOS.
I think large open source projects (like BitTorrent) and startups should start thinking about applications that build their own connectivity layer on top of IP, and that don't assume any two nodes will have direct IP connectivity. Obviously, this is already true of the Internet we use today, because a huge percentage of hosts don't expose direct connectivity for reasons of security (not addressing scarcity --- there are companies sitting on /16s that static-address every desktop, but allow none of them direct Internet connectivity).
If you believe TorrentFreak, BitTorrent accounts for double digit percentage points of all Internet traffic. BitTorrent is an overlay design (albeit not one that reliably avoids the need for direct point-to-point connectivity, unlike the Gnutella overlay). We already know designs like this are tenable. I'm saying we should explore them further as alternatives to an Internet design controlled by giant telecoms.
Nobody objects to building layers on top of TCP/IP. We've been doing that forever. I think we are all very confused by the idea that a layer built on top of TCP/IP will replace TCP/IP? It just doesn't make sense.
Actually, I'm saying we should replace IP, in the same way that IP "replaced" X.25 (if you want to call Frame Relay the successor X.25) by relegating it to a layer 2 connectivity technology.
I'm saying it would be great if all web pages were delivered over an overlay protocol running on top of some TCP friendly transport provided by an overlay; a "peer to peer" routing system like Skype.
The simplest way to describe the architecture I'm talking about is, "what if you made IRC a modern, fully routed binary connectivity fabric, then tunnelled the Internet over it?"
TCP/IP is peer to peer, and IRC is built on top of TCP/IP. Maybe I'm just misunderstanding your proposal, but it seems like what you want to do is what we already have.
How many of these P2P overlay networks could run without a centralized head/mapping server? BitTorrent has drastically reduced the size of theirs, but it still needs a mapping of other hosts somewhere. Same thing for Skype. I don't want to sign up for an internet where every connection needs a 'head' in order to determine how to traverse the overlay network.
It's probably also not a coincidence that Skype and Bittorrent are two of the hardest protocols to 'control' inside the network. This sounds good from a 'the internet should be a free democracy' standpoint but bad from a 'let's guarantee the performance of the application' standpoint. In terms of visibility into what is happening, being transferred, and where connections terminate there are more productive rather than draconian reasons for understanding those things.
Today with virtual hosting if the website next to you decides to host illegal content, the entire IP is taken down collateral damages or not. Well, that's part of your choice in choosing a provider as an end host. If the entire next generation internet and application space is based on overlays and other nebulous architectures, much of those same problems are going to exist to larger scales, including at transit points. Except it won't be as optional as changing hosting providers.
Since you've got BGP experience - think about the fun in troubleshooting eBGP neighbor problems when every connection is a GRE tunnel that may actually be tunneling through another BGP AS, which itself may be tunneled again. GRE tunnels are great immediate bandaids and generally awful long term solutions for enterprise networks to say the least. But the application guy gets his green light and thinks everything is great.
These are great points! I'm definitely not saying you could literally take IRC, give it a routing protocol, make it 8 bit clean, and use it as "the new Internet". It would break down in a lot of the ways you say.
I think we don't know enough yet about how to build reliable network topologies on top of reliable datagram delivery to have immediate answers to all of these issues.
I think one baby-step answer is to say, let's start figuring out how to do this on an application-by-application basis. But then, we're already saying that; that's how Skype worked, and Skype was (a) ambitious and (b) worked!
So then:
* I don't think things need to boil down to a single head. IRC doesn't. I remember IRC working even back during the NSFNet split in the mid-90's, when my ISP (Ripco) lost connectivity to university sites. That's because IRC is a mesh of lots of head-end services that leaf nodes rendezvous with.
* Skype and BT are hard to control because they are deliberately designed to be hard to control, because they're P2P (in the late-90's sense) schemes. Their deployment priority was to work in spite of network policy controls, not with them. But that doesn't need to be true of a next-gen system. Can you think of ideas to make protocols easier to moderate and make policy decisions about? I can, but wouldn't want to belabor the point.
* It's definitely true that running a dynamic routing protocol on top of a transport routed by its own dynamic routing protocol is a recipe for byzantine failure. But mitigating that issue is the fact that routing on top of a well-connected (but not completely-connected) transport is a simpler problem than traditional routing protocols handle; I remember feeling silly going through all the contortions OSPF goes through to carefully forward link state from hop to hop, knowing all along I could just be pulling the whole LSA table from a central database.
I'm not sure how to read your comment. Are you arguing that we can somehow make the internet work better by having everyone use whatever routing protocol they please, and throwing standards out the window?
Why is TCP/IP one of those standards you want to get rid of? Its flaws (particularly IPv6's flaws) appear to be primarily academic, and from my experience the cost of not having standardization is significant, so we'd need to see significant gains to make it worthwhile.
If you want to throw out TCP/IP, I hope you have specific dramatic improvements for its replacement in mind.
Careful: I don't want to ditch TCP, and I don't want to "ditch" IP so much as I want to push it down a level.
I have a bunch of problems with IPv6:
* IPv6 doesn't solve the multicast problem; multicast was attempted in an IP context and failed because when you try to express something as complex as group message delivery on IP, you needlessly complicate things, and because multicast on IP simply doesn't scale, because it conceptually requires core routers to maintain routing state for individual pieces of content.
* The topology of IPv6 is controlled by giant telecoms; also, some of what you are and aren't allowed to do with the connectivity you buy is controlled by telecoms. I don't have a moral problem with this, but it's a needless coupling between Internet and telephony businesses.
* IPv6 doesn't move us from our current state of "by default I don't have direct peer level connectivity with hosts", because in the IPv6 world, everyone will still be hiding behind firewalls, because for 2 decades the crappy software we had hanging out on TCP and UDP ports turned out to be too unsafe to expose. Overlays that would provide direct connectivity --- in the same sense as I can IM you on AIM or Gtalk right now --- would be starting fresh and would provide default direct connectivity to everyone. Anyone who's ever had to punch a hole through a firewall to allow inbound connectivity understands this.
* IPv6 is tremendously expensive to deploy: http://cr.yp.to/djbdns/ipv6mess.html --- forget the routers, we have so much software that has to get forklifted out to accomodate addresses that don't act like scalar integers.
What percentage of all Internet traffic is already carried over overlay networks of some sort? You need to account for all of BitTorrent and all of Skype, both of which are overlay designs. It is also worth noting that SMTP is an application overlay design; you just don't notice it because in its modern configuration everyone is one hop away from each other.
Open source project sounds more likely, though you did say "startup". Startups don't seem to be rushing to build network infrastructure (apart from their own).
Still , I struggle to imagine how such a network might work. Things like Kademlia seem inefficient and inflexible for general use.
My ISP has the technology and infrastructure in place to make sure that when I connect to 184.172.10.74 it actually goes to the correct place. They seem ideally suited for this because they have the physical wires.
How could I be sure I was talking to the right endpoint on an overlay network? PKI?
Do I have to choose an overlay provider to route my packets around?
Unfortunately getting it to run involved sacrificing at least a dozen newborn animals - which is maybe the reason it never went up.
Also, of course the mail delivery took tens of minutes even in a trivial ring of 4 machines connected on the local LAN. Maybe I had something wrong in the setup, though.
I'm really burying the lede on my crazy dream, because:
The great thing about overlay networks is you don't need anyone's permission to build them.
All you need is an app that really benefits from the advantages of an overlay network, some of which include:
* Point-to-multipoint or multipoint-multipoint delivery (ie, multicast)
* Logical addressing decoupled from network topology
* All points rendezvous between nodes that don't have inbound connectivity because of firewalls
I agree with you that it's unlikely anyone will be successful building this as a product that is simply "new Internet"; what I think is more likely is that people will starting having more and more Skype-like successes by ignoring the "rules" of IPv4 connectivity and routing around them in the "application layer"† like the brain damage they are.
† ... which is a stupid term that blinds people to how much power we have as developers
More likely that each startup cobbles together their own Internet to build their app on. (This muck also creates a nice barrier to entry against other startups.)
OK, so I have put together such a network. It's foolproof with NAT traversal and easy to use because I've integrated it into a portable OS. But I'm a do everything with netcat sort of guy and so I haven't written any sexy and compelling "apps" to run on it. Others would have to do that. My question is: What is the market demand for this? I mean, a few HN'ers say they want it. But that does not make a startup.
> Prince is hoping that the growth will not be steady-state but exponential, accelerating through the adoption curve. Even if that happens, however, CloudFlare predicts that full IPv6 adoption would take seven years, until January 2020.
> Not impressive.
Perhaps not impressive, but pretty much as-expected. RFC 2460 was finalized December 1998. I'm not sure why a 20-year +/- deployment process should surprise anyone for what is, ultimately, a telecom standard, one built before it was needed because we knew it would be.
Remember, APNIC is the first to hit its last /8 and only did so in 2011, and due to their exhaustion policy, hasn't exhausted that last /8 yet. Other regions are in better shape.
As we've always known, the big push will come when the IPv4 supply dwindles so low that it becomes reasonably impossible to accomodate new endpoints.
Still waiting on my ISP.
I'd love to implement IPv6 on my servers, but no way to test it atm. (unless using tunnels an alike)
Edit: Just rechecked at my provider ... they are rolling it out this year ... I'm actually surprised.
75 comments
[ 3.1 ms ] story [ 146 ms ] threadWhat needs to happen for IPv6 to take off? CloudFlare is doing a lot by allowing automatic IPv6 for many websites - it makes it very easy to get a foothold in IPv6.
1. Consumer grade routing hardware that always supports ipv6 out of the box.
2. ISPs transitioning into supplying both an ipv4 and ipv6 address to households.
3. Major network backbone services (google, amazon, etc) transition to both ipv4 and ipv6.
4. Once a majority of the critical network infrastructure is available on both, ISPs need to start dropping ipv4 addresses from customers, preferrably from new customers.
5. Slowly depreciate ipv4 by using a ISP provided ipv4 tunnel service for a while for those that still need to access the ipv4 web.
Basically, it requires most of the work be done on the backs of ISPs. In the US at least, the ISP business is an oligopoly of cable tv and Hollywood and will resist ipv6 because it enables more peer to peer trafficking and open communication over the Internet.
I actually think that the content hosting folks are behind the curve here. Amazon is one of the biggest hold-outs; their IPv6 rollout has thus far been a total joke. No IPv6 for Cloudfront? No IPv6 glue for Route53 authoritative nameservers? No IPv6 on VPC, barely on EC2 at all, etc., etc. From what I've heard, the management at Amazon doesn't really care about the network behind their infrastructure; instead of investing in building a backbone that could do a better job supporting this kind of stuff -- and all kinds of inter-region applications which would be useful -- they want pretend the network doesn't exist and that nothing needs to change. It's too bad. I think we'd see more IPv6 usage if Amazon would get their act together.
I switched to a manual tunnel from HE (tunnelbroker.net). I confirmed this worked, that my Apple devices behind the Airport Express all got IPv6 addresses and so forth, and that the performance was decent.
Experience in hand, I then tore it all down, since I couldn't think of a good reason why I really needed IPv6 at home. :-)
My hope is that by 2020, IP will have approximately the same importance to the global connectivity fabric that Ethernet has today; it'll be a particularly flexible next-hop technology, and we'll be routing something more clever than IP on top of it.
(Tone: Honest question. I'm curious.)
My email is in my profile.
My conclusion is that his application-level networking experience has fooled him into thinking he knows more than he does about actual network architecture, management, and performance issues. Or maybe he doesn't realize the issues even exist.
Regarding overlay networks: note that HTTP, WebSocket, Skype, BitTorrent and such are already looking quite a bit like an overlay network, and that increasing amounts of NAT-ting are making the classic point-to-point model of the internet more and more irrelevant. I'm not sure I particularly like this vision of the future, and I agree that the performance will be worse, but it's far from insane.
(As to performance, note how everything is routed over HTTP these days, despite it being rather inefficient for lots of use cases.)
The "overlay" networks you speak of are already hamstrung by the shrinking ability to directly address endpoints on the Internet. IPv4's fundamental flaws when applied to billions of nodes cannot be papered over forever.
But you have time to make multiple complaints about tptacek not knowing what he is talking about?
May I suggest you withdraw from this line of argument?
May I suggest that you look at the paper in question and realize it is one that requires careful and uninterrupted attention for an extended period of time to properly understand?
May I also suggest that you not make assumptions as to what another person is or is not doing, and how it may interfere with their ability to do some things but not others?
Or would you rather just be a judgmental, holier-than-thou asshole?
My conclusion is that his application-level networking experience has fooled him into thinking he knows more than he does about actual network architecture, management, and performance issues. Or maybe he doesn't realize the issues even exist.
My conclusion is that you made that comment having absolutely no idea what my experience was, and then decided to get pissy when the facts didn't line up with your bet.
I've made the same dumb bet on HN a bunch of times. "This is just some Ruby dork who knows fuck all about", I don't know, "tax law". It never works out for me, either.
The thread is now full of technical commentary about the point I made. Lots of smart people disagree with me; no surprise! You've contributed none of that commentary. Maybe now's a good time to start? You seem to feel passionately about the issue.
I made the comment based on the fact that the only thing you ever talk about actually doing is security-related. On everything else, you just pontificate context-free.
You implied earlier I should have looked at your "public linkedin profile". You should know that, for me, your profile contains nothing but your current and past employers, and your university. Nothing about what you did there. What was I supposed to conclude from that?
I've spent most of my career working for companies that do video-related work, but if you ask me a question about MPEG transport streams, I'll refer you to the guy who, y'know, actually deals with MPEG.
At this point I am pissy, yes, but that's not because I was wrong, it's because of the asshole who decided he knows better what my personal time commitments are than I do.
> The thread is now full of technical commentary about the point I made.
No, it isn't. The thread is full of cursory rejections of your equally cursory commentary. There is absolutely nothing interesting about any of it, because it's all surface bullshit that everybody already knows. There are no details. There's nothing concrete. There's nothing new. Boring, hand-wavy crap.
Propose something concrete, something that actually addresses the obvious criticisms you already know are going to be leveled, and I might get interested.
Right now, all I'm interested in is ignoring HN for a while.
Have a good weekend. Ignoring HN can definitely be a healthy thing sometimes.
which is a bad thing.
It would be less of a bad thing if IPv4 connectivity (specifically: what ASN was advertising the IPv4 prefixes your traffic was associated with when it traversed, say, Telecom Malaysia) wasn't the sine qua non of global connectivity, in the same sense that nobody cares what IP address you use when you connect to IRC.
Unless someone is using it already.
Then you need to choose another port.
Which you need to communicate to the other clients behind the NATs around the world. Fine, we'll just update the DNS server with an SRV record. You probably will always have a spare public IPv4 address to map the ports 53 to something on the inside network, so this might work.
Until your CGN does not have enough IPv4 addresses/ports to permanently allocate one to you. Oh, by the way this is a great way to introduce a "premium" tier service with a public IP address - if you really want to run a server then you can buy an address. So doing the PBX-style circuit switching with NATs is not really in the economic interest of the ISPs. (Not saying that they need to do all the accounting/logging).
If the app is awesome enough that the users will pay extra to get the public address - great.
That will help to further exhaust the IPv4 addresses and drive the prices up. Rinse, repeat.
I am of course exaggerating.
But this presentation from NANOG may be a less biased view (though Lee's employer does deploy IPv6, so you can argue it is biased):
http://www.nanog.org/sites/default/files/wed.general.howard....
Full disclosure: I am into this IPv6 thing. Before that, I spent ~10 years of my life at a random 5-letter vendor to find the bugs in the code and misconfigurations causing outages in people's NATs. It's refreshing to be able to use access-lists again. (the latter statement is only partially correct, but this is a topic for another rant, IPv6 has tons of its warts as well :-)
(I am actually opposed to overlays uber alles, partly because I think it may actually happen.)
A --- |> --- <| --- B
then getting from "A" to "B" is not easy. And the "cones" may be of varying degree of aggressiveness when it comes to filtering [1]. You can not guarantee that you can always establish connectivity between the two, even in a presence of the third party - with the addresses being more and more precious, the SPs will move from full-cone to more restricted NAPTs.
Besides these hurdles, moving large volumes of traffic atop overlays is not very optimal - I heard anecdotes that when Tor is abused in this manner, it is quite slow. Skype (and I guess probably Tor as well) uses supernodes that are not behind the NAPT to establish the communication. As there is less and less places for it.
So, the quality of communication over overlay degrades more and more.
Meanwhile, the bigger players will just place the caches close to eyeballs or directly peer with the eyeball providers. [2]
So, I think Lee's scenario is quite plausible.
Pretty much the only things keeping holding IPv4 in my home network are the client device bugs [3] and Skype. The web works quite well on DNS64.
I think the economics will take care of things. There are users on IPv6. A few clever interns last year made a website [4] that aggregates various IPv6-related stats. Take a look at the situation with users e.g. in Germany, Belgium, Switzerland. Yes, it is still not a whole lot but it is growing - and these three give the impression of how it might happen - sometimes steady, sometimes the switch will just turn on.
I am very curious to hear some VC chime in on this matter - what do they think about the prospects of business growth in the situation of the constrained address space and uneven competition. It is not the case today because ARIN still has numbers. Give it a couple of years.
[1] https://en.wikipedia.org/wiki/Network_address_translation [2] http://blogs.broughturner.com/2009/04/googles-peering-and-ca... [3] https://ripe66.ripe.net/archives/video/1196/ [4] http://6lab.cisco.com/
I think a lot of crappy 1980s protocol architecture is based on a fear of forwarding packets that (to invoke a shorthand) Moore's Law has addressed for us.
Then I took a job that required me to write the clientside for every protocol exposed on a "modern network" circa 1996, including FTP, TFTP, NFS, SunRPC, DNS (for which I had to write 3 different recursors and an authority server to do serverside-serverside testing), RIP(!), and lord knows what else. Our product was a network security scanner, before there was Nessus, that worked by actually testing for bugs instead of doing banner grabs.
I moved to a research role in that job and built a testsuite for network intrusion detection systems based on a line-by-line audit the 4.4BSD IP stack, then used it to reverse engineer the behaviors of the Windows NT and Solaris (then closed-source) IP stacks, and used that to discover (with my then-partner Tim Newsham) 3 new classes of attacks/bugs on network observation tools. Incidentally: I did that research because I have an ethical problem with unchecked network surveillance. Half this site seems to think I'm a secret government agent.
Then I started, with two friends, a chat company that pivoted into multicast content delivery. My role on the dev team was the routing system, which implemented a multicast link-state routing protocol and a multicast service model comparable to PIM; I was also the project manager for our global traffic management system, which took full BGP4 feeds from the networks of friends of mine that ran backbones, and used them to compute anycast DNS results.
When that company failed because our VCs and the CEO they installed decided we should go head to head with Akamai and invest all our money in global colocation fees, I thought long and hard about where to go next and decided I didn't yet know enough about networking.
So I took at a job as the team lead for Arbor Networks Peakflow DOS, an anti-DOS product now deployed on the backbones of to a first approximation all the Internets. Peakflow worked by consuming NetFlow records from the backbone Cats and GSRs of companies like AT&T; ie, it binned and computed stats for a large percentage of all the packets traversing the--- what would you call it? Yeah, the Internet.
Your call whether my career experience as a network designer, a network operator, a provider of network management and performance monitoring tools, and a kernel network software developer is germane to the discussion.
Have we dispensed with the part of this thread where we wonder whether I have any business having ideas about how the Internet might work, and move on to the part where you actually try to shoot down those ideas? The former conversation is irritating, because I'm re-explaining things you could have found out simply by reading my public LinkedIn profile. But I would love to be proven wrong on my ideas; that would involve me actually learning something.
I think you now owe me your best shot at taking down the actual notion of a next-gen Internet built as an application-layer overlay.
And I'd be happy to shoot down your ideas, if you had any. All you have is your regular content-free rant against IPv6 that you constantly re-post.
> Half this site seems to think I'm a secret government agent.
No, we just think you're a strict authoritarian who gives the American left a bad name.
https://news.ycombinator.com/item?id=5587264
https://news.ycombinator.com/item?id=3408523
All you've said so far is "Boo! New ideas bad!"
But neither thread is one of substance, only complaints and a vague "we need a general overlay network". Not one word about what it should look like. How it should work. Nothing.
If you want to argue about an idea, articulate one.
Edit: I see in the second thread you actually did say something about a directory service, so I guess that is one word. But as far as I can tell, you just described yet-another-DNS-replacement. If I were generous, maybe I would call it an Active Directory competitor.
Boo! You continue to say. New ideas bad! How dare you, of all people, assault the old ideas!
Edit: Ah, you've edited it to clarify. You're just going to be your usual self. How disappointing. Broadly speaking, you're an interesting, intelligent person I would love to know better and to learn from. But you can't stop yourself from being the get-off-my-lawn guy.
When I talk about next-hop transport, I mean, "the mechanism by which one node might connect to one or more other nodes in the topology". As soon as those hops form a connected graph --- a very easy problem to solve when your connection fabric is datagram delivery on IP --- routing over the topology is a Solved Problem. Think of what I mean by "next hop transport" as "the lowest common denominator of connectivity that allows any 2 hosts in the world, possibly by means of some number of intermediate hops, to talk to each other".
Then, to be clear about what I'm saying --- though this is probably not how you'd design such a system in the real word, imagine we did this:
* Stopped fussing about IPv6 (and IP multicast, although we've already stopped fussing about that).
* Took IRC
* Made it 8 bit clean and capable of delivering much larger messages.
* Gave it a real routing protocol so that it could express biconnected graphs.
* Built applications on top of that.
That system would provide a service model that would be compatible with a superset of all Internet applications today --- it would allow point-to-multipoint applications along with point-to-point applications, it would admit to client-server, it could express caching and CDN topologies, and it would admit to logical addressing that would enable you to attach to lots of different endpoints on the network with the same, static address, in the same way as I can talk to you now on HN as "tptacek" despite the fact that my IP address is temporarily owned by GoGo Inflight.
I did not intend for it to explode like that... but I'm on the Internet, so I'm not surprised.
I think I also see where you're going with this; we've got so many adhoc implementations of this already (I'll toss on XMPP as another example) that some sort of eventual agreement and standardization seems inevitable. Don't know where it'll come from or what it will look like, but Historical Imperative is calling for it.
I know you meant to say, "Failed experiment for consumer web connectivity" - but, I think it's important to remember that IPv6 has some very successful niches. Mesh networks, Smart Grids, anything with large scale flexible address and routing requirements has been using IPv6 very successfully for at least 5+ years now.
The only real competition IPv6 has over the next 20 years for global routing dominance though, is, ironically, IPv4.
The main reason is that all the Autonomous Systems have to have some common language to communicate, and share routing updates, and, for better or worse, IP is the "English" of network engineers when it comes to internetwork routing.
Frame Relay is a good example of what this design looks like: a "layer 2" technology with "layer 3" features (Frame had aspirations to be a layer 3 technology) that only network operators care about, because no Frame user ever expected global all-points connectivity to be provided by Frame.
IPvX should be demoted to Frame Relay status: a very flexible way of providing point-to-point connectivity between nodes that will provide global connectivity through routing done at a higher layer.
It looks like you went down the front page desperately looking for any non-NSA related articles to comment on, but rest assured the rest of us have been reading them and your boy Obama and his NSA have just blown the SaaS web startup industry out of the water. No web startup has a chance in hell of selling services to anybody outside the USA now (or anybody inside the USA with half a brain).
I'm not like, mad or anything about it; it's a self-evident kind of comment.
In other words, IPv6 adoption will accelerate by orders of magnitude once the real pain of IPv4 address exhaustion starts to be felt.
I think large open source projects (like BitTorrent) and startups should start thinking about applications that build their own connectivity layer on top of IP, and that don't assume any two nodes will have direct IP connectivity. Obviously, this is already true of the Internet we use today, because a huge percentage of hosts don't expose direct connectivity for reasons of security (not addressing scarcity --- there are companies sitting on /16s that static-address every desktop, but allow none of them direct Internet connectivity).
If you believe TorrentFreak, BitTorrent accounts for double digit percentage points of all Internet traffic. BitTorrent is an overlay design (albeit not one that reliably avoids the need for direct point-to-point connectivity, unlike the Gnutella overlay). We already know designs like this are tenable. I'm saying we should explore them further as alternatives to an Internet design controlled by giant telecoms.
I'm saying it would be great if all web pages were delivered over an overlay protocol running on top of some TCP friendly transport provided by an overlay; a "peer to peer" routing system like Skype.
The simplest way to describe the architecture I'm talking about is, "what if you made IRC a modern, fully routed binary connectivity fabric, then tunnelled the Internet over it?"
I'm not saying we get rid of IP (I'm not talking about TCP at all, for what it's worth).
I'm saying, we stop trying to make IP better, accept it for what it is, and building something better ON TOP OF IT.
But don't get me wrong: I'm not on a crusade to get rid of IPv6, and overlays work just fine on IPv6. I'm just on a crusade to put it into context.
It's probably also not a coincidence that Skype and Bittorrent are two of the hardest protocols to 'control' inside the network. This sounds good from a 'the internet should be a free democracy' standpoint but bad from a 'let's guarantee the performance of the application' standpoint. In terms of visibility into what is happening, being transferred, and where connections terminate there are more productive rather than draconian reasons for understanding those things.
Today with virtual hosting if the website next to you decides to host illegal content, the entire IP is taken down collateral damages or not. Well, that's part of your choice in choosing a provider as an end host. If the entire next generation internet and application space is based on overlays and other nebulous architectures, much of those same problems are going to exist to larger scales, including at transit points. Except it won't be as optional as changing hosting providers.
Since you've got BGP experience - think about the fun in troubleshooting eBGP neighbor problems when every connection is a GRE tunnel that may actually be tunneling through another BGP AS, which itself may be tunneled again. GRE tunnels are great immediate bandaids and generally awful long term solutions for enterprise networks to say the least. But the application guy gets his green light and thinks everything is great.
I think we don't know enough yet about how to build reliable network topologies on top of reliable datagram delivery to have immediate answers to all of these issues.
I think one baby-step answer is to say, let's start figuring out how to do this on an application-by-application basis. But then, we're already saying that; that's how Skype worked, and Skype was (a) ambitious and (b) worked!
So then:
* I don't think things need to boil down to a single head. IRC doesn't. I remember IRC working even back during the NSFNet split in the mid-90's, when my ISP (Ripco) lost connectivity to university sites. That's because IRC is a mesh of lots of head-end services that leaf nodes rendezvous with.
* Skype and BT are hard to control because they are deliberately designed to be hard to control, because they're P2P (in the late-90's sense) schemes. Their deployment priority was to work in spite of network policy controls, not with them. But that doesn't need to be true of a next-gen system. Can you think of ideas to make protocols easier to moderate and make policy decisions about? I can, but wouldn't want to belabor the point.
* It's definitely true that running a dynamic routing protocol on top of a transport routed by its own dynamic routing protocol is a recipe for byzantine failure. But mitigating that issue is the fact that routing on top of a well-connected (but not completely-connected) transport is a simpler problem than traditional routing protocols handle; I remember feeling silly going through all the contortions OSPF goes through to carefully forward link state from hop to hop, knowing all along I could just be pulling the whole LSA table from a central database.
(I think we could make the Internet work better by throwing some standards out the window. Not all of them.)
If you want to throw out TCP/IP, I hope you have specific dramatic improvements for its replacement in mind.
I have a bunch of problems with IPv6:
* IPv6 doesn't solve the multicast problem; multicast was attempted in an IP context and failed because when you try to express something as complex as group message delivery on IP, you needlessly complicate things, and because multicast on IP simply doesn't scale, because it conceptually requires core routers to maintain routing state for individual pieces of content.
* The topology of IPv6 is controlled by giant telecoms; also, some of what you are and aren't allowed to do with the connectivity you buy is controlled by telecoms. I don't have a moral problem with this, but it's a needless coupling between Internet and telephony businesses.
* IPv6 doesn't move us from our current state of "by default I don't have direct peer level connectivity with hosts", because in the IPv6 world, everyone will still be hiding behind firewalls, because for 2 decades the crappy software we had hanging out on TCP and UDP ports turned out to be too unsafe to expose. Overlays that would provide direct connectivity --- in the same sense as I can IM you on AIM or Gtalk right now --- would be starting fresh and would provide default direct connectivity to everyone. Anyone who's ever had to punch a hole through a firewall to allow inbound connectivity understands this.
* IPv6 is tremendously expensive to deploy: http://cr.yp.to/djbdns/ipv6mess.html --- forget the routers, we have so much software that has to get forklifted out to accomodate addresses that don't act like scalar integers.
No thanks.
I would want to build on top of something stable, not some proprietary standard that could change next year.
* The point of overlays is to moot the question of what IP connectivity you have; to provide "the full Internet" to people who only have NAT IPv4.
* Whether you build it on IPv6 or on IPv4, an overlay is in some sense "proprietary".
* But who's to say the most successful overlays won't be based on open source projects?
Still , I struggle to imagine how such a network might work. Things like Kademlia seem inefficient and inflexible for general use.
My ISP has the technology and infrastructure in place to make sure that when I connect to 184.172.10.74 it actually goes to the correct place. They seem ideally suited for this because they have the physical wires.
How could I be sure I was talking to the right endpoint on an overlay network? PKI?
Do I have to choose an overlay provider to route my packets around?
Unfortunately getting it to run involved sacrificing at least a dozen newborn animals - which is maybe the reason it never went up.
Also, of course the mail delivery took tens of minutes even in a trivial ring of 4 machines connected on the local LAN. Maybe I had something wrong in the setup, though.
The great thing about overlay networks is you don't need anyone's permission to build them.
All you need is an app that really benefits from the advantages of an overlay network, some of which include:
* Point-to-multipoint or multipoint-multipoint delivery (ie, multicast)
* Logical addressing decoupled from network topology
* All points rendezvous between nodes that don't have inbound connectivity because of firewalls
I agree with you that it's unlikely anyone will be successful building this as a product that is simply "new Internet"; what I think is more likely is that people will starting having more and more Skype-like successes by ignoring the "rules" of IPv4 connectivity and routing around them in the "application layer"† like the brain damage they are.
† ... which is a stupid term that blinds people to how much power we have as developers
> Not impressive.
Perhaps not impressive, but pretty much as-expected. RFC 2460 was finalized December 1998. I'm not sure why a 20-year +/- deployment process should surprise anyone for what is, ultimately, a telecom standard, one built before it was needed because we knew it would be.
Remember, APNIC is the first to hit its last /8 and only did so in 2011, and due to their exhaustion policy, hasn't exhausted that last /8 yet. Other regions are in better shape.
As we've always known, the big push will come when the IPv4 supply dwindles so low that it becomes reasonably impossible to accomodate new endpoints.