Here you go - it might not help much, this is a pretty specialized field of network engineering. If you were in a service provider, you were likely aware of this tech otherwise probably not at all.
LDP gave us BGP-free cores. Don't underestimate how much that changed the game in large networks. BGP converged faster and core routers didn't need to do anywhere near as much. SR is a better way to do that now and LDP/IGP sync was a huge pain, but IMO BGP-free cores changed the SP game as much as route reflectors did.
LDP also enabled L2VPN and was an IGP-bound mechanism to enable MPLS VPN, which was always the killer app for MPLS.
LDP by itself? Meh. But it opened up a whole lot of other possibilities.
We have just performed basic delay performance comparison with MPLS, SR-MPLS, and SRv6 in a small network case study with Cisco Modeling Lab (CML) and the initial results are very promising. SRv6 has about 48% improvement over original MPLS and 10% improvement over SR-MPLS. This delay improvement mainly due to the lack of LDP in Segment Routing approach. Perhaps later we can check other performance metrics as well and without LDP the node processing overheads will also be much better as well compared to the original MPLS. Scaling to the size of the global internet should further increase the improvement margins even more.
I always broke MPLS-TE into two categories - traffic engineering for bandwidth, and fast reroute (FRR).
Bandwidth engineering, like Bruce said, was religion for some people and anathema for others. It saved at least one company from bankruptcy, letting them run the network they had much hotter than IP would have.
FRR, though, was an absolute game-changer. Local protection to failure and sub-50ms (often sub-10ms) protection switching in an IP network? Immensely valuable. Link connectivity protection was easy to deploy and I will argue made a massive difference in an Internet network's ability to provide real-time services in the face of failure. Node protection and bandwidth protection? Less so. But MPLS-TE-based FRR was the first local protection mechanism that didn't require you to have a specific topology and which could cover 100% of all connectivity failures.
I'm surprised Bruce didn't call out this distinction, because they really are two separate things using the same underlying tech.
There is no doubt FRR had value, but like all tech, there was a complexity that came with it, so it really only made sense for certain applications. Heavy voice/video networks were a no brainer, but other TCP-based applications that were less sensitive were of less value, esp. as IGP convergence improved.
Yeah, for sure. I'm not saying FRR was always the right answer for all questions, but for a good decade or more it was a pretty solid choice, and particularly with auto-tunnel, it was easy to do. RSVP was far too complex a hammer for just FRR, and I like SR-LDP for FRR now, but from maybe 1998 (?) to 2015 (?) MPLS-TE FRR was the best tool we had.
Saving it for another post. It is actually just as controversial as there was a lot of work to improve convergence times in standard IP routing at the same time.
(Though, after 3 years of QA'ing the stuff, I gave up on MPLS and moved to other technologies. I was just not able to understand it to the deep level I wanted to.)
Same here. My wife was on a citizens' traffic commission that approved some roundabouts there a while back, and I was hoping I could show her some interesting stuff.
There is some extremely nice 2018 research paper on new routing paradigms that seems to be blowing out older stuff, and that is only possible because MPLS exists. The NSDI paper is linked below.
It's always a good time to re-read the Saltzer, Reed, and Clark End-to-End Arguments in System Design paper. "A too-real example" triggers a certain degree of paranoia.
I had always assumed MPLS (and the even more hideous, SRv6) were designed by software engineers. If you are designing a hardware network processor, traversing a variable stack of labels at line-rate is pretty horrible.
Eh? Each hop only looked at the outermost label, keeping it, doing a push or a pop. The label namespace is small, which made it possible to use it directly as an index into a small SRAM to do a lookup and determine what the next hop is. That's how MPLS made line rate possible at a time when there wasn't enough space on silicon to do full IPv4 route lookups. It was very much a design driven by hardware requirements for how to do hardware line rate forwarding with limited resources.
Correct, the (dumb, cheap) devices within the network have an easy time. The devices on the edge can have a lot of work to do, potentially popping multiple labels. And then there's a bunch of meta labels that have other meanings (I'm not actually sure if those are standardised or not).
Edge routers already have to cope with adding and removing arbitrary sized headers (just look at tunneling protocols like IPIP, L2TP or the more modern VXLAN), so it's not like MPLS with multiple labels is a new requirement for the data path. And it's only at the originating site that you have to push multiple labels onto a packet -- the receiving end should only have to pop at most 1 label off as the transport network would consume the rest. I had to implement a software MLPS data path once upon a time maybe a decade ago.
I've also written some Verilog code for adding and removing headers on packets on FPGAs. Once you have to support VLANs in hardware, you're already half way towards inserting an arbitrary number of bytes into a packet as higher data rates in FPGAs mean wide busses with all the corner cases of packets being aligned at various offsets into a 256 bit or 512 bit data path. Figuring out how to implement an ethernet CRC32 block that meets timing in a Kintex 7 was certainly fun!
MPLS was originally designed to be simple and fast. As you (and another) have commented, that was to help hardware. But it's grown into something that can be more complicated and even the P switches may read further down the stack to find hash labels etc (but yes, still only lookup using the top label). One benefit was to avoid using CAM but that's getting harder to realise.
We don't have just switching labels and these other types may not be consumed, leaving the egress PE popping multiple labels. This is fine and the datapath is designed to make these sorts of packet manipulations, it's just harder to do that at full speed as people get further from the nice, simple original MPLS cases. When using multiple layers of encapsulation, customers don't make the same performance assumptions.
Things are worse with SRv6 where the need to read much further into packets is approaching the pain point. Most (all?) network processors have a limit of what you can manipulate at full speed. There's another painful case but I don't think I can discuss that.
It's the oposite, trying to route or switch on variable length prefixes(as you would with IPv4 or V6), is pretty horrible to do in hardware, thanks to CAM it's atleast doeable. Route or switch directly on MAC addresses is ok, but creating policies becomes painful as there's no hierarchy.
With MPLS, you get
simple fixed length headers with single integer values is quite easy to push/pop and look up in hardware, and you can have software to generate the policies and manage hierarchies, placing the complexities and hard work at the MPLS edges, and having simple/dumb (aka. cheap) devices in the core.
Andrej Karpathy, an AI/ML researcher who does a lot of education and public outreach. I don’t know the answer to GP though. I dont even see a Karpathy mentioned anywhere in context of this article.
There's no Karpathy in MPLS(???) Did you mean Kireeti Kompella? His name is synonymous with BGP L2VPN (as opposed to Luca Martini, whose name is synonymous with LDP L2VPN.)
Also Andrej Karpathy was a schoolkid when MPLS was developed ;)
As always, Bruce raises a good discussion here, but I’m disappointed in the lack of depth of this analysis. The article, to me, characterises this as a religious discussion, choosing between simple ECMP/multipath and MPLS-TE. I think this ignores the business reality of why one looks at deploying traffic engineering within a network, and the available approaches. I’m also a little disappointed that it characterises Google’s B4 and Microsoft’s SWAN as networks that rely on RSVP-TE (to my knowledge they do not, see the B4 paper). To my mind, these are demonstrations of the utility of traffic engineering independently of the mess that distributed traffic engineering with RSVP-TE creates.
(My background: I’ve inherited RSVP-TE deployments in a number of continent-wide, and global networks — which has involved driving standards to improve its scalability, and subsequently driving segment routing in the industry and production deployments.)
The issue one has at any kind of scale is that it is non-trivial to acquire capacity that can be coherent in terms of different optimisation dimensions for your network. For example, a network I worked on could acquire limited capacity on Europe to India cable systems, alternate routes were significantly different latency - but there was significant EU-India demand in the network. What were the options for placing this traffic on the network? IGP weights - sure - but this means there is no selective placement of that traffic (i.e., everyone has to take the same route), which one might not be able to support commercially. Looking beyond that, there are limited options _other_ than MPLS-TE based on RSVP-TE. Path Computation Engine (PCE) support, even when it emerged, was RSVP-TE-centric. So, commercially, those networks didn’t have a choice to deploy traffic engineering — and it wasn’t for want of trying. Significant cable system deployments have been driven on routes such as the one that I mention above — so there was capital to be deployed to fix the problem, it’s just that building such systems take years to be built. Did their architects want to deploy RSVP-TE? Pre-SDN (and SDN-in-the-WAN like B4 and SWAN) what option did they have to meet that business requirement? I would postulate very little (at least at the time that I was engaged in these discussions there were no clear alternatives). In fact, I would postulate that the existence of TE in B4 and SWAN shows that there is value in traffic engineering practically. Greenfield/ground-up systems still implemented.
RSVP-TE itself though was not well thought through. The systems design discussion that I think is very interesting here is considering the lessons that we can learn from such a technology. Distributed state in the network, that causes large amounts of signalling following failures, and requires midpoints to be aware of all demands that traverse them and admit them is fragile by its very nature. The scaling analysis that was done during the architectural work (RFC5439 for example) did not think of the RSVP-TE distributed system’s different points of dynamism — it concentrated on steady-state cost, but we’ve demonstrated time and again in production (over many, many years) that practically the system’s scaling was to do with the cost and scaling of dynamic resignalling following events rather than steady-state utilisation (I’ve presented effusively on this, see https://research.google/pubs/pub45800/ and https://youtu.be/NtED7CUHLNE).
Rather than raising the question, from a systems design perspective, as to whether MPLS-TE was a religious mistake — let’s raise the one around how complex distributed systems that have multiple vendors of their equipment (i.e., not ecosystems that are controlled by one party) can iterate on solving business problems without religiously fill...
> I’m also a little disappointed that it characterises Google’s B4 and Microsoft’s SWAN as networks that rely on RSVP-TE (to my knowledge they do not, see the B4 paper).
I work for one of the vendors that makes the boxes that run B4 and SWAN. You are absolutely right.
Symmetrical paths with hard bandwidth control. I don't see how this can be achieved without on-path signaling like RSVP. Sure, in theory, we could do something similar with a PCEP centralized controller and SR extensions, but it is extremely difficult to tightly manage the bandwidth for all services from the controller. There will always be some inconsistency.
While we can argue that this is a niche use case, it would still be interesting to hear of any instances where this type of service is addressed using SR.
32 comments
[ 4.6 ms ] story [ 73.5 ms ] threador if you'd prefer wiki - https://en.wikipedia.org/wiki/Multiprotocol_Label_Switching
From my head:
LDP = Label Distribution Protocol
RSVP = ReSource Reservation Protocol
PCE = Path Computation Element
BGP = Border Gateway Protocol
L2VPN = Layer 2 Virtual Private Network
IGP = Interior Gateway Protocol
MPLS = Multi Protocol Label Switching
TE = Traffic Engineering
FRR = Fast ReRoute
SR = Segment Routing
SP = Service Provider
LDP also enabled L2VPN and was an IGP-bound mechanism to enable MPLS VPN, which was always the killer app for MPLS.
LDP by itself? Meh. But it opened up a whole lot of other possibilities.
running a PCE also adds a massive failure domain compared to the decentralised nature of RSVP.
Does RSVP have a million bells and whistles? yes. Segment routing also has a billion different bells and whistles, and those are far less mature.
Bandwidth engineering, like Bruce said, was religion for some people and anathema for others. It saved at least one company from bankruptcy, letting them run the network they had much hotter than IP would have.
FRR, though, was an absolute game-changer. Local protection to failure and sub-50ms (often sub-10ms) protection switching in an IP network? Immensely valuable. Link connectivity protection was easy to deploy and I will argue made a massive difference in an Internet network's ability to provide real-time services in the face of failure. Node protection and bandwidth protection? Less so. But MPLS-TE-based FRR was the first local protection mechanism that didn't require you to have a specific topology and which could cover 100% of all connectivity failures.
I'm surprised Bruce didn't call out this distinction, because they really are two separate things using the same underlying tech.
(Though, after 3 years of QA'ing the stuff, I gave up on MPLS and moved to other technologies. I was just not able to understand it to the deep level I wanted to.)
https://www.usenix.org/system/files/conference/nsdi18/nsdi18...
I guess it makes sense why the article says how it is central in Google's / Microsoft's network architecture to achieve performance gains.
I've also written some Verilog code for adding and removing headers on packets on FPGAs. Once you have to support VLANs in hardware, you're already half way towards inserting an arbitrary number of bytes into a packet as higher data rates in FPGAs mean wide busses with all the corner cases of packets being aligned at various offsets into a 256 bit or 512 bit data path. Figuring out how to implement an ethernet CRC32 block that meets timing in a Kintex 7 was certainly fun!
We don't have just switching labels and these other types may not be consumed, leaving the egress PE popping multiple labels. This is fine and the datapath is designed to make these sorts of packet manipulations, it's just harder to do that at full speed as people get further from the nice, simple original MPLS cases. When using multiple layers of encapsulation, customers don't make the same performance assumptions.
Things are worse with SRv6 where the need to read much further into packets is approaching the pain point. Most (all?) network processors have a limit of what you can manipulate at full speed. There's another painful case but I don't think I can discuss that.
With MPLS, you get simple fixed length headers with single integer values is quite easy to push/pop and look up in hardware, and you can have software to generate the policies and manage hierarchies, placing the complexities and hard work at the MPLS edges, and having simple/dumb (aka. cheap) devices in the core.
Also Andrej Karpathy was a schoolkid when MPLS was developed ;)
Bruce also co-wrote one of the earliest books on MPLS with the original author of MPLS [2].
[1]Was MPLS Necessary?
https://news.ycombinator.com/item?id=32988417
[2]MPLS: Technology and Applications:
https://ieeexplore.ieee.org/document/900633
(My background: I’ve inherited RSVP-TE deployments in a number of continent-wide, and global networks — which has involved driving standards to improve its scalability, and subsequently driving segment routing in the industry and production deployments.)
The issue one has at any kind of scale is that it is non-trivial to acquire capacity that can be coherent in terms of different optimisation dimensions for your network. For example, a network I worked on could acquire limited capacity on Europe to India cable systems, alternate routes were significantly different latency - but there was significant EU-India demand in the network. What were the options for placing this traffic on the network? IGP weights - sure - but this means there is no selective placement of that traffic (i.e., everyone has to take the same route), which one might not be able to support commercially. Looking beyond that, there are limited options _other_ than MPLS-TE based on RSVP-TE. Path Computation Engine (PCE) support, even when it emerged, was RSVP-TE-centric. So, commercially, those networks didn’t have a choice to deploy traffic engineering — and it wasn’t for want of trying. Significant cable system deployments have been driven on routes such as the one that I mention above — so there was capital to be deployed to fix the problem, it’s just that building such systems take years to be built. Did their architects want to deploy RSVP-TE? Pre-SDN (and SDN-in-the-WAN like B4 and SWAN) what option did they have to meet that business requirement? I would postulate very little (at least at the time that I was engaged in these discussions there were no clear alternatives). In fact, I would postulate that the existence of TE in B4 and SWAN shows that there is value in traffic engineering practically. Greenfield/ground-up systems still implemented.
RSVP-TE itself though was not well thought through. The systems design discussion that I think is very interesting here is considering the lessons that we can learn from such a technology. Distributed state in the network, that causes large amounts of signalling following failures, and requires midpoints to be aware of all demands that traverse them and admit them is fragile by its very nature. The scaling analysis that was done during the architectural work (RFC5439 for example) did not think of the RSVP-TE distributed system’s different points of dynamism — it concentrated on steady-state cost, but we’ve demonstrated time and again in production (over many, many years) that practically the system’s scaling was to do with the cost and scaling of dynamic resignalling following events rather than steady-state utilisation (I’ve presented effusively on this, see https://research.google/pubs/pub45800/ and https://youtu.be/NtED7CUHLNE).
Rather than raising the question, from a systems design perspective, as to whether MPLS-TE was a religious mistake — let’s raise the one around how complex distributed systems that have multiple vendors of their equipment (i.e., not ecosystems that are controlled by one party) can iterate on solving business problems without religiously fill...
I work for one of the vendors that makes the boxes that run B4 and SWAN. You are absolutely right.
Throw away because I don't want be identified.
While we can argue that this is a niche use case, it would still be interesting to hear of any instances where this type of service is addressed using SR.