That was an interesting read. Perhaps I'm misunderstanding, but doesn't this mean a rogue ISP could take down a large chunk of the internet if they wanted to? If Moratel hadn't fixed the issue at their end, would Google have stayed down indefinitely?
Perhaps a blog post on exactly how that is done. And after what particular time would that happen and is that something that needs to be coordinated or??
If there is a rogue actor on the internet, even at a large scale, the rest of the Internet blackholes their traffic, and treats them as a failure point. Various routing tables are updated across the internet to not consider them a viable destination or source, and traffic normalizes without them connected.
It isn't the action of one central authority, more a natural response to failure by separate parts of the internet.Its akin to shoppers and suppliers refusing to go to a particular store anymore. No one shuts the store down, but it no longer has products or customers.
Yes, a rogue BGP talker could take down large chunk of the internet, except that it would only be for an hour or so at most, before all the other BGP admins blacklisted the rogue BGP talker and simply completely disconnected the rogue BGP talker.
The BGP admins are those that are assigned that task by whatever company or organisation owns the AS they are responsible for.
There is no central authority, most people that are interested in that sort of stuff are on various mailing lists, the main one being NANOG.
Generally for larger ISP's they will have direct contacts with their counterpart on their BGP neighbour where the handoff from network to network exists, for smaller entities (for example a small business with two uplinks that announces their own /24) they may have a support contact that they can call about issues.
Is it a risk? Maybe. The Internet and BGP is built on mutual respect. You can and sometimes will filter certain routes from certain uplinks/providers to change how routing happens from your network onto the next hop, but overall there is no authentication, and no authorization.
Anyone that has an AS, and has a peer that is willing to accept their routes, can advertise whatever they want.
As for how many there are? How many CCNA/CCNP/CCIE's are there out there? How about comparable certs for Juniper/Alcatel Lucent and whatnot. Each of them could potentially be a BGP admin. It's not a difficult job. When I worked at a large ISP I remember announcing some new IP's out from our AS to the world from a new location and watching it propagate across the world. Had me and my colleague made one little typo we could have accidentally announced the wrong thing.
The only controls are policies that are set up by each AS for what routes they will or will not accept from a neighbouring AS. In general this is an accept all. Then the AS can choose to re-advertise it's received routes from it's neighbours to it's other neighbours, this is how route propagation works. You can also choose for example to receive routes from a neighbour but not re-advertise them (i.e. you don't become transit).
BGP changes happen every minute of every day. Routes get added, get removed, and those things propagate. This is not the first time nor the last time that a mistake happens, or that someone advertises an address space they shouldn't be advertising. Look at what happened when China accidentally leaked routes that took YouTube and Google off-line for a bit, or the BGP issues with Syria advertising everything in an attempt to take the internet offline in their country...
It is an admin for each AS. Each AS that has a direct connection to the abusing AS would need to put a stop to it, otherwise risk being seen as complicit themselves. Hops further down could blacklist an AS as well.
Large ISP's may have multiple people that can affect these policies.
Senior network/routing/noc staff at the top 5-10 backbones mostly. They move most/all bits at some points. For 2014 that was Level 3, Telia Sonera, Cogent, GTT, Tata, Verizon, Sprint and a few others.
One or more of these networks likely provides transit to the errand AS or the errand AS' upstream and they can lean on them pretty hard. Doing so will be in full compliance to whatever transit agreement exists between them as well. No need to be big brother about it, it's plain self-interest. They have SLAs to meet.
It happens when a provider advertises their internal routes to the world.
It boils down to how BGP works where networks (called Autonomous Systems) broadcast routes to all their peers (equal peers, customers, upstream providers, all peers.) These routes essentially say "for the IP prefix xyz.0.0.0/8, the AS 1234 is origin and you can get there in X hops through me".
Problems happen when an AS broadcasts to one or more of their peers routes they don't originate or provide transit for.
Network A might get a route to CloudFlare from 2 providers, one which is short and one that is long. If it accidentally tells the provider with the long route that it is a gateway toward the short route -- and if that provider believes it! -- then the provider with the long route will switch some or even all their traffic to go through network A and they might tell their own peers and so on.
Then suddenly half of cloudflare's traffic goes through Qatar or dies trying...
Anyone else experiencing issues with CF more frequently lately? We appreciate the service provided, but we're averaging about 1 customer-impacting issue per week over the last few months. And we are considering alternatives. Today's incident was particularly bad, and you'd have no idea that NA was impacted from the status page.
I've been with them for two years or so and in the beginning I had a streak of problems but actually the last year has been rock solid. Today's problem did take two of our large sites offline for ten to twenty minutes, though others were fine.
Fundamental flaw in how BGP was designed from the old days. Give me a transit or peering connection where filters aren't in place, and you can announce anyone's IP blocks with their AS (SS7 is very similar, as it was built for a closed telcom ecosystem where all participants were trusted: https://www.youtube.com/watch?v=lQ0I5tl0YLY).
There are efforts to move to a more secure version of BGP, but that'll be done around the same time the transition to IPv6 is complete.
The problem is these kinds of leaks were more local except for services which have a large internal network. Yes - Google has that problem, but it is naive to assume that more of these won't happen. Any service which has a super easy path to having all their routing tables polluted so quickly is more vulnerable than others.
40 comments
[ 2.8 ms ] story [ 91.8 ms ] threadIf you are interested in knowing about route leaks affecting different companies I recommend: https://twitter.com/bgpmon
Perhaps a blog post on exactly how that is done. And after what particular time would that happen and is that something that needs to be coordinated or??
It isn't the action of one central authority, more a natural response to failure by separate parts of the internet.Its akin to shoppers and suppliers refusing to go to a particular store anymore. No one shuts the store down, but it no longer has products or customers.
Just learning about this for the first time.
Similar to my question. Who are they, how many are there, how do they communicate, is the fact that they have this power in itself some kind of risk?
There is no central authority, most people that are interested in that sort of stuff are on various mailing lists, the main one being NANOG.
Generally for larger ISP's they will have direct contacts with their counterpart on their BGP neighbour where the handoff from network to network exists, for smaller entities (for example a small business with two uplinks that announces their own /24) they may have a support contact that they can call about issues.
Is it a risk? Maybe. The Internet and BGP is built on mutual respect. You can and sometimes will filter certain routes from certain uplinks/providers to change how routing happens from your network onto the next hop, but overall there is no authentication, and no authorization.
Anyone that has an AS, and has a peer that is willing to accept their routes, can advertise whatever they want.
As for how many there are? How many CCNA/CCNP/CCIE's are there out there? How about comparable certs for Juniper/Alcatel Lucent and whatnot. Each of them could potentially be a BGP admin. It's not a difficult job. When I worked at a large ISP I remember announcing some new IP's out from our AS to the world from a new location and watching it propagate across the world. Had me and my colleague made one little typo we could have accidentally announced the wrong thing.
The only controls are policies that are set up by each AS for what routes they will or will not accept from a neighbouring AS. In general this is an accept all. Then the AS can choose to re-advertise it's received routes from it's neighbours to it's other neighbours, this is how route propagation works. You can also choose for example to receive routes from a neighbour but not re-advertise them (i.e. you don't become transit).
BGP changes happen every minute of every day. Routes get added, get removed, and those things propagate. This is not the first time nor the last time that a mistake happens, or that someone advertises an address space they shouldn't be advertising. Look at what happened when China accidentally leaked routes that took YouTube and Google off-line for a bit, or the BGP issues with Syria advertising everything in an attempt to take the internet offline in their country...
They arrange with each other to connect their networks, sometimes payed, sometimes with no money exchanged.
These admins are not self-appointed admins over the 'internet', but they just maintain their own networks.
Large ISP's may have multiple people that can affect these policies.
Source: http://research.dyn.com/2015/02/bakers-dozen-2014-edition/
One or more of these networks likely provides transit to the errand AS or the errand AS' upstream and they can lean on them pretty hard. Doing so will be in full compliance to whatever transit agreement exists between them as well. No need to be big brother about it, it's plain self-interest. They have SLAs to meet.
It boils down to how BGP works where networks (called Autonomous Systems) broadcast routes to all their peers (equal peers, customers, upstream providers, all peers.) These routes essentially say "for the IP prefix xyz.0.0.0/8, the AS 1234 is origin and you can get there in X hops through me".
Problems happen when an AS broadcasts to one or more of their peers routes they don't originate or provide transit for.
Network A might get a route to CloudFlare from 2 providers, one which is short and one that is long. If it accidentally tells the provider with the long route that it is a gateway toward the short route -- and if that provider believes it! -- then the provider with the long route will switch some or even all their traffic to go through network A and they might tell their own peers and so on.
Then suddenly half of cloudflare's traffic goes through Qatar or dies trying...
Route leaks aren't really something they can defend against anymore than other ISPs until we get a better system in place.
Note the 10 references to qatar.net.qa. Hacker News uses CloudFlare so a traceroute from my current location was going to qatar.net.qa as well.
There are efforts to move to a more secure version of BGP, but that'll be done around the same time the transition to IPv6 is complete.
[1] https://twitter.com/feedly/status/480809578368487425