41 comments

[ 6.7 ms ] story [ 78.9 ms ] thread
I'm half sleepy but I liked the post. The analysis regarding path prepending really drives the accident theory home. If a state actor were trying to intercept traffic (MITM), the last thing they would do is pad the AS path multiple times because that tells the global routing table, "Don't come this way, I am the long scenic route" lol

This could be a classic fat finger config error, most likely a route map intended to manipulate traffic engineering for their own upstream links that inadvertently leaked widely because of a missing deny-all clause. Neverthless, a good reminder that BGP is still fundamentally a trust based system where a single typo in a config file can cascade globally. Never attribute to malice that which is adequately explained by a missing export filter.

(comment deleted)
(comment deleted)
The real question tho is: how would you become the mitm reserving for yourself the benefit of the doubt?
That’s a very new feeling for me. I read the entire post (with no prior knowledge of BGP at all) and I got chills from thinking how deeply intertwined US companies and the US government are.

I know this has always been the case, of course, but now I have lost trust. Whatever the reasons of this "leak" were, I am not accepting any information written in this message (search for the link to another coverage of the incident in the comments).

It is quite weird and quite logical at the same time: this is the end of an era.

If you look closely, you can see the color of the orange Cloudflare logo being slightly adjust to match a particular individual's facial color tone.
This is... hard to follow. You seem to be implying that Cloudflare is covering for USG's failed military op-sec surrounding a malicious BGP leak, and judging that this is such a bad action (on the part of Cloudflare) to undermine your trust, not only in Cloudflare, but in all companies and the US government entirely. I don't think the situation is so dire.

Cloudflare's post boils down to Hanlon's razor: a plausible benign interpretation of the facts is available, so we should give some scrutiny to accusations of malice.

Are there specific relevant facts being omitted in the article, or other factors that diminish Cloudflare's credibility? They're clearly a qualified expert in this space.

Let's assume for the sake of argument that the BGP leaks (all of them from the month of December, in fact) were the result of secret US military intelligence operations. The fact that militaries generally use cyber vulnerabilities to achieve their objectives is not news, and the US military is no exception. Keeping specific exploits secret preserves a valuable advantage over competitor states.

One could argue that Cloudflare's post helps to preserve USG's secrecy. We can't know publicly whether USG solicited the article. But even if we assume so (again assuming malice): Is Cloudflare wrong to oblige? I don't think so, but reasonable people could disagree.

Merely pointing out Hanlon's razor doesn't fundamentally change the facts of the situation. In Cloudflare's expert opinion, the facts don't necessarily implicate USG in the BGP leaks without an assumption of malice. Assuming Cloudflare is malicious without justification is just deeper belief in the conspiracy that they're arguing against.

If Cloudflare is distorting the facts, we should believe (rightly) that they're malicious. But I don't see any evidence of it.

EDIT: Clarity tweaks.

Respectfully your comment sounds like paranoid thinking.

The section of the article pointing out the AS prepending makes it really clear the route leak is a nothing Burger.

It's incredibly unlikely this leak change how any traffic was flowing, and is more indicative of a network operator with an understaffed/underskilled team. Furry evidence is that a similar leak has been appearing on and off for several weeks.

That's not to say the US government can't, doesn't or didn't use the Internet to spy, it's just that this isn't evidence of it.

Relevant section below: > Many of the leaked routes were also heavily prepended with AS8048, meaning it would have been potentially less attractive for routing when received by other networks. Prepending is the padding of an AS more than one time in an outbound advertisement by a customer or peer, to attempt to switch traffic away from a particular circuit to another. For example, many of the paths during the leak by AS8048 looked like this: “52320,8048,8048,8048,8048,8048,8048,8048,8048,8048,23520,1299,269832,21980”.

> You can see that AS8048 has sent their AS multiple times in an advertisement to AS52320, because by means of BGP loop prevention the path would never actually travel in and out of AS8048 multiple times in a row. A non-prepended path would look like this: “52320,8048,23520,1299,269832,21980”.

> If AS8048 was intentionally trying to become a man-in-the-middle (MITM) for traffic, why would they make the BGP advertisement less attractive instead of more attractive? Also, why leak prefixes to try and MITM traffic when you’re already a provider for the downstream AS anyway? That wouldn’t make much sense.

Slightly off topic, but if I want to understand the concepts discussed in this article, what all topics should I learn? Is this a good starting place or enough to understand everything in this article - https://beej.us/guide/bgnet/
Scary that so much of the basic internet infrastructure is being managed by US companies. Maybe now the rest of the world will change and become more independent. We should have learnt our lesson long ago though.
What do you mean? The internet is virtually entirely decentralized. There is no one central BGP router.
The rest of the world exports its talent to the US because they don't pay enough. There's no reason why the EU couldn't have made an Akamai or Cloudflare clone decades ago save for the money.
The comments here surprise me a bit. The common thread so far seems to be a general fear of US based companies, but how is that relates to the article?

Cloudflare's post is pretty boring here in that regard. They dig into how BGP works and propose that similar leaks seem common for the Venezuelan ISP in question.

Sure they could be wrong or even actively hiding the truth of what happened here, but the article mentions nothing of Cloudflare being involved in the action and they're describing a networking standard by pointing to publicly available BGP log data.

What am I missing here that everyone else seemed to zero in on?

(comment deleted)
I think it’s just bog standard, “USA bad, not USA good” thinking.
I don't think this article provides any evidence of anything to be scared of.

That said, based on what we know already, there is no reason to take everything is this article at face value necessarily.

Firstly, if anybody isn't aware of the history of Stuxnet, it's worth reading, because otherwise you'd underestimate the government's ability to use 0-days by an order of magnitude (we're talking full custom-written multi-month hacking projects with root-kits and custom fake drivers delivered successfully to an airgapped system, source wikipedia). Also worth learning about Dual EC DRBG debacle.

Secondly am immediate friend of mine worked at a FANG company that routinely sent a firehose of all sorts of things matching all sorts of filters directly to governments. In fact many ISPS have back-doors built in and that's not really disputed (wikipedia: room641A).

So the question to ask yourself is -- if this was a deliberate interaction that cloudfare was required to participate in via a warrant, would they legally even be allowed to publish a blog post that contradicted this?

So I think that is probably the default attitude of skepticism you are seeing, which in my opinion is a good default. Plus the primary claim of this article "Look it wasn't 1 routing issue, it's been happening for even longer! Therefore nothing to look at here!" seems really weak.

I mean, it's the context around the article...based on recent events...
> As news unfolds surrounding the U.S. capture and arrest of Venezuelan leader Nicolás Maduro, ... It is also noteworthy that these leak events begin over twelve hours prior to the U.S. military strikes in Venezuela. ...

This is how I imagine Russian companies in Russia write about the Russian war on Ukraine.

I probably glossed over it in all the posts but…

What is a BGP?

It's the protocol used by carriers to route traffic globally. (Make automated decisions about which core router should receive the traffic coming out of your AS ("network", kinda) )

It's entirely detached from anything else so you're pretty unlikely to have heard of it. In that way it's similar to SS7.

The Internet is a network of Networks, BGP (Boarder Gateway Protocol) is how routers tell other routers what networks they are connected to. This allows you to connect to any device on the Internet, even if you have to go through 5 different networks to get there.
I once was half way through a road trip when google maps routed me off the highway, through a walmart parking lot, and onto another highway.

I assumed it was a badly performing algorithm. But if it had instead routed me through a McDonalds drive through, I'd have assumed it was foul play.

I think the article makes a decent case that this was the former and not the latter, though it would be interesting to see route leaks visualized on a map over time. Too many odd coincidences could sway me the other way.

This is a good opportunity to assess what parts of your own online activity could be impacted by an attacker in the middle (assisted by a BGP leak or otherwise) and, if you're a service provider, how you can protect your customers.

At first pass you probably use HTTPS/TLS for the web, and you know that you shouldn't click through invalid certificate warnings. So the web, tentatively, looks pretty safe.

Email jumps out as vulnerable to eavesdropping, as we largely use opportunistic encryption when transferring messages between mail servers and an on-network-path attacker can use STARTTLS stripping or similar techniques. Most mail servers happily send using cleartext or without validating the TLS certificate. Check that you and your counter-parties are using DNSSEC+DANE, or MTA-STS to ensure that authenticated encryption is always used. Adoption is still quite low, but it's a great time to get started. Watch out for transactional email, like password reset messages, which virtually never validate encryption in transit (https://alexsci.com/blog/is-email-confidential-in-transit-ye... ; instead use multi-factor encryption).

TLS certificates themselves are at risk, unfortunately. An attacker who controls the network in-and-out of your DNS servers can issue domain-verified certificates for your domain; even removing protections like CAA records. DNSSEC is the classic solution here, although using a geographically distributed DNS provider should also work (see multi-perspective validation). Certificate transparency log monitoring should detect any attacker-issued certificates (a review of certificates issued for .ve domains would be interesting).

Ideally, we should build an internet where we don't need to trust the network layer. A BGP route leak would be a performance/availability concern only. We're not there yet, but now is a great time to take the next step in that direction.

Attackers hijacking domains to get certificates issued are generally hijacking registrar accounts, which DNSSEC doesn't help with, which is probably one of the many reasons DNSSEC is so rarely deployed.
There is a nuance here for Cloudflare users that makes this problematic.

While analyzing the Jabber.ru incident (which used this exact BGP->TLS vector), I discovered that Cloudflare's "Universal SSL" actively injects permissive CAA records that override user-defined restrictions.

If a user sets a strict accounturi CAA record (RFC 8657) to lock issuance to their specific account—specifically to prevent BGP hijackers from getting a cert—Cloudflare's system automatically appends a wildcard record alongside it to keep their automation working. Because CAs accept any valid record, this effectively nullifies the protection.

It creates a situation where you think you have mitigated the BGP risk via DNS, but the vendor has silently reopened the door.

I wrote up the technical analysis of this override behavior here: https://david-osipov.vision/en/blog/cybersecurity/cloudflare...

People are so wanting to believe there was an advanced cyber attack to Venezuela’s grid and ISPs that they forget this is a country that hasn’t updated its infrastructure in more than two decades while also not providing any significant maintenance. Most of the “new” technology deployed at the state and federal level comes from corrupt foreign and domestic “suitcase” companies that charged a lot of money to deliver poorly designed systems often even lacking the as-sold equipment. So Venezuela isn’t precisely the most formidable adversary when it comes to cybersecurity.
(comment deleted)
Whether the claims are true or not, this was a very entertaining BGP refresher. It made me wonder: 15+ years ago, I was network engineer and we used quite a bit of "BGP community magic" to get the routing outcomes we wanted.

If BGP only really needed to represent three types of peers (provider, customer, actual peer), wouldn't BGP configuration and perhaps even BGP be massively simplified?

> 8 min read

hah.

As someone who knows nothing about networking, this felt really easy to follow. Thanks for sharing!
This article existing at all is a bit suspicious IMO