Tell HN: Domain fronting to be blocked on Azure
"Action required: Azure Front Door/Azure CDN blocking domain fronting
Please take action to stop domain fronting on your application before 8 November 2023 You're receiving this email because you currently use Azure Front Door or Azure CDN Standard from Microsoft (classic).
Since 29 April 2022, we've changed the behavior of Azure Front Door and Azure CDN from Microsoft to align with our commitment to stop allowing domain fronting behavior on our platform. With that change, we offered the option to enable blocking domain fronting for existing or newly created Azure Front Door, Azure Front Door (classic) and Azure CDN Standard from Microsoft (classic) resources, through opening a support request. See details in <Generally available: Controls to block domain fronting behavior on customer resources | Azure updates | Microsoft Azure> https://azure.microsoft.com/updates/blocking-domain-fronting....
To continue our commitment, we're making changes in two phases to stop allowing domain fronting behavior on our platform.
1. Beginning 8 November 2022, all the newly created Azure Front Door, Azure Front Door (classic) or Azure CDN Standard from Microsoft (classic) resources will block any HTTP request that exhibits domain fronting behavior. Previously existing Front Door, Front Door (classic) and CDN from Microsoft (classic) resources aren't affected by these changes.
2. Beginning 8 November 2023, all existing Azure Front Door, Azure Front Door (classic) and Azure CDN Standard from Microsoft (classic) resources will block any HTTP request that exhibits domain fronting behavior.
Recommended action Between now and 7 November 2023, if you want to block domain fronting for any existing Azure Front Door or Azure CDN Standard from Microsoft (classic) resources created before 8 November 2022, please open a support request. Provide your subscription and Azure Front Door, Azure Front Door (classic), or Azure CDN Standard from Microsoft (classic) resource information in the support request. Once blocking of domain fronting has been enabled, Azure Front Door, Azure Front Door (classic), and Azure CDN Standard from Microsoft (classic) resources will block any HTTP requests that exhibit this behavior.
If your application uses a different TLS SNI extension during the TLS negotiation from the request Host header, you should prioritize changing this behavior on your application by 7 November 2023 to ensure they match. Otherwise, your application or API may be impacted by this change on 8 November 2023.
If you have any questions, please open a support request and provide your subscription details along with your Front Door or Azure CDN from Microsoft resource information.
If you have any questions, please contact us."
Posting it here in case it's of interest to anyone.
131 comments
[ 49.4 ms ] story [ 2109 ms ] threadhttps://en.m.wikipedia.org/wiki/Domain_fronting
”Many large cloud service providers, including Amazon and Google, now actively prohibit domain fronting, which has limited it as a censorship bypass technique. Pressure from censors in Russia and China is thought to have contributed to these prohibitions”
https://wikipedia.org/wiki/Domain_fronting
"Many large cloud service providers, including Amazon, Microsoft, and Google, actively prohibit domain fronting, which has limited it as a censorship bypass technique."
our venerable corporations would never bend to the will of foreign dictators.
0: https://twitter.com/sobollubov/status/1567152744812740610
I’ve personally seen domains on a shared provider banned without notice even though alleged pirate content was taken down within minutes. Very difficult to unban
If we were using it on a pentest, you'd best believe there are actors using it for far more nefarious purposes.
Morally, the question is which one is the most important?
“Because Security” arguments like this are increasingly used in place of “think of the children”.
The other 'possibilities' are not necessarily guaranteed to be allowed for egress traffic out of an organization. That's why the cloud providers blocking it is a big deal - most orgs WILL allow outbound traffic on more than a few ports to these platforms
> Domain fronting is a technique for Internet censorship circumvention that uses different domain names in different communication layers of an HTTPS connection to discreetly connect to a different target domain than is discernable to third parties monitoring the requests and connections.
https://en.wikipedia.org/wiki/Domain_fronting
Cool, so we are bowing down further to oppressive regimes now.
We'll have to ban rocks next though.
Then again, if you want to argue stone-age semantics: we can't ban anything and should just resort to continuous murder since rule-based society is too hard.
Once you've bought a knife, it will usually be illegal to carry it in public unless you have a good reason, and "self defence" is not a good reason.
Some particular knives, which have no apparent purpose except as weapons, are just illegal automatically except in some cases if you're a museum. In particular almost all swords (if they have a working blade) are in this category, and most things designed as concealed weapons (e.g. blade hidden in an umbrella, combs that are actually knives, push daggers)
I mean, are instances of kids <=17 year olds with knifes down? What about shivs and other things?
The reason I'm asking is because as a kid, if I had been told to not do something, I'd just find a way to do it. Not to use it, but to find loopholes and just say, _so what?, I can do x_
Reducing violence is a tricky thing because to a large extent it seems to me to be a symptom of broader and nearly intractable societal malaise. Not that it isn't worth removing nuclear weapons from circulation but the returns diminish quicker than most public policy folks seem to want to admit.
[1] https://commonslibrary.parliament.uk/research-briefings/sn04...
There was a spate of stabbing reports, about a decade ago. That seems to have died down. My guess (and it's only a guess) is that only drug dealers and people bent on mayhem carry guns or knives that aren't street-legal these days; what's the point in getting busted for carrying a penknife?
Incidentally, I don't get the point of the ban on locking knives. If you stab someone with a knife that doesn't lock, then I suppose there's a risk to your own fingers; but the dude you stabbed won't care if it was a locking knife or not. I do get the blade-length restriction though. You don't need a 3" blade to sharpen pencils.
My straight razors are street-legal, though, and I think they're quite a lot more dangerous than a 3" knife that locks. The sharpened edge of a razor is only about 2" long, so I assume it's street-legal; but you could really make a mess of someone's face with a razor (the preferred weapon of Pinky, the gangster from Brighton Rock).
My friend "got" a nice chat, and went on with his day, but did later comment that had he not had the outfit to go with them, and a vast array of obvious kitchen tools, he thought that they'd probably look at it a lot more seriously.
If airlines banned knives and someone posted a definition of “knife” that suggested they are only used for eating food, it would be perfectly reasonable for a reply to point out that there are dangerous uses which might have led to the bad, and that it’s disingenuous to suggest that the ban was obviously motivated by a desire to make eating difficult.
It’s still fair to debate the balance of use and the merits of the ban, but IMO it is not reasonable to be upset over a more complete definition and context.
https://hackernoon.com/domain-fronting-101-what-is-domain-fr...
"Amazon and Google bow to Russian censors in Telegram battle"
https://www.fastcompany.com/40568177/amazon-and-google-bow-t...
"U.S. Cloud Providers Face Backlash From China’s Censors"
https://www.wsj.com/articles/u-s-cloud-providers-face-backla...
https://archive.ph/qhFQ5
>China’s Internet censors have strengthened content screening in recent months, creating difficulties for businesses and disrupting more commonly used firewall-circumvention software called virtual private networks, which connect users to the Web through a proxy server overseas. President Xi Jinping has ordered tighter control of online content that may undermine the ruling Communist Party, with bloggers facing jail for spreading what the government says are false rumors.
The CEO even came on HN to try to frame it as an abuse mitigation, accusing Lantern of exploiting Cloudflare and arguing that they were not a customer. That was obviously false because you need to have a Cloudflare zone configured for domain fronting to work. They were a customer as much as the targeted hate websites they strenuously defend.
https://news.ycombinator.com/item?id=9234367
Companies show their color in selecting who they will stand up for.
What actually happened was we started to get reports from customers that we were serving content from unrelated sites under their domains. Wasn’t happening a lot but for a reverse proxy that’s terrifying and about as bad as it can get. And what made it even weirder was that the domain fronting was connecting to other proxy servers outside Cloudflare and so we literally had situation where a Cloudflare customer's website suddenly served up google.com (not a customer).
The issue was domain fronting where SNI didn’t match Host causing us to handle the traffic incorrectly sometimes (infrequently but not zero). Since standard use of the Internet doesn’t need SNI and Host to not match we blocked the use of domain fronting very fast to ensure the integrity of our service. Lantern was domain fronting tickling this bug and causing our customers trouble.
Hence we dropped domain fronting.
I remember this well because I actually debugged it myself and reported the problem in Jira. You weren’t involved and inventing a story about “China” is dead wrong.
I do remember one terrifying bug that Lantern was tickling which caused responses to cross streams, and I was involved in debugging that, but it was not due to the Host/SNI mismatch. It just happened around the same time that domain fronting was blocked. (I am going to respect my confidentiality agreement here, but if you want I can share what I remember here or in private.)
To be clear, that's not correct we did do this to mitigate the bug.
We were facing the bug that I described (the cross stream thing) showing up when Lantern was used. It was causing disruption to our service and customers were writing in. We were trying to understand what was happening and needed to stop it. One of the things we did to stop it was disable domain fronting. As we were seeing the customer reports we didn't know if this was an OpenSSL bug, something in NGINX, something in our code, but we did know that Lantern was somehow causing it and they were doing domain fronting which wasn't the standard use of our service and so we dropped it.
My point is that disabling domain fronting (or leaving it disabled after finding the bug's root cause) was a policy decision, not something necessary to mitigate the bug or prevent it from re-occurring.
If you'd please review and follow the site guidelines, we'd appreciate it: https://news.ycombinator.com/newsguidelines.html.
No organization I am part of will ever do business with you if I can stop it.
You fought for Turing but I suppose the Uyghur concentration camps mean nothing to you. They are not British so how could their lives be worth fighting for when there is money to be made (for yourself) without regard to morality or any sense of decency.
But good job doing Cloudflare PR.
There was also a prelude to all of this that I think made things stickier and bizarrely personal. Prince and I share a mutual friend who introduced us just a few weeks prior. Prince said he supported what we were doing, but asked that I not talk about it publicly, presumably because of the pending China deal. The problem was that literally moments after our friend had introduced us via email, and before he made that request, I had a call with the WSJ where I talked about precisely this. I did everything I could to walk back the article, but Prince didn't buy it and seemed to go ballistic over it. After the WSJ piece, we pulled back from talking more publicly in general.
Oh, I forgot! We also partly stayed silent because they didn't actually shut down what we were doing at all =). They matched the SNI to the Host header, sure, but they missed a little detail: we weren't using SNI. Hehe. Lantern worked for another six months or so, and then, through a similarly bizarre sequence of events, we essentially tipped them/you off to what was happening. We remained a customer throughout, and we're a customer to this day.
Either way, though, Cloudflare does great work, and everyone has their faults, so I'm generally sympathetic over the whole thing with the one caveat that I am truly unclear how much ultimately did relate to China, most clearly in terms of any public support for these internet freedom techniques.
Oh, and I've wanted you to work on Lantern forever btw. Oooh actually if you're not aware of it, the uTLS Go TLS fork is a hugely impactful project that's in widespread use (I would guess maybe 50 million monthly active users rely on it in censored regions via various projects) but needs updating - https://github.com/refraction-networking/utls
Oh, and if you think we were effective in China then, you should see what we're doing in Russia and especially Iran now!
What I'm wondering now is why one should use that instead of a proxy server that maps some.domain.com to something.else.com
Consider two clients, Alice and Bob, they are both connecting to some IP 10.20.30.40 using TLS or QUIC.
Alice wants to access naughty.thing.example and Bob wants to access bland.stuff.example. Perhaps Mallory is trying to prevent Alice from accessing naughty.thing.example (but they don't mind bland.stuff.example) or perhaps Alice just doesn't want Eve to know what she's accessing. Mallory and Eve are both on the network, able to interpose between Alice and Bob and 10.20.30.40 and neither Alice nor Bob can easily prevent that.
Under domain fronting, Bob is just honest, he tells the server I want bland.stuff.example, and then he uses bland.stuff.example as usual and everything works. This means for technical operations it's OK if the 10.20.30.40 server cares what Bob said during connection. For example maybe once Bob says bland.stuff.example, the server spins up an IPC to a Python server which only knows about bland.stuff.example and splices Bob's connection to the IPC.
Under domain fronting, Alice's situation is tricky. Alice says she wants bland.stuff.example but she actually uses naughty.thing.example and expects that to work. Technical ops people who've arranged that bland.stuff.example connections get spliced are faced with bug reports - why didn't naughty.thing.example work, 10.20.30.40 was the right server ???
Under ECH, both Alice and Bob are presenting a visible name (which might be bland.stuff.example or some other value entirely) which Mallory and Eve can easily read, but they're also providing an encrypted destination, Alice can encrypt naughty.thing.example while Bob encrypts bland.stuff.example† The 10.20.30.40 server can decrypt the name, and thus it knows which service Alice and Bob actually intend to access.
† If you thought "What about padding? Those names are different lengths" congratulations, you're now at step 1 of a long process which is why this was not trivial to design and implement.
DNS-over-HTTPS is only used in the last-mile --- recursive resolver to client, not NS to NS, so the request is still sent in plain. Besides, DNS-over-HTTPS still requires ... TLS ...
This whole thing is still a draft RFC [2] and currently called encrypted client hello.
What I don't get: The client will have to disclose, which public key it used to encrypt the client hello / SNI value. But don't we know "all" the public keys from the certificate transparency logs? Usually a public key is only valid for a limited set of domains, so it would be easy to associate the public key used for encrypting the SNI value with the relevant service. Just look up the public key, read it's SAN or CN, and you pretty much know the SNI value. Or at least the service, which might be good enough. What am I missing?
[1] https://blog.cloudflare.com/encrypted-sni [2] https://www.ietf.org/archive/id/draft-ietf-tls-esni-15.html
Actually, not necessarily, but first...
> But don't we know "all" the public keys from the certificate transparency logs? Usually a public key is only valid for a limited set of domains, so it would be easy to associate the public key used for encrypting the SNI value with the relevant service
The keys used to encrypt ECH are distinct from keys used to prove identity. So, no only are the CT logs irrelevant (they're logs of the identity information which uses separate keys) but also you would usually choose the same keys for all the names hidden behind some particular server.
Suppose I run big.huge.example a hypothetical CDN. We can use the same public key for clown-porn.example, abortion-rights.example, huge-corp.example, government-stuff.example, even though these services may be entirely segmented internally, we're only using that key to keep the names private from everybody else. As big.huge.example we know whether a client uses clown-porn.example or abortion-rights.example, but an on-path adversary doesn't know that and the key used doesn't help them.
Still back to the encryption problem. We don't actually necessarily need to tell people which key was used. So long as the number of possible keys in use is modest they can guess:
Suppose big.huge actually has four keys in use, maybe two sets of customers on different tiers of product, maybe we're switching keys and there's an overlap period of course - we don't need clients to signal necessarily, we can just decrypt with all four keys, incurring a modest 4x decrypt performance penalty and throwing away the 3 which fail in the ordinary case.
This is actually a small triumph for the people responsible for RFC 8446. With previous iterations of TLS it was always discovered shortly after release that idiots broke stuff and so a "fallback" was necessary to allow you to speak the previous version. Such fallback is dangerous because an adversary can thus forcibly downgrade you to an older protocol, and thus attack old protocols even if the new protocol is safe.
How is it done? That is, how does TLS 1.3 avoid downgrade attacks?
When a TLS 1.3 server finds itself talking to somebody over TLS 1.2 (for example maybe a rather archaic web browser is connecting) it scribbles over some of the bytes labelled "random" in its Hello message. It scribbles 44 4F 57 4E 47 52 44 01. Which in ASCII spells "DOWNGRD".
Those bytes don't mean anything special in TLS 1.2, they're just a strange coincidence. But if you're a TLS 1.3 client, seeing those bytes means a Downgrade attack was attempted. So you immediately give up, you are being attacked.
So you might think well, a bad guy could just change those bytes blind right? Nope. The "random" field is used by both parties to choose parameters they're going to verify in a moment to check everything is safe. If you can change the bytes the values will be different and the connection fails anyway.
Any company who does not want to lose a market of ~18% of global population will make sure it complies (example: Apple).
We need to think about real life here and not just technical implementation
Really? Tor springs to mind.
Only if by "their" you mean the ruling class, and by "self-determination" you mean their ability to control others. You can't really say their opinions represent the will of the people, especially when it's the people themselves ultimately choosing to engage in "illegal" activities.
And sure, "the will of the people" bakes in two very western individual/collectivist values. But as I get older I'm learning to not play the relativism card as much. We should certainly be critical of our culture - but to the point of making it better, not handicapping its spread.
Plus there's still a very easy answer for the poor oppressed tyrant who doesn't like freedom of communication - shut down all Internet connections.
Take a look at Iran, they are religious as a nation. Their religion supersedes any individual's will. Or china, the will if the party supersedes individuals' needs. You in your post-colonialist luxury worry about your own will but historically people worry about the well being of their children and society which means not getting killed/raped, having economic and academic opportunities,etc... and beyonf that also, the will of their god being implemented. Which people? The people with weapons just like in a democracy (else the US would still be under a monarchy). They self determined through violence and politics the state they are in. In China the economy is good so the CCP is actually popular, so they self-determined communism (at least by name).
There many countries where the US exported a revolution or a democracy and they are in shambles now (who cares so long as they are under western influence?) name one nation in europe that was did not prosper under a monarch before self-determining democracy? Yet your arrogant presumption robbed many nations of that opportunity. Because the people are unprepared and uninformed, the loudest asshole takes power by deceiving people and saying the right things the he gets super rich until they protest and he flies off with his money until the next asshole. This keeps happening and is your direct responsibility since it is because of your will your politicians are exporting chaos and installing puppet leaders in other countries. Civil war after civil war, genocide after genocide because humans are tribal by nature and there is no peaceful way a ruling tribe (see iraq) would peacefully let go of power.
Keep in mind that if it wasn't for the threat of violence by your own country's military you also would be part taker in the voiolence and chaos you are exporting.
Democracies cannot thrive when people are starving, destitute and uneducated and lack basic infrastructure by which they can be informed enough to critically analyze what their politicians are saying. "Your life sucks because of $tribe" does that sound familiar?
So you organize a revolt over Tor, I guess the other side will also use Tor to organize the civil war or genocide?
Read my comment again - I explicitly acknowledged this.
I'd say that most of your comment is attacking a top-down "exporting democracy" whether covertly, led by the State Department, outright invasion, etc. I agree that these things are evil, especially when "democracy" is used as the marketing for the primary concern of implementing USD-denominated markets.
So where we differ is the bottom up emergent behavior of people making their own choices.
> historically people worry about the well being of their children and society which means not getting killed/raped, having economic and academic opportunities,etc... and beyonf that also, the will of their god being implemented.
And yet, those are the same exact people choosing to use technology that provides things like (very imperfect) communications privacy. Your argument implies that their choices are wrong, so what you're really saying is that the larger population needs to be paternalistically protected from themselves. Which brings us to the huge unstated assumption of your comment that for every society we should respect some ambient "values" of the society, with some more powerful in-group protecting those values against the larger population.
I agree that's a descriptive statement about the power structure of basically every society. But I don't agree that it's a prescriptive model with inherent moral value.
And yes, I do know this viewpoint is a very "western" philosophy. I put "western" in quotes because it seems like a strong general attractor, as communications technology enables human-to-human communication unmediated by traditional top-down power structures. I'm also learning not to handicap myself by getting stuck in the doldrums of relativism. To the extent that it may be inherently western, spreading our own culture through arms length communication and voluntary buy in is a hell of a lot more defensible than the traditional ways of spreading culture - violent conquest and subjugation.
No it's not, see https://news.ycombinator.com/item?id=33446064
On most social media sites they're implemented the same way (ie. posts deleted/hidden from other users), but the objectives are totally different.
They also have different targets - moderation is about a forum, while censorship is about an idea
What are you losing here "nibbleshifter"?
Why do you put infosec in scare quotes? Why are they "wankers"? Why scare quote and name call a legitimate profession? Because you have qualms?
Tor's "meek" pluggable transport uses domain fronting for this purpose.
I work in the so called "infosec" field, and about half the field are myopic wankers who would readily sacrifice privacy wholesale to gain an ounce of so called security. Think: the kind of people who also want to cripple eSNI or DoH in the name of "network monitoring".
Historically these people and even journalists are targeted by Nation States utilizing Israeli made offense tools like Pegasus.
Domain fronting would not help in these cases to avoid censorship, maybe but you are attempting to circumvent a Nation State with almost unlimited money and resources the target would never be able to have.
Does it suck that domain fronting is gone? Yes. Is it a good thing it's gone? Yes.
The fact is the people that used domain fronting for your use case is heavily heavily outweighed by malicious actors.
One could argue that domain fronting and guns are the same. Why should I have my gun taken away when I am using it for legal purposes? Just because bad people use a gun for bad things I shouldn't be scrutinized for my legal use. I shouldn't have access taken away due to bad actors leveraging them for ill gotten gains.
Between encrypted SNI (Or domain name fronting), encrypted DNS and of course HTTPS. The biggest legitimate use case of Tor would vanish.
Your comment was on point up until this. Just because the Tor use you seemingly most identify with is access from oppressive regimes to the western web, doesn't mean that the rest should be shunned. If you want to make political points about specific other uses of Tor, do it explicitly rather than maligning the whole project.
your standard user agent (e.g. browser) will not send different values in SNI and HTTP Host header.
this is a deliberate action by the user agent to obscure the actual traffic destination.
this can of course be used both for censorship circumvention but also misleading corporate traffic inspection when TLS is not broken, though it's debatable whether that should work in the first place.
Azure originally started on this path in 2021: https://www.microsoft.com/en-us/security/blog/2021/03/26/sec...
Working in the pentest/red team field, I've seen various providers ban consulting companies and red teams from using domain fronting -- however, this doesn't stop the threat actors.
CDN customers that are having their stuff blocked because of that are not going to be happy, in general.
With domain fronting you may get 'nsfw_example.com' content from a TLS connection negotiated with 'bank_example.com'.
This can be a security threat (not able to properly filter outbound traffic) and customer A may be unhappy about Azure allowing nsfw content to be distributed over a channel secured under the name of their bank.
Lots of IoC base on DNS as well so that is out of the windows since the malicious traffic is inside TLS...:-/
The only part of the definition that applies is a subjective part... "for censorship circumvention".
How does the CDN detect this? The CDN only sees the encrypted domain, correct?
Much like back in the 1930s, various major corporations are sending a clear signal about how much they value human rights versus making money. Personally, I buy into the whole argument that we can keep capitalism ethical by "voting with our money" and it's up to the people to boycott unethical companies. I seem to be something of a radical though in today's society.
*) Or at least one of the reasons, none of which are good.