"This incident was the result of an infrastructure change that was made to our load balancers to prepare us for IPv6 enablement of GitHub.com. This change was deployed to a subset of our global edge sites."
Seems like this is a common error, systems that treat IPv4 and IPv6 addresses as entirely distinct things. The mapping is easily missed in new implementations. Wonder how many vulnerabilities there are out there with this same mistake.
It's a common bypass of server side request forgery filtering. Backends will try to validate that a user-submitted url doesn't resolve to an internal IPv4 address, but they'll happily allow an IPv6 mapped version for the same IPv4 address.
One potential scenario is bypassing rate limits, enabling abuse, scraping etc. because of the large IP blocks each customer is granted, and seemingly the lack of standardization there.
The assumption of there being a finite amount of IPs to cycle for an attack doesn’t entirely hold with IPv6, and seems like not all software is configured to take that into account.
Unsavory but true, it's nice in a way that ip addrs have a cost. Kinda like phone numbers.
The naive suggestion for ipv6 in that article is what I first thought, block a /64, but since that might still be too cheap, it also quotes a more robust algo of dynamically expanding the blocking scope.
I can just imagine that moving to IPv6 is as messy on Azure than it is on AWS.
Don’t get me wrong, actually using it is simpler, you can just have your range and then be free to do whatever you want. But the fact that it was added later to a stack never made for it shows everywhere, which makes UX tough.
Currently struggling with metadata service weirdness and slowness in IPv6 land on AWS :’)
With how weirdly complicated Azure was about IPv6 last time I looked into it, I wouldn't be surprised if it'd be more complicated than on AWS.
I'm not convinced it ever improved. Looking up a quick guide brings up https://learn.microsoft.com/en-us/azure/virtual-network/ip-s... which tells you to just... assign a random network from 2404:f800::? What even is this network? Are they using a routable IPv6 address as a substitute for an ULA for their NAT'ing load balancers? Why 2404:f800:8000:122::/64 specifically?
I just did the tutorial and I noticed that in the Azure portal it shows a public IPv6 address and a private IPv6 address. From my machine I connect to the public one and magically end up on the private one.
Curling what is my ip6 from the machine yield the public IPv6 address.
I suppose all of this is needed to ensure LB can be done? And it's easier to do with a range like this than a ULA which by default isn't routable.
They do, but they tell you to use a specific /64, which confuses me. I would've expected them to say something like "the address starting with 2404:f800 in your dashboard".
The "private" IPv6 address can be a ULA without any issues if the network is designed to be fully NAT'ed (i.e. for load balancing, maybe failover I guess). If you're not using the global address on your local machine and translate the public address into a private one, your local network doesn't need to have a routable IP address.
I suppose it works just as well, but it makes using IPv6 more confusing for now reason. It's as if Microsoft decide to use 20.64.0.0/10 for private networking on Azure, which they can do (they own that space after all, they can decide not to use it), but just doesn't make much sense.
Azure hands out contiguous blocks of 16 IPv6 addresses. No, not a /56 or anything useful like that. Sixteen addresses.
If you enable IPv6 in some virtual network, other peered virtual networks will have unrelated services just break. Like Postgres, Azure VPNs, and more.
There are no IPv6 to IPv4 gateways, and you can't even build such a thing yourself without enabling IPv6 in the whole virtual network... which breaks other networks!
Azure NATs IPv6, defeating the entire purpose of the thing. It's basically IPv4 with extra steps.
Azure doesn't support IPv6 for any of their PaaS offerings, especially not in their firewall rules.
Etc...
If you think there are excuses for any of this, consider this: IPv6 has been a standard for two decades and Windows has supported IPv6 since 2000.
I like to swap IPv4 and IPv6 in any sentence to gauge the insanity of it. E.g.: "Enabling IPv4 breaks unrelated services in other networks" would have you running for the hills, would it not?
>you think there are excuses for any of this, consider this: IPv6 has been a standard for two decades and Windows has supported IPv6 since 2000.
That's like complaining that Linux came out in the 1990s yet Photoshop doesn't support Linux. Like how it doesn't make economic sense for Adobe to support Linux, it doesn't make sense for a lot of organizations to additionally support ipv6 when they can just support ipv4.
Microsoft was an early IPv6 adopter, they even experimentally "forced" it onto their customers by making their DirectAccess VPN tech IPv6 only! They control both Azure, and the OS it runs on (Windows). Sure, bits of it runs on Linux, but they have their own distro of that too that they control (Mariner).
This would be more like complaining that Linus Torvalds prefers not to use open source software.
It’s easy to guess what happened: they developed an IPv4-only network stack and baked the limitations and constraints of IPv4 into it: private addresses are mandatory, public addresses are scarce, and NAT is required.
Then they got told to “do the needful” and make IPv6 happen, so they did… by weaving IPv6 support through the tangled briar patch of their codebase. They wove it through the NAT, the tiny public address blocks, and the mandatory private address spaces on virtual networks.
The result is IPv4 with a sticker on it with a hand-written label that says “IPv6”.
I think the problem with Py2 to Py3 was that it was too big of a migration to be easy but also too small of a migration to be game-changing. Like, switching for seriously improved packaging, parallelism, or performance would've been more compelling.
And yeah ipv6 could've been a much smaller change.
Which http are we talking about? HTTP3 that people wanted to use before the security portion of the spec. was even done? HTTP2 that had an entire section dedicated to possible exploits that where later found on major sites? Or HTTP 1 which enabled some of the exploits seen with 2 since implementations are neither consistent nor strict when parsing parameters?
Sometimes it feels as if the Internets security was supported by paper walls with a "please punch here to exploit" written all over the place.
Does it matter? Every HTTP version from HTTP/1 to HTTP/3 has each experienced its own set of vulnerabilities.
Never stopped us from using HTTP.
IPv4 has had multiple vulnerabilities in its stack as well, never stopped the internet from growing. (But the exhaustion of IPv4 _is_ hindering further expansion of the internet!)
I mean it's not a good comparison. IPv4->6 is a migration, with a lot of difficulty having to do with the fact that v4 is already in use. HTTP wasn't a replacement for something, but there is HTTP1->2.
When I was a network engineer I frequently experienced IPv6 skill rot. I'd go long periods of time not encountering it in the wild. It's hard to be confident with a thing you're not working with on a regular basis. I guarantee you I made mistakes because of it. There was only one maybe 2 year period where I would have considered myself an IPv6 expert.
I’m curious how this wasn’t caught in testing. When you set up a dual-stacked listener which is common when you omit the binding IP, you already get IPv4-mapped IPv6 address, and the request logs would have also shown the ::ffff:x.x.x.x addresses. Also curious as to why they are working with text representations of IP addresses instead of binary representations, although the incident GitHub described here could be very well a firewall deciding that IPv6 datagrams were invalid traffic.
However, to make this thread complete, I guess someone will again complain about the address format, without realizing that shoving in extra address bits on an IPv4 datagram is already a new protocol, the networking equivalent of Godwin’s law[1].
I'll be that guy. Glad to know I'm not the only one who just wants ipv4 with extra octets.
Edit: An aside, these addresses are different in both text or binary representation, but the article doesn't provide many details, so I can't entirely follow.
The other big problem I see to getting buy-in from non-networking people is the lack of good sources on how to prepare and roll out ipv6. It takes forever scrolling through pages of documentation that go through the history of ipv6 and the ipv6 numbering format. As though the world needs yet another primer on how long groups of 0's can be dropped to make ::1 be a valid address. Just link to Wikipedia's page explaining ipv6 addressing and move on.
I have firewalls but don't trust them to have safe defaults. Or more like, I don't trust every random router, including ones that each member of my extended family will inevitably call me about, to have safe firewall defaults until I test it myself.
I reasonably trust, not 100% but pretty well, that my boss's or grandma's PC with ip 192.168.1.3 on NAT (not even routable from the WAN) will not receive inbound TCP/UDP. That would take effort.
Upnp is where the shred of doubt comes from. Only a shred because the PC very likely won't use it, and nowadays few routers have it by default. And the solution is, upnp shouldn't exist.
Enough routers still have ipv6 firewall disabled by default, in which case, every port on every networked device is exposed. Upnp, I'm less worried about being enabled, and even if it is, Windows won't map SMB to itself.
In the long term with our world of cheap crappy routers, it's also much easier for upnp to die off than for ipv6 firewalls to become ubiquitously strong. Both of those are extra features that take effort to do right. So is NAT, but a broken NAT doesn't expose hosts.
My grandparent's crappy ISP-sponsored router from ~2017 (maybe 2016), TP-Link C1200 v1, had the IPv6 firewall enabled by default. (It has UPnP enabled by default too... which I turned off promptly. They don't need it anyway.)
My own Asus router from 2016, RT-AC1200G+, also came with IPv6 firewall enabled.
Both routers are 8 years old at this point. And they are far from being a prosumer product. What router are you using that it doesn't come with IPv6 firewall enabled?
edit: Oh, and my friend's cheapo Netis router (quite an obscure brand IMO) also had IPv6 firewall enabled out of the box.
>ipv6 firewalls to become ubiquitously strong
They don't need to be strong, the firewalls just have to be stateful so that they can drop unsolicited packets. It's that simple. Any other more sophisticated attacks would have made your NAT equally vulnerable as well.
Additionally, stateful firewalls have been available since decades ago. They are not exactly black magic.
I'll have to check the one I have at home when I get home. Airport Extreme is one example I remember not having ipv6 firewalling by default, even though it has it for v4.
Many internet users don't even know what the word firewall means. And there are a lot of devices with unpatched software. Obviously NAT shouldn't be seen as a firewall, but for many consumers it's pretty much the only line of defense.
I don't know if ISPs without NATs automatically deploy firewalls, but would hope so. IPV6 is harder to scan, but once you know a device's IP address (e.g. from web logs), it'd be easy to exploit older devices.
One example, AT&T cellular gives you an ipv6 address besides v4, and it's still behind a NAT (not just a firewall). Though I'm glad my home ISP lets me decide that for myself.
It's important to test this. I own an MSI router where the firewall is completely broken for IPV6. They even made the mistake of listening for DNS and HTTP (admin tool) on the external interface.
Exactly, even if my router says "ipv6 firewall enabled," I still can't trust that they did it right. I would rather go full NAT (which on most routers equals disabling IPv6) than have to pen-test my router.
"Full NAT" doesn't even make sense. Do you mean symmetric NAT or something in that vein?
>which on most routers equals disabling IPv6
Turning on NAT on IPv4 certainly does not entail turning off IPv6. Just _what_ exactly are you sprouting? It's not the first time you have made beginner mistakes like this.
NAT isn't a nice feature, it's a necessary evil to deal with address exhaustion in v4 and causes many problems of its own. Using it in v6 makes no sense.
It doesn't have a "default deny" behavior -- NAT doesn't block connections at all. It doesn't separate your LAN either, that's done by having a router. If you want those things then NAT isn't the thing you want.
You're right that in a technical sense, NAT isn't block connections, more like it allows connections in the first place that would otherwise not be possible. But that's the best part. It's very hard for a misconfigured NAT to result in devices being exposed, unlike a true firewall, which is actively blocking connections to devices that would otherwise be totally reachable.
NAT is sucky in plenty of server-side situations and is thus avoided, but in homes and offices, yeah I wouldn't want it any other way.
> The change had the unintended consequence of causing IPv4 addresses to start being passed as an IPv4-mapped IPv6-compatible address to our IP Allow List functionality.
Sounds like the opposite of this bug. GitHub's system represents 0.0.0.0/0 and ::ffff:0.0.0.0/96 as distinct values, when they should be merged/normalized.
I think it's best to treat IPv4-mapped addresses as an implementation detail of the socket API, and not leak them into general purpose code.
Though things get interesting when an external user provides "::ffff:1.2.3.4" as text.
67 comments
[ 3.0 ms ] story [ 136 ms ] threadThe assumption of there being a finite amount of IPs to cycle for an attack doesn’t entirely hold with IPv6, and seems like not all software is configured to take that into account.
Explained here better: https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/
The naive suggestion for ipv6 in that article is what I first thought, block a /64, but since that might still be too cheap, it also quotes a more robust algo of dynamically expanding the blocking scope.
Don’t get me wrong, actually using it is simpler, you can just have your range and then be free to do whatever you want. But the fact that it was added later to a stack never made for it shows everywhere, which makes UX tough.
Currently struggling with metadata service weirdness and slowness in IPv6 land on AWS :’)
I'm not convinced it ever improved. Looking up a quick guide brings up https://learn.microsoft.com/en-us/azure/virtual-network/ip-s... which tells you to just... assign a random network from 2404:f800::? What even is this network? Are they using a routable IPv6 address as a substitute for an ULA for their NAT'ing load balancers? Why 2404:f800:8000:122::/64 specifically?
I just did the tutorial and I noticed that in the Azure portal it shows a public IPv6 address and a private IPv6 address. From my machine I connect to the public one and magically end up on the private one.
Curling what is my ip6 from the machine yield the public IPv6 address.
I suppose all of this is needed to ensure LB can be done? And it's easier to do with a range like this than a ULA which by default isn't routable.
The "private" IPv6 address can be a ULA without any issues if the network is designed to be fully NAT'ed (i.e. for load balancing, maybe failover I guess). If you're not using the global address on your local machine and translate the public address into a private one, your local network doesn't need to have a routable IP address.
I suppose it works just as well, but it makes using IPv6 more confusing for now reason. It's as if Microsoft decide to use 20.64.0.0/10 for private networking on Azure, which they can do (they own that space after all, they can decide not to use it), but just doesn't make much sense.
Random examples:
Azure hands out contiguous blocks of 16 IPv6 addresses. No, not a /56 or anything useful like that. Sixteen addresses.
If you enable IPv6 in some virtual network, other peered virtual networks will have unrelated services just break. Like Postgres, Azure VPNs, and more.
There are no IPv6 to IPv4 gateways, and you can't even build such a thing yourself without enabling IPv6 in the whole virtual network... which breaks other networks!
Azure NATs IPv6, defeating the entire purpose of the thing. It's basically IPv4 with extra steps.
Azure doesn't support IPv6 for any of their PaaS offerings, especially not in their firewall rules.
Etc...
If you think there are excuses for any of this, consider this: IPv6 has been a standard for two decades and Windows has supported IPv6 since 2000.
I like to swap IPv4 and IPv6 in any sentence to gauge the insanity of it. E.g.: "Enabling IPv4 breaks unrelated services in other networks" would have you running for the hills, would it not?
That's like complaining that Linux came out in the 1990s yet Photoshop doesn't support Linux. Like how it doesn't make economic sense for Adobe to support Linux, it doesn't make sense for a lot of organizations to additionally support ipv6 when they can just support ipv4.
This would be more like complaining that Linus Torvalds prefers not to use open source software.
Then they got told to “do the needful” and make IPv6 happen, so they did… by weaving IPv6 support through the tangled briar patch of their codebase. They wove it through the NAT, the tiny public address blocks, and the mandatory private address spaces on virtual networks.
The result is IPv4 with a sticker on it with a hand-written label that says “IPv6”.
“Job done boss!”
How was this not caught in a pre-production environment?
Perhaps a smaller migration should have been considered first.
And yeah ipv6 could've been a much smaller change.
And that article proves them right.
Hardly "never going to happen".
How many bugs occur due to misconfigured HTTP servers every day? Do you conclude that HTTP is also a "no go"?
Sometimes it feels as if the Internets security was supported by paper walls with a "please punch here to exploit" written all over the place.
Never stopped us from using HTTP.
IPv4 has had multiple vulnerabilities in its stack as well, never stopped the internet from growing. (But the exhaustion of IPv4 _is_ hindering further expansion of the internet!)
However, to make this thread complete, I guess someone will again complain about the address format, without realizing that shoving in extra address bits on an IPv4 datagram is already a new protocol, the networking equivalent of Godwin’s law[1].
[1] https://news.ycombinator.com/item?id=39101523
Edit: An aside, these addresses are different in both text or binary representation, but the article doesn't provide many details, so I can't entirely follow.
I reasonably trust, not 100% but pretty well, that my boss's or grandma's PC with ip 192.168.1.3 on NAT (not even routable from the WAN) will not receive inbound TCP/UDP. That would take effort.
This argument is so overused we might as well add it to ipv6-excuses.txt.
And nowadays few routers have IPv6 firewall disabled by default... Do you see how fragile your argument is now?
In the long term with our world of cheap crappy routers, it's also much easier for upnp to die off than for ipv6 firewalls to become ubiquitously strong. Both of those are extra features that take effort to do right. So is NAT, but a broken NAT doesn't expose hosts.
My own Asus router from 2016, RT-AC1200G+, also came with IPv6 firewall enabled.
Both routers are 8 years old at this point. And they are far from being a prosumer product. What router are you using that it doesn't come with IPv6 firewall enabled?
edit: Oh, and my friend's cheapo Netis router (quite an obscure brand IMO) also had IPv6 firewall enabled out of the box.
>ipv6 firewalls to become ubiquitously strong
They don't need to be strong, the firewalls just have to be stateful so that they can drop unsolicited packets. It's that simple. Any other more sophisticated attacks would have made your NAT equally vulnerable as well.
Additionally, stateful firewalls have been available since decades ago. They are not exactly black magic.
I don't know if ISPs without NATs automatically deploy firewalls, but would hope so. IPV6 is harder to scan, but once you know a device's IP address (e.g. from web logs), it'd be easy to exploit older devices.
IPv6 is still confusing for many developers
>which on most routers equals disabling IPv6
Turning on NAT on IPv4 certainly does not entail turning off IPv6. Just _what_ exactly are you sprouting? It's not the first time you have made beginner mistakes like this.
It doesn't have a "default deny" behavior -- NAT doesn't block connections at all. It doesn't separate your LAN either, that's done by having a router. If you want those things then NAT isn't the thing you want.
NAT is sucky in plenty of server-side situations and is thus avoided, but in homes and offices, yeah I wouldn't want it any other way.
It's not hard to test if your firewall is working. Using NAT to avoid that is not the sensible approach.
Sounds like this bug: "Unable to reliably distinguish IPv4-mapped-IPv6 addresses from regular IPv4 addresses" https://github.com/golang/go/issues/37921 .
Use https://pkg.go.dev/net/netip instead.
I think it's best to treat IPv4-mapped addresses as an implementation detail of the socket API, and not leak them into general purpose code.
Though things get interesting when an external user provides "::ffff:1.2.3.4" as text.