My monitors started going berserk around 3:36 PM CT. Here's an example of the problem (I am not hostmaster for The Onion, but they are a well-known Linode customer):
$ whois theonion.com | grep -i 'name server'
Name Server: NS1.LINODE.COM
Name Server: NS2.LINODE.COM
Name Server: NS3.LINODE.COM
Name Server: NS4.LINODE.COM
Name Server: NS5.LINODE.COM
$ dig @ns1.linode.com theonion.com ns +short
$ dig @ns1.linode.com theonion.com a +short
DNS Performance Issue
Incident Report for Linode
Investigating
We are currently experiencing denial of service attacks that are targeting our DNS infrastructure.
Aug 23, 20:48 UTC
This incident affects: Hosted DNS Service.
Route 53 also has no IPv6 support, and no stated policy on how it deals with DDoS attacks, or if you get charged for the attack if it's directed at you. Good luck!
It might be more accurate to say R53 has partial v6 support. It'll return v6 records (AAAA) happily. The nameservers themselves do not respond on v6 (which is unfortunate and certainly means v6 support is lacking). Practically, however, it's pretty unlikely, that a v6 only host isn't going to also have access to a v6-to-v4 gateway. Not ideal, but generally transparent for most intents and purposes ...and generally used by just about anyone with a v6 address assignment today, everyday.
For DDoS attacks, I'm not sure what you mean by a "stated policy." They do a lot of countermeasures, some of which can be found via linked PDFs at https://aws.amazon.com/security/ Various R53 items are included in their DDoS document there. Officially you get charged. Unofficially, it depends on the circumstances. There's also a world of difference between a DDoS attack against a fully managed offering (like R53) versus, say, an EC2 instance.
Have you seen a better approach taken by other providers? I don't mean technical approach (which is near impossible to compare other than by track record), but from a policy perspective?
I can confirm this was not a DDoS, and we are actively working to fix the issue right now. We'll be updating our status page as soon as we have more updates to deliver.
I understand, and I don't have an ETA quite yet, but it appears that service is already starting to return. I've tested my domains hosted on Linode and a number of others that I know are and they appear to be working now. We're still working asap to ensure everything is coming back appropriately.
I recently migrated away from using Linode as secondary when they pushed everyone over to Cloudflare. That Internet-destroying company already has enough power!
Not only a gatekeeper, but also a keymaster. (As it were.) Cloudflare's role as a reverse proxy, and as a SSL CA, and as a DNS provider leaves users and webmasters both highly vulnerable to surveillance and/or content tampering by CF.
Not saying that they're doing anything malicious, of course. Merely that it'd be very easy for them to do so.
So far Cloudflare appears to be acting in a non-destroying way. They take as content-neutral a stance as they can, and have had to defend lawsuits as a result of that stance. They recognize that the internet only functions and they can only be part of the infrastructure of the internet if they take that stance.
The greater concern is this: Companies like Cloudflare should not be necessary. They often are. And, with today's internet architecture, companies like Cloudflare must operate at scale to be a useful service with pricing that people can afford.
In a world where we accept the Cloudflare model, we also risk a set of gatekeepers emerging, and that set is "People who can manage terabits/second of inter-network connectivity at PoPs all over the world". It's potentially a troublingly small small group of entities.
A pre-funding project I am working will achieve DDoS resilience without gatekeepers like Cloudflare, if I am able to bring it to market.
It turns out that there's actually a ton of academic research on a number of very promising techniques, in addition to the approaches I'm hoping to develop.
I see the issue more as UI/UX/ease of use/cost of adoption. Right now a lot of approaches you could probably defend a Ph.D. on the topic. They need to instead become approaches developers choose because it's the easiest and most cost-effective way to deliver a high-quality service.
I think content centric networking is very interesting, but my original comment was in the context of TCP/IP as it exists today.
My gut says that real solutions to DDOS problems are going to have to come from new approaches to networking in general and not just tweaks to TCP/IP or throwing massive amounts of infrastructure at the problem.
But I would point out a subtle distinction: I believe TCP/IP and DNS provide an adequate building block for what needs to happen in a backwards-compatible progressive enhancement way.
I distinguish this from IPv6 which - because of how we chose to implement it - has created a ton of chicken-egg issues with the incentives for adoption.
I know there are a lot of sheep to be shorn in the centralized httpasture, but that shouldn't prevent us earnestly appraising how various technologies effect society.
I don't think Bittorrent and IPFS really qualify for decentralisation of random content on public internet. Do you want to advertise to the world that you're distributing the content you found on a random service you visited? You can become responsible for redistribution of copyrighted just because the website includes it.
It could work of course for very restricted applications, like software updates, but general, unfiltered web? No thank you.
Unless you're thinking of lots of content mirroring services where you can publish your mirrored content... but then, what's the difference from cloudflare?
Nothing in the architecture of those protocols requires everyone to republish, especially publicly.
The difference is in the namespace of those protocols. Rather than just reusing L4 addresses, they create addresses that are conducive to decentralization.
The problem with site mirroring services is once again fundamentally comes down to the naming. Either the site operator has to setup and incorporate the mirrors, or site visitors have to discover and visit each mirror manually. There is no middleground where you can ad-hoc casually ask a peer for a copy and have the exact same functionality.
If you skip the P2P aspect of those networks, I'm not sure how much it really differs from distributing HTTP traffic over multiple domain names with possible anycast on IPs. You even get an automatic DNS split per region. What's the reason to go with IPFS / BT for distribution from only the endpoints that you control on the internet? (I do see good reasons for it in private networks)
Yes, it is non-trivial to setup, but if you need it, then automating this part may be more trivial than getting clients to switch to a new protocol.
The pathological case is the same (fill up publisher's pipes -> no publishing), but by giving clients flexibility on how to retrieve, they can set their own policy for how to augment this.
Some possible general alternatives:
- Wait to retrieve from publisher directly, never from third party cloud services
- Ask friends, if this conforms to their privacy goals. Perhaps via store-and-forward mix.
- Ask public altruistic peers.
- Ask for-profit "cache providers".
- Retrieve from multiple independent cloud service(s) using a technique such as PIR
It doesn't really matter who they'll provide service to, it matters how they alter the architecture of the network.
The two ends in a communication are the user and the site publisher [0]. Cloudflare inserts an intermediate proxy. It doesn't matter that it's woven in using IP, just how it doesn't matter that CableCo's proprietary VoD service delivers over IP.
[0] Or even two users wishing to chat or whatever - the same applies to the whole "web 2.0" bubble, but slightly less egregious than Cloudflare.
Note that Destination is the one choosing to replace Level3 with Cloudflare. If Cloudflare ceases to be a good choice, Destination can swap Cloudflare for an alternate provider.
The use case where Cloudflare handles the HTTPS decryption (MITM-as-a-service) is a little bit different. Although it's become quite common, that's not the only way Cloudflare can be used. And I agree that's something where it changes things for the worse in some cases (but if it's a case of "Amazon's machines decrypt it for you" vs "Cloudflare's machines decrypt it for you", I'm less clear that a new problem is being introduced by Cloudflare).
It's not a simple rerouting of a packet. It's a complete L5 decode/proxy/recode, inherently supporting only a few protocols. Putting this kind of functionality into the network is exactly what the end to end principle was reacting against.
(1) Cloudflare does do BGP-based protection (rerouting of a packet) on their enterprise tier.
(2) They really aren't adding functionality into the network. If I put a cluster of Nginx servers in front of my web server, or if I have Cloudflare do it for me, I don't see how that fundamentally changes anything.
(3) The Cloudflare approach of an application layer proxy is feasible for any protocol which can be proxied. I've worked on SMTP proxies which do to SMTP what Cloudflare does to HTTP. Cloudflare also has a DNS proxy. Proxying is a perfectly valid and often incredibly useful technique.
1. Sure, they can do multiple things. When most people talk about Cloudflare, they're referring to the L5 interdiction.
2. You could make a similar argument about network gear having code to facilitate streams. Having a third party provide this functionality is further down the path of centralization.
3. Sure. The point is that to create/tweak protocols, you have to convince Cloudflare (etc) to implement.
It's the end to end principle for a reason. It's not a hard and fast rule, in fact there are many immediate reasons for wanting centralization. It's just unsustainable in the far term.
Unless you want to rely on one company with anycast (like amazon or cloudflare), you can always just keep the same zone in a number of unrelated companies.
48 comments
[ 2.2 ms ] story [ 99.3 ms ] threadDNS Performance Issue Incident Report for Linode Investigating We are currently experiencing denial of service attacks that are targeting our DNS infrastructure. Aug 23, 20:48 UTC This incident affects: Hosted DNS Service.
Edit: Err, not a DDoS attack apparently. But catastrophic nonetheless.
A) Serves DNS from multiple domain names. A number of variants in .com .org and others etc
B) Serves DNS from multiple internet networks / sources. Queries from an end user don't all hit the same Anycast subnet.
For DDoS attacks, I'm not sure what you mean by a "stated policy." They do a lot of countermeasures, some of which can be found via linked PDFs at https://aws.amazon.com/security/ Various R53 items are included in their DDoS document there. Officially you get charged. Unofficially, it depends on the circumstances. There's also a world of difference between a DDoS attack against a fully managed offering (like R53) versus, say, an EC2 instance.
Have you seen a better approach taken by other providers? I don't mean technical approach (which is near impossible to compare other than by track record), but from a policy perspective?
Get it together Linode!
Silly because their backend servers appear to still be on the internet, e.g. ns1.linode.com at 69.93.127.10
They are not doing it for fun. If you think they are bad, give them suggestions or new tech how to solve this problem.
I thank cloudflare for making my life easier.
Not saying that they're doing anything malicious, of course. Merely that it'd be very easy for them to do so.
The greater concern is this: Companies like Cloudflare should not be necessary. They often are. And, with today's internet architecture, companies like Cloudflare must operate at scale to be a useful service with pricing that people can afford.
In a world where we accept the Cloudflare model, we also risk a set of gatekeepers emerging, and that set is "People who can manage terabits/second of inter-network connectivity at PoPs all over the world". It's potentially a troublingly small small group of entities.
I would love to hear about DDOS resilient service architectures that don't require something like Cloudflare.
It turns out that there's actually a ton of academic research on a number of very promising techniques, in addition to the approaches I'm hoping to develop.
I see the issue more as UI/UX/ease of use/cost of adoption. Right now a lot of approaches you could probably defend a Ph.D. on the topic. They need to instead become approaches developers choose because it's the easiest and most cost-effective way to deliver a high-quality service.
Three of the most important: * Blockchains * Protocol-level anti-DDoS changes (protocols specifically designed to make them less suitable for use in in DoS) * https://en.wikipedia.org/wiki/Content_centric_networking
My gut says that real solutions to DDOS problems are going to have to come from new approaches to networking in general and not just tweaks to TCP/IP or throwing massive amounts of infrastructure at the problem.
But I would point out a subtle distinction: I believe TCP/IP and DNS provide an adequate building block for what needs to happen in a backwards-compatible progressive enhancement way.
I distinguish this from IPv6 which - because of how we chose to implement it - has created a ton of chicken-egg issues with the incentives for adoption.
I know there are a lot of sheep to be shorn in the centralized httpasture, but that shouldn't prevent us earnestly appraising how various technologies effect society.
It could work of course for very restricted applications, like software updates, but general, unfiltered web? No thank you.
Unless you're thinking of lots of content mirroring services where you can publish your mirrored content... but then, what's the difference from cloudflare?
The difference is in the namespace of those protocols. Rather than just reusing L4 addresses, they create addresses that are conducive to decentralization.
The problem with site mirroring services is once again fundamentally comes down to the naming. Either the site operator has to setup and incorporate the mirrors, or site visitors have to discover and visit each mirror manually. There is no middleground where you can ad-hoc casually ask a peer for a copy and have the exact same functionality.
Yes, it is non-trivial to setup, but if you need it, then automating this part may be more trivial than getting clients to switch to a new protocol.
Some possible general alternatives:
- Wait to retrieve from publisher directly, never from third party cloud services
- Ask friends, if this conforms to their privacy goals. Perhaps via store-and-forward mix.
- Ask public altruistic peers.
- Ask for-profit "cache providers".
- Retrieve from multiple independent cloud service(s) using a technique such as PIR
They have:
* A reverse proxy service, open to anyone
* A network-level traffic volume management system, open to those that can afford it
I can come up with a ton of scenarios where both of these services actually help with the whole 'end-to-end' issue.
The two ends in a communication are the user and the site publisher [0]. Cloudflare inserts an intermediate proxy. It doesn't matter that it's woven in using IP, just how it doesn't matter that CableCo's proprietary VoD service delivers over IP.
[0] Or even two users wishing to chat or whatever - the same applies to the whole "web 2.0" bubble, but slightly less egregious than Cloudflare.
Or rather, how is Cloudflare making things worse or substantially different?
Example plausible flows in the life of a packet or HTTP transaction:
Before: Packet goes Comcast->Level3->HE->Destination After: Packet goes Comcast->Cloudflare->HE->Destination
Note that Destination is the one choosing to replace Level3 with Cloudflare. If Cloudflare ceases to be a good choice, Destination can swap Cloudflare for an alternate provider.
The use case where Cloudflare handles the HTTPS decryption (MITM-as-a-service) is a little bit different. Although it's become quite common, that's not the only way Cloudflare can be used. And I agree that's something where it changes things for the worse in some cases (but if it's a case of "Amazon's machines decrypt it for you" vs "Cloudflare's machines decrypt it for you", I'm less clear that a new problem is being introduced by Cloudflare).
(2) They really aren't adding functionality into the network. If I put a cluster of Nginx servers in front of my web server, or if I have Cloudflare do it for me, I don't see how that fundamentally changes anything.
(3) The Cloudflare approach of an application layer proxy is feasible for any protocol which can be proxied. I've worked on SMTP proxies which do to SMTP what Cloudflare does to HTTP. Cloudflare also has a DNS proxy. Proxying is a perfectly valid and often incredibly useful technique.
2. You could make a similar argument about network gear having code to facilitate streams. Having a third party provide this functionality is further down the path of centralization.
3. Sure. The point is that to create/tweak protocols, you have to convince Cloudflare (etc) to implement.
It's the end to end principle for a reason. It's not a hard and fast rule, in fact there are many immediate reasons for wanting centralization. It's just unsustainable in the far term.
One assumes some behind the scenes infrastructure exists to push and sync updates to Cloudflare.