HN Explorer

Why are we being DDoSed by Cloudflare?

118 points by cpncrunch ↗ HN
Earlier today we experienced a DDoS HTTP attack, which was automatically mitigated by OVH, so only caused minimal disruption. However it's concerning that it happened at all, as all the ips were Cloudflare ips, e.g.:

162.158.202.23 - - [01/Dec/2023:23:30:02 -0500] "GET / HTTP/1.1" 301 244

162.158.202.23 - - [01/Dec/2023:23:30:02 -0500] "GET / HTTP/1.1" 301 244

172.69.90.116 - - [01/Dec/2023:23:30:02 -0500] "GET / HTTP/1.1" 301 244

108.162.221.143 - - [01/Dec/2023:23:30:02 -0500] "GET / HTTP/1.1" 301 244

162.158.235.85 - - [01/Dec/2023:23:30:02 -0500] "GET / HTTP/1.1" 301 244

162.158.110.143 - - [01/Dec/2023:23:30:02 -0500] "GET / HTTP/1.1" 301 244

162.158.110.142 - - [01/Dec/2023:23:30:02 -0500] "GET / HTTP/1.1" 301 244

162.158.202.23 - - [01/Dec/2023:23:30:02 -0500] "GET / HTTP/1.1" 301 244

162.158.202.22 - - [01/Dec/2023:23:30:02 -0500] "GET / HTTP/1.1" 301 244

162.158.202.23 - - [01/Dec/2023:23:30:02 -0500] "GET / HTTP/1.1" 301 244

(it was around 5800 requests per second to a site that normally gets 1 request per second or less at this time of day, and lasted 21.5 minutes).

As it is http, I don't think it could have been spoofed. I tried contacting cloudflare, but they are impossible to contact unless you are a customer. All their social media chat just uses bots and it is impossible to connect to a human if you're not a customer. I tried calling their emergency DDoS line, but the person just said that cloudflare doesn't DDoS people (even though I explained that the attack definitely came from their network, and someone was likely using their service to DDoS us). They simply refused to direct me to anyone else in their company. The phone number is just for people looking to purchase DDoS protection, and they don't have any other method of contacting them.

So, can someone explain why Cloudflare seems to be DDoSing us, and exactly how it happened?

There is no way to report abuse to Cloudflare unless it is a cloudflare website. (I tried, and it refuses to submit the report). I suspect it may be someone abusing their WARP VPN service, but there doesn't seem to be any way of reporting abuse about it.

52 comments

[ 5.3 ms ] story [ 130 ms ] thread
philprx
Well they certainly can’t feel good about not detecting egress DDoS from their customer usage.

It could be VPN originating or maybe Worker originating maybe?

Don’t they have a security.txt and security@ email address?

Looks like posting to social media the full analysis of the attack is the sensible thing to do.

cpncrunch
>Don’t they have a security.txt

Yes, but it just covers bug bounties and also links to the abuse page (which only accepts websites hosted on cloudflare).

cookiengineer
Check out the crimeflare project (not the firewall thing).

A lot of hacking groups, terror organizations and other malicious actors have been using cloud flare for a while without them doing shit about it.

It's their business model. More DDoS means more cloudflare customers, yaaay.

A guy ran a DNS and logged all the suspicious domains linking to cloud flare (e.g. cname entries etc): he eventually gave up cause he was sued into oblivion (he was a Swiss guy operating from Switzerland).

http://web.archive.org/web/20210826102143/http://www.crimefl...

And this kinda speaks for itself:

http://web.archive.org/web/20210826102230/http://www.crimefl...

clwg
I don't believe these groups should have free rein on the internet. However, I am also not in favor of Cloudflare acting as the police of the internet. Ideally the internet shouldn't be so centralized around Cloudflare as it is, while the regulations and laws that would govern such actions to take care of the problem need to be put in place.

It's a tricky situation, and whether or not Cloudflare endorses, facilitates, or turns a blind eye to DDoS attacks shouldn't matter as much as it currently does.

edent
They don't have to be the police. Cloudflare are a private company, not a public utility. They can deny service to anyone they want (within the bounds of the law).

If they want to be a neutral public utility, they can rearrange their business to that effect.

Joel_Mckay
No, as a business they are technically liable for their users illegal theft of services once notified of the problem.

i.e. unless they want to host in a lawless foreign data-center begging for scrip-coin they should care.

=)

cookiengineer
> lawless foreign data-center

Anecdotal forest city / woodlands in Malaysia / Singapore note. The territorial dispute led to most organizations there doing shady business online, as nobody is ruling these lands, effectively.

Joel_Mckay
Singapore is hardly a place anyone would want to get in trouble, and visitors are specifically warned some folks have weaponized the most punitive aspects of local laws.

i.e. if you aren't careful and someone drops a pill in your luggage, they do shoot people without trial on the following Tuesday.

And yes, having stuff reverse engineered at the local university care of an unscrupulous US company is common. One really can't take it personally, as it is legal there. =)

clwg
They can also allow service to anyone they want and accept the liability, whatever that may be for doing so, which is what they are doing.

I'm not disagreeing with you, or trying to defend Cloudflare – I just see a very complicated situation, especially when you consider the DNS ecosystem, which in itself is extremely shady. You also run the risk of DMCA-style procedures that would require equal application across service providers and would undoubtedly be abused by bad actors.

layer8
They shouldn't be acting as the police, but they should have an effective procedure to report abuse coming from their IPs and having it handled.
[deleted]
(comment deleted)
ThinkBeat
Cloudflare is SIGIT not law enforcement.

All that delicious data to snoop at and share with the government, or possibly UK in order to avoid legal problems.

It would be difficult to build a better listening station for the Internet.

Given that Google,FB,etc are all open silos for the spooks as well.

The government has even declared that the US near monopoly is of vital national security importance.

(This was in regard to stop the administration from pushing for a breakup of the internet giants).

greyface-
I think it's more complicated than "open silos" - there's an element of denial that enables this. My experience is that many people working in these organizations are blind to the realities of where their companies are situated, and the level of interest intelligence agencies have in them, and get upset and defensive when you suggest building internal countermeasures.
WhyNotHugo
> I don't believe these groups should have free rein on the internet.

I agree, we need strong legislation to cut down on Cloudflare's activity.

Joel_Mckay
Started preemptively black-listing many cloud blocks years ago due to scans, scams, and spam.

Cut down a further 23% of traffic annoyances, and booted many VPN users who never seemed to participate in a constructive manner anyway.

Play possum with a temporary route ban, it is usually the only way they will move on to find other targets. If the server appears already down for days, they will get bored. =)

EdwardDiego
That website was 2 "under construction" GIFs away from being a 90s conspiracy website.

I'm really not sure what you thought those links proved.

cookiengineer
> I'm really not sure what you thought those links proved.

1. Pick any domain from the website

2. dig A <thedomain>

3. dig CNAME <thedomain>

4. ???

6. Proof

You know, it's not like how to DNS is a secret or something. You can easily confirm this yourself whether or not the author of said article told the truth or not.

Cpoll
> And this kinda speaks for itself

That's just a list of URLs with the substring "hack" in them.

URLs like ryansugarshack.com (Ryan's Sugar Shack: Taste of New Hampshire).

solardev
If you put your own website behind Cloudflare while it's under attack from Cloudflare... does the internet explode?

OP you should try it and see what happens. It's only like $20 for the basic plan, and then you can ask their support once you log all the attacks coming from themselves to themselves...

foothebar
$20 to stop a big tech company allowing attacking you from their network they are responsible for?
toyg
It's one of the oldest business models in the world, something about racquets... /s
traveler1
The free plan would cover this sort of attack.
rollcat
The problem is you are asked to move your DNS SOA/NS to CloudFlare, they won't let you just slap a CNAME/A+AAAA, and that could be disruptive to your setup - there's no single standard for managing DNS programmatically, many DNS vendors provide extensions (such as GeoDNS, AWS "ALIAS" records, fallback) etc.

I've done my share of nasty DNS hacks to make a customer happy; even without hacks you do want some order in your setup so you'll end up with a huge piece of Terraform or a glued-together Python script. Changing DNS providers can be extremely painful.

relatedtitle
Are you sure this is not from their Cloudflare Warp service (VPN)? If it is, and you are a CF customer, you can see the real user's IP from a header.
Am4TIfIsER0ppos
How do you think they get people to sign up for their services?
buggeryorkshire
At a previous role we had our origin servers attacked by Cloudflare IP ranges even though we were locked down behind them. Seemed at the time anybody could spin up a CF worker and bypass origin restrictions.
georgyo
More than likely someone with a high traffic site fat fingered an IP address and sent all the traffic to you.

I am curious if you were able to capture the headers at all. The 'Host' header would allow you to figure out the site these people were actually trying to hit. I'm fairly confident it was not your domain. You may want to to put the Host header in your access logs. This may also explain why everyone got a 301 and using http.

No matter what you should be able to see this header 'Cf-Connecting-Ip' to know the true source.

While cloudflare is somewhat masking the origin IPs, a similar mistake without cloudflare would send at least the amount of traffic towards you.

hartator
Yes, I agree.

And if all requests are from the homepage, why not adding a header to ask Cloudflare to cache it for 1 day?

cpncrunch
That doesn't seem like a solution to a DDoS attack that is automatically mitigated .
freitasm
Yes, very likely. The website owner might not even realise if they use more than one IP address, which are then presented in a round-robin way to clients, so errors loading the pages aren't easily spotted.

If you find out the hostname used you could try contacting the website owner and check with them, bypasing Cloudflare (assuming you could either load the site and use a contact form, or find registrant's information).

cpncrunch
I think that is unlikely, as the logs only show the 301 redirect, but not the request to load the site itself. And it only ever loaded the main home page (/).
tyingq
The other posts may be correct in that someone with a cloudflare account accidentally pointed a service at your host.

That said, it is a pretty attractive platform if you want to build a scraper that operates either at no cost, or very low cost. Especially if you can aggregate multiple free level accounts without Cloudflare catching on. And it's pretty easy to write a bad scraper that handles things like redirects poorly, and becomes an unintentional DDOS machine.

Dumping the headers of some of the requests could help narrow down which type of Cloudflare service is hitting you.

cpncrunch
There is no way to get headers now unfortunately. Even during an attack it would be difficult, as it is mitigated by ovh so quickly.
injeolmi_love
This is one of Cloudflares well documented tactics to force people to purchase their subscription services. I suspect but have no proof that it is overlooked by the authorities because an internet centralized under Cloudflare is easier to censor.
victorbjorklund
Why would cloudflare ddos you with their own servers? If they actually wanted to attack random sites to get them to become customers (very much doubt that conspiracy) they probably would use someone elses IP:s.
injeolmi_love
Deniability. The executives can claim third parties are abusing the platform while simultaneously enabling the abuse. Whether or not that’s true, the outcome is the same; cloudflare’s actions force smaller operators to switch to cloudflare.
victorbjorklund
So they switch to cloudflare free tier and uses cloudflares stuff for free.
jdefr89
Security Researcher here. There could be lots of reasons for this. Do you have any more header information? It is hard to tell with what you have provided. Either way I would still report it to them so they can take a closer look.
cpncrunch
No, that is the only info recorded (I just looked in all the apache logs for some of the ips).
neurostimulant
Is it possible to abuse Cloudflare Workers to perform ddos on the cheap?
oneplane
Hanlon's razor suggests either human error (someone putting in the wrong DNS entry on their end) or someone abusing a cloudflare service.

All the other suggestions of malice are rather inefficient, especially considering the net value is lower than the cost to push those malicious actions.

As for the reporting: yeah, it seems rather dumb there is no easy to reach form for this, but you can message their abuse address and NOC which are listed in the WHOIS you probably queried anyway to find out Cloudflare owns those addresses:

  RNOCHandle: NOC11962-ARIN
  RNOCName:   NOC
  RNOCPhone:  +1-650-319-8930 
  RNOCEmail:  noc@cloudflare.com
  RNOCRef:    https://rdap.arin.net/registry/entity/NOC11962-ARIN

  RAbuseHandle: ABUSE2916-ARIN
  RAbuseName:   Abuse
  RAbusePhone:  +1-650-319-8930 
  RAbuseEmail:  abuse@cloudflare.com
  RAbuseRef:    https://rdap.arin.net/registry/entity/ABUSE2916-ARIN
A phone call might be quicker if you're really curious about what is happening.
sdflhasjd
The phone number is listed on CloudFlare's website as their international sales number, so I am skeptical that it goes anywhere useful to OP
oneplane
When I used to call them I just asked for the NOC and they connected me to the NOC first line handler. Might be different now (~3 years difference).
anonymousiam
I'm surprised that you could reach a NOC as recently as three years ago. A few decades ago, it was fairly easy to contact a NOC. Nowadays many NOCs don't even allow phones that are reachable from the PSTN. This is done to increase "security" by making social engineering attacks impossible.
cpncrunch
Yes, I just tried, and the phone number just goes to a menu system with no way to talk to a human. If you choose abuse, it tells you to report it online. If you choose support it tells you that if you're an enterprise customer you can create a ticket. If you choose the "under attack" option, then do send you to a human who tries to sell you cloudflare service and doesn't understand that cloudflare could possibly be attacking you (and refuses to transfer you).

I tried emailing the abuse address, but I get an automated reply saying to report it using the online form (which won't let you report anything not hosted on a cloudflare website).

I also emailed their noc address and haven't had an automated reply yet, so perhaps that will actually reach a human.

oneplane
As far as I could tell, the handler wasn't inside the NOC, but was able to contact them directly. I suppose 1st and 2nd level don't actually have to be inside the NOC to be of use.

Wouldn't be surprised it they do some sort of CLIP filtering/routing, or if they no longer use humans on the front lines, it's the way of the enterprise once they exist long enough :-(

quags
2-3 years ago in a support role I would get requests about domains masked cloudflare pointing to someone’s site. So the other domain would show the content of the other site and be masked by cloudflare. Dumping all the $server variables would pinpoint better where it was coming from and it could be blocked.
cpncrunch
I think it's unlikely to be this because it's logging the 301 redirect (which redirects a different page of our site), but nothing after that.

Not really possible to get any more info after the fact.

r1ch
Is your DNS on Cloudflare? Their network will proxy requests to your domain even if you don't use their proxy service. You have to add a WAF rule to prevent this (I also got DDoSed this way).
ravenstine
Yes, we are being Dados attacked by Cloudflare due to how they've made over half the web impossible to use without enabling browser anti-features to support fingerprinting and telemetry.