I would assume the fastest way to actually make this stop would be to setup a bunch of honeypot exploits, trigger their detection and someone will figure out what they did wrong.
Other than not, with these huge companies you have 0 recourse.
Meta not only hasn't noticed, but is currently sending about 11 requests per second to my site. I've also seemingly trapped one of those TV proxy scraper nets as I'm getting absolutely hammered by requests from all over the place now. I get maybe 10 legit visitors per day, and I'm currently blocking 406,787 IPs from things that have fallen into my honeypot.
I've tweaked my site to return empty status responses a configurable amount of time but the traffic has been so intense that Traefik is now struggling, so I'm going to have to figure out something else. I was returning over-capacity errors and I think that was a mistake, I've swapped to 400 range status codes now. I don't want to use Cloudflare so I'm not sure what to do after this.
The people at these companies are either incompetent or malicious.
Often, the folks you want to ban are not the hosts running the scans.
One's best bet is to play possum, and use your clients last login IPs falling in your service area geo-IP ranges for a firewall white-list. Then redirect the other traffic for a black hole route.
If the nuisance hosts assume they have driven the host offline, they will eventually give up and move on. =3
I wonder if source IP can be ping triangulated, then total ping count displayed as "Tesla MAU: +/-x% today", "Suspected ownership changes this month: xxx cars" by apparent home location changes, then residential and Tesla-unrelated locations excluded server side, and plotted on the map, then finally the whole system exposed to the public Internet and shared to SpaceX fans.
"Hey Texas Model S #345 just left KXYZ, moving at >100mph towards the pad. Everyone get cameras out!"
In case of the very likely title only reader posting:
> Speculation: Assetnote pulled in everything it could find under tesla.com, including pool-ntp.tesla.com, which CNAMEs to pool.ntp.org, which can resolve to my machine — 67.215.249.229. The asset inventory saves this as a Tesla asset, and starts throwing exploits at me, a stranger.
> Not a vuln in Tesla, and I'm not asking for anything, but I just wanted to let you know that you may unintentionally be being a nuisance.
Apparently they didn't just do that, they also had them make requests way more frequently than they needed to (I guess particularly surprising given that they'd already have to be fairly negligent to have the server hard-coded):
> We learned that these packets appeared to be legitimate, well-formed Simple Network Time Protocol (SNTP) version 1 queries, albeit at an inexplicably high rate from each client host. For instance, during one trace, many clients produced about one query per second. This would be highly unusual for a properly constructed SNTP client, since an application which uses SNTP is merely interested in setting its own clock relatively accurately so that its host has some reasonable notion of the current time. One query per second is ridiculous, and is far from best practice for NTP client behavior.
On another note, back when I ran a web hosting business we hosted a few NTP servers in the pool. It’s such a simple thing to give back, and worth anyone who can make a stable contribution doing so.
Note that in the past I've had companies writing embedded linux based firmware using ntppool for time sync request their own vendor zones, however a lot of those requests were ignored so it's unclear if that's still expected. In the end they ended up just using the default ntppool domains since they never got their own vendor zones.
If you do run an NTP server, please make sure it's not vulnerable to DDoS amplification (monlist, readvar, etc need to be disabled) and apply some rate limiting to make it less useful for reflection attacks. And be proactive about monitoring its traffic volume.
If you see high packet rate from a specific IP address or prefix, it's very likely not them abusing your service, but rather you attacking them by responding to spoofed requests.
CNAME'ing pool-ntp.tesla.com to something they do not control is already quite risky as it would allow someone to e.g. request pool-ntp.telsa.com certificate though it might take quite a few tries.
I thought about trying this, but MPIC makes it very very very difficult (the round-robin has some geolocation magic baked in regarding what server it connects you to).
Thankfully it doesn't seem to be much traffic, but still... weird. You'd hope at somepoint the weird responses would get looked at in some log, but I won't hold my breath for that haha.
Tangential, but I love the design of your blog. That's so freakishly accurate to old GNOME 2 Ubuntu, amazing work.
This happens EVERY day to EVERY web server out there. I have a personal site that gets thousands of requests per day from bots.
Running a public server (like NTP) means you will get tons of strange requests. Moreso if you run a web server on the same IP because bots will scrape certificate transparency logs. The entire IPv4 space is scanned continuously.
This may sound harsh, but you cannot stop it. It is whack-a-mole. Filter it and move on, go outside and touch grass, seriously. This is not worth being upset over.
I treat these as an opportunity to tune my filters and firewall rules.
You don’t think there’s a difference between “hackers try to attack everything“ and “Tesla decided that I personally need to be tested as one of their systems due to a lazy misconfiguration“ are different?
People are trying to compromise Tesla and because this guy provides NTP services, and Tesla set their NTP up wrong, it appears to other people like his machine is part of Tesla.
Got it. The way it was phrased it made it sound like Assetnote was the party actually sending the exploits (which made me assume it was intentional testing going to the wrong target), not that they were coming in from unknown senders in the outside world.
As a bug bounty researcher, my systems would do the same thing if they ended up georouted to this IP. *.tesla.com is marked as in scope on https://bugcrowd.com/engagements/tesla, and my agents will probe anything under there as it is presumed to have explicit authorization.
Not sure if there is a great solution, but I'm inclined to say that attack traffic like this is the new normal. In fact, the attack volume they got is quite small compared to the volume I have seen on other tech company subdomains - the new normal is probably much worse.
So many of the news on HN, such as present one, can be actual stories/scenes from a cyberpunk game/movie these days, that we can safely assume this (otherwise imaginary) future has already arrived.
I'd try to contact Assetnote. Most (sadly not all) managed vuln scan companies are pretty sensitive to scanning stuff that doesn't belong to their client and could expose them to liabilities because they don't have permission.
I've been consistently attacked by ShadowServer who have the following sponsors,
Akamai,
APNIC Foundation,
Arctic Security,
AusCERT,
Avast,
Backblaze,
Canadian Center for Cyber Security,
CERT.AT,
CERT.br,
CERT.LV,
CIRA,
CIRCL,
Craig Newmark Philanthropies,
CSIRT.LI,
CSIS Security Group,
DFN‑CSIRT,
Digital Trust Center,
EURid,
HelseCERT,
ICANN,
Identity Digital,
KPN,
Mastercard,
NASK (CERT.pl),
NCSC Ireland,
NICS,
Nihon Cyber Defence,
Nucleus Security,
Orange Polska,
Precursor Security,
Protect.ngo,
Public Interest Registry (PIR),
Red Hat,
SURFcert,
SWITCH,
Team Cymru,
Trend Micro,
Trivest AG,
Tucows,
Verisign,
VulnCheck,
I don't care what they say they're doing, I hate how corporations can act with impunity with these types of things while everyone else would get a felony for it.
Calling vuln scanning from a non-profit a felony is a bit of a stretch. Many for-profit companies do similar vuln scanning and then threaten companies with security "scorecards" that is borderline extortion.
If the non-profit was walking down the road and rattling everybody’s door lock to see which are unlocked, and having a look around the windows to see if any are open, would that be a crime?
Because that is exactly what all of these vulnerability scanning companies are doing, and all of us sort of just… let them.
I just checked my own webserver logs (I also run a sever in the ntp pool) and I too see some hits in my webserver logs.
They look to all be log4j vuln scanning activity (CVE-2021-44228), and the volume isn't that high (a few a day, and not every day). They just have some overzealous vuln scanning. And yes, they shouldn't have the NTP pool under their DNS name.
I've had all sorts of strange things happen because of my ntp pool membership, this one is pretty far on the benign end of things.
56 comments
[ 4.4 ms ] story [ 71.4 ms ] threadOther than not, with these huge companies you have 0 recourse.
Meta not only hasn't noticed, but is currently sending about 11 requests per second to my site. I've also seemingly trapped one of those TV proxy scraper nets as I'm getting absolutely hammered by requests from all over the place now. I get maybe 10 legit visitors per day, and I'm currently blocking 406,787 IPs from things that have fallen into my honeypot.
I've tweaked my site to return empty status responses a configurable amount of time but the traffic has been so intense that Traefik is now struggling, so I'm going to have to figure out something else. I was returning over-capacity errors and I think that was a mistake, I've swapped to 400 range status codes now. I don't want to use Cloudflare so I'm not sure what to do after this.
The people at these companies are either incompetent or malicious.
Might make them scan themselves instead.
One's best bet is to play possum, and use your clients last login IPs falling in your service area geo-IP ranges for a firewall white-list. Then redirect the other traffic for a black hole route.
If the nuisance hosts assume they have driven the host offline, they will eventually give up and move on. =3
"Hey Texas Model S #345 just left KXYZ, moving at >100mph towards the pad. Everyone get cameras out!"
It'll be gone by lunchtime that day.
> Speculation: Assetnote pulled in everything it could find under tesla.com, including pool-ntp.tesla.com, which CNAMEs to pool.ntp.org, which can resolve to my machine — 67.215.249.229. The asset inventory saves this as a Tesla asset, and starts throwing exploits at me, a stranger.
> Not a vuln in Tesla, and I'm not asking for anything, but I just wanted to let you know that you may unintentionally be being a nuisance.
https://www.google.com/search?&q=university+ntp+server+netge...
https://pages.cs.wisc.edu/~plonka/netgear-sntp/
I love articles like that.
> We learned that these packets appeared to be legitimate, well-formed Simple Network Time Protocol (SNTP) version 1 queries, albeit at an inexplicably high rate from each client host. For instance, during one trace, many clients produced about one query per second. This would be highly unusual for a properly constructed SNTP client, since an application which uses SNTP is merely interested in setting its own clock relatively accurately so that its host has some reasonable notion of the current time. One query per second is ridiculous, and is far from best practice for NTP client behavior.
Neat! Didn’t know about this command that’s very helpful
If it were 8000 requests per second, this might be worthy of some investigation.
But 8000 ntp requests alone consume far less than 1 us cent of compute + bandwidth. This isn't worth lifting a finger over.
This is standard bot crawler traffic. Anyone who runs a home server sees attempts to load wp paths all the time
The way a vendor embedding NTP is _meant_ to do so is documented here: https://www.ntppool.org/en/vendors.html
On another note, back when I ran a web hosting business we hosted a few NTP servers in the pool. It’s such a simple thing to give back, and worth anyone who can make a stable contribution doing so.
Note that in the past I've had companies writing embedded linux based firmware using ntppool for time sync request their own vendor zones, however a lot of those requests were ignored so it's unclear if that's still expected. In the end they ended up just using the default ntppool domains since they never got their own vendor zones.
If you see high packet rate from a specific IP address or prefix, it's very likely not them abusing your service, but rather you attacking them by responding to spoofed requests.
When was this page last updated? I would expect that number to be in the hundreds of millions these days, perhaps even billions.
Maybe running a web server on the same IP as an NTP server is a bad idea.
Tangential, but I love the design of your blog. That's so freakishly accurate to old GNOME 2 Ubuntu, amazing work.
This happens EVERY day to EVERY web server out there. I have a personal site that gets thousands of requests per day from bots.
Running a public server (like NTP) means you will get tons of strange requests. Moreso if you run a web server on the same IP because bots will scrape certificate transparency logs. The entire IPv4 space is scanned continuously.
This may sound harsh, but you cannot stop it. It is whack-a-mole. Filter it and move on, go outside and touch grass, seriously. This is not worth being upset over.
I treat these as an opportunity to tune my filters and firewall rules.
ref: https://www.ntppool.org/en/vendors.html
Not sure if there is a great solution, but I'm inclined to say that attack traffic like this is the new normal. In fact, the attack volume they got is quite small compared to the volume I have seen on other tech company subdomains - the new normal is probably much worse.
Akamai, APNIC Foundation, Arctic Security, AusCERT, Avast, Backblaze, Canadian Center for Cyber Security, CERT.AT, CERT.br, CERT.LV, CIRA, CIRCL, Craig Newmark Philanthropies, CSIRT.LI, CSIS Security Group, DFN‑CSIRT, Digital Trust Center, EURid, HelseCERT, ICANN, Identity Digital, KPN, Mastercard, NASK (CERT.pl), NCSC Ireland, NICS, Nihon Cyber Defence, Nucleus Security, Orange Polska, Precursor Security, Protect.ngo, Public Interest Registry (PIR), Red Hat, SURFcert, SWITCH, Team Cymru, Trend Micro, Trivest AG, Tucows, Verisign, VulnCheck,
I don't care what they say they're doing, I hate how corporations can act with impunity with these types of things while everyone else would get a felony for it.
Because that is exactly what all of these vulnerability scanning companies are doing, and all of us sort of just… let them.
They look to all be log4j vuln scanning activity (CVE-2021-44228), and the volume isn't that high (a few a day, and not every day). They just have some overzealous vuln scanning. And yes, they shouldn't have the NTP pool under their DNS name.
I've had all sorts of strange things happen because of my ntp pool membership, this one is pretty far on the benign end of things.
As opposed to intentionally being a nuisance?