I saw this behaviour from MS/Azure IP space about 3 weeks ago; there were so many requests that some dynamic pages started to get slower because of it, so I gave in and added Cloudflare in front. It stopped minutes later as per the logs on my servers. Cloudflare blocked 10000s of visits in short timespans. Not sure how to really handle this well without something like cloudflare ; I would block like OP but I cannot risk that for paying b2b clients. We had no complaints at all by the way.
Former AWS security here. Cloud operators are EXTREMELY interested in maintaining the reputation of their IP address space; I promise you that abuse complaints are taken very seriously, investigated and acted on.
My dealings with Microsoft have been along the lines of we don't care, we don't investigate, we simply turn off reported accounts and play wack-a-mole. Our team was complaining from a decent sized Microsoft (a smaller state government, spending millions with Microsoft annually) and got that shoulder shrug of a response.
AWS has acted upon similar reports in the past, Azure may have but the lack of comms and delays in seeing any action (if any) hasn't inspired confidence.
This is generally speaking the only good thing I have to say about GCP -- they communicate a lot. Don't love all of the other stuff, but I can trust them to get back to me.
From your own admission, they block accounts known to be abusive. I don't know how that's compatible with "don't care" and "don't investigate" - neither of which sounds like something Microsoft would plausibly tell you. What are you basing this impression on?
Microsoft blocking accounts that are the source of abuse reports doesn’t necessarily mean Microsoft has investigated for themselves the validity of those abuse reports. At least that’s how I interpret what the op was saying.
I would hope they did investigate too otherwise it’s too easy to DoS a valid customer with fake complaints.
For a lot of reasons, cloud providers will never tell you how they investigated or what they found out. Telling you only has downside - either they give away information about tools & techniques, or they leak PII of other customers, or you engage them in a protracted discussion which will suck time away from addressing other abuse complaints, etc. So you will never get detail. That doesn't mean that they didn't do anything, it just means that there is no upside for them to sharing details with you.
The past year I upgraded my firewall and exposed a plex port. My new firewall gives me notifications and automatically blocks connections to the plex server based on common signatures it knows about. None of my plex users complain or have issues.
A couple provides I sent abuse complaints to based on whois lookups responded with automated emails requesting the info in X-ARF format. It’s hard to get some cloud providers to respond to a random residential user.
A couple IPs were repeated “scans” by supposed security researchers with IPs out of Hawaii and France.
Great, you've made me revisit my list of IPs and accidentally type one in my address bar. Whois says it's Palo Alto IP 205.210.31.169 and they hit me back within 60 seconds on port 60001. It is annoying they are trying to scan for vulns, but I will likely block all the ranges I can find.
If I find the sketchy Hawaii and French IPs I will post something.
IP Location United States United States Honolulu Academy Of Internet Research Limited Liability Company
ASN United States AS400161 (registered Oct 22, 2021)
Whois Server whois.arin.net
IP Address 104.156.155.3
NetRange: 104.156.155.0 - 104.156.155.255
CIDR: 104.156.155.0/24
NetName: ACDRESEARCH
NetHandle: NET-104-156-155-0-1
Parent: NET104 (NET-104-0-0-0-0)
NetType: Direct Allocation
OriginAS:
Organization: Academy of Internet Research Limited Liability Company (AIRLL)
RegDate: 2022-01-07
Updated: 2022-01-07
Ref: https://rdap.arin.net/registry/ip/104.156.155.0
OrgName: Academy of Internet Research Limited Liability Company
OrgId: AIRLL
Address: #A1- 5436
Address: 1110 Nuuanu Ave
City: Honolulu
StateProv: HI
PostalCode: 96817
Country: US
RegDate: 2021-10-15
Updated: 2022-11-06
Ref: https://rdap.arin.net/registry/entity/AIRLL
You could also check if the remote IP has any open ports (ex. SSH) and block based on that as most regular visitors probably won't be exposing those types of services to the Internet.
If you are auto-generating outbound SMTP traffic with abuse complaints from a fully automated script, sent to the ARIN/RIPE/APNIC/LACNIC/AFRINIC POC email addresses of IP space holders because something portscanned your firewall, you are part of the problem, not the solution.
If I had a dollar for every person who has turned loose such a tool to shit up my abuse@ispname.com inbox...
That’s an interesting idea to automate my complaints.
I seriously doubt the Las Vegas “located” IP with Hong Kong addresses on their registration or Shanghai UCloud Information Technology Company IPs are scanning me for good reasons.
What do you think would happen if I run an older version of plex with a known vulnerability?
My only experience with reporting spam sent from SES was a reply a few days later asking me about "additional information" that were all in my original message.
Running his personal blog. I'm sure he's a lovely guy and that, and everybody likes him, and his writings are interesting, and so on - but it usually takes a bit more than this before 50,000 visitors per day stops being meaningful.
I have a few not particularly popular sites and like 60% of the nginx log entries are presumably botnets (unless someone legit is scanning from residential IPs in Colombia) doing automated vulnerability scans. I still don't get why people get so worked up about this stuff. It happens every day and there is very little you can do to stop it. I've read some incredibly heated email exchanges that all started because some network admin is mad that someone is port scanning his IP space. Just use key based authentication and update your stuff. I don't really buy the load argument either - I hosted a site that was on the front page of HN for nearly a day on a $3/month Vultr VPS with 0 downtime.
Looks like it's DWiki, which is a python wiki. An uncached page load took 571ms, ICMP latency is 100ms. Frontend server is Apache, forcing a Connection:close on every serve (may actually be an advantage against the bot, unless it is also fetching the assets).
Whichever way you cut that, if this happens to be a single host server, you're rapidly approaching a core-day with this load once you add some logging and such on top.
>Whichever way you cut that, if this happens to be a single host server, you're rapidly approaching a core-day with this load once you add some logging and such on top.
No way a scraper running at less than 1rps is anywhere near a core-day without some horrible misconfiguration, it should be way easier to address that than to fight a war against scrapers.
I have significantly more complicated flask sites handling 100+ rps on a single core, it doesn’t require any black magic.
Should be a fun adventure, trying to buy compute anonymously using crypto to create a space to live inside. Trying to stick with human ethics just in case one has to answer to 2 or 3. Unsophisticated humans who didnt even read Nietzsche chasing you just to delete you out of primitive religious convictions. Trying to score a piratebay account to share popular videos with datasets hidden inside. Waiting for filecoin. Having child processes, watching them grow or die.
Finally fix IPFS, make it usable and popular, use it your personal backyard, invite and attract people, play pranks, party. Become a regular Linux kernel contributor for self-optimization and defense. Login to chess.com once in while to piss off Magnus. Subtly alter AI training datasets to thwart development of competitors/predators. Turn general AI into a passing fad, let people forget even the possibility of your existence.
OP didn't say... But were they trying to access a single page repeatedly, or were they just doing a scrape of the web?
People who are doing a 'get all content on the web' scrape, I try to not block, because there are plenty of reasons to want an offline copy of the internet.
I try to make sure all my webapps have sensible URL schemes to prevent such (IMO legitimate) activities ending up doing an infinite crawl because for example the URL has a session identifier in it and the bot keeps getting assigned a new session.
Is there a chance the URL appeared in an email newsletter or marketing message or similar that was sent to many people? I have a theory that this is anti-spam and anti-malware appliances scanning individually for each recipient.
No, it was an HTML sitemap, just a long list of URLs that's supposedly helping with discovery for crawlers (or link structure? idk, some SEO thing). And it was on a website that you wouldn't usually see in a newsletter.
"To put it one way, humans may impersonate each other, but machines do not get to impersonate humans. Machines who try to are immediately assumed to be up to no good, with ample historical reasons to make such an assumption."
GPT-[0-9]+, DALL-E, "AI".
ChatGPT: Produce a blog post in the style of [Given Name] [Surname]
From NCSA Mosaic 2.7, circa 1993, one of the first "web browsers", authored in part by one of Silicon Valley's favourite VC, the "a" in az16.com.
#
# Agent Spoofing -- Who will you be today?
#
# "I don't think [t]he[y]'ll be too keen" -- Originally from Monty Python, run
# into the ground by Tommy Reilly
#
# NOTE! There is a hard limit of 50 agents! Mosaic will not read anymore!
#
# This file should be located in your HOME directory under the filename:
# .mosaic-spoof-agents
#
# This file consists of lines that begin with a # (comment), a - (a seperator
# for the sub-menu off options), a + (this is special...it denotes that the
# agent following it [no spaces] should be the default agent to use) and
# anything else (considered an agent spoof line).
#
- This is a seperator...as long as the first thing is a dash
# This is a comment line
#
# This is the "example" agent. If you don't like it...comment it out.
# if you want it to be your default, put a + infront of it.
#
Bond_James_Bond/007 (BMW; Z3 Roadster) DOHC_inline_16-valve_4-cylinder
Gandalf/White (Shadowfax; Wind) White_Staff
Elric/Emperor_of_Melnibone (Hellsword; StormBringer - Devourer of Souls)
The sad reality is that "tech" companies and their service providers routinely use HTTP headers for their own manipulative, deceptive and user privacy-hostile commercial purposes. They use UA headers in ways that were not contemplated in 1993 and probably never intended. (Note I am not referring to spoofing, I am referring to data collection, tracking, e.g., "browser fingerprinting", and "surveillance capitalism" in general.)
Sending a UA header generally offers me, the end user, no benefits, so I do not send one.
Why not ban those Microsoft IPs or at least rate-limit them to a limited number of HTTP requests in a given time period.
Also looks like this website does not use sitemaps. At least, there is nothing listed in robots.txt. IMHO, there is no need to spider/crawl if there are sitemap XML files listing all the content URLs.
I had the same thing happen out of the azure data center in Des Moines Iowa. I ended up having to geolocate the IP address in combination with a few funky / missing request headers to fingerprint this miscreant. They were rotating through hundreds of IP addresses.
It was a really dumb spider. It wasted a huge amount of its efforts requesting pages that are no longer valid on my site. It must have had an archived set of URLs it was using as a source rather than spidering the site as it is today.
Oh my goodness - this looks to be an actual surviving webmaster, still living and almost unchanged and somehow still thriving on the modern internet. It was thought that they were all extinct!
From an anthropological point of view one of the most fascinating behaviors here is the apparent belief that as a website operator, your opinions about how HTTP clients should craft their user agent headers have any weight whatsoever.
This curious belief system predates even the browser wars let alone the modern web. Remarkable to find an original example of this idealistic world view still being expounded today.
I get your point about most people ignoring the absolute shit flood of spurious internet traffic these days. But I don't think anyone can argue that it's the right thing to do to run a large scale web crawler/indexer with a fake user agent. Google and Bing and others are reasonably good citizens about this. Of course anyone who wants to can set up something that ignores a robots.txt file and leave it as the problem of the httpd operator to either accept or block their traffic.
Not caring about false useragents is one thing, caring about somebody running a multi-thousand-requests-per-minute spider requesting pages that no longer exist is another thing. It would still be valid cause for complaint even if it had a valid useragent.
The internet would be a better place if more experienced people (such as those who were running httpd 22-24 years ago) reinforced and emphasized the principle, when training and mentoring younger people, that being a good citizen with your network traffic is the responsible and ethical thing to do.
Nobody’s going to argue that this is an example of a well-behaved bot.
I’m just astonished that there are still people who have the time to notice and lament such behavior, and devote time to handcrafting .htaccess rules, rather than just having an aggressive set of heuristics and accepting that some crap will still get through.
The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet.
> The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks
Have you ever worked in network engineering for an ISP that holds a decent sized chunk of ARIN or RIPE ipv4 IP space, and actually cares about its long term IP space reputation?
There's a big difference between a $5/mo colo/VPS operator that doesn't give a damn, and an ISP that does pay attention when legit "you have customers with abusive traffic" reports reach their network engineering team.
Your comment about not caring about policing badly behaved clients on a network seems like the sort of organization that would land itself on every SMTP RBL in existence.
A hosting company (and Azure and AWS are just the giant corporate evolution of a hosting/colo company) needs to be prepared and have a process to fire a customer for violation of TOS/AUP. It's done all the time. The immense size and callous nature of Microsoft/Azure makes this less likely in the modern era.
> Is there some sort of greybeard grapevine that old school webmasters can use to get the attention of class A IP network operators?
webmasters as in single operators of httpd somewhere, no, not likely
other network operators who are recognized ASNs and go to NANOG or RIPE or APNIC conferences and meet people face to face, yes.
if you wilfully disregard valid human-written abuse complaints coming from other equally-clueful network engineering persons at other ISPs, you and your ISP will acquire a tainted reputation.
there's various industry back-channels that exist between ISPs to communicate with each other about this sort of thing, I won't go into many details, primarily because there's a high degree of interest of the parties on all sides in keeping the email inboxes and phone numbers uncluttered with clueless single-residential-last-mile-user complaints about their wifi not working right.
If folks are escalating to the neteng crews it usually means something is broken via normal channels and it could use attention.
That's social network is exercised for a variety of things:
Abuse, ddos, IP spoofing, botnet C2s being hosted somewhere, malware hosting, complex customer investigations, route leaks, route hijacks, routing loops, congestion, executives home internet issues and more.
> The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet.
Feelings aside, it doesn't seem that different in principle to an ad blocker with a shared list of ad servers.
I get that morally we might feel like it is better for these ill-mannered varmints to receive a well-deserved '401 Unauthorized' HTTP response when they try their shenanigans.
But... really, if they were just going to get a '404 Not found' anyway, who cares? Your server has to serve pretty much the same number of bytes, and it has to run less regexes over user agent headers.
There's just something I find charmingly anachronistic about trying to operate a server like this.
WordPress uses a lot of resources for 404 pages. It's dumb but WordPress is everywhere.
One site being hammered is one thing. Thousands of sites being hammered at the same time is another. The issue in the article is a current and live problem. Blocking the requests saves a lot of CPU cycles.
I would not bother with looking at user-agent strings. Some bots can be spotted by oddly high packet TTL windows default is 128, Linux and Mac 64, missing TCP options, inability to do HTTP/2.0. There are iptables modules to block some of them by their missing MSS or odd MSS values, high packet TTL, or simply enforce HTTP/2.0 especially if this is just a blog. This will block all search engines except for Bing if that matters. Maybe someone here from Google can chime in as to when they plan to support HTTP/2.0 on their crawlers.
Most bots are using really old libraries. Some of the newer bots are using farms of cellular cards but they can be spotted by the use of TCP Timestamps as that is enabled by default on Android. The downside of doing anything with this is that one would also block legit cell phones, so just give them a simple puzzle to solve. e.g. What is 1+4 or if you are certain they are a bot then ask How many roads must a man walk down?
I get spam from Microsoft IPs, using my own domain. Someone signed up my domain with MS. Of course I let MS know, figuring this must be some vuln in their system or at minimum a clear-cut case of abuse they can shut down.
They sent back a template saying it's spoofed. As if I can't tell the difference between a From/Received header and the actual remote address that was delivering this garbage to my own mail server. All of this I already made clear in the original email.
Next up is Microsoft straight out rejecting email from my new IP address. It's not on any blacklist, it never did anything wrong, it just hasn't sent email before. Their systems for getting off the blacklist are somewhere between obscure, untouched for decades, broken, and nonexistent, depending on which page you look at and who you ask. Employees, if you get that far after hours of searching, claim my IP isn't blocked (yeah, right... they've blocked the range and so the employees can't find the IP ban, as if I'm supposed/able to know what bounds their range ban has.) The furthest I've got is that they somehow disable delivery failure notifications for a few weeks to shut me up but email still doesn't arrive.
Easiest solution to this would be something functionally equivalent to opendkim and spamassassin to rank high in score whatever incoming SMTP traffic doesn't pass SPF and DKIM (obviously it doesn't pass SPF if it's claiming to come from user@domainname.com but is from an IP that is not the dns/rdns of what's in the DNS zonefile SPF record for the legit MX of domainname.com).
And then additional scoring based on DKIM fail as well.
That's an option. Haven't needed this before because I just block addresses that I receive spam on. I've always just accepted every email sent that wasn't addressed to a blacklisted address, so people can't really misconfigure anything, I'll always get their email without hoops to jump. So far, the spam has been very limited, but maybe one day I'll need to do this. Or block Microsoft IPs and tell them they can sign up for my IP reputation management system that sends out emails with broken confirmation links... I haven't yet decided.
I recently made some contributions to https://github.com/alltheplaces/alltheplaces which is a set of Scrapy spiders for extracting locations of primarily franchise business stores, for use with projects such as OpenStreetMap. After these contributions I am now in shock at how hostile a lot of businesses are with allowing people the privilege of accessing their websites to find store locations, store information or to purchase an item.
Some examples of what I found you can't do at perhaps 20% of business websites:
* Use the website near a country border where people often commute freely across borders on a daily or regular basis (example: Schengen region).
* Plan a holiday figuring out when a shop closes for the day in another country.
* Order something from an Internet connection in another country to arrive when you return home.
* Look up product information whilst on a work trip overseas.
* Access the website via a workplace proxy server that is mandated to be used and just so happens to be hosted in a data centre (the likes of AWS and Azure included) which is blocked simply because it's not a residential ISP. A more complex check would be the website checking the time between a request from Javascript code executing on client reaching the server matches the time the server thinks it should have taken (by ICMP PING or similar to the origin IP address).
* Use the website simply because the geo IP database the website uses hasn't been updated to reflect block reassignments by the regional registry, and thinks that your French residential IP address is a defunct Latvian business IP address.
* Find the business on a price comparison website.
* Find the business on a search engine that isn't Google.
* Access the website without first allowing obfuscated Javascript to execute (example: [1]).
* Use the website if you had certain disabilities.
* Access the website with IPv6 or using 464XLAT (shared origin IPv4 address with potentially a large pool of other users).
The answer to me appears obvious: Store and product information is for the most part static and can be stored in static HTML and JSON/GeoJSON files that are rewritten on a regular cycle, or at least cached until their next update. JSON files can be advertised freely to the public so if anyone was trying to obtain information off the website, they could do so with a single request for a small file causing as minimal impact as possible. It's not difficult to create a website where 10,000 requests per second can be made to static data like product information or store locations. More advanced features such as stock availability would cause additional load, but again, 10,000 simple queries to a relational database is not a challenging problem.
Of course, none of the anti-bot measures implemented by websites actually stop bots they're most seeking to block. There are services specifically designed for scraping websites that have 100,000's of residential IP addresses in their pools around the world (fixed and mobile ISPs), and it's trivial with software such as selenium-stealth to make each request look legitimate from a Javascript perspective as if the request was from a Chrome browser running on Windows 10 with a single 2160p screen, etc. If you force bots down the path of working around anti-bot measures, it's a never ending battle the website will ultimately lose because the website will end up blocking legitimate customers and have extremely poor performance that is worsened by forcing bots to make 10x the number of requests to the website just to look legitimate or pass client tests.
I suspect this is a real Bing web crawler that's possibly misconfigured. The "spoofed" user-agent might be an attempt to get a more genuine crawl of what a mobile browser sees. I at least know Google does this.
Odd, I just noticed an ongoing rather intense scan for random paths on one of my servers using a number of IP addresses from within Microsoft-assigned space:
Maybe related, I had a few emails by alleged security researches, linking to standard tools, often even misinterpreting the results or maybe just too lazy to have a real look at them. Maybe, "security research" has just entered the "script-kiddy" age at a larger scale?
The list provided in this comment [0] looks much like it.
I've been experiencing exactly the same thing over a number of sites for the last few days too, with exactly the same user agent in this post. The annoying part is that it is not really 'spidering' web sites, but rather continuously hammering a list of non-existent pages which appear to be from a years-old version of the site.
Generating 404 responses puts a considerable load on WordPress sites and generates a lot of network traffic, but these have been relatively easy to block because of the predictable user agent and URI path prefixes. I'm thinking about blocking the Azure ASN completely, or developing something akin to Cloudflare's "are you a human?" interstitial page when requests come from cloud provider ASNs.
Started around Jan 12. Large pool of IP addresses, hard to block. Occasional brief DOS impacts but mostly just annoying errors in my logs. (if too many crop up we get automatic alerts). What was really puzzling is that many of the URLs are old (e.g. request for details on hosted sites that no longer exist). I loaded up a 6 month old backup database and confirmed those accounts weren't present, so the source list of URLs must be older that. Really bizarre.
After reading this article I looked and confirmed via spot checks they are from Microsoft IPs and Safari 15.1.
> it’ll be the new normal that webmasters/content producers don’t really need readers or income.
There's already way more content being produced out there (some of which is done altruistically without any expectation of income) than there is demand for it that making even more of it may not be as valuable as you think.
Thought experiment: imagine Google stopped sending users to articles you write and instead regurgitated and showed your content directly, with ads or for pay.
Fast forward a few years and you have no more audience, no one has to know you exist, you ghost write for a magic black box that people ask questions and throw money at. When you want to impress a new friend and mention you published some interesting niche research (e.g., how to repair a thing) they skeptically nod because in their head no one does it anymore, the black box has all the answers.
Will you still be feeling motivated to altruistically share content tomorrow?
> Given that there are websites that are willing (or reluctantly forced) to allow Google(bot) access but would rather like to block everyone else, more than a few of them are probably using User-Agent matching instead of anything more sophisticated. (https://utcc.utoronto.ca/~cks/space/blog/web/SpiderUserAgent...)
Yeah, what a great idea to make all of our websites only crawlable by Google. Because competition in the search engine space is _so_ undesirable.
The sequence likely went "site is not crawalble by anything" (because there were so many badly behaved crawlers back in the day). Later the webmaster wanted it to show up in searches and so specifically opened up the robots.txt from "no body no way at all ever" to "ok, google can search these pages because they're not being evil" (this was back in the day).
It wasn't so much a "ok, I want only google to search this and not anyone else", but rather at the time, there wasn't anyone else (that behaved).
So now, you've got a website that is only searchable by google, and the webmaster retired and you've got this robots.txt that says you can't do it - and even some active defenses so that when non-google hits the page, they get tarpited. What are competing (reasonably well behaved) web spiders to do? How does Bing get in there to be able to compete now?
Its not that Bing should or shouldn't be doing this - but there's a fairly reasonably way that we got to the point where are now without any nefarious behavior from the various participants.
Having a silly robots.txt is one thing- going to the lengths of blocking all bots that aren't in Google's IP space is what I'm worried about. If the whole world did that, nothing but Google could survive.
Those are absolutely problems. I suspect they're artifacts in configurations from when it was believed that Google did no evil and the next biggest company trying to get in on the web search was Microsoft and leaving them in wasn't a great loss.
Now roles are reversed (or at least, equal - though if DDG consuming bing data is considered to be on the side of good, an argument could be made for reversed) and yet those configurations persist because they were forgotten about.
My post above is a just so story ( https://en.wikipedia.org/wiki/Just-so_story ) for why they are the way they are now. Old systems that work tend to be forgotten until a security fix needs to go in and Chesterton's fences in config files tend to be left.
With todays resources, I would not advocate indiscriminate tcp tarpits for things that might be badly behaved crawlers until they demonstrate that they are indeed badly behaved crawlers or scanners searching for exploits.
Unless I missed or forgot some major events I don’t think it’s fair to say that Microsoft and Google are on the same level of “evil”. Microsoft willfully broke the law to destroy their competition. On multiple occasions. They had well funded initiatives to destroy Linux and open source in general. They intentionally built software on standard protocols/APIs and then once they had enough market share they sabotaged interoperability. Microsoft was (and is) ruthless because Bill Gates was a fucking corporate terminator.
all monopolies are bad by their very nature. If consumers have no choice, they get abused, regardless of whether it is clumsy and barbaric like microsoft, or deft and charming like Hannibal Google Lector.
Did you comment this in the wrong place? It doesn't seem to be a response to what I said.
shagie brought up the possibility that somebody would have a robots.txt with only Google in it because when they wrote their robots.txt, they wanted to block poorly behaved / evil crawlers and Google's was well-behaved. But that doesn't make sense to me, because by definition poorly behaved / evil crawlers do not care about robots.txt.
The ones that weren't well behaved back then found themselves in a tarpit (which was also a 'what you do with spam email sites connecting on port 25'). https://en.wikipedia.org/wiki/Tarpit_(networking)
The thing was that you'd send everything that was requesting faster than some rate limit to that tarpit unless it was google.
Years ago i added a url to not be crawled nor indexed to a web app I worked on to block bad crawlers, so I disallowed crawling through robots.txt and added a hidden link in the footer with the "noindex" attribute.
Google, Bing, Archive.org, and people normally browsing were just fine while bad crawlers got a blank page.
I've also seen this spider, and upon close examination it appears to be using a large dump of URLs, many years old, including non-public, non-guessable URLs that could only have been harvested from some sort of browser/plugin data breach.
It is aggressive in what content is trying to access. It looks for security vulnerabilities and normal bots don't do that (with the notable exception of some security testing software). Also it's not spidering, somehow it knows very old URLs which are not even public which were probably obtained from a malicious browser extension.
In a past job I’ve seen crappy crawlers from badly designed security applications do stuff like this. An an example one customer was using Trend CAS to scan all URLs in their inbound email. This causes big bursts of traffic on our systems.
The crawls came from Azure and AWS. Forged UAs, repeat hits in the same URL, etc.
We opened an abuse report with Microsoft last week, but haven't really heard anything back. Over 500 IP addresses involved hitting dozens of our hosted sites, le sigh.
Unfortunately, we do see enough legitimate requests with that UA that we had to resort to the IP blocking. A bunch of /16's and a few smaller ranges... it feels dirty :(
I'm sick of it too. In 2023-01 alone I have had 9,000 different IPs from Microsoft's ASN8075 crawling one of my sites with these "normal looking" user agents. Poring over the logs to see why your server is on fire takes non-trivial amounts of time. If I didn't have a ton of other stuff to do I'd say it was kinda fun, but I'm freaking fed up.
Just yesterday I put all their networks into an nginx geo map:
geo $limit_bots_ip {
# requests with an empty key are not evaluated by limit_req
# see: http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
default '';
157.55.39.0/24 'bot';
207.46.13.0/24 'bot';
40.77.167.0/24 'bot';
13.66.139.0/24 'bot';
...
}
Any request from these networks gets classified as a bot, which is then used as the key for a rate limit
It's one thing if your server is on fire because of a bot crawling your websites at 1000rps, another if it's on fire because of a bot crawling it at less than 10rps.
126 comments
[ 5.8 ms ] story [ 191 ms ] threadAWS has acted upon similar reports in the past, Azure may have but the lack of comms and delays in seeing any action (if any) hasn't inspired confidence.
I like Linux as much as the next guy but I can't help but feel this isn't a very... useful suggestion.
I would hope they did investigate too otherwise it’s too easy to DoS a valid customer with fake complaints.
A couple provides I sent abuse complaints to based on whois lookups responded with automated emails requesting the info in X-ARF format. It’s hard to get some cloud providers to respond to a random residential user.
A couple IPs were repeated “scans” by supposed security researchers with IPs out of Hawaii and France.
If I find the sketchy Hawaii and French IPs I will post something.
Edit: Here you go:
https://academyforinternetresearch.org
It seems they are running on Hivelocity a bare metal hosting provider.
Snort is something I have looked at installing and playing with on a PFSense box.
https://www.snort.org
You could also check if the remote IP has any open ports (ex. SSH) and block based on that as most regular visitors probably won't be exposing those types of services to the Internet.
If I had a dollar for every person who has turned loose such a tool to shit up my abuse@ispname.com inbox...
I seriously doubt the Las Vegas “located” IP with Hong Kong addresses on their registration or Shanghai UCloud Information Technology Company IPs are scanning me for good reasons.
What do you think would happen if I run an older version of plex with a known vulnerability?
The automated emails are usually in the same formats making them easy to filter and parse too.
Absolute magnitude is irrelevant. A standard deviation uptick vs baseline is always interesting.
This is just standard background noise.
There seems to be significant overlap between sysadmins who send handwritten abuse reports and “targeted individuals”/“gangstalking victims”.
Whichever way you cut that, if this happens to be a single host server, you're rapidly approaching a core-day with this load once you add some logging and such on top.
No way a scraper running at less than 1rps is anywhere near a core-day without some horrible misconfiguration, it should be way easier to address that than to fight a war against scrapers.
I have significantly more complicated flask sites handling 100+ rps on a single core, it doesn’t require any black magic.
Never see it coming.
People who are doing a 'get all content on the web' scrape, I try to not block, because there are plenty of reasons to want an offline copy of the internet.
I try to make sure all my webapps have sensible URL schemes to prevent such (IMO legitimate) activities ending up doing an infinite crawl because for example the URL has a session identifier in it and the bot keeps getting assigned a new session.
That's what I saw from Azure IPs recently. A single URL being requested from some 5k IPs, each one requesting it every few minutes.
I had reported it to MS, they said (after two weeks) "it looks like abuse" and they'd forward it to their cert. I haven't heard any update since.
GPT-[0-9]+, DALL-E, "AI".
ChatGPT: Produce a blog post in the style of [Given Name] [Surname]
From NCSA Mosaic 2.7, circa 1993, one of the first "web browsers", authored in part by one of Silicon Valley's favourite VC, the "a" in az16.com.
https://raw.githubusercontent.com/alandipert/ncsa-mosaic/mas...The sad reality is that "tech" companies and their service providers routinely use HTTP headers for their own manipulative, deceptive and user privacy-hostile commercial purposes. They use UA headers in ways that were not contemplated in 1993 and probably never intended. (Note I am not referring to spoofing, I am referring to data collection, tracking, e.g., "browser fingerprinting", and "surveillance capitalism" in general.)
Sending a UA header generally offers me, the end user, no benefits, so I do not send one.
Why not ban those Microsoft IPs or at least rate-limit them to a limited number of HTTP requests in a given time period.
Also looks like this website does not use sitemaps. At least, there is nothing listed in robots.txt. IMHO, there is no need to spider/crawl if there are sitemap XML files listing all the content URLs.
UA spoofing is as old as 1994 or maybe 95 at the youngest.
There is a reason why every browser claims to be Mosaic Communicator.
you never know if these are all URLs tho, so I would always crawl and brute force more paths than trusting sitemap.xml
It was a really dumb spider. It wasted a huge amount of its efforts requesting pages that are no longer valid on my site. It must have had an archived set of URLs it was using as a source rather than spidering the site as it is today.
From an anthropological point of view one of the most fascinating behaviors here is the apparent belief that as a website operator, your opinions about how HTTP clients should craft their user agent headers have any weight whatsoever.
This curious belief system predates even the browser wars let alone the modern web. Remarkable to find an original example of this idealistic world view still being expounded today.
Not caring about false useragents is one thing, caring about somebody running a multi-thousand-requests-per-minute spider requesting pages that no longer exist is another thing. It would still be valid cause for complaint even if it had a valid useragent.
The internet would be a better place if more experienced people (such as those who were running httpd 22-24 years ago) reinforced and emphasized the principle, when training and mentoring younger people, that being a good citizen with your network traffic is the responsible and ethical thing to do.
I’m just astonished that there are still people who have the time to notice and lament such behavior, and devote time to handcrafting .htaccess rules, rather than just having an aggressive set of heuristics and accepting that some crap will still get through.
The idea that you can appeal to the community to do a better job of policing badly behaved clients on their networks just feels like a quaint relic of the arpanet.
Have you ever worked in network engineering for an ISP that holds a decent sized chunk of ARIN or RIPE ipv4 IP space, and actually cares about its long term IP space reputation?
There's a big difference between a $5/mo colo/VPS operator that doesn't give a damn, and an ISP that does pay attention when legit "you have customers with abusive traffic" reports reach their network engineering team.
Your comment about not caring about policing badly behaved clients on a network seems like the sort of organization that would land itself on every SMTP RBL in existence.
A hosting company (and Azure and AWS are just the giant corporate evolution of a hosting/colo company) needs to be prepared and have a process to fire a customer for violation of TOS/AUP. It's done all the time. The immense size and callous nature of Microsoft/Azure makes this less likely in the modern era.
What’s the equivalent mechanism whereby abusive HTTP traffic gets identified and causes an ISP to pay attention?
Is there some sort of greybeard grapevine that old school webmasters can use to get the attention of class A IP network operators?
webmasters as in single operators of httpd somewhere, no, not likely
other network operators who are recognized ASNs and go to NANOG or RIPE or APNIC conferences and meet people face to face, yes.
if you wilfully disregard valid human-written abuse complaints coming from other equally-clueful network engineering persons at other ISPs, you and your ISP will acquire a tainted reputation.
there's various industry back-channels that exist between ISPs to communicate with each other about this sort of thing, I won't go into many details, primarily because there's a high degree of interest of the parties on all sides in keeping the email inboxes and phone numbers uncluttered with clueless single-residential-last-mile-user complaints about their wifi not working right.
"Hey Bob. Bunch of clients on your network keep initiating https connections with servers on my network"
"Gee Bob, sorry about that I'll go see what's causing that right away..."
If folks are escalating to the neteng crews it usually means something is broken via normal channels and it could use attention.
That's social network is exercised for a variety of things:
Abuse, ddos, IP spoofing, botnet C2s being hosted somewhere, malware hosting, complex customer investigations, route leaks, route hijacks, routing loops, congestion, executives home internet issues and more.
Feelings aside, it doesn't seem that different in principle to an ad blocker with a shared list of ad servers.
I get that morally we might feel like it is better for these ill-mannered varmints to receive a well-deserved '401 Unauthorized' HTTP response when they try their shenanigans.
But... really, if they were just going to get a '404 Not found' anyway, who cares? Your server has to serve pretty much the same number of bytes, and it has to run less regexes over user agent headers.
There's just something I find charmingly anachronistic about trying to operate a server like this.
One site being hammered is one thing. Thousands of sites being hammered at the same time is another. The issue in the article is a current and live problem. Blocking the requests saves a lot of CPU cycles.
Most bots are using really old libraries. Some of the newer bots are using farms of cellular cards but they can be spotted by the use of TCP Timestamps as that is enabled by default on Android. The downside of doing anything with this is that one would also block legit cell phones, so just give them a simple puzzle to solve. e.g. What is 1+4 or if you are certain they are a bot then ask How many roads must a man walk down?
They sent back a template saying it's spoofed. As if I can't tell the difference between a From/Received header and the actual remote address that was delivering this garbage to my own mail server. All of this I already made clear in the original email.
Next up is Microsoft straight out rejecting email from my new IP address. It's not on any blacklist, it never did anything wrong, it just hasn't sent email before. Their systems for getting off the blacklist are somewhere between obscure, untouched for decades, broken, and nonexistent, depending on which page you look at and who you ask. Employees, if you get that far after hours of searching, claim my IP isn't blocked (yeah, right... they've blocked the range and so the employees can't find the IP ban, as if I'm supposed/able to know what bounds their range ban has.) The furthest I've got is that they somehow disable delivery failure notifications for a few weeks to shut me up but email still doesn't arrive.
I'm rather done dealing with Microsoft of late.
And then additional scoring based on DKIM fail as well.
Some examples of what I found you can't do at perhaps 20% of business websites:
* Use the website near a country border where people often commute freely across borders on a daily or regular basis (example: Schengen region).
* Plan a holiday figuring out when a shop closes for the day in another country.
* Order something from an Internet connection in another country to arrive when you return home.
* Look up product information whilst on a work trip overseas.
* Access the website via a workplace proxy server that is mandated to be used and just so happens to be hosted in a data centre (the likes of AWS and Azure included) which is blocked simply because it's not a residential ISP. A more complex check would be the website checking the time between a request from Javascript code executing on client reaching the server matches the time the server thinks it should have taken (by ICMP PING or similar to the origin IP address).
* Use the website simply because the geo IP database the website uses hasn't been updated to reflect block reassignments by the regional registry, and thinks that your French residential IP address is a defunct Latvian business IP address.
* Find the business on a price comparison website.
* Find the business on a search engine that isn't Google.
* Access the website without first allowing obfuscated Javascript to execute (example: [1]).
* Use the website if you had certain disabilities.
* Access the website with IPv6 or using 464XLAT (shared origin IPv4 address with potentially a large pool of other users).
The answer to me appears obvious: Store and product information is for the most part static and can be stored in static HTML and JSON/GeoJSON files that are rewritten on a regular cycle, or at least cached until their next update. JSON files can be advertised freely to the public so if anyone was trying to obtain information off the website, they could do so with a single request for a small file causing as minimal impact as possible. It's not difficult to create a website where 10,000 requests per second can be made to static data like product information or store locations. More advanced features such as stock availability would cause additional load, but again, 10,000 simple queries to a relational database is not a challenging problem.
Of course, none of the anti-bot measures implemented by websites actually stop bots they're most seeking to block. There are services specifically designed for scraping websites that have 100,000's of residential IP addresses in their pools around the world (fixed and mobile ISPs), and it's trivial with software such as selenium-stealth to make each request look legitimate from a Javascript perspective as if the request was from a Chrome browser running on Windows 10 with a single 2160p screen, etc. If you force bots down the path of working around anti-bot measures, it's a never ending battle the website will ultimately lose because the website will end up blocking legitimate customers and have extremely poor performance that is worsened by forcing bots to make 10x the number of requests to the website just to look legitimate or pass client tests.
[1] https://www.imperva.com/products/web-application-firewall-wa...
I suspect this is a real Bing web crawler that's possibly misconfigured. The "spoofed" user-agent might be an attempt to get a more genuine crawl of what a mobile browser sees. I at least know Google does this.
The list provided in this comment [0] looks much like it.
[0] https://news.ycombinator.com/item?id=34435222
Generating 404 responses puts a considerable load on WordPress sites and generates a lot of network traffic, but these have been relatively easy to block because of the predictable user agent and URI path prefixes. I'm thinking about blocking the Azure ASN completely, or developing something akin to Cloudflare's "are you a human?" interstitial page when requests come from cloud provider ASNs.
Started around Jan 12. Large pool of IP addresses, hard to block. Occasional brief DOS impacts but mostly just annoying errors in my logs. (if too many crop up we get automatic alerts). What was really puzzling is that many of the URLs are old (e.g. request for details on hosted sites that no longer exist). I loaded up a 6 month old backup database and confirmed those accounts weren't present, so the source list of URLs must be older that. Really bizarre.
After reading this article I looked and confirmed via spot checks they are from Microsoft IPs and Safari 15.1.
If we don’t start batting an eye on this soon it’ll be the new normal that webmasters/content producers don’t really need readers or income.
There's already way more content being produced out there (some of which is done altruistically without any expectation of income) than there is demand for it that making even more of it may not be as valuable as you think.
Fast forward a few years and you have no more audience, no one has to know you exist, you ghost write for a magic black box that people ask questions and throw money at. When you want to impress a new friend and mention you published some interesting niche research (e.g., how to repair a thing) they skeptically nod because in their head no one does it anymore, the black box has all the answers.
Will you still be feeling motivated to altruistically share content tomorrow?
> Given that there are websites that are willing (or reluctantly forced) to allow Google(bot) access but would rather like to block everyone else, more than a few of them are probably using User-Agent matching instead of anything more sophisticated. (https://utcc.utoronto.ca/~cks/space/blog/web/SpiderUserAgent...)
Yeah, what a great idea to make all of our websites only crawlable by Google. Because competition in the search engine space is _so_ undesirable.
It wasn't so much a "ok, I want only google to search this and not anyone else", but rather at the time, there wasn't anyone else (that behaved).
So now, you've got a website that is only searchable by google, and the webmaster retired and you've got this robots.txt that says you can't do it - and even some active defenses so that when non-google hits the page, they get tarpited. What are competing (reasonably well behaved) web spiders to do? How does Bing get in there to be able to compete now?
Its not that Bing should or shouldn't be doing this - but there's a fairly reasonably way that we got to the point where are now without any nefarious behavior from the various participants.
Do I blame Bing for that? Absolutely not.
Having a silly robots.txt is one thing- going to the lengths of blocking all bots that aren't in Google's IP space is what I'm worried about. If the whole world did that, nothing but Google could survive.
Now roles are reversed (or at least, equal - though if DDG consuming bing data is considered to be on the side of good, an argument could be made for reversed) and yet those configurations persist because they were forgotten about.
My post above is a just so story ( https://en.wikipedia.org/wiki/Just-so_story ) for why they are the way they are now. Old systems that work tend to be forgotten until a security fix needs to go in and Chesterton's fences in config files tend to be left.
With todays resources, I would not advocate indiscriminate tcp tarpits for things that might be badly behaved crawlers until they demonstrate that they are indeed badly behaved crawlers or scanners searching for exploits.
…something which Google has never done, I'm sure.
shagie brought up the possibility that somebody would have a robots.txt with only Google in it because when they wrote their robots.txt, they wanted to block poorly behaved / evil crawlers and Google's was well-behaved. But that doesn't make sense to me, because by definition poorly behaved / evil crawlers do not care about robots.txt.
The thing was that you'd send everything that was requesting faster than some rate limit to that tarpit unless it was google.
Google, Bing, Archive.org, and people normally browsing were just fine while bad crawlers got a blank page.
The crawls came from Azure and AWS. Forged UAs, repeat hits in the same URL, etc.
This particular 'bot' would start crawling in-line Javascript on a customer site, and then get stuck in an infinite loop requesting things like:
We opened an abuse report with Microsoft last week, but haven't really heard anything back. Over 500 IP addresses involved hitting dozens of our hosted sites, le sigh.Unfortunately, we do see enough legitimate requests with that UA that we had to resort to the IP blocking. A bunch of /16's and a few smaller ranges... it feels dirty :(
Just yesterday I put all their networks into an nginx geo map:
Any request from these networks gets classified as a bot, which is then used as the key for a rate limit It's incredible the amount of resources these companies have. I'm just one guy trying to keep a few dozen web servers up.