Absolutely, and this is why I refuse to run connected devices on my network that do run software I can reasonably trust. This is also why every computer, phone or appliance I own runs Linux. This includes my TV box, which I recently migrated to an x86 port of Android with Claude's help.
That is the least bad thing they could do btw. Unless your internet is data restricted, go for it if you want free cable. If questioned by your ISP (extremely unlikely but theoretically possible) pretend you have no idea what they're talking about.
Firewall it from your local network if you want to - it'll still work - they're not after that.
Interesting! According to this, a lot more going on beyond the residential proxy element, which was the only bit I had considered (or heard about). I wish there was an accompanying article with some more specifics.
So many downvotes. This is a technical forum. The transcript provides almost no technical insights but manages to mentions China 18x. Would be nice to have a discusion on that.
> I get a hit in my vulnerability log, like in the threat log on my Palo Alto firewall, and it’s for a SCADA vulnerability. ... A SCADA vulnerability. This makes no sense. SCADA is the control systems used in large-scale industrial settings. Think pumps, valves, conveyor belts, compressors, elevators, railway switches. This is where SCADA systems live. Why in the world is this box that’s here to deliver TV and movies attempting to trigger a SCADA exploit on D3ada55’ network? This is very concerning.
Such things are threats to humanity imho. They can get us back to the times we couldn't exploit the internet to run a society.
There are plenty of people selling IPTV access on g2g, few bucks for a month, all TV channels from entire world. I am not sure if it is profitable to let such piracy boxes work forever, even with them being used as residential proxy, but I also don't know how low you can go if you skip middlemen.
Yes. I've also seen them sold on other online retailers, including Amazon. They're pre-infected boxes, acting as a voluntary network implant under the guise of being a sneaky way to consume otherwise-paid-for content.
I mean supply chain-attacks are everywhere. Smart-TV's with certain marketplace apps, IoT devices. Also, I'm surprised the white house's network is not monitored more.
Some good answers in this thread. I had the same question but I'm still a tad confused:
* I initially thought this meant someone in the WH network was using commercial proxies to appear as if they were exiting from a different network (which would align with "some user is firing up a proxy on their phone to exit in a different location")
* But then the description of residential proxies seemed to align more with traffic hitting an endpoint in the White House, and _exiting_ from the network there. But this didn't explain the timing (why isn't it 24/7).
* Finally it sounds like some person, legitimately on the WH network, has a device (cell phone, TV, whatever) that, when activated, is proxying outside traffic into the WH network.
And making DDoS-attacks nigh impossible to IP block.
Residential proxies are in many cases at least shady, both on the side of who uses them and for what and on providers that are often not above using dark pattern to hide them in some innocuous app for a smart-whatever that silently runs a proxy on a unaware user's connection. And in many places in the world if illegal stuff in done using your connection you are liable, regardless if you knew about it.
Residential proxies are a very important part of the internet
So important that we have to trick people into using them. Remember when folks were running SETI@Home? Of course you do, because it was widely thought to be a good thing, and folks willingly signed up.
How many would buy the "box" if the advertising were honest about what's actually in the box?
Seems like the president found a business opportunity and gave internal orders to rent out the network infrastructure to 'ethically sourced' providers :^) Jokes aside, might be some infected IoT crap.
The danger here, of course, is that while these compromised devices are currently being monetized in a relatively benign way (other than bandwidth costs to the user, which to be sure can be nontrivial)… it would be naive to assume that there isn't a Command and Control capability that can assign less benign tasking - or can be compromised to do so.
This also puts some of the troubling reports that politicians were bringing phones into SCIFs into context…
On a server that used to get below 1k HTTP requests a day, I've had several cases of mass scraping from these botnets exceeding 100req/s. The web is getting harder and harder to use due to all the anti-bot mechanisms site operators had to put up.
> The web is getting harder and harder to use due to all the anti-bot mechanisms site operators had to put up.
Indeed. I recently set up a bot on my local machine that I can interact with via Telegram. I gave it the ability to retrieve the contents of a URL via Playwright (with modifications). Basic things like getting the Readability view and getting the site as Markdown with links.
I was essentially mimicking the feature OpenAI etc offer where it can do a web search for you and get you the answer you seek. I just didn't want to pay a company for that feature.
It's not autonomous. It only acts when I tell it to. The actual search is via the Kagi API, but the URL contents is via Playwright. I also ensured it can't do lots of requests at a time. There's a mandatory (random) sleep of between 1 and 5 seconds between two requests.
In less than a week, I've been flagged. When I do a Google search (as a human), I have to prove I'm human. Many sites have simply blocked me - I can't buy stuff from various vendors[1], I cannot access several news sites, etc.
The thing is, I didn't really access that many sites with the bot! At most it's the equivalent of someone mildly addicted to the web. Yet they caught me quickly.
Obviously, I disabled the bot's capability to get site content. I wonder how long my IP will be in the blacklist.
I know some people will say "Good riddance" to the outcome and to what I did. I'd love to have a real, serious discussion on whether what I did was immoral. But whatever your thoughts on the matter, do ponder over the fact that the system has made it hard for average Joes like you and me to have this "feature", but if you pay a giant tech company a monthly fee, you can have it.
[1] Thankfully, Amazon didn't block me. In the past I'd avoid them, but now I'm in the dilemma where I don't want to buy stuff from sites that make it difficult for me to buy stuff from them!
That is entirely ISP dependent and many don't actually work like that. Your cable modem gets a specific IP for as long as Comcast or whoever sees fit, restarting does nothing on the WAN side.
On cable ISPs, oftentimes, changing the router's WAN interface's MAC address and then rebooting the modem will get you a fresh IP address, but rebooting the modem without first changing the router's WAN MAC address will get you the same IP address.
Doesn't work like that, only on mobile phones you can toggle airplane mode to get a new ip assigned and cgnat makes sure mobile ips are "trusted" , but they fingerprint you beyond ip address
It depends on the DHCP part and your provider lease time. When you access the Internet through a modem and landline, it's an highly dynamical system, where it's difficult to guarantee every caller/user gets its own IP. Especially when IP sharing is used among the clients. Even more difficult it is with the mobile Internet. It's a constant connection loosing/reconnecting. Actually resetting and freeing up IPs for new devices, the whole time. With cable Internet (the TV cable) one is having a kind of stationary access point. The network provider can see it's topology and it never changes once a leafe has joined the network. I typically have the same access creds over months. Even when I restart my cable modem, it's still the same IP. When I was on modem, I could get a new IP every second of I wanted. Now I have to think twice, because I just can't get a new IP, even if is forcedly, not.
Expect to remain blocked, that's how defense works, some services plenty use will have is bot or known proxy exit node set to true, and some websites use the same defense services, request velocity is just one factor, if you don't have random human like scrolling and jitter with a bot, it's a dead profile walking, metaphorically speaking
I think that major gateways like Google and Cloudflare and Akamai are doing things like TLS-fingerprinting and smearing.
I normally use a VPN to access the Internet. It's an ordinary consumer-grade VPN, and I use it for pretty innocuous things like watching the international versions of Netflix or avoiding being geographically-profiled and charged more when I book travel. The VPN itself, I'm sure, is being used for all sorts of naughty stuff: when I look it up in whatismyip, it comes up as VPN, "abuse detected", and "residential proxy detected". I expect to get blocked occasionally on it and just pause the VPN then.
What I found surprising is that pausing the VPN doesn't always clear the CAPTCHAs. The next request will also fail, sometimes. For a while I was wondering if I have a residential proxy on my network, but I went hunting with lsof and bandwidth monitoring and a list of all the open connections through my router, and it seems clear. Yet even with VPN disabled, I'll get CAPTCHAs from Google and Cloudflare.
I suspect they're using browser/TLS fingerprinting and then smearing the results of that across bad IPs. If you connect to them through a known-bad IP, they not only will CAPTCHA you, but they will record a fingerprint of any data they can. Then if you connect again with that fingerprint, regardless of the IP, they block that too.
Ironically, if you are actually malicious, you can get around this pretty easily. Because you'll know not to use a fingerprintable browser, or to rotate fingerprints. So you have one completely separate identity on a given IP (that is not yours), and if it gets blocked, you roll up another completely separate identity on a different IP. But this is too much work for an ordinary consumer who just wants to keep their IP private. So you end up with this excluded middle where the only rational strategies are to either do exactly what the website expects you to do (with no funny business, even if it's innocuous), or to go whole-hog as the aggressor where you're determined to skirt their policies no matter what, and if you're doing that you might as well amortize the effort by extracting everything of value you can from them.
Yeah, I didn't want to put too many details as it would obscure my point. I was actually using a modified Playwright that's been designed to better act as a human. It works, in that I got fewer captchas etc than regular Playwright, but apparently not well enough.
That modified Playwright was based on Firefox, and that's what I use for my regular browsing. So that didn't help.
I can actually launch Chrome and not deal with this - but I'm not going to. In fact, what surprises me is that I can create a new Firefox profile and not get those captchas. I don't know how my using that modified Playwright as a bot got them hooked on to my Firefox profile. I suspect they had my FF profile as a "suspicious" entity even before I made the bot (I use a number of privacy/anti-cookie extensions), and the bot pushed the score over the edge.
I'll probably start fiddling with my profile - disabling extensions, until things seem to work. Thankfully, no one else in the household is impacted.
For now, I got around all this by just paying for the Kagi extract API. It's not as good as my modified Playwright had been, though. I thought of a VPN, but I suspect that won't last long (not a residential proxy, etc), and I don't want an ongoing monthly fee. I like Kagi's pay as you go option.
But as I said, it's a weird world where suddenly Amazon is coming out the "better" site/vendor. It's funny - in the old days I was pretty anti-Walmart, but a few years ago I started preferring them over Amazon (and even several local stores) for a number of reasons (better collection than local stores, not having to deal with fakes for online orders, etc).
I've not been liking the state of the web for years prior to the advent of LLMs, but this is feeling like a final straw where I accept "the web is dead". We need a fundamentally different protocol - the model of "Set up a server anyone can reach, put content out there, and then put up a lot of sophisticated technology so your server isn't hammered" is not tenable. The protocol itself needs to manage the load.
I just did this as well. I also added paywall bypass by default along with inline article chat. The internet is such trash these days.
I made my modifications at the network level and auto redirect a configurable set of sites to go to reader mode/paywall bypass by default with a MITM proxy and a custom CA cert I trust
My residential network has been on Google's shit list for a few years now, I don't think it will ever go away. Like others on this thread if I see a Google recaptcha that site won't get my business as I have to exit since the captcha is an infinite loop.
While your server handles that, what does your CPU do with the other 99% of available cycles?
I run https://bunnystomp.com on an Always Free VM from a major cloud provider. It handles 1,930 req/s at 33 ms latency with no tuning/optimization. It's literally just python behind cloudflare.
You seem to know a lot about my server. From what I could see, all cores were pinned to 100%, but maybe my budget AMD CPU is hiding fantastical amounts of compute from the kernel. I'd love to know how to make use of it.
I have no idea what the site you linked does, but it doesn't look like a cgit instance to me; that's what got hammered on my end, and one very quickly finds out that a lot of git operations are CPU-bound when hosting it.
If you want credibility from the get-go, you need to be up front about your application. I certainly took it to be a generic host for an unpopular site in your first comment.
Funny. My site is a search engine and cgit is one of many tarpits that I block. Otherwise the index gets filled with garbage and the crawler hits TCP conn limits because the bytes trickle in too slowly.
I'm a screen reader user (which is as "weird and unusual" as it gets when fingerprinting mechanisms are concerned), and I am not really noticing much difference versus 2022 or so. Sure, there's little bits of anubis here or there, and a few sites (which I know for a fact are experiencing targeted DDoS attacks unrelated to AI scraping) having annoying Cloudflare verification screens, but that's about it.
My setup (other than the screen reader) is as vanilla as it gets. Safari, AdGuard for Ad blocking, no VPN except work VPN when necessary, no PiHole, no settings changed. If you're using something more esoteric, maybe try going mainstream and see if that improves your experience.
1) The proxy isn't the scraper. There are plenty of scrapers not using residential proxies.
2) They're clearly talking from the perspective of the user and network admin. A residential proxy virus is in fact quite benign for both.
3) No matter what, DoS is quite a low priority in terms of security. And it's not even really DDoS, you'll know a real DDoS when you're on the receiving end.
Let's not get ahead of ourselves... If someone plants a bomb under your car, you drive to work, then it blows up while you're out for lunch, should you be prosecuted?
I'm all for creating consequences for people who negligently don't secure their systems and people get hurt because of it, but people who installed a mobile game that has a proxy library in it? Please, chill.
Why are you so surprised that the US gov't is being spied on? Of course they are. Why/how could you ever suspect they would not be? How Polyanna can one be to think this way?
I'll just note that "compromised" is perhaps too strong. These services often come from VPN plugins (eg Hola VPN) that lets users watch sports from different restricted geographic regions. There is a trade here that's in the TOs: you get someone else's IP address, they get yours.
If I'm using one of these services while connected to a guest wifi network, with the usual configuration for guest access (like no intranet connections), is that guest wifi network compromised? What's the threat model?
I'd say yes since you've allowed random people unauthorized use of the connection. It makes no difference if it's your connection or not. For example, if that proxy gets used as part of a DDoS attack, your device compromised that network.
how is this different than Tor? You seem a bit on the surveillance side. The computer/network is compromised if someone gains unauthorized access to it not if you share it.
> The computer/network is compromised if someone gains unauthorized access to it not if you share it.
Yes. And someone did get unauthorized access to it. You don't have the permission to delegate authorization to a third party.
Did the third party connect to the guest wifi or did they connect to your computer and you have allowed them access to the guest wifi?
Did they agree to the ToS that the wifi network requires for people to use it?
Are you certain that the wifi network isn't intentionally limiting the area where it is broadcast, allowing only people within that geographic area the ability to connect to it?
If there is a purely 100% open wifi network who is only limited geographically due to technological constraints but would allow anyone in the world to connect to it if the technology allowed, then using your device to solve the technological constraint of limited wifi range would not fall under compromising the network IMO. Assuming again that the operator of the network publicized it as such and you aren't just making assumptions about their intentions.
That's mind bending gimnastics. So if you connect a device(i.e. a TV) to WiFi and there is a Netflix app that's downloading movies in background then Netflix has gained unauthorized access to the network because somehow is not `directly` connected to the home router(like a kind of long cable from directly from netflix server into the home router ) ? Did Netflix agree to the ToS of the Wi-Fi network provider? What a stupid thing to say...I would expect more from this community.
> So if you connect a device(i.e. a TV) to WiFi and there is a Netflix app that's downloading movies in background then Netflix has gained unauthorized access to the network
There are two main differences. Netflix is working on your behalf. Connections initiated from netflix are no different from connections initiated from your web browser or your email client. Your use of the internet, for you, is authorized. With a proxy you have no idea who is connected or what they are doing with that connection. That network activity uses you, but isn't by you, or for you.
Finally, netflix isn't likely to be hacking, spamming, or attempting to bypass bans or IP blacklists over that connection, but those are exactly the kinds of activities that shady residential proxies are used for. You basically backdoored your own computer, and now access to the internet through it will be sold to people who want to use it for something that they couldn't get away with by using a VPN or any other legitimate provider.
Depending on the sophistication of the wifi network you may be held liable and accountable for everything that takes place over your connection, including what ever nasty things the people who bought access to your proxy get up to. If the wifi system isn't set up to keep careful track of who is using it and when the liability will fall to the entity offering the wifi access, and if that connection is repeatedly used for activity in violation of the law, or the AUP of the ISP being used, that same entity will face the repercussions. Their internet connection may be temporarily suspended, and they may even be forced to stop providing wifi access to guests entirely unless they are willing/able to spend the money needed to log and monitor who is using their connection and respond to notifications of abuse over their network as well as legal requests.
If that guest WiFi network is running inside the White House? Yes, I'd say that is compromised. The threat model being that guests on that network may be diplomatic officials and staff.
You've confused intranet access with internet access. Guest networks rarely have intranet access, even between clients. They usually exist for "I don't trust this person on my network, but they want internet access".
What part of that isn’t technically “compromised”? You can argue there’s no evidence of it currently being weaponized but I’d say some shitbird running a free vpn inside the White House is not a good look regardless.
That's not how Hola
works. You're not sharing your IP with some other Hola user. You're sharing it with bots used for all sorts of good and bad things.
The joke that S in "IoT" stands for Security is as old as the term itself, but there were no such clear incentives for exploiting them before, beyond basic DDoS and more recently crypto mining.
If your IP address is part of a residential proxy network, it will be flagged.
Also, I recommend seeing the device count data we have for you. If you are on a NATed IP, you have virtually no control to remove the residential proxy SDK from your system. Let me know if you have any questions, as I work for IPinfo.
It's been some time since I ran this myself but I used to have an openwrt router sit behind my ISP modem/router (it was one unit).
Multiple times I caught unexpected traffic and was able to block it piecewise including a lightbulb(!) that was sending mb packets of god knows what home.
If they were ethernet based I reckon some smart VLAN capable switches would be nice even at home to keep them out of each others business.
I moved home a few years ago and now I'm back with the ISP router and do often feel a little in the dark about what my network is doing!
I had to recently inform my sister and didn't realize just how unaware the general public was of how utterly terrible things have gotten, like with LG TVs or the Meta Quest's access to your family.
When someone tells you "I have nothing to hide, I don't care", let them know that companies like LG make 5x as much money selling your data as selling $1000 TVs, so clearly your data is valuable and you shouldn't be giving it away for free. That's a message normal people can understand.
It detects if your IP is a residential proxy, not if you are using a residential proxy. From the perspective of the outside Internet, they are the same thing, the only one who cares about the difference is you.
We do not run any kind of detection systems when you visit the page. What we do is reference our resproxy detection database, which has a schema like this:
We collect the residential proxy data continuously from independent systems; when you visit the page, we take your IP address and look it up in our database. Note the percent_day_seen field, which is frequency metadata. You can also complement it with our device count data to understand if you are on a NATed IP.
If you are on a static IP address and a single user, we flag your IP address as a residential proxy. I think you should install some DNS monitoring service and start uninstalling unnecessary apps from internet-connected devices.
Neat. I almost said "surely this will only detect if your internet connection is itself being routed through a residential proxy", and then I realised that's effectively the same thing!
Freaky. They have a page explaining the concept (https://ipinfo.io/data/residential-proxy); when I checked it, there was a FOUC showing a full-screen, stretched image of what I think was Dario Amodei's face.
I would actually feel better if this were the case, but given that we have "the very best people" currently running our cybersecurity infrastructure I doubt it.
Most likely scenario is a 20-something holdover from DOGE who accidentally had his phone p0wned while downloading groyper memes.
Most likely scenario is a janitor who installed an app (I'm guessing it was called LegitWorldCupWatch420696969) on his phone (which might be connecting to the wifi network from whatever locker he leaves it in while he's on the clock, the WH is not a giant SCIF). Or any number of other mundane, easy to overlook employees who work outside of the actual secured areas of the building.
It is tricky. Monitoring network traffic should be enough, the infected device has to connect to Command and Control, to receive orders and set up tunnels.
It got harder after malware creators started using Telegram, Discord or even good old IRC for those purposes - it will be asy to notice only if they use a service no one else uses in your network.
I work for IPinfo. I recommend users use ipinfo.io/my as we do connect to residential IPs and verify.
Now, how to prevent this. If you are on a NATed connection, it is nearly impossible to prevent this from happening because you cannot control what apps people install who share the same IP address with you. If you have control over your network, then I suggest you install a DNS monitoring service. You can periodically monitor the DNS requests to find unusual patterns.
I have T-mobile 5g. My IP shows up as a residential proxy. I can't control the T-Mobile 5g device. It's a black-box. And, I only have one Linux desktop on the home network (so no Windows/Mac malware).
Not sure why my IP is listed as a residential proxy.
There’s only that many ipv4 addresses and I believe all 5G are shared ones. That’s why they’re among the most valuable ip ranges for scraping. You can’t just naively ban that IP or you’ll ban a large group of users
It should be a balancing act though, somewhere between what you said and my first sentence.
Rate limiting, or additional layers of input sanitisation for IP addresses known to have been malicious in the past.
It's unfortunate that good users will have to endure some difficulties because of bad users on the same network, but if security is the priority, then measures must be taken, you shouldn't just leave to door open to potentially known bad users / traffic.
IPv4 is a shared resource, but some ranges have worse reputations than others, and all are owned by organisations that have the ability to monitor and filter for malicious traffic. It gets into a fraught discussion about censorship, but it is possible for network owners to monitor for, and block / restrict traffic detected as 'bad'. It's just a matter of cost and scale, and those that put the effort in should have their users rewarded with fewer 'trust-issues' roadblocks.
The blocking of residential proxies might be the thing that moves the world to ipv6. These days what we need is a dynamic map of all the IPs running proxies so servers can block them. If every device gets it's own v6 address, you could block just the TV running the proxy. Modern crowd sourced security products could identify the proxies and share their IP live so you can ban and unban addresses as the device moves.
> It's unfortunate that good users will have to endure some difficulties because of bad users on the same network, but if security is the priority, then measures must be taken, you shouldn't just leave to door open to potentially known bad users / traffic
Is this like a corporate support response? I legit can't tell. Try explaining to a retail consumer why they can't buy from your site because they have TMobile service.
They’re saying T-Mobile should be monitoring traffic on their network and block traffic from bad actors that ruin it for everyone for everyone else? Or at least they do in the next paragraph.
When the bad actor is grandma who has an infected computer, T-Mobile will be dealing with a lot of support calls from objectively non-malicious actors.
>you shouldn't just leave to door open to potentially known bad users / traffic
Why not? If security is the priority, the worst that could happen from that is clutter in logs and higher resource use. Nobody wastes 0days spraying them on random hosts, and a real motivated attacker targeting your server will not be stopped by an ip blacklist.
286 comments
[ 0.23 ms ] story [ 105 ms ] threadFirewall it from your local network if you want to - it'll still work - they're not after that.
https://darknetdiaries.com/episode/172/
There's a transcript if you want to read, not listen. At the start, you'll be "hey! I gotta get me one!" and later you'll be like "burn it with fire!"
“ Most of the infected devices were manufactured in China. ” uh… most devices are manufactured in China period. Very strange article/podcast.
Such things are threats to humanity imho. They can get us back to the times we couldn't exploit the internet to run a society.
* I initially thought this meant someone in the WH network was using commercial proxies to appear as if they were exiting from a different network (which would align with "some user is firing up a proxy on their phone to exit in a different location")
* But then the description of residential proxies seemed to align more with traffic hitting an endpoint in the White House, and _exiting_ from the network there. But this didn't explain the timing (why isn't it 24/7).
* Finally it sounds like some person, legitimately on the WH network, has a device (cell phone, TV, whatever) that, when activated, is proxying outside traffic into the WH network.
While they have grown a bit too much, they are a natural response to the absurd amount of effort every website is pouring into anti-scraping tech.
> and can be used for extra anonymity
Great, so you can conduct illegal transactions and have some unsuspecting grandma take the blame and have her front door kicked in?
Technically correct statement though
Residential proxies are in many cases at least shady, both on the side of who uses them and for what and on providers that are often not above using dark pattern to hide them in some innocuous app for a smart-whatever that silently runs a proxy on a unaware user's connection. And in many places in the world if illegal stuff in done using your connection you are liable, regardless if you knew about it.
So important that we have to trick people into using them. Remember when folks were running SETI@Home? Of course you do, because it was widely thought to be a good thing, and folks willingly signed up.
How many would buy the "box" if the advertising were honest about what's actually in the box?
I buy and use residential proxies, though.
This also puts some of the troubling reports that politicians were bringing phones into SCIFs into context…
On a server that used to get below 1k HTTP requests a day, I've had several cases of mass scraping from these botnets exceeding 100req/s. The web is getting harder and harder to use due to all the anti-bot mechanisms site operators had to put up.
How is DDOSing every site on the web benign?
Indeed. I recently set up a bot on my local machine that I can interact with via Telegram. I gave it the ability to retrieve the contents of a URL via Playwright (with modifications). Basic things like getting the Readability view and getting the site as Markdown with links.
I was essentially mimicking the feature OpenAI etc offer where it can do a web search for you and get you the answer you seek. I just didn't want to pay a company for that feature.
It's not autonomous. It only acts when I tell it to. The actual search is via the Kagi API, but the URL contents is via Playwright. I also ensured it can't do lots of requests at a time. There's a mandatory (random) sleep of between 1 and 5 seconds between two requests.
In less than a week, I've been flagged. When I do a Google search (as a human), I have to prove I'm human. Many sites have simply blocked me - I can't buy stuff from various vendors[1], I cannot access several news sites, etc.
The thing is, I didn't really access that many sites with the bot! At most it's the equivalent of someone mildly addicted to the web. Yet they caught me quickly.
Obviously, I disabled the bot's capability to get site content. I wonder how long my IP will be in the blacklist.
I know some people will say "Good riddance" to the outcome and to what I did. I'd love to have a real, serious discussion on whether what I did was immoral. But whatever your thoughts on the matter, do ponder over the fact that the system has made it hard for average Joes like you and me to have this "feature", but if you pay a giant tech company a monthly fee, you can have it.
[1] Thankfully, Amazon didn't block me. In the past I'd avoid them, but now I'm in the dilemma where I don't want to buy stuff from sites that make it difficult for me to buy stuff from them!
I will instead say "skill issue".
Using Playwright results in some very obvious and easy to detect fingerprints. I'd expect better of someone on hacker news.
Mine hasn't been detected as far as I can tell, so I know it's possible to do better.
I normally use a VPN to access the Internet. It's an ordinary consumer-grade VPN, and I use it for pretty innocuous things like watching the international versions of Netflix or avoiding being geographically-profiled and charged more when I book travel. The VPN itself, I'm sure, is being used for all sorts of naughty stuff: when I look it up in whatismyip, it comes up as VPN, "abuse detected", and "residential proxy detected". I expect to get blocked occasionally on it and just pause the VPN then.
What I found surprising is that pausing the VPN doesn't always clear the CAPTCHAs. The next request will also fail, sometimes. For a while I was wondering if I have a residential proxy on my network, but I went hunting with lsof and bandwidth monitoring and a list of all the open connections through my router, and it seems clear. Yet even with VPN disabled, I'll get CAPTCHAs from Google and Cloudflare.
I suspect they're using browser/TLS fingerprinting and then smearing the results of that across bad IPs. If you connect to them through a known-bad IP, they not only will CAPTCHA you, but they will record a fingerprint of any data they can. Then if you connect again with that fingerprint, regardless of the IP, they block that too.
Ironically, if you are actually malicious, you can get around this pretty easily. Because you'll know not to use a fingerprintable browser, or to rotate fingerprints. So you have one completely separate identity on a given IP (that is not yours), and if it gets blocked, you roll up another completely separate identity on a different IP. But this is too much work for an ordinary consumer who just wants to keep their IP private. So you end up with this excluded middle where the only rational strategies are to either do exactly what the website expects you to do (with no funny business, even if it's innocuous), or to go whole-hog as the aggressor where you're determined to skirt their policies no matter what, and if you're doing that you might as well amortize the effort by extracting everything of value you can from them.
That modified Playwright was based on Firefox, and that's what I use for my regular browsing. So that didn't help.
I can actually launch Chrome and not deal with this - but I'm not going to. In fact, what surprises me is that I can create a new Firefox profile and not get those captchas. I don't know how my using that modified Playwright as a bot got them hooked on to my Firefox profile. I suspect they had my FF profile as a "suspicious" entity even before I made the bot (I use a number of privacy/anti-cookie extensions), and the bot pushed the score over the edge.
I'll probably start fiddling with my profile - disabling extensions, until things seem to work. Thankfully, no one else in the household is impacted.
For now, I got around all this by just paying for the Kagi extract API. It's not as good as my modified Playwright had been, though. I thought of a VPN, but I suspect that won't last long (not a residential proxy, etc), and I don't want an ongoing monthly fee. I like Kagi's pay as you go option.
But as I said, it's a weird world where suddenly Amazon is coming out the "better" site/vendor. It's funny - in the old days I was pretty anti-Walmart, but a few years ago I started preferring them over Amazon (and even several local stores) for a number of reasons (better collection than local stores, not having to deal with fakes for online orders, etc).
I've not been liking the state of the web for years prior to the advent of LLMs, but this is feeling like a final straw where I accept "the web is dead". We need a fundamentally different protocol - the model of "Set up a server anyone can reach, put content out there, and then put up a lot of sophisticated technology so your server isn't hammered" is not tenable. The protocol itself needs to manage the load.
I made my modifications at the network level and auto redirect a configurable set of sites to go to reader mode/paywall bypass by default with a MITM proxy and a custom CA cert I trust
While your server handles that, what does your CPU do with the other 99% of available cycles?
I run https://bunnystomp.com on an Always Free VM from a major cloud provider. It handles 1,930 req/s at 33 ms latency with no tuning/optimization. It's literally just python behind cloudflare.
I have no idea what the site you linked does, but it doesn't look like a cgit instance to me; that's what got hammered on my end, and one very quickly finds out that a lot of git operations are CPU-bound when hosting it.
What are you doing to make this true?
I'm a screen reader user (which is as "weird and unusual" as it gets when fingerprinting mechanisms are concerned), and I am not really noticing much difference versus 2022 or so. Sure, there's little bits of anubis here or there, and a few sites (which I know for a fact are experiencing targeted DDoS attacks unrelated to AI scraping) having annoying Cloudflare verification screens, but that's about it.
My setup (other than the screen reader) is as vanilla as it gets. Safari, AdGuard for Ad blocking, no VPN except work VPN when necessary, no PiHole, no settings changed. If you're using something more esoteric, maybe try going mainstream and see if that improves your experience.
2) They're clearly talking from the perspective of the user and network admin. A residential proxy virus is in fact quite benign for both.
3) No matter what, DoS is quite a low priority in terms of security. And it's not even really DDoS, you'll know a real DDoS when you're on the receiving end.
That's only because it isn't being prosecuted appropriately.
I'm all for creating consequences for people who negligently don't secure their systems and people get hurt because of it, but people who installed a mobile game that has a proxy library in it? Please, chill.
I'd say yes since you've allowed random people unauthorized use of the connection. It makes no difference if it's your connection or not. For example, if that proxy gets used as part of a DDoS attack, your device compromised that network.
Yes. And someone did get unauthorized access to it. You don't have the permission to delegate authorization to a third party.
Did the third party connect to the guest wifi or did they connect to your computer and you have allowed them access to the guest wifi?
Did they agree to the ToS that the wifi network requires for people to use it?
Are you certain that the wifi network isn't intentionally limiting the area where it is broadcast, allowing only people within that geographic area the ability to connect to it?
If there is a purely 100% open wifi network who is only limited geographically due to technological constraints but would allow anyone in the world to connect to it if the technology allowed, then using your device to solve the technological constraint of limited wifi range would not fall under compromising the network IMO. Assuming again that the operator of the network publicized it as such and you aren't just making assumptions about their intentions.
There are two main differences. Netflix is working on your behalf. Connections initiated from netflix are no different from connections initiated from your web browser or your email client. Your use of the internet, for you, is authorized. With a proxy you have no idea who is connected or what they are doing with that connection. That network activity uses you, but isn't by you, or for you.
Finally, netflix isn't likely to be hacking, spamming, or attempting to bypass bans or IP blacklists over that connection, but those are exactly the kinds of activities that shady residential proxies are used for. You basically backdoored your own computer, and now access to the internet through it will be sold to people who want to use it for something that they couldn't get away with by using a VPN or any other legitimate provider.
Depending on the sophistication of the wifi network you may be held liable and accountable for everything that takes place over your connection, including what ever nasty things the people who bought access to your proxy get up to. If the wifi system isn't set up to keep careful track of who is using it and when the liability will fall to the entity offering the wifi access, and if that connection is repeatedly used for activity in violation of the law, or the AUP of the ISP being used, that same entity will face the repercussions. Their internet connection may be temporarily suspended, and they may even be forced to stop providing wifi access to guests entirely unless they are willing/able to spend the money needed to log and monitor who is using their connection and respond to notifications of abuse over their network as well as legal requests.
If your IP address is part of a residential proxy network, it will be flagged.
Also, I recommend seeing the device count data we have for you. If you are on a NATed IP, you have virtually no control to remove the residential proxy SDK from your system. Let me know if you have any questions, as I work for IPinfo.
It could be in a security lab for all they know.
I was aghast when my partner connected a pet watering bowl to the WiFi.
Multiple times I caught unexpected traffic and was able to block it piecewise including a lightbulb(!) that was sending mb packets of god knows what home.
If they were ethernet based I reckon some smart VLAN capable switches would be nice even at home to keep them out of each others business.
I moved home a few years ago and now I'm back with the ISP router and do often feel a little in the dark about what my network is doing!
https://openwrt.org/
I had to recently inform my sister and didn't realize just how unaware the general public was of how utterly terrible things have gotten, like with LG TVs or the Meta Quest's access to your family.
When someone tells you "I have nothing to hide, I don't care", let them know that companies like LG make 5x as much money selling your data as selling $1000 TVs, so clearly your data is valuable and you shouldn't be giving it away for free. That's a message normal people can understand.
{
}We collect the residential proxy data continuously from independent systems; when you visit the page, we take your IP address and look it up in our database. Note the percent_day_seen field, which is frequency metadata. You can also complement it with our device count data to understand if you are on a NATed IP.
If you are on a static IP address and a single user, we flag your IP address as a residential proxy. I think you should install some DNS monitoring service and start uninstalling unnecessary apps from internet-connected devices.
— Abdullah | DevRel, IPinfo
Most likely scenario is a 20-something holdover from DOGE who accidentally had his phone p0wned while downloading groyper memes.
There is no evidence that physically ties this to the WH. It could be in a security lab for all they know. Or a guest network for the news media.
They don't need to be physically inside that building to be a network security problem.
That seems worse.
Let me guess. The SDK being used does properly handle this and this whole article is anti residential proxy FUD.
Step 2: have you bought any devices made by large companies like LG or Samsung?
Now, how to prevent this. If you are on a NATed connection, it is nearly impossible to prevent this from happening because you cannot control what apps people install who share the same IP address with you. If you have control over your network, then I suggest you install a DNS monitoring service. You can periodically monitor the DNS requests to find unusual patterns.
Not sure why my IP is listed as a residential proxy.
It should be a balancing act though, somewhere between what you said and my first sentence.
Rate limiting, or additional layers of input sanitisation for IP addresses known to have been malicious in the past.
It's unfortunate that good users will have to endure some difficulties because of bad users on the same network, but if security is the priority, then measures must be taken, you shouldn't just leave to door open to potentially known bad users / traffic.
IPv4 is a shared resource, but some ranges have worse reputations than others, and all are owned by organisations that have the ability to monitor and filter for malicious traffic. It gets into a fraught discussion about censorship, but it is possible for network owners to monitor for, and block / restrict traffic detected as 'bad'. It's just a matter of cost and scale, and those that put the effort in should have their users rewarded with fewer 'trust-issues' roadblocks.
The lists generated by this tool can be useful as well: https://github.com/maximilianfeix/proxy-scraper (which point to here: https://github.com/maximilianfeix/free-proxy-list)
There is also this bot proxy IP list: https://github.com/antoinevastel/avastel-bot-ips-lists
Is this like a corporate support response? I legit can't tell. Try explaining to a retail consumer why they can't buy from your site because they have TMobile service.
Why not? If security is the priority, the worst that could happen from that is clutter in logs and higher resource use. Nobody wastes 0days spraying them on random hosts, and a real motivated attacker targeting your server will not be stopped by an ip blacklist.