> There were many times where I saw a big traffic jump and I realized the traffic was coming from the same ASN, and likely from the same company. I tried reaching out to these companies when I saw it but they rarely ever replied. Some even became extremely hostile to my emails.
A hostile reply from a netblock operator seems like a perfectly valid reason to block their traffic.
The problem is that you don't know what the source of the traffic is. It could be an incompetent network operator/sysadmin, but it could just as well be something like an IP camera that people bought in good faith. If you block the CGNAT system of an operator that has a hundred million subscribers because it all seems to come from a single IP range you know nothing about, you could be hurting innocent users with the block.
That being said, a service like this doesn't come with any guarantees and if it'd disappear from the net tomorrow, I wouldn't blame the author. Blocking is a perfectly valid solution to this problem, but assuming malice isn't always the right answer.
Were I in this situation, I'd rate limit networks per /24 (maybe even /16?) as much as I could, and work together with antivirus companies to help identify infections of malware known to use the service to discourage criminals from abusing the system. I wouldn't even bother hosting the site on IPv6 since those addresses are supposed to be public anyway. The author clearly has more patience than I do.
Looking at the icanhazip.com site, I wonder how much any kind of rate-limiting per address/block would even help.
At the HTTP level it's probably cheaper to just return the HTTP 200 response. I suppose if you're doing TLS handshakes then a packet-level rate-limit would help significantly, but at the same time I'd be wary of triggering any kind of retry-behavior.
Worst-case scenario for a service like this would be having an error response/timeout trigger some kind of unlimited retry flood.
The block route I'd go with is blackholing the entire range into nothing through BGP or similar so the servers wouldn't have to deal with the traffic, similar to how anti DDOS tools often work. Might even redirect the DNS for that subnet to the IP of the people running the network, let them deal with the abuse. That'd be a very offensive approach, though.
I probably wouldn't bother with TLS either, just a plain HTTP 0.1 response with minimum information should be enough.
In some sense, it might not matter. If an ASN/company admin responds to emails in a hostile fashion, does it matter if they bought their devices in good faith? They're still assholes.
Hostility can often come from a place of ignorance or misunderstanding. I can't say much for the former, but the latter can easily go wrong with the cultural and linguistic barrier between operators.
The guy operating the NOC may be a dick, but is taking down the IoT networks for all of their customers unknowingly relying on your services really the right way?
Personally, I'd say yes, it'd help. However, there's an argument to be made that the hostile ASN operator doesn't represent the people behind the network in the slightest. I can understand that someone may give such an asshat the benefit of doubt and drop it despite their abuse.
This isn’t “one person bought a bad camera”, it was certain ASNs accounting for a huge portion of the traffic. If the operators are unresponsive to the abuse request (making them incompetent network operators), then you absolutely block them. At that point the fallout is the fault of the network operators for operating an abuse friendly network.
This is how cloudflare handles it for normal web services. If you’re coming from trash IPs there is no chance a curl request is going to make it through to a backend without an onerous captcha.
I wouldn't expect one person with one camera to cause such a load, but popular, cheap internet cameras pull this crap all the time. I remember reading a story here about one company that hardcoded a particular IP address for their NTP bootstrapping in their firmware, with thousands of devices all across the world and no way to easily update them. Such a thing can easily happen with consumer routers and other networking equipment, generating a publicly accessible link for their user's convenience.
If I saw the Time Warner ASN send too many requests, my first thought wouldn't be to just block a huge ISP. Who knows what mihjt be causing these issues and what you could be breaking by interrupting service.
The Time Warner NOC wouldn't be able to completely fix the problem if the source of the issue is the firmware of a certain shitty IoT device. If someone emailed their NOC about some weird IP cams installed by their customers causing load on their servers, they could feel like that's a problem between icanhazip and the camera manufacturer, not something they can fix.
The author is quite tolerant of the obviously malicious behaviour others are attacking his servers with. I'd have taken more aggressive measures instead of scaling up capacities myself. Because the problem is volume and not necessarily anything complex, I'd wager that even a simple block could be quite expensive because that traffic and the associated retries will be going somewhere. Directing the traffic towards the last router in their ASN through DNS would be something I'd consider, making it the problem of the network operators.
This is not an unimportant or victimless problem, however said problem is the network operator's entire job. Making them deal with this is not uncalled-for.
This raises something I've wondered for a while: is there a service or database that can give an indication of how many humans are behind a particular IP address? e.g. with CGNAT, there might be many thousands of people sharing a single IP. For some residential services, it might be 1-2 people.
It feels like this sort of data (even if only providing order of magnitude estimates) would help greatly with deciding on appropriate rate limits for small operators who don't have the time to research all the traffic they're receiving.
As a network operator, you would be surprised how many AS operators are hostile or simply don’t respond. It’s unfortunately very common, even Tier-1’s are hostile.
Out of the last month, I sent out 191 abuse reports, of which 10 got replied to, 2 were resolved 6 were “no f** off” style, and 2 were told “can’t fix / won’t fix / don’t know how to fix”.
I’m not just referring to Chinese ASNs either, some US Telco’s, German, Australia even.
We're entering an era where the content providers and the major clouds are >50% of demand on new subsea capacity. Feels like this blurs 'Tier 1' and the role of backbones because major eyeball networks now interconnect directly with e.g. Facebook or Google.
Anyway, on abuse@ response rates, my probably unpopular but realistic take based on looking at tens of thousands of such complaints over the years and having worked for ASNs which have received millions, I'd hazard everyone has an SNR and ROI problem with handling these. There's just too many of them and most aren't actionable.
Some examples, "I saw a failed SSH login attempt from 1.2.3.4 and OMG that's a huge issue, you have been compromised, and you must solve this immediately!". OK, well, the subscriber might have: a) Typoed your IP address, b) Been running nmap/zmap over a wide range of IPs for research purposes; c) You're on an IP with a provider who recycled it to you, subscriber has outdated DNS records.
What do you expect a 'Tier 1' to do with your report?
Many ASNs are now just looking at the pattern of reports per IP address or subscriber, are automating scanning for e.g. open mail relays when whatever processes abuse@ determines the person is complaining about spam, or automating looking for anomalies in flows for DDoS complaints, a human may not even see the ticket unless the automation was able to confirm a problem may exist, and the human will probably only engage the subscriber and won't respond to the 1-1000 things received to abuse@ related to the issue.
In Major's case with icanhazip.com it looks like pretty bad behavior from the Chinese ASNs mentioned, but could just be IOT configured to fetch its IP every minute instead of every 60 minutes of 24 hours because someone misunderstood cron. Unfortunate that nobody responded but 30B a day is ~350kRPS (which isn't a lot, in the grand scheme of the internet). I'm sure 30B requests per day is nothing at Cloudflare's scale and they have options to cure these ASNs behavior should they choose, including stuff like IP-based or ASN-based ratelimiting, or even IP/ASN restrictions.
I'm sure Cloudflare will learn some interesting things about both the accidental contributors (e.g. cron) and intentional contributors (e.g. botnets) from analyzing the sources generating the requests, and I'm ultimately glad it is them picking this up, their other initiatives like 1.1.1.1 have had been positive for the internet (IMHO).
I've been using ipinfo.io for several years -- checking a dynamic ip address every 10 minutes. My thanks for supplying this service! Is there a reason to change over to icanhazip ?
No of course no reason to switch away from us! We provide many more details than icanhazip and others (geo, asn, anycast etc). We handle 40 billion requests a month,and plan to be around forever!
I'm just glad that icanhazip is staying up - it sucks when useful services go offline, and negatively impact all their users.
Chinese originated spam and abuse is so outrageously widespread, I don’t understand why there isn’t a conversation going on about cutting them off from the wider internet. They blocked most of it anyway.
I believe in reciprocity. China has blocked a lot of the western traffic. So, the west should block China. If they open up, we should welcome them with open arms. Similar spirit as some open source licenses - reciprocity creates fairness and increases collaboration, prevents hawks in a population of doves and improves stability.
We are already doing this with trade. The amount of leeway and free lunch China has gotten from the west is insane. I don’t blame China, I blame the west and the rest of the world for not preventing it. Asymmetrical policies are often exploited by capitalism and governments have been caught off guard.
I’m not an Anti-China lunatic. It’s just common sense.
I really don't think blocks and embargoes are going to help anyone. They just suck for all the affected people, but I don't think they are very effective at convincing foreign governments to open up.
We're talking about blocking IPs originating from China. How would that hurt the affected people outside of China?
In regards to trade war, HN has discussed this ad-nauseum, I think we should restrict the discussion to internet traffic even though I brought it up as an analogy about asymmetric response from the west in general: https://hn.algolia.com/?q=trade+war
The lives of people in China trying to use Internet services outside are already miserable; let's not make it worse and alienate others. We should treat them just like the rest: if extensive malicious traffic arrives, we drop it, but we don't ban the entire country.
The internet is built on institutional trust. You can’t have a properly functioning network when a sizable part is just not giving a shit about its users abusing your users.
It very much echoes the problems of intellectual property theft in China.
This supposed 'intellectual property theft' is mostly just reverse engineering of technology.
It's not really a problem anyway. If some capitalists in the US and Europe don't get to skim off a slice of profit from another country's manufacturing output, then so what?
Autonomous system (AS) operators should not act in bad faith towards other AS networks they peer with. BGP hijacking is an example of abuse of this institutional trust, same for DNS hijacking.
If this trust is repeatedly broken, peering networks may be forced to depeer the AS as a result, like what happened to McColo when they were depeered.
The Chinese government operates their Internet blocks (the “Great Firewall”). But overwhelmingly, it is the Chinese people who are trying to access information on the public Internet.
Blocking the entire country will do little to hurt the government (who can employ state resources to get whatever information they want) and do quite a bit more to harm the Chinese people by reducing whatever level of information independence they still have.
If there is going to be significant change in China, it will have to come from the Chinese people. Cutting them off from the Internet vindictively does not advance that goal.
There are specific people in China doing specific bad things with specific computing resources. It would be far better for the U.S. government to dedicate more resources to finding and partnering with orgs and projects (like icanhazip or Cloudflare) to find the info they need to apply targeted mitigations.
“China does it, so we should do it too” only makes sense as a strategy if our goal is to become exactly like China is today. I don’t think that should be our goal.
> “China does it, so we should do it too” only makes sense as a strategy if our goal is to become exactly like China is today. I don’t think that should be our goal.
I very strongly disagree. An eye for an eye is exactly what needs to be done and should have been done from the beginning. Unfortunately, it is too late. 1989 massacre should have been condemned more solidly and trade restrictions should have been placed in the 90's. The bet that western alliances made is that China would open up in the 2000s leading into 2010s. That has gone horribly wrong.
> I believe in reciprocity. China has blocked a lot of the western traffic. So, the west should block China. If they open up, we should welcome them with open arms.
This sounds like a couple of people I've met, who have a philosophy of "treating people the way they treat me". And what if the other person/side also "believes in reciprocity", what happens then? This seems to rely on other people being nice first, and then always treating them how they treated you, imitating their behaviour, like Tit-for-tat[0]—except Tit-for-tat begins by being nice. It's not easy to put my finger on what seems fishy about that strategy, but it doesn't at all seem the easy solution to being fair and just (or whatever word you most prefer here) its proponents seem to think it is.
One can have a tit-for-tat policies on a geopolitical scale but also excercise forgiveness in most situations in life. I think Tit-for-tat is a terrible strategy on the whole and agree with your philosophical stance - I hope you didn't mean it on the personal level.
Apologies, I was hoping you didn't mean to judge my personal character from a single data point on China's policy. I think I am pretty forgiving and excercise tolerance because I know when these things are difficult is exactly when it matters the most. Tolerance is the price we pay for freedom and liberty.
This false dichotomy is impressive. A single country accounting for for 50+% sets up the choice to be, “a global network with a lot less spam and a regional island with a lot of spam” vs “a global network with a ton of spam barely connected to a regional network that much of the spam originates from”.
Calling China "a single country" minimizes the fact that it contains 18% of the world's people. It's hard to call something a "global network" if it leaves out that much of the world.
Any criticism of China can be construed as being pro-US and we can't have that. Criticism of Chinese traditional medicine, which is a forcing function for the endangering and extinction of species across the globe, is also unacceptable since criticism of any aspect of a culture is just racism (unless it's the one culture that's okay to criticize).
As long as the truth doesn't match what the preferred narrative is we'll continue to suffer the consequences, which is true of so many things beyond just attitudes towards China.
Many of us in China are absolutely saying Chinese Traditional Medicine is garbage and there s no "western medicine", just medicine.
It's not racist in China to say the truth, why is it where you are ? You probably live in an oppressive political regime with a biparty dictating what you can think ? :P
> Many of us in China are absolutely saying Chinese Traditional Medicine is garbage and there s no "western medicine", just medicine.
I sure hope that "just medicine" extends to Po Chai pills. So welcome for treating diarrhea symptoms (despite my initial skepticism) when loperamide wasn't available while I was traveling in China!
China currently makes some absurdly large percentage of the world's consumer goods, and the discussions about producing them are probably being had over the internet. Cut them off of the internet and we have to rebuild manufacturing capacity everywhere else.
Which might not be a bad thing overall, but it's sure not gonna make any transnational corporation's bottom line happy over the next few quarters, so they'll be waving a lot of money at politicians to make this not happen.
Because doing so would essentially push China towards a China-only internet, which they're already halfway towards.
The benefits of gobalization and the spread of democracy (or even just alternative governance models) via exposure to other cultures cannot be understated
Not a strong reason. I would be shocked if the average Internet user has heard of any of the top ten most visited websites in China. Their entire infrastructure, from the technological layer to the bureaucratic layer, has ensured that the average Chinese Internet user knows very little about the outside world that hasn’t been pre-vetted or filtered out completely by the GFW.
Im in China and met a Xinhua journalist once. At day she would edit propaganda stories that she knew were complete horseshit, at night her and her colleagues would go to their boss watch netflix together because he had a working VPN.
Even if just one port is left open, people will be curious enough to find it and use it. Chinese people are humans too :D
I liked this hypothesis overall—that exposure to democracy through trade is sufficient to breed democracy in China. It’s a confident and peaceful approach, and I’m glad that we tested it. However, in this case, I believe we’ve disproven the hypothesis; continuing to run the same experiment unmodified and expecting improving results is signing up for disappointment.
> continuing to run the same experiment unmodified and expecting improving results is signing up for disappointment.
Maybe that's an acceptable price to pay for not being the ones to take the next step toward war. If war is a game in which "the only winning move is not to play", then maybe it's also true that when it comes to doing the peaceful thing, the only winning move is to keep on playing, even if it hurts us.
I disagree: I m not sure what's making you say it's not working but from a Chinese point of view, this was also an experiment, to try and open up a little.
Are you sure you're doing your part of the bilateral exchange? It cant just be China changing, the US must learn too to adapt and accept a larger, more powerful country, with a widely different model.
Living in China, I can tell you the american model is known, and not particularly impressive to them. They care a lot less about freedom of speech, maybe because they never had it, than they care qbout order, unity and crime rate for instance. And what I always hear is that throwing themselves at the communists in revolt to get the same shitty system as the US is not so seductive.
Maybe become a role model and people will beg to ressemble you ? I have a hard time convincing them voting for their government is gonna work better because "if even idiots can vote, look at who they elect" :s
It's totally an option if you're in a country effectively ruled by soulless transnational corporations who rank "making lots of money" several orders of magnitude more important than "any kind of ethical concern".
It's probably totally an option if you want to work for one of those corporations, too.
It's not even a new trend either. Back in 2003 when I worked at eBay and PayPal doing security, the bulk of the attempts came from China and Romania (Romania at the time had one ISP for the whole country that was fast but didn't care about abuse at all).
It stands to reason that an especially large volume of abuse will originate from the most populous country in the world. I don't think that's a reason to cut them off from the global Internet. If it's true that their government is already oppressing their own people (I don't know what's truth and what's propaganda), then the rest of us shouldn't make it worse for those people by cutting off whatever outside connections they manage to have.
Also, I'm generally bothered by comments like this one that stir up the general human tendency toward xenophobia. We should be fighting that tendency within ourselves, not fighting the out group. Whichever group of people we want to demonize, we should remember that they're people just like us. We shouldn't punish the majority of them for what a minority are doing to us.
They are not the most connected country, but a long shot. Why isn’t India no. 2 by your logic? Stop apologizing for unacceptable behavior from a country that openly purports to become “the superpower of the world”.
I'm not apologizing for anyone's bad behavior; I just don't want us to escalate an already tense situation. "The only winning move is not to play", right?
I don’t think number of people with internet access is a good measure of connectedness at all. Curiously, you dodged my question on why India isn’t the same when your list also shows they should be.
I mean, if their government doesn’t stop, and often even encourages the behavior, what are we supposed to do? Just roll over and show them the other cheek?
I agree you don’t want to cut them off, but on the other hand, I don’t want 90% of all global malicious traffic to originate from a specific country.
> I don’t want 90% of all global malicious traffic to originate from a specific country.
Is that actually true? I guess I'm inclined to believe that claims like that are more likely to be propaganda from western governments and/or western-owned companies.
If it is true, I wonder why their government isn't stopping it. They must realize that it's giving them a bad reputation in the wider world.
It was true (well, in the same ballpark, don’t remember exact numbers) for the website I was in a position to see it for. It may be different for others, but like 98% of the malicious traffic comes from 3 or 4 countries.
sorry but this opinion is ridiculous on its face. Blaming china because of population is a complete joke. China (by a large margin), Russia, and poorly configured proxies are 95% of all malicious traffic targeting US-based businesses. It's not even a question.
If you run a small-medium sized business in the US blocking all of countries you can't do business with anyway will save a ton of trouble.
> China (by a large margin), Russia, and poorly configured proxies are 95% of all malicious traffic targeting US-based businesses.
My reasoning for these kinds of stats is usually: Of course it makes sense to attack targets in jurisdictions which can't catch you or equally hide in a country which won't extradite you. (But I never looked into it to any depth, so it's baseless reasoning.)
yup. toxik's comment disgusts me. So much for being against the government and the bad apples and not against the common citizen. Let's just cut off all the devs doing their jobs every day from accessing github, or let's just cut off everyone who was curious enough to bypass the GFW and look around on the outer internet, or let's just cut off someone simply trying to contact an international friend. The truth is: people like toxik could not care less about any person who happens to live inside the borders of "public enemy no. 1".
Your comment contains no information and only presents an extended ad hominem. Go read the site rules please. This isn’t the place for moral grandstanding.
Most Chinese internet users would not miss Western internet for a second, a fact you would be aware of if you actually had any insight into Chinese culture.
This attitude that you cannot give consequences to abuse because THINK OF THE POOR CHINESE is so utterly laughable.
> Also, I'm generally bothered by comments like this one that stir up the general human tendency toward xenophobia.
Most countries cooperate internationally in getting bad actors from hackers over pirates to pedos booted off the Internet and into jail.
The exceptions are China and Russia who won't do anything against any bad actor and India which is a big base for phone scams (as is Turkey for the European Union, but even Erdogan's regime is cooperating with EU police in taking down scammers).
I agree, the line between demanding at least some sort of common decency standards and xenophobia is thin in these days, but we have to get everyone on board to protect everyone else from rampant abuse.
15+ or so years ago I worked in the NOC of the 3rd or so largest ISP in the US and a random network engineer did this one evening. We got a big influx of customers complaining about email not working to their family, etc, until I finally figured it out.
That network guy (classic long hair "security" guy) was a lazy asshole for doing it then and the internet needs to have the technology to deal with bad actors beyond AS/geo-level blocking now.
I was using icanhazip to check if my Tor circuit was complete, and probably made 50-100 requests per week. The site was getting slow, and I thought it is just a random site that the author didn't really care too much.
I dropped my jaw when I read it was getting 30B req/day.
Thank you for running this site for so long, and thank you for keeping it up for free, and deciding to not monetize it.
I got a lot of mileage out of neverssl.com before somebody fixed the process to log into various "guest wifi" setups...ones that would intercept/redirect any http request.
I'm somewhat curious what fixed things, as I've not had to use neverssl.com for some time.
From what I can tell, most operating systems will now ping their own version of neverssl as you connect to a network to find out whether they need to show you a login prompt. It looks like they basically just check to see if they get the content they expect from a domain they own, and if not they serve you that page so you can see whatever it is your network injected. (You can usually see the OS domain in the address bar.)
Apple has at least ten or more of these I’ve seen - on badly configured networks you sometimes see it in the address bar - because cached responses could destroy the utility.
Ah, that's interesting. I remember it being very broken for a long time...especially for "normal" users that wouldn't understand why navigating to an https site wouldn't work in that captive wifi situation.
Annoyingly, I've seen some networks that try to "fix" problems by letting that go through even when not logged in. I'm pretty sure it had been fixed last time I flew, but Qantas in-flight WiFi used to be one example I'd seen. You'd connect, and then nothing would happen and no SSL connections would work. You'd either need a non-HTTPs site to get the redirect, or go to 'wifi.qantas.com' to accept the terms and conditions before you could browse. I was trying to work out why my iPhone and Mac weren't popping up that page as soon as I connected, and bizarrely captive.apple.com came up with 'Success' instead of redirecting, which means they must have misguidedly put in an explicit rule to let that through, completely breaking the feature!
I set up my own version of both neverssl and icanhazip, with nothing but Nginx on a cheap VPS. I already had the server up for other purposes, and I feel better knowing that I'm not mooching off of other people's effort (and money).
Neverssl has done some pretty nifty work to avoid caching. It redirects you to a random subdomain over plain HTTP just to make sure the browser has a cold cache. Maintaining a domain, the redirects, and making sure to _not_ accidentally obtain a certificate is a burden I wouldn't want to do, although it is not that difficult to do.
I was reading from neverssl maintainer that they get a _lot_ of traffic, questionable ones more than it is not. Its DNS runs on AWS IIRC, and we all know Route53 isn't the cheapest.
Having lived through the debut of Firesheep, which prompted the industry to get serious about using TLS, its an amusing triumph of cybersecurity that today a site has to be careful to not accidentally get issued a certificate. Back in 2010, when certificates cost substantial sums and needed some expertise to apply for and install, I wouldn't have guessed we'd ever get to this point.
Ha, I could've used this ~a year ago when I moved into my current rental apartment. Fiber optic internet is included in the rent (in the form of an RJ45 jack in my wall), but there's a captive portal requiring you to enter your contract ID, which resulted in having to scour my memory and bookmarks for a non-HTTPS site.
I’ve seen packages that do ”internet-detection” by calling out to icanhazip.com, and I just thought that was so irresposnible. What if your package got popular, how much money are you costing the hoster? For services like this, people just don’t consider the fact that there’s someone on the other side.
I feel the same about dependency steps in CI, without a cache or any similar structure. Package repos like Rubygems, NPM and PyPi get utterly rinsed by the continual downloading and redownloading of stuff the client should have already stored.
This. And both with GitHub and GitLab it takes quite a bit of an extra effort to setup caching. It hurts to see 'npm ci' download half the internet every time a developer pushes to dev server.
It used to be possible to have a squid or even a backwards varnish cache handle lots of this but https everything has made that much harder to do. Still possible, however.
If nothing else, it is patently wasteful and, as a user, you don't really see CI billed in terms of network bandwidth. Just indirectly through the equivalent of mainframe minutes. And even then, that's not enough to discourage anyone from building a suboptimal pipeline.
That's why the first step of CI for me, when possible, is to rsync a .tar.gz file from the server I'm deploying to. The tarball contains statically-linked binaries and other stuff I'll need for the build.
It's also a good reason for CI providers to mirror package repositories.
The article was about abusive floods accounting for 90% of the traffic. The author was happy with legitimate use cases like packages doing detection, contrary to your comment.
Requesting "yoursite.tld/ip" will then return your IP address. I set up something like this on all my servers and recommend that others do the same. It's easy to do the same for Apache and Caddy configs. That should help spread the load.
I'm curious as to what other overused utilities can be trivially done with pure server configs.
I used to use this site until I found https://checkip.amazonaws.com/. Switched because I wasn't sure who was behind icanhazip.com and it's tough to beat AWS. Glad to hear that it will likely be maintained for awhile longer!
I think the point is to prevent middleboxes (eg. caching proxy servers) from interfering with the request. Otherwise I don't really see the issue with the ip address being affected by X-Forwarded-For. You can just... not specify the header.
HTTPS encrypts headers, thereby preventing other people from adding headers to your request. Typically people are not adding X-Forwarded-For to their own requests.
I'm not arguing either point, I just pointed out that headers are independent of whether you use encryption. But now that I'm thinking about it for a sec, you might want to know what the proxy's exit IP is, and if the proxy adds an XFF Header then you just learn your own IP which wasn't what you wanted. If that is what GGGP meant.
I feel like in theory google should be returning this site, instead of the ad-filled sites when one searches "my ip address." But it always seems like Google heavily over-values the domain name and search term matches.
The "first result" in a Google query for "my ip" and other combinations is a box with your public IP. There's no reason to click in any of the ad-filled sites anymore.
This had been true for me for several years, but recently, I have found that Google no longer provides that info box and it's necessary to click into one of the search results. (I am using Firefox, but I am logged into a Google account of it makes any difference.)
Yes that's true, but besides the point. My point is that obviously relevant (to a human) results like icanhazip don't show up anywhere just because the domain name is... creative. This happens to more verticals than this one, where the top results happen to be websites with some SEO tricks like having a domain name that matches the search term.
Thanks for all your hard work! icanhazip.com / icanhazptr.com have been incredibly useful.
Small feature request: back in the day {ipv4,ipv6}.{icanhazptr,icanhazip}.com only had A / AAAA records, but now it seems they have both and thus a simple "curl ipv4.icanhzptr.com" can also give me a v6 address (of course, "curl -4" works). Would Cloudfare be OK with separating them again?
Yep, same here -- no AAAA record for v4.icanhazip and no A record for v6.icanhazip . The documented interfaces were ipv4/ipv6 though https://major.io/icanhazip-com-faq/ and {v4,v6}.icanhazptr.com have both A and AAAA.
This kind of service is exactly what STUN servers are made for. Designed to be used with webrtc, but it works perfectly alright by itself.
There are a plethora of unauthenticated STUN servers around, and while there's still room for abuse, the protocol is a bit more lightweight than full-blown http requests, and faster, too!
I've dabbled with doing this on my own, but I've found `myip` to do the job nicely and without hassle:
It is, but utility above queries multiple public STUN servers concurrently. As soon as quorum of servers replied with matching addresses, result is returned. This way it's more reliable and offers decent latency guarantees.
Wow so this person has been running this site for so many years, paying bills, answering god knows how many idiots and even getting close to trouble with 3 letter agencies and senators for absolutely nothing.. hats off to you sir, any other person would have thrown in the towel a long time ago.
Also i feel little bad you didn't get any money out of it whether the site was designed to make money or not. It would have been a wonderful end to the story if you got something back for all the years of hardwork you put into running it. You do have my appreciation if that means anything though.
P.S. this story is very similar to rawgit which was a wonderful site but also fell prey to malware aholes.
> Also i feel little bad you didn't get any money out of I
Most likely it got them a much higher paying job than they would have otherwise gotten. Walking in and saying you single handedly run a site with billions of requests per day and petabytes of traffic will get you noticed.
Being good is neither necessary or sufficient for getting a high paying job.
Being able to get in front of a hiring manager who is offering a high paying job and convincing them to hire you is the only thing. And having a very popular website with a ton of traffic is more likely to get you in front of a hiring manager with a good job than actually being good.
To drive this home, if anyone were to tell me they’re keeping ~400kRPS afloat singlehandedly, that’s “I’ll write you a req, what do you want to do?” territory.
Make sure you do that floating effect (where part of the text 'rises' from the text) IE4 and some other early CHM files had when you opened it to show off that there was a browser engine underneath with full javascript and all :-)
The certs are more just to tick some checkboxes when working with certain enterprise clients (my guess).
I hate certs, but if you want in on certain projects, it doesn't hurt to have the desired certs, especially if you can get your employer to foot the bill.
Anyways, knowing him personally, I can attest he's a really good dev and wholesome person. Sad to see this site get bogged down with malicious activity, because like downforeveryoneorjustme, I kind of have it committed to muscle memory. Hope CloudFlare can keep it going strong!
Yeah he seems like an awesome dude and engineer/leader. Definitely will be following his blog now. Interestingly however, I had never heard of icanhazip before today. Had always used ifconfig.me.
The certs prove you have the knowledge, so you don't have to boast. I've only got one certificate (AWS generic one), and I underestimated how much you need to study for things like that. I mean that was more of a nice to have, I don't actually have aspirations to go into AWS or the "ops" side of things, I just felt like it was something I needed to know more about.
Yeah once your career is somewhat established, they don’t mean all that much. My RHCA (Administrator) aged out in 2017, and I’ve felt little need to renew it.
There's a lot more helpful info you can return, too. Try https://ifconfig.me. Works great in bash scripts too, as it only returns the IP when called with a curl user agent.
I run a very simple, completely free API service as well. Currently using Google Cloud Run, handling a constant 10 rps for ~$8/mo. Pretty happy with it. I could probably cost optimize more. I sure hope I never have to deal with 30 billion requests per day, though. I'm sure my patience would run thin as well. Thank you to the author for running this site for so many years!
It will be bots using it. I have a python script that calls it every 5 minutes to get my current IP so that I can update my DNS records on Cloudflare. This is because my server is self hosted on a Pi and I don't have a static IP.
For those behind a home router an alternative is to use UPNP, e.g., through the miniupnpc package on Debian which ships the `/usr/bin/external-ip` script that postprocesses the `upnpc -s` output.
259 comments
[ 3.2 ms ] story [ 279 ms ] threadA hostile reply from a netblock operator seems like a perfectly valid reason to block their traffic.
That being said, a service like this doesn't come with any guarantees and if it'd disappear from the net tomorrow, I wouldn't blame the author. Blocking is a perfectly valid solution to this problem, but assuming malice isn't always the right answer.
Were I in this situation, I'd rate limit networks per /24 (maybe even /16?) as much as I could, and work together with antivirus companies to help identify infections of malware known to use the service to discourage criminals from abusing the system. I wouldn't even bother hosting the site on IPv6 since those addresses are supposed to be public anyway. The author clearly has more patience than I do.
At the HTTP level it's probably cheaper to just return the HTTP 200 response. I suppose if you're doing TLS handshakes then a packet-level rate-limit would help significantly, but at the same time I'd be wary of triggering any kind of retry-behavior.
Worst-case scenario for a service like this would be having an error response/timeout trigger some kind of unlimited retry flood.
I probably wouldn't bother with TLS either, just a plain HTTP 0.1 response with minimum information should be enough.
The guy operating the NOC may be a dick, but is taking down the IoT networks for all of their customers unknowingly relying on your services really the right way?
Personally, I'd say yes, it'd help. However, there's an argument to be made that the hostile ASN operator doesn't represent the people behind the network in the slightest. I can understand that someone may give such an asshat the benefit of doubt and drop it despite their abuse.
This is how cloudflare handles it for normal web services. If you’re coming from trash IPs there is no chance a curl request is going to make it through to a backend without an onerous captcha.
If I saw the Time Warner ASN send too many requests, my first thought wouldn't be to just block a huge ISP. Who knows what mihjt be causing these issues and what you could be breaking by interrupting service.
The Time Warner NOC wouldn't be able to completely fix the problem if the source of the issue is the firmware of a certain shitty IoT device. If someone emailed their NOC about some weird IP cams installed by their customers causing load on their servers, they could feel like that's a problem between icanhazip and the camera manufacturer, not something they can fix.
The author is quite tolerant of the obviously malicious behaviour others are attacking his servers with. I'd have taken more aggressive measures instead of scaling up capacities myself. Because the problem is volume and not necessarily anything complex, I'd wager that even a simple block could be quite expensive because that traffic and the associated retries will be going somewhere. Directing the traffic towards the last router in their ASN through DNS would be something I'd consider, making it the problem of the network operators.
It feels like this sort of data (even if only providing order of magnitude estimates) would help greatly with deciding on appropriate rate limits for small operators who don't have the time to research all the traffic they're receiving.
Out of the last month, I sent out 191 abuse reports, of which 10 got replied to, 2 were resolved 6 were “no f** off” style, and 2 were told “can’t fix / won’t fix / don’t know how to fix”.
I’m not just referring to Chinese ASNs either, some US Telco’s, German, Australia even.
Anyway, on abuse@ response rates, my probably unpopular but realistic take based on looking at tens of thousands of such complaints over the years and having worked for ASNs which have received millions, I'd hazard everyone has an SNR and ROI problem with handling these. There's just too many of them and most aren't actionable.
Some examples, "I saw a failed SSH login attempt from 1.2.3.4 and OMG that's a huge issue, you have been compromised, and you must solve this immediately!". OK, well, the subscriber might have: a) Typoed your IP address, b) Been running nmap/zmap over a wide range of IPs for research purposes; c) You're on an IP with a provider who recycled it to you, subscriber has outdated DNS records.
What do you expect a 'Tier 1' to do with your report?
Many ASNs are now just looking at the pattern of reports per IP address or subscriber, are automating scanning for e.g. open mail relays when whatever processes abuse@ determines the person is complaining about spam, or automating looking for anomalies in flows for DDoS complaints, a human may not even see the ticket unless the automation was able to confirm a problem may exist, and the human will probably only engage the subscriber and won't respond to the 1-1000 things received to abuse@ related to the issue.
In Major's case with icanhazip.com it looks like pretty bad behavior from the Chinese ASNs mentioned, but could just be IOT configured to fetch its IP every minute instead of every 60 minutes of 24 hours because someone misunderstood cron. Unfortunate that nobody responded but 30B a day is ~350kRPS (which isn't a lot, in the grand scheme of the internet). I'm sure 30B requests per day is nothing at Cloudflare's scale and they have options to cure these ASNs behavior should they choose, including stuff like IP-based or ASN-based ratelimiting, or even IP/ASN restrictions.
I'm sure Cloudflare will learn some interesting things about both the accidental contributors (e.g. cron) and intentional contributors (e.g. botnets) from analyzing the sources generating the requests, and I'm ultimately glad it is them picking this up, their other initiatives like 1.1.1.1 have had been positive for the internet (IMHO).
they'll soon learn
No of course no reason to switch away from us! We provide many more details than icanhazip and others (geo, asn, anycast etc). We handle 40 billion requests a month,and plan to be around forever!
I'm just glad that icanhazip is staying up - it sucks when useful services go offline, and negatively impact all their users.
We are already doing this with trade. The amount of leeway and free lunch China has gotten from the west is insane. I don’t blame China, I blame the west and the rest of the world for not preventing it. Asymmetrical policies are often exploited by capitalism and governments have been caught off guard.
I’m not an Anti-China lunatic. It’s just common sense.
In regards to trade war, HN has discussed this ad-nauseum, I think we should restrict the discussion to internet traffic even though I brought it up as an analogy about asymmetric response from the west in general: https://hn.algolia.com/?q=trade+war
Most of the affected people would obviously be inside China.
It very much echoes the problems of intellectual property theft in China.
It's not really a problem anyway. If some capitalists in the US and Europe don't get to skim off a slice of profit from another country's manufacturing output, then so what?
No it’s not. What is this even supposed to mean?
If this trust is repeatedly broken, peering networks may be forced to depeer the AS as a result, like what happened to McColo when they were depeered.
Blocking the entire country will do little to hurt the government (who can employ state resources to get whatever information they want) and do quite a bit more to harm the Chinese people by reducing whatever level of information independence they still have.
If there is going to be significant change in China, it will have to come from the Chinese people. Cutting them off from the Internet vindictively does not advance that goal.
There are specific people in China doing specific bad things with specific computing resources. It would be far better for the U.S. government to dedicate more resources to finding and partnering with orgs and projects (like icanhazip or Cloudflare) to find the info they need to apply targeted mitigations.
“China does it, so we should do it too” only makes sense as a strategy if our goal is to become exactly like China is today. I don’t think that should be our goal.
I very strongly disagree. An eye for an eye is exactly what needs to be done and should have been done from the beginning. Unfortunately, it is too late. 1989 massacre should have been condemned more solidly and trade restrictions should have been placed in the 90's. The bet that western alliances made is that China would open up in the 2000s leading into 2010s. That has gone horribly wrong.
The west is finally waking up: https://en.wikipedia.org/wiki/Inter-Parliamentary_Alliance_o...
This sounds like a couple of people I've met, who have a philosophy of "treating people the way they treat me". And what if the other person/side also "believes in reciprocity", what happens then? This seems to rely on other people being nice first, and then always treating them how they treated you, imitating their behaviour, like Tit-for-tat[0]—except Tit-for-tat begins by being nice. It's not easy to put my finger on what seems fishy about that strategy, but it doesn't at all seem the easy solution to being fair and just (or whatever word you most prefer here) its proponents seem to think it is.
[0] https://en.wikipedia.org/wiki/Tit_for_tat (See particularly "Problems" and the next section)
Why isnt 82% of the worlds's people global?
As long as the truth doesn't match what the preferred narrative is we'll continue to suffer the consequences, which is true of so many things beyond just attitudes towards China.
That would be one's own culture, whichever one that is.
It's not racist in China to say the truth, why is it where you are ? You probably live in an oppressive political regime with a biparty dictating what you can think ? :P
mmm, I have had a number of Chinese people explain to me the “truth” about people of various races over the years
I sure hope that "just medicine" extends to Po Chai pills. So welcome for treating diarrhea symptoms (despite my initial skepticism) when loperamide wasn't available while I was traveling in China!
Which might not be a bad thing overall, but it's sure not gonna make any transnational corporation's bottom line happy over the next few quarters, so they'll be waving a lot of money at politicians to make this not happen.
The benefits of gobalization and the spread of democracy (or even just alternative governance models) via exposure to other cultures cannot be understated
Even if just one port is left open, people will be curious enough to find it and use it. Chinese people are humans too :D
Maybe that's an acceptable price to pay for not being the ones to take the next step toward war. If war is a game in which "the only winning move is not to play", then maybe it's also true that when it comes to doing the peaceful thing, the only winning move is to keep on playing, even if it hurts us.
When do we accept the fact that Xi's ambitions extend far beyond the borders of Mainland China and pose a threat to the very idea of human dignity?
Is it when he invades Taiwan? Floods the Uyghur camps with gas? Bombs Japan? Lands an army at the port of Darwin?
Are you sure you're doing your part of the bilateral exchange? It cant just be China changing, the US must learn too to adapt and accept a larger, more powerful country, with a widely different model.
Living in China, I can tell you the american model is known, and not particularly impressive to them. They care a lot less about freedom of speech, maybe because they never had it, than they care qbout order, unity and crime rate for instance. And what I always hear is that throwing themselves at the communists in revolt to get the same shitty system as the US is not so seductive.
Maybe become a role model and people will beg to ressemble you ? I have a hard time convincing them voting for their government is gonna work better because "if even idiots can vote, look at who they elect" :s
It's probably totally an option if you want to work for one of those corporations, too.
Also, I'm generally bothered by comments like this one that stir up the general human tendency toward xenophobia. We should be fighting that tendency within ourselves, not fighting the out group. Whichever group of people we want to demonize, we should remember that they're people just like us. We shouldn't punish the majority of them for what a minority are doing to us.
I agree you don’t want to cut them off, but on the other hand, I don’t want 90% of all global malicious traffic to originate from a specific country.
Is that actually true? I guess I'm inclined to believe that claims like that are more likely to be propaganda from western governments and/or western-owned companies.
If it is true, I wonder why their government isn't stopping it. They must realize that it's giving them a bad reputation in the wider world.
[1] https://en.m.wikipedia.org/wiki/Great_Cannon
If you run a small-medium sized business in the US blocking all of countries you can't do business with anyway will save a ton of trouble.
My reasoning for these kinds of stats is usually: Of course it makes sense to attack targets in jurisdictions which can't catch you or equally hide in a country which won't extradite you. (But I never looked into it to any depth, so it's baseless reasoning.)
Sigh.
Most Chinese internet users would not miss Western internet for a second, a fact you would be aware of if you actually had any insight into Chinese culture.
This attitude that you cannot give consequences to abuse because THINK OF THE POOR CHINESE is so utterly laughable.
Thanks for the blind assumption. I'm Chinese myself.
Most countries cooperate internationally in getting bad actors from hackers over pirates to pedos booted off the Internet and into jail.
The exceptions are China and Russia who won't do anything against any bad actor and India which is a big base for phone scams (as is Turkey for the European Union, but even Erdogan's regime is cooperating with EU police in taking down scammers).
I agree, the line between demanding at least some sort of common decency standards and xenophobia is thin in these days, but we have to get everyone on board to protect everyone else from rampant abuse.
That network guy (classic long hair "security" guy) was a lazy asshole for doing it then and the internet needs to have the technology to deal with bad actors beyond AS/geo-level blocking now.
I dropped my jaw when I read it was getting 30B req/day.
Thank you for running this site for so long, and thank you for keeping it up for free, and deciding to not monetize it.
I'm somewhat curious what fixed things, as I've not had to use neverssl.com for some time.
I was reading from neverssl maintainer that they get a _lot_ of traffic, questionable ones more than it is not. Its DNS runs on AWS IIRC, and we all know Route53 isn't the cheapest.
It's also a good reason for CI providers to mirror package repositories.
Sure you could attempt to generate a bunch of keys and cycle them but it would be easier to just cache your gems.
I'm curious as to what other overused utilities can be trivially done with pure server configs.
``` <!--#echo var="REMOTE_ADDR" --> ```
Small feature request: back in the day {ipv4,ipv6}.{icanhazptr,icanhazip}.com only had A / AAAA records, but now it seems they have both and thus a simple "curl ipv4.icanhzptr.com" can also give me a v6 address (of course, "curl -4" works). Would Cloudfare be OK with separating them again?
Cool, I didn't know about that one.
Glad it will live on!
There are a plethora of unauthenticated STUN servers around, and while there's still room for abuse, the protocol is a bit more lightweight than full-blown http requests, and faster, too!
I've dabbled with doing this on my own, but I've found `myip` to do the job nicely and without hassle:
https://github.com/Snawoot/myip
http://manpages.ubuntu.com/manpages/bionic/man1/stun.1.html
Also i feel little bad you didn't get any money out of it whether the site was designed to make money or not. It would have been a wonderful end to the story if you got something back for all the years of hardwork you put into running it. You do have my appreciation if that means anything though.
P.S. this story is very similar to rawgit which was a wonderful site but also fell prey to malware aholes.
Most likely it got them a much higher paying job than they would have otherwise gotten. Walking in and saying you single handedly run a site with billions of requests per day and petabytes of traffic will get you noticed.
Being able to get in front of a hiring manager who is offering a high paying job and convincing them to hire you is the only thing. And having a very popular website with a ton of traffic is more likely to get you in front of a hiring manager with a good job than actually being good.
Rackspace:
- Principal Architect, OpenStack - Cloud Architect, Cloud Servers
- Chief Security Architect, Director
I'd love to set one up like that.
Since I have mostly Windows Experience, it would probably need to be a .chm file :)
That cert is no joke.
RHCA(dminstrator) is a great cert to get (not super difficult but not super easy, but like all RH certs you need to actually know the material).
RHCE is awesome to have, and a good bit harder than RHCAdministrator. But when I grow up I want RHCArchitect...
I hate certs, but if you want in on certain projects, it doesn't hurt to have the desired certs, especially if you can get your employer to foot the bill.
Anyways, knowing him personally, I can attest he's a really good dev and wholesome person. Sad to see this site get bogged down with malicious activity, because like downforeveryoneorjustme, I kind of have it committed to muscle memory. Hope CloudFlare can keep it going strong!
Took me 5 minutes of work and exactly one line of code.
It's time to put this service to the first place.
This is absolutely mind blowing.
[1]: <https://news.ycombinator.com/item?id=26634476>
dig +short @resolver2.opendns.com myip.opendns.com