Ask HN: Someone is proxy-mirroring my website, can I do anything?
I'm trying to deal with a very interesting (to me) case. Someone is proxy-mirroring all content of my website under a different domain name.
- Original: https://www.saashub.com
- Abuser/Proxy-mirror: https://sukuns.us.to
My ideas of resolution:
1) Block them by IP - That doesn't work as they are rotating the IP from which the request is coming.
2) Block them by User Agent - They are duplicating the user-agent of the person making the request to sukuns.us.to
3) Add some JavaScript to redirect to the original domain-name - They are stripping all JS.
4) Use absolute URLs everywhere - they are rewriting everything www.saashub.com to their domain name.
i.e. I'm out of ideas. Any suggestions would be highly appreciated.
p.s. what is more, Bing is indexing all of SaaSHub's content under sukuns.us.to ¯\_(ツ)_/¯. I've reported a copyright infringement, but I have a feeling that it could take ages to get resolved.
309 comments
[ 46.6 ms ] story [ 7139 ms ] threadhttps://blog.cloudflare.com/introducing-scrapeshield-discove...
EDIT: or building up on what user zhouyisu says above you can generate your perfect match IP blacklist by calling urls via the abusing site that automatically puts any caller into the blacklist.
https://developers.cloudflare.com/fundamentals/get-started/c...
[1] https://github.com/brianhama/bad-asn-list
BTW: if they're removing/replacing domain name of your site, try obscuring it with HTML entities. This may dodge simple find'n'replace.
Now let's say that your website will show the ip of whoever visit it, in one of textbox. When you access it it shows your ip. When the proxy sever access it it shows the proxy sever's ip. When you access the proxy site, the proxy site will access your site, having their ip on one of the text box, then return the page with their ip to you.
The more advanced method is to encrypt the ip and put it hidden somewhere, on later for you to decrypt it, get the ip and black list them.
Bots will trigger it by walking through all pages, but real human would not click in since the paging is non-sense and titles are non-sense.
https://webmasters.stackexchange.com/questions/56326/canonic...
I noticed that the other domain is hotlinking your images. So you can disable image hotlinking, by only allowing certain domains as the referers. If you block hotlinked images then the other domain will not look as good. Remember to do it for SVGs too.
https://ubiq.co/tech-blog/prevent-image-hotlinking-nginx/
Finally I also see they are using a CDN called Statically to host some assets off your domain. You can block their scrapers by user agent listed here:
http://statically.io/docs/whitelisting-statically/
2. Create fake html elements and put unique strings inside. And you can search that string in search engines for finding similar fake sites on different domains.
3. Create fake html element and put all request details in encrypted format. Visit adversary's website and look for that element and flag that ip OR flag the headers.
4. Buy proxy databases, and when any user requests your webpage, check if its a proxy.
5. Instead of banning them, return fake content (fake titles and fake images etc) if proxy is detected OR the ip is flagged.
6. Don't ban the flagged ip's. She/He's gonna find another one. Make them angry and their user's angry so they give up on you.
7. Maybe write some bad words to the user on random places in the HTML when you detect flagged ip's :D So the user's will leave the site and this will reduce the SEO point of the adversary. Will be downranked.
8. Enable image hotlinking protection. Increase the cost of proxying for them.
9. Use @document CSS to hide the stuff when the URL is different.
10. Send abuse mail request to the hosting site.
11. Send abuse mail request to the domain provider.
12. Look for the flagged IPs and try to find the proxy provider. If you find, send mail to them too.
Edit: More ideas sparkled in my mind when I was in toilet:
1. Create fake big css files (10MB etc). And repeatedly download that from the adversary's website. This should cost them too much money on proxies.
2. When you detect proxy, return too big fake HTML files (10GB) etc. That could crash their server if they load the HTML into the memory when parsing.
Reminds me of a time some real estate website hotlinked a ton of images from my website. After I asked them to stop and they ignored me I added an nginx rewrite rule to send them a bunch of pictures of houses that were on fire.
For some reason they stopped using my website as their image host after that.
I'm curious if they are stealing anything else, e.g. are they selling ads/tracking, do they replace order forms with their own...
Also to teach them an important lesson about the internet.
there was another time a site hotlinked to a js file. After asking them to stop, i found that they had a contact form with a homebrew captcha which created the letters image like http://evilsite.com/cgi-bin/captcha.jpg?q=ansr
A little while later, their captcha form had a hidden input appended with the correct answer value, and the word to solve was changed to a new 4 letter word from a dictionary of interesting 4 letter words. The form still worked because of the hidden input. I might have changed the name on the "real" input also.
Additionally if they decide to blackhole the fake/honeypot url, since you mentioned they pass along the user agent, you could mixin some token in a randomized user agent string that your scraper uses so that you could duck-type the request on your end to signal when to capture the egress ip.
[0]: https://caniuse.com/mdn-css_at-rules_document
A little soft NSFW would also greatly accelerate them being added to a block list, especially if you were to submit their site to the blocklists as soon as you started including it. You can include literally anything that won't get you arrested. Terrorist manifestos, the anarchists cookbook, insane hentai porn... Use all those block categories - gore/extreme, terrorist, adult, etc.
SRI is for the situation where a CDN has been poisoned, not this.
A. Blank page
B. Let the find and replace update the CSS. Generate new hashes in the HTML.
C. Find someone new to pick on.
B is time and potentially computationally expensive, so it makes C a better option.
> 6. Don't ban the flagged ip's. She/He's gonna find another one. Make them angry and their user's angry so they give up on you.
There's a popular blog that no longer gets linked on HN.
The author didn't like the discussions HN had around his writing, so any visitors with HN as the referer are shown goatse, a notorious upsetting image, instead of the blog content.
Fortunately, there are many upsetting images for the OP to choose from!
One bad example is relying on the referer only for log-out, everything else works. That site also runs massive js on log-out, as if it really needs to rely on explicit log-out, and not just the user disappearing.
> Create fake big css files (10MB etc). And repeatedly download that from the adversary's website. This should cost them too much money on proxies.
Be careful when doing things like this, including the shock image option mentioned in other comments, as then it could become an arsehole race with them trying to DoS your site in retribution. Then again, going through more official channels could also get the same reaction, so…
> When you detect proxy, return too big fake HTML files (10GB) etc. That could crash their server if they load the HTML into the memory when parsing.
Make sure you are setup to always compress outgoing content, so that you can send GBs of mostly single-token content with MBs of bandwidth.
Then, write a little script that repeatedly hits that honeypot URL. I quite like this idea.
You can also write some obfuscated inline JavaScript that checks the current hostname and compares to the expected one and redirects when not aligned.
As a side note, their domain is linked in this thread so they are seeing HN in their access logs and probably reading this. It should make for an interesting arms race. Or red/blue team event.
For example, I had an app developer start stealing API content, so once I determined points to key from them, instead of blocking them I simply randomized the API content details returned to their user's apps.
Hey, API calls look good, the app looks like it is working, no problem right? Well, the users of the app were pissed and the negative reviews rolled in. It was glorious.
Doesn't that also cost you an equal amount? You'll be serving them an equal amount that they proxy to the end user.
It's not even necessarily a cost for them; you're assuming that the host is owned and paid for by the abuser. If it's simply been hijacked (quite possible), you're just racking up costs for another victim.
Not sure how you actually do it and if it serves your purpose but sounded neat.
[1] https://www.youtube.com/watch?v=jnDk8BcqoR0
Nope, since anybody doing this and it has at least minimum intelligence are using residential botnets as proxies.
Then instead of blocking the fingerprint, poison the data. Introduce errors that are hard to detect. Maybe corrupt the URLs, or use the incorrect description or category. Be creative, but make it kind of shit.
It's easy to work around blocks. Working around poisoned data is much harder.
i.e. ssl_cipher, http_x_requested_with, http_accept... and the order of all headers supplied... the casing of all headers supplied... TLS client HELO.
It is relatively easy, if you have enough signals, to essentially create a fingerprint that they won't understand how it works. Yet it will be effective at blocking it regardless of the IP.
Once you add enough of these together it will be hard for them to get around it without being obvious as they do so.
Super aggressive... those same fingerprints will reveal legit browser traffic and the fingerprints for things like Google-bot... so you could go towards a whitelist rather than blocklist. But this is a place you'd have to actively manage as new variations arise constantly.
However, then you're left with the really hard problem of when real browsers are used. But hey, you went a long way before you had to actually look at traffic patterns and in the meantime you've significantly raised the costs for those operating the bots.
It's also worth noting that if you really get enough signals, that bot writers cannot control them all. Everyone can rewrite a HTTP header, but can you pick the right HTTP headers in the right order with the right TLS cipher and TLS HELO to appear to be the same as Chrome on Windows? Good luck.
If you change subtle details about spelling, spacing, formatting, etc by the source IP, then you can look at one of their pages and figure out which IP it was scraped from.
Then, just add goatse to all pages requested by that IP. Alternatively, replace every other sentence with GPT-generated nonsense.
EDIT: it should be quite easy to use JS to fingerprint the scraper. The downside is that you will also block all NoScript users.
1) Add a watermark to your images when they proxy to you.
Stolen image from {url}
2) Add a js script when the url differs from yours and display a message + redirect
By the way, I've also reported the abuser as a phishing/fraud website through https://safebrowsing.google.com/safebrowsing/report_phish/?u...
> 4) Use absolute URLs everywhere - they are rewriting everything www.saashub.com to their domain name.
If the TLS ciphers the client proposes for negotiation doesn’t align with the client’s User-Agent they get a CAPTCHA.
I would suspect that whoever is doing this proxy-mirroring isn’t smart enough to ensure the TLS ciphers align with the User-Agent they’re passing through.
Edit: you could maybe add a <meta> tag to define a CSP in but I guess they will remove it [1].
[1] https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP