HN Explorer

Ask HN: How can a website copy mine in real time?

14 points by francescochi ↗ HN
Hey everyone,

I am literally going insane since yesterday night as my website is being cloned in real time and I cannot explain how that is possible.

I use NextJS and host it on Vercel.

As soon as I deploy a change it gets copied instantly, even the API routes.

I just added a check to see if the referrer is allowed and, since they copy everything, their requests return a 401.

Mine is a .com and theirs is a .xyz

Does anyone have any idea?

Thank you

20 comments

[ 3.1 ms ] story [ 64.9 ms ] thread
cranberryturkey
its easy. is your site accessible on the internet? it can be copied in realtime.
calculito
ok, it's easy for you to understand. For some other people, me for example, not that easy. The question is also, how is is possible. So, I don't expect a detailed explanation or a step-by-step howTo but do you have any hints?
cranberryturkey
its called 'web scraping' look it up. You can send a cease and desist or block their ips.
jrvieira
Server side code is not publicly accessible, that's not what webscraping means, and nothing is being copied in this case, it's being proxied.
ahazred8ta
Other commenters have suggested that the xyz site is not using telepathy to instantly update itself whenever you make a change, but rather they have no local copy of your site at all, and every time you load an xyz page, it fetches the contents directly from your site. Some are assuming that the foo.xyz DNS is pointing at your own server, but you say you've checked for that.
sim7c00
not sure about nextjs / vercel, but don't you have access logs for your webserver which you can check to see who's doing the cloning? Perhaps you can find some user-agent or certain ip-range or something is responsible, and block that.

there are tons of automated systems copying everything everywhere unfortunately. a lot are fairly ok to block.

There's also services you can hire or hosting providers who do some filtering for you, but those are generally more expensive than basic ones. (added security tooling around their platform)

petercooper
Might they just be reverse proxying it? Make a request for an unusual URL and see if it appears in your logs.
francescochi
Yes, that's exactly what they are doing.

How can I stop this?

petercooper
One naive approach would be to look at what IP address(es) they're using when proxying and blocking them. (Or detecting them in code and redirecting them to your domain, perhaps.)
roosgit
I had a similar thing happen to one of my websites. In Varnish I used something like this:

if (req.http.host ~ "^(?i)(example.com|www.example.com)") { #redirect to https } else { return(synth(403, "Not allowed.")); }

It basically checks if the host is my domain. I don’t know know what the equivalent of `req.http.host` is on the web server you use. This "solution" might run into issues with Google Translate, but I’m not sure.

solardev
In Next.js you can use middleware to sniff the domain and do something (block or redirect to your own): https://nextjs.org/docs/app/building-your-application/routin...

On the frontend you can also use Javascript to do the same thing and just redirect their traffic to your own (or block them).

Make sure to specify canonical tags etc too so Google doesn't think those duped domains are yours.

If you have any API routes, make sure you set up CORS and CSPs and such to block requests from unauthorized domains.

(Edit: You can probably also just throw Cloudflare in front of the whole setup: https://vercel.com/docs/integrations/external-platforms/clou...)

nicbou
Configure which domains your server accepts requests from. Only serve requests coming from the domain that's actually yours.
mlhpdx
Put CORS in place and a strict CSP.

Look the Host header in your API backend and block it. It might be forged, but probably not.

Identify their host (reverse IP lookup) and block requests from that ASN. If they host in same place as you, complain and get them shut down (or move).

The obfuscated JS based domain checking others suggested.

None of these is perfect, but they don’t have to be. Just keep adding layers until they go away for good.

gary_0
If your site is only meant for human end users, you can get lists of datacenter IPs and use that to block them (assuming they are running in a datacenter that's on the list). That tends to block all sorts of abuse.

If you are worried about blocking legit users with this method, you can show a CAPTCHA instead on every request from a suspect IP.

At any rate, get their IP from your logs and run a whois on it to get their ISP and location. You could also try emailing the ISP's abuse contact or filing a DMCA complaint with their hosting provider, although of course that might not work.

beardyw
Check the domain and if it's not yours redirect it. If they copy that you are done.
ActorNightly
Just add obfuscated java script code that checks the host and errors out if its not your domain. Can still be bypassed, just will take some reverse engineering.

Ive added JSFuck code for this specifically on a few occasions

curtisblaine
Insert some heavily obfuscated javascript code that checks the domain at runtime and, if it's the wrong one, redirects to distasteful images. If the site is in China, redirecting to prohibited content is another fun thing to do.
RecycledEle
My guess is that your web hosting provider sold you out.

DirectNIC did it to me.

You can test this by adding a file like inxed.htm (note the deliberate misspelling) in your web root folder without any links to it anywhere and seeing if you can hit it in their web server.