HN Explorer

Ask HN: Dealing with a competitor who is scraping my content and ranking higher

224 points by rakjosh ↗ HN
One of my competitors has been scrapping my site and providing service to their users without paying anything. And now they surpassed me on google ranking.

I've created a website which gets around 5K unique hits every day. It's a free service for users but I've to pay a monthly fee to a third party service provider.

Because my site is free for users and doesn't require users to register it's been very hard to keep up with this guy. If I change certain things, they counter it immediately and make it work. And they use several proxies to send the request, it's virtually impossible to block based on IP.

Please suggest, if there is anything I'm missing that can be done.

129 comments

[ 2.7 ms ] story [ 213 ms ] thread
maxpupmax
ideas:

1. Create content that can't be scraped. I'm not sure exactly what your "content" is in this case, but images can be watermarked, text can be given lots of references to your own brand and service, ect.

2. Submit legal requests to google to remove the content. Enough violations can get their domain blacklisted. I've done this successfully in the past for competitors using my trademark without permission to get it removed from Google. Ads. https://support.google.com/legal/answer/3110420

3. Talk to the hosting provider, if applicable. If someone is repeatedly breaking copyright law using their platform, they may have some incentive to stop providing hosting.

babaganoosh89
Can you implement a captcha somewhere? Or figure a way to obfuscate your html if that’s what he’s scraping. Something that can scramble the css class names each build.
notyourwork
It really depends on how complex/simple the page is. If its small site with little context you can usually traverse from parent body ignoring classes/ids. However, for more complex sites changing class/id values on each build can certainly help make this a lot more challenging.

Ideally changing the children counts to prevent things like $(body).children().children()[3].html().

PopeDotNinja
Can you figure out how the person is scraping your site? If so, start putting in content that only shows up for that scraper, such as...

"this content was unabashedly stolen from <your site URL>"

...or maybe...

"my favorite movie is Mac & Me, check it out! https://youtu.be/vNjACYfQlbI"

kazinator
"this content was unabashedly stolen from <your site URL> *by <IP address of scraper> on <date time>".

Unless the content must be 100% static, you can embed the caller's IP address in it.

Eridrus
Google ReCaptcha is a pretty low friction way for (most) users to get your content.

You can do this while serving your real content to Google.

SyneRyder
The spammers & scrapers seem to have found their way around ReCaptcha. I made a simple spam filter for my website comment form that logs every submission, and a lot of the spam being sent (maybe 20%?) has a valid Google Recaptcha. I'm thinking of removing Recaptcha because simple keyword based filters have been more effective than ReCaptcha has been.
Jaruzel
ReCaptcha and the original two-word based one are just Google using your visitors as 'mechanical turks' - by solving ReCaptcha et al, they are actually helping Google apply identification tags to images in streetview, and previously the Google books collection.

An afternoons coding would get you an independent self-hosted Captcha system that would be just as effective (if not more so, as it's proprietary) and probably less annoying to your visitors.

orf
> and probably less annoying to your visitors.

As someone who has used both, absolutely 100% no in every way.

Google's captcha is so superior to what you would write, please don't tell people not to.

Jaruzel
How so ? It's really not that hard. I've done it, and it zeroed spam coming through a web form almost 100%.
danielbarla
Both (custom and recaptcha) are essentially security by obscurity, in the sense that a skilled attacker will eventually bypass it. The problem is not that difficult. The advantage Google has is being able to move the goalposts, potentially on a daily basis, while you presumably have better things to do. On the other hand, they are a juicier target, so will draw more effort. Ymmv.

Google tends to strike a good balance between convenience and irritation.

ericpauley
So would disabling submissions. The real question is how many legitimate submissions weren't made successfully because of a bad captcha.
mockingbirdy
I've build a ReCaptcha breaker in an evening (which beats Google 100% of the time). The only reason why most spammers don't evade ReCaptcha is because they make enough money through other means (and most spammers can't program).

There are also a lot of services where you can simply send your challenge token and some guy will solve it for a fraction of a penny.

As soon as other money-making opportunities will dry up, spammers will evade ReCaptcha and spam will increase regardless of your security mechanisms.

senorjazz
this won't work in this case. The site is being specifically targetted. A custom captcha will be trivial to bypass. The reason it worked for you would be your site was just bulk scraped and no patterns could be found in your captcha to match it to a solving solution so the spammer moved on to the next.

When your site is specifically targetted, captchas will not work

exikyut
They're probably using https://2captcha.com/

They pay 50c per 2 hours to the workers, and API access costs 50c-$1 per 2000 solved captchas.

I think most of the people doing this are in India, where US 50c for a couple hours translates well enough. (Completely naive about the truth of the situation though.)

SyneRyder
One of the regular spam messages mentions the software they're using ("You read this, so you know it works!"), so it's pretty clear that ReCaptcha has already been defeated. I'd mention the software, but I'd rather not give them publicity.

Others have been able to beat reCaptcha too, though:

https://www.youtube.com/watch?v=fsF7enQY8uI

senorjazz
didn't watch, but the Russian xrumer forum spammer has been at the forefront of captcha solving for the last decade or more.
dingaling
ReCaptcha is very high friction for many users, particularly those not logged into Google services

For example I have never reached the end of their 'identify the road signs' challenge and if I encounter a site that uses them I'll just close the tab. Even Google Search.

Much better to use rotating series of questions related to your domain of data.

is_true
If you can detect them try to serve them wrong data
chrischen
Yea I was thinking a "hellban" type tactic might wreak more havoc. Just subtle enough to de-rank them.
slededit
This is pretty much what the DMCA and other copyright mechanisms were made for. Send your request to Google and have them delisted.

https://support.google.com/legal/answer/3110420?hl=en

rovr138
He can also go after them with this. He can write them to take it down. If they don’t, go to the host and have them take the site down.
acct1771
"Excuse me, could you put down my TV?"
twothamendment
Asking nice can work, but I find it depends on what is in my hand at the time.
exikyut
This is an excellent idea, especially because the system is automated and allows anything and everything through.

Just submit whatever politically incorrect thing you like in with a bunch of P2P links and watch the dissent disappear :)

Instructions and discussion here: https://news.ycombinator.com/item?id=17787302

winkeltripel
Yes, I would combine this with a couple of fake entries, and when those show up on their site, you file DMCA claims with google, and with their hosting provider. An american company has no choice but to shut their site down under threat of DMCA claims. If the site gets back up, add another fake entry, and do it again when they scrape you, until they give up.
JonoBB
I had my entire marketing site copied, word-for-word. I still have no idea why (other than perhaps to damage google ranking due to duplicate content?).

I reported it to google, and they removed it from search results in a few days. I also reported it to the site's host, who took it down completely.

ddgflorida
Almost the same here - they copied my whole website, improved the user interface, and added a little content. Luckily I still outrank them :)
graeme
Does anything stop the scraper from using DCMA on the original site?
jasonjayr
They'd lose their source of data
jordan801
Anything receiving 5k visits a month, is likely crawled at least once a day unless it's told not to. In this case, Google could check crawls and determine the original source of content between the two competitors.
jklein11
If the source that is copying is also being crawled once per day isn't possible that it would be crawled first and appear to Google that it is the original source?
setr
presumably site A got popular before site B started scraping it; else there wouldn't be much point in scraping A. There is ofc the scenario was A was a generally unknown resource, and B realized it had the potential to be valuable if you just SEO'd it right.. but there's still probably evidence somewhere that A cropped up first (how did B stumble into it?)
oo0shiny
I would think that the scraper wouldn't want the source of their material to disappear, for one.
laurieg
If you are able to detect them, serve them garbage data instead of blocking them. Mix in some false entries etc. If you completely block them they will work hard to work around the block, but if you give them some bad data randomly it will take them longer to notice.

Also, you can use the fake data as evidence they scraped you.

coding123
And also to try to obtain a pattern of what request headers are common... Try to create a special hash from the request headers. Then find a way to infiltrate the data with it so you can identify them better
acct1771
It's not nearly as valuable as cryptographically provable evidence of the pages having identical content as yours, with yours being of an early publication date.
rakjosh
That is what I did initially. My API was sending garbage data Saying this website is shutting down, please visit [mysite] for more info :D. But later on, they figured, I was sending garbage data and people were visiting my site because of that. I even put in a tracked link to see how many visitors I'd get from there site and I was getting around 500 people every day.

It was like a cat and mice game for a few weeks.

hkai
I thought Cloudflare's "under attack" mode can prevent scraping bots. Correct me if I'm wrong.
ishanjain28
It can be bypassed and they don't make it very difficult to bypass it.
[deleted]
(comment deleted)
lubujackson
Write a few things. Register them with the copyright office. Then post them and wait for them to copy it. Now you can sue them for the max and strong-arm them to take it all down or face obvious legal fees and punishments over 100k.
setr
Can you sue for monetary damages if you yourself were not making any money off the content? If anything the other site could be called a mirror, and saving our op money on the hosting cost by diverting traffic.. (although maybe not from scraping?)
pluma
I'm not sure about how this works in the US but in my country (Germany) "commercial use" does not actually require money to change hands or operating at a net profit. And last I checked copyright laws (if they apply to the content) apply to everything you create, whether you monetize it or not.

That HN comment of yours? If it's sufficiently non-trivial, it's protected by copyright. Even though neither you nor HN directly make money of it. If someone scrapes HN and republishes that comment, you could go after them and demand they delete it.

nutjob2
You can sue for the loss of value of your website, even if it doesn't make an income it remains an asset with monetary value.
shiado
Find a way to contact the owner of the site and bait them with something like an acquisition offer, you might be able to use the email in the whois data if there is one. The point is to get the owner to click a link you provide, behind this link you want to log IP/Headers, maybe do some JS fingerprinting/etc... It is unlikely they will open this link over VPN/Tor. If you feel like it check your server logs to see if they have accessed your site from that IP and when. Then send a C&D containing all this information and tell them that you have legally contacted their ISP for their subscriber information and not only will they be sued for violating DMCA for possessing your data but that they will also get hit with the CFAA for circumventing protections against access and that they will enjoy a five year sentence in a federal pound me in the ass prison.
hluska
That's well thought out!
staticautomatic
Please don't invoke the CFAA for scraping.
vntok
Why not? Scraping seems like exactly what the law is aiming to prevent, in spirit and in letter.
sp332
The scraper is 100% authorized to access the data. It's the rehosting that is the problem.
robin_reala
I don’t disagree with the content of your post and I know this is a US cultural trope at this point, but trivialising rape benefits precisely no-one.
yakshaving_jgt
"Federal pound me in the ass prison" is a quote from the movie Office Space.

If you already knew this, then I guess the applicable movie quote this time is "Lighten up, Francis" from Stripes.

ninjakeyboard
Are you using server side rendering? There are ways to make it harder. But ultimately you don't want to sacrifice the user experience. Why don't you require registration provided by a third party like facebook or google - it's one click sign on and you can make your access requirements extremely low.
wolco
Loads of great suggestions. The DMCA request is the tool you want. That will cause a downrank if you keep doing it. Problem solved.
SyneRyder
If you haven't already, try adding some "trap streets" to your data. Map makers occasionally include streets that don't exist, so if a competitors map includes it too, it's clear that the competitor copied it:

https://en.wikipedia.org/wiki/Trap_street

I did that with an online marketing dictionary I wrote years ago, some of the definitions included strange usage examples that contained the names of several of my friends. When a competitor scraped us, instead of shutting them down, the boss negotiated a data licensing arrangement with the scraper instead, so we ended up getting a revenue stream & backlinks out of the incident.

If that fails, and talking to them directly doesn't work, then the DMCA is often effective. I've made DMCA requests against websites that distributed cracks of my software & they often disappeared in a couple of days.

partycoder
Similar to trap streets are "phantom settlements", aka "paper towns", which are fake towns rather than streets.

Now, this idea is not limited to maps: Google used trap search results to catch Microsoft using Internet Explorer to scrap Google search results: https://googleblog.blogspot.com/2011/02/microsofts-bing-uses...

[deleted]
(comment deleted)
m0nty
I was told all these phenomena (on maps) are referred to as "cartographers' follies", so the other names are interesting to hear.

In the "real world", it was commonplace for compilers of mailing lists to include several "phantom names" in their lists. If those names received a mail, the list-holder would send an invoice to the person who sent it. Simple, elegant, very difficult to bypass way to protect your knowledge-based business.

stef25
This is the same as "poison pills" right? A directory website would include some fake entries so that if they pop up on other sites they could have only come from copying the original directory.

A scraper could get around that by x-referencing the data with another source, if possible.

S-E-P
Though that creates quite a bit of extra overhead, and you only really have to catch them once.
codedokode
Copyright protects only creative works, not any information as I understand.
bena
It's a little weird. Trap streets often aren't protected by copyright as the existence or non-existence of something is a fact and facts aren't copyrightable.

And maps are nothing more than a collection of facts. As are recipes. You can take the recipes in a cookbook, write them down, and disseminate them how you choose and no one could say boo. Which is why Coca-Cola guards its recipe carefully. Once it is known, they can't do anything about it. You can't copyright "a lot of sugar, carbonated water, caramel coloring, mix".

But you also can't just photocopy a map and sell it as your own. Presentation falls under copyright. The look of the map is a distinct piece of art that has copyright protections. The combination of colors, fonts, line thickness, placement of labels, etc, are all things you can't reproduce.

But the information that Bob Street crosses Bill and Jill streets at these points, that's just facts.

bausshf
Include a base64 encoded image as a street name.
jlengrand
Something I haven't seen mentioned yet, but did you try to simple get in contact with them to discuss the issue?

Since your website is free, it might be worth combining forces to serve your users better. I know it's hard to swallow, but in the hand what matters is that what you do is useful to people isn't it? And if your users are moving away it probably is because `some` of the things they do is right?

ivolimmen
Usually scrapping means it will go though all links you provide. If you add non visible links on your site that link to rubbish content he will automatically add them as wel.
[deleted]
(comment deleted)
Witesse
When you say "my content", are you referring to content that you created, like a blog or online resource or are you referring to content your users created? Also is the information they're copying factual in nature (like a map or results of a calculation)?

If it's factual in nature: It's not protected by copyright.

If it's content your users created: You are not the copyright holder and cannot file a DMCA takedown notice. Your best bet is really going to be making yourself unscrapable.

If it's content you created: File a DMCA takedown notice.

Doctor_Fegg
> If it's factual in nature: It's not protected by copyright

Compilations of facts can however be protected in the EU under the Database Directive.

Also, Terms of Use etc.

amelius
> If it's content your users created: You are not the copyright holder and cannot file a DMCA takedown notice.

But you can still have copyright on some of your data. If the other company simply copies everything, you can take legal action.

rajadigopula
Google is smart enough to penalize plagiarizing sites. If the content appeared on your site first, it is highly unlikely they get ranked higher. Work on your own SEO strategy.
pacuna
I was thinking the same thing. I believe with the latest changes in the algorithm this should be even harder.
smilesnd
Learn to accept it. Only way to stop scrapers is to make it a financial burden for them to scrape your website. There are to many ip address, proxy, vpn, and botnets for you to try and block them all. You can write code to try and tell legit traffic from scrapers traffic but then they will just figure out how to bypass that through trial and error. You can try to take legal action, but that will cost you money up front and might take longer then you think. You can try to get there service terminated, but most host won't do it without a court order. You can make you html/css hell to read randomize autogen all your class and id tags. Put all you elements in random order so the people writing the scraping code don't have a default template they can update in minutes. Use javascript to actually send the data which then you use to identify people and have a call home function so they cannot hide behind proxies and vpn and such.

These are all just things that will slow them down, but won't stop them. Make it cost them more money then they make using your data will be the only way to truly stop them.