Tell HN: A case of negative SEO I caught on my service and how I dealt with it
For example, about 2 years ago, something similar happened. While digging through my Search Console I discovered that Russian websites generated thousands of links pointing to a page on Next Episode with pornographic keywords used as link anchors. This was so effective that they managed to get those keywords to the top of the "Top linking text" in Google Search Console - naturally (most likely) resulting in drop in rankings for the regular keywords and the domain in general.
About a week ago, while trying to investigate the current drop in rankings and browsing through my "Latest links" external links export from Google Search Console, I noticed something funny. There were thousands of links in there (from 3 domains) following the same structure as on Next Episode: domain/show-name domain/show-name/browse domain/show-name/season-1, etc.
Following these links revealed something even funnier: all of them displayed content directly from my site! Not even scraped/cached content - they were dynamically pulling content from my server and displaying it on their domain. Even the search worked, the news archive and the top charts. Here is a list of those domains as an image: https://i.imgur.com/PjNKh0b.png. I've since blocked their access, so opening any of them will not show my website right now, but here is how it looked: https://i.imgur.com/HBiL3yh.png
Now, my first thought was that those were maybe scraping the content as part of a link farm (to spam with ads?), but I also wanted to know more. I experimented with Google searches that included pages from my website, like "Hot Shows - Next Episode" and ones with very specific news posts subjects like "Streaming Services Availability added to Episodes and Movies" (posted in September last year). Imagine my surprise when I discovered that not only the domains above were indexed by Google (and were listed in the Search results), but there were 4-5 more domains that did the same thing and some of them even outranked mine!
Here is a full list of domains that I discovered by searching for my news posts subjects: https://i.imgur.com/dAm1CzI.png. If you Google for site:domain.com you'll see some of them have thousands of pages indexed by Google. Trying out more keyword searches, I was also able to discover these domains: https://i.imgur.com/s5YjJWK.png (as they've cached the content, they still work). Those all seem to be part of the same operation, but they serve a different purpose - they have only scraped the home page of Next Episode and all their links point to inside pages on the other domains. I suspect this is to generate incoming links to the other domains and give them some credibility.
As with the links with adult keywords text anchors mentioned above - I suspect this whole thing is a negative SEO campaign - I don't see any other reason for it to be happening and it seems to be achieving its goal. Once I found all I could find about the domains involved in this, I took some action:
1) disavowed all those domains through the Google disavow tool
2) investigated if I could redirect their pages to mine (as they were dynamically pulling the content - I could change it to whatever I wanted). I managed ...
90 comments
[ 3.4 ms ] story [ 153 ms ] threadLooking at Google's search results, it's obvious that these tactics are rampant and really winning the war here.
We need a new search engine that cannot be gamed so easily. I know it's non trivial but the stakes are high as is the reward for making such.
This is a real engineering challenge. I'm excited about the problem space and opportunity.
So you mean a search engine that's 100% human curated? Or rather, a directory, it wouldn't really be a search engine.
Any algorithmic signal can be gamed. Although I'd be curious to hear how I'm wrong about that.
And Wikipedia, where power tipping super editors can control whatever they want.
Imagine if you could upvote/downvote Google search results, and got rewarded for being "right" or something...
A user-voted search engine is a good idea, I think.
Once upon a time, back in 1994, a startup used this idea as their first product. It was very popular, very useful. It was called Jerry and David's Guide to the World Wide Web.
But, as the WWW grew insanely fast in the 1990s, the human approach couldn't keep up with the algorithms. Eventually it was abandoned.
https://en.wikipedia.org/wiki/Yahoo!_Directory
Some smaller scope things can be made completely watertight, for example mathematically proven cryptography, but even that often leaks to government pressure.
https://news.ycombinator.com/item?id=26104087
Your YC application practically writes itself.
Google's algorithm is smart enough to recognise Neg SEO attacks. Sure five years ago you could buy a blast of spammy links using Xrumer or GSA with some viagra anchor text and boom you're competition is gone.
From a quick glance, most of your pages have pretty thin content, and I assume it's pulling from an API, so none of it is unique. If there was one thing I would do is try to build some content on pages. A great tool to analyse and develop content that is SEO friendly is SurferSEO - highly recommend it.
I'm surprised your forum doesn't rank as well as your main site as it looks fairly active. However, I'm not sure about how PunBB does SEO wise.
This is mysterious to me. That bad links hurt your site unless someone else bought them. Google's smart and all, but?
This is not an isolated thing, negative SEO attacks happen on a daily basis, mostly intention but I bet you there are some accidental ones too.
If neg SEO still worked efficiently, then SEO's would be focusing on spamming their competition instead of investing in quality content and backlinks.
Like any case, disavow the links if you need, then move on and continue to build content Google likes with quality backlinks.
https://docs.google.com/forms/d/1Pw1KVOVRyr4a7ezj_6SHghnX1Y6...
The fact they removed it tells me they're confident enough in their algorithms nowdays, but that doesn't mean they're perfect.
I can see my own content being outranked by the scraping domains in this very case.
Just google "Streaming Services Availability added to Episodes and Movies" and you'll see.
If they can outrank me, I don't see how this will not affect me negatively by my content being considered duplicate ...
Ot it could mean something completely different.
I'm aware of the algo update that happened in December (and that it correlated very closely with my drop in rankings).
However, in my experience - even though jumps or drops in ranking are almost always triggered by such updates - there is a good reason for it to happen.
But you're right, what is happening here may've also had no effect whatsoever.
In any case - I found it because I dug up to try and what was going on and I can't explain what I found in any other way.
Even if it didn't affect ranking - does it look like negative SEO to you or do you think something else is going on here?
I think it's just an opportunistic person who saw an easy way to rank their own stuff by taking advantage of your hard work
Like you said -- they ranked higher in some cases (without doing any work besides copying you).
Only the domains that display the stolen content, with no ads or anything.
What is in it for them?
It doesn't make much sense to do it other than if you're trying to affect someone else's ranking by causing them to be penalized for duplicate content. No?
view-source:https://[redacted].com/titansgrave--the-ashes-of-valkana/sea...
unless you have a competitor you really hates you for some personal vendetta, it's exceptionally rare to see this level of effort for the purpose of negative SEO in 2021.
like I said just looks like a spammer going for low hanging fruit, throwing everything at the wall and seeing what sticks
The AD code there is mine and the only thing they added is some JavaScript code for a Russian web page counter ...
Can you please remove the link to the offending site as not to give them any link juice?
I'm not sure that is a correct assessment, just (hopefully) clarifying the OP's intent.
I still think the main problem for you is thin content. Google doesn't seem to be as strict as they are about duplicate content. But if you're taking the movie/episode description from TMDb, there are most likely thousands of other sites doing it too.
Google could have put you in a category of thin content [1], you may have got a notification in Search Console (but unlikely, I'm not a fan of the new Search Console since literally all the data in there is so vague which makes it not very helpful).
Another thing is that when I look in Ahrefs for pages with the most links [2], they are forum profiles that have been hit with spammy Xrumer/GSA link. They're referred to as profile links. They are pretty useless, but people still build them as there is literally $0 cost to do so.
If I were to offer my advice, my first step would be to build some content. First, add some content on the homepage (if you get SurferSEO it will tell how many words to write based on your competition) and then if you're willing to build a blog in (you seem to be a developer). You could write some roundup articles that are SEO focused and then link to the relevant Show/Movie, which would also help your internal linking.
If you wanted to explore building backlinks (depending on if you want to go on a whitehat or greyhat road) - you can look into paid guest posts or niche edits.
[1] https://www.vertical-leap.uk/blog/what-is-thin-content-why-d... [2] https://i.imgur.com/xZ8VEJQ.png
Good catch with the spammy forum profiles - I moderate the forum quite heavily, so there is no spam there, but I never thought to monitor inactive profiles. I'll clean them up and will see what I can do to try and prevent those from popping up in future.
Thanks
Your advice will help OP, but it's sad to see it on here.
Adding content to sites that don't need it ruins the experience. Don't build for the bot, build for humans.
SEO is a cancer on the web. A few days ago Google directed me to a recipe for a particular type of bread. I swear I had to scroll through the authors entire life story... how their grandmother handed down this recipe from her grandmothers mother, how it feels to make you own bread, how to save your sanity with bread, how best to store bread.
The recipe at the very bottom of the page could have fit neatly above the fold.
Here are two options for OP, either of which will improve his website, unlike your suggestion: https://i.imgur.com/9VlBguW.png
I distinctly remember 8 years ago dealing with negative SEO and reading the same thing everywhere while researching it. "Negative SEO used to work in the olden days of the internet, but now in the modern era Google is all over it."
I wonder if in 5 years people will be admitting that negative SEO worked 5 years ago.
This was within the last 3 years. Until then, I too believed Google were 'smart enough' to recognise such things.
Right now getting links indexed is a 15 day+ affair for some, some have no luck at all. From search results that I have been getting, it's almost like nothing makes sense anymore. Pure garbage content is at the top, or worst of all content from 3+ years ago and content freshness is considered a key ranking factor.
The thing is, I have "nofollow"-ed all links to the forum from the main site. Honestly, at this point - I don't even remember why I did it, it must've been close to 10 years ago.
Now this makes me think if I should remove those nofollows ...
Nofollow is another thing that isn't important as used to be. Sure dofollow does give you more link juice, but nofollows are still good. If you have 99% dofollow links to your site it's going to look unnatural, so a good mix of dofollow and nofollow always heps.
Would you be able to make them do the same negative SEOing but to their own site?
Fill their site with unrelated garbage and internal links with undesirable anchor text.
* unbock their IP * create content that links back to their site with the undesirable keywords * only show this content to them and not regular visitors * don't let them grab much / any legitimate content
I can either fill their content with pornographic stuff or simply redirect it to adult websites.
I think if that happens - Google will de-rank and de-list them super quickly.
I chose not to, because I'm not sure if it's not going to affect real people ...
Do you see anything wrong?
You have a very straight forward value prop. "Next episode" of some-show. I think these sort of optimized results are probably things that Google has been algorithmically adjusting for.
Looking at the Ranked 1-3 terms you dropped for, it seems you dropped some pretty big terms and even keyword terms.
You were #1 for "seal team next episode", but now you rank #3. #1 got replaced by CBS's page, which is arguably a better result.
"black clover new episode" also dropped from #1. Replaced by Wikipedia.
"the good place next episode" similar story.
I don't know what the best move is here. Algorithmic changes are really hard to combat without major changes and even then, you don't have a ton of room to wiggle with next-episode content.
Apparently, they expanded their pool of available IPs they pull data from and now they seem to be endless (so some of the scraping domains actually work now).
I'm investigating what I can do about it. I'd appreciate any advice!
You could look at the http request headers and perhaps identify the scrapper script. You could also put a javascript challenge that is required to solve before pulling more data, and disable it for Google and Bing ips, so it's more work for them to pull data for some time.
Instead of simply blocking, you could detect them and do some kind of http slowloris response.
I'll try and find out and also I'll have to learn exactly what "slowloris" is. It may be helpful indeed!
It's usually targeted to webservers, before most HTTP servers got fixed you could DDOS a server with a tiny connection, but some HTTP clients can also be vulnerable.
But you may find better usage of your time than implementing this.
In this case though, to defeat scrapers maybe create some link which only the scraper sees and leave a "gzip-bomb"' like described here https://blog.haschek.at/2017/how-to-defend-your-website-with... and see how their scraper handle that :-)
Personally I just used a html-fuzzer to generate 5 MiB of junk html and named it wp-login.php :-) And a ssh-tarpit
As an aside, I’ve fought credential stuffers by returning real looking but actually false data, and initiating password resets... start serving different data on each hit, you may need to be annoying enough that they give up.
Problem is - right now I'm over 250 (new) IPs and they keep piling up (their domains now rarely use an IP more than once).
I may have to block entire ranges of IPs or whole ASNs.
Then, setup a script on your laptop or whatever to search this string on their domains every half hour or so.
It even prepares the expression snippet for me to paste directly into a CloudFlare firewall rule.
That's how I got to quickly identify and ban almost 2000 different IPs.
If they continue to expand the IP pool I may need to automate it though.
I wonder how banning so many IPs affects CloudFlare performance and if I should optimize it to block whole IP ranges instead ...
You might also be able to find common user agent headers through the CF firewall page and block them based on their UA. That'd not work if their scraper tool was randomizing the UA string but quite a few of them don't
As for user agent - they're using a very common, real browser user agent that's impossible to distinct from legit users.
Russia has no GDPR or something. So you could put (special key in) a cookie? They probably do not process it so subsequent requests without a cookie are to be discarded?
It is entirely permissible under the GDPR to use cookies for security purposes.
I blocked what I could and then I blocked the whole country (Russia) behind a CloudFlare javasript challenge (so that any legit traffic that wanted to pass it - could).
Everything stayed blocked for about a day and now it seems they gave up:
- all domains that pulled dynamic content from my site now show some other content and do not try to scrape Next Episode
- all static domains that delivered the cached version of the Next Episode homepage now deliver some other website(s)
- CloudFlare shows no further traffic on the firewall rules I've set (so the bots from those IPs are, for now, gone)
As this post is relatively old now (more than 3 days) I doubt many people will go back and check for new developments, but I wanted to give this the proper closure and let you know of the apparent happy end.
Also, given the way they were using your site, effectively reverse-proxying you and adding ads, it implies that you have access, in your server logs at least, to all of their traffic! And that might give you insight into their motivations, and maybe other elements of their operations. I mean, it sounds like a reasonably clever, small scale scam operation in Russia; but if they proved out the technique with your niche site, then they can easily duplicate with other sites, in which case it is effectively a new kind of malware that has to be solved by Google!
Last but not least, I wanted to encourage you, and others, to consider whether this kind of attack would work in a decentralized world, what search looks like in that world, and therefore how this kind of attack might be mitigated.
That happens when it is legal to hack/steal/cause damage to people from other countries
Didn't think I'd see the author but since you're here, thanks, this has been my go-to over the years.
1. Compile a list of domains and sitemaps that are 100% stealing and mirroring your content.
2. Go to Google's DMCA request page: https://www.google.com/webmasters/tools/legal-removal-reques...
3. Fill out all relevant data, and submit the offending domains and URL's.
Wait a few days, and you'll be happy to see that those pages are blocked from Google entirely. Not many people know what to do when Google DMCA's them, so it could solve your problem permanently (or you can automate it).
Regarding physically blocking them from scraping your site, you've got a few options. Put Cloudflare up if it isn't already. They've got at least one anti-scraping application (Scrape Shield) that may help.
Another thing you can do is automate the scraping of their websites using distinct query parameters and try to exhaust their list of proxies by automatically logging and filtering them. This might be a fruitless endeavor if they're using rotating residential proxies though.
Hope this helps, and good luck!
I've added filing a DMCA removal request to Google to the list of things to do if this continues ...
The rest of what you mentioned we've discussed in prior comments and are indeed helpful in mitigating this.