HN Explorer

Stopping web scrapers with HTML character codes?

2 points by icu ↗ HN
How robust is the idea to protect my website content from web scrapers by converting all my text into HTML character codes like this: <xmp>&#72;&#101;&#108;&#108;&#111;&#32;&#87;&#111;&#114;&#108;&#100;</xmp>

7 comments

[ 3.6 ms ] story [ 32.5 ms ] thread
v2interactive
I believe this could negatively affect SEO
icu
Thank you I didn't even consider this.
mooism2
If the scrapers are targeting your site specifically, then it's not even remotely robust.
icu
Yes this seems to be the case, thanks for your reply.
pixeloution
Its not effective vs any scraping; how do you imagine scraping works? In the most basic case, a program makes a HTTP request, gets the source of your page, and processes it via a HTML parser.

In some cases, the same HTML source will be run through a headless browser to parse it, in which case javascript trickery can't protect it either.

The basic answer is: if an end user can see it in a browser, you can't hide it from a scraper.

icu
I run scraping with Batch, CURL, SED, BCP and SQL Server 2012 or MySQL. My scripts basically download the source code into a text file, run a regex with SED to get content and then put the info into a DB with BCP. I don't take into account any HTML codes for text so such a technique would break my scraping technique.
Lorenz-Kraft
You can't stop scraping. The only thing you can do is detect them. If detected, there are several fun stuff to do. Like showing different content. BUT: Be aware that you might block a "legal" SEO scraper!