HN Explorer

Ask HN: What countries are safe for web scraping?

4 points by alexmarin ↗ HN
If I wanted to start a site that scrapes listing data from another site. And provides analytics, market intelligence. The source site has TOS that disallows scraping. And they have shutdown another scraping site previously. The source site is not big and I’m providing value and aligned with their mission but seems they don’t care.

I want to host it in a country that is outside of their jurisdiction just to be safe from cease and desist/getting sued.

Anyone know favorable countries for web scraping? Or what other methods I can take to protect?

9 comments

[ 3.5 ms ] story [ 29.4 ms ] thread
rolph
how about providing a security service [APP] that inspects the web scrape made by the client browser, to provide awareness of the sites security/threat properties ? Its semantic but scraping to protect users should be different from scraping just to snoop a site. IANAL so the line between hacking, and threat model identification should be investigated somehow.

And of course, the development of such a system is a time and cost concern.

sebst
First, it's their site. If they don't want it to be scraped (even if their reasoning sounds silly to you), I'd advise you to obey their IP. (That's the moral pov)

Second, IANAL, but the hosting location should not matter as much as you might think (YMMV depending on your jurisdiction): It's you who is responsible for the (probably unlawful) scraping. So, they will sue you in your jurisdiction, not the machines hosted anywhere. So, your question boils down to finding a hosting provider which basically ignores abuse complaints and protects your anonymity. (That's [probably] the legal pov)

Third, even if you found a provider and they are unable to effectively take your site down and/or sue you, they might just block the IPs they identified the scraping originates from. So you'd have to use IPs from different subnets to obfuscate what you're doing. (That's the technical pov)

So, in conclusion, why not scrape their site, build whatever product you have in mind on top of their data and show it to them (and only to them) to arrange a deal. More often than not, you can convince people more easily when they can visualize what you're talking about. So, show them your prototype.

If they still reject a collaboration with you, they might have their reasons. It's not your cup of tea to judge them for their business decisions, then. Better find another business partner who matches your views and ideas.

tldr; Why would you build a business on a never-ending fight with someone you assume you share a mission with?

100100010001
Alter the scrapping so it looks like normal activity.
duxup
>providing value and aligned with their mission

What does that mean exactly?

thedevindevops
Could you maybe do whatever-it-is-you're-planning-to-do-with-the-scraped-data under the guise of a review site?
dangerface
As far as I know there is no legal recourse against scrapping even if they specifically disallow scraping, people have tried to sue google for scraping and lost.

You using the data will break copyright in pretty much every country, it doesn't matter how you got the data it's how you use it. They will sue you for copyright breach and most likely they will win. There is no legal loophole even for people with googles money they end up paying for a licence for the data or remove it.

infinitone
Then how come companies like https://www.airdna.co/ continue to operate?
tjkrusinski
Simply because they have that data doesn't mean they sourced it illegally. Their value is likely the way in which they assemble disparate data sets, obtained with permission or purchased.