Ask HN: How do you feel about web scraping?
Does the purpose of your scraping make a difference, if one use is just a project but another would be selling the data?
Are comments on sites like this public data or private?
What about sports statistics that sometimes are "private" by the league, rather than just open for people to use and write interesting articles about it?
Overall thoughts?
I actually created a site (I edited and deleted the mention of the name here because apparently people don't believe that I don't care about people looking at the site.) that scrapes comments and posts from Reddit that link to Amazon products and shows that information. I'm thinking of adding Amazon links from HN and other sites, but just not sure about how people would feel about scraping from sites like this.
168 comments
[ 6.2 ms ] story [ 218 ms ] threadGoogle won't bombard your site with requests and at worst will cache data but not try to make use of it.
This is changing, though, with Google's slow but steady movement of organic search results down the fold.
Someone putting up a bunch of words doesn't bind you to a contract with respect to scraping or anything else. (At most they can give up a right. They can put something in the public domain, for example, by giving up their copyright.)
To give some extreme examples: What if the site says you owe them a million dollars if you even look at the site? What if someone put a sign on their lawn that says anyone stepping on the grass can be legally shot? Are you OK with those just because they said so?
Scraping is copying, which is an exclusive privilege of the copyright owner under copyright. In the absence of a license or a copyright exception, you have no legal right to exercise that privilege. Someone putting words on a page might give you a license for the use you intend.
You don't need to be bound to a contract, because this isn't about restricting a right you have in the absence of license.
Is it necessarily copying in a sense that is different from how a web browser acquires the resource to render it? For example, scraping data to derive some non-protected facts could be considered consumption in the same sense a human reading it via a client would, and doesn't require storing a copy any more permanently than a browser.
Intent is probably a lot more important, legally, than the fact that you are technically copying something simply by "accessing" it in a browser. If you are scraping with the intent to extract and copy the copyright protected content, sure, that's a good point, but that's by no means the only use of scraping.
Personally, and I understand that this isn't how it works legally, I think that the contract is implied by the protocol. I request copies of information by means of HTTP. If the distributor doesn't want me to have a copy, they can choose not to respond with a copy. I never make a copy in that case: the server is, hence I have not reproduced the work. It's not me taking a book to a photocopier, it's me calling the publisher and asking them to send me a copy of page 237.
It's different in purpose of not mechanism, and insofar as there is an implicit license to copy in a browser for display, it is another step to establish that such a license extends to other purposes even if the mechanism is the same.
Also not that the existence and scope of implied license can be affected by the presence and terms of an explicit license, which is one reason you might want to read the T&C prior to scraping.
Also, the validity of a set of T&C that I didn't explicitly agree to and likely never read is dubious, inconsistently enforcable at best.
No, I'm not. Fair use is one (of several) copyright exceptions and, as such, is addressed in my post.
https://media.giphy.com/media/EouEzI5bBR8uk/giphy.gif
Can you be a bit more specific?
You don't really get to decide how somebody else's data gets used.
Using your sports stastics example, this will become a grey area as writing becomse more automated, but at the moment, a writer gets a 'statistic' like a score which is made publicly available. There are no limits on using that statistic. But you didn't automate the process of spreading the stats, you, in theory found a fact and wrote about it.
This is different from just giving a feed of stats, or linking through a bunch of services.
https://www.reddit.com/help/useragreement/
Doesn't really matter how we all feel about it. If you hit their radar, they'll go after you. Can be expensive, whether you're in the right or not.
You're more likely to hit their radar if you're trying to make money. I suspect you're using affiliate links, right?
And I did have affiliation links initially where I included the non-affiliate links next to them if people didn't want the affiliation. But then I got rid of those cause people didn't like it. I don't care about making money on it, just think it's interesting.
If you aren't trying to monetize, you will be lower on their radar, but not immune.
"By submitting user content to reddit, you grant us a royalty-free, perpetual, irrevocable, non-exclusive, unrestricted, worldwide license to reproduce, prepare derivative works, distribute copies, perform, or publicly display your user content in any medium and for any purpose, including commercial purposes, and to authorize others to do so."
The original comment submitters could grant you a license on their own, but that might be difficult to coordinate.
If anyone, I'm infringing on the copyright of the original author of the comment, while Reddit has no grounds for a copyright infringement lawsuit.
I view this as significant mainly because it changes what they could sue you for, and any fair use evaluation if copyright does enter into it at some point.
I suspect Reddit might claim a Collection Copyright in the compilation of posts/comments - so even if you could acquire your own license to a post and each of the comments - if you tried publishing that as a book, Reddit could claim the organisation of those individually copyrighted items is owned by them...
Where? I don't see anything that prohibits scraping. There are provisions against rehosting content, but that isn't scraping. You can scrape content without rehosting if you are locally scraping in an app, for example.
"I actually created a site...that scrapes...from Reddit...and shows that information."
"that scrapes comments and posts from Reddit that link to Amazon products and shows that information"
It's ambiguous, but my interpretation is that it's the Amazon content (or perhaps a summary thereof) that's being rehosted, not the reddit content.
I got some comments on this about people not liking me throwing the url in here but figure it'd explain things better.
And also, this post isn't just about whether or not what I'm doing here is legal and ok, overall thoughts on it as well.
And seems to surface funny or interesting things based on popularity, like the 55 gallon drums of "personal lube". So, the aggregation has some purpose.
Good luck with it, seems an interesting idea.
There are some websites whose primary business model is providing content in exchange for something: a subscription fee, or advertising eyeballs. They have a very strong financial interest in your not scraping their content and providing it to others on different terms.
There are other websites who make some content available and explicitly authorize people to use it: various datasets and RSS feeds and such.
And then there is a wide swath of websites that have adopted generic TOS that prohibit scraping, or they prohibit it because they haven't given it much thought and can't think of any particular reason off the top of their heads to permit it.
So what you really want to know is what sites in the third category would consider a sensible scraping policy, if they had to give it sufficient thought.
In other words, if they don't just default to a prohibition because it's already in a TOS template or because they haven't thought it through, what's the rationale for either blocking or not blocking scrapers?
"it's naughty for anyone else to do it" if that everyone are companies that will be using or monetizing that content without providing any kind of attribution or compensation to the source/creator.
It would be ideal to define the terms for scraping (similar to how API calls work) as opposed to the acceptable authors of scrapers.
Given the difference in the size of the internet now and then, the backlash faced for poor practices may not be comparable.. so perhaps not.
Of course there are people building services that scrape certain sites that appear to be off limits to you.
Those people scraping sites that are explicitly prohibited either:
a. are breaking the rules, potentially the law if it's explicitly prohibited in a ToS, and will eventually have to deal with getting banned, or sued. It's quite a gray area legally but here are some laws that could be used against you:
Violation of the Computer Fraud and Abuse Act (CFAA). Violation of California Penal Code. Violation of the Digital Millennium Copyright Act (DMCA). Breach of contract. Trespass. Misappropriation. Source: Linkedin v. Doe Defendants
b. have an agreement with the website owners allowing them to scrape certain portions of their site.
c. scraping data with no rules concerning it.
For example, Facebook. has a ToS for scraping: https://www.facebook.com/apps/site_scraping_tos_terms.php At the bottom there is a form for those that want to get permission to scrape the site. And their robots.txt is heavily used to control crawlers with User-Agents they know. http://facebook.com/robots.txt
It's rare you would run into legal issues, but possible. The question is whether it's morally okay for you to scrape any data you want.
A scraper company, funded by magic money (Knight Foundation grants) and $1m of VC, convinced a (UK) Government department to pay them to scrape our site for some analysis the department wanted. They'd never contacted us, never asked for permission, never asked if we could supply the data. Our company was bumping along at this point and having to lay people off. Income from a nice lucrative Government contract would have kept a couple more people in work.
The scraper company's FAQ was, in my view, full-on unethical:
> "we check the robots.txt file. If the site permits robots in general to scrape their site (NOT just GoogleBot!), then we will do so. We will make no effort to look for other terms and conditions as well."
You will ostentatiously "make no effort to look" for T&Cs in case they prohibit the significant contract you're about to sign with the Government? Whoa.
So how I feel about web scraping is simple: "don't be evil". If you're diverting income or traffic from the original site, don't do it. If you're genuinely adding value, go for it, but be open, be prepared to work with the original site, and be prepared to accede to their wishes.
We're talking a custom scraper written for this site and this site only.
Yes, I am expecting the people who spend hours inspecting the source of my site, and then writing a custom scraper for it, to spend 30 seconds reading the T&Cs first.
If you want people to read it put your content behind a sign up with a checkbox.
Seen here: https://kazuar.github.io/scraping-tutorial/
Incidentally, I'll be shopping later. May I give you $5 for you to drive to London and take me there? I was going to steal your car, but then I figured I'd be a good citizen and demand you provide it to me at your own, prohibitive, loss.
If you're using my data to hijack my traffic, without asking, you could have all the right justifications in the world but you're still a prick. Who knows, maybe your orphanage building app will move me tears once I hear about it and I'll give you free access.
EDIT - I can't reply to your comment below, but FWIW I agree that scraping sites in this manner is unethical. I am merely describing the logic that most scrapers go through for self justification and legal protection.
But whatever. It just saddens me that the internet is a constant "don't be a dick" battle with companies like the scraper guys.
(edit - understood :) )
My own take on it in general is that for personal/research use I'm not morally opposed to scraping, even when it's in violation of the ToS, with two conditions: that it doesn't place an unreasonable burden on the server, and that it doesn't invade people's privacy. The legal significance of the ToS is murky at best (disclaimer: I'm not a lawyer) but if the site asks you specifically to stop scraping them or puts up a technical barrier you should stop (morally and, in the US at least, legally: see craigslist v 3taps)
Data yearns to be free, stop fighting it!
So to answer your questions:
There is no correct answer, at least unless you are willing to wait a decade and spend millions while the cases make their way through the byzantine legal system of districts, circuits, appeals and supremes. Unlike Science & Technology where there is a "correct" answer, you should approach legal system with different perspective using instincts and acceptable risk tolerance.The history is littered with people who took a bet, and ended up succeeding or failing upwards.
Finally ignorance is actually preferable to knowledge. By writing this question or say having this conversation over an email you are simply creating a paper trail that can only harm you if you get sued tomorrow. [3]
[1] https://photo.stackexchange.com/questions/53304/what-is-the-...
[2] https://www.nytimes.com/2014/05/07/business/media/rap-genius...
[3] https://www.fastcompany.com/1588353/steal-it-and-other-inter...
Web scraping is going the way webbots work now. Have to scrape data to automate stuff.
We were--at the time, scraping for lead information to add to our marketing database, and this isn't the thing I'm exactly the most proud of in my career. But we all make mistakes. I wouldn't do that again.
At the same time, we rotated things so that we weren't killing the websites in the niche market that we were trying to scrape for leads.
The algorithm was that we would seed Google, Yahoo, and Bing with certain keyword searches that were relevant. Then we'd take the search results from the APIs and stuff them into an array. Then we would sort them proportionally. If we (like we did) most often get the most hits from google, followed by yahoo, and then by Bing, we'd stuff the results into an array and intersperse them.
So if we had 3x google results and 2x yahoo, and 1x bing, we wouldn't hit the google results first. We'd hit a google result 3x then a yahoo 2x, then a bing 1x and cycle.
It was a decent way of doing things.
We never broke anyone's stuff. Even if it should have been.
In the grand scheme of things, I think that I did the most responsible thing I could have with the task I was assigned. But the task falls into a general category of things I don't approve of.
It certainly wasn't illegal, and it probably wasn't unethical, but it was definitely gross, and my internal standards tell me to avoid things that make me feel gross.
That said, I'm kind of pleased with the technical results. Up until that time in my career, I'd never encountered a sorting algorithm that handled things based on the proportion of similar items in an array. I'm sure that other people have done this and that there's no way it's novel at all. But it was a cool challenge to make a shady task perform in a way that didn't break other people's shit just so that we could maximize our own efficiency.
https://en.wikipedia.org/wiki/Sui_generis_database_right
Instead of violating the ToS we have business contracts with each company, that give us the permission to scrape. We use this as a way to take control of integrations, and put the ball in our court, as most of these companies have little to no technical expertise or resource. By doing this we can create an integration as quickly as we want, instead of waiting months or even never managing to get one if it were to be done through an API.
Scraping can be a powerful tool in this respect, make sure you have permission, but ToS saying no scraping doesn't necessarily mean you can't get special permission.
ssh into the machine, select using a db commandline tool, return data back from stdout in csv format, parse data at our side. It's surprisingly fast and secure.
Or do you also plan on suing Google once they scrape your site?
> "scraping" site every 15 minutes with massive ddos
Google doesn't scrape every 15 minutes. It was implied that they scraped the site often enough to cause a DDoS.
Additionally, it clearly wasn't a DDoS. Quoting from the parent:
> Some swedish equal rights server was "scraping" site every 15 minutes with massive ddos. We put capacha for the server ip only (let them scratch their heads now) and let it be
If there is only one server ip, it is by definition not distributed. Just a misuse of the term DDoS.
Google has a variable timing crawler. Google News crawls in almost realtime, but all Google crawlers back off if the site responds slow.
Additionally, Google respects `/robots.txt` (which allows the site owner to define a crawl delay) and uses sitemaps for hints, so it doesn't have to re-crawl every single HTTP object "every 15 minutes".
The simplest thing to do is NOT parallelize your scraper, and sleep for a few seconds between requests.
Did you mean "We couldn't handle the traffic caused by their scraping which lead to a denial of service"?