Ask HN: Legality of Web Scraping User-Submitted Content?
Is it legal to scrape another website for user submitted content? For example, if there was a news site that got its news solely from users submitting their own personally-written articles and stories -- who owns that content? Does the end user relinquish ownership once they submit that content?
Is it legal for another company to "scrape" that content and use it on their site, only removing it if the user who submitted it in the first place asks them to, either directly or via legal means?
Thoughts and insight would be appreciated!
47 comments
[ 3.6 ms ] story [ 94.2 ms ] threadIn the real world, however, people steal information all the time. Its not polite though. Usually people ask for attributes. I don't think a policy of only removing it when someone asks you to would be polite. That's like stealing something when no-one is home and leaving a note saying you'll return it if they ask you to.
Also, pure scraping, even of non-copyrighted information can get you into trouble if the other person had paid for that information, like a news site. They pay for their news. Scraping it and making your own news site (with full content, not just headlines) is illegal.
So the short answer is that the original creator still owns the content, and no you probably can't have it.
What if the user is submitting content that they would very clearly want re-distributed? I guess my question with that is, if you take the end-user who is submitting the content out of the equation, does the site that is being scraped from have any kind of leg to stand on if they don't want you scraping their content (excluding technical means that they might implement), assuming that the site being scraped from does not force their end-user who is submitting the content to agree that the site that they are submitting the content to "owns" the content, once it is submitted?
But like someone else said, if someone wants their content distributed, the user is not going to give you any trouble, and if they do, you're very protected if you take the offended material down right away, as someone else mentioned.
The website you're stealing from does have legal measures they can take, especially if you're directly competing with them. I think someone else mentioned that they could also just reconfigure their website to mess up your scrape, which is probably what they'll keep doing. They'll also likely publicize your bad business practices and you'll end up with a horrible reputation. Nobody likes a copier. Its like people who steal designs. They hardly ever outperform the website they're copying.
I'm not sure how this relates to other kind of content from a legal standpoint, but I've used it to ask myself 'did this person intend for this information to be public' as sort of an ethical guideline.
I've noticed there are plenty of menu services that take this same approach.. just do a google search and you will see there are many guides putting menus online without a license from the restaurant. What am I missing?
So, I called my friend who does have a menu service. His lawyer told him that case law may support the content of menus being public domain, but NOT the trademarks (like logos).
More important, he says, is the business issue. That is, the restaurant (or as it applies to this thread, the content owner) must PROVE that damages were caused by you, in order to take legal action.
Public domain is what it sounds like. Things in the public domain are available for use by anyone for any (legal) purpose. Only once copyright expires (or voluntarily relinquished) does a work enter the public domain. Menus are copyrightable, and automatically granted copyright. Regardless of the "business use", the business owns them and their use.
Who are these lawyers and why are they making such a hash of copyright law? Why are you guys trusting them?
Did you read the definition of public domain on the link I just sent you? Apparently not. Here you go:
* their copyrights have expired; or * the copyright owner didn’t follow certain required formalities (so they didn’t get a valid copyright); or * the works weren’t eligible for copyright in the first place; or * their creators dedicated them to the public domain.
You still think that the content of restaurant menus is copyrighted without question? Go down the street and let me know where I can find the formal copyright for 'Ben's Kabob's' - or whatever local diner is in your neighborhood. Even something like Chili's could be argued to be public domain because of the way they are already allowing use of their content for marketing purposes.
Who are these lawyers? And, who are the lawyers for the dozens of menus guides I can find on google? Why do all these entrepreneurs 'believe' them that it is a safe venture endeavor? Well, it seems like they are telling a consistent story, there have been zero problems after several years, and overall, its just a question of risk like any other decision you make in your business. Its not a perfect world, and sometimes you just have to take the advice of the best domain experts you can find and move forward to focus on building a profitable enterprise.
RIAA does not have proof of loss of damages in a lot of cases. Neither does MPAA. http://blog.wired.com/27bstroke6/2008/06/mpaa-says-no-pr.htm....
Besides that is irrelevant. What you meant was you need proof if you are suing for damages. You don't need proof of damages if you want someone to stop using your property.
Yes, restaurant menus are still copyrighted without question. Arguing with you about this is getting very tiresome, I suggest you go read the Wikipedia article on copyright as a starting point, or the other article I posted. Trust me, try reproducing the entire Chili's menu and see what happens. It'll be an interested experiment. Their menu is not in the public domain.
I very much doubt most entrepreneurs believe that it is a safe venture. Most people won't fight you on reproducing things if it benefits them, which is why people do it. But in reality the copyright owners are perfectly within their rights to demand a licensing fee or the removal of the property.
However, compare "basic facts" with an example from the menu from my favourite restaurant (www.chachachar.com.au) (and an example of fair use!):
"Eye Fillet Age 28-36 mths old Lean with sweet, clean, toasty flavours Served with truffled mash potato, salad of grilled pear & pancetta with smoked tomato relish
Hereford sourced from fertile pastures of NSW raised on Cattle Care accredited properties that are managed under stringent quality & environmental systems and are finished on feedlots in Jindalee."
IANAL, but depending on the nature of your service and specifically how the content is collected, you may qualify for DMCA safe harbor protections. This means that if you remove things in a timely manner upon request nobody can sue you. This is how Google caches the whole internet without getting sued.
That's all legal mumbo-jumbo. The real world answer is that some people will get mad regardless of the law, so take their content down and apologize. Follow robots.txt guidelines. Don't post takedown replys a la PirateBay. Generally act sane. If you do all of the above you'll probably be ok.
Examples are an ISP that merely provides access to the internet. The ISP cannot be sued for copyright infringement just because infringing bits passed through its servers.
Also, a message board that posts whatever a user writes would obtain DMCA safe harbor. They just provide a service and don't screen out for content. An example is Craigslist.
This person's site DOES NOT enjoy DMCA safe harbor. He scraped the other site and populated his own. He is not even acting as a service provider as the DMCA statute prescribes.
If the site doesn't have one of those pages and you don't ask for permission, you're not only putting yourself in danger of legal action, but you're also depending on a data source that isn't reliable.
Remember, it'd only be a matter of time before they notice you scraping, and take measures to stop it.
The truth is, it’s hard to find a site without the mention of copyright in terms and conditions page.
Overall it will depend of the use you make of the copy content. Trouble starts when you use someone’s material in a line of business that competes with the owners.
That's the theory anyway. Talk to a lawyer.
http://yro.slashdot.org/article.pl?sid=08/07/08/1245204
I think what most of the responses so far are missing is the importance of accrediting the content to the content owner (likely the site, not the contributing users), and providing a link to the source.
Check out this pdf <a href="http://www.law.berkeley.edu/journals/btlj/articles/vol16/sab...">Sableman's authorized linking</a> and search on Google v. Perfect 10.
You haven't really given much to go on with respect to what you are scraping, and what you plan to do with it. But I think a bunch of common sense and ensuring that your site in no way harms the original source's site (defamation, etc), are the most important things to consider.
Hope this helps.
The user owns copyright to the article or story that he wrote. This ownership of copyright gives the user the right to decide how it is distributed. For you to use the material legally, you must get permission of the copyright holder. The user may be the copyright holder, or the site may be if the user transfers the ownership. Either way, someone owns the copyright and it is not you. You need permission.
There are two defenses to copyright infringement: 1) fair use, and 2) parody.
Parody probably doesn't fit here. So that means you need to make a case for fair use.
Fair use has 4 elements:
1: The purpose of the use (commercial vs. non-commercial/educational) - if you are going to make money on this, fair use is out.
2. Nature of the copyrighted work - This doesn't really apply here, so I won't go into the lengthy explanation.
3. Amount of portion used in relation to the whole - Did you extract a quote? That is probably ok. Did you copy the entire article? Probably not ok.
4. Effect upon the market - if your site harms the market of the other site, no fair use.
"While commercial copying for profit work may make it harder to qualify as fair use, it does not make it impossible."
see: http://en.wikipedia.org/wiki/Fair_use
The use in the case you are referring to had several extra elements which prompted the court to not find copyright infringement.
2 Live Crew was fair use because their work was transformative. They took a beat and changed words to a Roy Orbison song. They did not blindy copy, as the question asker will be doing. They took the score, not the words.
The poster is taking the story, not quotes. His site is not transformative, and therefore his commercial use is not fair
Seriously, though: of course the person asking the question should retain a lawyer for definitive guidance. However, asking around to get a feel for the opinions of intelligent people makes sense, too. It might, at the very least, clarify the questions, and it will help detect if the lawyer's words make sense. While attorneys are valuable, they are also fallible, and their advice should be as critically considered as any other.
The danger is that misinformation is often worse than no information at all, and asking a bunch of non-attorneys for legal advice is sure to net you a ton of it.
Well, I don't know as I want a lawyer to tell me what I cannot do. I hire him to tell how to do what I want to do. ~ J. P. Morgan
But some answers are so blazingly obvious or common knowledge that it's worth asking. It might at least speed up a conversation with a lawyer saving many hundreds of $s.
http://www.google.com/support/webmasters/bin/answer.py?answe...
I don't know if this "opt-out" strategy has been tested in court, but I wouldn't assume that wholesale copying of user content is analogous to Google's situation unless the users have some similarly accepted way of opting out of the copying.
They scrape, re-present abstracts from and supposedly do calculations based on of the entirety of user-submitted data collected by a number of sites.
Does that violate fair use in anyone's opinion?
It depends on the terms of service of the site.
#1 The TOS of the site may not let you use a robot on their site at all
#2 The TOS will define who owns the user generated content (UGC), either the user or the site
#3 Depending who owns the UGC you may or may not be able to scrape it, if it's the site it's against their TOS if it's the user you would need permission from the user.
#4 As other people have said fair use might come into play. If the site owns the material using a single user contribution might be fair use within the context of the whole site. If the users own the content you are likely to be using all their content, thereby not able to use fair use.
Again all of these are my observations. Hopefully it will give you something to think about. If you are starting a business based on this, you do need to consult a lawyer. Also starting a business based on page scraping is a pretty risky thing to do. If the scraped site turn you off you could be pretty screwed.