I used to work for PriceRunner, and I remember that while scraping a site and extracting content from the results was easy, avoiding rate limits and handling all the edge cases was hard. From the feature list it seems that CloudScrape is running full headless browsers, so handles JavaScript, allows screengrabs and that stuff is expensive.
What you put online is public. Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.
OK, how about hiring hordes of cheap human scrapers from Mechanical Turk? Everything visible can and will be scraped if need arises, the only variable is price.
You're making a distinction that, if you were to actually explain it in a way that makes sense, would reveal a much more relevant solution to this problem.
What's the difference between a machine scraper and a human scraper? Humans use machines to access websites. All your users are machines. And behind every machine is a human.
If you don't want someone to access all your data in a short time-frame, then don't give it all away in a short time-frame.
There have been lawsuits where the company running the site-being-scraped sued the scraper and won. So while it's technically possible, there can be legal issues.
That is incorrect. Even if you don't republish the data, the scraping alone can be a violation of the site's Terms of Service, and you can be sued (and lose). See the Wikipedia article I linked above.
Think about it from the point of view of the site being scraped. Suddenly your webserver is being slammed with traffic that's way beyond what any human would cause. It's costing you bandwidth and compute resources. You have every right to say no, scraping is not allowed (in your Terms of Service). It's then up to a court to decide if those Terms of Service are legally binding. Some courts have decided that yes, they are.
Well that sure is an arrogant reply, and not even close to true. In the US everything published (online or not) is copyrighted.
Since you're clearly not a content creator, and have decided since you didn't work on it it has no value, but putting something online does not make content public domain.
Not correct. Google will completely ignore the rules in robots.txt if it deems it acceptable. I think there's a link to this somewhere in this comment page.
Putting it online allows me to scrape it, but not to republish it.
I might just want to train a neural network on extracting data out of websites, and, for that usecase, wget -m every website I can find.
How I use the data you present to me, as long as I don’t republish it, is my decision. If you give me a license to read it, you also give me the license to copy it onto up to 7 different media at the same time, and to show it to up to 7 friends at the same time.
No. Your site is public. If it is available via HTTP without authorization, I can scrape it (but not republish). There have been in many countries (except the US) court decisions about this.
There are ways to force authorization, if you don’t use them, your fault.
If I just use TOR or VPNs or similar to get around your block, also, again, your issue. Unless you require authorization, your service is, by law, public.
And if I want to store the site offline for later reading, or read it now in a browser, is nothing that you have to decide.
Well, copyright does prohibit redistributing the scraped content. And that is what all 3rd party content scraping services essentially do. That violates copyright law in quite a few countries. If you scrape with your own tools, own servers and use it only within your company, I would think it's a different legal matter.
No, it's not "by law" public, or could you cite a few laws for that, both US and EU. Would appreciate that.
It is public because the HTTP standard defines it as public, and, if you claim to use this standard, its definitions apply to you (if you claim to support a standard, you have to accept that its definitions will be used in court against you)
HTTP directly says you should use authorization to make stuff non-public.
And third-party content scraping tools are NOT violating copyright law, as they do not redistribute it freely.
Otherwise services like Opera Turbo – which scrapes a website, removes cruft, compresses data, and sends it to your phone – would also break copyright law, and there is a nice exception for this case, under US law, it’s covered under fair use.
(Unless the scraping service makes your content accessible publicly – and even then, fair use applies usually, as in the case of pure archival like archive.org or archive.is)
You can have your content blocked from archive.org also by robots.txt - didn't check archive.is
Are you a lawyer? I am not - anyway, I can't really agree with that falls under fair use.
If I charge money to scrape someones site and then hand over the material in whatever form my customer uses - I am bound run into problems sooner or later.
If these guys are serious about their business they are using proxies to obfuscate who they are. There are many services now that are claiming to have a ton of IP addresses Luminati, Shadio.io and Nohodo are just a few examples.
Awesome, thank you. They do look interesting, but their pricing confuses me. Are they just selling individual proxies? Or does $1.80 get you single exit nodes or threads basically?
I use to think web scraping was limited to obscure proprietary companies like connotate so I'm glad to see more of these tools becoming available for everyone to utilize and hopefully create something cool.
Intercom.io is a very cool tool indeed. We use it to extract unique product links from product overview pages of webshops for adding them to Pricepin (our own tool).
Nice timing, as just this week I had my first exposure to the Cloudscrape platform. Credit where credit is due, I found the platform refreshingly feature rich even at lower tier (i.e. free) account levels compared to other offerings from competitors.
Downsides were like so many other crawlers/spiders - too much in trying to meet the needs of all. Aiming for the 95% is easy, 95 to 99% though, not so much. Hence all the corpses littered along this path.
Recommendations:
Define solid, real-world examples of your platform in practice and really, no bullshit, what makes you different from every other platform that's 97% polished out there?
You didn't meet my needs because I needed much more in the way of parsing network requests from pages, mostly in handling if/then scenarios. A relatively simple scenario, to ME, and there-in lies the challenge of the crawler market, it's the last 5% that's custom to each scenario that so often delivers 98& of the value.
Suggestion: Identify and use the talent you obviously have to destroy some high value markets. Hint: Prioritize targets that still champion a desktop app at any point in the top 5 in terms of data collection.
Outside of that and far more niche (and subsequently far lower priority, allow for parsing / handling handling network requests and responses via custom app/API instead of just block all, some, or none)
Thank you for the recommandations! Would love to know more about your exact use cases - we're aiming for the 100% :) If you'd reach out on support@cloudscrape.com - we'd be happy to investigate if we can't make CloudScrape work for you as well.
Also, do they respect robots.txt? And how are they going to avoid being blocked by websites that don't want to be scraped? (I guess it would be easy to determine the IP addresses of their scrape robots).
Hit a certain threshold and you'll most likely get IP banned by the site you're scraping or heavily throttled unless the site does not care to minimize bot traffic (which can cost the site owner valuable bandwidth and server resources)
You may also get sued...
If the site owner wants its data to be available for automated extraction they'd provide an API and can price it to compensate for cost of serving all those bots.
Google has a unique position in that any site that wants to be found has to let Google bot index its content. Google does not build a derived product from your site's content that ends up competing with your site.
If someone needs to build a search site for real estate etc why couldn't they just scrape the Google search result, filter it (white list domains), extract the actual link and present it? In that case you'l need a Google specific scraper that can be based on open source scraping libraries.
Update:
Google actually will IP ban you if it thinks you're a bot trying to scrape for search results.
First of all, Google does take your content and make a product out of it by selling ad space on search result pages. Those pages would have no value for advertisers if it weren't for the content producers Google scraped to fill those pages up with.
Second, you linked to an api page that clearly says it was deprecated 5 years ago, come on. No one should ever feel bad about scraping Google, considering Google is the world's largest scraper themselves.
I fail to see a problem you are trying to present.
Even if identification was hard, which is not true because of how HTTP works, it is irrelevant because HTTP doesn't discriminate. If someone does, that is their problem, and should be solved by them, and not a committee or law.
> If you don't want someone to access your page, then don't respond to their request
> there's no easy way to always reliably identify the requester
That's the problem: you can't identify the person to block them in the first place.
Robots.txt is actually an explicit signal of intention for reputable search engines but that's all we have today and is easily ignored and does not work with these scrapers or anyone else.
You haven't quite laid out your argument so I have to guess what it is.
When you say "That is not how HTTP works" it suggests that your claim is that anything that HTTP allows is ethically OK to do. However that is clearly a ridiculous stance, since a DDoS attack is a stream of valid HTTP requests and that's clearly not OK.
So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK.
I find this an interesting question, because while I would love for protcols to also define ethics, I feel that would be scope creep for the poor protocol designers. There's a wide variety of conduct and ethics questions that a protocol cannot address.
Where I myself draw the line is at protocol behavior intentionally designed to obscure my intentions. For example, sending my requests from a wide variety of IP addresses is behavior that is specifically designed to obscure where I'm coming from; my only intent in doing so would be to circumvent the intent of the serving machine from providing lots of content to a single requestor. At that point I'm engaging in deceptive behavior; I've crossed an ethical line.
When you say "That is not how HTTP works" it suggests that your claim is that anything that HTTP allows is ethically OK to do. However that is clearly a ridiculous stance, since a DDoS attack is a stream of valid HTTP requests and that's clearly not OK.
That wasn't a response made to your comment, and you are mixing two different arguments there. You guess in not correct.
So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK.
I never even suggested such an argument.
The behavior you described in the last paragraph is only deceptive from the eyes of an information and privacy surveillant state actor.
Anonymity is not unethical, it is a human right.
DDoS attacks are malicious events that disrupt service. In almost 100% of cases, scrapers don't want to disrupt service, because they need the data they're scraping. They want to be able to continue to get it, so they won't do things that may harm their ability to do that (including presenting honest IPs and user agents).
Services like this one actually make scraper-related unavailability, which IMO is already greatly exaggerated, less likely, since there will be fewer amateurs trying to write their own bots and accidentally breaking things.
To the extent that a scraper harms the other business, the scraping company can be held civilly liable on several accounts without specifically bringing scraping as a practice into the picture. All that matters is that they damaged the target site's ability to operate, not that they were saving [portions of] the pages (that'd be a separate copyright claim, unrelated to the disruption of service).
You have the right to disagree, but that's the way World Wide Web was built. Feel free to use alternative service(s) or stop publishing your stuff. Put it behind password or don't answer to my scrapers or browsers requests. Fair and simple.
Web is for people from people, not solely for company(s) financial interests.
What's the WWW or the way it's built have to do with it? And the web is just technology, it's not "for" anyone or anything in particular.
Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't see how that's realistic.
>What's the WWW or the way it's built have to do with it?
What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default.
Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins.
So yes, intentions matter. The question is how we learn about them. Sometimes the choice of technology implies particular intentions by default.
Putting a up a webserver that can be publically reached is not authorization to access it. I really can't say this in any other way - just because you can do something doesn't mean you are allowed to, whether it's online or offline.
We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against actions on the web in general which is why people can get away with scraping but please don't mistake it for somehow being OK or allowed by the owner of that content. It's just not that simple.
">Putting a up a webserver that can be publically reached is not authorization to access it."
This is legally incorrect. Without any further information or protection measures by the publisher it is legal to access content on a public web server.
">I really can't say this in any other way - just because you can do something doesn't mean you are allowed to"
You are allowed to do everything that is not expressly forbidden by law. Accessing a public web server is not forbidden by law unless the owner takes steps to prevent you from accessing it or at least clearly signals that intention. Terms of service do not constitute an implied contract, so you are not required to read the TOS before accessing a public page.
">We already have an explicit signal called robots.txt"
Exactly. That is part of what I meant when I said that any restrictions had to be implemented on top of the default, which is that everyone who can is allowed to access anything on a public webserver.
[Edit] But initially, this wasn't a thread about legality but about ethics. I think there are unethical reasons to scrape and there are unethical reasons to block scrapers. We simply need to know more about the purpose of any scraping before making a judgement.
2) This directly contradicts your previous comment: "Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins."
I agree that this is about ethics and any scraper that doesnt honor robots.txt and explicitly uses different IPs, user-agents, and other methods to mainly disguise itself as a machine service is unethical in this context.
Absolutely not. The default intention of putting an HTTP server online is not "letting stuff lying around", it is publishing stuff. And yes, the default can be overruled in various ways.
>There is already case law precedent regarding this exact type of publically accessible information not being authorized
You're grasping for straws here. In this specific case, it was completely obvious that this information was not supposed to be public. It was an embarrassing security failure that the defendant wanted to expose.
I think we agree on a lot. robots.txt should be honored and scraping in way or for a purpose that negatively impacts the website's viability or business model is unethical. But usually, such purposes are covered by copyright law anyway.
You still have copyright to your online content and can dictate how it can be used. Websites can also have TOS for their content. If you're arguing ethics then it could be unethical to use content in a way that the copyright owner doesn't want it to be used.
IANAL, but copyright governs redistribution of content not consumption (That's what pirates get busted for). I aslo recall that there was a ruling that footer TOSs aren't enforceable unless the user actively and explicitly agrees to them.
I agree with the GP in that public content is fair game. How do you thing google works?
Google technically respects robots txt and noindex metatags. OP is arguing the ethics of scraping, not if people are ignoring bot meta tags.
Copyright governs how the content is used, including distribution. The reason people who download videos are not liable is because you have to download the complete content to see the copyright. File sharers have already downloaded the content and are subject to copyright. Bots that scrape can interpret meta tags in the header of the dom, which is why scraping and violating copyright is unethical.
I don't think it's unethical to pull down a copy of public information. If you pull too fast it might be considered rude (heavy load on the server). That's why some sites reflexively block all scrapers, hence the rotating IP feature. Hopefully this tool is rate-limited so it's not rude.
In terms of copyright, what matters is what you do with the scrape. If you scrape a public website for personal use, it's no different from just browsing it for personal use. If you try to republish the content for your own benefit, you'll run afoul of copyright law.
Not hard to code for that, especially if your static content like images are served by alternate servers, so you can just focus on your content servers.
The problem I am dealing with lately is massive bot farms where you do not see a repeating IP for an hour. I catch them eventually but it takes a toll.
But this is why you preemptively block all of aws and servers like them.
CAPTCHA are used to separate real people from automated scripts like here so all of your registered users are real human beings.
If this has automated CAPTCHAs, there will be work into making much more difficult CAPTCHas. I don't know if this is sustainable. I remember 2 different sites where I havent been able to solve their captchas
First thing, it's just a suggestion. Don't use duplicate account to reply to the comments. Do it from your original account.
I think what the user "chdir" means by wrong is that you're not honoring "robots.txt" and how do you account for ethical scraping ( eg. running 50 concurrent connection on a single website and overloading the website, technically its DoS attack)
Websites use captchas & IP based limits to prevent abuse of their resources & make it harder for copycats to mirror their data. There are often cases where copycats outrank original content in search rankings. (see this example : https://news.ycombinator.com/item?id=10103545 ).
If I were a content owner/producer and I see automated scraping from IP addresses owned by Cloudscrape that violate the ToS, I would sadly treat the entire pool of IPs as violators (even though some might be genuine users who are respecting the limits).
I'd like to know what's a legitimate use case of auto-resolving captchas and IP rotation other than circumventing limits imposed by webmaster.
There's already a tool to stop "copycats". It's called copyright (and for inventions, patents). You can and should use that to enforce your rights to your IP. It's not too hard to start issuing DMCA requests, and it's not even that expensive to have a lawyer do it if you're making money. It doesn't matter whether the illegal copy is obtained by a bot or a human.
While I agree that captchas and IP blocks can be employed by target sites, I don't agree that it should be illegal to circumvent them. I also don't agree that it's necessarily unethical (though in some cases, it may be). If you have public information posted on the public web, I don't think you have the right to mandate that it only be accessed by certain tools. You should plan and expect that it will be accessed by every tool capable of doing so.
If something is disrupting your business by "clogging the tubes" or whatever, that's another thing, and they can be held liable for that. But it doesn't matter that they clogged the tubes with one type of program or another; what matters is that the tubes were clogged by their actions, and that's the part that should be focused on in the subsequent legal proceedings. The specific tool or tools used to clog the tubes is at most a tangential curiosity. We don't want to make certain programs illegal.
Maybe we need a new amendment with "the right to bear code". We do not want to get down a rabbit hole where certain programs are legal and certain programs are not (at least not anymore than we already are with the DMCA et al). Down with code control!
I agree. It's ironic they protect their page so aggressively against scraping, being the biggest scraper in the universe at the same time.
Often they ignore robots.txt also, they still continue scraping, sometimes they even publish things in their SERPS.
From Google's FAQ:
However, robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. In this case, you should not disallow the page in robots.txt, because the page must be crawled in order for the tag to be seen and obeyed.
Thank your for the kind words :) While you're right that we do run a headless browser-ish thing what sets us apart from most of our competitors, other than the point-and-click approach is that we autodetect everything that's going on in the browser - which also means that in most cases you dont need to know what's going on. What this effectively means is that you'll often spend no time reverse-engineering and be able to scrape even wildly complex javascript-heavy sites in minutes instead of hours - and have them be a lot more stable than they would otherwise since there's no "Wait for 5 seconds" which will only be enough 95% of the time.
We see a lot of our clients manage to do what they need done - even daily scrapes - for as little as $29 / month since scraping a news site daily will often take up no more than a few minutes.
Absolutely. People will always tell you they want everything for either practically or actually free. What they say they'll pay and what they'll actually pay are usually two very different things, assuming you provide something that can't be easily replaced.
Thanks for the link to my service! (PhantomJsCloud)
FYI, I have a public preview of our new version, totally free. Check it out if you find this kind of service interesting! http://stage2.phantomjscloud.com
I built a little Swift/iOS "search engine". It contains about 2200 urls. I only search page titles and tags, that I've manually added. What would be my best option for crawling the links and allowing the search to include the text of each page?
Diffbot offers a series of awesome automatic APIs for data extraction -- no setting up manual rules, just provide a URL you want to extract data from and they'll visually process and extract automatically. Also provide a crawler and bulk extractor, and a free trial -- www.diffbot.com
It's great the see discussions going on here - would like to tie a few comments to the questions of ethical aspects of web scraping:
As some has pointed out scraping is not exactly a new thing and a lot of the biggest sites out there are built on the basis of web scraping or crawling. We provide a tool and expect you use that tool while abiding the law - and if not we will of course shut your account down immediately. Breaking the law includes violating copyrights and performing DDoS attacks (Although they will be rather small attacks since even 50 concurrent agents is no big deal for most websites).
We consider ourselves good netizens. We wish nothing more than to provide a good, easily accessible and safe tool for extracting valuable information from the internet, be it for a price comparison site in a market that lacks transparency, business intelligence for your company to make informed and wiser decisions, or a PhD project that requires access to millions of data points available online in unstructured form.
Additionally if you feel we're providing services that has ill-intent - we are not providing any services (Captcha and proxy rotation) that anyone with a bit of programming skill can not easily use in their own software. The main difference is that we are actively improving and focusing not only on making a good experience for our users - but also on minimizing the impact on the sites being scraped. This involves several things like automated throttling and slow-site detection, request caching, and blocking requests to services such as google analytics - to not interfere with site owners stats.
The problem is "the law" is murky on web scraping. For example, did you know that even if your users are only extracting non-copyrighted (even non-copyrightable) data from a page, a judge once ruled that the act of storing the entire page in RAM constituted copyright infringement, since it contained some copyrighted elements that were immediately disposed of after extraction (like the company's logo)? This was Ticketmaster v. RMG Technologies, and it was used against Power Ventures in Facebook's case against them.
Contrast with Feist Publications, Inc. v. Rural Telephone Service Co., where it was ruled that it was legal to copy data from a phonebook and republish, since it was non-copyrightable factual data.
There are several other ridiculous early rulings that were made while the internet was still coming of age, and I think before many judges really understood the way it worked. Recent cases have been bucking these precedents, but you can still get the book thrown at you based on those rulings.
Read about 3Taps and please understand that you will be sued, as they were, unless you fold the moment you get a C&D, which would make your site fairly useless.
Google and all other search engines are illegal in the US in most cases. They just don't get in trouble for most of their activity because people usually want to be on Google. If you end up collecting data in a way that someone doesn't like, things won't go so well for you. See Facebook Inc. v. Power Ventures, Inc.. That guy got raked over the coals; I'm sure Facebook was trying to make an example of him.
Data portability is a threat to the business model of many web incumbents, and that means they want scraping, a critical tool for ensuring that portability, to remain in a nebulous grey area; this allows them to use it for their own purposes (which they often do) and also to try to block people who are using data found on their platform in a way they don't like. This basically results in the bigger company getting their way, because only other multi-billion dollar companies really have the resources to fight against the army of $1k/hr lawyers that public companies hire to try to enforce their opinions on upstarts.
What we really need is serious internet law reform that favors a fair and open platform. Unfortunately, whenever we hear about "internet law reform", it's skewed to the interests of the megacorps who want more tools to shut down innovators that may threaten their business models, not toward creating an open and fair environment for innovation.
Consider, for instance, how ridiculous it would be if every time you opened a book one of the title pages contained a "Terms of Reading" that bound you not to use the information in the book, even the non-copyrightable information, in any way that the book's publisher didn't like, required you to only read the book using the publisher's approved reading methods (perhaps only Oakleys and Ray-bans are publisher-approved eyeglasses, only Herman-Miller publisher-approved seating, and only GE bulbs publisher-approved lighting), required you to agree that you'd never sue the publisher in court but always use private arbitrators that the publisher can easily, even implicitly, buy off, and so forth.
Consider the viability of the argument that you committed copyright infringement by looking at the pages of the book when the author didn't want you to, because the reflection of the content on your eyes constituted an illegal copy.
These things would get laughed out of court, but the digital equivalent is frequently upheld when it comes to online activity.
I think eventually things will stabilize and scraping non-copyrighted data will unambiguously not be a crime, but unfortunately, I think it may still be a few more decades until that happens. I really hope your company is able and willing to help us set the right precedents by committing the tens of millions...
Saying 50 concurrent agents is no big deal for most websites is kinda flippant. It all adds up and not every popular website can handle loads of traffic. By using a headless browser to gain a proper version of a webpage, you also strain a webserver with serving static files costing them bandwidth as well.
Looks good. I've used WebScraper.io (running in Chrome on AWS EC2 instances) before for many projects. It's quite poerwful and free. http://webscraper.io
This is just what I was looking for! Have you looked at the website? You get free webs craping hours every month and start out with 20 hours. That will take me som time to consume. http://cloudscrape.com/
108 comments
[ 3.6 ms ] story [ 168 ms ] threadhttp://codecondo.com/web-scraping-tools-extracting-data/
What you put online is public. Treat the web the same as a public bulletin board. Everything you post will be readable by everyone. Trying to prevent this is pointless.
What's the difference between a machine scraper and a human scraper? Humans use machines to access websites. All your users are machines. And behind every machine is a human.
If you don't want someone to access all your data in a short time-frame, then don't give it all away in a short time-frame.
See https://en.wikipedia.org/wiki/Web_scraping#Legal_issues
If I just download a site – for local use, or similar cases – it’s not an issue.
And if I provide a scraper where the user customly has to enter the URL, it’s also technically just a very advanced webproxy, and also okay.
Think about it from the point of view of the site being scraped. Suddenly your webserver is being slammed with traffic that's way beyond what any human would cause. It's costing you bandwidth and compute resources. You have every right to say no, scraping is not allowed (in your Terms of Service). It's then up to a court to decide if those Terms of Service are legally binding. Some courts have decided that yes, they are.
Since you're clearly not a content creator, and have decided since you didn't work on it it has no value, but putting something online does not make content public domain.
Google scraps webpages as a core competency.
I might just want to train a neural network on extracting data out of websites, and, for that usecase, wget -m every website I can find.
How I use the data you present to me, as long as I don’t republish it, is my decision. If you give me a license to read it, you also give me the license to copy it onto up to 7 different media at the same time, and to show it to up to 7 friends at the same time.
My site, my robots.txt - if the scrapers do not obey, then I'll block them.
There are ways to force authorization, if you don’t use them, your fault.
If I just use TOR or VPNs or similar to get around your block, also, again, your issue. Unless you require authorization, your service is, by law, public.
And if I want to store the site offline for later reading, or read it now in a browser, is nothing that you have to decide.
No, it's not "by law" public, or could you cite a few laws for that, both US and EU. Would appreciate that.
HTTP directly says you should use authorization to make stuff non-public.
And third-party content scraping tools are NOT violating copyright law, as they do not redistribute it freely.
Otherwise services like Opera Turbo – which scrapes a website, removes cruft, compresses data, and sends it to your phone – would also break copyright law, and there is a nice exception for this case, under US law, it’s covered under fair use.
(Unless the scraping service makes your content accessible publicly – and even then, fair use applies usually, as in the case of pure archival like archive.org or archive.is)
Are you a lawyer? I am not - anyway, I can't really agree with that falls under fair use.
If I charge money to scrape someones site and then hand over the material in whatever form my customer uses - I am bound run into problems sooner or later.
I use to think web scraping was limited to obscure proprietary companies like connotate so I'm glad to see more of these tools becoming available for everyone to utilize and hopefully create something cool.
Downsides were like so many other crawlers/spiders - too much in trying to meet the needs of all. Aiming for the 95% is easy, 95 to 99% though, not so much. Hence all the corpses littered along this path.
Recommendations:
Define solid, real-world examples of your platform in practice and really, no bullshit, what makes you different from every other platform that's 97% polished out there?
You didn't meet my needs because I needed much more in the way of parsing network requests from pages, mostly in handling if/then scenarios. A relatively simple scenario, to ME, and there-in lies the challenge of the crawler market, it's the last 5% that's custom to each scenario that so often delivers 98& of the value.
Suggestion: Identify and use the talent you obviously have to destroy some high value markets. Hint: Prioritize targets that still champion a desktop app at any point in the top 5 in terms of data collection.
Outside of that and far more niche (and subsequently far lower priority, allow for parsing / handling handling network requests and responses via custom app/API instead of just block all, some, or none)
Also, do they respect robots.txt? And how are they going to avoid being blocked by websites that don't want to be scraped? (I guess it would be easy to determine the IP addresses of their scrape robots).
Wouldn't be hard to file abuse@ reports to their hosting provider as well. Just pipe their scrapper IPs into WHOIS.
You may also get sued...
If the site owner wants its data to be available for automated extraction they'd provide an API and can price it to compensate for cost of serving all those bots.
Countless sites depend on web scraping. You can scrape and be a good netizen, the two are not mutually exclusive.
If someone needs to build a search site for real estate etc why couldn't they just scrape the Google search result, filter it (white list domains), extract the actual link and present it? In that case you'l need a Google specific scraper that can be based on open source scraping libraries.
Update: Google actually will IP ban you if it thinks you're a bot trying to scrape for search results.
But they have an API:
https://developers.google.com/web-search/docs/?hl=en
Second, you linked to an api page that clearly says it was deprecated 5 years ago, come on. No one should ever feel bad about scraping Google, considering Google is the world's largest scraper themselves.
Just because you have access to something doesn't give you permission to access or access it in any manner possible.
Nobody is taking anything. If you don't want someone to access your page, then don't respond to their request.
Most scrapers - including this one - advertise how they use multiple servers/locations/ips/etc to get around this.
Even if identification was hard, which is not true because of how HTTP works, it is irrelevant because HTTP doesn't discriminate. If someone does, that is their problem, and should be solved by them, and not a committee or law.
> there's no easy way to always reliably identify the requester
That's the problem: you can't identify the person to block them in the first place.
Robots.txt is actually an explicit signal of intention for reputable search engines but that's all we have today and is easily ignored and does not work with these scrapers or anyone else.
Not sure what your last sentence means.
When you say "That is not how HTTP works" it suggests that your claim is that anything that HTTP allows is ethically OK to do. However that is clearly a ridiculous stance, since a DDoS attack is a stream of valid HTTP requests and that's clearly not OK.
So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK.
I find this an interesting question, because while I would love for protcols to also define ethics, I feel that would be scope creep for the poor protocol designers. There's a wide variety of conduct and ethics questions that a protocol cannot address.
Where I myself draw the line is at protocol behavior intentionally designed to obscure my intentions. For example, sending my requests from a wide variety of IP addresses is behavior that is specifically designed to obscure where I'm coming from; my only intent in doing so would be to circumvent the intent of the serving machine from providing lots of content to a single requestor. At that point I'm engaging in deceptive behavior; I've crossed an ethical line.
That wasn't a response made to your comment, and you are mixing two different arguments there. You guess in not correct.
So I'm left wondering what your argument actually is for why unwelcome scraping is ethically OK.
I never even suggested such an argument.
The behavior you described in the last paragraph is only deceptive from the eyes of an information and privacy surveillant state actor. Anonymity is not unethical, it is a human right.
Services like this one actually make scraper-related unavailability, which IMO is already greatly exaggerated, less likely, since there will be fewer amateurs trying to write their own bots and accidentally breaking things.
To the extent that a scraper harms the other business, the scraping company can be held civilly liable on several accounts without specifically bringing scraping as a practice into the picture. All that matters is that they damaged the target site's ability to operate, not that they were saving [portions of] the pages (that'd be a separate copyright claim, unrelated to the disruption of service).
Web is for people from people, not solely for company(s) financial interests.
Intentions matter - on both sides. This is what most of the legal framework of the entire world is based on. You can disagree with that but again the ability to do something doesn't grant permission to do it. You're saying the solution to that is to remove the ability, but I don't see how that's realistic.
What it has to do with it is that putting an HTTP server on the public Web signals the intention to serve up resources to anyone who sends an HTTP request. Any restrictions to this default must be implemented explicitly on top of the default.
Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins.
So yes, intentions matter. The question is how we learn about them. Sometimes the choice of technology implies particular intentions by default.
We already have an explicit signal called robots.txt which major search engines use. The problem is that there's no way to enforce this and there's just very little enforcement against actions on the web in general which is why people can get away with scraping but please don't mistake it for somehow being OK or allowed by the owner of that content. It's just not that simple.
This is legally incorrect. Without any further information or protection measures by the publisher it is legal to access content on a public web server.
">I really can't say this in any other way - just because you can do something doesn't mean you are allowed to"
You are allowed to do everything that is not expressly forbidden by law. Accessing a public web server is not forbidden by law unless the owner takes steps to prevent you from accessing it or at least clearly signals that intention. Terms of service do not constitute an implied contract, so you are not required to read the TOS before accessing a public page.
">We already have an explicit signal called robots.txt"
Exactly. That is part of what I meant when I said that any restrictions had to be implemented on top of the default, which is that everyone who can is allowed to access anything on a public webserver.
[Edit] But initially, this wasn't a thread about legality but about ethics. I think there are unethical reasons to scrape and there are unethical reasons to block scrapers. We simply need to know more about the purpose of any scraping before making a judgement.
2) This directly contradicts your previous comment: "Leaving my stuff lying around does not signal my intention for anyone to take it, unless I let it lying around next to the bins."
3) There is already case law precedent regarding this exact type of publically accessible information not being authorized: http://www.net-security.org/secworld.php?id=14614
I agree that this is about ethics and any scraper that doesnt honor robots.txt and explicitly uses different IPs, user-agents, and other methods to mainly disguise itself as a machine service is unethical in this context.
And you are a lawyer?
>This directly contradicts your previous comment
Absolutely not. The default intention of putting an HTTP server online is not "letting stuff lying around", it is publishing stuff. And yes, the default can be overruled in various ways.
>There is already case law precedent regarding this exact type of publically accessible information not being authorized
You're grasping for straws here. In this specific case, it was completely obvious that this information was not supposed to be public. It was an embarrassing security failure that the defendant wanted to expose.
I think we agree on a lot. robots.txt should be honored and scraping in way or for a purpose that negatively impacts the website's viability or business model is unethical. But usually, such purposes are covered by copyright law anyway.
That doesn't mean that it doesn't happen.
I agree with the GP in that public content is fair game. How do you thing google works?
Copyright governs how the content is used, including distribution. The reason people who download videos are not liable is because you have to download the complete content to see the copyright. File sharers have already downloaded the content and are subject to copyright. Bots that scrape can interpret meta tags in the header of the dom, which is why scraping and violating copyright is unethical.
In terms of copyright, what matters is what you do with the scrape. If you scrape a public website for personal use, it's no different from just browsing it for personal use. If you try to republish the content for your own benefit, you'll run afoul of copyright law.
Humans don't read pages at dozens per minute.
Not hard to code for that, especially if your static content like images are served by alternate servers, so you can just focus on your content servers.
The problem I am dealing with lately is massive bot farms where you do not see a repeating IP for an hour. I catch them eventually but it takes a toll.
But this is why you preemptively block all of aws and servers like them.
- Automatic IP rotation
That's just wrong. Is there a website where we can blacklist IP addresses of such violators ?
You haven't explained how is it wrong and why. None of those thing are "wrong" by itself. It is the malicious use, of any tool, that is unethical.
Is there a website where we can blacklist IP addresses of such violators ?
What exactly do you think is being violated here?
If this has automated CAPTCHAs, there will be work into making much more difficult CAPTCHas. I don't know if this is sustainable. I remember 2 different sites where I havent been able to solve their captchas
First thing, it's just a suggestion. Don't use duplicate account to reply to the comments. Do it from your original account.
I think what the user "chdir" means by wrong is that you're not honoring "robots.txt" and how do you account for ethical scraping ( eg. running 50 concurrent connection on a single website and overloading the website, technically its DoS attack)
If I were a content owner/producer and I see automated scraping from IP addresses owned by Cloudscrape that violate the ToS, I would sadly treat the entire pool of IPs as violators (even though some might be genuine users who are respecting the limits).
I'd like to know what's a legitimate use case of auto-resolving captchas and IP rotation other than circumventing limits imposed by webmaster.
P.S. Why the throwaway ?
While I agree that captchas and IP blocks can be employed by target sites, I don't agree that it should be illegal to circumvent them. I also don't agree that it's necessarily unethical (though in some cases, it may be). If you have public information posted on the public web, I don't think you have the right to mandate that it only be accessed by certain tools. You should plan and expect that it will be accessed by every tool capable of doing so.
If something is disrupting your business by "clogging the tubes" or whatever, that's another thing, and they can be held liable for that. But it doesn't matter that they clogged the tubes with one type of program or another; what matters is that the tubes were clogged by their actions, and that's the part that should be focused on in the subsequent legal proceedings. The specific tool or tools used to clog the tubes is at most a tangential curiosity. We don't want to make certain programs illegal.
Maybe we need a new amendment with "the right to bear code". We do not want to get down a rabbit hole where certain programs are legal and certain programs are not (at least not anymore than we already are with the DMCA et al). Down with code control!
Appeal to fear.
No comment.
No being able to determine a right cause doesn't prove a wrong.
Loaded question.
Often they ignore robots.txt also, they still continue scraping, sometimes they even publish things in their SERPS.
From Google's FAQ:
However, robots.txt Disallow does not guarantee that a page will not appear in results: Google may still decide, based on external information such as incoming links, that it is relevant. If you wish to explicitly block a page from being indexed, you should instead use the noindex robots meta tag or X-Robots-Tag HTTP header. In this case, you should not disallow the page in robots.txt, because the page must be crawled in order for the tag to be seen and obeyed.
The tooling and intuitivity is awesome but besides that at the heart doesn't it do what any other headless javascript enabled browser does?
For example: https://phantomjscloud.com/site/pricing.html
Is there a way to get a less feature rich version of this i.e. sans auto-resolve CAPTCHA etc?
Great work none the less though.
We see a lot of our clients manage to do what they need done - even daily scrapes - for as little as $29 / month since scraping a news site daily will often take up no more than a few minutes.
FYI, I have a public preview of our new version, totally free. Check it out if you find this kind of service interesting! http://stage2.phantomjscloud.com
I then wanted to back out, of course there is no way to remove your account.
While I know what scrape/robot/etc does, I don't know how to start using this one after spent some minutes reading through on the website.
http://www.h4labs.com/dev/ios/swift.html
This is a weekend project so I don't want to spend a lot of money on it.
As some has pointed out scraping is not exactly a new thing and a lot of the biggest sites out there are built on the basis of web scraping or crawling. We provide a tool and expect you use that tool while abiding the law - and if not we will of course shut your account down immediately. Breaking the law includes violating copyrights and performing DDoS attacks (Although they will be rather small attacks since even 50 concurrent agents is no big deal for most websites).
We consider ourselves good netizens. We wish nothing more than to provide a good, easily accessible and safe tool for extracting valuable information from the internet, be it for a price comparison site in a market that lacks transparency, business intelligence for your company to make informed and wiser decisions, or a PhD project that requires access to millions of data points available online in unstructured form.
Additionally if you feel we're providing services that has ill-intent - we are not providing any services (Captcha and proxy rotation) that anyone with a bit of programming skill can not easily use in their own software. The main difference is that we are actively improving and focusing not only on making a good experience for our users - but also on minimizing the impact on the sites being scraped. This involves several things like automated throttling and slow-site detection, request caching, and blocking requests to services such as google analytics - to not interfere with site owners stats.
Contrast with Feist Publications, Inc. v. Rural Telephone Service Co., where it was ruled that it was legal to copy data from a phonebook and republish, since it was non-copyrightable factual data.
There are several other ridiculous early rulings that were made while the internet was still coming of age, and I think before many judges really understood the way it worked. Recent cases have been bucking these precedents, but you can still get the book thrown at you based on those rulings.
Read about 3Taps and please understand that you will be sued, as they were, unless you fold the moment you get a C&D, which would make your site fairly useless.
Google and all other search engines are illegal in the US in most cases. They just don't get in trouble for most of their activity because people usually want to be on Google. If you end up collecting data in a way that someone doesn't like, things won't go so well for you. See Facebook Inc. v. Power Ventures, Inc.. That guy got raked over the coals; I'm sure Facebook was trying to make an example of him.
Data portability is a threat to the business model of many web incumbents, and that means they want scraping, a critical tool for ensuring that portability, to remain in a nebulous grey area; this allows them to use it for their own purposes (which they often do) and also to try to block people who are using data found on their platform in a way they don't like. This basically results in the bigger company getting their way, because only other multi-billion dollar companies really have the resources to fight against the army of $1k/hr lawyers that public companies hire to try to enforce their opinions on upstarts.
What we really need is serious internet law reform that favors a fair and open platform. Unfortunately, whenever we hear about "internet law reform", it's skewed to the interests of the megacorps who want more tools to shut down innovators that may threaten their business models, not toward creating an open and fair environment for innovation.
Consider, for instance, how ridiculous it would be if every time you opened a book one of the title pages contained a "Terms of Reading" that bound you not to use the information in the book, even the non-copyrightable information, in any way that the book's publisher didn't like, required you to only read the book using the publisher's approved reading methods (perhaps only Oakleys and Ray-bans are publisher-approved eyeglasses, only Herman-Miller publisher-approved seating, and only GE bulbs publisher-approved lighting), required you to agree that you'd never sue the publisher in court but always use private arbitrators that the publisher can easily, even implicitly, buy off, and so forth.
Consider the viability of the argument that you committed copyright infringement by looking at the pages of the book when the author didn't want you to, because the reflection of the content on your eyes constituted an illegal copy.
These things would get laughed out of court, but the digital equivalent is frequently upheld when it comes to online activity.
I think eventually things will stabilize and scraping non-copyrighted data will unambiguously not be a crime, but unfortunately, I think it may still be a few more decades until that happens. I really hope your company is able and willing to help us set the right precedents by committing the tens of millions...
Thank you.
EDIT: found that in your FAQ:
"Since disclosing IP’s and user agents would allow anyone to identify all traffic coming from our system – we naturally never do."
That is the opposite of being a good netizen and I hope I'll be able to sue you once I find out your services are helping to scrape my content.
2nd EDIT: Found out that you reside in Denmark and therefore in the EU, that makes it way easier then.