The main caveat is that this may violate a site's terms of use and thus website owners may feel called upon to sue you. Depending on circumstances, the legal situation here can be a long story.
Yes, it is complicated. That said, this is partly just because there aren't enough cases - and partly because the law hasn't stabilised (took a century to stabilise after invention of printing press). It isn't clear what rights society should grant yet, for maximising business.
Create an script that scraps proxies and then use those to scrap twitter, use it in a Russian domain, claim 140 characters can't be copyrighted, claim that the tweets are being extracted from third party sites that use the twitter API but lack any kind of TOS and disclaimer; sell API access, profit!
scraping for fun is okay, but if you would like to build a business most people would prefer to abide by the terms of use and still use the API. remember the story of Pad Mapper ?
I've written some in-browser JS to download all my tweets without needing to resort to the API and oAuth nightmare. It is indeed possible to write a client, but not recommended at all...
Web scraping, as fun as it is (and btw, this title again abuses "Fun and Profit"), is not a practice we should encourage. Yes, it's the sort of dirty practice many people do, at one point or another, but it shouldn't be glorified.
I can see site operators being against the practice though, as it (usually):
- generates no ad revenue
- often enables someone else to use data that you struggled to put together, allowing others to profit with no gain for you
- hit's edge cases that were never optimized for (as it does not follow real user access)
Right, and this is why sites like Craigslist explicitly forbid scraping. If the site operators wanted, explicitly, to share their data with you, they would provide an API or give you permission to scrape.
The reality of scraping was really known many years ago. If you're doing if for above-board reasons like for research etc., you'll probably get a pass - if you're doing it in order to profit from someone else's work because you are too lazy to do it yourself, it's probably unethical and you won't get a pass --- these concepts have been around for at least a thousand years or more.
Full Disclosure: I have also scraped data - but only from government websites where the scraped data is explicitly public domain to begin with and APIs were not available.
1. That doesn't address search engines, which are doing it to profit from someone else's work. If you open the door for search engines then how many search engine like things do you give passes to?
2. What if I'm scraping it just for me, because I want a different interface? How many friends can I share that with? Can I open source the program?
3. What if I read a bunch of these sites to do research and write up a story on something about it? Not plagiarizing, just summarizing and providing analysis on craigslist rental prices? What if I do this every day? What if I automate that process? The data is transformed just as much as if I had read it myself and crunched the numbers myself, I made just as many requests to the site as my browser would have.
Concepts that have been around a thousand years or more are not fully applicable. Like the printing press, some things alter the scarcity equation for ideas and data distribution and ownership. Considering how little we've agreed on about print after 500 years I have some doubts that this is as closed an issue as you say.
Good points, but...
1. I do think it addresses search engines because site operators do explicitly give search engines permission to scrape their sites via something called "robots.txt" files otherwise known as the "robots exclusion standard".
2. Like all other scenarios, this one is also likely between you and the site operator. Are you breaking the site's TOU? The answer to that question might help. If you are asking me for the answer to a moral dilemma, I might suggest that you try Shakespeare for some relevant insight to your question(s).
3. See (2).
I believe you are incorrect in your last sentence, and in a number of ways, but feel free to disagree.
Good point. I meant to include a mention of "robots.txt" but I forgot or delete it editing. That's the motivation for number 3. A "robots.txt is the law" philosophy makes some sense to me, but number 3 is an example of a time when I think it falls down. I don't see a distinction between scripting my daily bookmark visits and manually doing it as a meaningful one. What about extensive browser plugins?
This isn't settled legally certainly and it certainly doesn't seem like this is settled ethically either considering the various insane statements that occur when politicians comment on the subject.
Some examples of the specific concepts from a thousand years ago that apply and answer these questions would help me see what you see. I know the basic rules for music sampling and referencing other works when writing and where the line for plagiarism is drawn and the rights for using photography. Don't know the rules for accessing network resources that are open or for using their data.
On #1, you mentioned before that Craigslist disallows scraping, yet unless you are OmniExplorer, it seems scraping is mostly fair game if robots.txt means anything. The robots.txt standard mentions nothing about it being for search engines [2], so there are no special exclusions for search engines specifically.
Additionally, robots.txt is really for automated link traversal, not scrapers in general. If your scraper is initiated by a user, there is no need to follow robots.txt. Not even Google does when the request is user-initiated [3].
From there, the waters just become really murky. Is lynx a scraper because it doesn't render the way most web browsers do? Does it get a pass because it still adheres to web standards? What if a real scraper adheres to web standards? Maybe it is the storage of scraped data that is the issue? What about caches? I could go on, but I'm sure you see what I'm getting at. It's a very complex issue that is not at all understood.
If you don't want your data used by others, don't send it to them.
You explicitly give them permission to have it by going out of your way to install a program on a common port, with a common API, and giving it a directory full of documents to distribute, and not using any form of authentication. The way the web works is that answering is equivalent to granting permission to ask and sending a file is tantamount to granting permission. When you receive a file you don't first receive a permissions document, you receive the file - authentication and contractual obligations come first because there is no later. (This is like the tide, you may not like it but that doesn't mean you can change it, especially not with laws.)
You have many ways to check authentication and legally they can be VERY weak, 1-bit passwords are sufficient, but if you don't restrict access it is open - not just because it's the default, but because it's the technical reality: they didn't hack into your computer to get that file, they asked your document server and it gave it to them!
Robots.txt is a suggestion, for the scraper's benefit! It suggests better links. You're allowed to see the rest (the server sends them to you without a password) but you're unlikely to find good content.
If you're afraid of someone examining data you send them, don't send them the data if they ask. Expecting them to not ask, or once they've received it, to not manipulate it in certain ways because you can't then extract a fee for them doing so is controlling and more-over, doomed to fail.
- Respect robots.txt (as mentioned elsewhere) which will often provide a limited subset of all data available
- Give something in return (potential traffic) for the data they reap.
I fully agree that scraping is great, and do it myself frequently. Site operators do have legitimate concerns in some situations though, and it probably comes from feeling as if they are being 'ripped off' somehow.
No one in their right mind is going to object to incidental scraping for personal use.
However, scraping is often scripted into cron or the like and that data is then used to profit someone else. I'm usually cool with that, but if someone is running a web site and they are dependent upon ad revenue to keep the servers running, I understand objecting to it.
I'm not actually in favor of scraping. But I think it is possibility that needs to be considered on both ends. If a site has valuable info and doesn't provide a decent API, it naturally is going to encourage scraping.
So you're not so hot on the whole search engine thing?
The article does slide into the sketchy side (I've always wanted an excuse to do that client side javascript trick too) but I found it more interesting because of that, these aren't secrets. Maybe if I put my "won't somebody please think of the children" hat on I agree that glorifying using trojan code to potentially ddos someones server to get around rate limits desired by the owners of the server is bad. Adults and especially adult self described hackers should be able to read this without mock outrage, it's interesting and it's happening all the time.
You can't condemn web scraping though, that's the backbone of the services we all depend on for most internet related things. That's the whole point of structured markup and the world wide web itself.
> So you're not so hot on the whole search engine thing?
They scrape to generate links for users to go to the site. That's quite different than scraping for...any other purpose? So it seems. Would you (anyone) argue otherwise? (genuine curiosity).
Does it have to be a search based interface to the indexed data?
Does finding a link to the scrapee have to be the primary purpose of the site (and therefore google would be constantly getting "worse" by this scale)?
So how prominent does the link back have to be for it to be ok?
What about the summarized data from there that search engines are adding these days, so you don't need to leave the google results page to get your answer but the data still comes from some site that you rarely notice the name of?
edit: as to your curiosity, I honestly do not see the line that you see. Unless it's that the link back to the source is required. I don't know that I agree with that but I would understand it, although that gets harder and harder the more you massage your dataset to be useful to users.
They are also using title, description, some snippets from the page and taking a cached version of the site and images you can view without having to visit the site itself. They are also using this data as a product to sell advertising against.
If there wasn't so much benefit for most of all sites to be in search engine indexes you would thinking at least some would object to this scraping.
There would be lots of other scraping that websites want to prevent that takes even less data than this. It just doesn't provide much in return for the website.
Google is even moving into the territory of scraping content to display. Relevant wikipedia snippets are now being displayed on the search page as a side bar. While Wiki probably doesn't care...there are plenty of other sites that would not like Google to scrape the content and display it on the search page.
Yeah, Wikipedia is creative commons so that should be okay?
You are right though I wonder if they have the rights to sports results and weather that they are pulling.
They have even convinced us all to go mark up our page to help them pull stuff like ratings and reviews out.
Wikipedia explicitly allows that kind of thing with CC-BY-SA licenses, and indeed gets substantial funding from companies like answers.com that do it. (Incidentally, answers.com was the only way to see TeX equations on Wikipedia on my Android phone last time I checked, so it's not like they're adding no value.)
From what I understand, Google uses crawled data as a learning set for their translation service. There is no "this phrase was learned from: www.nytimes.com" when I do a translation, so I guess Google is still guilty?
There is tons of data on the govt websites in India and the only way to get to that is by scrapping the websites. Knowing that you can scrap it and be on your way feels very liberating at times. Example: Rates for Indian postal department services. Minutes of parliament houses. (great for building machine translation systems. A lot of research in MT has benefit from the availability of parallel corpus consisting of parliament proceedings in 2 or more languages. Hansard corpus from Canada. European Parliament corpus. No such luck in India.)
The issue with web scraping is that it relies on the scraper to keep up with changes made to the site.
If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.
In most cases, the site doesn't have an api... so we scrape and take the risk that the structure will change. One thing that helps is using tools which give you jquery-like selectors because they give a lot of freedom and are very easy to write/update.
This should be stressed - sites like Facebook do exactly this. Constant changes mean constantly updating your scraper. When it comes to A/B testing? Your scraper needs to intelligent find the data, which might not always be in the same place.
Sidenote: I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. I guess this would be a caching nightmare, though.
I bet you would only need to randomly shuffle between a few alternatives for all of them. You'd need a dedicated effort to work that one out and the cache implications could be managed. No getting around the trade-off of possible page alternatives vs cache nightmare-ness though, and doing that to json apis would get ugly fast.
At least it's easier to code these tricks than to patch a scraper to get around them.
I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping.
In my spare time, I've been playing around with "scrapers" (I like to call them web browsers, personally) that don't even look at markup.
My first attempt used a short list of heuristics that proved to be eerily successful for what I was after. To the point I could throw random websites with similar content (discussion sites, like HN), but vastly dissimilar structures, at it and it would return what I expected about, I'd say, 70% of the time in my tests.
After that, I started introducing some machine learning in an attempt to replicate how I determine what blocks are meaningful. My quick prototype showed mix results, but worked well enough that I feel with some tweaking it could be quite powerful. Sadly, I've become busy with other things and haven't had time to revisit it.
With that, swapping variables and similar techniques to thwart crawlers seems like it would be easily circumvented.
I would be really interested in knowing which heuristics or machine learning techniques produced decent results. That's if I can't convince you to open source the code. I'm working on the same problem at the moment.
We're fine with scrapers and scraping infrastructure, although tubes.io is a very interesting idea.
I'm more interested in what I can do to write fewer scrapers since the content is, at a high level, relatively similar. I've just started with experiments writing "generic" scrapers that try and extract the data without depending on markup. It's going to eventually work well enough but to get the error rate down to an acceptable level is going to take a lot of tweaking and trial and error.
There's a few papers on this, but not much out there. That's why I was interested in someone else working on the same problem in a different space.
> Sidenote: I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. I guess this would be a caching nightmare, though.
These guys do a stellar job on the IP addresses: http://www.hidemyass.com/proxy-list -- the good thing is the data is available for an amazing price.
Other sites I have some across will use large images and css sprites to mask price data.
I write a lot of scrapers for fun, rarely profit, just for the buzz
>The issue with web scraping is that it relies on the scraper to keep up with changes made to the site.
The OP addresses that point. His contention is, there's a lot more pressure on the typical enterprise to keep their public-facing website in tip-top shape than there is to make sure whatever API they've defined is continuing to deliver results properly.
Of course, part of the art of (and fun of) scraping is to see if you can make your scraper robust against mere cosmetic changes to the site.
Of course, part of the art of (and fun of) scraping is to see if you can make your scraper robust against mere cosmetic changes to the site.
I once had to maintain a (legal) scraper and I can tell you there is no fun in making your scraper robust when the website maintainers are doing there best to keep you from scraping there site.
I've seen random class-names and identifiers, switching of DIVs and SPANs (block display). Adding and removing SPANs for nesting/un-nesting elements. And so on.
Ofcourse the site likes to keep the SEO, but most of the time it's easy to keep parts out of context for a scraper.
This is indeed painful. I was scraping the Pirate Bay last year for a school project, their HTML would occasionally change in subtle ways that would break my scraper for hours or days until I noticed it.
when i need to scrap a webpage, i use phpQuery (http://code.google.com/p/phpquery/), it's dead simple if you have experience with jQuery and i get all the benefits of a server-side programming language.
For Perl you can use Web::Query (but don't use the default HTML::TreeBuilder::XPath with it, it's extremely slow - use http://search.cpan.org/dist/HTML-TreeBuilder-LibXML/), or Mojo::DOM (part of Mojolicious: http://mojolicio.us/). Both use the nowdays standard CSS selectors for comfortable handling.
JavaScript can be scraped using WWW::Mechanize::Plugin::JavaScript or even WWW::Mechanize::Firefox).
In the past, I have successfully used HtmlUnit to fulfill my admittedly limited scraping needs.
It runs headless, but it has a virtual head designed to pretend it's a user visting a web application to be be tested for QA purposes. You just program it to go through the motions of a human visting a site to be tested (or scraped). E.g., click here, get some response. For each whatever in the response, click and aggregate the results in your output (to whatever granularity).
Alas, it's in Java. But, if you use JRuby, you can avoid most of the nastiness that implies. (You do need to know Java, but at least you don't have to write Java.)
Hartley, what is your recommended toolkit?
I note you mentioned the problem of dynamically generated content. You develop your plan of attack using the browser plus Chrome Inspector or Firebug. So far, so good. But what if you want to be headless? Then you need something that will generate a DOM as if presenting a real user interface but instead simply returns a reference to the DOM tree that you are free to scan and react to.
Headless: Xvfb on Linux. (Virtual Framebuffer, let's you run apps that require a GUI) You can use one of the many options that include Webkit (like phantom.js, the capybara-webkit gem, or Selenium if you want a real browser like Firefox to do the work)
well, it wasn't due to anything in phantomjs actually -- it was because qt introduced project lighthouse & their qt platform abstraction. project lighthouse was a fork that got integrated into qt 4.8 that phantomjs includes. (you can see the entire qt source tree if you git-clone phantomjs)
Not every site. There is data I would really love to access on Facebook without having to gain specific authorization from the user. It's odd that for most user profiles the most you can extract via the graph API (with no access token) is their name and sex. Whereas I can visit their profile page in the browser, see all sorts of info and latest updates (and not even be friends with them)
Tried scraping Facebook. They have IP blocks and the like.
Do it in JS, client-side with 3 second delays. I have used this to get available data (location, name, status, etc) from the latest 500 available likes of a page I manage
I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques, you're driving a real browser and your requests look real (if you make 10000 requests to index.php and never pull down a single image, you might look a bit suspicious). There's a bit more overhead, but micro instances on EC2 can easily run 2 or 3 Selenium sessions at the same time, and at 0.3 cents per hour for spot instances, you can have 200-300 browsers going for 30-50 cents/hour.
My 2c about scraping - when you try to obtain data from large websites, always go for javascript content. Pages like Newegg or Amazon * may change html outline very often even without a single alteration to the front-user and even your smartest regex can have a brain fart. In contrast, even when site gets major overhaul, most likely old javascript will be left in place with all up to date variables, because engineers will be concerned of removing that code not to break some functionality .
* given you have rights to scrap.
not that there are no tools to debug the site; but I found websites like mentioned plus youtube, and bunch others just not fiddling too much with js.
In regards to the javascript problem. I'd suggest checking out the mobile versions of the sites first before you hop to a weighty solution like Selenium. Could be a very simple solution to the problem :)
I recently built a twitter bot that did some scraping and posting. I beat my head against a wall for a couple of hours trying to find a good tool to deal with all the javascript driven stuff. I happened to get an update on my phone when it dawned on me that the mobile.twitter site is, for the most part, simple html stuff. Once I realized that, I was able to programatically log into my account with no problems, and the rest of twitter was unlocked for me. I could scrape and post to (almost) my heart content.
However, there were a few very big problems. Which makes me feel that scraping is not the way to go about things. I certainly wouldn't build a service based around scraping a particular site's data.
When I had my twitter bot operational, I would get blocked from twitter for hours at a time. It seems anytime I hit their servers too hard, or crossed some threshold, I would be locked out. I'm assuming it was some kind of IP level ban, because I wasn't even able to access the site from an actual browser.
I was able to deal with the setback by setting up a script to repeatedly check its access the site, and then relaunch the scraper upon discovering access, but the solution was just a band-aid. That would translate to significant downtime if I was running a service with counted on access to their data. The ban-hammer is too easily laid down.
Finally, just as a word of caution, I'd warn prospective scrapers to be careful of just who you scrape. I've inadvertently "DDoS'd" a site when a multiprocessed script got away from me. It spawned 1000+ instances of this particular request, all of which were doing their best to beat the bejesus out of this small websites servers. The site ended up going down for a couple of hours; I assume because of a bandwidth cap or something.
So, my point being, scraping is cool, but (1) I'm unsure if I agree with relying on it over a proper API, and (2) with great power comes great responsibility! Be nice to smaller guys, and don't punish their servers to bad.
My scraping is part of a transactional B2B service, not a high traffic social or B2C thing, so it's a different set of problems than those who want their hands on Twitter data. These are Fortune 100's, so if I can bring down their site, they have bigger problems. :-)
I wouldn't make that assumption. Do check return codes and load times, and back off if you see issues. If these sites are business partners/suppliers you have a lot to lose if things go wrong. It's worth it to develop your relationship with the business owners of the services you're touching in the correspondent organizations. And do set a User-Agent string that declares who you are and provides a link for information; if you are doing business with them, it should be on a basis of honesty.
For my twitter bot I extracted the xauth keys from twitter's official Mac client(s) (Tweetie 1 and 2 have different keys) and used those to access the API. To twitter the bot looked like the official client and they couldn't ban it without banning their official clients.
And XAuth made account creation and log in a breeze as there was no need for OAuth tokens - username/password was enough.
But you're not always as lucky as that and many websites are heavily JS driven. For Reddit I had to resort to selenium.
I think his comment was quite appropriate, and did not feel there was an implication that he thought anyone was dumb for not having known about the alternative approach to getting Reddit data.
Often times programmers and the managers that drive them are way too quick to get going building or solving something with brute force. If they would just be patient and stop for a moment. Spending even a mere 30 minutes extra doing your homework on a problem can save hours or days in dev time.
Saying that that was a proof of spending 15 minutes to search about Reddit API would save his time implied that kybernetyk didn't do that research.
But kybernetyk already said he did the research before and that Reddit's API is not good enough for his requirement.
So this is not the case that 15 minutes of research will save the time. And his comment meant he assumed kybernetyk didn't do research, i.e. being dumb for not doing search.
I did not assume that kybernetyk was dumb or anything, I simply chuckled and thought to myself ouch haven't I done similar mistake before?! Please don't assume the worst when reading someone's comment.
If you're using ruby, I've found watir (http://watir.com/) to be very nice to use. There might be better alternatives now but it made my life easier when I had to scrape a bunch of our supplier's crappy B2B sites that required JavaScript.
Not quite ready for prime time but I am working on a project that makes it really easy to grab content from any site using a point and click interface no xpaths selectors or regex.
You enter the url you want to capture data from, it gets loaded in an iframe, you click on the texts you need and set a schedule to receive updates and how(email/twitter dm) that's it.
It supports javascript driven content and can handle practically any website.
In my experience, you seldom need a full browser to extract data from javascript-heavy sites. You often can make your way with a little bit of reverse engineering, starting from a traffic capture and looking after parameters you dont understand in the HTML/JS code. Usually, there is nothing hidden. Though, when they're effectively trying to make your life harder with JS, it is easily solved by feeding a JS interpreter (like python spidermonkey) the offending algorithm.
Depending on your use case, headless may be simpler, but it has also many drawbacks that don't show at first, the main being that they're not simple to drive from remote processes as queue-consuming devices.
The article suggests BeautifulSoup as a parsing library for python. If I'm not mistaken BeautifulSoup is not actively maintained anymore, and other cleaner and faster solutions exists, like lxml.html. Ian Bicking made a good article on that topic : http://blog.ianbicking.org/2008/03/30/python-html-parser-per...
I hear it recommended the most among Pythonistas, and it's plenty clean and fast for my use. But if you're skeptical, I'd still look for a more up to date benchmark (or run your own) rather than rely on results from >4 years ago.
Looks like things have changed since the last time I checked. Thank you for pointing this out. Next time I'll check y facts twice before posting.
Still, lxml being basically a binding to libxml2 the performance comparison of the two libs should still hold. I heard it recommended too, in a python talk about scraping like 1 or 2 (at most) years ago.
BeautifulSoup may still be better for parsing broken documents, though I never had problems with lxml while using it on a very large variety of sites.
You can use BeautifulSoup with lxml if you like, although I just use the HTMLParser in lxml these days and don't use BeautifulSoup any more. It seems to work a little better, at least for my uses.
I love HTML scraping.
But Javascript???...The juiciest data sets these days are increasingly in JS.
For the love of me i can't get around scraping JS :(
I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share??
I would be eternally grateful.
I've been doing this on a site that is 100% Javascript-driven for over a year, very successfully.
It's really no different than hitting a static site with Selenium. Figuring out the proper XPath to use is often the biggest challenge: Chrome Developer tools help immensely. Also, you need to watch for delays in JS rendering, so put a lot of pauses in your scripts.
It's of course slow, so if you want to distribute it across several machines, use Selenium Grid or a queue system (SQS, Resque, etc). Setup Xvfb to run on headless Linux instances.
Yippie...that worked. Thanks a ton for sharing that code snippet.
Am unfortunately more of an analyst. Have only just started picking up coding for data analysis. JS scraping was one area that I always had difficulty with.
Not anymore :D
If you can't get the data from the endpoints the javascript hits then write your scraper in javascript and have it run in a headless browser, and it's the webkit engine so most sites test their site against it heavily.
Either pull the data out of the javascript objects or trigger your extraction from the html by attaching to the events in the javascript.
I've started playing with zombie.js recently as well - much lighter and faster than the ones that instrument a completely full browser. But has a full Javascript engine.
We're also trying it for integration tests, as it is much quicker than Phantom or Selenium. Even there, where we control the standards-compliant site, it isn't quite good enough yet.
Would love to see more people helping make it so, though!
zombie.js is not a full browser. It's a poor emulation using jsdom as its backing. http://zombie.labnotes.org/guts Beware, for some applications, jsdom is super buggy.
In general, any Web functional testing tool can be used as a scraper. Scraping and testing are extremely similar. In both cases, one uses XPath or (hopefully) CSS to locate an element and examine certain aspects of that element's state. A scraper is only different from a functional test in that a scraper is focused only on the state of nodes (potentially) containing human-readable content. That, and a scraper saves the data it collects rather than discarding all data at the end of a test run.
Here's a very old Selenium 1.0 example that scrapes the full, rendered HTML of a page. After performing a scrape like this, I would then feed the HTML into a parser such as Nokogiri http://snipplr.com/view/7906/rendered-wget-with-selenium/
Yes, but if you want the DOM you would have to use something like webkit. So something like pyphantomjs might hit the right spot. It's a python re-implementation of phantomjs.
Another issue not covered: file downloads. Let's say you have a process that creates a dynamic image, or logs in and downloads dynamic PDFs. Even Selenium can't handle this (the download dialog is an OS-level feature). At one point I was able to get Chrome to auto-download in Selenium, but had zero control over filename and where it was saving. I ended up using iMacros (the pay version) to drive this (using Windows instances: their Linux version is very immature comparably).
Mechanize doesn't really have a proper doc, but just about everything you'd need could be figured out from the very lengthy examples page on their site.
Playing with it now, and while it seems to hit my download need, I can't seem to get it to play nice with sites that are JavaScript dependent. Am I missing something, or is there a way to plugin an underlying WebKit engine?
PhantomJS is capable of downloading binary content from js dependent sites but it is a journey to get it working as it is not an out-of-the-box feature. Instead use CasperJS to drive Phantom and get a ton of snazzy features including simple binary downloads. Happy scraping!
(shameless plug) I can scrape asynchronously, anonymously, with JS wizardry, and feed it into your defined models in your MVC (e.g. Django). But! I need to get to a hacker conference on the other side of the world (29c3). Any other time of year, I'd just drop a tutorial. See profile if you'd like to help me with a consulting gig.
EDIT: Knowledge isn't zero-sum. Here's an overview of a kick-ass way to spider/scrape:
I use Scrapy to spider asynchronously. When I define the crawler bot as an object, if the site contains complicated stuff (stateful forms or javascript) I usually create methods that involve importing either Mechanize or QtWebKit. Xpath selectors are also useful for the ability to not have to specify the entire XML tree from trunk to leaf. I then import pre-existing Django models from a site I want the data to go into and write to the DB. At this point you usually have to convert some types.
I find Scrapy cleaner and more like a pipeline so it seems to produce less 'side effect kludge' than other scraping methods (if anybody has seen a complex Beautiful Soup + Mechanize scraper you know what I mean by 'side effect kludge'). It can also act as a server to return json.
Being asynchronous, you can do crazy req/s.
I will leave out how to do all this through Tor because I don't want the Tor network being abused but am happy to talk about it one on one if your interest is beyond spamming the web.
Through this + a couple of unmentioned tricks, it's possible to get insane data, so much so it crosses over into security research & could be used for pen-testing.
The title makes it sound as if there is going to be some discussion of how the OP has made web scraping profitable, but this seems to have been left to the reader's imagination.
Otherwise, great article! I agree that BeautifulSoup is a great tool for this.
This is a shameless plug but I've created a service that aims to help with a lot of the issues that OP describes such as rate limiting, JS and scaling. It's a bit like Heroku for web scraping and automation. It's still in beta but if anyone is interested then check out http://tubes.io.
This illustrates the significant difference between the use-cases of "web APIs" and conventional APIs, that the former are more like a database CRUD (including REST), rather than a request for computation. They (usually) are an alternative interface to a website (a GUI), and that's how most websites are used. e.g. an API for HN would allow story/comment retrieval, voting, submission, commenting.
They could be used for computation, but (mostly) aren't.
There is just one major trouble with not needing stinking API. You can not POST as a possible client without requiring them to give their password to you, which actually would give you full access to their account instead of limited access with API.
To all HN: All this being said, how do we prevent our sites from being scraped in this way? What can you not get around, and what are the potential uses for an 'unscrapeable' site to your mind.
if you don't want something to be scrapped, don't publish it on the internet. scrapping prevention reminds me of blocking right-click and other ridiculous solutions back in the day. hey, if I can view it, it means the data reached my end point.
Yah, I don't understand why people make things so complicated once Javascript gets involved. Just inspect the XHR traffic to your browser ("Network" tab in Web Inspector, Firebug, etc) as you update the information on the page. You'll quickly discover what are essentially undocumented APIs returning the data used to generate the page. You don't need to use or even read through the Javascript that's calling them, you just need to figure out what parameters and cookies are being sent, and tweak those as you wish.
You might have to spoof the Referer header so that it thinks the request is still coming from their website.
The problem with scraping instead of using the API is that when the website makes even a slight change to their markup it breaks your code. I have had that experience and it's a living hell. I can say it's not worth it to scrap when there is an API available.
I had to do some scraping of a rather Javascript-heavy site last year - I found the entire process was made almost trivial using Ruby and Nokugiri. Particularly relevant for a non-uber-programmer like me, it's simple to use, as well as powerful.
If it's not automated and a fewer times, I will prefer IMacro to perform tasks on my behalf. The best part of it that you can integrate a Db to record your desired data.
175 comments
[ 2.0 ms ] story [ 202 ms ] threadThe main caveat is that this may violate a site's terms of use and thus website owners may feel called upon to sue you. Depending on circumstances, the legal situation here can be a long story.
My take on it, from ScraperWiki's point of view: http://blog.scraperwiki.com/2012/04/02/is-scraping-legal/
scraping for fun is okay, but if you would like to build a business most people would prefer to abide by the terms of use and still use the API. remember the story of Pad Mapper ?
Web scraping, as fun as it is (and btw, this title again abuses "Fun and Profit"), is not a practice we should encourage. Yes, it's the sort of dirty practice many people do, at one point or another, but it shouldn't be glorified.
I can see site operators being against the practice though, as it (usually): - generates no ad revenue - often enables someone else to use data that you struggled to put together, allowing others to profit with no gain for you - hit's edge cases that were never optimized for (as it does not follow real user access)
The reality of scraping was really known many years ago. If you're doing if for above-board reasons like for research etc., you'll probably get a pass - if you're doing it in order to profit from someone else's work because you are too lazy to do it yourself, it's probably unethical and you won't get a pass --- these concepts have been around for at least a thousand years or more.
Full Disclosure: I have also scraped data - but only from government websites where the scraped data is explicitly public domain to begin with and APIs were not available.
2. What if I'm scraping it just for me, because I want a different interface? How many friends can I share that with? Can I open source the program?
3. What if I read a bunch of these sites to do research and write up a story on something about it? Not plagiarizing, just summarizing and providing analysis on craigslist rental prices? What if I do this every day? What if I automate that process? The data is transformed just as much as if I had read it myself and crunched the numbers myself, I made just as many requests to the site as my browser would have.
Concepts that have been around a thousand years or more are not fully applicable. Like the printing press, some things alter the scarcity equation for ideas and data distribution and ownership. Considering how little we've agreed on about print after 500 years I have some doubts that this is as closed an issue as you say.
This isn't settled legally certainly and it certainly doesn't seem like this is settled ethically either considering the various insane statements that occur when politicians comment on the subject.
Some examples of the specific concepts from a thousand years ago that apply and answer these questions would help me see what you see. I know the basic rules for music sampling and referencing other works when writing and where the line for plagiarism is drawn and the rights for using photography. Don't know the rules for accessing network resources that are open or for using their data.
Additionally, robots.txt is really for automated link traversal, not scrapers in general. If your scraper is initiated by a user, there is no need to follow robots.txt. Not even Google does when the request is user-initiated [3].
From there, the waters just become really murky. Is lynx a scraper because it doesn't render the way most web browsers do? Does it get a pass because it still adheres to web standards? What if a real scraper adheres to web standards? Maybe it is the storage of scraped data that is the issue? What about caches? I could go on, but I'm sure you see what I'm getting at. It's a very complex issue that is not at all understood.
[1] http://www.craigslist.org/robots.txt
[2] http://www.robotstxt.org/robotstxt.html
[3] http://support.google.com/webmasters/bin/answer.py?hl=en&...
You explicitly give them permission to have it by going out of your way to install a program on a common port, with a common API, and giving it a directory full of documents to distribute, and not using any form of authentication. The way the web works is that answering is equivalent to granting permission to ask and sending a file is tantamount to granting permission. When you receive a file you don't first receive a permissions document, you receive the file - authentication and contractual obligations come first because there is no later. (This is like the tide, you may not like it but that doesn't mean you can change it, especially not with laws.)
You have many ways to check authentication and legally they can be VERY weak, 1-bit passwords are sufficient, but if you don't restrict access it is open - not just because it's the default, but because it's the technical reality: they didn't hack into your computer to get that file, they asked your document server and it gave it to them!
Robots.txt is a suggestion, for the scraper's benefit! It suggests better links. You're allowed to see the rest (the server sends them to you without a password) but you're unlikely to find good content.
If you're afraid of someone examining data you send them, don't send them the data if they ask. Expecting them to not ask, or once they've received it, to not manipulate it in certain ways because you can't then extract a fee for them doing so is controlling and more-over, doomed to fail.
- Respect robots.txt (as mentioned elsewhere) which will often provide a limited subset of all data available
- Give something in return (potential traffic) for the data they reap.
I fully agree that scraping is great, and do it myself frequently. Site operators do have legitimate concerns in some situations though, and it probably comes from feeling as if they are being 'ripped off' somehow.
No one in their right mind is going to object to incidental scraping for personal use.
However, scraping is often scripted into cron or the like and that data is then used to profit someone else. I'm usually cool with that, but if someone is running a web site and they are dependent upon ad revenue to keep the servers running, I understand objecting to it.
> No one in their right mind is going to object to incidental scraping for personal use.
It would almost certainly involve stripping ads when re-purposing the content.
I'm not actually in favor of scraping. But I think it is possibility that needs to be considered on both ends. If a site has valuable info and doesn't provide a decent API, it naturally is going to encourage scraping.
And isn't a search engine a kind of scraper?
The article does slide into the sketchy side (I've always wanted an excuse to do that client side javascript trick too) but I found it more interesting because of that, these aren't secrets. Maybe if I put my "won't somebody please think of the children" hat on I agree that glorifying using trojan code to potentially ddos someones server to get around rate limits desired by the owners of the server is bad. Adults and especially adult self described hackers should be able to read this without mock outrage, it's interesting and it's happening all the time.
You can't condemn web scraping though, that's the backbone of the services we all depend on for most internet related things. That's the whole point of structured markup and the world wide web itself.
They scrape to generate links for users to go to the site. That's quite different than scraping for...any other purpose? So it seems. Would you (anyone) argue otherwise? (genuine curiosity).
Does finding a link to the scrapee have to be the primary purpose of the site (and therefore google would be constantly getting "worse" by this scale)?
So how prominent does the link back have to be for it to be ok?
What about the summarized data from there that search engines are adding these days, so you don't need to leave the google results page to get your answer but the data still comes from some site that you rarely notice the name of?
edit: as to your curiosity, I honestly do not see the line that you see. Unless it's that the link back to the source is required. I don't know that I agree with that but I would understand it, although that gets harder and harder the more you massage your dataset to be useful to users.
If there wasn't so much benefit for most of all sites to be in search engine indexes you would thinking at least some would object to this scraping.
There would be lots of other scraping that websites want to prevent that takes even less data than this. It just doesn't provide much in return for the website.
They have even convinced us all to go mark up our page to help them pull stuff like ratings and reviews out.
If a site owner changes the layout or implements a new feature, the programs depending on the scraper immediately fail. This is much less likely to happen when working with official APIs.
Sidenote: I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. I guess this would be a caching nightmare, though.
At least it's easier to code these tricks than to patch a scraper to get around them.
but who cares, no one can beat Xpath :)
In my spare time, I've been playing around with "scrapers" (I like to call them web browsers, personally) that don't even look at markup.
My first attempt used a short list of heuristics that proved to be eerily successful for what I was after. To the point I could throw random websites with similar content (discussion sites, like HN), but vastly dissimilar structures, at it and it would return what I expected about, I'd say, 70% of the time in my tests.
After that, I started introducing some machine learning in an attempt to replicate how I determine what blocks are meaningful. My quick prototype showed mix results, but worked well enough that I feel with some tweaking it could be quite powerful. Sadly, I've become busy with other things and haven't had time to revisit it.
With that, swapping variables and similar techniques to thwart crawlers seems like it would be easily circumvented.
I'm more interested in what I can do to write fewer scrapers since the content is, at a high level, relatively similar. I've just started with experiments writing "generic" scrapers that try and extract the data without depending on markup. It's going to eventually work well enough but to get the error rate down to an acceptable level is going to take a lot of tweaking and trial and error.
There's a few papers on this, but not much out there. That's why I was interested in someone else working on the same problem in a different space.
These guys do a stellar job on the IP addresses: http://www.hidemyass.com/proxy-list -- the good thing is the data is available for an amazing price.
Other sites I have some across will use large images and css sprites to mask price data.
I write a lot of scrapers for fun, rarely profit, just for the buzz
The OP addresses that point. His contention is, there's a lot more pressure on the typical enterprise to keep their public-facing website in tip-top shape than there is to make sure whatever API they've defined is continuing to deliver results properly.
Of course, part of the art of (and fun of) scraping is to see if you can make your scraper robust against mere cosmetic changes to the site.
I once had to maintain a (legal) scraper and I can tell you there is no fun in making your scraper robust when the website maintainers are doing there best to keep you from scraping there site. I've seen random class-names and identifiers, switching of DIVs and SPANs (block display). Adding and removing SPANs for nesting/un-nesting elements. And so on. Ofcourse the site likes to keep the SEO, but most of the time it's easy to keep parts out of context for a scraper.
JavaScript can be scraped using WWW::Mechanize::Plugin::JavaScript or even WWW::Mechanize::Firefox).
In the past, I have successfully used HtmlUnit to fulfill my admittedly limited scraping needs.
It runs headless, but it has a virtual head designed to pretend it's a user visting a web application to be be tested for QA purposes. You just program it to go through the motions of a human visting a site to be tested (or scraped). E.g., click here, get some response. For each whatever in the response, click and aggregate the results in your output (to whatever granularity).
Alas, it's in Java. But, if you use JRuby, you can avoid most of the nastiness that implies. (You do need to know Java, but at least you don't have to write Java.)
Hartley, what is your recommended toolkit?
I note you mentioned the problem of dynamically generated content. You develop your plan of attack using the browser plus Chrome Inspector or Firebug. So far, so good. But what if you want to be headless? Then you need something that will generate a DOM as if presenting a real user interface but instead simply returns a reference to the DOM tree that you are free to scan and react to.
Tried scraping Facebook. They have IP blocks and the like.
My 2c about scraping - when you try to obtain data from large websites, always go for javascript content. Pages like Newegg or Amazon * may change html outline very often even without a single alteration to the front-user and even your smartest regex can have a brain fart. In contrast, even when site gets major overhaul, most likely old javascript will be left in place with all up to date variables, because engineers will be concerned of removing that code not to break some functionality .
* given you have rights to scrap.
not that there are no tools to debug the site; but I found websites like mentioned plus youtube, and bunch others just not fiddling too much with js.
If you're using regex to solve this sort of problem, your code deserves to break, I'm sorry.
Regex for HTML is a bad idea ... http://stackoverflow.com/questions/590747/using-regular-expr...
I recently built a twitter bot that did some scraping and posting. I beat my head against a wall for a couple of hours trying to find a good tool to deal with all the javascript driven stuff. I happened to get an update on my phone when it dawned on me that the mobile.twitter site is, for the most part, simple html stuff. Once I realized that, I was able to programatically log into my account with no problems, and the rest of twitter was unlocked for me. I could scrape and post to (almost) my heart content.
However, there were a few very big problems. Which makes me feel that scraping is not the way to go about things. I certainly wouldn't build a service based around scraping a particular site's data.
When I had my twitter bot operational, I would get blocked from twitter for hours at a time. It seems anytime I hit their servers too hard, or crossed some threshold, I would be locked out. I'm assuming it was some kind of IP level ban, because I wasn't even able to access the site from an actual browser.
I was able to deal with the setback by setting up a script to repeatedly check its access the site, and then relaunch the scraper upon discovering access, but the solution was just a band-aid. That would translate to significant downtime if I was running a service with counted on access to their data. The ban-hammer is too easily laid down.
Finally, just as a word of caution, I'd warn prospective scrapers to be careful of just who you scrape. I've inadvertently "DDoS'd" a site when a multiprocessed script got away from me. It spawned 1000+ instances of this particular request, all of which were doing their best to beat the bejesus out of this small websites servers. The site ended up going down for a couple of hours; I assume because of a bandwidth cap or something.
So, my point being, scraping is cool, but (1) I'm unsure if I agree with relying on it over a proper API, and (2) with great power comes great responsibility! Be nice to smaller guys, and don't punish their servers to bad.
And XAuth made account creation and log in a breeze as there was no need for OAuth tokens - username/password was enough.
But you're not always as lucky as that and many websites are heavily JS driven. For Reddit I had to resort to selenium.
Often times programmers and the managers that drive them are way too quick to get going building or solving something with brute force. If they would just be patient and stop for a moment. Spending even a mere 30 minutes extra doing your homework on a problem can save hours or days in dev time.
But kybernetyk already said he did the research before and that Reddit's API is not good enough for his requirement.
So this is not the case that 15 minutes of research will save the time. And his comment meant he assumed kybernetyk didn't do research, i.e. being dumb for not doing search.
I did not assume that kybernetyk was dumb or anything, I simply chuckled and thought to myself ouch haven't I done similar mistake before?! Please don't assume the worst when reading someone's comment.
You enter the url you want to capture data from, it gets loaded in an iframe, you click on the texts you need and set a schedule to receive updates and how(email/twitter dm) that's it.
It supports javascript driven content and can handle practically any website.
http://www.followwww.com
Depending on your use case, headless may be simpler, but it has also many drawbacks that don't show at first, the main being that they're not simple to drive from remote processes as queue-consuming devices.
The article suggests BeautifulSoup as a parsing library for python. If I'm not mistaken BeautifulSoup is not actively maintained anymore, and other cleaner and faster solutions exists, like lxml.html. Ian Bicking made a good article on that topic : http://blog.ianbicking.org/2008/03/30/python-html-parser-per...
http://www.crummy.com/software/BeautifulSoup/
I hear it recommended the most among Pythonistas, and it's plenty clean and fast for my use. But if you're skeptical, I'd still look for a more up to date benchmark (or run your own) rather than rely on results from >4 years ago.
Still, lxml being basically a binding to libxml2 the performance comparison of the two libs should still hold. I heard it recommended too, in a python talk about scraping like 1 or 2 (at most) years ago.
BeautifulSoup may still be better for parsing broken documents, though I never had problems with lxml while using it on a very large variety of sites.
http://lxml.de/elementsoup.html
I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.
I've been doing this on a site that is 100% Javascript-driven for over a year, very successfully.
It's really no different than hitting a static site with Selenium. Figuring out the proper XPath to use is often the biggest challenge: Chrome Developer tools help immensely. Also, you need to watch for delays in JS rendering, so put a lot of pauses in your scripts.
It's of course slow, so if you want to distribute it across several machines, use Selenium Grid or a queue system (SQS, Resque, etc). Setup Xvfb to run on headless Linux instances.
first install Xvfb and pyvirtualdisplay then try this snippet https://gist.github.com/4243582
selenium is great, it can even wait for ajax requests to finish (see WebDriverWait) ..
You really don't need xvfb anymore. Use xserver-xorg-video-dummy.
If you can't get the data from the endpoints the javascript hits then write your scraper in javascript and have it run in a headless browser, and it's the webkit engine so most sites test their site against it heavily.
Either pull the data out of the javascript objects or trigger your extraction from the html by attaching to the events in the javascript.
I worry that it's not going to replicate a real browser accurately enough, but I'm excited to try it out a bit.
We're also trying it for integration tests, as it is much quicker than Phantom or Selenium. Even there, where we control the standards-compliant site, it isn't quite good enough yet.
Would love to see more people helping make it so, though!
Here's a very old Selenium 1.0 example that scrapes the full, rendered HTML of a page. After performing a scrape like this, I would then feed the HTML into a parser such as Nokogiri http://snipplr.com/view/7906/rendered-wget-with-selenium/
https://github.com/kanzure/pyphantomjs
[0]http://jeanphix.me/Ghost.py/
If you are planning to use phantomjs, import sh and it's commandline all the way to payday :D
e.g.
Mechanize doesn't really have a proper doc, but just about everything you'd need could be figured out from the very lengthy examples page on their site.I used it to crawl for freelance web dev gigs, but it can be re-purposed to do anything.
EDIT: Knowledge isn't zero-sum. Here's an overview of a kick-ass way to spider/scrape:
I use Scrapy to spider asynchronously. When I define the crawler bot as an object, if the site contains complicated stuff (stateful forms or javascript) I usually create methods that involve importing either Mechanize or QtWebKit. Xpath selectors are also useful for the ability to not have to specify the entire XML tree from trunk to leaf. I then import pre-existing Django models from a site I want the data to go into and write to the DB. At this point you usually have to convert some types.
I find Scrapy cleaner and more like a pipeline so it seems to produce less 'side effect kludge' than other scraping methods (if anybody has seen a complex Beautiful Soup + Mechanize scraper you know what I mean by 'side effect kludge'). It can also act as a server to return json.
Being asynchronous, you can do crazy req/s.
I will leave out how to do all this through Tor because I don't want the Tor network being abused but am happy to talk about it one on one if your interest is beyond spamming the web.
Through this + a couple of unmentioned tricks, it's possible to get insane data, so much so it crosses over into security research & could be used for pen-testing.
Otherwise, great article! I agree that BeautifulSoup is a great tool for this.
google as a search engine crawls most all sites, but offers very few usable stuff to other bots
http://www.google.com/robots.txt
Disallow 247 Allow 41
They could be used for computation, but (mostly) aren't.
The answer is HttpFox. It records all http-requests.
1. Start recording
2. Do some action that causes data to be fetched
3. Stop recording.
You will find the url, the returned data, and a nice table of get and post-variables.
https://addons.mozilla.org/en-us/firefox/addon/httpfox/
You might have to spoof the Referer header so that it thinks the request is still coming from their website.
http://mitmproxy.org/