I am a proponent of web scraping. As long as the overall volume does not overload the available resources I don't see why automated retrieval of publicly available info is an issue. If the data in my website could help someone else's research, a simple reference to the source would make me super happy.
> We strongly feel that more researchers should be developing code to help conduct their research, and then sharing it with the community. If manual data collection has been an issue for your project, a web scraper could be the perfect solution and a great beginner coding project.
I totally agree with the general idea of sharing code and tools, in an open research community but incentivising someone from a field completely unrelated to CS to 'just code' sounds like pretty bad advice. This is very likely to lead to a lot of time spent, frustration, and underwhelming results.
Why not incentivise building a professional Software team at inter-departmental University level and have that team act as multi-department shared resource?
I guess it depends on the complexity of the task. If it's something I could knock up in a few hours time, and experiment with the results myself, why would I engage in endless inter-departmental politics, of who's project is more important, etc. On the other hand, there's a threshold where project is significantly complicated and requires professional assistance.
That would require something that is mostly missing in universities: strategy. Cooperation between departments and colleagues without direct sight on short term benefits is very slim. I'm quite close to my local university (father worked there, alumnus, wife works there, friends who are professor; it's a quite good Euro-business and law school). The thing I've learned: a university is more akin to a village than to a business even though the administrators try to run it as a business (but have terms a lot shorter than most professors). There is no clear line of control. Professors are tenured and some have super-star tendencies. Budgets are fragmented between departments, groups, and in our country are built upon subsidies for specific research tasks. There is a lot of cooperation between individual researchers in fields. But that is a response to it satisficing the demands of subsidies. Ýou could call that strategy, but I'd call that 'external motivation'. Strategy as in internal motivation: "How can we get greater research results from all our research staff by leveraging technology" is a question hardly asked.
Within the UK there is a growth in Research Software Engineering[1] groups at universities, primarily due to the work of the software sustainability institute[2].
There is a ton of programming in research, and almost all academic code is written by the researchers, whether they are in biology, physics, humanities, whatever. This has some of the drawbacks you mention, but the solution is not so obvious.
First, almost all academic code is really simple from a software engineering perspective, but really complex from a subject matter perspective. Having a deep understanding of both the data and the relevant hypotheses is critical, and is often actually helped by writing the code yourself. Trying to communicate every feature requirement perfectly, and making sure every assumption is met, to a third party CS person might be possible but is definitely non-trivial.
Second, most of these projects are one-time use (code up some project over a few months, write a paper, never touch again), and so spending a ton of time + money making it robust and efficient is not really worth it. For things like open source tools that are expected to be used by a lot of people it's much more feasible to get engineers involved. The Chan Zuckerberg initiative is actually funding a program that essentially does this [1].
It's kinda funny that this isn't widely appreciated. It used to be that research programs were most programs. Use of computers for other things was a kind of spin-off technology. (Now it's the other way around, of course.)
> but incentivising someone from a field completely unrelated to CS to 'just code' sounds like pretty bad advice.
I’m sorry but learning to code very custom or short lived software is just not that hard. I think you’re making programming too precious.
Don’t get me wrong — writing good, maintainable, long lived software, and basically anything with a UI for non programmers, and certainly anything for sale, is much harder and usually something professionals or their equivalents will need to do especially for an institution.
But writing scripts in interpreted languages like python or ruby for the research purposes of an individual or small group is not something people need to (or more importantly can) hire a dedicated team of engineers to do.
IMO it’s good that programming has become more democratic. It’s an ongoing process. And one that has happened in basically all other creative industries to some extent.
I admit being a bit disappointed that a well-known disadvantage of web scraping was not mentioned: Web scraping is fragile!
Web sites change, web frameworks evolve, and just some subtle reordering of some <divs> or renaming of CSS classes, and your perfect scraping code from yesterday will break tomorrow -- maybe not leaving you empty-handed, but probably missing some data or delivering the wrong one.
If there is an API you can use, use it. If your budget allows to pay for API access, buy it. APIs tend to be more stable than scraping, and the data provider will probably inform you if it changes. Contacting them might even get you more interesting data, as not every column they have in their database might become published on the web site.
For the purposes of their research and study, web scraping is stable enough, given that many websites (especially government sites), aren’t overhauled frequently. And many government sites, like those run by coroners, aren’t likely to have APIs.
Contacting the agency for the data is always a good step, but even if they are responsive, they may not be willing to email you on a daily basis with data updates.
We used to despise web scraping and tried to dump data from an industrial system by listening to raw Modbus RS-485 traffic. Turned out that there was no way to get the register maps from the vendor nor the firmware.
We ended up writing a scraper for the main unit. Once they have been installed and configured, they're likely not going to be updated unless absolutely necessary (such as when adding new, unsupported controllers to the bus).
That is a good argument, and I should have mentioned it, yes. For a one-off job, web scraping will probably be the best choice, and maybe even the fastest to implement. I have done my own share of web scraping for personal projects (and thus know about the fragility), but I didn't care much about broken results in the long run.
But in the article they mentioned re-running their program to update their data, so it could be a long-term effort. And anyone planning a long-term project reading that article and taking their advice should at least be warned about this possible problem.
After 'scraping' some forums via their APIs for weeks, I ended up realising that the data and metadata given to me by the API was so restricted (e.g. providing 'recent comments' instead of all comments) that a pure vanilla web scraping approach became the preferred option.
I agree with all the points you mention about the shortcomings though and your argument is sound. This is my opinion in the other direction, APIs come with an element of trust.
You both are right. Apis are extortion and scraping is fragile. I once amused myself by crating random invisible divs when generating a server side html page. It made scraping resulting files impossible, and made no difference to the look of the page.
Random invisible divs aren't likely to defeat a moderately motivated scraper, though. Depending on what they're looking for, it could be as simple as getting the inner text of a sufficiently high-up element and matching a regex.
More complex scrape defeating measures I've seen are blobs of JS that need evaluating in order to generate URL parameters (all that needs doing is extract the JS and run it in a JS engine, if you don't want to drive a headless browser, with care of course!) or that need a captcha defeating (just buy some deathbycaptcha API calls).
I've seen this approach backfire a bit too. Rather than having to scrape web content, my work is reduced to pulling out my favorite sandboxed JS interpreter bindings, running the snippet, and extracting the rich object they just created with exactly the data I wanted. You only need a headless browser if there's a meaningful interplay between the JS and the rest of the site.
I add a delay on the server side for IPs that seem scrappy and throw heavy javascript to blast off the resources. So far, it seems to work well in some cases.
a better approach is dynamically generated css, absolute positions and randomized html output. In my SEO days, I wrote a tool to do this exact thing for a different reason.
We had a network of sites, and they all looked the exact same to the user, but to google, each site had a completely different structure, and it kept the network safe for years before a google employee (or we assumed google employee - @google.com email) signed up for the service without us knowing, and discovered the entire network by placing a large order which gave him links across the entire network. Within 1 week of them signing up, our entire network of 10k domains was dead, and everything they linked to was delisted from google. We had to shut down the network, and refund all unused credits from our customers.
I didn't say I was smart. Also if you block @google emails you will just get them signing up with another email address, so why even try to block them. And once they pay for a service, you cant not deliver, because that would constitute fraud.
We tried to just fly under their radar, and avoid any automated trigger that would arouse their suspicion.
What do you mean? We sold links on websites filled with unique content we paid for...
We were masking those domains from google because google penalizes selling backlinks to justify paying for their ads. My conscience is quite clear. When google delisted our network, we refunded our customers and moved on to a smaller invite only network that ran well for years. I left that company 6 or 7 years ago, but I'm sure they are still making some money off hosting and managing private blog networks.
> And your conscience still allowed you to sleep at night?
The average American endorses slavery in their clothes, Christmas decorations and electronics. By comparison creating some bad links in a search engine is so low on my list of moral failings it doesn't even register.
SEO is already to snake-oily to do something like that. We had principals, we did our best for our customers and our clients. If we did fail in trying our best at least we could sleep knowing we weren't actively trying to screw over our paying customers.
I'm surprised I don't see this discussed more in the context of web scraping, but XPath is not only much more powerful, but can also be made robust against such techniques.
Sure, if you change the page structure enough you could defeat it, but it would require more than just adding a few divs. XPath easily lets you mix and match matching against not just CSS classes, but also the page's structure itself, inner text, attributes, and so on. As a result, you can get some really powerful queries without having any kind of complex post-processing of the results.
Xpath is one of the dinosaur technologies that I didn't learn until some time ago, and man was it a great way to find the right element, and pass that to the tool doing traversing and other things - being able to find a div that contains a string in a forest of divs was so damn nice
Eh, you might be surprised how effective it is to just use regular expressions instead of something that parses the DOM. Usually there is something to key off of, and while regular expressions aren't good for parsing HTML, they still work just as well as they always have for matching text patterns, which is often what scraping ends up being.
One example, unfortunately not very scientific: Years ago, I had a bunch of Web comics I visited daily. To make this more efficient, I had a script on one of my servers which scraped the corresponding sites in the morning and produced the daily collection on a single page.
Every few weeks or months, one of the comics would just not appear or not be updated (i.e. there was the same strip day after day). I had to update the code each time, checking the source code and finding the new page and location of the image. Later, some pages started using JavaScript to load the image file, and I lost interest in sophisticating my script. So, no single page comic collection for me anymore. :)
brittle, but in a way more stable, in the sense that the provider can't cut off access suddenly without also changing their website.
Also the breaks that do happen tend to be trivial to fix. The entire project needs to be managed differently. I'd look at another frequently updated project (youtube-dl comes to mind) where breaks are _expected_
Yep, we've had exactly this issue with the web portal on one of our applications. A customer used some 'quick and dirty' convenience tool that used web scraping to invoke the portal application pages, and wrapped that up with part of their admin workflow.
When the pages changed, even slightly, it broke their admin tool.
We had said not to do this, but the workflow/web-scraping tool was surely quicker and cheaper to implement than custom API development within that organisation.
Going way back in time, well before www and soap, I recall having to develop an application that would talk to another system using serial comms, effectively pretending to be a VT52-type terminal.
Had to send commands, verify responses, send subsequent commands based on available options, etc.
That was used to integrate between two different types of Telco exchange equipment, for provisioning of X25 circuits on the national network in the UK.
Supplier of the kit did not have any form of API, just a terminal-level application, so client-side code had to pretend to be an operator.
Ahh, those old days, using green-screen terminals, eh
Been there, done that.... even to hack an old ciscos into a bodged-up production/testing enviroment (take config from production, change two addresses from production to testing values, and apply the whole config to a testing cisco).
The good thing with ciscos and most of the old technology is, that once you write that script, it works for years... commands never change, outputs never change, some perl, a regex or five, and you're done.
Doing the same with a webpage, where it's "pride month" today, "womens day" tomorrow, "day against aids" the day after, and each means a new div, a new popup, a new redirect, a backend update inbetween, to make the new banner possible, etc., is a pain in the ass.
Good point. Yet a good remedy is to add real-time asserts on your scraped data and get yourself notified as soon as you encounter unexpected data formats.
I've found the opposite to be true -- when an entity is maintaining an API and their website with the same data, the website is their core business. The API is prone to being incomplete, buggy, subject to sudden deprecation, unreasonably rate limited (crippling access to some objects below what a casual human user has), and so on.
Conversely, overall document structure doesn't change much over time. I know it _can_; there's a social contract that APIs should change slowly while documents can change whenever, but that isn't what I observe in the wild. Even on fairly major redesigns, the overall structure has minimal edits.
A technique I've used before (wasted effort in hindsight since web pages are stable and I never have to update my scrapers) is to come up with several semantically different ways of accessing a piece of data on a page. It serves two purposes; you can recover from small page changes by having the different methods vote, and you can detect most kinds of page changes by noticing discrepancies, notifying yourself that the scraper needs to be updated soon.
I think it really depends on the type of product that the business makes money on. If one of the main products is data then I'd wager their API will have significantly more information compared to their website. If they make money via the website then yes, they're less inclined to spend resources on the API.
All of our own websites are built on-top of the same public API that everybody else uses and scraping used to be a nuisance. It was also confusing because they would be able to get more data using the same free account just by using the API instead of scraping. Exactly like the OP mentioned we only show a small number of properties via the website but most scrapers never took the time to actually compare API vs website.
I think this depends entirely on what you are indexing. From my experience with some 100 ish scrapers for news sites a few would break literally every day.
And the only thing we really wanted was article title and date.
Granted, but there are lots and lots of ways they can break scrapers in the pursuit of their core business, such as a website redesign. For example, moving from static HTML to a web framework would require your scraper to actually run the JavaScript to generate the DOM in the state that a reader might view it in, and this is quite a lot more complicated than walking the static HTML.
Its not too complicated, you just need a headless browser. Having done a ton of web scraping projects, I’d recommend just starting with this approach as even sites that look pretty static use Javascript in subtle ways.
"only" is a bit of an overstatement. The data is always coming from somewhere, it just depends on how much effort needed to reverse engineer the JavaScript code path to the data
I don't find this as a concern - in all the scraping I've done, the only bottleneck was the intentional throttling/rate limiting, not the speed and resources spent by the headless browser; a small, cheap machine could easily process many, many times more requests than it would be reasonable to crawl.
Data is usually embedded in json or available from an internal api when it's an SPA. Headless browser resources are pretty huge. When doing large scale scraping, headless browser should be a last resort
Looking at this sentence, I have the impression that it is nowadays taken for granted that "web framework" means "front end web framework". I come from a time in which it was perfectly fine to generate static HTML via a (server-side) web framework.
For example, moving from static HTML to a web framework would require your scraper to actually run the JavaScript to generate the DOM in the state that a reader might view it in
Or, as is often the case, the content is already there or fetched via an API in far more easily-consumed JSON format that you can use directly.
Granted, lots of APIs make it prohibitively difficult to authenticate such that it’s easier to simply scrape. Such is the case with just about every Microsoft product I’ve ever used, most recently the XBox Live API. I genuinely wonder what kind of nonsense goes on in Microsoft design review meetings.
I've found that the breaking point is for websites that consume their own public APIs. On those, the API is usually very well maintained, documented, and stable.
Those that don't use their own APIs almost always end up with an open API in the state you describe (except maybe the very big players like FB, where the open API is overall good).
That is actually a good criteria for code quality in general. Don’t prepare a Java method which is not used. Because it will _never_ be right. Just implement today’s story, and leave the rest for another day. Same goes for rarely used functions: They are usually very buggy, where as function in the middle of everyone’s workflow are flawless. Hence the work of a good product owner is to streamline everyone as much as possible on a few central functions. But of course, in an enterprise environment, there are a few functions that are required to work (XML export, backup and restore...)
The best explanation I've come up with is that as a naive developer, it's impossible to know the nuances of any sufficiently complex process or workflow.
I've long wanted a really robust way of defining page areas for scraping, that could handle even relatively major HTML shifts.
My best idea has been to simply maintain a collection of "reference" URL's (e.g. of different products or articles) and identify unique start/end text for those specific instances.
Then automatically extract as many possible different "rules" for locating the desired content (pure structure and ordering, class hierarchies, classes/ids, surrounding text, etc.) and find the ones that are consistent across different instances.
And then just use those rules until they break on the reference page... and when they break, develop new ones.
I've seen a few academic papers and a few closed products that convert your selection of content you care about into a scraper capable of acquiring that content in the future. Last I checked there wasn't anything readily available as a FOSS library for doing so.
I'm having trouble finding those papers at the moment, but here are a couple commercial products that sound similar in spirit to what you're describing.
Edit: I hadn't searched recently enough. See the sibling comment recommending this library. Haven't used it yet, but at first glance it looks nice. https://github.com/alirezamika/autoscraper/
I've tried autoscraper now, and I don't like it (not yet).
(1) Its wrapper generation code isn't much more advanced than that similar data will be similarly nested in similar parent blocks. It looks more brittle than I'd like.
(2) It has zero tests, comments, docstrings, types, or any other niceties so far (and minimal documentation).
(3) When things go wrong it strongly prefers returning no information and not throwing any errors. None of the examples in the README actually run (or rather, they give you a `None` response that's all but useless) without changes.
despite being undocumented, it really works well.
I tried the readme examples and all work. maybe you didn't update the wanted list in the examples because it has changed in the page.
IMO the biggest problem is the lack of js enabled content support.
It works well enough. I tried a few other sites and had mixed results even when providing the raw HTML so that I knew its http logic wasn't the issue.
> maybe you didn't update the wanted list in the examples
Yeah, that was my only real problem with the readme examples. Those could just as easily be provided as local data (e.g. how `sklearn.datasets` works) so that the end user starts with working code, especially since there are no errors/warnings/etc when anything goes wrong.
> IMO the biggest problem is the lack of js enabled content support.
Haha, unless I'm seriously misunderstanding you this is one of the only things I don't mind :) Since you can pass raw html to the library, you can use your favorite headless browser to navigate (or in the happy case just load a non-interactive js-enabled site) to your content and pass it through to this library to do the data extraction. I rather like those features being decoupled and kind of wish this library didn't attempt to do any of the crawling itself. I know that's just a personal preference, but it's my account, so I'll say what I like about it.
Interesting approach using multiple locators, might use it in the future. Although in my limited experience I agree that the avg site which usually needs scraping keeps its "interface" mostly unchanged.
I guess a lot of stuff I've needed to scrape are from old CMSes or sites where its viewed as part of a cost center they're unlikely to invest in and just maintain it
I've had good luck just running this sort of thing as an offline process, especially for external dependencies. We used to get 'pink' for this lookup and now it's "<span>pink </span>" or somesuch.
It depends on the SLA, of course, but it's cheaper to check every few hours than on every request, and you get a couple of alerts instead of a constant stream of them.
Approaching the web as an end user, I have also found this to be true. Most websites rarely change their document structure in such a way that breaks simple text-editing scripts. Keyword: "Most". In most cases no specialised tools or libararies are needed for extracting text or other resources. Again, keyword: "most". Personally, just because there may be a few exceptions does not mean I am going to change a strategy that works almost 100% of the time.
Understanding "web APIs", which did not exist when I first starting using the www in 1993, other than as a way to try to control and/or monetise scraping continues to escape me. I do like the increased usage of "endpoints" though, serving only data with no markup. Although XML and JSON are too bloated compared to something sensible like netstrings.
Similarly, on the client side, I fail to understand all the parsing tools and libraries and related promotion; it is just as easy to break any solution that depends on them and in many cases they are obviously overkill, more brittle than simple scripts using generalised text-editing tools.
One example is "jq". In many cases it is clearly overkill and is slower than sed.
As a data source, the web is messy. "Standards" cannot be relied on 100%. Some people try to pretend the web is clean and can be tamed, or they "give up" because it is not "perfect" and things can break. Getting hands dirty works the best and most things do not break if kept simple, IME
I find that's the difference between public API endpoints and those APIs written for the page/app itself... if it's tightly coupled and the specs aren't published to the public for general use, I treat it as breakable.
That's just my own take. I've worked in environments with stronger versioning, and depending on your data needs and structures it can work. It's usually not worth it for most use cases though.
About 15 years ago, I worked on an add-on for a large online service, using their official, versioned, documented API.
We had to build a test suite just to verify that the API working as expected, because they would break it so often, and also the documentation didn't always match reality.
This was a paid API we used on a pay-per-use basis, IIRC, and had official support for.
In the beginning, we had a false sense of security about the version numbers and such. The first couple of breaks seemed to be "just this one time". Then we realized that it was happening all the time, and so the test suite. (I was a junior, so I can't take credit for this work, just a witness.)
API is often no more reliable than scraping human UI most of the time, with the added disadvantage of being second-level importance.
Personally, I've tried to combine human UI with API as much as possible. For example, I added a feature for being able to post via direct URL entry, like so: http://example.com/hello+world
most browsers will convert the spaces for you too, so you can just type your text into the address bar.
There are multiple ways of doing web scraping, and some are definitely more fragile than others.
I've found fully specified XPaths to be a mistake for example. It only takes one tiny change on the page to mess up the script. On the other hand, despite numerous warnings that it would be a disaster I've found I have a lot of luck maintaining regexes, even after major page reworks.
Sure, but A) As implemented in code, the path doesn’t necessarily have to be fully explicit in terms of tags. You could look for a child with text containing rather than a specific class or id, for example (or get fancier with semantic similarity on tags or text). B) There’s a trade off between speed and fragility that could make a difference if your tree is deep enough that traversing it iteratively is slow enough compared to a long xpath that it becomes a limiting factor. Granted you don’t typically encounter this in a standard issue html tree but the lxml docs, for example, correctly note that xpaths can be way faster when the nesting is super deep.
Keep in mind this is a research publication. In general research systems aren't meant to be maintained long term. You do the study, or make the proof of concept, and if it has long term value then you find a long term solution.
I know they mention maintenance, but having been in both fields what a web person means by maintenance and what a research person means by maintenance are an order of magnitude different.
Still, para 7 excerpt. " Once your program is written, you can recapture these data whenever you need to, assuming the structure of the website stays mostly the same."
It all makes sense if you believe in an information is universal idea where it should all be freely available.
Though, you you get cases like celebritynetworth.com (no affiliation to me) that was featured on HN where Google wanted to APIize the data and when that was not available they decided to scrape it. It effectively killed the business.
I think if a company decides to offer an API or other structured data then fine, they've decided that their data can be shared in a way that makes it OK to collect en masse.
I wouldn't say we should "learn to love" scraping when there are sites that put a lot of man hours into the data that's seen on public html pages only for someone else to spend a few hours scraping it and repurpose it.
Now, if the major traffic drivers like search engines and social networks were able to differentiate who the true authoritative source is, that'd be great. But they don't.
There are a significant number of missing steps, in terms of moral justifiability, between "scraping information from the web for academic research purposes" and "using the scraped information to add value to our primary generator of advertisement-derived revenue".
What is there that needs justification for what the proverbial eyes can plainly see? That aside what is the moral license to academic research and what gives the right to gatekeep what everyone can see if they took an uncommon action? Both are in fact doing so to make a living indirectly for their institutions by using what was already out there and rely upon arcane means to derive income from the data through layers of separation.
The article avoids directly addressing this, but I think is the real elephant in the room -- whether you have the right to web scrape for research.
The article says, "You might be able to use what you scrape, but it’s worth checking that you can also legally share it. Ideally, the website content licence [sic] will be readily available."
In practice, the data you want to scrape either has 1) no license/information about whether they are okay with you scraping it (and if they are okay with it, they usually offer the source data anyways), or more commonly they 2) strictly prohibit it in the terms of service, in which case it's not clear whether a scraper which mimics a browser falls under fair use.
It's an area where when you ask your university attorney, they exchange a few emails with you and then avoid making any decision. I think it's just because it's not well tested in court (at least when I encountered these situations), and depends on whether you'd be subject to a takedown notice or lawsuit.
Much scraping is done with automated control of a (possibly headless) web browser. It runs all the same scripts and has all the same features as a standard web browser for sighted users.
One thing I tend to see a lot (90% of requests) is from people who are stuck with mediocre CRMs and want better analytics and email reporting than they’re given. The only way to accomplish anything is to spoof login and scrape because the only api is through “partners” who either pay a premium or are giant.
I turn these requests down because it’s a ticking time bomb when you add in the element of login (password resets, 2FA, another point of change). On the other hand, I wonder how different this is from what Plaid does...
I've written scrapers and web automation bots going as far back as 1998 (using VB5 and VB6 automating IE)
Nowadays mostly I write custom scrapers use PHP (Guzzle, Curl, etc) and Python (I was introduced to Beautiful Soup, by the "Python for Secret Agents" book)
I've tried several commercial scraper tools and services over the years but few stuck for various reasons.
UbotStudio had great potential but in the end was buggy and painful, almost abandonware.
Scrapinghub.com is decent enough but bit expensive for my projects.
80legs.com is cool for massive scale but was overly robots.txt restrictive at the time I tried using it (for what I was scraping) and I don't like the syntax.
A scraper colleague likes using Winautomation however it's no longer for sale separately, because Microsoft acquired the company and rolled it into their Power Automate SaaS (RPA/RDA focused)
There is a new tool called RTILA that I used for the very first time a 9 days ago, which actually is the easiest way to create and run scrapers I have found in 20 years.
The RTILA software currently has minimal documentation (apparently is being worked on now), however the new features are being developed fast, all releases are here on GitHub (see the frequency of releases)
https://github.com/IKAJIAN/rtila-releases/releases
I am a genuine customer and have no connection in any way with the solo founder developing it, (except for a few emails and support forum messages). Genuine recommendation for writing a bot more easily.
Having written a few scrapers, including the browser kind. I found the trick was to specialize.
Each website respond differently, some can just use requests, others would need selenium with pre-existing profile. To develop a common utility is almost futile.
I do have useful building blocks, but for each individual things I want to scrape I scale out using project specific code. It's never too slow either - the time it would take to fill in all of the required bits in a do it all tool would have been similar.
How are you doing it? Automating an actual browser using the developer tools is probably the most under-the-radar way, and also quite nice to work with.
This is a good point, and I've noticed this phenomenon on some video sites lately.
Although the caveats for this particular library[1] imply enough false positives and false negatives that it seems mostly useless. Sites that take this seriously must be doing something smarter.
[1] "Doesn't work if DevTools is undocked and will show false positive if you toggle any kind of sidebar."
Shameless plug for my open source library Helium [1]. It extends Selenium's API and makes it easier to use. The name comes from Helium also being a chemical element, but lighter than Selenium.
In the 2-scraping folder, there's a bunch of scraping examples such as bypassing captchas, building an Amazon price checker, blocking CSS and JavaScript resources during scraping to make the process much quicker, and a few others. Hope it's useful.
The article mentions Python, Requests and BeautifulSoup.
Here's a short example that scrapes HN Favorites. [0]
#!/usr/bin/env python3
import requests
from bs4 import BeautifulSoup
username = input('username: ')
# session uses connection pooling, often resulting in faster execution.
session = requests.Session()
base = 'https://news.ycombinator.com/'
path = f'favorites?id={username}'
while path:
r = session.get(base + path)
s = BeautifulSoup(r.text, 'html.parser')
for a in s.select('a.storylink'):
print(a.text, a['href'])
more = s.select_one('a.morelink')
path = more['href'] if more else None
To be fair, the code above is for scraping favorites which I do not believe is included in the official API. There was a very popular story[0] a few days ago that used scraped data to compile a list of the most favorite posts, which many users didn't even know was a feature until reading the story. Dang even joined in.
People love BeautifulSoup, but I've had better luck with fully emulating a browser with Selenium. While more complicated it seems to run into less issues. Is there a rule of thumb as to when to use one over the other?
I looked into using Selenium, but it seems like you have to download an 80 MB executable. If you noticed above, I also posted JavaScript that you could run in the browser that you already have. It seemed like a better way to go. I admit Selenium does have some useful features, too.
Nowadays you could just use headless Chrome[1] or Firefox[2], though it looks like Firefox is lagging behind a bit and still wants to require Selenium.
Personally, I loved using BS for hobbies until the SPA era started, and then I had to either use headless (selenium back then was great) and/or monitor the network for their API.
This seems like the tip of the iceberg for their work. The authors automated document collection but are left with the task of searching through the docs and crawling onward to linked documents. What kind of libraries are available today for crawling pdf documents?
I failed a webscraping project due to strong anti bot detection.
They checked for bots through useragent/screen size, maybe mouse movements, trends in searches (same area code), etc... (Can they really detect me through my internet connection headers, despite proxies?)
there are 2 approaches they use that make developing bots very difficult.
1. they detect device input. if there is no mouse movement, while the website is being loaded, they will consider it's a bot.
2. they detect the order of page visiting. A human visitor will not enumerate all paths, instead, they follow certain patterns. This is detectable with their machine learning model.
The beautiful thing about web scraping is that it automates so much tedious work that the benefits in saved time offsets the brittleness inherent in scraping data.
And with Puppeteer (also Playwright) it's never been easier. Recaptcha solving, Ad blocking etc. in just a few lines of code[1].
I've built a business on the back of Puppeteer - https://simplescraper.io. Ten months in and we've just passed 100 customers so there's mucho opportunities in solving these kind of problems.
Genuinely thought this was an editorial by nature.com explaining how they stopped worrying about researchers scraping their website for data and instead embraced them, and the rationale behind this decision. Turned out to be a just a 101 about web scraping in general. Disappointed.
My favourite kind of web scraping is when you have to pick apart some undocumented API that serves up data to an SPA, then figure out how to tidy up the response(s) into a single pandas dataframe. Always a satisfying feeling to solve one of those little puzzles.
Surprised no one has mentioned weboob. If your interest isn't specifically along the lines of "go to all pages, and get all of the data" and more along the lines of "go to my bank, log in using username and password, get the transactions" etc, weboob is ideal. It seems to be extremely well thought out and easily extendable.it has plugins for a whole bunch of websites already, including many, many banks, some of the biggest websites such as reddit, Google, and many more. The only downside I've had so far is that it will fall flat on it's arse if the website has to load must do javascripty things (it can handle APIs and html easily though).
I really like the options of node + cheerio, or node + puppeteer for these kinds of jobs. Language of the browser, easy enough selection/usage, cheerio if it's plain html, puppeteer if it's dynamic/spa content.
In the end, not hammering a site is key... also, on your own end, possibly hashing the main content, so you aren't creating duplicate entries on your own side are important.
As to fragility, that happens... in general, you need to update to match site updates, but most sites won't be dramatically updated more than a couple times a year if they aren't in active development.
Used to do web scrapping and took very long as most websites change formatting and tags quite often. This required engineering just to maintain few sites all the time.
found this lib in the comments, tried it and looks nice.
can be good for finding the xpath automatically and save the manual traversing overhead.
https://github.com/alirezamika/autoscraper
158 comments
[ 3.1 ms ] story [ 252 ms ] threadI totally agree with the general idea of sharing code and tools, in an open research community but incentivising someone from a field completely unrelated to CS to 'just code' sounds like pretty bad advice. This is very likely to lead to a lot of time spent, frustration, and underwhelming results.
Why not incentivise building a professional Software team at inter-departmental University level and have that team act as multi-department shared resource?
edit: missed words
[1] https://society-rse.org/ [2] https://www.software.ac.uk/
First, almost all academic code is really simple from a software engineering perspective, but really complex from a subject matter perspective. Having a deep understanding of both the data and the relevant hypotheses is critical, and is often actually helped by writing the code yourself. Trying to communicate every feature requirement perfectly, and making sure every assumption is met, to a third party CS person might be possible but is definitely non-trivial.
Second, most of these projects are one-time use (code up some project over a few months, write a paper, never touch again), and so spending a ton of time + money making it robust and efficient is not really worth it. For things like open source tools that are expected to be used by a lot of people it's much more feasible to get engineers involved. The Chan Zuckerberg initiative is actually funding a program that essentially does this [1].
[1]https://chanzuckerberg.com/rfa/essential-open-source-softwar...
I’m sorry but learning to code very custom or short lived software is just not that hard. I think you’re making programming too precious.
Don’t get me wrong — writing good, maintainable, long lived software, and basically anything with a UI for non programmers, and certainly anything for sale, is much harder and usually something professionals or their equivalents will need to do especially for an institution.
But writing scripts in interpreted languages like python or ruby for the research purposes of an individual or small group is not something people need to (or more importantly can) hire a dedicated team of engineers to do.
IMO it’s good that programming has become more democratic. It’s an ongoing process. And one that has happened in basically all other creative industries to some extent.
Web sites change, web frameworks evolve, and just some subtle reordering of some <divs> or renaming of CSS classes, and your perfect scraping code from yesterday will break tomorrow -- maybe not leaving you empty-handed, but probably missing some data or delivering the wrong one.
If there is an API you can use, use it. If your budget allows to pay for API access, buy it. APIs tend to be more stable than scraping, and the data provider will probably inform you if it changes. Contacting them might even get you more interesting data, as not every column they have in their database might become published on the web site.
Contacting the agency for the data is always a good step, but even if they are responsive, they may not be willing to email you on a daily basis with data updates.
We ended up writing a scraper for the main unit. Once they have been installed and configured, they're likely not going to be updated unless absolutely necessary (such as when adding new, unsupported controllers to the bus).
But in the article they mentioned re-running their program to update their data, so it could be a long-term effort. And anyone planning a long-term project reading that article and taking their advice should at least be warned about this possible problem.
After 'scraping' some forums via their APIs for weeks, I ended up realising that the data and metadata given to me by the API was so restricted (e.g. providing 'recent comments' instead of all comments) that a pure vanilla web scraping approach became the preferred option.
I agree with all the points you mention about the shortcomings though and your argument is sound. This is my opinion in the other direction, APIs come with an element of trust.
More complex scrape defeating measures I've seen are blobs of JS that need evaluating in order to generate URL parameters (all that needs doing is extract the JS and run it in a JS engine, if you don't want to drive a headless browser, with care of course!) or that need a captcha defeating (just buy some deathbycaptcha API calls).
We had a network of sites, and they all looked the exact same to the user, but to google, each site had a completely different structure, and it kept the network safe for years before a google employee (or we assumed google employee - @google.com email) signed up for the service without us knowing, and discovered the entire network by placing a large order which gave him links across the entire network. Within 1 week of them signing up, our entire network of 10k domains was dead, and everything they linked to was delisted from google. We had to shut down the network, and refund all unused credits from our customers.
We tried to just fly under their radar, and avoid any automated trigger that would arouse their suspicion.
All this seems like a nice illustration of how the web ecosystem encourages parasitic behavior on so many fronts. It's sad.
We were masking those domains from google because google penalizes selling backlinks to justify paying for their ads. My conscience is quite clear. When google delisted our network, we refunded our customers and moved on to a smaller invite only network that ran well for years. I left that company 6 or 7 years ago, but I'm sure they are still making some money off hosting and managing private blog networks.
The average American endorses slavery in their clothes, Christmas decorations and electronics. By comparison creating some bad links in a search engine is so low on my list of moral failings it doesn't even register.
Underperform, and then offer them a goodwill refund if they ask for it?
Sure, if you change the page structure enough you could defeat it, but it would require more than just adding a few divs. XPath easily lets you mix and match matching against not just CSS classes, but also the page's structure itself, inner text, attributes, and so on. As a result, you can get some really powerful queries without having any kind of complex post-processing of the results.
Can you give any specific examples (sites and data needed from them).
Every few weeks or months, one of the comics would just not appear or not be updated (i.e. there was the same strip day after day). I had to update the code each time, checking the source code and finding the new page and location of the image. Later, some pages started using JavaScript to load the image file, and I lost interest in sophisticating my script. So, no single page comic collection for me anymore. :)
Set up a cron job to run `dosage @` every day, it will check for new comics and download them.
The good thing with ciscos and most of the old technology is, that once you write that script, it works for years... commands never change, outputs never change, some perl, a regex or five, and you're done.
Doing the same with a webpage, where it's "pride month" today, "womens day" tomorrow, "day against aids" the day after, and each means a new div, a new popup, a new redirect, a backend update inbetween, to make the new banner possible, etc., is a pain in the ass.
Conversely, overall document structure doesn't change much over time. I know it _can_; there's a social contract that APIs should change slowly while documents can change whenever, but that isn't what I observe in the wild. Even on fairly major redesigns, the overall structure has minimal edits.
A technique I've used before (wasted effort in hindsight since web pages are stable and I never have to update my scrapers) is to come up with several semantically different ways of accessing a piece of data on a page. It serves two purposes; you can recover from small page changes by having the different methods vote, and you can detect most kinds of page changes by noticing discrepancies, notifying yourself that the scraper needs to be updated soon.
All of our own websites are built on-top of the same public API that everybody else uses and scraping used to be a nuisance. It was also confusing because they would be able to get more data using the same free account just by using the API instead of scraping. Exactly like the OP mentioned we only show a small number of properties via the website but most scrapers never took the time to actually compare API vs website.
And the only thing we really wanted was article title and date.
People are usually hesitant to constantly change how a customer interacts. All to willing to change internal details.
Granted, but there are lots and lots of ways they can break scrapers in the pursuit of their core business, such as a website redesign. For example, moving from static HTML to a web framework would require your scraper to actually run the JavaScript to generate the DOM in the state that a reader might view it in, and this is quite a lot more complicated than walking the static HTML.
Certainly more resource-intensive.
Looking at this sentence, I have the impression that it is nowadays taken for granted that "web framework" means "front end web framework". I come from a time in which it was perfectly fine to generate static HTML via a (server-side) web framework.
Or, as is often the case, the content is already there or fetched via an API in far more easily-consumed JSON format that you can use directly.
Granted, lots of APIs make it prohibitively difficult to authenticate such that it’s easier to simply scrape. Such is the case with just about every Microsoft product I’ve ever used, most recently the XBox Live API. I genuinely wonder what kind of nonsense goes on in Microsoft design review meetings.
Those that don't use their own APIs almost always end up with an open API in the state you describe (except maybe the very big players like FB, where the open API is overall good).
The best explanation I've come up with is that as a naive developer, it's impossible to know the nuances of any sufficiently complex process or workflow.
My best idea has been to simply maintain a collection of "reference" URL's (e.g. of different products or articles) and identify unique start/end text for those specific instances.
Then automatically extract as many possible different "rules" for locating the desired content (pure structure and ordering, class hierarchies, classes/ids, surrounding text, etc.) and find the ones that are consistent across different instances.
And then just use those rules until they break on the reference page... and when they break, develop new ones.
I'm curious if anyone's built this type of thing?
I'm having trouble finding those papers at the moment, but here are a couple commercial products that sound similar in spirit to what you're describing.
https://scraper.ai/
https://www.diffbot.com/ (kind of)
Edit: I hadn't searched recently enough. See the sibling comment recommending this library. Haven't used it yet, but at first glance it looks nice. https://github.com/alirezamika/autoscraper/
(1) Its wrapper generation code isn't much more advanced than that similar data will be similarly nested in similar parent blocks. It looks more brittle than I'd like.
(2) It has zero tests, comments, docstrings, types, or any other niceties so far (and minimal documentation).
(3) When things go wrong it strongly prefers returning no information and not throwing any errors. None of the examples in the README actually run (or rather, they give you a `None` response that's all but useless) without changes.
It works well enough. I tried a few other sites and had mixed results even when providing the raw HTML so that I knew its http logic wasn't the issue.
> maybe you didn't update the wanted list in the examples
Yeah, that was my only real problem with the readme examples. Those could just as easily be provided as local data (e.g. how `sklearn.datasets` works) so that the end user starts with working code, especially since there are no errors/warnings/etc when anything goes wrong.
> IMO the biggest problem is the lack of js enabled content support.
Haha, unless I'm seriously misunderstanding you this is one of the only things I don't mind :) Since you can pass raw html to the library, you can use your favorite headless browser to navigate (or in the happy case just load a non-interactive js-enabled site) to your content and pass it through to this library to do the data extraction. I rather like those features being decoupled and kind of wish this library didn't attempt to do any of the crawling itself. I know that's just a personal preference, but it's my account, so I'll say what I like about it.
I guess a lot of stuff I've needed to scrape are from old CMSes or sites where its viewed as part of a cost center they're unlikely to invest in and just maintain it
It depends on the SLA, of course, but it's cheaper to check every few hours than on every request, and you get a couple of alerts instead of a constant stream of them.
Understanding "web APIs", which did not exist when I first starting using the www in 1993, other than as a way to try to control and/or monetise scraping continues to escape me. I do like the increased usage of "endpoints" though, serving only data with no markup. Although XML and JSON are too bloated compared to something sensible like netstrings.
Similarly, on the client side, I fail to understand all the parsing tools and libraries and related promotion; it is just as easy to break any solution that depends on them and in many cases they are obviously overkill, more brittle than simple scripts using generalised text-editing tools.
One example is "jq". In many cases it is clearly overkill and is slower than sed.
https://stackoverflow.com/q/59806699
As a data source, the web is messy. "Standards" cannot be relied on 100%. Some people try to pretend the web is clean and can be tamed, or they "give up" because it is not "perfect" and things can break. Getting hands dirty works the best and most things do not break if kept simple, IME
That's just my own take. I've worked in environments with stronger versioning, and depending on your data needs and structures it can work. It's usually not worth it for most use cases though.
We had to build a test suite just to verify that the API working as expected, because they would break it so often, and also the documentation didn't always match reality.
This was a paid API we used on a pay-per-use basis, IIRC, and had official support for.
In the beginning, we had a false sense of security about the version numbers and such. The first couple of breaks seemed to be "just this one time". Then we realized that it was happening all the time, and so the test suite. (I was a junior, so I can't take credit for this work, just a witness.)
API is often no more reliable than scraping human UI most of the time, with the added disadvantage of being second-level importance.
Personally, I've tried to combine human UI with API as much as possible. For example, I added a feature for being able to post via direct URL entry, like so: http://example.com/hello+world
most browsers will convert the spaces for you too, so you can just type your text into the address bar.
I've found fully specified XPaths to be a mistake for example. It only takes one tiny change on the page to mess up the script. On the other hand, despite numerous warnings that it would be a disaster I've found I have a lot of luck maintaining regexes, even after major page reworks.
I know they mention maintenance, but having been in both fields what a web person means by maintenance and what a research person means by maintenance are an order of magnitude different.
I recently sold a 'business' that does web scraping. Does anyone have insight into which industries need more web scraping 'experts'?
Though, you you get cases like celebritynetworth.com (no affiliation to me) that was featured on HN where Google wanted to APIize the data and when that was not available they decided to scrape it. It effectively killed the business.
I think if a company decides to offer an API or other structured data then fine, they've decided that their data can be shared in a way that makes it OK to collect en masse.
I wouldn't say we should "learn to love" scraping when there are sites that put a lot of man hours into the data that's seen on public html pages only for someone else to spend a few hours scraping it and repurpose it.
Now, if the major traffic drivers like search engines and social networks were able to differentiate who the true authoritative source is, that'd be great. But they don't.
Not part of the Berne convention, and not applicable in the US.
The article says, "You might be able to use what you scrape, but it’s worth checking that you can also legally share it. Ideally, the website content licence [sic] will be readily available."
In practice, the data you want to scrape either has 1) no license/information about whether they are okay with you scraping it (and if they are okay with it, they usually offer the source data anyways), or more commonly they 2) strictly prohibit it in the terms of service, in which case it's not clear whether a scraper which mimics a browser falls under fair use.
It's an area where when you ask your university attorney, they exchange a few emails with you and then avoid making any decision. I think it's just because it's not well tested in court (at least when I encountered these situations), and depends on whether you'd be subject to a takedown notice or lawsuit.
Or what about a 'human scraper' a la Amazon MTurk?
Very strange fact.
Too bad I can't find a measure of English content written by country. https://en.wikipedia.org/wiki/List_of_countries_by_English-s...
b) what does "switching to WASM" mean for you? If it still generates a DOM or access a private API, many scraping techniques don't care.
I turn these requests down because it’s a ticking time bomb when you add in the element of login (password resets, 2FA, another point of change). On the other hand, I wonder how different this is from what Plaid does...
Nowadays mostly I write custom scrapers use PHP (Guzzle, Curl, etc) and Python (I was introduced to Beautiful Soup, by the "Python for Secret Agents" book)
I've tried several commercial scraper tools and services over the years but few stuck for various reasons.
UbotStudio had great potential but in the end was buggy and painful, almost abandonware.
Scrapinghub.com is decent enough but bit expensive for my projects.
80legs.com is cool for massive scale but was overly robots.txt restrictive at the time I tried using it (for what I was scraping) and I don't like the syntax.
A scraper colleague likes using Winautomation however it's no longer for sale separately, because Microsoft acquired the company and rolled it into their Power Automate SaaS (RPA/RDA focused)
There is a new tool called RTILA that I used for the very first time a 9 days ago, which actually is the easiest way to create and run scrapers I have found in 20 years.
The RTILA software currently has minimal documentation (apparently is being worked on now), however the new features are being developed fast, all releases are here on GitHub (see the frequency of releases) https://github.com/IKAJIAN/rtila-releases/releases
Another user of the software has produced several video tutorials here showing how it works: https://www.youtube.com/channel/UCH6ov8LnB8-4ZF0yraxjw8Q/vid...
You download it from GitHub but also still need to buy a license key from here https://codecanyon.net/user/ikajian
The RTILA home page is here https://rtila.ikajian.com/
I am a genuine customer and have no connection in any way with the solo founder developing it, (except for a few emails and support forum messages). Genuine recommendation for writing a bot more easily.
Each website respond differently, some can just use requests, others would need selenium with pre-existing profile. To develop a common utility is almost futile.
I do have useful building blocks, but for each individual things I want to scrape I scale out using project specific code. It's never too slow either - the time it would take to fill in all of the required bits in a do it all tool would have been similar.
See: https://github.com/jawj/web-scraping-for-researchers
Although the caveats for this particular library[1] imply enough false positives and false negatives that it seems mostly useless. Sites that take this seriously must be doing something smarter.
[1] "Doesn't work if DevTools is undocked and will show false positive if you toggle any kind of sidebar."
1: https://github.com/mherrmann/selenium-python-helium
I'm keeping the codebase open on GitHub: https://github.com/umaar/learn-browser-testing/ so anyone who wants to follow along can do so for free.
In the 2-scraping folder, there's a bunch of scraping examples such as bypassing captchas, building an Amazon price checker, blocking CSS and JavaScript resources during scraping to make the process much quicker, and a few others. Hope it's useful.
Here's a short example that scrapes HN Favorites. [0]
[0] Python: https://github.com/gabrielsroka/gabrielsroka.github.io/blob/...[1] JavaScript, runs in browser with a UI: https://github.com/gabrielsroka/gabrielsroka.github.io/blob/...
[0] https://news.ycombinator.com/item?id=24351073
Personally, I loved using BS for hobbies until the SPA era started, and then I had to either use headless (selenium back then was great) and/or monitor the network for their API.
[1] https://developers.google.com/web/updates/2017/04/headless-c...
[2] https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Hea...
They checked for bots through useragent/screen size, maybe mouse movements, trends in searches (same area code), etc... (Can they really detect me through my internet connection headers, despite proxies?)
It was impossible for me to scrape, they won.
there are 2 approaches they use that make developing bots very difficult.
1. they detect device input. if there is no mouse movement, while the website is being loaded, they will consider it's a bot.
2. they detect the order of page visiting. A human visitor will not enumerate all paths, instead, they follow certain patterns. This is detectable with their machine learning model.
I really don't have a solution for #2
If you record, you can probably teach AI to emulate.
Think about websites that have every reason to stop you from scraping.
It's not reasonable to disect their huge obfuscated js code. So headless doesn't really work.
And with Puppeteer (also Playwright) it's never been easier. Recaptcha solving, Ad blocking etc. in just a few lines of code[1].
I've built a business on the back of Puppeteer - https://simplescraper.io. Ten months in and we've just passed 100 customers so there's mucho opportunities in solving these kind of problems.
[1] https://github.com/berstend/puppeteer-extra/tree/master/pack...
Python Tutorial: Web Scraping with BeautifulSoup and Requests
https://youtu.be/ng2o98k983k
https://spencerbaucke.com/2020/04/29/web-scraping-in-power-b...
In the end, not hammering a site is key... also, on your own end, possibly hashing the main content, so you aren't creating duplicate entries on your own side are important.
As to fragility, that happens... in general, you need to update to match site updates, but most sites won't be dramatically updated more than a couple times a year if they aren't in active development.