I wrote a highly complex analysis engine for Wikipedia and Wikidata that fetched Wiki markup through the API and tried to parse the Wiki markup.
It was big mistake.
The feedback loop that Wikipedia editors work on is "does the HTML output look OK?" and not "does the markup conform to an unknown specification?" so part of the problems is fixing up "errors" that exist against what you think the markup meaning is that don't exist when the document is rendered.
I wrote something that turned the HTML to a DOM tree that could get photo metadata out of a flickr page knowing just a few things about the document and URL structure; I gave up on maintaining the API-based tool and was able to get highly accurate (better than a week of work on the markup parser) extraction of Wikiiedia Commons metadata in about 20 minutes out of the HTML.
As an engineer I also know the business folk tend to design APIs to maximize their business value and minimize my business value. Usually I look at a web site, see information I want to scrape and I "just do it". With an API you tend to make a big investment before you realize exactly how restricted your access is.
Frequently it is "RPA and done" as opposed to "you can't get from here to there" with APIs in commercial practice.
Author here. I would say that even though you're grabbing the markup from an API, trying to get structured data from presentation logic still seems closer to RPA than not.
And Wikipedia might be an unique example - the markup structure is fairly stable vs. commercial SaaS.
If a competitor offered API access with the data you wanted, you would always choose the API over screen scraping. If a screen scraping business becomes big enough, they usually either partner with the existing sites (i.e. Plaid) or a competitor does.
How about legacy systems? Thats where RPA was used when I worked for a client some years back. If there was an API there’d be no question what to use but in a lot of cases that is not possible and RPA fits the bill
APIs will surely be the way forward for new, larger scale solutions. However the number of legacy, written to be temporary but never replaced systems is just stunning and anything beyond manual data entry and shuffling is still like magic for _alot_ of business types, so I see the two co-existing for years and decades to come
Most APIs I’ve encountered expose stunning amounts of implementation details, and/or are less stable over time than the visual presentation.
Of course, there are counter-examples, but if there’s no strong business reason to make the API clean and stable over time, it’s unlikely to happen. Also, for many companies, there are strong financial motivations to obfuscate the API.
Pretty silly take. APIs supplement RPA, they don’t “disrupt” it. Additionally, most B2B companies outside of tech-land have 0 interest in creating APIs. Are you going to wait for the API that never comes, or do RPA?
Also the whole “chrome has no API” is wrong. Ever heard of selenium or puppeteer?
Whether a website offers a useful API depends, in part, on whether its owner reacts to automated use of the site with encouragement, indifference, or hostility.
API's are offered by sites that either (a) actively encourage automated usage, or (b) are indifferent to users' use of their API, but provide it to users because it's easier to develop their own pages' JS that way.
Robot-style automation is a sort of "backdoor" that allows automated interaction with any website, no matter how hostile its owner is. You can't do much about automation that pretends to be a human. (Except maybe deploy a CAPTCHA, but solving a CAPTCHA is an annoyance users generally won't put up with more than once per login. And your RPA could simply defer to a human to login and solve the CAPTCHA, then take over for the automation part.)
In some domains, a competitor that offers an automated interface to the same data will outcompete a site that doesn't. But in other domains, you can't start that competitor, because the data to back the API isn't open.
For example, if we lived in a world where say Facebook provided only pre-rendered pages and had no API, you couldn't make an API to allow automated access to Facebook data. If you made a Facebook clone with all the functions of Facebook plus an API, it still wouldn't be the same site as Facebook plus an API -- the users and network effect would still all be on Facebook. Your clone would likely never be able to outcompete Facebook.
Regarding the first sentence: it really depends on whether or not it benefits the company. Most companies seem to restrict or remove their APIs once they become large enough to hold your data hostage.
9 comments
[ 3.2 ms ] story [ 30.3 ms ] threadI wrote a highly complex analysis engine for Wikipedia and Wikidata that fetched Wiki markup through the API and tried to parse the Wiki markup.
It was big mistake.
The feedback loop that Wikipedia editors work on is "does the HTML output look OK?" and not "does the markup conform to an unknown specification?" so part of the problems is fixing up "errors" that exist against what you think the markup meaning is that don't exist when the document is rendered.
I wrote something that turned the HTML to a DOM tree that could get photo metadata out of a flickr page knowing just a few things about the document and URL structure; I gave up on maintaining the API-based tool and was able to get highly accurate (better than a week of work on the markup parser) extraction of Wikiiedia Commons metadata in about 20 minutes out of the HTML.
As an engineer I also know the business folk tend to design APIs to maximize their business value and minimize my business value. Usually I look at a web site, see information I want to scrape and I "just do it". With an API you tend to make a big investment before you realize exactly how restricted your access is.
Frequently it is "RPA and done" as opposed to "you can't get from here to there" with APIs in commercial practice.
And Wikipedia might be an unique example - the markup structure is fairly stable vs. commercial SaaS.
If a competitor offered API access with the data you wanted, you would always choose the API over screen scraping. If a screen scraping business becomes big enough, they usually either partner with the existing sites (i.e. Plaid) or a competitor does.
Of course, there are counter-examples, but if there’s no strong business reason to make the API clean and stable over time, it’s unlikely to happen. Also, for many companies, there are strong financial motivations to obfuscate the API.
Also the whole “chrome has no API” is wrong. Ever heard of selenium or puppeteer?
API's are offered by sites that either (a) actively encourage automated usage, or (b) are indifferent to users' use of their API, but provide it to users because it's easier to develop their own pages' JS that way.
Robot-style automation is a sort of "backdoor" that allows automated interaction with any website, no matter how hostile its owner is. You can't do much about automation that pretends to be a human. (Except maybe deploy a CAPTCHA, but solving a CAPTCHA is an annoyance users generally won't put up with more than once per login. And your RPA could simply defer to a human to login and solve the CAPTCHA, then take over for the automation part.)
In some domains, a competitor that offers an automated interface to the same data will outcompete a site that doesn't. But in other domains, you can't start that competitor, because the data to back the API isn't open.
For example, if we lived in a world where say Facebook provided only pre-rendered pages and had no API, you couldn't make an API to allow automated access to Facebook data. If you made a Facebook clone with all the functions of Facebook plus an API, it still wouldn't be the same site as Facebook plus an API -- the users and network effect would still all be on Facebook. Your clone would likely never be able to outcompete Facebook.