This is a quite arrogant post. I do not make APIs but I can understand some needs of the API providers.
First of all you are being arrogant because you if you just want an URL and nothing more means that for you every API must be free and this is totally bullshit. Also giving you a URL and nothing more gives me no control on what you do. You could harm my service by doing too many request or god knows what and I could do (almost)nothing without hurting maybe other developers. Conclusion: think of the others before ranting.
P.S.
the code samples are there help all the people that are not great developers as you.
I could harm your service by making too many API-key signed, OAuth-signed requests too. I could harm your service by hitting your website a lot too. We have ways of dealing with people who intentionally or unintentionally launch the equivalent of denial of service attacks: you block their IPs and move on. There's no need to have a special magical way of doing it with an API.
The point is the whole concept of an API should be unnecessary. We have a way of saying where data is: URLs. We have a way of specifying what format the client wants it in: content negotiation (Accept headers). We have a way of retrieving that data: HTTP GET. There's a reason why BugMeNot exists for websites. API keys are basically pointless registration pages for access to the same data that is being published on the web.
As for the code samples: if there's little more needed than "here's the URL of our data", I don't need a code sample. I only need a code sample when it's been made ridiculously over-complicated.
(My favourite API recently: clockworksms.com - all of the other SMS providers I've looked at want me to talk to some salesman and/or read complicated docs. Clockwork just let me send an HTTPS POST message. They have an API key, sure, but they required only an email address to get it. And I can pay for credits with PayPal. Ridiculously simple. I like.)
As for arrogancy? Guilty as charged. I'll say in my defence that it's more that the data I wanted to retrieve from the service in question (which I won't name) was exceptionally simple, had no commercial value in itself, but would send referrals to the site that they could monetise (and there's no affiliate scheme, I wasn't gonna profit off this). There literally is no business reason to lock that kind of API down. It's just cargo-cult API design: everyone else has API keys, they must have a reason, so I better have that too.
That post is totally wrong, though. There's no way in which URLs with opaque IDs in are better for humans than URLs with stubs. If the stub happens to jog a users memory about the content of the post, it might be useful, and even if it doesn't sum up the post very well, it's still no worse than an opaque ID.
Try writing down "example.org/posts/1234" on a post-it note and then try writing down "example.org/posts/thing-with-a-long-name-with-lots-of-dashes". One is a lot more fun than the other.
I think the point of API keys is so that if you were to build and publish an abusive client (think badly written iOS app), it's easier to stop the DDoS, rather than having to ban thousands of end-user IPs.
This helps stop developers from putting lazy API calls on the client side instead of caching, for example.
For instance if my website includes a feed of my latest posts to a third party service I should be caching them myself and serving direct to visitors, rather than using a client side request coming from every visitor's browser.
Obviously this sounds like common sense but you might be surprised :)
Psst... Not sure if you care, but viewing your site on iPad shows links overlapping with surrounding text, some text inexplicably clipped as well as few other rendering issues.
In general, blocking IPs is not really useful anymore since far too much abuse comes over Tor, open proxies, or potentially short-lived dynamic IPs (major German ISPs let you switch IPs as often as you want) and if you care about legitimate users, you cannot block these.
The point that websites without API-key get hit by this anyway is valid though and legitimate users won't be annoyed too much by IP-blocking abusers of the API for API access only (block all Tor exit nodes for the API too, who cares ...).
You could also mention that authentication, registration etc. probably make APIs slower and more of a burden for the servers too ...
So, on the Tor front: Wikipedia has blocked Tor users from editing since, well, since forever. For the occasional Chinese dissident or whoever who actually wants to legitimately edit over Tor or another blocked IP from an account, they can request IP block exemption from an administrator.
Another solution is to actually use User-Agent strings. Nominatim, OpenStreetMap's reverse geocoder, recommends that legitimate users put an email address in their User-Agent string. So it might read "My Craptastic Mashup v1. Maintained by: <whoever@gmail.com>". If there's a legitimate problem, email them.
"cargo-cult". That's the term I was looking for. Monkey see, monkey do. Code monkey mimicry.
If thinking for yourself is "arrogance", then we need more "arrogant" people involved in web design.
It's not the "API" that is the method of providing and restricting access. It's the IP address. Any reasonably smart user can figure out the "API". They do not need a lengthy manual. A few examples, mere hints, is all that is needed.
I don't particularly care about the price. I'm using it for things like sending myself a small number of simple notifications. I have a script that monitors a web app I maintain for work, and if it isn't responding, it sends me an SMS.
When the site is down, people can't do any work. If they can't do any work, they lose clients. If they lose clients, I don't have a job anymore.
If the point of using SMS were to actually build an SMS service, I'd spend more time worrying about it and choose one of the more fully-featured SMS providers. And if it were to be a commercial thing, sure, I'd talk to their business development people. But everything starts at the micro level: what could be a business idea in six months starts as a crappy little hack now.
Here's a point of comparison. Esendex are usually considered one of the better SMS providers. How much do they cost? Oh, let's have a look at their pricing page. Doesn't tell me. Lots of blather. And a nice phone number I can ring and talk to a salesman or I can sign up for a no-commitment free trial. I don't want to talk to a salesman, I want to send a flipping text to myself in this crappy little Python script I'm writing.
In the general case, the data is going to be in an enterprise datawarehouse divided over 200 tables each of which could be terabytes in size.
The schema design will mean that each table individually is meaningless, and you will probably not understand it all without a lot of documentation.
If you were allowed to pull a table using a GET it would almost certainly be very expensive for the service provider, and might kill your download connection.
The data is there in an HTML page. I'm looking at it. I just want to stick ".xml" or ".json" on the end and see the exact same thing in a machine readable form.
Whatever complex enterprise data warehousing you are doing to get it in my browser as an HTML page is very impressive, I'm sure.
It can be as complicated as you want it to be if you want to offer something more complicated than that. But for the basic use case, I fail to see why exactly it has to be more complicated than using HTTP to GET things that are on the damn web. And all the talk of APIs makes people think it needs to be complicated when it doesn't.
On your web page you say
"I just want the data on your website in a machine readable format. XML, JSON, RDF, CSV, YAML, I don’t particularly give a fuck.
...
I don’t want a relationship with you, I just want to have a fiddle with your data."
But in your comment you say
"The data is there in an HTML page. ... I just want to stick .xml or .json on the end and see the exact same thing in a machine readable form."
In the first case, you are asking for the absence of an API. In the second case, you are asking for an API that apes the user interface.
I just wanted to point out that that's an important difference.
EDIT: It is not correct to say the data is in the web page already, it is more accurate to say the needed query result is in the web page.
No, I'm specifically making the point that content negotiation and RESTful "APIs" aren't really APIs, they are just serving data up in the same way that you are serving HTML up. The whole API thing is just a social construction we have with these bizarre conventions around it.
It may be necessary to have big complicated APIs some of the time. Great. Mostly, it's not necessary: you have a URI structure, you have pointers between records (hyperlinks), you just need a machine readable representation of the data. HTTP and web architecture already does that. You don't need a special fancy API with things I need to learn and understand.
On reading your clarification: okay. You term what I call "data" as a "query result". Then it's a merely semantic distinction you are drawing. And, I'd suggest, probably an irrelevant one from the perspective of the data consumer. There's a URL for a resource. I want the stuff there. I don't care whether it's in a database or how it's stored. That's just plumbing.
Paid APIs should have the minimum amount of bullshit to determine who pays what. Free APIs should have no bullshit. Request throttling doesn't require API keys.
Just the other day I went to use the good old fashioned twitter API to find that it had been deprecated in favor of a new one that requires that I do everything differently in order to get the same results. Really irked me. I can empathize with this guy.
Yeah. I wanted to read some tweets, but apparently I now need a stupid API token and have to do OAuth! What happened to the good old days of session-less read only-mode APIs?! :(
Justin Bieber is my internal shorthand for: Twitter's API changes are based around the fact that they care more about celebrities and media than around being a good web API citizen or whatever. ;-)
I would be very interested in knowing more about what exactly his problems were. What APIs what he trying to use? I work on backend systems and build APIs or related systems everyday so getting a rant is not of much help. :(
* Auth Tokens - needed.
* Code samples - good place to get started.
* Documentation - really, I am yet to see a good API without documentation.
* Blog - Okay not always helpful but other times its a way to tell users about new features, downtimes, changes.
* business development/relationship/ecosystem - not important but can be nice to have in case you get stuck or are building something that is more than just a crappy weekend hack.
Ha. I've had the same thoughts many times. Never thought anyone else felt this way.
At some stage, people started to get weird ideas about URL schemes (aka "API's") like it's some sort of marketing thing. API keys. WTF? Imagine if early Google was required to get an "API key" for every site they crawled. Who came up with this silly idea of "API keys"?
In the 90's, I remember decipering URL schemes in order to do crawling long before anyone used the term "API's" to describe URL's. They are usually quite predictable, since almost every website used the same software. And they're even more predictable today. Crwaling got easier. Hmmm, I wonder why.
I also remember how most URL's used to be non-descriptive. Not true anyomre. Google has changed everything. Sites _want_ to be crawled now. But funnily enough they think they can make money from it. Heh, good luck with that.
If you want to charge money, then put the data behind a password and a paywall. And watch traffic plummet.
API keys are a perfectly valid and useful concept for a situation where you, as the owner of a (possibly paid) account can create those keys and hand them to 3rd parties or programs that provide a service using (reading only) some of the data related to your account (you don't want to give them
your login info, right?).
I have to admit, I do actually really like the EVE API a lot. It's very simple, but the community has done amazing things with it: iOS and Android clients, EVE Fitting Tool, EVEMon and many other things.
Oh god, I'm being tempted back to exploring New Eden again.
I just noticed the "Sign in" link on the top left. Clicking that opens up a windows popup with Mozilla's new Persona login feature. I didn't enter my email address but instead closed the popup. Yet the site refreshed and shows: "You are signed in as ". Is this normal behavior?
I'm not very familiar with Mozilla's Persona. Been meaning to try it but I've got to find the time and a suitable project.
Nope, it isn't. I'm just trying it out. If you did sign in, you wouldn't get much. It's just for writing posts. ;-)
It's a case of building my site from scratch, and I wanted a super-light-weight authentication system. So I picked Persona. (Incidentally, I picked it because unlike Twitter or Facebook or whatever, I didn't need any API keys to get started.) There are a few bugs with my implementation. But as it's just to authenticate me to access the admin panel, it's no biggie.
It's a real shame that this rant has made it to the number one spot (as I of now). I don't mind rants, but there's nothing to really take away from this post. The author doesn't provide any motivation as to why any of these things are problematic, nor does he try to appreciate why some of these exist for some APIs. Rather than offensively ranting at people providing you with data, usually at no cost, why not try and provide examples of good web service design? This post could have been a whole lot more insightful.
I didn't plan for it to get to the top of Hacker News. It's an off-the-cuff rant, basically a long tweet. I could write a long and detailed post about it, but this was written in the middle of hacking, while waiting for a long database query to run.
It's an extreme position, but such views are still valuable.
I wouldn't advocate implementing an API to those bullet points, but it's certainly worth remembering when you are making an API that you aren't making it for you, the service provider.
As nothing but a reminder of that, the post is valid. It's certainly why I upvoted it... not because I think it's deeply insightful, but that it's good to have the reminder that on the other side of an API is a developer who I might be frustrating and causing problems for.
What a ridiculously limited view of what apis do. What about being able to make a credit card payment, customize a google map, or do anything that involves interacting with a user account?
If the complaint is that there are needlessly complicated apis out there, it's a valid one. But the solution completely ignores any problem that does't fall into a very simple data provider model.
hmmm... this post is bewildering. Is there a particular API this is targeted at ? Some examples for each complaint would have been great to add more context. Lets look at your points...
I don’t want to build an application for your service.
I don’t want a relationship.
I don’t want to talk to any business development people.
I don’t want to read your developer blog.
I don’t want to participate in your “ecosystem”.
I don't recall any API agreement that dictates any of those. No one is forcing you to read a blog or change your relationship in facebook to 'in a relationship with twitter API'. I don't get this.
I don’t want to get an API key.
Fat chance. Some APIs need rate limiting / authentication. What do you have against obtaining an API key ? Its not that complicated. Most APIs provide BASIC auth or let you get OAuth tokens.
I don’t want to read your code samples.
ಠ_ಠ
What likely happened was that you were pissed with the API that someone provided. It had complicated requests where HTTP methods and simple structures alone were not enough to describe the request and to add to that you were given a freakin complicated way to retrieve auth tokens. You then decided to take it out on all API providers without providing enough context on your rant.
It's not taking anything out on anyone. It's a joke, a mild diversionary amusement, that for some reason that I can't understand people seem to think should be the #1 post on Hacker News.
My main issue is that the article doesn't make logical sense and you signed it off:
Tom, a grumpy developer.
I think you are tapping into the negative sentiments of some other recent articles. Namely the "Do You Really Want to be Doing This When You're 50?" article.
I find it difficult to understand how you have come to this stance; if they are providing the API as a free service, why should they put themselves out in any way or more specifically risk the chance of abuse?
Also, on one hand you ask for unadulterated access to their data, but you explicitly don't want their free code samples? - "Give me free stuff, but not that, I don't want that!"
51 comments
[ 3.4 ms ] story [ 102 ms ] threadP.S. the code samples are there help all the people that are not great developers as you.
Currently, no API key is required, but this will likely change so we can better monitor usage and enforce the Terms of Use (below).
I could harm your service by making too many API-key signed, OAuth-signed requests too. I could harm your service by hitting your website a lot too. We have ways of dealing with people who intentionally or unintentionally launch the equivalent of denial of service attacks: you block their IPs and move on. There's no need to have a special magical way of doing it with an API.
The point is the whole concept of an API should be unnecessary. We have a way of saying where data is: URLs. We have a way of specifying what format the client wants it in: content negotiation (Accept headers). We have a way of retrieving that data: HTTP GET. There's a reason why BugMeNot exists for websites. API keys are basically pointless registration pages for access to the same data that is being published on the web.
As for the code samples: if there's little more needed than "here's the URL of our data", I don't need a code sample. I only need a code sample when it's been made ridiculously over-complicated.
(My favourite API recently: clockworksms.com - all of the other SMS providers I've looked at want me to talk to some salesman and/or read complicated docs. Clockwork just let me send an HTTPS POST message. They have an API key, sure, but they required only an email address to get it. And I can pay for credits with PayPal. Ridiculously simple. I like.)
As for arrogancy? Guilty as charged. I'll say in my defence that it's more that the data I wanted to retrieve from the service in question (which I won't name) was exceptionally simple, had no commercial value in itself, but would send referrals to the site that they could monetise (and there's no affiliate scheme, I wasn't gonna profit off this). There literally is no business reason to lock that kind of API down. It's just cargo-cult API design: everyone else has API keys, they must have a reason, so I better have that too.
This helps stop developers from putting lazy API calls on the client side instead of caching, for example.
For instance if my website includes a feed of my latest posts to a third party service I should be caching them myself and serving direct to visitors, rather than using a client side request coming from every visitor's browser.
Obviously this sounds like common sense but you might be surprised :)
The point that websites without API-key get hit by this anyway is valid though and legitimate users won't be annoyed too much by IP-blocking abusers of the API for API access only (block all Tor exit nodes for the API too, who cares ...).
You could also mention that authentication, registration etc. probably make APIs slower and more of a burden for the servers too ...
Another solution is to actually use User-Agent strings. Nominatim, OpenStreetMap's reverse geocoder, recommends that legitimate users put an email address in their User-Agent string. So it might read "My Craptastic Mashup v1. Maintained by: <whoever@gmail.com>". If there's a legitimate problem, email them.
If thinking for yourself is "arrogance", then we need more "arrogant" people involved in web design.
It's not the "API" that is the method of providing and restricting access. It's the IP address. Any reasonably smart user can figure out the "API". They do not need a lengthy manual. A few examples, mere hints, is all that is needed.
When the site is down, people can't do any work. If they can't do any work, they lose clients. If they lose clients, I don't have a job anymore.
If the point of using SMS were to actually build an SMS service, I'd spend more time worrying about it and choose one of the more fully-featured SMS providers. And if it were to be a commercial thing, sure, I'd talk to their business development people. But everything starts at the micro level: what could be a business idea in six months starts as a crappy little hack now.
Here's a point of comparison. Esendex are usually considered one of the better SMS providers. How much do they cost? Oh, let's have a look at their pricing page. Doesn't tell me. Lots of blather. And a nice phone number I can ring and talk to a salesman or I can sign up for a no-commitment free trial. I don't want to talk to a salesman, I want to send a flipping text to myself in this crappy little Python script I'm writing.
It is a way of naming a data query.
HTTP GET is not a way of retrieving data.
It is a a way of launching a query.
In the general case, the data is going to be in an enterprise datawarehouse divided over 200 tables each of which could be terabytes in size.
The schema design will mean that each table individually is meaningless, and you will probably not understand it all without a lot of documentation.
If you were allowed to pull a table using a GET it would almost certainly be very expensive for the service provider, and might kill your download connection.
APIs were invented to deal with these cases.
Whatever complex enterprise data warehousing you are doing to get it in my browser as an HTML page is very impressive, I'm sure.
It can be as complicated as you want it to be if you want to offer something more complicated than that. But for the basic use case, I fail to see why exactly it has to be more complicated than using HTTP to GET things that are on the damn web. And all the talk of APIs makes people think it needs to be complicated when it doesn't.
But in your comment you say "The data is there in an HTML page. ... I just want to stick .xml or .json on the end and see the exact same thing in a machine readable form."
In the first case, you are asking for the absence of an API. In the second case, you are asking for an API that apes the user interface.
I just wanted to point out that that's an important difference.
EDIT: It is not correct to say the data is in the web page already, it is more accurate to say the needed query result is in the web page.
It may be necessary to have big complicated APIs some of the time. Great. Mostly, it's not necessary: you have a URI structure, you have pointers between records (hyperlinks), you just need a machine readable representation of the data. HTTP and web architecture already does that. You don't need a special fancy API with things I need to learn and understand.
On reading your clarification: okay. You term what I call "data" as a "query result". Then it's a merely semantic distinction you are drawing. And, I'd suggest, probably an irrelevant one from the perspective of the data consumer. There's a URL for a resource. I want the stuff there. I don't care whether it's in a database or how it's stored. That's just plumbing.
At some stage, people started to get weird ideas about URL schemes (aka "API's") like it's some sort of marketing thing. API keys. WTF? Imagine if early Google was required to get an "API key" for every site they crawled. Who came up with this silly idea of "API keys"?
In the 90's, I remember decipering URL schemes in order to do crawling long before anyone used the term "API's" to describe URL's. They are usually quite predictable, since almost every website used the same software. And they're even more predictable today. Crwaling got easier. Hmmm, I wonder why.
I also remember how most URL's used to be non-descriptive. Not true anyomre. Google has changed everything. Sites _want_ to be crawled now. But funnily enough they think they can make money from it. Heh, good luck with that.
If you want to charge money, then put the data behind a password and a paywall. And watch traffic plummet.
Example: http://wiki.eveonline.com/en/wiki/EVE_API_Functions
Oh god, I'm being tempted back to exploring New Eden again.
For non-EVE players: http://wiki.eveonline.com/en/wiki/3rd_party_tools
I'm not very familiar with Mozilla's Persona. Been meaning to try it but I've got to find the time and a suitable project.
It's a case of building my site from scratch, and I wanted a super-light-weight authentication system. So I picked Persona. (Incidentally, I picked it because unlike Twitter or Facebook or whatever, I didn't need any API keys to get started.) There are a few bugs with my implementation. But as it's just to authenticate me to access the admin panel, it's no biggie.
I apologise for the lack of insight.
I wouldn't advocate implementing an API to those bullet points, but it's certainly worth remembering when you are making an API that you aren't making it for you, the service provider.
As nothing but a reminder of that, the post is valid. It's certainly why I upvoted it... not because I think it's deeply insightful, but that it's good to have the reminder that on the other side of an API is a developer who I might be frustrating and causing problems for.
If the complaint is that there are needlessly complicated apis out there, it's a valid one. But the solution completely ignores any problem that does't fall into a very simple data provider model.
I don’t want to build an application for your service.
I don’t want a relationship.
I don’t want to talk to any business development people.
I don’t want to read your developer blog.
I don’t want to participate in your “ecosystem”.
I don't recall any API agreement that dictates any of those. No one is forcing you to read a blog or change your relationship in facebook to 'in a relationship with twitter API'. I don't get this.
I don’t want to get an API key.
Fat chance. Some APIs need rate limiting / authentication. What do you have against obtaining an API key ? Its not that complicated. Most APIs provide BASIC auth or let you get OAuth tokens.
I don’t want to read your code samples.
ಠ_ಠ
What likely happened was that you were pissed with the API that someone provided. It had complicated requests where HTTP methods and simple structures alone were not enough to describe the request and to add to that you were given a freakin complicated way to retrieve auth tokens. You then decided to take it out on all API providers without providing enough context on your rant.
[EDIT] - formatting
Tom, a grumpy developer.
I think you are tapping into the negative sentiments of some other recent articles. Namely the "Do You Really Want to be Doing This When You're 50?" article.
Perhaps people are laughing at you, not with you?
Also, on one hand you ask for unadulterated access to their data, but you explicitly don't want their free code samples? - "Give me free stuff, but not that, I don't want that!"