> However, a technically-poor product can make it nearly impossible to build an elegant API. That’s because API design usually tracks the “basic resources” of a product (for instance, Jira’s resources would be issues, projects, users and so on). When those resources are set up awkwardly, that makes the API awkward as well.
One issue I have with weird resources are those that feel like unnecessary abstraction. It makes it hard for the human to read and understand intuitively, especially someone new to these set of APIs. Also, it makes it so much harder to troubleshoot during an incident.
I think the only thing here that I don't agree with is that internal users are just users. Yes, they may be more technical - or likely other programmers, but they're busy too. Often they're building their own thing and don't have the time or ability to deal with your API churning.
If at all possible, take your time and dog-food your API before opening it up to others. Once it's opened, you're stuck and need to respect the "never break userspace" contract.
A big difference is you can tell internal users to update or else. It’s not free and should be reserved for good business reasons, but it can happen on a shorter time frame as you have the internal organisation to enforce it.
It’s not really an option in the same way with end users or customers, as they aren’t part of your organisation, by definition.
I'm a bit of a different opinion on API versioning, but I can see the argument. I definitely disagree about idempotency: it's NOT optional. You don't have to require idempotency tokens for each request, but there should be an option to specify them. Stripe API clients are a good example here, they automatically generate idempotency tokens for you.
Things that's missing from this list but that were important for me at some points:
1. Deadlines. Your API should allow to specify the deadline after which the request is no longer going to matter. The API implementation can use this deadline to cancel any pending operations.
2. Closely related: backpressure and dependent services. Your API should be designed to not overload its own dependent services with useless retries. Some retries might be useful, but in general the API should quickly propagate the error status back to the callers.
3. Static stability. The system behind the API should be designed to fail static, so that it retains some functionality even if the mutating operations fail.
The reminder to "never break userspace" is good, but people never bring up the other half of that statement: "we can and will break kernel APIs without warning".
It illustrates that the reminder isn't "never change an API in a way that breaks someone", it's the more nuanced "declare what's stable, and never break those".
In software engineering the statement "interfaces, not implementations" has been used for a long time (certainly at least Robert "Uncle Bob" C. Martin started teaching), which is a generalization on the "we don't break userspace". In essence it cooks down to declaring an interface without announcing or depending on the implementation. With OOP languages like C++, a code base would aggressively use interfaces as types, never concrete class types (which implement the interface), so that it can make it easier to reason about how and whether the program behaves when one implementation of an interface is swapped for another.
With Linux, which is a C codebase by and large, they load and pass pointers to structures to kernel procedures which can do as they please -- as long as the documentation on said structures (which usually says which fields and how are retained with which values and so on) remains unchanged. That's their "object oriented programming" (yeah, I know Linus would likely have hated the comparison).
Anyone else old enough to remember when "API" also meant something that had nothing to do with sending and receiving JSON over HTTP? In some cases, you could even make something that your users would install locally, and use without needing an Internet connection.
Cursor based pagination was mentioned. It has another useful feature: If items have been added between when a user loads the page and hits the next button, index based pagination will give you some already viewed items from the previous page.
Cursor based pagination (using the ID of the last object on the previous page) will give you a new list of items that haven't been viewed. This is helpful for infinite scrolling.
The downside to cursor based pagination is that it's hard to build a jump to page N button.
Most people who see "API" today only think "it's a web app I send a request to, and I pass some arguments and set some headers, then check some settings from the returned headers, then parse some returned data."
But "API" means "Application Programming Interface". It was originally for application programs, which were... programs with user interfaces! It comes from the 1940's originally, and wasn't referred to for much else until 1990. APIs have existed for over 80 years. Books and papers have been published on the subject that are older than many of the people reading this text right now.
What might've those older APIs been like? What were they working with? What was their purpose? How did those programmers solve their problems? How might that be relevant to you?
You are talking in past tense, but there are still many non-web APIs. Every software library has an API. I still find it incredibly annoying that the web folks have hijacked the term "API" as a short hand for "web API".
Really?! That’s amazingly early. There were barely even subroutine libraries at that time. I’d love to see an example of "Application Programming Interface" from that time.
(I don’t remember seeing the term until Microsoft started using it when talking about Windows in the 1990s; before then it was things like library functions or supervisor calls - but I didn’t have much experience at that point so I was probably missing some of the more collar-and-tie programmer lingo.)
You are talking as if it were a thing of a past. Yet I am 20 and when I read API, only ever think of it as in API/ABI. I don't think a protocol endpoint is an API.
> How should you store the key? I’ve seen people store it in some durable, resource-specific way (e.g. as a column on the comments table), but I don’t think that’s strictly necessary. The easiest way is to put them in Redis or some similar key/value store (with the idempotency key as the key).
I'm not sure how would storing a key in Redis achieve idempotency in all failure cases. What's the algorithm? Imagine a server handling the request is doing a conditional write (like SET key 1 NX), and sees that the key is already stored. What then, skip creating a comment? Can't assume that the comment had been created before, since the process could have been killed in-between storing the key in Redis and actually creating the comment in the database.
An attempt to store idempotency key needs to be atomically committed (and rolled back in case it's unsuccessful) together with the operation payload, i.e. it always has to be a resource-specific id. For all intents and purposes, the idempotency key is the ID of the operation (request) being executed, be it "comment creation" or "comment update".
> many of your users will not be professional engineers. They may be salespeople, product managers, students, hobbyists, and so on.
This is not just true for authentication.
If you work in a business setting, your APIs will be used by the most random set of users. They be able to google for how to call your api in python, but not be able to do things like converting UTC to their local time zone.
They suggest storing the idempotency key in redis. Seems like if possible, you should store them in whatever system you are writing to in a single transaction with the write mutations.
The quality of the API is inversely correlated to how difficult it is to obtain API documentation. If you are only going to get the API documentation after signing a contract, just assume it’s dismally bad.
Pragmatism rules here, but yeah - the common way to do this (at least if you have keys generatable by the client), eg. using REST, is to not allow POSTs, but only PUT. Most APIs I've seen use PUT solely for updates (of existing items), but as is obvious from the wording it's not the original intention.
API versioning mostly just means things perpetually stuck at v1. You might have the intention to change things up, but you never will.
Putting version numbers in a URL is a bit of a kludge. v1 is the most common version, by far, you will ever see in a url. v2 is rare. v3 is more common strangely. I don't think I've seen a v4 or v5 or higher in the wild very often. That's just not a thing.
My theory is that v1 is the quick and dirty version that developers would like to forget exists. v2 is the "now we know what we're doing!" version and that's usually quickly followed by v3 because if you can change your mind once you can do it twice. After which people just tell developers to quit messing with the API already and keep things stable. v4 and v5 never happen.
Another observation is that semantic versioning for API urls here seems rare. Reason: it's inconvenient for clients to have to update all their URLs every time some developer changes their mind. Most clients will hard code the version. Because it never changes. And because it is hard coded, changing the version becomes inconvenient.
My attitude towards URL based versioning is that you could do it but it's not a tool that you get to use much. Therefore you can safely skip it and it won't be a problem. And in the worst case where you do need it, you can easily add a v2 URL space anyway. But you probably never will as you are unlikely to deprecated the entirety of your API.
There are other ways to deal with deprecating APIs. You can just add new paths or path prefixes in your API as needed. You can use a different domain. Or you can just remove them after some grace period. It depends. Versioning is more aspirational than actually a thing with APIs.
We do version our API but via client headers. Our API client sends a version header. And we check it server side and reject older versions with a version conflict response (409). This enables us to force users of our app to update to something we still support. The version number of our client library increments regularly. Anything falling behind too far we reject. This doesn't work for all use cases. But for a web app this is completely fine.
Pagination: do not force me to drink from a paginated coffee stir. I do not want 640 B of data in a response, and then have to send another response for the next 640 B. And often, pagination means the calls are serialized, so I'm just doing nothing but waiting for round trip latency after round trip latency for the next meager 640 B of data.
Azure I'm looking at you. Many of their services do this, but Blob storage is something else: I've literally gotten information-free responses there. (I.e., 0 B of actual data. I wish I could say 0 B were used to transfer it.)
When you're designing, think about how big a record/object/item is, and return a reasonable number of them in a page. For programmatic consumers who want to walk the dataset, a 640 KiB response is really not that big, and I've seen so many times responses orders of magnitude less, because someone thought "100 items is a good page size, right?" and 100 items was like 4 KiB of data.
> If you have thirty API endpoints, every new version you add introduces thirty new endpoints to maintain. You will rapidly end up with hundreds of APIs that all need testing, debugging, and customer support.
You version the one thing that's changing.
As much as I hate the /v2/... form of versioning, nobody reversions all the /v1/... APIs just because one API needed a /v2. /v2 is ghost town, save for the /v2 APIs.
It’s certainly been my experience that page sizes should be bigger than you initially expect. Paginated endpoints are typically iterated all the way through meaning you’re going to return that data anyway. May as well save the additional overhead from multiple requests.
Not implementing pagination at the outset can be problematic, however. If you later want to paginate data (e.g. if the size of your data grows) then it’s going to be a breaking change to implement that later. Big page sizes but with pagination can be a reasonable balance.
> You should let people use your APIs with a long-lived API key
This is an extremely unpopular opinion, but I would go even further. I think you should let people use your API with just an username and a password.
It should by no means be the only way people can use your API. Put very low users-per-IP rate limits on that approach if you want to, to force lazy but professional software developers to go the oAuth route before their app gets to production. For one-off scripts though, APIs that let you do this are a breath of fresh air.
If your API is based on API keys, you will be tempted to do things that really annoy new users of that API. People don't want to tell you what their app name is, they don't know that yet. They're certainly not picking a purpose they need this API for from a list of five, not if it doesn't include "completing a classroom assignment I don't really care about and want to finish as quickly as possible." They for sure don't yet know what scopes they might possibly need, even if to you, their names are descriptive and obvious. If you allow user-password authentication, you take away the ability to shoot yourself in the foot in this way.
47 comments
[ 2.7 ms ] story [ 70.1 ms ] threadSigh... I wish this were not true. It's a shame that no alternatives have emerged so far.
You cannot predict the future and chances are there will be some breaking change forced upon you by someone or something out of your control.
One issue I have with weird resources are those that feel like unnecessary abstraction. It makes it hard for the human to read and understand intuitively, especially someone new to these set of APIs. Also, it makes it so much harder to troubleshoot during an incident.
If at all possible, take your time and dog-food your API before opening it up to others. Once it's opened, you're stuck and need to respect the "never break userspace" contract.
It’s not really an option in the same way with end users or customers, as they aren’t part of your organisation, by definition.
Things that's missing from this list but that were important for me at some points:
1. Deadlines. Your API should allow to specify the deadline after which the request is no longer going to matter. The API implementation can use this deadline to cancel any pending operations.
2. Closely related: backpressure and dependent services. Your API should be designed to not overload its own dependent services with useless retries. Some retries might be useful, but in general the API should quickly propagate the error status back to the callers.
3. Static stability. The system behind the API should be designed to fail static, so that it retains some functionality even if the mutating operations fail.
It illustrates that the reminder isn't "never change an API in a way that breaks someone", it's the more nuanced "declare what's stable, and never break those".
With Linux, which is a C codebase by and large, they load and pass pointers to structures to kernel procedures which can do as they please -- as long as the documentation on said structures (which usually says which fields and how are retained with which values and so on) remains unchanged. That's their "object oriented programming" (yeah, I know Linus would likely have hated the comparison).
Versioning, etc. matter (or don’t) for binary UDP APIs (aka protocols) just as much as for any web API.
I'd like to introduce more fields or flags to control the behavior as params, not asking user to change the whole base url for single new API.
Cursor based pagination (using the ID of the last object on the previous page) will give you a new list of items that haven't been viewed. This is helpful for infinite scrolling.
The downside to cursor based pagination is that it's hard to build a jump to page N button.
But "API" means "Application Programming Interface". It was originally for application programs, which were... programs with user interfaces! It comes from the 1940's originally, and wasn't referred to for much else until 1990. APIs have existed for over 80 years. Books and papers have been published on the subject that are older than many of the people reading this text right now.
What might've those older APIs been like? What were they working with? What was their purpose? How did those programmers solve their problems? How might that be relevant to you?
Really?! That’s amazingly early. There were barely even subroutine libraries at that time. I’d love to see an example of "Application Programming Interface" from that time.
(I don’t remember seeing the term until Microsoft started using it when talking about Windows in the 1990s; before then it was things like library functions or supervisor calls - but I didn’t have much experience at that point so I was probably missing some of the more collar-and-tie programmer lingo.)
I'm not sure how would storing a key in Redis achieve idempotency in all failure cases. What's the algorithm? Imagine a server handling the request is doing a conditional write (like SET key 1 NX), and sees that the key is already stored. What then, skip creating a comment? Can't assume that the comment had been created before, since the process could have been killed in-between storing the key in Redis and actually creating the comment in the database.
An attempt to store idempotency key needs to be atomically committed (and rolled back in case it's unsuccessful) together with the operation payload, i.e. it always has to be a resource-specific id. For all intents and purposes, the idempotency key is the ID of the operation (request) being executed, be it "comment creation" or "comment update".
This is not just true for authentication. If you work in a business setting, your APIs will be used by the most random set of users. They be able to google for how to call your api in python, but not be able to do things like converting UTC to their local time zone.
The quality of the API is inversely correlated to how difficult it is to obtain API documentation. If you are only going to get the API documentation after signing a contract, just assume it’s dismally bad.
I worked in an org where idempotency meant: if it threw an exception this time, it needs to throw the same exception everytime.
Bravo!
Putting version numbers in a URL is a bit of a kludge. v1 is the most common version, by far, you will ever see in a url. v2 is rare. v3 is more common strangely. I don't think I've seen a v4 or v5 or higher in the wild very often. That's just not a thing.
My theory is that v1 is the quick and dirty version that developers would like to forget exists. v2 is the "now we know what we're doing!" version and that's usually quickly followed by v3 because if you can change your mind once you can do it twice. After which people just tell developers to quit messing with the API already and keep things stable. v4 and v5 never happen.
Another observation is that semantic versioning for API urls here seems rare. Reason: it's inconvenient for clients to have to update all their URLs every time some developer changes their mind. Most clients will hard code the version. Because it never changes. And because it is hard coded, changing the version becomes inconvenient.
My attitude towards URL based versioning is that you could do it but it's not a tool that you get to use much. Therefore you can safely skip it and it won't be a problem. And in the worst case where you do need it, you can easily add a v2 URL space anyway. But you probably never will as you are unlikely to deprecated the entirety of your API.
There are other ways to deal with deprecating APIs. You can just add new paths or path prefixes in your API as needed. You can use a different domain. Or you can just remove them after some grace period. It depends. Versioning is more aspirational than actually a thing with APIs.
We do version our API but via client headers. Our API client sends a version header. And we check it server side and reject older versions with a version conflict response (409). This enables us to force users of our app to update to something we still support. The version number of our client library increments regularly. Anything falling behind too far we reject. This doesn't work for all use cases. But for a web app this is completely fine.
Azure I'm looking at you. Many of their services do this, but Blob storage is something else: I've literally gotten information-free responses there. (I.e., 0 B of actual data. I wish I could say 0 B were used to transfer it.)
When you're designing, think about how big a record/object/item is, and return a reasonable number of them in a page. For programmatic consumers who want to walk the dataset, a 640 KiB response is really not that big, and I've seen so many times responses orders of magnitude less, because someone thought "100 items is a good page size, right?" and 100 items was like 4 KiB of data.
> If you have thirty API endpoints, every new version you add introduces thirty new endpoints to maintain. You will rapidly end up with hundreds of APIs that all need testing, debugging, and customer support.
You version the one thing that's changing.
As much as I hate the /v2/... form of versioning, nobody reversions all the /v1/... APIs just because one API needed a /v2. /v2 is ghost town, save for the /v2 APIs.
Not implementing pagination at the outset can be problematic, however. If you later want to paginate data (e.g. if the size of your data grows) then it’s going to be a breaking change to implement that later. Big page sizes but with pagination can be a reasonable balance.
This is an extremely unpopular opinion, but I would go even further. I think you should let people use your API with just an username and a password.
It should by no means be the only way people can use your API. Put very low users-per-IP rate limits on that approach if you want to, to force lazy but professional software developers to go the oAuth route before their app gets to production. For one-off scripts though, APIs that let you do this are a breath of fresh air.
If your API is based on API keys, you will be tempted to do things that really annoy new users of that API. People don't want to tell you what their app name is, they don't know that yet. They're certainly not picking a purpose they need this API for from a list of five, not if it doesn't include "completing a classroom assignment I don't really care about and want to finish as quickly as possible." They for sure don't yet know what scopes they might possibly need, even if to you, their names are descriptive and obvious. If you allow user-password authentication, you take away the ability to shoot yourself in the foot in this way.