A possible reason is the fear of losing intellectual property. Providing a machine readable format makes duplication much easier (although you'd still have to deal with the implementation). The funny part is that the Oracle vs Google API Lawsuit in effect sets a precedent that protects API specs - so this should really not be the case.
Not a single mention of the human element, which I will theorize is the main reason.
Psychologically, looking at a formal specification is pretty intimidating. How many people do you know who enjoy sitting down with a cup of coffee/tea and their favorite time-worn leather-bound copy of RFC7231?
Most developers I know love getting something to work. They love the feeling of having mastered something to the point that just reading "GET /widgets" is enough for them to go off to curl and implement the rest ad-hoc as they're visually inspecting the JSON and occasionally referring back to the docs.
Creating formal specs is probably the most efficient way to go in a purely computational sense, but reading specs is hard work and it's not especially inviting or rewarding, especially to a class of workers that would much rather jump in, get their hands dirty, and figure it out themselves. It's why developers love frameworks/APIs that just feel simple and familiar. The low cognitive overhead, and low barrier to entry gets you into a wonderfully endorphin-driven flow that's hard to replicate.
Formal specifications are great. But they won't likely get read by people. And people, not machines, are going to drive your profit as a business offering a shiny API.
So should you focus on your formal spec first, or human-readable API documentation? Probably the latter. And good luck convincing the business owners that a spec is a high priority now that you technically have everything covered.
Hi, the author here. I think its a good point you make. However, I believe being based on JSON or YAML, many of the specification formats provide at least some degree of readability. Also, one idea of them is to provide the basis for creating consistent, human-targeted documentations (see the right side of http://editor.swagger.io/). One might say these documentations are not good enough, but that would be a discussion about the quality of the tooling, rather than an argument against formal specifications per se.
I agree with the argument: developers like reading web pages that describe how to use API, covering rationale of API design, gotchas, code samples.
However, what if the specs were able to capture what is in such human-readable pages (for example https://docs.cloudant.com/)? What if API spec could be automatically derived out of HTML pages? Embed some metadata in HTML page and then we have both cake (human-readable) and eat it (machine-readable)?
That sounds very much like [API Blueprints](https://apiblueprint.org/), which [Apiary](https://apiary.io/) back. I tried to get the organisation I work for to standardise on them (being able to run tests from a versioned API specification seemed like a fantastic way to hit two birds with one stone), but sadly, couldn't convince enough people.
Mostly inertia. We're a large-ish company; I work in a remote office away from the majority of dev, and technical direction needs marketing.
We ended up using Swagger, which doesn't solve as many of our problems (having documentation generated from code isn't as robust from a testing POV, particularly when your RESTful API is versioned), but my "perfect" is the enemy of the company's "good" :).
Cool projects, thanks for pointing them out! So, the scrape-html-to Swagger one would basically allow you to keep Swagger up to date as you change the human-readable API documentation? In that case, someone still has to make sure the API documentation is in sync with the source code, right?
I don't think anyone is arguing that machine readable specs should take the place of human readable documentation. Specs are helpful for testing, integrations, and generating HTML docs.
I agree the API provider's primary concern should be Developer Experience, and understand why creating a spec often takes a back seat. But I suspect that soon it will be seen as critical for any production grade API to have a machine readable spec.
I view specs as analogous to unit tests - it doesn't get you closer to a deliverable product, but it makes the development lifecycle much less painful.
The IETF's RFCs are hardly ever "formal". At best they have a formal grammar, but never a formal semantics.
And I disagree that formal specifications are intimidating. A formal specification is just code, so if you have trouble understanding something, you can enlist the computer's help. On the other hand, if an informal specification isn't clear to you, you are more or less screwed.
I'm wondering if there's actually a widespread demand/need for public API specs.
Contentful's API specification (in the blue print format) has been open for a while as a side effect of opening up our documentation. I'm not aware of any user ever using for anything but the generated documentation (and there have been bugs in the past that would have made things like generated test servers break quite nicely).
What do you think about automatic generation of client-code in various languages?
With regards to the bugs, were they a result of the implementation and the specification getting out of sync? Would automatic means for creating / maintaining specs be able to help here?
I spend the last few weeks trying to find an appropriate SDL for my project but have otherwise no experience in creating APIs.
In my opinion, API specs should probably be generated from code and its documentation, and not the reverse. This is because it is the code and not the spec which is subject to most change and only the endpoint interfaces/signatures can be generated. API specifications could also be more easily transformed by middleware (e.g. when renaming endpoints, combining APIs) if they were machine generated.
As was described above, only few people are acquainted with service description languages, and deeper changes in their semantics over time or the choice of an alternative would require a manual rewrite. Also, some SDLs seem to impose their own higher level ontologies and name spaces, which is another major obstacle to usage for simple systems.
First, it is too bad that you spent weeks and probably didn't find the "right" solution (because it doesn't seem to exist).
Second, I like your enthusiasm for specs from code ... the problem (opportunity?) today is that the specifications intertwine human readable descriptions / summaries with the actual endpoint / payload information.
I would hate for my devops pipeline to kick-off because somebody fixed a typo in some human focused in-line documentation. I personally don't think it belongs in the code but absolutely agree with the spirit of your comment (if you keep them separate it is more work and they will get out of sync!).
The best luck I have is working within the specification (i.e. Swagger) and avoiding doing things that don't model well. This seems backwards, but for most of the development I've been involved with to date it has served the projects well.
Interested if others have had luck embedding API specs into code.
Mixing code with documentation is one concern. Another issue is that extracting the information needed to create a Swagger from source code is hard, especially when using dynamic languages (JavaScript, Python). One approach is to use dedicated annotators to relate code with parts of an API spec - but then you have the first problem again...
As an API provider, would indexing and discovery of your APIs be of value to you?
As a developer with discovery may enable synthesis and composition of APIs.
Sites like ProgrammableWeb (http://www.programmableweb.com) and API Harmony (http://apiharmony-open.mybluemix.net) would do better if they could understand what changed and what is available.
I lead the API Harmony team at IBM Research and this information would be extremely valuable.
Our [API](https://www.modio.se/pages/api-doc.html) documentation is public, but that makes for little value, as identification is made using TLS certificates.
Yes, the API is public, but you need a Cert bound to your user data to get anything from it.
Spidler, what you show is nice, to the point documentation for human consumption. The API specification mentioned is related to things like OpenAPI Specification (http://swagger.io/specification/) or RAML (http://raml.org) which are used to model APIs in a machine readable way and then lend to automating things like documentation, SDKs or mock test servers.
This is an orthogonal issue to whether or not the API itself is publicly accessible.
I think API owner need to see some value in publishing spec.
And SDK generation, not a strong argument for publishing since it can be done on API owner side with more control and better quality.
IMHO key component here is automatic integration, you simply publish a link to your spec and you magically have integration with a number of 3rd-party tools/services.
IvanGoncharov, you've done much work in this space in trying to innovate around Web APIs (thanks for that!).
I think realityking nailed it in terms of the API community needing to define compelling use cases. The catch 22 here is that it is hard for people to innovate around API specifications when they are so hard to get! Per the SDKs ... what if there was a NPM.org or rubygems equivalent of high quality SDKs that get automatically generated whenever I push a new API specification? That seems like it could be cool.
And your example of automated composition is fantastic! Curious if you are able to share how much work it is for you to get access to the specifications in order to make this possible? If we invented a common practice to publish something like api.company.com/spec.yaml could you have spent a lot more time on making cool integrations vs mining specs?
If you've a Swagger/OpenAPI spec, you can use Swagger Codegen (https://github.com/swagger-api/swagger-codegen), a free and open-source project, to generate API clients (SDKs) in C#, Ruby, PHP, Java, ObjC, etc.
> The catch 22 here is that it is hard for people to innovate around API specifications when they are so hard to get!
Yes, this is exactly the problem which I try to solve with my collection.
> Per the SDKs ... what if there was a NPM.org or rubygems equivalent of high quality SDKs that get automatically generated whenever I push a new API specification?
You can run the same tool on API owner side you don't need to publish your spec for that. I did a couple of interviews with API owner and they fear to loose control over SDKs, Docs, etc.
> Curious if you are able to share how much work it is for you to get access to the specifications in order to make this possible?
No, it's public info since my entire work is open-sourced under MIT license. I try to make process automatic as much as possible, so it starts from scraping. For example, I scrape Github for Swagger specs: https://morph.io/IvanGoncharov/API_specifications
When I need to filter test, example, etc. APIs and find specs for real-life APIs. But get spec source is just first step, I need to fix errors in it(~80% have them), get additional info(logo, link to API key registration, etc.).
But most time consuming is update them every day.
My update/convert/validation scripts not ideal so I need to manually inspect all changes.
> ould you have spent a lot more time on making cool integrations vs mining specs?
Actually not I learn a lot from it, for example, you can fix all mistakes in API owner specs. Instead, you should give them a tool which will automatically inspect API spec and output errors/warnings/recommendations/hints. Together with my friend we working on OpenAPI/Swagger linter.
If you want to discuss more here is my Skype(ivangon4arov) and Hangouts(ivan.goncharov.ua@gmail.com) or APIs.guru public chat https://gitter.im/APIs-guru/api-models
One of the things that striked me when I worked on my first API was that all of service registration, search, payments and certification incur a lot of overhead. Most of these problems are addressed by using third party platforms and middleware (for example Mashape + Kong). But this seems to restrict findability to this one platform and also introduces nontrivial dependencies and fees even for small projects.
Therefore, I am trying to find or create systems that do this without this requirement, in a mostly trustless p2p fashion, starting with an extension to the OpenAPI spec:
I wonder what comes after that. Most probably different transports and patterns of interactions. http://iris.karalabe.com/ might show how it will look like.
You blew my mind a little bit there ... so you're introducing the notion of a super light-weight subscription / payment system to sidecar the API specification itself? This something that would make sense with state-less services vs those requiring some investment by the service provider (i.e. some account with more backend overhead)?
Having the API specification cover more of the business/service aspects is very interesting (could you possibly define cost per call per endpoint or per payload size for instance).
The only costs that would occur would be transaction fees for changing state in the Ethereum network (which is only required once for registering the service and at the order of half a cent at the current scalability level) and a similar amount for opening and closing the payment channels. I'm currently implementing a prototype extending http://hug.rest. If executed correctly, the following three lines of Python:
@hug.get("/pay_world", requires=pay("$0.001"))
def pay_world():
return "You paid for this"
would result in the framework generating the OpenAPI spec amended with the payment metadata in the background and making it available under the service root. A middleware layer would check every ingoing request for payment signatures, closing the payment channel if necessary. Interesting things would become possible, although it is not clear if it all will succeed. Many different kinds of payment mechanisms would be conceivable, from pay-per-call to subscription based. Hypothetically, you could even pay APIs with gold backed cryptocurrency or tokens you issued yourself.
29 comments
[ 3.1 ms ] story [ 67.3 ms ] threadPsychologically, looking at a formal specification is pretty intimidating. How many people do you know who enjoy sitting down with a cup of coffee/tea and their favorite time-worn leather-bound copy of RFC7231?
Most developers I know love getting something to work. They love the feeling of having mastered something to the point that just reading "GET /widgets" is enough for them to go off to curl and implement the rest ad-hoc as they're visually inspecting the JSON and occasionally referring back to the docs.
Creating formal specs is probably the most efficient way to go in a purely computational sense, but reading specs is hard work and it's not especially inviting or rewarding, especially to a class of workers that would much rather jump in, get their hands dirty, and figure it out themselves. It's why developers love frameworks/APIs that just feel simple and familiar. The low cognitive overhead, and low barrier to entry gets you into a wonderfully endorphin-driven flow that's hard to replicate.
Formal specifications are great. But they won't likely get read by people. And people, not machines, are going to drive your profit as a business offering a shiny API.
So should you focus on your formal spec first, or human-readable API documentation? Probably the latter. And good luck convincing the business owners that a spec is a high priority now that you technically have everything covered.
However, what if the specs were able to capture what is in such human-readable pages (for example https://docs.cloudant.com/)? What if API spec could be automatically derived out of HTML pages? Embed some metadata in HTML page and then we have both cake (human-readable) and eat it (machine-readable)?
We ended up using Swagger, which doesn't solve as many of our problems (having documentation generated from code isn't as robust from a testing POV, particularly when your RESTful API is versioned), but my "perfect" is the enemy of the company's "good" :).
Scrape HTML to swagger: https://github.com/bobby-brennan/scrape-to-swagger
Swagger to HTML docs: https://github.com/lucybot/lucy-console
We also have a commercial version of the latter which you can see at any-api.com
I agree the API provider's primary concern should be Developer Experience, and understand why creating a spec often takes a back seat. But I suspect that soon it will be seen as critical for any production grade API to have a machine readable spec.
I view specs as analogous to unit tests - it doesn't get you closer to a deliverable product, but it makes the development lifecycle much less painful.
And I disagree that formal specifications are intimidating. A formal specification is just code, so if you have trouble understanding something, you can enlist the computer's help. On the other hand, if an informal specification isn't clear to you, you are more or less screwed.
Contentful's API specification (in the blue print format) has been open for a while as a side effect of opening up our documentation. I'm not aware of any user ever using for anything but the generated documentation (and there have been bugs in the past that would have made things like generated test servers break quite nicely).
What (potential) use case am I missing?
With regards to the bugs, were they a result of the implementation and the specification getting out of sync? Would automatic means for creating / maintaining specs be able to help here?
In my opinion, API specs should probably be generated from code and its documentation, and not the reverse. This is because it is the code and not the spec which is subject to most change and only the endpoint interfaces/signatures can be generated. API specifications could also be more easily transformed by middleware (e.g. when renaming endpoints, combining APIs) if they were machine generated.
As was described above, only few people are acquainted with service description languages, and deeper changes in their semantics over time or the choice of an alternative would require a manual rewrite. Also, some SDLs seem to impose their own higher level ontologies and name spaces, which is another major obstacle to usage for simple systems.
Second, I like your enthusiasm for specs from code ... the problem (opportunity?) today is that the specifications intertwine human readable descriptions / summaries with the actual endpoint / payload information.
I would hate for my devops pipeline to kick-off because somebody fixed a typo in some human focused in-line documentation. I personally don't think it belongs in the code but absolutely agree with the spirit of your comment (if you keep them separate it is more work and they will get out of sync!).
The best luck I have is working within the specification (i.e. Swagger) and avoiding doing things that don't model well. This seems backwards, but for most of the development I've been involved with to date it has served the projects well.
Interested if others have had luck embedding API specs into code.
Swagger-Codegen (https://github.com/swagger-api/swagger-codegen) is a pretty mature, free and open-source project for generating API clients (e.g. ObjC, Java, C#, Ruby, Python, etc) and server stubs (e.g. PHP Slim, Java Spring, etc). Please give it a try and let us know your feedback by opening a ticket via https://github.com/swagger-api/swagger-codegen/issues
Disclosure: I'm a top contributor to the project.
I lead the API Harmony team at IBM Research and this information would be extremely valuable.
Yes, the API is public, but you need a Cert bound to your user data to get anything from it.
This is an orthogonal issue to whether or not the API itself is publicly accessible.
IMHO key component here is automatic integration, you simply publish a link to your spec and you magically have integration with a number of 3rd-party tools/services.
I currently work on catalog which does exactly this, and you can see list of integrations here: https://github.com/APIs-guru/api-models#existing-integration...
I think realityking nailed it in terms of the API community needing to define compelling use cases. The catch 22 here is that it is hard for people to innovate around API specifications when they are so hard to get! Per the SDKs ... what if there was a NPM.org or rubygems equivalent of high quality SDKs that get automatically generated whenever I push a new API specification? That seems like it could be cool.
And your example of automated composition is fantastic! Curious if you are able to share how much work it is for you to get access to the specifications in order to make this possible? If we invented a common practice to publish something like api.company.com/spec.yaml could you have spent a lot more time on making cool integrations vs mining specs?
If you've a Swagger/OpenAPI spec, you can use Swagger Codegen (https://github.com/swagger-api/swagger-codegen), a free and open-source project, to generate API clients (SDKs) in C#, Ruby, PHP, Java, ObjC, etc.
Here is an example: https://github.com/wing328/petstore-php
You can also use Swagger Codegen to generate server stub in SpringMVC, PHP Slim and more.
Disclosure: I'm a top contributor to Swagger Codegen.
(To convert RAML, API blueprint, WSDL, etc to Swagger/OpenAPI spec 2.0, please try https://github.com/lucybot/api-spec-converter - an open-source converter written in NodeJS)
Yes, this is exactly the problem which I try to solve with my collection.
> Per the SDKs ... what if there was a NPM.org or rubygems equivalent of high quality SDKs that get automatically generated whenever I push a new API specification?
You can run the same tool on API owner side you don't need to publish your spec for that. I did a couple of interviews with API owner and they fear to loose control over SDKs, Docs, etc.
> Curious if you are able to share how much work it is for you to get access to the specifications in order to make this possible?
No, it's public info since my entire work is open-sourced under MIT license. I try to make process automatic as much as possible, so it starts from scraping. For example, I scrape Github for Swagger specs: https://morph.io/IvanGoncharov/API_specifications When I need to filter test, example, etc. APIs and find specs for real-life APIs. But get spec source is just first step, I need to fix errors in it(~80% have them), get additional info(logo, link to API key registration, etc.). But most time consuming is update them every day. My update/convert/validation scripts not ideal so I need to manually inspect all changes.
> ould you have spent a lot more time on making cool integrations vs mining specs?
Actually not I learn a lot from it, for example, you can fix all mistakes in API owner specs. Instead, you should give them a tool which will automatically inspect API spec and output errors/warnings/recommendations/hints. Together with my friend we working on OpenAPI/Swagger linter.
If you want to discuss more here is my Skype(ivangon4arov) and Hangouts(ivan.goncharov.ua@gmail.com) or APIs.guru public chat https://gitter.im/APIs-guru/api-models
Therefore, I am trying to find or create systems that do this without this requirement, in a mostly trustless p2p fashion, starting with an extension to the OpenAPI spec:
http://i.imgur.com/4mmckZ9.png https://gist.github.com/void4/523f23b50e1572e8ef60
I wonder what comes after that. Most probably different transports and patterns of interactions. http://iris.karalabe.com/ might show how it will look like.
Having the API specification cover more of the business/service aspects is very interesting (could you possibly define cost per call per endpoint or per payload size for instance).
The only costs that would occur would be transaction fees for changing state in the Ethereum network (which is only required once for registering the service and at the order of half a cent at the current scalability level) and a similar amount for opening and closing the payment channels. I'm currently implementing a prototype extending http://hug.rest. If executed correctly, the following three lines of Python:
@hug.get("/pay_world", requires=pay("$0.001")) def pay_world(): return "You paid for this"
would result in the framework generating the OpenAPI spec amended with the payment metadata in the background and making it available under the service root. A middleware layer would check every ingoing request for payment signatures, closing the payment channel if necessary. Interesting things would become possible, although it is not clear if it all will succeed. Many different kinds of payment mechanisms would be conceivable, from pay-per-call to subscription based. Hypothetically, you could even pay APIs with gold backed cryptocurrency or tokens you issued yourself.
See here for a similar project and reasoning: https://github.com/etherapis/etherapis