Show HN: APIRank.dev – We crawled and ranked public APIs from the internet (apirank.dev)
Why we did that?
During a YC meetup I spoke with a fellow founder that told me how hard it was to pick the right external APIs to use within your own projects. I realized that most of what we build relies on public APIs from external vendors, but there was no benchmark to help developers compare and evaluate public APIs before picking one. So we decided to do it ourselves. Say hi to apirank.dev.
Why is ranking public APIs hard? Automating Public API technical assessment is a tough problem. First, we needed to find all the public APIs and their specifications - mostly OpenAPI files.
We used several strategies to find those:
- Crawl API repositories like apis.guru
- Crawl Github for openapi.json and openapi.yaml files
- A cool google dork
Those strategies enabled us to gather around ~20.000 OpenAPI specs.
Then lies the hard part of the problem:
We want to dynamically evaluate those APIs' security, performance, and reliability.
But APIs take parameters that are tightly coupled to the underlying business logic.
A naive automated way would not work: putting random data in parameters would likely not pass the API's validation layer, thus giving us little insight into the real API behavior.
Manually creating tests for each API is also not sustainable: it would take years for our 10-people team. We needed to do it in an automated way.
Fortunately, our main R&D efforts at Escape aimed to generate legitimate traffic against any API efficently.
That's how we developed Feedback-Driven API exploration, a new technique that quickly asses the underlying business logic of an API by analyzing responses and dependencies between requests. (see https://escape.tech/blog/feedback-driven-api-exploration/)
We originally developed this technology for advanced API security testing. But from there, it was super easy to also test the performance and the reliability of APIs.
How we ranked APIs?
Now that we have a scalable way to gather exciting data from public APIs, we need to find a way to rank them. And this ranking should be meaningful to developers when choosing their APIs.
We decided to rank APIs using the following five criteria:
- Security - Performance - Reliability - Design - Popularity
Security score is computed as a combination of the number of OWASP top 10 vulnerabilities, and the number of sensitive information leaks detected by our scanner
The performance score is derived from the median response time of the API, aka the P50
The reliability score is derived from the number of inconsistent server responses, either 500 errors or responses that are not conform with the specification
The Design score reflects the quality of the OpenAPI specification file. Having comments, examples, a license, and contact information improves this score
The popularity score is computed from the number of references to the API found online
If you are curious about your API's performance, you can ask us to index your own api for free at https://apirank.dev/submit
61 comments
[ 0.49 ms ] story [ 175 ms ] threadLooking forward to some more browsing on your site.
Google is not an OpenAPI spec indexer either ;)
And most of them have status pages that let me know their reliability.
The next step could be adding tags/categories so you can look for specific APIs ;)
A well designed API with no OpenAPI support or 3.1.x spec support will get a poor design score.
I think user generated content might be the way to go for Design. Let people score and review the design quality.
Thanks for your feedback! APIRank is new, and we are here to improve it and make it useful to the community.
APIRank only considers APIs with an OpenAPI or Swagger specification. It supports any version of Swagger v2+ or OpenAPI v3+.
However, I agree in practice for people, the "design criterium" is not only the compliance to the spec ;)
Antoine
On a more serious note, if the purpose is to be able to compare candidate APIs that you're looking at using, here are some features that I think might be helpful:
* Tagging, and filtering based on those tags. Example: "Show me APIs that deal with SMS"
* Select two or more APIs for direct comparison. Example: "Compare Twilio, CleverTap, and Amazon SES"
* Sorting by column heading. Example: "Sort by median response time, descending"
Also, I can't quite get the search to work for me. The term "Paul" returns the top entry ("P J S Paul Ministries API") and one other ("PoGoSnap API") that doesn't include the search term. "Twilio" returns no results, even though there are three entries on the bottom of the first page that start with that string.
This is kind of a side project for us, but I'll definitely think about it
I run a SaaS that helps people get API data into Google Sheets.
For very popular services, we make customized connectors that hide all the complexity.
For medium popularity services, we do some automated transforms to hide complexity in the requests, and massage the response shape to an easier-to-consume format.
For low popularity services, we'll do a best-effort to point customers towards docs and common pitfalls.
All of that is to say: I've seen a lot of APIs!
So it's very bizarre to see Xero ranked #3 out of 5,651. Xero has this convoluted system where you need to query to get your tenant ID, and then pass that as a header. If you don't pass it, you get an opaque error message. That's the 3rd best API, eh?
Also, you sort of have to wonder how much human review went into this. GitHub has an OpenAPI spec, is a hugely popular service, and seems to be absent from this index.
(edit: Perhaps Github is present? I got no results when I searched "github", but I also get no results if I search "xero", despite Xero Assets API being the 3rd hit.)
Is the plan to actually build this out to be a usable tool or is it just a way for you to get people to submit API:s and get some marketing?
Thanks for the great ideas, I share your opinion on those
Apple Itunes : https://developer.apple.com/library/archive/documentation/Au...
last fm : https://www.last.fm/api
acoustid : https://acoustid.org/webservice
spotify : https://developer.spotify.com/documentation/web-api/
discogs : https://www.discogs.com/developers
deezer : https://developers.deezer.com/login?redirect=/api
music brainz : https://musicbrainz.org/doc/MusicBrainz_API
gracenote : https://developer.tmsapi.com/Sample_Code
---
Additionally, why not list API rate limits?
Or some lookups don't need any security, like querying Apple Itunes does not require anything except for being nice and rate limiting.
* https://boto3.amazonaws.com/v1/documentation/api/latest/refe...
* e.g. https://cloud.google.com/iam/docs/reference/rest
I'm sure there's an Azure one, too, I just don't currently play in that sandbox
I know that DataDog also uses openapi but since this site's search it broken, hard to know if it's hiding on page 57 or what
Had the search said "error while searching: 403" (or whatever) versus "no results" there would have been far fewer complaints in this thread
The search appears to be broken, because it returns no results for spotify.
Thanks for your feedback! APIRank is new, and we are eager to make it better.
Please note APIRank we are only compatible with Swagger & OpenAPI specifications. You can propose to index any API you want here: https://apirank.dev/submit
Concerning Github APIs, indeed, we missed it. We will add the following in our next update: https://raw.githubusercontent.com/github/rest-api-descriptio...
A sample response takes this form: `{ type: 'data', nodes: [ { type: 'data', data: [Array], uses: [Object] }, ... ] }`
Things get a little strange (for me at least!) when I start poking around in the data array. At index 0 in this array is an object with some information on what I think is mostly just paging and the length of the data array (sans this first element). At index 1 is an array filled with numbers that correspond to array indices in the initial data array, and the objects at those specified indices are the actual data sources. While the keys in those objects are actual descriptors of relevant information, the VALUES for those keys are also array indices. Essentially the incoming data structure is a super flattened array with some objects/ additional arrays that just point to other indices in the data array.[1]
Is there a name for this type of pattern? Is it just an artifact of using some query library to handle data fetching? Apologies for the naivety, I still consider myself a fairly new dev (especially when I see something like this!).
[1] More complete example (not sure if I explained this well):
Say service A has an API:
- GET /users/<id> - 100ms
Service B has multiple APIs:
- GET /users/<id> - 100ms
- GET /users - 250ms
- GET / - 500ms
Which service is faster according to your algorithms?
I'm also a bit skeptical of your other metrics. "P J S Paul Ministries API" is 5/5 in popularity while Adyen is 3/5?
There are limitations indeed, though if you compare similar APIs with similar business intent, their structure should also have similarity, unlike your example.
For the popularity metrics, I will take a look - I admit we can do better :)
In order for this to have any use, it seems to me you'd need to at least group the API's by use case/product type.
Similarly: how can you meaningfully measure the performance, security, or reliability of these APIs without first determining whether you're communicating meaningfully with them? I assume that many (most?) of these APIs require some kind of credential; it's hard to glean anything of interest from a server's ability to reliably send an error message,
Given generic external scanners propensity for giving false positives, I'm very skeptical.
I clicked because I was indeed curious how they'd rank, but this being the first point tells me that no sensible ranking could be found
The only objective metric in the set is response time, but anyone would agree that this isn't the only thing you use to select what api to use
I'm really curious if a lot of those < 100 OWASP scores are really anything that matters in the real world.
I also think there's way more metadata that could be useful in ranking. Things like API documentation, communities, usage cost, complexity of API, statefulness, age of the API...
Anyway, looks like a cool project that will be useful to some people
I have a habit of sleeping under rocks but why do people do this? It seems incredibly brittle and strictly worse than leftpad-esque dependency hell – you can’t freeze or fork an API.
I can understand that for payments, maps or even email sending it’s more convenient, but that’s at least a bounded issue.
Are the apps we are building today gonna work in 3 years without meticulous maintenance? Are we doomed to suffer flaky experiences from compound latencies and rate limiting? And what’s the point of your 5 nines if you rely on third parties anyway?
We're implementing a new (cloud) platform that's pretty central to our primary business. Some sub-projects have already spun off, before we had even decided how to set up and maintain the platform, because I said to meet the timeline we need to focus on more broad, general integrations first.
The decision was, instead, to bring in additional developers whom several months later we're still helping learn the APIs and troubleshoot problems (amidst the rest of our integration tasks).
Meanwhile a few weeks ago, I finally got the go-ahead and a DBA's time to do the integration I wanted. We've got a live database of 90-some tables dumped from the platform, including everything these projects are fetching (but with a greater delay between updates), accessible to the entire org. Reporting teams and business analyst are already writing queries to drive the services they need, freeing up our developers to work on other tasks.
A good API can provide very rapid, very specialized development; which is great when that development _is_ your business and so you're paying developers to keep up with it, or it's not so important that is kept up on (personal projects, prototypes, and temporary services). And a well designed and managed API shouldn't change _that_ much _that_ frequently.
But ignoring that an external API is always something you don't control and needs to be maintained is a costly mistake I've seen my org fall into over and over again.