Ask HN: RESTful API Pagination Strategies

Hi HN,

I'm hoping to start a discussion on the topic of paging/pagination as there seems to a hole in most implementations that I'm only seeing addressed by some API's.

The example is pretty simple:

GET /some_resource/

Returns: 2,000 results (200 per page)

Loop 10 times to get all results(GET page=1; GET page=2, etc, etc...)

If any record is updated on page 1 while I'm on page 9 (just an example), I will miss it unless I go back and do a reconciliation.

Facebook's Graph API (https://developers.facebook.com/docs/graph-api/using-graph-api -> go to Cursor-based Pagination) tries to address this with a Cursor-based Pagination approach, where they lock a given data set to be within two hashed id's. I don't fully grok the implementation of this strategy though at first glance - [it] seems vast.

Stripe's API tries to do the same https://stripe.com/docs/api#pagination . I don't work with Stripe's API but it appears they want you to specify an "object ID", which I'm guessing is some hashed id + timestamp. Then if a record was updated on page 1 (per my example above), a new hashed ID is generated and would likely be returned on a subsequent page (say page 11 in my example above).

I might be interpreting those implementations incorrectly (by all means correct me here) but I would love to hear from some HN members who have dealt with this issue.

Immediate questions:

- Is there an API out there that solves this really well? - Any further reading folks here might recommend?

4 comments

[ 0.28 ms ] story [ 42.7 ms ] thread

evm9 10y ago ↗

You don't need to know of changes unless the user is on that page.

You can use websockets to broadcast/listen for changed data and updated the DOM accordingly (if you're on the web).

anonfunction 10y ago ↗

http://jsonapi.org/format/#fetching-pagination

mnort9 10y ago ↗

Twitter has some good points on the problem with paging and uses a "max_id" & "since_id" instead.

https://dev.twitter.com/rest/public/timelines

meric 10y ago ↗

We follow the github approach and put it all in the headers:

    < HTTP/1.1 200 OK
    * Server nginx/1.9.11 is not blacklisted
    < Server: nginx/1.9.11
    < Date: Thu, 10 Mar 2016 03:19:48 GMT
    < Content-Type: application/json
    < Content-Length: 247620
    < Connection: keep-alive
    < Count: 120
    < X-Frame-Options: SAMEORIGIN
    < Vary: Cookie
    < Link: <https://xxx.com/api/someresource/?limit=50&offset=50; rel="next">
    < Allow: GET, POST, HEAD, OPTIONS
    < X-Server: zzz
    < X-Frame-Options: SAMEORIGIN
    <