Hypolite Petovan

I couldn't find any implementation info on Twitter's API "cursoring" pagination system. I'd like to replicate it for #Friendica 's #API since we emulate their API, but so far I only have the "next_cursor" feature, no idea how to compute the "previous_cursor" value.

I'm looking for either a high-level explanation of this system that doesn't seem to rely on the usual "max_id"/'since_id" boundaries, or an actual implementation of it.

Thanks!

#help #webdev

Brad Koehn ☑️

4 years ago

Wouldn’t it be similar to a SQL cursor? An index into the result of a query?

Hypolite Petovan

4 years ago

@Brad Koehn ☑️ Probably, but I'm not sure how to implement that either in the context of stateless API calls. I don't want to hold the full result in memory not knowing if the cursor will soon be used backward. Going forward is simpler with the cursor parameter behaving like a since_id parameter to filter out previous results.

@Brad Koehn ☑️

Brad Koehn ☑️

4 years ago

Well, all sql databases support queries with an index into the results. So the first page has an index of zero, then 10, 20, and so on. The next page increments the index by the count, and the previous page decrements it. You repeat the query for each request. That’s the most common implementation, though it has weaknesses, esp. around data inserts/deletes between requests.

Sorry if all this is obvious to you and you were thinking of something more sophisticated!

Hypolite Petovan

4 years ago

@Brad Koehn ☑️ No, I'm grateful someone is answering my question. I know of 2 stateless pagination systems: the classic page system, and the since_id/max_id system I discovered implementing the Mastodon API endpoint in Friendica and used by Facebook as well.

With the page system, whether we have a previous page is given away by the page number we receive. page > 1 means that there probably is a page - 1 page. The problem like you said is that if data is prepended to the list, the results are inconsistent with the same page number.

With the since_id/max_id system, previous/next pages are guessed using query boundaries. The next page URL sets since_id that ignores all record with the order field lower than the maximum provided value in the current result list, and the previous page URL sets max_id that ignores all records with the order field greater than the minimum in the current result list. This provides consistent results with a list where items are regularly prepended like a timeline but we can't actually know if the previous/next page actually have records in them.

Here, I'm stumped. The cursor values Twitter uses are unrelated to the result set data which makes me assume there's an additional data structure to handle result sets but I can't imagine a simple system without having a massive in-memory cache containing all the result sets for future use.

@Brad Koehn ☑️

Hypolite Petovan

4 years ago

Never mind, the previous cursor value is negative, which allows to discriminate between since and max even with a single named parameter.

Thank you Brad for entertaining my question, I believe it was pivotal in my breakthrough.

@%:https://diaspora.koehn.com/u/bkoehn

⇧