`CollectionSearchExtension` + `TokenPaginationExtension` unsupported?

[fmigneault]

Using CollectionSearchExtension + TokenPaginationExtension simultaneously (as configured in https://github.com/crim-ca/stac-app/blob/8eaeafc0972cb4ed3c16e8d8d12785ae0704ee12/src/stac_app.py#L117-L148) does not seem to be considered.

The OffsetPaginationExtension seems to be applied regardless, as shown by the response's next link:

In contrast, doing the same with Items listing works as intended:
https://github.com/crim-ca/stac-app/blob/8eaeafc0972cb4ed3c16e8d8d12785ae0704ee12/src/stac_app.py#L150-L172

---

These behaviours seem to align with PagingLinks used for items:

stac-fastapi-pgstac/stac_fastapi/pgstac/models/links.py

Line 139 in 340fd66

class PagingLinks(BaseLinks):

And CollectionSearchPagingLinks for collections:

stac-fastapi-pgstac/stac_fastapi/pgstac/models/links.py

Line 195 in 340fd66

class CollectionSearchPagingLinks(BaseLinks):

Each using their respective hardcoded token/offset keywords.

However, it seems counter-intuitive that the PG integration does not let the flexibility between token/offset paging extensions offered by the base (https://github.com/stac-utils/stac-fastapi/tree/main/stac_fastapi/extensions/stac_fastapi/extensions/core/pagination), and that they are used inconsistently between Collections/Items.

Is there something else to configure, or is this imposed by stac-fastapi-pgstac for some technical reason?

[vincentsarago]

Does PgSTAC support token pagination for collections?

[fmigneault]

On SQL side, it looks like only offset is defined as well:
https://github.com/stac-utils/pgstac/blob/main/src/pgstac/sql/004a_collectionsearch.sql

Another question for us is not only if STAC API / pgstac could support token pagination, but whether STAC browser will also reflect this type of pagination (relying only on the next link), or whether it encodes offset as well?

We can switch to offset in our implementation to work around this, but proper warning should be provided in the pagination extension (in https://github.com/stac-utils/stac-fastapi docs) since the provided classes make it seem like they can be interchanged seamlessly.

[vincentsarago]

We can switch to offset in our implementation to work around this, but proper warning should be provided in the pagination extension (in https://github.com/stac-utils/stac-fastapi docs) since the provided classes make it seem like they can be interchanged seamlessly.

Well it really depends on the backends 🤷. I'm not sure what we could do more to be honest. In main.py we list all the supported extensions. Then it's up to the users to enable them or not. If someone creates its own app, them it's expected that they know the backend.

Another question for us is not only if STAC API / pgstac could support token pagination, but whether STAC browser will also reflect this type of pagination (relying only on the next link), or whether it encodes offset as well?

That's a stac-browser question 🤷

@fmigneault any action item for this repo or we're good to close?

[fmigneault]

I think it is fine to not have it implemented and have it marked as a known limitation.

I suggest that this repo adds a clear warning about PagingExtension not being directly applicable with all variants everywhere since it is not easy to follow all the very deep SQL dependencies and how that plugs into the https://github.com/stac-utils/stac-fastapi interfaces.

[vincentsarago]

Happy to review PRs 🙏

[bitner]

@fmigneault the reason that there is a difference in pagination between collection search and item search has to do with the general scale of the collections table and the items table. If you have more than several thousands of collections, the partition mechanisms within pgstac are not going to scale well anyway. Items, on the other hand can easily be millions to hundreds of millions of rows. Offset pagination is much much easier to work with than keyset pagination, BUT it does not scale well at all -- in order to use an offset, all rows up to the offset must be scanned in any query meaning that using a limit of 100, an offset of 10000 could theoretically take 100 times longer to return than with an offset of 0 (generally, it is not quite that bad, but it is still a linear trend). That is why we use keyset pagination that can use indexes to just skip rows for items, but there really are never enough total rows in the collections table for it to make that much of a difference.

When developing a client for STAC, it really should not matter which pagination method is being used. As per the STAC API spec, you should just always be looking for the "prev" and "next" links and just following them. There is no offset parameter defined in STAC API, this can be exposed as a parameter in any STAC API, BUT the STAC compliant approach to using it is not to manage the offset on the client side, but to still follow the prev and next links as far as my reading of the spec.

[fmigneault]

@bitner
Thanks for the technical explanation. The use of offset and the scaling issue was also my intuition, hence why I attempted to apply token everywhere. It is true that I would not expect to ever have enough collections to make the offset/token distinction noticeable, but I figured if it made a difference on the Item side, the Collection paging could also use token for the sake of consistency rather than mixing methods. But I agree that the client does not need to worry about either and just follow the links.

@vincentsarago
I can open a PR.
Would a note somewhere in https://stac-utils.github.io/stac-fastapi-pgstac/settings/ be the right place to document it, or somewhere else should be considered (eg: README), or both places?

[fmigneault]

Took me a while to get back to this, but here's the PR: #375