Skip to content

Add hot reload collections#42

Open
chorng wants to merge 2 commits intostac-utils:mainfrom
chorng:feat/hot-reload-collections
Open

Add hot reload collections#42
chorng wants to merge 2 commits intostac-utils:mainfrom
chorng:feat/hot-reload-collections

Conversation

@chorng
Copy link

@chorng chorng commented Jan 28, 2026

I am serving data with stac-fastapi-geoparquet with a pipeline regularly adds new items to existing parquets and updates the extent of collections.json stored remotely. Since collection.json is load only once when the app starts, we have to restart the service every time after the collection.json is updated.

This is a draft which implements a mechanism to reload and cache collections.json at a set time interval, and all requests will go through the latest cached collection.json. With the hot reload mechanism the updated collections.json can be reflected without restarting the service.

@gadomski What do you think having a hot reload mechanism at the backend? Please let me know if this feature fits to the project, thank you!

@gadomski gadomski self-requested a review January 28, 2026 13:49
@gadomski
Copy link
Member

gadomski commented Feb 2, 2026

Thanks for this! I haven't had time to look at it yet, but just dropping a note to say I've seen it an it's in the queue. I'm not against the idea of hot-reloading on principal, but I'll want to take a think to see if there's any alternatives.

Copy link
Member

@gadomski gadomski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It makes sense, but what about something simpler -- just add a new API endpoint that tells the lambda to reload the collection file? If you're worried about other people hitting it you could put it behind auth. E.g. POST /collections/reload?

Comment on lines +120 to +122
app.state._collections_refresher = asyncio.create_task(
collections_cache_refresher(settings)
)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on creating a function to list and store the collections in the state https://github.com/developmentseed/tipg/blob/47e1f091ecd8e8962808753e891e9bb0d6947c31/tipg/main.py#L40-L41

Copy link
Member

@gadomski gadomski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you wanted to keep things auto-magic, tipg has an example that doesn't rely on globals: https://github.com/developmentseed/tipg/blob/47e1f091ecd8e8962808753e891e9bb0d6947c31/tipg/middleware.py#L73-L118

@chorng
Copy link
Author

chorng commented Feb 3, 2026

Thank you for reviewing. Will try to implement sth similar to tipg as suggested :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants