Skip to content

Separate Read and Write QPS Limits for Persistence #9108

@roshchha

Description

@roshchha

Is your feature request related to a problem? Please describe.
We experienced an issue where we couldn't view schedules via UI due to Namespace Persistence Max QPS Reached errors. This happened when tuning global persistence QPS per namespace.

Root cause: A single DescribeSchedule request can generate 5-20+ persistence calls (GetCurrentExecution, GetWorkflowExecution, ReadHistoryBranch, etc.) in a burst. This number increases based on recent/running workflows in the schedule. These are mostly read operations, which are less costly than writes. But with a single combined QPS limit, it's easy to hit the limit quickly - especially on smaller DB sizes or when workflows are concentrated on a single pod.

Describe the solution you'd like
Separate read and write QPS limits:

history.persistenceNamespaceReadMaxQPS: 500
history.persistenceNamespaceWriteMaxQPS: 100

This would allow read-heavy operations (like Schedule APIs, UI browsing) to work without risking DB overload from writes.

Describe alternatives you've considered
As of now, we have increased the namespace QPS quota to accommodate schedules

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions