Temporal scaler: add a generic worker-capacity signal to prevent premature scale-down under activity fan-out

[Sanil2108]

Proposal

We run the Temporal scaler in production at Atlan. The stock scaler scales on task-queue
backlog, which scales our workers down while they're still busy: a single workflow can
fan out into hundreds of concurrent activities, so the queue backlog is momentarily empty
even though every worker slot is occupied. To keep workers alive we forked KEDA and
extended the Temporal scaler — but we'd much rather not carry a fork. The core of the
change is generic, so we want to upstream it, help the wider community, and retire our fork.

What's missing today

• Backlog drives the replica count. The metric HPA scales on is ApproximateBacklogCount.
  Once the queue drains, the scaler reads "no load" and scales toward the floor — even when
  workers are still mid-execution.
• includeRunningWorkflowCount (Temporal Scaler: add composite metric (backlog + running workflow count) #7460) only partly compensates. It prevents the hard
  scale-to-zero while workflows are running, but it only flips the isActive keep-warm gate — it
  never raises the reported metric, and it's a Workflow-level, effectively boolean signal: one
  workflow driving 1 activity and one driving 500 look identical. It can't see fan-out, and gives
  nothing to activity-only worker pools.

Proposed change

Add an opt-in worker-capacity signal:

• New includeWorkerCapacity flag (default false — no behavior change unless enabled).
• New metadata: workerCapacityServerAddress + workerCapacityQuery, plus optional
  workerCapacityHeaders, workerCapacityBearerToken, workerCapacityUnsafeSsl.
• Queries any Prometheus-compatible endpoint the operator already runs (Prometheus / Thanos /
  Mimir). The PromQL is entirely user-supplied — KEDA makes no assumption about metric names,
  labels, or how workers are scraped.
• Per-version/build-id attribution is just a label matcher in the user's query (e.g.
  sum(temporal_worker_task_slots_used{temporal_worker_deployment_version="svc:v1"})), so it
  composes cleanly with the deployment-version targeting from feat(temporal): support scaling Temporal workers using Deployment Versions #7672 and per-version ScaledObjects
  don't double-count each other's workers.

Use-Case

Temporal workers whose load is driven by activity fan-out rather than queue depth. Today they
either disable scale-to-zero (idle workers burning money) or accept workers being scaled down
mid-execution (dropped/retried work, latency spikes). A generic capacity signal lets the community
keep aggressive scale-down while staying correct under fan-out — and lets us retire our fork.

Is this a feature you are interested in implementing yourself?

Yes

Anything else?

I worked on our fork fixing these issues, experimented creating a generic version, can raise a PR for this. I also wanted to ensure we align on the exact interface this exposes.

[rickbrouwer]

Could this already be solved with a second prometheus trigger on the same ScaledObject, running the same query you would pass to workerCapacityQuery? Multiple triggers already keep the object active if any of them is active, and advanced.scalingModifiers lets you compose the backlog and the capacity metric into one value.

Does this cover your use case, or is there something it misses?

[Sanil2108]

Thanks @rickbrouwer, fair point — I wasn't aware you could compose triggers like that, and I think you're mostly right. For the keep-warm case a second prometheus trigger with the same query does the job: with multiple triggers the object stays active as long as any one of them is, so it won't scale to zero while slots are in use.

The one case I don't think two triggers can cover is the worker-deployment-version handling. While a version is draining we want it to stay up on its in-flight work, but once it's drained we want it to scale down even if the slot metric is still reporting non-zero from a lagging scrape.