Support Kubernetes resource requests below limits (bounded bursting)

[KnutZuidema]

Summary

Metaflow currently only supports two effective CPU/memory configurations on Kubernetes, selected via the @kubernetes(qos=...) attribute:

• Guaranteed — requests == limits (forced equal).
• Burstable (default) — requests only, no limit (unbounded).

There is no way to specify a request that is lower than a limit — i.e. the standard Kubernetes pattern where a pod is guaranteed a baseline and may burst up to a bounded ceiling.

Motivation

A common use case is workloads that benefit from being able to burst above their requested resources when spare capacity is available, while still being capped so they can never consume unlimited resources on a node. Batch workloads are one general example of this. This is the canonical use case for requests < limits in Kubernetes.

The two currently-available options do not cover this:

• Guaranteed prevents any bursting (request == limit).
• Burstable allows bursting but with no ceiling, which is undesirable for shared/multi-tenant clusters and interacts poorly with namespace LimitRanges.

Prior discussion

This was raised in #1544 ("Support resource limits in Kubernetes"), which was closed by #2155 (the qos attribute). See also #2046. However, #2155 only implemented the Guaranteed/Burstable distinction. The specific need for independent request/limit values was explicitly noted in the #1544 thread but never addressed:

Requests are "I need at least X", limits are "I want this pod to use at most Y". These should not really be exclusive... often you want to specify both of these.

So the underlying request — bounded bursting via request < limit — remains unaddressed, and there is currently no open issue tracking it.

Proposal

Add a way to configure a limit that is higher than the request for CPU/memory. I do not want to prescribe a specific design; possible approaches include:

• Explicit limit attributes on the decorator (e.g. cpu_limit / memory_limit) alongside the existing cpu / memory (treated as requests).
• A multiplier applied to requests to derive limits (e.g. a burst factor).
• Some other mechanism that composes cleanly with the existing qos attribute and @resources.

Happy to discuss the design and/or contribute an implementation.

Relevant code

• metaflow/plugins/kubernetes/kube_utils.py — qos_requests_and_limits() is the single source of truth; currently takes one scalar per resource.
• Call sites: kubernetes_job.py, kubernetes_jobsets.py, argo/argo_workflows.py.

[palakbhati]

Hi @KnutZuidema, thanks for the detailed writeup I'd like to work on this if it's still open. Could you please assign it to me?

I'll start by looking at qos_requests_and_limits() in kube_utils.py and propose a design (leaning toward explicit cpu_limit/memory_limit decorator attributes alongside the existing cpu/memory, since it maps most directly onto Kubernetes' own request/limit model) before touching the call sites in kubernetes_job.py, kubernetes_jobsets.py, and argo_workflows.py. Happy to adjust based on any maintainer feedback on the approach.

[KnutZuidema]

@palakbhati I'm not a maintainer of this project, so I cannot assign this to you.

I would first wait for feedback from a maintainer, regarding whether this feature fits into the Meta flow vision and regarding possible preferences on the design.