Upcoming Mantis 5.x: Reservation-Based Scheduling for Resource Clusters

[Andyz26]

Mantis 5.x: Reservation-Based Scheduling for Resource Clusters

Hey everyone,

We're sharing an early heads-up on a significant scheduling change landing in Mantis 5.x. This post covers what's changing, why, and what you should be aware of when upgrading.

---

TL;DR

The resource cluster scheduler is moving from a fire-and-forget allocation model to a reservation-based, priority-queued model. This is enabled by default in 5.x.

---

What's Changing

Today, when a job needs workers, the JobActor asks the scheduler to immediately find and assign a TaskExecutor. If nothing is available, the request fails and retries are driven by heartbeat timeouts. This works, but it breaks down at scale:

• No priority fairness -- a burst of new jobs can starve worker replacements and scale-up events.
• Scaler is blind to pending demand -- the autoscaler only sees idle vs. busy executors, not queued demand, so it can't proactively scale up.
• Retry storms -- under resource scarcity, many jobs retry scheduling simultaneously, each running a full executor selection cycle.
• No batching -- 100 workers of the same SKU means 100 individual allocation requests.

The new model

In 5.x, scheduling requests become reservations that enter a prioritized queue per resource SKU. A ReservationRegistryActor processes these queues on a periodic tick (default: 1s), dispatching batch allocation requests to an ExecutorStateManagerActor one constraint group at a time.

Key properties:

• Three-tier priority: REPLACE (highest) > SCALE > NEW_JOB (lowest), with job tier and FIFO as tiebreakers. Worker replacements always get resources before new low-priority jobs.
• Back-pressure by design: only one in-flight allocation per SKU at a time. No thundering herd under scarcity.
• Batch allocation: a single reservation for N workers of the same SKU is sent as one batch request.
• Reservation-aware autoscaling: the scaler now sees pending reservation counts and computes effectiveIdleCount = idle - pendingReservations, preventing premature scale-down and enabling proactive scale-up.

---

Impact and Upgrade Notes

Feature flag (opt-out fallback)

Reservation scheduling is controlled by MasterConfiguration.isReservationSchedulingEnabled() and defaults to true in 5.x. To fall back to legacy behavior:

mantis.master.reservationSchedulingEnabled=false

When disabled, the system uses the previous ResourceClusterAwareScheduler with direct scheduling. The flag is global (applies to all resource clusters). However due to the large scope of all changed components the legacy mode will not guarantee the exact same behavior thus we don't recommend running in legacy mode in production. Please remain on 4.x until you are ready to upgrade to the new mode. We recommend running with the flag enabled in a staging/test environment before rolling out to production.

Breaking changes

• MantisScheduler interface: the new ResourceClusterReservationAwareScheduler reports schedulerHandlesAllocationRetries() = true. If you have custom MantisScheduler implementations, note that scheduleWorkers() and unscheduleWorker() now throw UnsupportedOperationException in the reservation-aware path. The entry points are upsertReservation() and cancelReservation() instead.
• ResourceClusterActor refactored: executor state management, disabled executor tracking, and assignment logic have been extracted into child actors (ExecutorStateManagerActor, AssignmentHandlerActor). If you have custom code that directly interacts with ResourceClusterActor internals or extends its message protocol, you'll need to adapt.
• Scaler protocol change: when reservation scheduling is enabled, the scaler issues GetReservationAwareClusterUsageRequest instead of GetClusterUsageRequest. Custom scaler integrations that rely on the old request/response types should be updated.

No breaking changes if...

• You use Mantis as-is without custom scheduler or resource cluster extensions -- the change is transparent. Jobs submit the same way; the scheduling path is different under the hood.
• You interact with Mantis only through the job submission API -- no API changes.

Recovery behavior change

Reservation state is in-memory only (not persisted). On master failover:

1. All pending reservations are lost.
2. After the new leader initializes, each JobActor scans its stages for workers in Accepted state and re-submits them via upsertReservation(), rebuilding the queue.

This means there is a brief window after failover where pending reservations don't exist yet. In practice, this window is bounded by leader election time + job cluster re-initialization time. If you have tight SLAs around worker replacement latency during master failover, test this path.

Autoscaler tuning

If you've tuned your ResourceClusterScalerActor thresholds (maxIdleToKeep, minIdleToKeep, cooldown periods), be aware:

• Scale-down now uses effectiveIdleCount = idle - pendingReservations. Machines about to be used by pending reservations won't be scaled down.
• Scale-down cooldown is 5x the normal cooldown when reservation scheduling is enabled, to account for the non-atomic two-phase usage computation.
• Scale-up step calculation changes to: step = pendingReservations + minIdleToKeep - actualIdleCount.

You may want to revisit your idle thresholds after enabling reservation scheduling, as the scaler is now more conservative about scale-down and more aggressive about scale-up.

---

Known Limitations (5.x initial release)

1. No reservation persistence -- crash recovery relies on JobActor re-initialization (see above). A future release may add durable reservation state.
2. Non-atomic scaler usage computation -- the two-phase query across ReservationRegistryActor and ExecutorStateManagerActor can transiently see stale data. Mitigated by the conservative scale-down cooldown.
3. No reservation TTL -- if a job crashes without sending CancelReservation, its reservations remain in memory indefinitely. A TTL/sweep mechanism is planned for a follow-up.

---

New Components at a Glance

Component	Role
ReservationRegistryActor	Owns reservation queues, processes by priority per SKU on tick
ExecutorStateManagerActor	Manages executor lifecycle, handles batch assignment requests
AssignmentHandlerActor	Submits tasks to executors, retries with fresh gateway on failure
ResourceClusterReservationAwareScheduler	New MantisScheduler impl that delegates to reservation APIs

---

Feedback Welcome

We'd love to hear from anyone running Mantis at scale:

• Are there scheduling pain points this doesn't address?
• Any concerns about the in-memory-only reservation state?
• Would per-cluster feature flag granularity be important for your rollout?

Please share your thoughts in this thread. If you run into issues during upgrade, open an issue or use this thread to let us know.

Thanks!

[Andyz26]

cc @kmg-stripe @andresgalindo-stripe