Summary
Two hard-coded timeouts make large-model loading (100 GB+ shards per node) structurally impossible to complete on heterogeneous clusters, because healthy-but-busy nodes/runners get killed mid-load:
1. Node inactivity timeout: 30 s (master/main.py)
if now - time > timedelta(seconds=30):
logger.info(f"Manually removing node {node_id} due to inactivity")While a node loads a 100 GB+ shard (mmap + materialization), its main event loop can starve for well over 30 s — especially on unified-memory devices (Jetson Thor/GB10, page-cache pressure) and WSL2 (slow I/O). The master then kicks the node, which invalidates the whole multi-node instance → every load attempt of our 368 GB model (GLM-5.2, 5-rank pipeline) died this way. Raising to 900 s made loads survive this phase.
2. Runner initialize_timeout: float = 400 (worker/runner/supervisor.py)
On CUDA ranks with cold JIT caches, shard initialization can exceed 400 s (we observed a runner grinding at ~300 % CPU for 10+ min on an RTX 5090 after a fresh boot wiped the JIT cache). The supervisor kills the runner at 400 s, the instance retries, hits the same wall, and after max retries the instance is deleted.
Suggestion
Make both timeouts configurable (env or config), and/or scale the defaults with the shard size being loaded. A 30 s liveness window that is fine for 0.6B models is ~2 orders of magnitude too tight for 300 GB+ models on heterogeneous hardware.
Context: same 5-node cluster as #2219 (2× Mac, GB10, Jetson Thor, RTX 5090/WSL2; 368 GB GLM-5.2 pipeline-parallel; loads reached Ready×5 once these timeouts were patched to 900 s / 1800 s locally).
Summary
Two hard-coded timeouts make large-model loading (100 GB+ shards per node) structurally impossible to complete on heterogeneous clusters, because healthy-but-busy nodes/runners get killed mid-load:
1. Node inactivity timeout: 30 s (
master/main.py)While a node loads a 100 GB+ shard (mmap + materialization), its main event loop can starve for well over 30 s — especially on unified-memory devices (Jetson Thor/GB10, page-cache pressure) and WSL2 (slow I/O). The master then kicks the node, which invalidates the whole multi-node instance → every load attempt of our 368 GB model (GLM-5.2, 5-rank pipeline) died this way. Raising to 900 s made loads survive this phase.
2. Runner
initialize_timeout: float = 400(worker/runner/supervisor.py)On CUDA ranks with cold JIT caches, shard initialization can exceed 400 s (we observed a runner grinding at ~300 % CPU for 10+ min on an RTX 5090 after a fresh boot wiped the JIT cache). The supervisor kills the runner at 400 s, the instance retries, hits the same wall, and after max retries the instance is deleted.
Suggestion
Make both timeouts configurable (env or config), and/or scale the defaults with the shard size being loaded. A 30 s liveness window that is fine for 0.6B models is ~2 orders of magnitude too tight for 300 GB+ models on heterogeneous hardware.
Context: same 5-node cluster as #2219 (2× Mac, GB10, Jetson Thor, RTX 5090/WSL2; 368 GB GLM-5.2 pipeline-parallel; loads reached Ready×5 once these timeouts were patched to 900 s / 1800 s locally).