runtime: make rollout seeds selectable and reproducible via RolloutSpec - #149
Merged
Merged
Conversation
Follow-up to NVlabs#128. force_determinism and the inference_seed hookup (session.seed + inference_count) shipped since that discussion, so per-inference seeds already derive deterministically from the session seed. What was still missing is choosing or recovering the session seed itself: driver_service.py and traffic_service.py draw it from random.randint at session creation and RolloutSpec has no seed field, so no rollout can be re-run. Add RolloutSpec.random_seed (fixed64, field 5), following the same 0-means-default convention start_time_offset_us already uses in that message. Rollout k of a spec gets random_seed + k as its session seed, mirroring the additive derivation the driver already uses for inference_seed, so any individual rollout is reproducible on its own. 0 keeps today's behaviour: every service picks a random seed. The seed travels the exact path start_time_offset_us takes: RolloutSpec -> PendingRolloutJob -> AssignedRolloutJob -> UnboundRollout -> Driver/TrafficSessionConfig -> the session request. Both services fall back to random.randint when no seed is supplied. The traffic service currently only logs its seed downstream; wiring trafficsim to consume it is deliberately out of scope here. Tests: seed derivation and the unset default in test_daemon_request_plumbing.py; both services honouring a configured seed (and the random fallback) in tests/services. Targeted suites pass 30/30; the full runtime suite passes except three test_with_mocks failures that reproduce identically with pristine sources under the same containerized proto-regeneration harness, i.e. a harness artifact, plus the pre-existing test_route_generator_map failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
amtellezfernandez
force-pushed
the
pr/rollout-seed
branch
from
August 13, 2026 16:05
b59e729 to
c5f5d6b
Compare
Collaborator
|
/ok to test c5f5d6b |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #128.
force_determinismand theinference_seedhookup (session.seed + session.inference_count) have shipped since that discussion, so per-inference seeds already derive deterministically from the session seed. What was still missing is choosing or recovering the session seed itself:driver_service.pyandtraffic_service.pydraw it fromrandom.randintat session creation, andRolloutSpechas no seed field, so no rollout can be re-run.This adds
fixed64 random_seed = 5;toRolloutSpec, with0(default) meaning the runtime picks a random seed per session, the same conventionstart_time_offset_usalready uses in that message. Rollout k of a spec getsrandom_seed + kas its session seed, mirroring the additive derivationinference_seedalready uses, so a single rollout is reproducible on its own. Existing callers see no change.The value travels the exact path
start_time_offset_ustakes:RolloutSpectoPendingRolloutJobtoAssignedRolloutJobtoUnboundRolloutto the session configs to the session request. Both services fall back torandom.randintwhen it's unset.Out of scope on purpose: making trafficsim consume its seed. Today
request.random_seedonly appears in a log line (trafficsim/grpc/servicer.py:213), and how much reproducibility is realistically achievable there needs its own discussion. The video-model renderer path also has its own sampling, so this is about re-running a rollout and getting the same policy behaviour, not bit-exact replay, which matches the NuRec caveat in #128 anyway.Tests: seed derivation and the unset default in
test_daemon_request_plumbing.py; both services honouring a configured seed, plus the random fallback, intests/services/. Targeted suites pass 30/30, and the full runtime suite is unaffected beyond failures that reproduce identically without this change.