fix(dist): preserve new_group options across reload#1495
Conversation
There was a problem hiding this comment.
Code Review
This pull request refactors the process group reloading mechanism to preserve the original arguments and keyword arguments (such as timeout and pg_options) when recreating process groups. It extracts helper functions for identifying Gloo groups and retrieving group ranks, and adds a comprehensive unit test suite. The review feedback suggests two improvements to prevent potential TypeErrors: checking if old_new_group is None before calling it, and handling cases where group_args might be explicitly passed as None.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
65d39dd to
d07f362
Compare
|
@fzyzcjy @yueming-yuan @guapisolo @maocheng23 @jybsuper @Zhichenzzz @Shi-Dong Could I get a quick review on the PR? This preserves new_group args/kwargs across reloadable process-group reloads. That avoids silently dropping backend/timeout semantics etc. |
Port of THUDM/slime#2095 for miles.
Summary:
torch.distributed.new_grouppositional args and kwargs{ranks, backend=nccl}Validation:
uv run --with pytest --with torch --with psutil --with numpy pytest --confcutdir=tests/fast/utils tests/fast/utils/test_reloadable_process_group.py -q-> 1 passed