Skip to content

Fix TP-only parallelism config: always include dp_shard in device mesh - #3986

Closed
songhappy wants to merge 1 commit into
huggingface:mainfrom
songhappy:fix_tp
Closed

Fix TP-only parallelism config: always include dp_shard in device mesh#3986
songhappy wants to merge 1 commit into
huggingface:mainfrom
songhappy:fix_tp

Conversation

@songhappy

@songhappy songhappy commented Mar 21, 2026

Copy link
Copy Markdown

What does this PR do?

Fixes a crash when using TP-only parallelism (e.g., [tp_size=8, dp_shard_size=1] with error of:
KeyError: "Invalid mesh_dim_names ('dp_shard_cp',) specified. Valid mesh_dim_names are ['tp']."

Root cause: _get_mesh() only includes dimensions with size > 1 in the mesh, so dp_shard is excluded when dp_shard_size=1. However, build_device_mesh() unconditionally tries to create a dp_shard_cp submesh from dp_shard + cp, which fails because neither dimension exists in the mesh.

Fix (two changes):

  1. _get_mesh(): Always include dp_shard in the mesh dimensions (even when size == 1) so that FSDP2 has a valid submesh. A size-1 shard dimension is a no-op but keeps the composable API consistent.

  2. build_device_mesh(): Build the dp_shard_cp submesh from dimensions actually present in the mesh instead of relying on dp_shard_cp_dim_names which assumes both dimensions exist.

Impact:
No effect on existing FSDP-only or FSDP+TP configs (dp_shard was already present). Only fixes the TP-only case where dp_shard was missing.

@github-actions

Copy link
Copy Markdown
Contributor

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

@github-actions github-actions Bot closed this Apr 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant