fsdp(scripts): RL scripts for representative dense + MoE models

[Zhichenzzz]

Stacked on #1469 (the FSDP adaptations/ refactor) — base is zhichen/fsdp-refactor, so the diff is just the scripts.

FSDP-backend GRPO RL launch scripts, one per model family (one dense + one MoE). Shared policy for all:

• train: DAPO-math-17k · eval: gsm8k · seq: 8k response len
• GRPO, AdamW, colocate sglang rollout, no checkpointing

scripts/fsdp_rl/common.sh holds the shared launcher; each per-model script just sets MODEL + GPU layout and sources it. GPU counts are sized per model (4/8 GPU, CPU-offload for big ones; multi-node for the huge MoEs). See scripts/fsdp_rl/README.md for the table.

Dense: Qwen3-4B, Qwen3.5-4B, Nemotron-3-Nano-4B, Gemma-4-31B
MoE: Qwen3-30B-A3B, Qwen3.5-35B-A3B (GatedDeltaNet), GLM-4.7-Flash (glm4_moe_lite), Gemma-4-26B-A4B, Nemotron-3-Nano-30B-A3B (nemotron_h), gpt-oss-20B, Qwen3-Next-80B-A3B, DeepSeek-V3 671B, Kimi-K2.5 ~1T

[gemini-code-assist]

Code Review

This pull request introduces FSDP-backend RL reference scripts for various dense and MoE models, featuring a shared launcher script (common.sh) and model-specific configurations. Feedback on the shared launcher focuses on improving robustness and portability: replacing the aggressive pkill -9 python with targeted process termination, adding a validation check to prevent multi-node runs from hanging on localhost, dynamically resolving the repository root directory instead of hardcoding /root/miles, and allowing the SGLang attention backend to be configurable to support non-Hooper GPU architectures.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Using pkill -9 python is extremely aggressive and dangerous. It will abruptly terminate all Python processes owned by the current user on the machine. This has several severe consequences:

1. On shared multi-user clusters, it will kill other unrelated training jobs, Jupyter notebooks, or background tasks the user is running.
2. If this script is invoked or orchestrated by a Python-based wrapper (e.g., a custom launcher, a benchmarking harness, or a CI/CD runner), it will kill its own parent process, causing the run to terminate prematurely and fail to report results.

Instead, you should target only the specific processes related to this run (e.g., train.py and sglang) using the -f flag to match the command line pattern.

Suggested change

	pkill -9 sglang 2>/dev/null; ray stop --force 2>/dev/null; pkill -9 ray 2>/dev/null; pkill -9 python 2>/dev/null; sleep 2
	pkill -9 -f sglang 2>/dev/null; ray stop --force 2>/dev/null; pkill -9 -f ray 2>/dev/null; pkill -9 -f train.py 2>/dev/null; sleep 2

[gemini-code-assist]

When running a multi-node setup (NNODES > 1), if MASTER_ADDR is left at its default value of 127.0.0.1 (localhost), the worker nodes on other machines will not be able to connect to the Ray head node. This will cause the multi-node training run to fail or hang indefinitely.

We should add a validation check to ensure that MASTER_ADDR is set to a non-loopback IP address when NNODES > 1.

Suggested change

	ray start --head --node-ip-address "$MASTER_ADDR" --num-gpus "$GPUS_PER_NODE" --disable-usage-stats
	if [ "$NNODES" -gt 1 ]; then
	echo ">> NNODES=$NNODES: start ray on the other $((NNODES-1)) node(s) with"
	echo " ray start --address=$MASTER_ADDR:6379 --num-gpus=$GPUS_PER_NODE"
	echo " (or launch via your multi-node cluster tooling) so all $((NNODES*GPUS_PER_NODE)) GPUs join before training."
	fi
	if [ "$NNODES" -gt 1 ] && [ "$MASTER_ADDR" = "127.0.0.1" ]; then
	echo "ERROR: NNODES=$NNODES (> 1) but MASTER_ADDR is 127.0.0.1 (localhost)."
	echo "Please set MASTER_ADDR to the head node's reachable network IP address so other nodes can connect."
	exit 1
	fi
	ray start --head --node-ip-address "$MASTER_ADDR" --num-gpus "$GPUS_PER_NODE" --disable-usage-stats
	if [ "$NNODES" -gt 1 ]; then
	echo ">> NNODES=$NNODES: start ray on the other $((NNODES-1)) node(s) with"
	echo " ray start --address=$MASTER_ADDR:6379 --num-gpus=$GPUS_PER_NODE"
	echo " (or launch via your multi-node cluster tooling) so all $((NNODES*GPUS_PER_NODE)) GPUs join before training."
	fi

[gemini-code-assist]

The default value for MILES_DIR is hardcoded to /root/miles, which is typically only accessible by the root user and assumes a specific directory structure. If a non-root user runs this script, or if the repository is cloned to a different location, the script will fail at cd "$MILES_DIR".

We can dynamically resolve the repository root directory relative to the script's location using BASH_SOURCE. This makes the script work out-of-the-box in any environment without requiring manual environment variable overrides.

Suggested change

	: "${MILES_DIR:=/root/miles}"
	: "${MILES_DIR:=$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}"

[gemini-code-assist]

Hardcoding --sglang-attention-backend fa3 restricts these scripts to Hopper GPUs (H100/H200). If a user attempts to run these scripts on Ampere (A100) or other GPU architectures, SGLang will fail to start because FlashAttention-3 is not supported.

To make the scripts more portable and robust across different GPU architectures, we should allow overriding the attention backend via an environment variable (e.g., SGLANG_ATTN_BACKEND), defaulting to fa3.

Suggested change

	SGLANG_ARGS=( --rollout-num-gpus-per-engine 1 --sglang-mem-fraction-static "$SGLANG_MEM" --sglang-decode-log-interval 1000 --sglang-chunked-prefill-size 4096 --sglang-attention-backend fa3 )
	SGLANG_ARGS=( --rollout-num-gpus-per-engine 1 --sglang-mem-fraction-static "$SGLANG_MEM" --sglang-decode-log-interval 1000 --sglang-chunked-prefill-size 4096 --sglang-attention-backend "${SGLANG_ATTN_BACKEND:-fa3}" )