Skip to content

perf(nsys): reduce CPU-side overhead in profiling defaults #3311

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
dingqingy-nv wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

perf(nsys): reduce CPU-side overhead in profiling defaults #3311

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
de22181
perf(nsys): reduce CPU-side overhead in nsys profiling defaults
dingqingy-nv Apr 13, 2026
075257d
fixup: remove unrelated --cuda-event-trace=false from interactive script
dingqingy-nv Apr 13, 2026
35ee200
refactor: surgically modify nemo_run nsys defaults instead of replacing
dingqingy-nv Apr 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions scripts/performance/perf_plugins.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,12 +121,16 @@ def setup(self, task: Union["run.Partial", "run.Script"], executor: "run.Executo
if self.nsys_trace is not None:
launcher.nsys_trace = self.nsys_trace

# Combine default extra args with user-provided extra args
# Reduce CPU-side collection overhead: disable context-switch tracing,
# backtrace collection, and CUDA graph node tracing.
existing = launcher.nsys_extra_args or []
existing = [a for a in existing if not a.startswith("--cuda-graph-trace")]
existing.extend(["--cpuctxsw=none", "--backtrace=none"])
launcher.nsys_extra_args = existing

# Combine with user-provided extra args (user args first for precedence)
if self.nsys_extra_args is not None:
# Get existing launcher extra args (nemo_run defaults)
existing_extra_args = launcher.nsys_extra_args or []
# Combine user args with existing args (user args first for precedence)
launcher.nsys_extra_args = self.nsys_extra_args + existing_extra_args
launcher.nsys_extra_args = self.nsys_extra_args + launcher.nsys_extra_args
logger.info(f"Combined nsys_extra_args: {launcher.nsys_extra_args}")

if isinstance(executor, SlurmExecutor):
Expand Down
2 changes: 2 additions & 0 deletions scripts/performance/rubin_launch_interactive.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,8 @@ if [ "${ENABLE_NSYS}" = "1" ]; then
NSYS_PREFIX=(
nsys profile
-s none
--cpuctxsw=none
--backtrace=none
-t "${NSYS_TRACE}"
-o "${NSYS_OUTPUT}"
--force-overwrite true
Expand Down
7 changes: 7 additions & 0 deletions src/megatron/bridge/recipes/run_plugins.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -297,6 +297,13 @@ def setup(self, task: Union["run.Partial", "run.Script"], executor: "run.Executo
launcher.nsys_profile = True
launcher.nsys_trace = self.nsys_trace or ["nvtx", "cuda"]

# Reduce CPU-side collection overhead: disable context-switch tracing,
# backtrace collection, and CUDA graph node tracing.
existing = launcher.nsys_extra_args or []
existing = [a for a in existing if not a.startswith("--cuda-graph-trace")]
existing.extend(["--cpuctxsw=none", "--backtrace=none"])
launcher.nsys_extra_args = existing

if isinstance(executor, SlurmExecutor):
# NOTE: DO NOT change to f-string, `%q{}` is Slurm placeholder
launcher.nsys_filename = "profile_%p_%q{SLURM_JOB_ID}_node%q{SLURM_NODEID}_rank%q{SLURM_PROCID}"
Expand Down
Loading