Skip to content

Comments

Add Option for Warp Renderer backend with PhysX simulation engine#4612

Open
bdilinila wants to merge 44 commits intoisaac-sim:developfrom
bdilinila:dev/bdilinila/physx-warp
Open

Add Option for Warp Renderer backend with PhysX simulation engine#4612
bdilinila wants to merge 44 commits intoisaac-sim:developfrom
bdilinila:dev/bdilinila/physx-warp

Conversation

@bdilinila
Copy link

@bdilinila bdilinila commented Feb 13, 2026

TODO: Changelog, testing

Description

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

ooctipus and others added 27 commits February 3, 2026 16:25
This commit adds the foundation for Newton Warp rendering support:
Implemented full state synchronization:
Added comprehensive documentation and tools:
@github-actions github-actions bot added the isaac-lab Related to Isaac Lab team label Feb 13, 2026
@@ -67,6 +67,52 @@ class RslRlPpoActorCriticRecurrentCfg(RslRlPpoActorCriticCfg):
"""The number of RNN layers."""


@configclass
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this was carried over from this change.

@@ -0,0 +1,47 @@
# Copyright (c) 2022-2026, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -116,6 +116,28 @@ def __post_init__(self):
policy: PolicyCfg = PolicyCfg()


@configclass
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -26,7 +26,7 @@
from isaaclab.utils import replace_strings_with_slices
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file and source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/reach/reach_env_cfg.py both come from https://github.com/ooctipus/IsaacLab/tree/feature/dexsuite_vision/source/isaaclab_tasks

@github-actions github-actions bot added the documentation Improvements or additions to documentation label Feb 18, 2026
@greptile-apps
Copy link
Contributor

greptile-apps bot commented Feb 18, 2026

Greptile Summary

This PR adds an optional Newton Warp renderer backend for tiled cameras while keeping PhysX as the simulation engine. It introduces a new isaaclab.renderer sub-package with a registry pattern, a NewtonManager singleton that synchronizes PhysX rigid body states into a Newton model via GPU kernels, and integration into TiledCamera via a new renderer_type config field. The PR also adds Hydra config group variants for scene/agent configuration, CNN actor-critic support for RSL-RL, vision-based Dexsuite Kuka-Allegro environments, and bumps rsl-rl-lib to 3.3.0.

Key issues found:

  • Missing gym environment classes: The __init__.py registers 4 vision-based gym environments, but 3 reference classes (DexsuiteKukaAllegroLiftSingleCameraEnvCfg_PLAY, DexsuiteKukaAllegroReorientSingleCameraEnvCfg, DexsuiteKukaAllegroReorientSingleCameraEnvCfg_PLAY) that are either commented out or never defined, causing runtime failures on use.
  • Pervasive debug instrumentation: print(f"[PERF]...") statements and import time as _t calls are injected into hot paths (simulation_context.py reset()/step()/render(), manager_based_rl_env.py) that affect all users, not just Newton Warp users.
  • Overly broad exception handling: except Exception: pass in manager_based_env.py silently swallows real errors from NewtonManager.set_scene().
  • Hardcoded temp path with rmtree: NewtonWarpRenderer.__init__ unconditionally deletes /tmp/newton_renders on every instantiation, which is fragile for multi-camera and multi-process setups.
  • No-op device string conversion: str(self.device).replace("cuda:", "cuda:") in tiled_camera.py.
  • Missing save_run_artifacts.sh: train.py references a shell script that doesn't exist in the repo.
  • The PR description notes TODO items for changelog, testing, and formatting that are still unfinished (tests checkbox unchecked, changelog not updated).

Confidence Score: 2/5

  • This PR has runtime-breaking issues (missing classes for gym registrations) and pervasive debug instrumentation that affects all users, making it not ready for merge as-is.
  • Score of 2 reflects: (1) gym environments referencing non-existent classes will cause ImportErrors at runtime; (2) debug print statements in simulation_context.py hot paths affect all users globally; (3) overly broad exception handling hides real errors; (4) hardcoded /tmp path with rmtree is fragile; (5) missing shell script reference. The core Newton Warp rendering architecture is sound, but the PR needs cleanup before merge.
  • Pay close attention to: kuka_allegro/__init__.py (missing classes), simulation_context.py (debug prints in hot paths), manager_based_env.py (broad exception catch), newton_warp_renderer.py (hardcoded /tmp rmtree), tiled_camera.py (no-op device replace)

Important Files Changed

Filename Overview
scripts/reinforcement_learning/rsl_rl/train.py Adds --renderer_backend CLI arg, timing summary, and save_run_artifacts call referencing a non-existent shell script. Contains debug instrumentation.
source/isaaclab/isaaclab/envs/manager_based_env.py Adds NewtonManager.set_scene() call with overly broad exception handling that hides errors, plus verbose debug prints during startup.
source/isaaclab/isaaclab/envs/manager_based_rl_env.py Wraps simulation step and observation compute in Timer contexts, adds debug prints. Adds isaac_sim_version to metadata. Debug prints affect all users.
source/isaaclab/isaaclab/renderer/newton_warp_renderer.py New Newton Warp renderer implementation with Warp kernels. Has hardcoded /tmp path with unconditional rmtree, debug image saving every 50 frames, and verbose print statements.
source/isaaclab/isaaclab/sensors/camera/tiled_camera.py Core integration point for Newton Warp rendering. Contains no-op device string replace, but otherwise correctly branches between RTX and Newton Warp paths.
source/isaaclab/isaaclab/sim/_impl/newton_manager.py New 507-line Newton Manager singleton for PhysX-to-Newton state sync. Uses class-level state, has debug logger.warning calls that should be logger.debug, and GPU kernel for pose copying.
source/isaaclab/isaaclab/sim/simulation_context.py Adds extensive PERF print instrumentation to reset(), step(), and render() hot paths that affects all users regardless of renderer backend choice.
source/isaaclab/setup.py Loosens flatdict dependency from ==4.0.1 to >=3.4.0 with no upper bound, risking future incompatibilities.
source/isaaclab_rl/setup.py Bumps rsl-rl-lib from 3.1.2 to 3.3.0. Major version bump needs validation.
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/dexsuite/config/kuka_allegro/init.py Registers 4 new vision gym environments, but 3 reference classes that don't exist (commented out or never defined), causing runtime ImportError on use.
source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/dexsuite/config/kuka_allegro/dexsuite_kuka_allegro_vision_env_cfg.py New vision env configs with RTX and Newton Warp variants. Missing PLAY and Reorient classes that are referenced by gym registrations in init.py.
source/isaaclab_tasks/isaaclab_tasks/utils/hydra.py Major extension adding Hydra group variant system (register_hydra_group, resolve_hydra_group_runtime_override, helper functions). Complex new functionality with lambda-based logic.

Sequence Diagram

sequenceDiagram
    participant Train as train.py
    participant RLEnv as ManagerBasedRLEnv
    participant SimCtx as SimulationContext
    participant Scene as InteractiveScene
    participant TiledCam as TiledCamera
    participant NM as NewtonManager
    participant NWR as NewtonWarpRenderer
    participant Warp as SensorTiledCamera (Warp)

    Train->>RLEnv: step(action)
    RLEnv->>SimCtx: sim.step(render=False)
    Note over SimCtx: PhysX physics step
    RLEnv->>Scene: scene.update(dt)

    RLEnv->>RLEnv: observation_manager.compute()
    RLEnv->>TiledCam: _update_buffers_impl()

    alt renderer_type == "newton_warp"
        TiledCam->>NM: update_state_from_physx_tensors_gpu()
        NM->>NM: GPU kernel: copy PhysX poses → Newton body_q
        TiledCam->>NWR: render(pos_w, quat_w, intrinsics)
        NWR->>NWR: _prepare_camera_transforms()
        NWR->>Warp: SensorTiledCamera.render()
        Warp-->>NWR: color_image, depth_image
        NWR->>NWR: _copy_outputs_to_buffers()
        NWR-->>TiledCam: output buffers (rgba, depth)
    else RTX (default)
        TiledCam->>TiledCam: annotator.get_data()
        TiledCam->>TiledCam: reshape_tiled_image kernel
    end

    TiledCam-->>RLEnv: observation tensors
Loading

Last reviewed commit: 42dbb9b

Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

30 files reviewed, 10 comments

Edit Code Review Agent Settings | Greptile

Comment on lines 43 to 46
disable_env_checker=True,
kwargs={
"env_cfg_entry_point": f"{__name__}.dexsuite_kuka_allegro_env_cfg:DexsuiteKukaAllegroLiftEnvCfg",
"rl_games_cfg_entry_point": f"{agents.__name__}:rl_games_ppo_cfg.yaml",
"rsl_rl_cfg_entry_point": f"{agents.__name__}.rsl_rl_ppo_cfg:DexsuiteKukaAllegroPPORunnerCfg",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing classes will cause ImportError at registration time

The gym registrations reference four classes from dexsuite_kuka_allegro_vision_env_cfg that do not exist:

  • DexsuiteKukaAllegroLiftSingleCameraEnvCfg_PLAY — class definition is commented out in vision_env_cfg (line 248)
  • DexsuiteKukaAllegroReorientSingleCameraEnvCfg — never defined anywhere
  • DexsuiteKukaAllegroReorientSingleCameraEnvCfg_PLAY — never defined anywhere

Attempting to create any of these gym environments (e.g., Isaac-Dexsuite-Kuka-Allegro-Lift-Single-Camera-Play-v0) will fail with an AttributeError/ImportError at runtime.

Comment on lines +126 to +129
self._save_dir = "/tmp/newton_renders"
if os.path.exists(self._save_dir):
shutil.rmtree(self._save_dir)
os.makedirs(self._save_dir, exist_ok=True)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardcoded /tmp path and unconditional rmtree on construction

This constructor unconditionally deletes /tmp/newton_renders via shutil.rmtree every time a NewtonWarpRenderer is instantiated. If multiple renderers are created (e.g. multi-camera setups), the second instance will wipe the first's saved images. This is also fragile in multi-process scenarios.

Consider making the save directory configurable (e.g. via cfg) and/or appending a unique suffix (PID, timestamp) to avoid collisions.

Comment on lines 246 to +254
# -- step interval events
if "interval" in self.event_manager.available_modes:
self.event_manager.apply(mode="interval", dt=self.step_dt)
# -- compute observations
# -- compute observations (includes camera/tiled camera rendering)
# note: done after reset to get the correct observations for reset envs
self.obs_buf = self.observation_manager.compute(update_history=True)
if self.common_step_counter <= 3 or self.common_step_counter % 50 == 0:
msg = (
f"[PERF][manager_based_rl_env] Computing observations (camera/tiled render) "
f"step #{self.common_step_counter}..."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug print statements in hot loop for all users

The conditional print(msg, flush=True) executes for all users on steps 1-3 and every 50th step, regardless of whether Newton Warp is being used. This adds unwanted console noise for all training runs. Consider using logger.debug() or gating behind a verbosity/debug flag.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Comment on lines +300 to +308

@classmethod
def _body_path_to_newton_idx_lookup(cls, body_path: str, root_path: str, body_name: str) -> int:
"""Resolve Newton body index: try exact path, then match body_key by path components."""
idx = cls._body_path_to_newton_idx.get(body_path, -1)
if idx >= 0:
return idx
# Newton's body_key may use different path format; match by root + body_name as last component
suffix = "/" + body_name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Debug logger.warning calls left in production code

The _build_physx_to_newton_mapping method uses logger.warning for "DEBUG" messages (e.g., "[NewtonManager] DEBUG sample Newton body_key (first 15)"). These are development debugging aids that will appear in production logs for all users. Use logger.debug instead of logger.warning for these, or remove them.

Suggested change
@classmethod
def _body_path_to_newton_idx_lookup(cls, body_path: str, root_path: str, body_name: str) -> int:
"""Resolve Newton body index: try exact path, then match body_key by path components."""
idx = cls._body_path_to_newton_idx.get(body_path, -1)
if idx >= 0:
return idx
# Newton's body_key may use different path format; match by root + body_name as last component
suffix = "/" + body_name
if not _debug_done:
newton_keys = list(cls._body_path_to_newton_idx.keys())
logger.debug("[NewtonManager] sample Newton body_key (first 15): %s", newton_keys[:15])
if len(newton_keys) > 20:
logger.debug("[NewtonManager] Newton body_key (last 5): %s", newton_keys[-5:])

Comment on lines +197 to +199
if not hasattr(NewtonManager, "_is_initialized") or not NewtonManager._is_initialized:
device_str = str(self.device).replace("cuda:", "cuda:")
NewtonManager.initialize(num_envs=self._num_envs, device=device_str)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No-op string replace for device conversion

str(self.device).replace("cuda:", "cuda:") replaces "cuda:" with the identical string "cuda:", making this a no-op. If this was intended to strip or transform the device string in some way, the logic is incorrect. Otherwise, it should just be str(self.device).

Suggested change
if not hasattr(NewtonManager, "_is_initialized") or not NewtonManager._is_initialized:
device_str = str(self.device).replace("cuda:", "cuda:")
NewtonManager.initialize(num_envs=self._num_envs, device=device_str)
device_str = str(self.device)

Comment on lines 268 to 281
try:
_repo_root = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..", ".."))
_save_script = os.path.join(_repo_root, "scripts", "save_run_artifacts.sh")
_dest = os.path.join(_repo_root, "run_artifacts", datetime.now().strftime("%Y-%m-%d_%H-%M-%S"))
os.makedirs(_dest, exist_ok=True)
if _timing_lines is not None:
_timing_file = os.path.join(_dest, "timing_summary.txt")
with open(_timing_file, "w") as f:
f.write("Training time: " + str(round(time.time() - start_time, 2)) + " seconds\n")
f.write("[Timing summary]\n")
f.write("\n".join(_timing_lines) + "\n")
if os.path.isfile(_save_script):
import subprocess

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

References non-existent save_run_artifacts.sh script

The code references scripts/save_run_artifacts.sh but this file doesn't exist in the repository. When the script isn't found, the os.path.isfile check prevents a crash, but the timing summary file and os.makedirs side-effect still occur, and the "Run artifacts saved" message is never printed, which may confuse users. Either add the script to the PR or remove this dead code path.

"gym",
], # rl-games still needs gym :(
"rsl-rl": ["rsl-rl-lib==3.1.2", "onnxscript>=0.5"], # linux aarch 64 requires manual onnxscript installation
"rsl-rl": ["rsl-rl-lib==3.3.0", "onnxscript>=0.5"], # linux aarch 64 requires manual onnxscript installation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Major version bump of rsl-rl-lib (3.1.2 → 3.3.0)

This bumps rsl-rl-lib from 3.1.2 to 3.3.0. This is a significant version jump that could introduce breaking API changes (e.g., the new ActorCriticCNN class used by RslRlActorCriticCNNCfg). Make sure this is intentional and that all existing RSL-RL training configurations remain compatible with v3.3.0. Is the bump to rsl-rl-lib==3.3.0 intentional and have existing training configs been validated against the new version?

"junitparser",
"coverage==7.6.1",
"flatdict==4.0.1",
"flatdict>=3.4.0",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Loosened dependency constraint for flatdict

Changing flatdict==4.0.1 to flatdict>=3.4.0 allows any version from 3.4.0 upward, including potential future versions with breaking changes. Unless there's a specific need for an older version, a safer pattern would be flatdict>=3.4.0,<5 to at least pin the major version.

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Feb 18, 2026

Additional Comments (1)

source/isaaclab/isaaclab/sim/simulation_context.py
Excessive debug print statements in production code paths

The reset(), step(), and render() methods now contain numerous import time as _t calls and print(f"[PERF]...") statements that execute in hot paths. These will:

  1. Pollute stdout for all users even when Newton Warp is not in use
  2. Duplicate timing that is already handled by the Timer class
  3. Import time at function level repeatedly (though this is cached by Python, it adds noise)

This instrumentation should be removed or gated behind a debug flag / logging level before merge.

self._data_types = []

# output buffer format: dict mapping data type -> list of buffers (one per camera)
# TODO: Document the standard format of the output data buffers. Need discussion.
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants