Adds visual-based tactile sensor with shape sensing example by JuanaDd · Pull Request #3420 · isaac-sim/IsaacLab

JuanaDd · 2025-09-10T06:18:34Z

Description

This is an implementation of TacSL integrated with Isaac Lab, which demonstrates how to properly configure and use tactile sensors to obtain realistic sensor outputs including tactile RGB images, force fields, and other relevant tactile measurements.

Fixes # (issue)

Type of change

New feature (non-breaking change which adds functionality)

Screenshots

The screenshots of added documentation and simulation outputs.

Checklist

I have run the pre-commit checks with ./isaaclab.sh --format
I have made corresponding changes to the documentation
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
I have updated the changelog and the corresponding version in the extension's config/extension.toml file
I have added my name to the CONTRIBUTORS.md or my name already exists there

clean up license and configs; use full import path; add doc for visuo_tactile_sensor

JuanaDd · 2025-09-10T06:22:21Z

docs/source/overview/core-concepts/sensors/visuo_tactile_sensor.rst

+
+References
+~~~~~~~~~~
+


@iakinola23 Could you help view this documentation here? Thanks!

Hi @iakinola23 updated the documentation with your edits.

JuanaDd · 2025-09-10T06:35:59Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_r15_data/polycalib.npz

@kellyguo11 Hi Kelly, I will upload the npz binaries, and the USD files to nucleus once the licensing concerns are resolved. Thanks!

kellyguo11

just a few high-level comments, haven't gone through the code in detail yet

kellyguo11 · 2025-09-23T19:48:23Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_render.py

+    taxim_gelsight = gelsightRender("gelsight_r15")
+    import ipdb
+
+    ipdb.set_trace()


why is there a breakpoint here?

deleted in the newest commit.

kellyguo11 · 2025-09-23T19:50:47Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sensor.py

+        self._camera_sensor = TiledCamera(self.cfg.camera_cfg)
+
+        # Initialize camera if not already done
+        # TODO: Juana: this is a hack to initialize the camera sensor. Should camera_sensor be managed by TacSL sensor or InteractiveScene?


maybe we can do something similar as the Raycaster camera?

Hi Kelly, I checked the RayCasterCamera implementation, and it differs slightly from the tacsl sensor logic. The tacsl_sensor includes a camera sensor that can be enabled or disabled via the cfg, while RayCasterCamera only uses CameraData without having a camera or other sensor as a member. I’ve kept the current implementation and cleaned up the comments. Please let me know if you have any further concerns or suggestions.

kellyguo11 · 2025-09-23T19:51:17Z

docs/source/_static/overview/sensors/tacsl_demo.png

images should be .jpg

all changed to jpg format and documentation is updated accordingly.

kellyguo11 · 2025-09-23T19:52:14Z

docs/source/overview/core-concepts/sensors/visuo_tactile_sensor.rst

+.. code-block:: bash
+
+    conda activate env_isaaclab
+    pip install opencv-python==4.11.0 trimesh==4.5.1 imageio==2.37.0


do we need to add these to setup.py?

Added opencv-python to setup.py; trimesh is already included, and imageio's dependency is not required in the newest commit.

kellyguo11 · 2025-09-23T19:54:35Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_render.py

+license agreement from NVIDIA CORPORATION is strictly prohibited.
+
+----------------
+Tensorized implementation of RGB rendering of gelsight-style visuo-tactile sensors


can you also add any licenses required for gelsight under docs/licenses?

Updated the license header to be consistent with other files.

kellyguo11 · 2025-09-23T21:22:11Z

please also make sure to run ./isaaclab.sh -f for the linter checks

…ency and readability; Clean comments

iakinola23 · 2025-09-25T01:40:17Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_render.py

+    return dzdx, dzdy
+
+
+def generate_normals(height_map):


generate_normals not needed and deprecated. only the tensorized version is used.

iakinola23 · 2025-09-25T01:41:21Z

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_render.py

+        self.background_tensor = torch.tensor(self.background, device=self.device)
+        print("Gelsight initialization done!")
+
+    def render(self, heightMap):


Delete render as it is not used. Only the tensorized version (render_tensorized ) is used.

deleted render , generate_normals, and compute_image_gradient , padding as well.

zxhuang97 · 2025-09-30T19:31:02Z

Thank you for making TacSL available on IsaacLab! I have three quick questions:

Have you evaluated the physical plausibility of the GelSight finger? For example, would it allow stable grasps of objects like pegs or bolts, similar to what TacSL demonstrated in Isaac Gym?
I noticed that only the USD for gelsight_r15 is currently available. Do you plan to migrate gelsight_mini as well? Alternatively, could you share guidance on converting the URDF to USD while preserving realistic physics?
Is there a way to change the resolution of the tactile image? Currently it's hard-coded in configuration to be 320 x 240

Thanks!

kellyguo11 · 2025-10-17T21:40:16Z

@ooctipus @Mayankm96 could you please help do a review of this one?

JuanaDd · 2025-10-18T04:03:00Z

@zxhuang97 Hi, thanks for your questions and for checking out TacSL on IsaacLab.

Yes. A full task environment (like object grasping) will be supported later after this PR. This PR mainly serves as a reference for how to implement a custom visual-based sensor in the IsaacLab framework.
The GelSight Mini will likely be supported later, as long as there aren’t any license issues. In the meantime, you can try using the URDF importer tool in IsaacSim or the convert_urdf.py script in IsaacLab to convert it yourself. You can tweak the physical properties — such as compliance_stiffness and compliant_damping — in the tacsl sensor config if needed.
The tactile image resolution can be modified through the camera config. Right now, it’s using the values defined in the camera prim included in the asset.

Hope this helps!

docs/source/_static/overview/sensors/tacsl_diagram.jpg

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_render.py

…der.py Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com> Signed-off-by: Juana <yvetted@nvidia.com>

…sor.py Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com> Signed-off-by: Juana <yvetted@nvidia.com>

…sor_cfg.py Co-authored-by: James Tigue <166445701+jtigue-bdai@users.noreply.github.com> Signed-off-by: Juana <yvetted@nvidia.com>

…g.py Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com> Signed-off-by: Juana <yvetted@nvidia.com>

…aming and doc

YuHoChau · 2025-12-29T15:17:19Z

Thank you very much for your contribution! @JuanaDd
Just a quick question: for the Signed Distance Filed (SDF) computation, the TacSL version in IsaacGym uses the built-in function gym.acquire_sdf_view_tensor which supports GPU parallelization. I was wondering which method you used in this Lab version implementation, for example NVIDIA Warp? Thank you.

Signed-off-by: iakinola23 <147214266+iakinola23@users.noreply.github.com>

Signed-off-by: Juana <yvetted@nvidia.com>

JuanaDd · 2025-12-30T06:20:31Z

Thank you very much for your contribution! @JuanaDd Just a quick question: for the Signed Distance Filed (SDF) computation, the TacSL version in IsaacGym uses the built-in function gym.acquire_sdf_view_tensor which supports GPU parallelization. I was wondering which method you used in this Lab version implementation, for example NVIDIA Warp? Thank you.

@YuHoChau Hi, thanks for asking. This implementation uses the PhysX tensor API create_sdf_shape_view. You can see how it's used in the source code here.

YuHoChau · 2026-01-04T19:34:06Z

Thank you very much for your contribution! @JuanaDd Just a quick question: for the Signed Distance Filed (SDF) computation, the TacSL version in IsaacGym uses the built-in function gym.acquire_sdf_view_tensor which supports GPU parallelization. I was wondering which method you used in this Lab version implementation, for example NVIDIA Warp? Thank you.

@YuHoChau Hi, thanks for asking. This implementation uses the PhysX tensor API create_sdf_shape_view. You can see how it's used in the source code here.

Thank you for your reply!

Mayankm96

Merging this for now. I haven't tested it for functionality but the overall MR looks good. Thank you for the contribution.

…m#3420) # Description This is an implementation of TacSL integrated with Isaac Lab, which demonstrates how to properly configure and use tactile sensors to obtain realistic sensor outputs including tactile RGB images, force fields, and other relevant tactile measurements. ## Type of change - New feature (non-breaking change which adds functionality) ## Screenshots The screenshots of added documentation and simulation outputs. <img width="1121" height="878" alt="image" src="https://github.com/user-attachments/assets/5772a87b-474c-4a6c-87f5-b65aab102259" /> <img width="1311" height="669" alt="image" src="https://github.com/user-attachments/assets/8c2d4e66-2c12-4724-b6fd-8180f2fe9960" /> <img width="765" height="281" alt="image" src="https://github.com/user-attachments/assets/ad0e0899-7e3c-429f-9848-700d9310ae1b" /> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by: Juana <yvetted@nvidia.com> Co-authored-by: iakinola23 <147214266+iakinola23@users.noreply.github.com>

visual-based tactile sensor impl. and shape sensing example

6808a09

clean up license and configs; use full import path; add doc for visuo_tactile_sensor

JuanaDd requested review from Mayankm96, jtigue-bdai, kellyguo11 and ooctipus as code owners September 10, 2025 06:18

JuanaDd commented Sep 10, 2025

View reviewed changes

JuanaDd mentioned this pull request Sep 10, 2025

visual-based tactile sensor impl. and shape sensing example #3341

Closed

6 tasks

Mayankm96 changed the title ~~visual-based tactile sensor impl. and shape sensing example~~ Adds visual-based tactile sensor with shape sensing example Sep 10, 2025

Mayankm96 added the enhancement New feature or request label Sep 10, 2025

Merge branch 'main' into tacsl_demo

5f83dd2

github-actions bot added documentation Improvements or additions to documentation isaac-lab Related to Isaac Lab team labels Sep 11, 2025

kellyguo11 reviewed Sep 23, 2025

View reviewed changes

Merge branch 'main' into tacsl_demo

8eecdc0

JuanaDd added 4 commits September 24, 2025 13:28

load assets from nucleus

7c90c47

cherrypick document update

ec93980

update documentation and setup.py, formatting

9115636

Optimized tensor operations in VisuoTactileSensor to improve effici…

e86b370

…ency and readability; Clean comments

iakinola23 reviewed Sep 25, 2025

View reviewed changes

Delete unused functions

a8e1c98

Mayankm96 added this to Isaac Lab Oct 10, 2025

Mayankm96 moved this to In review in Isaac Lab Oct 10, 2025

Mayankm96 reviewed Oct 21, 2025

View reviewed changes

docs/source/_static/overview/sensors/tacsl_diagram.jpg Show resolved Hide resolved

Mayankm96 reviewed Oct 21, 2025

View reviewed changes

source/isaaclab/isaaclab/sensors/tacsl_sensor/gelsight_render.py Outdated Show resolved Hide resolved

JuanaDd and others added 12 commits December 24, 2025 18:34

Merge branch 'main' into tacsl_demo

f657072

Update source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_ren…

cf70346

…der.py Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com> Signed-off-by: Juana <yvetted@nvidia.com>

Update source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_ren…

648b93c

…der.py Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com> Signed-off-by: Juana <yvetted@nvidia.com>

Update source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sen…

09844e2

…sor.py Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com> Signed-off-by: Juana <yvetted@nvidia.com>

Update source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sen…

b47e826

…sor.py Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com> Signed-off-by: Juana <yvetted@nvidia.com>

Update source/isaaclab/isaaclab/sensors/tacsl_sensor/visuotactile_sen…

e959e4c

…sor_cfg.py Co-authored-by: James Tigue <166445701+jtigue-bdai@users.noreply.github.com> Signed-off-by: Juana <yvetted@nvidia.com>

Update source/isaaclab/isaaclab/sim/spawners/from_files/from_files_cf…

811ce7c

…g.py Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com> Signed-off-by: Juana <yvetted@nvidia.com>

Improve docstrings

18ac547

Removed the benchmarking code from the sensor

e3e5cd1

Support applying physics materials to multiple rigid bodies, better n…

3b085da

…aming and doc

velocity correction

a93c29e

Call create_sdf_shape_view from physx view directly

49e1d65

iakinola23 and others added 4 commits December 29, 2025 19:24

Minor docstring updates to visuotactile_render.py

498d02c

Signed-off-by: iakinola23 <147214266+iakinola23@users.noreply.github.com>

Merge branch 'main' into tacsl_demo

a1fe606

Signed-off-by: Juana <yvetted@nvidia.com>

apply logging update

f331ef3

code format

439187c

JuanaDd added 5 commits January 5, 2026 19:25

Fixed typo

95a32cd

Merge branch 'main' into tacsl_demo

781e03e

Merge branch 'main' into tacsl_demo

127a357

Merge branch 'main' into tacsl_demo

eb4aca4

Mayankm96 approved these changes Jan 8, 2026

View reviewed changes

Mayankm96 merged commit bd7cd1b into isaac-sim:main Jan 8, 2026
7 checks passed

github-project-automation bot moved this from In review to Done in Isaac Lab Jan 8, 2026


		References
		~~~~~~~~~~

Conversation

JuanaDd commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Screenshots

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JuanaDd Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kellyguo11 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kellyguo11 commented Sep 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zxhuang97 commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kellyguo11 commented Oct 17, 2025

Uh oh!

JuanaDd commented Oct 18, 2025

Uh oh!

Uh oh!

Uh oh!

YuHoChau commented Dec 29, 2025

Uh oh!

JuanaDd commented Dec 30, 2025

Uh oh!

YuHoChau commented Jan 4, 2026

Uh oh!

Mayankm96 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

JuanaDd commented Sep 10, 2025 •

edited

Loading

JuanaDd Sep 10, 2025 •

edited

Loading

zxhuang97 commented Sep 30, 2025 •

edited

Loading