Shadow Hand Environments Newton Implementation#4616
Shadow Hand Environments Newton Implementation#4616nv-rgresia wants to merge 10 commits intoisaac-sim:dev/newtonfrom
Conversation
|
Too many files changed for review. ( |
nv-rgresia
requested review from
ClemensSchwarke,
Mayankm96,
Toni-SM,
hhansen-bdai,
jtigue-bdai,
kellyguo11,
masoudmoghani,
matthewtrepte,
ooctipus,
ossamaAhmed,
pascal-roth,
peterd-NV and
xyao-nv
as code owners
github-actions
bot
added
documentation
kellyguo11
left a comment
kellyguo11
left a comment
There was a problem hiding this comment.
Thanks, looks good overall! Does the Newton renderer actually provide segmentation data? or that's just an empty buffer for now?
I also wonder if the difference in training performance compared to PhysX might be from the tendon setup or missing sensor data?
|
The segmentation data is indeed empty buffers; I wonder if the lack of color data is impacting it as well. I'll have to do some benchmarking between all the environments to determine whats going on. The current allegro hand checkpoint looks to be underperforming as well and I can compare it against the basic environment, but i'll need to do the training first. Also I've added another commit the OpenAI FF environment, the EventCfg has some limitations due to Newton currently, but its able to start training. The OpenAI LSTM should be working as well, although I get a key error in the rl_games train script, is this a known issue? |
| # if dict, then parse the regular expression | ||
| indices, _, values = string_utils.resolve_matching_names_values(cfg_value, self.joint_names) | ||
| indices_global = [self._joint_indices[i] for i in indices] | ||
| indices_global = [self._joint_indices[i].cpu() for i in indices] |
There was a problem hiding this comment.
Is that strictly needed?
|
|
||
| if len(joint_names) == self.num_joints: | ||
| joint_indices = slice(None) | ||
| else: | ||
| joint_indices = torch.tensor(joint_indices, device=self.device) |
There was a problem hiding this comment.
Is that actually needed?
There was a problem hiding this comment.
That might explain the cpu() in the base actuator?
AntoineRichard
left a comment
AntoineRichard
left a comment
There was a problem hiding this comment.
Thanks for PR! I've left a couple comments. I think there is an issue with the joint_pos_limits, but i'm working on making that homogeneous across PhysX and Newton.
| def write_joint_position_limit_to_sim( | ||
| self, | ||
| lower_limits: wp.array | float, | ||
| upper_limits: wp.array | float, | ||
| limits: wp.array | float, | ||
| joint_ids: Sequence[int] | None = None, | ||
| env_ids: Sequence[int] | None = None, | ||
| joint_mask: wp.array | None = None, | ||
| env_mask: wp.array | None = None, | ||
| ) -> None: | ||
| """Write joint position limits into the simulation. | ||
|
|
||
| Args: | ||
| lower_limits: Joint lower limits. Shape is (num_instances, num_joints). | ||
| upper_limits: Joint upper limits. Shape is (len(env_ids), len(joint_ids)) or (num_instances, num_joints). | ||
| joint_ids: The joint indices to set the targets for. Defaults to None (all joints). | ||
| env_ids: The environment indices to set the targets for. Defaults to None (all environments). | ||
| joint_mask: The joint mask. Shape is (num_joints). | ||
| env_mask: The environment mask. Shape is (num_instances,). | ||
| """ | ||
| # Resolve indices into mask, convert from partial data to complete data, handles the conversion to warp. | ||
| if isinstance(upper_limits, torch.Tensor): | ||
| if self._temp_joint_pos is None: | ||
| self._temp_joint_pos = wp.zeros( | ||
| (self.num_instances, self.num_joints), dtype=wp.float32, device=self.device | ||
| ) | ||
|
|
||
| if isinstance(limits, torch.Tensor): | ||
| upper_limits = make_complete_data_from_torch_dual_index( | ||
| upper_limits, | ||
| limits[:, 0, :, 0], | ||
| self.num_instances, | ||
| self.num_joints, | ||
| env_ids, | ||
| joint_ids, | ||
| dtype=wp.float32, | ||
| device=self.device, | ||
| out=self._temp_joint_pos, | ||
| ) | ||
| if isinstance(lower_limits, torch.Tensor): | ||
| if self._temp_joint_vel is None: | ||
| self._temp_joint_vel = wp.zeros( | ||
| (self.num_instances, self.num_joints), dtype=wp.float32, device=self.device | ||
| ) | ||
|
|
||
| if isinstance(limits, torch.Tensor): | ||
| lower_limits = make_complete_data_from_torch_dual_index( | ||
| lower_limits, | ||
| limits[:, 0, :, 1], | ||
| self.num_instances, | ||
| self.num_joints, | ||
| env_ids, | ||
| joint_ids, | ||
| dtype=wp.float32, | ||
| device=self.device, | ||
| out=self._temp_joint_vel, | ||
| ) |
There was a problem hiding this comment.
Why remove the lower and upper limits? in a pure warp path, what would happen?
3 participants
Description
Imports the shadow hand environments to Newton. Currently the working environments are Isaac-Repose-Cube-Shadow-Direct-v0 and 'Isaac-Repose-Cube-Shadow-Vision-Direct-v0', but all around model performance is lacking during testing.
Type of change
Checklist
pre-commitchecks with./isaaclab.sh --formatconfig/extension.tomlfileCONTRIBUTORS.mdor my name already exists there