Inconsistent Success Rates Across All Tasks with Provided Checkpoints (PyBullet)

[yazidmarzuk]

Hi, thanks for releasing M2Diffuser and the official checkpoints!

I attempted to reproduce the results reported in the paper using the provided inference scripts and pretrained checkpoints, running in PyBullet mode (headless). However, I am observing noticeably different success rates across all three tasks: Pick, Place, and Goal-Reach.

Command I used to run inference:

bash ./scripts/model-m2diffuser/<task>/inference.sh checkpoints/checkpoints/MK-M2Diffuser-<task>/2024-07-14-09-38-10/

My System(Docker):

OS: Ubuntu 20.04
Python: 3.8
PyTorch: 1.13.1+cu116
PyBullet: 3.2.6
sim_gui: true

So I just wanted to know:

1. Are there any known sources of variation between the inference scripts and the final paper results?
2. Could the discrepancy be due to the use of PyBullet instead of Isaac Sim for evaluation? Was the paper's reported success based on Isaac Sim or PyBullet inference?

[sixu-yan]

Hi, thank you for your interest in M2Diffuser and for taking the time to reproduce our results!

Regarding your question: the PyBullet-based inference is primarily used to verify whether the generated trajectories satisfy physical constraints, such as collision avoidance and motion smoothness. However, task success—i.e., whether the robot successfully completes the Pick or Place task—must be evaluated in Isaac Sim, where full task dynamics and success metrics can be reliably assessed.

The success rates reported in the paper are based on evaluations conducted in Isaac Sim, and reflect both physical feasibility and task completion. The discrepancies you're observing in PyBullet are expected, as PyBullet only checks for physical plausibility, not task-specific outcomes.

Currently, the Isaac Sim-based evaluation code is undergoing internal review and requires approval from BIGAI before we can release it. We appreciate your patience while we complete this process.

Thanks again for your attention and support!

[yazidmarzuk]

Hi team,

Thanks for the clarification—good to know the paper’s numbers were obtained entirely in Isaac Sim.

Could you share an approximate timeline for when the Isaac Sim evaluation scripts will be released? Even a rough ETA would help us plan our own benchmarks and ablation studies.

Appreciate the quick response and the continued support.

[sixu-yan]

Thanks for your patience. We’ve checked with our colleagues at BIGAI—Isaac Sim evaluation scripts will be released this week as part of the M3Bench repository. The M2Diffuser-specific version may take a bit longer to sync up, but the usage is essentially the same.

If you're in a hurry, we recommend trying out the version in M3Bench first. Let us know if you run into any issues or need help getting started.