Add Pico sim2real recording support

Author: BotRunner64

AGENTS.md

@@ -146,6 +146,9 @@ target_dof_pos = clip(action, -10, 10) × action_scale + default_dof_pos
 - `ARMS` entering/exiting/resume resets policy/reference alignment and uses Kp ramp; offline BVH sim2real does not use `ARMS`, and Unitree remote `B` remains BVH replay
 - Realtime mode switches and pause/resume use a retargeter-preserving soft reset: policy/reference state, smoothers, and reference alignment are reset, while the GMR IK warm-start is retained
 - Optional LinkerHand control uses `hands.enabled=true`, `hands.driver=linkerhand_l6|linkerhand_o6`, and `hands.mode=gripper|vr_hand_pose`; default is disabled
+- Optional Pico sim2real LeRobot v3 recording uses `--config-name sim2real_record` or `recording.enabled=true`; it requires `input.provider=pico4`, `input.video.enabled=true`, `input.video.source=realsense`, an interactive terminal, and the `recording` extra
+- Recording is manual only: terminal `R` starts an episode, `S` saves, `D` discards the active episode, and `Q` shuts down; `STANDING`, `MOCAP`, `ARMS`, and paused mocap are recordable
+- Recording captures `observation.images.d435i_rgb` RealSense RGB video at 30Hz plus `observation.state(68)`, `observation.mode(1)`, and `action(36)`; RealSense capture lives in `pico_input` through the normal `input.video` path
 - `gripper` mode reuses `Pico4InputProvider.get_controller_snapshot()` for Pico grip/trigger open-close control and supports LinkerHand L6 and O6
 - `vr_hand_pose` mode reuses `Pico4InputProvider.get_hand_snapshot()` and somehand 0.2.0 public `somehand.api` for continuous Pico hand-pose retargeting; do not start a second `PicoBridge` for hand control
 - Teleopit owns Pico 26-joint hand-state to 21-landmark conversion; do not import `somehand.pico_input`

README.md

@@ -84,6 +84,23 @@ python train_mimic/scripts/data/build_dataset.py \
     --spec data/pico_motion/pico_recorded.yaml --force
 ```
 
+## Sim2Real LeRobot Recording
+
+Pico sim2real can also record manual LeRobot v3 episodes from the real G1:
+
+```bash
+pip install -e '.[recording]'
+python scripts/run/run_sim2real.py --config-name sim2real_record \
+    controller.policy_path=track.onnx \
+    recording.task="walk forward"
+```
+
+Recording uses the terminal controls `R` start, `S` save, `D` discard, and `Q`
+shutdown. `STANDING`, `MOCAP`, `ARMS`, and paused mocap can be recorded. The
+dataset schema is `observation.images.d435i_rgb` video at 30 Hz,
+`observation.state(68)`, `observation.mode(1)`, and `action(36)` as the aligned
+reference qpos sent to the policy path.
+
 ## Documentation
 
 Full docs at **[BotRunner64.github.io/Teleopit](https://BotRunner64.github.io/Teleopit/)**, covering installation profiles, all tutorials, configuration reference, and architecture.
@@ -95,6 +112,7 @@ Full docs at **[BotRunner64.github.io/Teleopit](https://BotRunner64.github.io/Te
 - Added Pico sim2real `ARMS` mode: Pico/controller `B` toggles between whole-body `MOCAP` and stand-pose body/legs with live retargeted arms.
 - Bumped Pico input support to pico-bridge 0.2.1 and its corrected tracking pose semantics.
 - Added optional LinkerHand L6 sim2real modes under `hands.*`: `gripper` from Pico grip/trigger and `vr_hand_pose` from Pico hand pose through somehand 0.2.0 public API.
+- Added manual Pico sim2real LeRobot v3 recording with RealSense D435i RGB video, 68D robot state, mode labels, and 36D reference-qpos action labels.
 - Added LinkerHand O6 support for Pico `gripper` mode with an O6-specific grasp pose.
 - Set LinkerHand L6 `vr_hand_pose` control to maximum speed while keeping `gripper` at the configured default speed.
 - Switched default `vr_hand_pose` to a low-latency somehand path with 60 Hz hand retargeting and reduced smoothing.

docs/docs/configuration/config-reference.md

@@ -151,6 +151,48 @@ calling somehand 0.2.0 through `somehand.api` only.
 | `hands.somehand.temporal_filter_alpha` | somehand input landmark smoothing alpha; `1.0` disables smoothing delay | `1.0` |
 | `hands.somehand.output_alpha` | somehand qpos output smoothing alpha; `1.0` disables smoothing delay | `1.0` |
 
+### LeRobot Recording (Pico sim2real)
+
+`recording.enabled=true` is supported only with `input.provider=pico4`,
+`input.video.enabled=true`, `input.video.source=realsense`, and an interactive
+terminal. The recorder is manual: `R` starts an episode, `S` saves the active
+episode, `D` discards the active episode, and `Q` shuts down. `STANDING`,
+`MOCAP`, `ARMS`, and paused mocap can be recorded.
+
+`sim2real_record.yaml` enables both recording and the required RealSense
+`input.video` path. Recording does not open a second camera; it consumes the
+same frames produced by `pico_input`.
+
+| Field | Description | Default |
+|-------|-------------|---------|
+| `recording.enabled` | Enable manual LeRobot v3 recording | `false` |
+| `recording.output_dir` | Dataset root directory | `data/lerobot` |
+| `recording.repo_id` / `dataset_name` | LeRobot dataset identity | `null` |
+| `recording.task` | Task string stored with frames | `demo` |
+| `recording.fps` | Recording/video clock rate | `30` |
+| `recording.min_episode_seconds` | Discard saved episodes shorter than this duration | `1.0` |
+| `recording.record_modes` | Modes that allow recording start and frame writes | `[standing, mocap, arms, pause]` |
+| `recording.camera.key` | LeRobot video feature key | `observation.images.d435i_rgb` |
+| `recording.camera.width` / `height` / `fps` | RealSense RGB capture settings | `640` / `480` / `30` |
+| `recording.camera.device` | Optional RealSense serial | `null` |
+
+Camera failure behavior is controlled by `input.video.fail_on_error`.
+
+LeRobot features:
+
+```text
+observation.images.d435i_rgb   video [480,640,3] uint8
+observation.state              float32[68]
+observation.mode               float32[1]
+action                         float32[36]
+```
+
+`observation.state` is ordered as `joint_pos(29)`, `joint_vel(29)`,
+`base_quat_wxyz(4)`, `base_ang_vel(3)`, and `projected_gravity(3)`.
+`observation.mode` is a numeric categorical: `standing=0`, `mocap=1`,
+`arms=2`, and `pause=3`. `action` is the current reference qpos:
+`root_pos(3) + root_quat_wxyz(4) + joint_pos(29)`.
+
 ## Critical: `default_dof_pos`
 
 The RL policy outputs action **offsets** relative to the default standing pose, not absolute joint angles:

docs/docs/getting-started/installation.md

@@ -73,6 +73,15 @@ scripts/setup/download_somehand_l6_assets.sh
 
 This extra is only required when `hands.enabled=true`.
 
+### Sim2Real Recording
+
+```bash
+pip install -e '.[recording]'
+```
+
+Adds the Pico sim2real stack plus LeRobot, RealSense, and video encoding
+dependencies used by `sim2real_record.yaml`.
+
 ## Verify Installation
 
 ```bash

docs/docs/tutorials/pico-sim2real.md

@@ -95,6 +95,30 @@ python scripts/run/run_sim2real.py \
     real_robot.network_interface=eth0
 ```
 
+## Optional LeRobot Recording
+
+Install the recording extra on the machine that owns Pico input and RealSense:
+
+```bash
+pip install -e '.[recording]'
+```
+
+Run the recording config:
+
+```bash
+python scripts/run/run_sim2real.py \
+    --config-name sim2real_record \
+    controller.policy_path=track.onnx \
+    real_robot.network_interface=enp130s0 \
+    recording.task="walk forward"
+```
+
+Terminal controls are `R` start episode, `S` save, `D` discard, and `Q`
+shutdown. `STANDING`, `MOCAP`, `ARMS`, and paused mocap can be recorded;
+saved episodes cannot be discarded afterward. The v1 schema records
+`observation.images.d435i_rgb`, `observation.state(68)`,
+`observation.mode(1)`, and `action(36)` at 30 Hz.
+
 ## Operator Flow
 
 Keep the Unitree remote in hand. `L1+R1` is the emergency stop path into

docs/i18n/zh-Hans/docusaurus-plugin-content-docs/current/configuration/config-reference.md

@@ -169,3 +169,43 @@ Teleopit 会先将 Pico 手部状态转成 21 个 landmarks，再只通过 someh
 | `hands.somehand.max_iterations` | `vr_hand_pose` 的 somehand solver 迭代上限 | `12` |
 | `hands.somehand.temporal_filter_alpha` | somehand 输入 landmarks 平滑 alpha；`1.0` 表示关闭平滑延时 | `1.0` |
 | `hands.somehand.output_alpha` | somehand qpos 输出平滑 alpha；`1.0` 表示关闭平滑延时 | `1.0` |
+
+### LeRobot 录制（Pico sim2real）
+
+`recording.enabled=true` 只支持 `input.provider=pico4`、
+`input.video.enabled=true`、`input.video.source=realsense`，并且需要交互式终端。
+录制是手动控制：`R` 开始 episode，`S` 保存当前 episode，`D` 丢弃当前 episode，
+`Q` 关闭。可以录制 `STANDING`、`MOCAP`、`ARMS` 和暂停状态的 mocap。
+
+`sim2real_record.yaml` 会同时启用录制和必需的 RealSense `input.video`
+路径。录制不会打开第二路相机，而是消费 `pico_input` 已经产生的同一批帧。
+
+| 字段 | 说明 | 默认值 |
+|---|---|---|
+| `recording.enabled` | 启用手动 LeRobot v3 录制 | `false` |
+| `recording.output_dir` | 数据集根目录 | `data/lerobot` |
+| `recording.repo_id` / `dataset_name` | LeRobot 数据集标识 | `null` |
+| `recording.task` | 写入 frame 的任务字符串 | `demo` |
+| `recording.fps` | 录制/视频主时钟频率 | `30` |
+| `recording.min_episode_seconds` | 保存时短于该时长的 episode 会被丢弃 | `1.0` |
+| `recording.record_modes` | 允许开始录制和写帧的模式 | `[standing, mocap, arms, pause]` |
+| `recording.camera.key` | LeRobot 视频 feature key | `observation.images.d435i_rgb` |
+| `recording.camera.width` / `height` / `fps` | RealSense RGB 采集设置 | `640` / `480` / `30` |
+| `recording.camera.device` | 可选 RealSense 序列号 | `null` |
+
+相机失败时的行为由 `input.video.fail_on_error` 控制。
+
+LeRobot features：
+
+```text
+observation.images.d435i_rgb   video [480,640,3] uint8
+observation.state              float32[68]
+observation.mode               float32[1]
+action                         float32[36]
+```
+
+`observation.state` 的顺序是 `joint_pos(29)`、`joint_vel(29)`、
+`base_quat_wxyz(4)`、`base_ang_vel(3)` 和 `projected_gravity(3)`。
+`observation.mode` 是数值类别：`standing=0`、`mocap=1`、
+`arms=2`、`pause=3`。`action` 是当前 reference qpos：
+`root_pos(3) + root_quat_wxyz(4) + joint_pos(29)`。

docs/i18n/zh-Hans/docusaurus-plugin-content-docs/current/getting-started/installation.md

@@ -72,6 +72,15 @@ scripts/setup/download_somehand_l6_assets.sh
 
 只有在 `hands.enabled=true` 时才需要安装这个 extra。
 
+### Sim2Real 录制
+
+```bash
+pip install -e '.[recording]'
+```
+
+该配置包含 Pico sim2real 栈，以及 `sim2real_record.yaml` 使用的 LeRobot、
+RealSense 和视频编码依赖。
+
 ## 验证安装
 
 ```bash

docs/i18n/zh-Hans/docusaurus-plugin-content-docs/current/tutorials/pico-sim2real.md

@@ -91,6 +91,29 @@ python scripts/run/run_sim2real.py \
     real_robot.network_interface=eth0
 ```
 
+## 可选 LeRobot 录制
+
+在负责 Pico 输入和 RealSense 的机器上安装 recording extra：
+
+```bash
+pip install -e '.[recording]'
+```
+
+运行录制配置：
+
+```bash
+python scripts/run/run_sim2real.py \
+    --config-name sim2real_record \
+    controller.policy_path=track.onnx \
+    real_robot.network_interface=enp130s0 \
+    recording.task="walk forward"
+```
+
+终端控制为：`R` 开始 episode，`S` 保存，`D` 丢弃，`Q` 关闭。可以录制
+`STANDING`、`MOCAP`、`ARMS` 和暂停状态的 mocap；已经保存的 episode 不支持再丢弃。
+v1 schema 以 30 Hz 记录 `observation.images.d435i_rgb`、`observation.state(68)`、
+`observation.mode(1)` 和 `action(36)`。
+
 ## 操作流程
 
 始终把 Unitree 遥控器拿在手里。`L1+R1` 是进入 `DAMPING` 的急停路径。

pyproject.toml

@@ -53,6 +53,13 @@ pico4 = [
     "pico-bridge[camera] @ https://github.com/BotRunner64/pico-bridge/releases/download/v0.2.1/pico_bridge-0.2.1-py3-none-any.whl",
     "teleopit[sim2real]",
 ]
+recording = [
+    "teleopit[pico4]",
+    "lerobot",
+    "pyrealsense2",
+    "opencv-python",
+    "imageio[ffmpeg]",
+]
 dexhand = [
     "linkerhand-python-sdk @ file:third_party/linkerhand-python-sdk",
     "somehand @ file:third_party/somehand",

teleopit/configs/pico4_sim2real.yaml

@@ -18,6 +18,27 @@ reference_anchor_velocity_smoothing_alpha: 0.25
 reference_steps: [0]
 reference_debug_log: false
 
+recording:
+  enabled: false
+  format: lerobot_v3
+  output_dir: data/lerobot
+  dataset_name: null
+  repo_id: null
+  task: demo
+  fps: 30
+  control: terminal
+  min_episode_seconds: 1.0
+  discard_on_shutdown: true
+  record_modes: [standing, mocap, arms, pause]
+  camera:
+    enabled: true
+    key: observation.images.d435i_rgb
+    source: realsense
+    width: 640
+    height: 480
+    fps: 30
+    device: null
+
 runtime:
   host: 127.0.0.1
   base_port: 39700