Skip to content

Commit 746baee

Browse files
BotRunner64BotRunner64
committed
docs: reorganize reference guides and resources
1 parent 439a7e5 commit 746baee

36 files changed

Lines changed: 881 additions & 848 deletions

‎docs/docs/configuration/faq.md‎

Lines changed: 0 additions & 29 deletions
This file was deleted.

‎docs/docs/getting-started/installation.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -91,8 +91,8 @@ python scripts/setup/download_assets.py \
9191

9292
The inference bundle creates `track.onnx`, the G1 model files, GMR files and a
9393
sample BVH under their expected project paths. See
94-
[Asset Reference](../reference/assets) for the complete inventory and asset
95-
group mapping.
94+
[Assets](../reference/resources/assets) for the complete inventory
95+
and asset group mapping.
9696

9797
## 5. Additional Setup for a Physical G1
9898

@@ -104,8 +104,8 @@ bash scripts/setup/setup_g1_bridge.sh
104104
```
105105

106106
The bridge is required for both Pico and BVH control on a real G1. See
107-
[G1 Bridge SDK](../reference/g1-bridge-sdk) if the build or robot connection
108-
fails.
107+
[Companion Projects](../reference/companion-projects#g1-bridge-sdk) if the
108+
build or robot connection fails.
109109

110110
## 6. Optional Hardware
111111

‎docs/docs/intro.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -40,5 +40,6 @@ hand during hardware operation; `L1+R1` is the emergency path to `DAMPING`.
4040

4141
The user guides intentionally keep internals out of the main flow. See
4242
[Architecture](reference/architecture) for the runtime pipeline and technical
43-
specifications, [Asset Reference](reference/assets) for every downloaded file,
44-
or [Configuration](configuration/overview) for Hydra options.
43+
specifications, [Assets](reference/resources/assets) for every
44+
downloaded file, or
45+
[Configuration](reference/configuration/overview) for Hydra options.

‎docs/docs/reference/architecture.md‎

Lines changed: 126 additions & 103 deletions
Original file line numberDiff line numberDiff line change
@@ -1,135 +1,158 @@
11
---
2-
sidebar_position: 1
2+
sidebar_position: 2
33
---
44

55
# Architecture
66

7-
This page collects the runtime pipeline, supported boundaries and exact
8-
dimensions that are intentionally omitted from the task-based user guides.
7+
This page defines Teleopit's runtime pipelines, repository layout, supported
8+
technical surface, and public entry points.
99

1010
## Pipeline
1111

12-
```text
13-
InputProvider (BVH file / Pico4)
14-
-> Retargeter (GMR)
15-
-> ObservationBuilder (167D)
16-
-> Controller (dual-input TemporalCNN ONNX)
17-
-> Robot (MuJoCo sim or Unitree G1)
18-
```
12+
![Teleopit runtime pipelines](/img/diagrams/architecture-pipeline.svg)
1913

20-
Offline/online inference is assembled by `teleopit/runtime/` and `teleopit/pipeline.py`. The hardware state machine runs through the process-isolated runtime in `teleopit/sim2real/mp/`. Training is provided by `train_mimic/`.
14+
The main tracking path converts BVH or live PICO body motion into a time-aligned
15+
G1 reference. `VelCmdObservationBuilder` combines that reference with robot
16+
state, and the dual-input TemporalCNN ONNX controller produces 29 joint offsets.
17+
The same observation and controller path drives MuJoCo and the real G1.
2118

22-
## Full-Embodiment Pico Path
19+
Pico hand and active-vision paths are optional process-isolated workers. They
20+
reuse the same in-process `PicoBridge` receiver and never add fields to the 167D
21+
tracking-policy observation. A hand or neck failure must not stop G1 body
22+
control. These optional hardware paths are supported by onboard deployment;
23+
external-host Pico deployment supports whole-body control only.
2324

24-
One Pico frame can feed three independent control paths:
25+
Host-policy deployment is independent from the Pico runtime. A separate host
26+
environment receives JPEG RGB and `observation.state(68)`, then returns
27+
canonical `float32[T,50]` action chunks over strict ZeroMQ/msgpack messages.
28+
The onboard validator and scheduler convert the body portion into a 36D
29+
reference for the existing motion tracker; host output never bypasses that
30+
tracker or becomes a direct motor command.
2531

26-
```text
27-
Pico full-body tracking
28-
-> GMR retargeting -> tracking policy -> G1 whole-body joints
32+
The Teleopit and host environments share semantic data and one identical
33+
`hand_calibration.json`, but do not import each other's Python packages. The
34+
current client/server code and protocol tests define the network structure, so
35+
both repositories must change together when that protocol changes.
2936

30-
Pico hand tracking or controller input
31-
-> Teleopit hand adapter -> somehand or gripper mapping -> LinkerHand L6/O6
37+
## Runtime Boundaries
3238

33-
Pico HMD rotation + same-frame Spine3 rotation
34-
-> relative yaw/pitch mapping -> OpenNeck
35-
```
39+
- Offline core components communicate through `InProcessBus` without copying
40+
array payloads.
41+
- Sim2real robot control, reference generation, camera, recording, hand, neck,
42+
and host-policy client work are process-isolated where blocking or hardware
43+
failure could disturb the 50 Hz control loop.
44+
- Local sim2real workers use localhost ZeroMQ and shared-memory video rings.
45+
- The external host-policy boundary uses msgpack and non-pickle float32 arrays.
46+
- Shared component contracts are `typing.Protocol` definitions in
47+
`teleopit/interfaces.py`.
3648

37-
Whole-body control is the required path. Hands and OpenNeck are optional
38-
process-isolated workers for onboard deployment; their failure must not stop G1
39-
body control. External-host Pico deployment supports the whole-body path only.
40-
All active paths reuse the same in-process PicoBridge receiver.
41-
42-
Host-served imitation policies use a second, independent deployment path:
49+
## Repository Layout
4350

4451
```text
45-
lerobot-teleopit host environment
46-
policy server -> strict ZeroMQ/msgpack messages
47-
|
48-
Teleopit onboard environment
49-
RealSense/state -> non-critical client worker -> validated action scheduler
50-
-> existing 50 Hz motion tracker -> G1 joint-angle targets
51-
-> dedicated LinkerHand O6 and OpenNeck workers
52+
teleopit/ — Core inference and deployment package
53+
├── interfaces.py — Robot, controller, input and retargeting protocols
54+
├── pipeline.py — Thin offline simulation facade
55+
├── runtime/ — Config/path resolution, factories and CLI validation
56+
├── configs/ — Hydra runtime configuration
57+
├── bus/ — In-process zero-copy publish/subscribe
58+
├── inputs/ — BVH, PICO and realtime input adapters
59+
├── retargeting/gmr/ — Self-contained whole-body GMR implementation
60+
├── controllers/ — Observation builder and ONNX policy controller
61+
├── robots/ — MuJoCo robot adapter
62+
├── sim/ — 200 Hz PD / 50 Hz policy simulation loop
63+
├── sim2real/
64+
│ ├── mp/ — Process supervisor, IPC and robot-control state machine
65+
│ ├── hands/ — Optional LinkerHand drivers and input mapping
66+
│ └── neck/ — Optional OpenNeck mapping and worker
67+
├── high_level_policy/ — Host protocol, frame transforms and action scheduler
68+
└── recording/ — Sim2real dataset schema and recording workers
69+
70+
train_mimic/ — Training package
71+
├── app.py — Shared train/play/benchmark assembly
72+
├── tasks/tracking/ — General-Tracking-G1 task and TemporalCNN model
73+
├── data/ — Dataset construction and motion loading
74+
└── scripts/ — Training, playback, benchmark and ONNX export
75+
76+
scripts/ — User-facing runtime and maintenance entry points
77+
├── run/ — Simulation, sim2real and recording commands
78+
├── setup/ — Asset download and hardware setup
79+
├── render/ — Offline video rendering
80+
├── view/ — Recording review
81+
└── dev/ — Validation and calibration utilities
82+
83+
third_party/ — Optional hardware SDKs and somehand
84+
tests/ — Unit, protocol and integration tests
5285
```
5386

54-
The host and onboard environments share semantic data and one identical
55-
`hand_calibration.json`; they do not import each other's Python packages. The
56-
current client/server code and protocol tests define the network structure, so
57-
both repositories must change together during active development. Pico
58-
teleoperation and host-policy deployment also have separate run scripts and
59-
process assemblies.
60-
61-
## Code Structure
62-
63-
```text
64-
configs / scripts
65-
-> runtime
66-
-> interfaces + pipeline state machines
67-
-> adapters (inputs / retargeting / controller / robot / recording)
68-
69-
train_mimic/scripts
70-
-> train_mimic/app.py
71-
-> single task registry / env builder / runner cfg
72-
-> mjlab / rsl_rl
73-
74-
train_mimic/scripts/data
75-
-> train_mimic/data/dataset_builder.py
76-
-> dataset_lib / motion_fk / convert_pkl_to_npz
77-
```
78-
79-
## Core Boundaries
80-
81-
| Module | Role |
82-
|--------|------|
83-
| `teleopit/interfaces.py` | Stable protocols: InputProvider, Retargeter, Controller, Robot, ObservationBuilder |
84-
| `teleopit/runtime/` | Config parsing, path normalization, component assembly, CLI validation |
85-
| `teleopit/pipeline.py` | Lightweight facade for offline sim |
86-
| `teleopit/sim2real/mp/` | Process-isolated sim2real state machine, IPC, and robot-control loop |
87-
| `teleopit/high_level_policy/` | Host-policy protocol, session-local frame transform, validation, and 30-to-50 Hz scheduler |
88-
| `teleopit/controllers/observation.py` | ObservationBuilder |
89-
| `teleopit/controllers/rl_policy.py` | Accepts dual-input ONNX whose observation dimension matches the runtime builder |
90-
| `train_mimic/app.py` | Shared train/play/benchmark assembly |
91-
| `train_mimic/tasks/tracking/config/` | Single task registration (`General-Tracking-G1`) |
92-
| `train_mimic/data/dataset_builder.py` | Sole official dataset construction entry |
93-
9487
## Technical Specifications
9588

96-
| Spec | Value |
97-
|------|-------|
98-
| Supported robot | Unitree G1, 29 actuated joints |
89+
| Specification | Supported value |
90+
|---------------|-----------------|
91+
| Robot | Unitree G1 with 29 actuated joints |
9992
| Simulator | MuJoCo |
100-
| Motion retargeting | GMR (General Motion Retargeting) |
93+
| Whole-body retargeting | GMR (General Motion Retargeting) |
10194
| Policy / PD rates | 50 Hz / 200 Hz |
10295
| Training task | `General-Tracking-G1` |
10396
| Inference observation | `velcmd_history` (167D) |
104-
| ONNX signature | Dual-input `obs` (167D) + `obs_history` |
97+
| ONNX signature | Dual input: `obs` (167D) + `obs_history` |
10598
| Policy action | 29D joint offsets from `default_dof_pos` |
106-
| Actor/Critic | TemporalCNN (2048, 1024, 512, 256, 128) |
107-
| Training sampling | Default `rewind`; also supports `uniform`; playback uses `start`; benchmark pins exact clips and disables clip-end resampling |
108-
| Training `window_steps` | `[0]` |
109-
| Data format | Minimal recursive HDF5 shards (`shard_*.h5`) |
110-
| Optional hands | LinkerHand L6 or O6, gripper or Pico hand-pose input |
99+
| Actor / critic | TemporalCNN (2048, 1024, 512, 256, 128) |
100+
| Training sampling | `rewind` by default; `uniform` supported; playback uses `start`; benchmark pins exact clips and disables clip-end resampling |
101+
| Training window | `window_steps=[0]` |
102+
| Distributed motion data | Minimal recursive HDF5 `shard_*.h5` files |
103+
| Optional hands | LinkerHand L6/O6 with gripper or PICO hand-pose input |
111104
| Optional active vision | OpenNeck yaw/pitch in physical degrees |
112105
| Host-policy observation | JPEG RGB + `observation.state(68)` |
113-
| Host-policy action | `float32[T,50]` canonical reference at 30 Hz |
106+
| Host-policy action | `float32[T,50]`, 30 Hz source horizon, `T` in `[1,50]` |
114107
| Host-policy body control | 36D root/joint reference through the existing 50 Hz motion tracker |
115108

116109
## Constraints
117110

118-
- `controller.policy_path` must be explicitly provided and the file must exist
119-
- Offline BVH runs require explicit `input.bvh_file`
120-
- `viewers` is the sole viewer configuration entry
121-
- Observation/ONNX dimension mismatch causes immediate startup error
122-
- sim2real also requires a dual-input ONNX whose observation dimension matches the runtime builder
123-
- Host-policy message-envelope or schema mismatches are rejected while the robot remains in `STANDING`
124-
- Host action chunks are validated and interpolated onboard; the host cannot bypass the motion tracker or send motor commands
125-
- Policy entry remains internal to `STANDING` only while one host session waits for its first valid chunk; that chunk enters `POLICY` directly, with no candidate alignment, entry Kp ramp, or second session/reset, and the 50 Hz limiter starts from the measured robot reference captured at session start
126-
- Temporal root, yaw, and joint-reference discontinuities are accepted at chunk boundaries and inside chunks, then rate-limited at the 50 Hz scheduler output so recorded pause/resume transitions remain usable
127-
128-
## Public Surface
129-
130-
**Stable run modes:** offline sim2sim, offline sim2real playback, Pico4 sim2sim,
131-
G1 sim2real, independent host-policy G1 sim2real
132-
133-
**Stable training entry points:** `train.py`, `play.py`, `benchmark.py`, `save_onnx.py`
134-
135-
**Stable data entry points:** `build_dataset.py`, `precompute_dataset.py`
111+
- `controller.policy_path` must be explicit and point to an existing file.
112+
- Offline BVH runs require an explicit, existing `input.bvh_file`.
113+
- `viewers` is the only viewer configuration key.
114+
- Observation definitions and ONNX signatures must match exactly; startup fails
115+
instead of padding or trimming data.
116+
- `default_dof_pos` must come from the selected robot's default standing angles.
117+
- Sim2real requires the same dual-input observation contract used in simulation.
118+
- Host message-envelope or schema mismatches are rejected while the robot
119+
remains in `STANDING`. Shape, finiteness, session, sequence, quaternion,
120+
staleness, and safety violations reject the whole action chunk.
121+
- Host actions are validated, scheduled, and rate-limited onboard. The host
122+
cannot bypass the motion tracker or send G1 motor commands.
123+
- Policy entry remains an internal `STANDING` flow while one host session waits
124+
for its first valid chunk. That chunk enters `POLICY` directly, with no
125+
candidate alignment, entry Kp ramp, or second session/reset. The 50 Hz limiter
126+
starts from the measured robot reference captured at session start.
127+
- Temporal root, yaw, and joint-reference discontinuities are accepted at chunk
128+
boundaries and inside chunks, then rate-limited at the 50 Hz scheduler output
129+
so recorded pause/resume transitions remain usable.
130+
- PICO input, RealSense preview, recording, hand, and neck failures are
131+
non-critical; the Unitree remote and robot-control loop remain available.
132+
133+
## Public Entry Points
134+
135+
Supported run modes are offline sim2sim, offline sim2real playback, PICO
136+
sim2sim, PICO G1 sim2real, and independent host-policy G1 sim2real.
137+
138+
Runtime commands:
139+
140+
- `scripts/run/run_sim.py` — offline BVH and live PICO sim2sim
141+
- `scripts/run/run_sim2real.py` — BVH or PICO G1 sim2real
142+
- `scripts/run/run_high_level_policy_sim2real.py` — independent host-policy G1 deployment
143+
- `scripts/run/record_pico_motion.py` — record retargeted motion clips from PICO
144+
- `scripts/render/render_sim.py` — render mocap, retargeting, and sim2sim videos
145+
- `scripts/view/view_recording.py` — review synchronized sim2real recordings
146+
147+
Training and data commands:
148+
149+
- `train_mimic/scripts/train.py`, `play.py`, `benchmark.py`, `save_onnx.py`
150+
- `train_mimic/scripts/data/build_dataset.py`
151+
- `train_mimic/scripts/data/precompute_dataset.py`
152+
153+
Public Python surfaces:
154+
155+
- Protocols in `teleopit/interfaces.py`
156+
- `TeleopPipeline`
157+
- `VelCmdObservationBuilder`
158+
- `RLPolicyController`

0 commit comments

Comments
 (0)