Gemini 435Le (PoE): supported way to synchronize RGB↔Depth *capture* within a single camera? (color/depth appear to free-run on independent clocks, ~150 ppm relative drift)

[iamanshulchauhan1]

Summary

For RGB-D fusion on dynamic scenes I need tight (single-digit-ms) RGB↔Depth capture-time alignment within a single camera. With pipeline.enable_frame_sync() enabled, the per-frameset color_ts − depth_ts gap is large, drifts over time, and starts at a random offset each run. Empirically the RGB and depth sensors appear to free-run on independent clocks.

I'd like to know the supported way to lock RGB to Depth within one 435Le — and specifically what these OBPropertyID entries are for and how to use them:

• OB_PROP_INTRA_CAMERA_SYNC_REFERENCE_INT (236)
• OB_STRUCT_COLOR_SYNCED_EXPOSURE_PARAM (1077)

Related: #150 (same question for Astra2).

Environment

• Device: Orbbec Gemini 435Le, PID 2069 (reproduced on more than one unit)
• Firmware: 1.3.19 (≥ recommended 1.3.6 per README, so not a stale-firmware bug)
• pyorbbecsdk: __version__ = 2.1.1, core get_version = 2.8.6
• OS: Ubuntu 22.04 (x86_64), Python 3.10 — Linux, not Windows, so Windows metadata registration is N/A; the SENSOR_TIMESTAMP frame metadata is present and reads correctly.
• Streams: Color 1280×800@10 (MJPG/RGB) + Depth 1280×800@10 (Y16), pipeline.enable_frame_sync() ON

What I observe

I logged the folded color_ts − depth_ts gap within each frameset over 60 s, computed in every timestamp domain the SDK exposes — device (get_timestamp_us), hardware SENSOR_TIMESTAMP metadata, get_global_timestamp_us, get_system_timestamp_us.

1. The gap drifts ~9 ms/min, including in the raw hardware SENSOR_TIMESTAMP domain (i.e. not a host-clock artifact). 60 s summary:

device  mean +26.8 ms   drift −9.6 ms
sensor  mean +24.6 ms   drift −9.6 ms   <-- hardware exposure clock
global  mean +26.9 ms   drift −9.4 ms

2. Random per-startup offset. Across restarts the within-frameset gap is a different near-constant each boot (e.g. +0.5 ms one session, −37 ms another). If RGB and Depth shared a trigger, this would be a fixed small constant every boot.

3. Drift persists with enable_multi_device_sync ON and OFF (and with/without timer_sync_with_host) — so periodic host time-discipline is not the cause.

4. Multi-device sync mode doesn't help within a camera. Setting SOFTWARE_SYNCED (256) does not change the within-camera RGB↔Depth relationship.

5. Delays can't fix it from software:

• Equal color_delay_us / depth_delay_us shift the whole frameset, leaving the within-camera RGB↔Depth gap unchanged.
• An unequal delay shifts the gap but cannot track the ~9 ms/min drift.
• set_multi_device_sync_config is rejected while streaming (errorCode 3, propertyId 1038), so live retuning isn't possible regardless.

Taken together (random per-startup phase + ~150 ppm relative drift in the hardware SENSOR_TIMESTAMP domain), this looks like RGB and Depth capturing on independent, free-running clocks rather than a shared trigger.

Questions

1. On the 435Le, are the RGB and Depth(IR) sensors hardware-co-triggered, or do they free-run on independent oscillators?
2. What is the supported way to achieve few-ms RGB↔Depth capture alignment within one 435Le? In particular:
   • OB_PROP_INTRA_CAMERA_SYNC_REFERENCE_INT (236) — what does it do, what are valid values/units, when must it be set (before start? per sensor?), and does it co-trigger RGB+Depth?
   • OB_STRUCT_COLOR_SYNCED_EXPOSURE_PARAM (1077) — does it tie color exposure to the depth frame?
   • Do HARDWARE_TRIGGERING (64) / SOFTWARE_TRIGGERING (32) latch both color and depth on a single trigger (making the within-camera gap deterministic)? What are the per-stream semantics of trigger_to_image_delay_us, color_delay_us, depth_delay_us?
3. Is set_multi_device_sync_config only writable with the pipeline stopped (I see errorCode 3, propertyId 1038 when live)? Is there a live path?
4. Is the ~150 ppm RGB-vs-Depth timestamp drift expected / within spec?

Repro

Enable color + depth, pipeline.enable_frame_sync(), stream 60 s, and per frameset log:
color.get_timestamp_us() − depth.get_timestamp_us() and the SENSOR_TIMESTAMP metadata for both frames. The gap drifts ~9 ms/min and starts at a random offset each run.

[gongye2]

Hi, @iamanshulchauhan1
Thanks for the detailed report. Here are the findings from our code-level investigation:

1. Are RGB and Depth(IR) hardware-co-triggered?

They free-run on independent oscillators. The Color and Depth(IR) sensors each have their own clock source within the 435Le. There is no shared hardware PLL or trigger connecting them internally. The observed ~150 ppm relative drift is the natural consequence of two independent crystal oscillators.

2. OB_PROP_INTRA_CAMERA_SYNC_REFERENCE_INT (236)

Gemini 435Le does not support this property.

3. OB_STRUCT_COLOR_SYNCED_EXPOSURE_PARAM (1077)

Gemini 435Le does not support this property.

4. HARDWARE_TRIGGERING (64) / SOFTWARE_TRIGGERING (32)

These are multi-device external synchronization modes. They control how the 435Le interacts with other cameras via the VSYNC_IN/VSYNC_OUT pins on the sync port. They do not alter the internal Color↔Depth clock relationship within a single camera.

When an external trigger signal arrives (or the host calls trigger_capture() in software trigger mode), both Color and Depth sensors receive the trigger simultaneously. However, because they run on independent oscillators, their actual exposure timing will drift apart as frames accumulate — the trigger provides a shared starting point but cannot compensate for the ongoing ppm-level clock divergence.

Per-stream delay semantics:

         TRIGGER signal
              |
    |<-- depthDelayUs -->|  Depth sensor starts exposure
    |<-- colorDelayUs ------>|  Color sensor starts exposure

Parameter	Meaning
colorDelayUs	Delay from trigger signal arrival to Color sensor exposure start (µs)
depthDelayUs	Delay from trigger signal arrival to Depth sensor exposure start (µs)

These two parameters are independent on the 435Le. They control the per-trigger phase offset between the two sensors' exposure start times. trigger2ImageDelayUs is a shared delay used on devices where Color and Depth cannot be independently delayed — not the active control on 435Le.

Critical limitation: Setting unequal delays can shift the mean gap between Color and Depth frames, but cannot track or eliminate drift over time. The drift originates from the independent oscillators' ppm mismatch.

5. Is set_multi_device_sync_config writable while streaming?

No. This configuration is a pre-stream hardware register write — there is no live path. The pipeline must be stopped, the config written, and restarted.

6. Is the ~150 ppm drift within spec?

Yes. The Color and Depth sensors each have independent crystal oscillators. A relative drift of ~150 ppm falls within normal manufacturing variation and is expected behavior on the 435Le.