HandTracker: sample timestamps of hands injected via push devices are lost; MCAP `sample_time` is the query time

オープン
  • 何か問題がありますか？
#1,382 コメント 1 件 リアクション 0 件 担当者 1 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@nv-jakob がすでに取り組んでいます。

2026年10月28日 から。

評価

この issue はまだ評価されていません。

説明

Summary

We are developing an IsaacCapture plugin for a data glove (RevoHuman) and plan to open a PR later. Before that, we would like to confirm the timestamp convention for hand data.

When the plugin pushes hand data, it fills XrPushDeviceHandTrackingDataNV.timestamp with the glove's sample time. In our tests, CloudXR Runtime 6.3.0 does not use this timestamp when selecting a frame, and LiveHandTrackerImpl then records the query time as sample_time. As a result, the hand pose sample_time in the MCAP is later than the actual sample time, and the original sample time cannot be recovered from the recording. Tactile data from the same glove frame goes through the tensor path, where sample_time is recorded correctly, so the two end up about 10–15 ms apart in the same MCAP.

Background: the RevoHuman plugin

The plugin is structured like the Manus plugin, with one pipeline per hand:

  • Input: the vendor SDK delivers joint angles and tactile data for each hand at about 200 Hz, with the glove sample time (CLOCK_MONOTONIC) on every frame.
  • Hand joints: FK produces the 26 OpenXR hand joints; the wrist pose is provided externally. Joints are injected with HandInjector::push(joints, t), where t is the glove sample time converted to XrTime. In steady state, t is about 9.5 ms before the push.
  • Tactile: pushed as a tensor via SchemaPusher, with sample_time also set to the glove sample time.
  • Stall handling: if the glove stream stalls for more than 50 ms, the plugin destroys the injector and the reader goes inactive.

Current behavior

  1. The timestamp field of the push struct has no documented semantics (XR_NVX1_device_interface.h#L106-L116). Existing hand plugins all pass the push time, e.g. Manus (#L1285) and Sharpa (#L702).
  2. Runtime 6.3.0 returns only the most recently arrived frame and ignores the pushed timestamp (measurements below).
  3. LiveHandTrackerImpl records DeviceDataTimestamp(last_update_time_, last_update_time_, xr_time), i.e. available = sample = the query time of the current update (live_hand_tracker_impl.cpp#L188). This matches the convention in live_trackers/AGENTS.md#L16-L19 and appears to be intentional, following the OpenXR query/prediction model.
  4. However, timestamp.fbs defines sample_time as the time "when the sample was taken", which "enables synchronization across different data sources". The tensor path (schema_tracker_base.cpp#L287-L295) satisfies this definition; the push-device hand path does not.

Impact: for hands coming from a push device, MCAP sample_time − actual sample time = (sample-to-push latency) + (phase offset between push and query). For RevoHuman at 200 Hz, this is about 9.5 ms + 0–5 ms. The second term grows as the push rate drops (p99 of 98 ms at 10 Hz).

Reproduction

No glove is needed; a minimal OpenXR program is enough (C++, depends only on the OpenXR loader; we can share it):

  1. Start CloudXR Runtime 6.3.0 with NV_CXR_ENABLE_PUSH_DEVICES=true.
  2. Process A: xrCreatePushDeviceNV (right hand), then call xrPushDevicePushHandTrackingNV in a loop with timestamp = now + Δ, encoding the frame index in the joint positions.
  3. Process B: call xrLocateHandJointsEXT the same way IC does, with various time values, and decode which frame was returned from the joint positions.

Results (3 same-process runs and 7 cross-process runs, all consistent):

Action Result
Push timestamp Δ ∈ {−100, −20, 0, +20, +100} ms Same frame returned; the pushed timestamp is only echoed back
Query time swept from −1 s to +5 s Always the most recently arrived frame: no interpolation, no extrapolation (velocityFlags always 0), no history
Out-of-order push (the later frame carries an older timestamp) The later-arriving frame overwrites the earlier one
Process alive but stops pushing for 30 s Stays isActive=true, frozen at the last frame
Push jointCount=0 / destroy the push device / process exits Inactive within one read cycle (<10 ms)

Expected behavior (for discussion)

Any one of the following would work for us:

  • A: For push-device hands, HandTracker records the pushed timestamp as sample_time. This requires the runtime to expose that timestamp to the reader.
  • B: If the hand path intentionally keeps the query time, recommend an official side channel, e.g. the plugin also pushes a tensor carrying sample_time (the existing soma_hand_joint_poses? trackers.toml#L56-L61), and describe how it maps to HandPose.
  • C: At minimum, document three points: timestamp is not used for frame selection; HandPose sample_time is the query time; producers are responsible for stall detection.

Questions

  1. Should XrPushDeviceHandTrackingDataNV.timestamp carry the sample time or the push time?
  2. Is the 6.3.0 behavior above (latest arrival wins, timestamp ignored, no expiry) by design, or a property of the current implementation? Will it change in later versions?
  3. Are there plans for HandPose to record the actual sample time? If not, which option do you recommend?
  4. While a process is alive but has stopped pushing, the hand stays frozen and is reported as active. Is there a recommended threshold for how long a producer should wait after a stall before destroying the injector? We use 50 ms; existing plugins range from 200 ms to 3 s, e.g. Wuji destroys the injector after 200 ms (#L554-L560), while Sharpa treats the glove as disconnected only after 3 s without data (#L497-L500). (Related: #1319)

Environment

  • IsaacCapture: code references at main a74babb0; tests run on 9aee7654 (hand-related files are identical in both)
  • CloudXR Runtime: 6.3.0
  • OS: Ubuntu 22.04 (x86_64)
  • No headset: XR_MND_headless + XR_EXTX_overlay, STAGE space
主要言語
Python
スター
425
フォーク
101
平均マージ
1日 14時間
マージ済み PR（30日）
165

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/IsaacCapture のほかの issue

NVIDIA/IsaacCapture の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。