Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

HandTracker: sample timestamps of hands injected via push devices are lost; MCAP `sample_time` is the query time

Đang mở
  • Có gì không ổn?
#1,382 1 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@nv-jakob đang làm issue này rồi.

Từ ngày 28/10/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

Summary

We are developing an IsaacCapture plugin for a data glove (RevoHuman) and plan to open a PR later. Before that, we would like to confirm the timestamp convention for hand data.

When the plugin pushes hand data, it fills XrPushDeviceHandTrackingDataNV.timestamp with the glove's sample time. In our tests, CloudXR Runtime 6.3.0 does not use this timestamp when selecting a frame, and LiveHandTrackerImpl then records the query time as sample_time. As a result, the hand pose sample_time in the MCAP is later than the actual sample time, and the original sample time cannot be recovered from the recording. Tactile data from the same glove frame goes through the tensor path, where sample_time is recorded correctly, so the two end up about 10–15 ms apart in the same MCAP.

Background: the RevoHuman plugin

The plugin is structured like the Manus plugin, with one pipeline per hand:

  • Input: the vendor SDK delivers joint angles and tactile data for each hand at about 200 Hz, with the glove sample time (CLOCK_MONOTONIC) on every frame.
  • Hand joints: FK produces the 26 OpenXR hand joints; the wrist pose is provided externally. Joints are injected with HandInjector::push(joints, t), where t is the glove sample time converted to XrTime. In steady state, t is about 9.5 ms before the push.
  • Tactile: pushed as a tensor via SchemaPusher, with sample_time also set to the glove sample time.
  • Stall handling: if the glove stream stalls for more than 50 ms, the plugin destroys the injector and the reader goes inactive.

Current behavior

  1. The timestamp field of the push struct has no documented semantics (XR_NVX1_device_interface.h#L106-L116). Existing hand plugins all pass the push time, e.g. Manus (#L1285) and Sharpa (#L702).
  2. Runtime 6.3.0 returns only the most recently arrived frame and ignores the pushed timestamp (measurements below).
  3. LiveHandTrackerImpl records DeviceDataTimestamp(last_update_time_, last_update_time_, xr_time), i.e. available = sample = the query time of the current update (live_hand_tracker_impl.cpp#L188). This matches the convention in live_trackers/AGENTS.md#L16-L19 and appears to be intentional, following the OpenXR query/prediction model.
  4. However, timestamp.fbs defines sample_time as the time "when the sample was taken", which "enables synchronization across different data sources". The tensor path (schema_tracker_base.cpp#L287-L295) satisfies this definition; the push-device hand path does not.

Impact: for hands coming from a push device, MCAP sample_time − actual sample time = (sample-to-push latency) + (phase offset between push and query). For RevoHuman at 200 Hz, this is about 9.5 ms + 0–5 ms. The second term grows as the push rate drops (p99 of 98 ms at 10 Hz).

Reproduction

No glove is needed; a minimal OpenXR program is enough (C++, depends only on the OpenXR loader; we can share it):

  1. Start CloudXR Runtime 6.3.0 with NV_CXR_ENABLE_PUSH_DEVICES=true.
  2. Process A: xrCreatePushDeviceNV (right hand), then call xrPushDevicePushHandTrackingNV in a loop with timestamp = now + Δ, encoding the frame index in the joint positions.
  3. Process B: call xrLocateHandJointsEXT the same way IC does, with various time values, and decode which frame was returned from the joint positions.

Results (3 same-process runs and 7 cross-process runs, all consistent):

Action Result
Push timestamp Δ ∈ {−100, −20, 0, +20, +100} ms Same frame returned; the pushed timestamp is only echoed back
Query time swept from −1 s to +5 s Always the most recently arrived frame: no interpolation, no extrapolation (velocityFlags always 0), no history
Out-of-order push (the later frame carries an older timestamp) The later-arriving frame overwrites the earlier one
Process alive but stops pushing for 30 s Stays isActive=true, frozen at the last frame
Push jointCount=0 / destroy the push device / process exits Inactive within one read cycle (<10 ms)

Expected behavior (for discussion)

Any one of the following would work for us:

  • A: For push-device hands, HandTracker records the pushed timestamp as sample_time. This requires the runtime to expose that timestamp to the reader.
  • B: If the hand path intentionally keeps the query time, recommend an official side channel, e.g. the plugin also pushes a tensor carrying sample_time (the existing soma_hand_joint_poses? trackers.toml#L56-L61), and describe how it maps to HandPose.
  • C: At minimum, document three points: timestamp is not used for frame selection; HandPose sample_time is the query time; producers are responsible for stall detection.

Questions

  1. Should XrPushDeviceHandTrackingDataNV.timestamp carry the sample time or the push time?
  2. Is the 6.3.0 behavior above (latest arrival wins, timestamp ignored, no expiry) by design, or a property of the current implementation? Will it change in later versions?
  3. Are there plans for HandPose to record the actual sample time? If not, which option do you recommend?
  4. While a process is alive but has stopped pushing, the hand stays frozen and is reported as active. Is there a recommended threshold for how long a producer should wait after a stall before destroying the injector? We use 50 ms; existing plugins range from 200 ms to 3 s, e.g. Wuji destroys the injector after 200 ms (#L554-L560), while Sharpa treats the glove as disconnected only after 3 s without data (#L497-L500). (Related: #1319)

Environment

  • IsaacCapture: code references at main a74babb0; tests run on 9aee7654 (hand-related files are identical in both)
  • CloudXR Runtime: 6.3.0
  • OS: Ubuntu 22.04 (x86_64)
  • No headset: XR_MND_headless + XR_EXTX_overlay, STAGE space
Ngôn ngữ chính
Python
Star
425
Fork
101
Merge trung bình
1 ngày 14 giờ
Pull request đã merge (30 ngày)
165

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của NVIDIA/IsaacCapture

Tất cả issue của NVIDIA/IsaacCapture

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.