SchemaTrackerBase: XRT_ERROR_INVALID_TENSOR_ID from xrGetTensorDataNV aborts the teleop session
メンテナーはふだん 1 日以内に返信
まだ誰も着手していません。
評価
調査の方向性
src/core/live_trackers/cpp/schema_tracker_base.cpp の SchemaTrackerBase::read_next_sample() から始め、次に tensor 拡張のエラー処理と DeviceIOSession.update() の呼び出し経路を確認してください。利用可能であれば、文書化されている CloudXR、MANUS、IsaacTeleop のセットアップで再現してください。一時的に無効な tensor 参照を、テレオペレーションを終了せずに更新または復旧でき、永続的なランタイム障害は引き続き致命的なままであることが完了条件です。
索引モデルが issue の本文から書いたものです。
説明
Summary
During a live IsaacTeleop session with MANUS sensor collections, CloudXR’s tensor service reported XRT_ERROR_INVALID_TENSOR_ID. The OpenXR API returned XR_ERROR_RUNTIME_FAILURE, which SchemaTrackerBase treated as fatal, terminating the teleoperation session.
The immediate failure is reproducible only intermittently. The stale tensor may result from a producer disappearing or recreating its collection, but that root cause is not yet confirmed.
Environment
- IsaacTeleop version:
1.6.x - IsaacTeleop commit:
5b5bceb7257733b6a4a7880ac5e02d3319a0de28 - CloudXR Runtime:
6.3.0 - Integration: GR00T
xr_server - Additional device: MANUS gloves
- GR00T release: Dexter v1.2.1
Symptom
The MANUS plugin connected and registered both sensor collections:
[manus-plugin] Successfully connected to Manus host after 1 attempts
[manus-plugin] Initialized with wrist source: HandTracking
[manus-plugin] left sensors=on
[manus-plugin] right sensors=on
Shortly afterward, the next DeviceIO update failed:
ERROR [get_tensor_data] ipc_receive: Service side get_tensor_data failed:
XRT_ERROR_INVALID_TENSOR_ID
[/builds/cloudxr/cloudxr-openxr-runtime/deps/monado/src/xrt/ipc/client/ipc_client_tensor_overseer.c:111]
XR_ERROR_RUNTIME_FAILURE in xrGetTensorDataNV: Failed to get tensor data
The exception propagated through:
TeleopSession.step()
-> TeleopSession._execute_step_request()
-> DeviceIOSession.update()
-> SchemaTrackerBase::read_next_sample()
-> xrGetTensorDataNV()
xr_server then exited. The available log does not show that the OpenXR runtime process itself crashed.
Reproduction
- Start CloudXR Runtime 6.3.0.
- Start an IsaacTeleop session configured with the MANUS plugin and the
human,sensors,hapticdatasets. - Connect the headset and wait for
manus_sensors_leftandmanus_sensors_rightto begin streaming. - Continue teleoperation.
- Intermittently,
DeviceIOSession.update()fails withXRT_ERROR_INVALID_TENSOR_ID.
The problem has occurred alongside MANUS USB/device instability. Restarting the stack or moving the MANUS dongle and license key to different USB ports can temporarily clear it, but this is not reliable.
Expected behavior
If a tensor producer or collection temporarily disappears, the tracker should invalidate its cached collection reference and rediscover it. A transient tensor-list change should not terminate the entire teleoperation session.
Code analysis
SchemaTrackerBase::read_next_sample() handles XR_ERROR_TENSOR_LOST_NV as recoverable, but throws for every other XrResult.
The tensor extension describes XR_ERROR_TENSOR_LOST_NV as the result expected when a cached tensor-list element is no longer valid. In this case, however, the runtime’s internal XRT_ERROR_INVALID_TENSOR_ID was surfaced as the generic XR_ERROR_RUNTIME_FAILURE.
Possible explanations:
- A MANUS producer exited or recreated its tensor collection between generation checking and data retrieval.
- The CloudXR runtime retained or resolved a stale internal tensor ID.
- A producer failure occurred between IsaacTeleop’s periodic plugin-health checks, so the tensor error surfaced first.
Mitigations tested
- Restarting the complete teleoperation stack: temporarily recovers.
- Moving the MANUS USB devices to other ports: sometimes recovers, but the issue returns.
- Upgrading the GR00T deployment to Dexter v1.2.1: issue still observed.
Questions
- Why is
XRT_ERROR_INVALID_TENSOR_IDmapped toXR_ERROR_RUNTIME_FAILURErather thanXR_ERROR_TENSOR_LOST_NV? - Should
SchemaTrackerBaseinvalidate and refresh its tensor list after this failure? - Would one forced refresh and retry allow recovery while preserving fatal handling for persistent runtime failures?
- Can the runtime log the missing tensor ID, current generation, and owning producer when this occurs?
Additional context
The same investigation also observed separate MANUS plugin processes terminating with SIGABRT and SIGSEGV. Those failures may explain a disappearing tensor producer, but they are being tracked separately and should not be assumed to share this root cause.
Full station logs are available. Sensitive MANUS license information has been omitted.
- 主要言語
- Python
- スター
- 412
- フォーク
- 97
- 平均マージ
- 2日 3時間
- マージ済み PR(30日)
- 141
環境構築
- Dockerfile・Docker Compose ファイルなし
- プルリクエストのテンプレートあり
- コントリビューションガイドを読む
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
NVIDIA/IsaacCapture のほかの issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
NVIDIA/IsaacCapture#910 ·
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 88/100
NVIDIA/IsaacCapture#891 ·
メンテナーはふだん 1 日以内に返信
-
難易度 5/5 1週間以上 初心者へのやさしさ 35/100
NVIDIA/IsaacCapture#1319 ·
メンテナーはふだん 1 日以内に返信
-
Add OneRobotics A1 asset and Isaac Teleop example対応中かも @WangXuechun-woan が 6 日前に担当しました。 オープン
難易度 5/5 1週間以上 初心者へのやさしさ 35/100
NVIDIA/IsaacCapture#1291 · コメント 2 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 4/5 3〜5日 初心者へのやさしさ 25/100
NVIDIA/IsaacCapture#1217 ·
メンテナーはふだん 1 日以内に返信
NVIDIA/IsaacCapture の issue をすべて見る
似ている issue
-
README.zh-CN/ja/es still hard-code 501 test counts after README.md went count-free対応中かも @Aaqibhafeezkhan が今日担当しました。 オープンbeginner documentation good first issue
難易度 2/5 1〜3時間 初心者へのやさしさ 75/100
BistaDinesh03/critique#164 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
bug IO
難易度 2/5 1〜3時間 初心者へのやさしさ 65/100
MrNeRF/LichtFeld-Studio#3824 ·
メンテナーはふだん 1 日以内に返信
-
bug confirmed issue
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
open-webui/open-webui#33200 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
難易度 1/5 1時間未満 初心者へのやさしさ 72/100
-
The changelog workflow excludes otelbot but not dependabot, so every Dependabot pull request fails the changelog check対応中かも @ocelotl が今日担当しました。 オープン
難易度 1/5 1時間未満 初心者へのやさしさ 82/100
open-telemetry/opentelemetry-python#5825 ·
メンテナーはふだん 2 日以内に返信