[Bug] SO-101 can make discontinuous joint jumps during Grip motion and stationary re-clutch

エントリーポイント examples.isaac_teleop_to_so101.teleoperate から始め、提示されたコマンドとプラットフォームの詳細を使って2つのクラッチシナリオを再現します。Isaac Teleop の動作を下流の修正 f611bba と比較し、engage-frame hold または joint-state rebase を統合境界に置くべきか判断します。完了条件は、関節間の動作が連続しており、静止したコントローラーを再度有効にしたときに動作しないことです。

Describe the bug

When teleoperating an SO-101 with a Meta Quest 3S Touch Plus controller through CloudXR and Isaac Teleop, I observed sudden discontinuous robot motion in two situations:

1. A joint jump while continuously holding Grip/Squeeze and moving the controller.
2. A downward jump after moving the robot to a pose, releasing Grip, keeping the controller stationary, and pressing Grip again to re-engage the clutch.

The Quest controller pose and button input otherwise reached the Ubuntu host correctly. Grip/Squeeze acted as the clutch and Trigger controlled the gripper.

This was reproduced through the LeRobot SO-101 integration. I am filing it here because I would like guidance on the intended Isaac Teleop clutch/re-engagement behavior for underactuated arms and whether joint-continuity handling should be exposed or documented at the Isaac Teleop integration boundary.

To reproduce

1. Start CloudXR Runtime 6.2.0 on Ubuntu and connect a Quest 3S through the official CloudXR web client.
2. Connect an SO-101 follower and run the LeRobot Isaac Teleop example:

python -m examples.isaac_teleop_to_so101.teleoperate \
  --robot.type=so101_follower \
  --robot.port=/dev/serial/by-id/<SO101_SERIAL_DEVICE> \
  --robot.id=so101_follower_main \
  --teleop.type=xr_controller \
  --reset_to_origin=false

3. Reproduction A:

   • Hold Grip continuously.
   • Move the right Quest controller through the workspace.
   • Observe that the arm can suddenly switch to a discontinuous elbow/wrist configuration.

4. Reproduction B:

   • Hold Grip and move the arm to a reachable pose.
   • Release Grip and confirm that the arm holds.
   • Keep the controller stationary.
   • Press Grip again.
   • Observe an immediate downward jump on re-engagement.

Expected behavior

• Motion should remain joint-continuous while Grip is held.
• Releasing Grip should hold the robot.
• Re-engaging Grip with a stationary controller should produce no robot movement on the engage frame.
• The following IK frames should continue from the measured/current joint configuration rather than switching to another local solution.

Observed behavior

In a quantified re-clutch run before the downstream fix, three physical Grip re-engagements produced first-frame maximum joint-target changes of:

• 6.422 degrees with 14.006 mm end-effector position change
• 3.116 degrees with 6.815 mm end-effector position change
• 3.186 degrees with 7.151 mm end-effector position change

The reported orientation change was 0 degrees in all three cases. The largest changes were mainly in elbow_flex, with wrist_flex also visibly involved in another event.

Videos

• Video 1 - jump while Grip is continuously held and the controller is moving:

  https://github.com/user-attachments/assets/f249e4d1-d16d-401f-91fb-02b7f3e34ddf

• Video 2 - downward jump when re-engaging Grip while the controller is stationary:

  https://github.com/user-attachments/assets/2e6d228d-d09b-4497-ae2c-a6e52b0d9927

Platforms

• Isaac Teleop Version: 1.3.131
• CloudXR Runtime: 6.2.0
• OS: Ubuntu 24.04 LTS, kernel 6.8.0-134-generic, x86_64
• GPU: NVIDIA GeForce RTX 4080, driver 595.71.05
• Integration: LeRobot v0.6.0, examples/isaac_teleop_to_so101
• XR Device: Meta Quest 3S with Touch Plus controllers
• Robot: SO-101 follower, 5-DoF arm plus gripper
• Control target rate: 30 Hz

Investigation and downstream workaround

The investigation in the LeRobot integration found two relevant behaviors:

• A calibrated/measured joint value can be slightly outside the URDF nominal limit (for example, shoulder_lift=-103 degrees while the URDF lower limit is approximately -100 degrees). A same-pose q -> FK(q) -> IK(FK(q)) operation can therefore project the state into the solver's feasible joint space instead of returning the measured joint configuration.
• On this 5-DoF arm, the same-pose projection may require multiple solves to settle, and a small first controller displacement near an underconstrained or near-singular pose can select a more distant local joint solution.

A downstream fix was implemented in commit f611bba, currently included in the independent draft LeRobot PR #4046.

The workaround keeps the measured arm joints exactly unchanged on the engage frame, pre-converges the same-pose IK state, adds native dq regularization, and rebases subsequent solver increments onto the measured joint configuration. The gripper remains an absolute 0-100 command.

After the fix, 47 physical re-engagements on the SO-101 produced a 0.000000-degree engage-frame joint jump, and the original elbow/wrist jump did not recur during small and larger XYZ motions.

Questions

1. Is there an intended Isaac Teleop pattern for guaranteeing joint-space continuity at clutch re-engagement on underactuated arms?
2. Should the clutch/integration API provide an engage-frame hold or joint-state rebase hook before Cartesian IK resumes?
3. Would NVIDIA prefer this behavior to remain entirely integration-specific, or should the Isaac Teleop documentation/example cover it?

Additional context

No hand tracking was used. The test used only the Quest 3S right Touch Plus controller. No cameras, data recording, or leader arm were involved.