OpenVINO: LeRobot policies that do not run yet, and what blocks them

Offen
  • Stimmt etwas nicht?
#24,325 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Anfängerfreundlichkeit
15/100
Issue-Typ
Bug
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
python, pytorch
Bereich
backend, machine-learning

Rechercherichtung

This is a tracking issue with several independent blockers, so start by picking one item rather than the whole list. For item 1, read TorchFXPythonDecoder.get_shape_for_value and compare it with get_found_shape/get_found_dtype; reproduce with a small Qwen3-VL DeepStack model or groot/vla_jepa. For item 2, start at modeling_pi0.py around line 633 and modeling_pi05.py around line 906. Done means the policy exports and runs through OpenVINO with results matching eager PyTorch.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

module: openvino

This issue tracks LeRobot policies exported to ExecuTorch with OpenVINO: which ones run on an Intel Xeon CPU, how fast, and what each project still needs to look at. The TensorRT column is copied from #24353 as a reference. The other CPU backends are tracked in #24667 (XNNPACK), #24668 (Qualcomm) and #24669 (Core ML and MLX), the CUDA backend alone in #24684, Vulkan in #24688, and Arm in #24697.

Why this matters for LeRobot: a LeRobot arm plugs into an ordinary computer over USB, and that computer can be an Intel laptop or mini PC with no NVIDIA GPU. OpenVINO can use the Intel CPU, the integrated GPU and, on Core Ultra chips, the NPU (neural processing unit) of those machines. ExecuTorch's OpenVINO backend lists all 3 as targets. This issue measures the Xeon CPU only so far.

Coverage

Policy OpenVINO, Xeon TensorRT, H100 (from #24353) Programs
act $\color{#2da44e}{\checkmark}$ 42.9 (2.2x) $\color{#2da44e}{\checkmark}$ 3.15 (2.5x) OpenVINO 1
diffusion $\color{#2da44e}{\checkmark}$ 2558 (6.8x) $\color{#2da44e}{\checkmark}$ 535 (1.6x) OpenVINO 3
dm05 ❌ $\color{#d4a017}{\checkmark}$ 81.0 (5.7x)
eo1 $\color{#2da44e}{\checkmark}$ 2868 (1.4x) $\color{#2da44e}{\checkmark}$ 58.3 (5.0x) OpenVINO 3 + CPU 3
evo1 ❌ $\color{#2da44e}{\checkmark}$ 28.7 (6.6x)
fastwam ❌ $\color{#2da44e}{\checkmark}$ 54.0 (8.1x)
fineart_vla $\color{#2da44e}{\checkmark}$ 6373 (1.2x) $\color{#2da44e}{\checkmark}$ 104 (2.3x) OpenVINO 1
flux3 ❌ ❌
gaussian_actor $\color{#2da44e}{\checkmark}$ 1.82 (2.0x) $\color{#2da44e}{\checkmark}$ 0.57 (3.7x) OpenVINO 4 + CPU 7
groot $\color{#2da44e}{\checkmark}$ 998 (2.3x) $\color{#2da44e}{\checkmark}$ 19.4 (6.8x) OpenVINO 11 + CPU 2
lawam $\color{#2da44e}{\checkmark}$ 1269 (2.3x) $\color{#d4a017}{\checkmark}$ 35.8 (5.1x) OpenVINO 2 + CPU 1
lingbot_va $\color{#2da44e}{\checkmark}$ 311633 (0.5x) $\color{#2da44e}{\checkmark}$ 2114 (1.4x) OpenVINO 18
lingbot_vla_v2 ⏳ $\color{#2da44e}{\checkmark}$ 76.0 (7.6x)
molmoact2 ❌ $\color{#d4a017}{\checkmark}$ 48.6 (4.4x)
multi_task_dit $\color{#2da44e}{\checkmark}$ 568 (3.1x) $\color{#2da44e}{\checkmark}$ 80.0 (3.8x) OpenVINO 3
pi0 ⏳ $\color{#2da44e}{\checkmark}$ 37.8 (4.7x)
pi05 ⏳ $\color{#2da44e}{\checkmark}$ 31.9 (6.1x)
pi0_fast $\color{#2da44e}{\checkmark}$ 51546 (1.0x) $\color{#2da44e}{\checkmark}$ 1592 (1.7x) OpenVINO 3
smolvla $\color{#2da44e}{\checkmark}$ 1140 (1.4x) $\color{#2da44e}{\checkmark}$ 17.3 (11.3x) OpenVINO 4 + CPU 55
tdmpc $\color{#2da44e}{\checkmark}$ 152 (2.4x) $\color{#2da44e}{\checkmark}$ 7.83 (4.5x) OpenVINO 1
vla_jepa $\color{#2da44e}{\checkmark}$ 967 (1.3x) $\color{#2da44e}{\checkmark}$ 17.2 (4.9x) OpenVINO 3 + CPU 3
vqbet $\color{#2da44e}{\checkmark}$ 10.3 (3.1x) $\color{#2da44e}{\checkmark}$ 2.36 (3.3x) OpenVINO 1
wall_oss_05 $\color{#2da44e}{\checkmark}$ 5622 (1.2x) $\color{#2da44e}{\checkmark}$ 68.0 (4.9x) OpenVINO 3 + CPU 7
wall_x ❌ $\color{#d4a017}{\checkmark}$ 93.4 (4.9x)
xvla $\color{#2da44e}{\checkmark}$ 2906 (1.4x) $\color{#2da44e}{\checkmark}$ 65.7 (1.6x) OpenVINO 1

Legend:

  • The number: milliseconds to produce one action chunk (batch size 1, median of 20 chunks) in lerobot-rollout --bundle.path, with PyTorch never loaded.
  • In brackets: how many times faster than eager PyTorch (lerobot-rollout --policy.path) on the same device, checkpoint and dtype.
  • $\color{#2da44e}{\checkmark}$ and a number: works with torch 2.15 and the ExecuTorch 1.6 build under "Built on".
  • $\color{#d4a017}{\checkmark}$ and a number (TensorRT column only): needs PyTorch fixes that are not in torch 2.15, see #24353.
  • ❌: measured, and it fails; the cause is in the tables below.
  • ⏳: being measured, or waiting for the LeRobot fix in the LeRobot table (pi0, pi05, lingbot_vla_v2).
  • Empty: not measured, or no like-for-like reference (see the notes).

"Programs": how many OpenVINO partitions (OpenVINO) the model is split into, and how many operators stay on ExecuTorch's CPU kernels (CPU).

Notes:

  • OpenVINO runs on a server CPU and TensorRT on a datacenter GPU. Each number in brackets compares a backend with eager PyTorch on its own device, so do not divide one column by the other.
  • The TensorRT column uses the same LIBERO checkpoints of evo1, lingbot_va and pi0_fast (lerobot/pi0fast-libero) as this issue, and flux3 uses black-forest-labs/flux-3-action-droid in both.
  • 3 rows compare different dtypes. xvla: the TensorRT number is its float32 run, like OpenVINO (#24353 lists the bf16 run, 16.5 ms). groot: float32 on TensorRT, bf16 here. pi0_fast: the checkpoint's bf16 setting on TensorRT, float32 here.
  • lawam's TensorRT number needs the PyTorch fixes listed in #24353 (yellow mark). Its OpenVINO number uses stock torch 2.15.
  • OpenVINO runs each policy in the same dtype as its eager run: mostly bf16 for eo1, groot, lawam, lingbot_va, smolvla, vla_jepa and wall_oss_05, float32 for the other 9 timed rows. Those 9 stay float32 because ExecuTorch's OpenVINO runtime asks for it (INFERENCE_PRECISION_HINT). With OpenVINO's own default on this Xeon, which has bf16 instructions, even act runs some layers in bf16.
  • Every timed bundle first passed LeRobot's check against eager PyTorch on the same weights: tolerance 0.075 for the bf16 checkpoints (evo1, groot, lawam, smolvla, vla_jepa), 2e-05 for float32.
  • dm05, wall_x, molmoact2: stock torch 2.15 cannot save or load these programs (PyTorch fixes in #24353). fastwam runs with an ExecuTorch fix that is merged into main after 1.6 (ExecuTorch table).
  • tdmpc's random checkpoint names its camera observation.image, so it is fed with --rename_map='{"observation.images.image": "observation.image"}'.

Built on

One build: the official ExecuTorch 1.6.0 rc3 CPU wheel from the PyTorch test index, with nothing added.

  • ExecuTorch: 1.6 rc3. These OpenVINO fixes are in it, merged into release/1.6:
    • #24222: crashes, bool-mask and bf16 inputs, stray assert nodes (pick: #24298)
    • #24294: 0-d tensors, float32 by default, data-dependent sizes, int64 cumsum and prod, integer cumsum over constants computed during lowering; molmoact2's partitions run (pick: #24492)
    • #24396: bool min, max, clamp and sign (gaussian_actor), tensor_meta filled before conversion, and a test runner that fails when a test fails instead of exiting with success (pick: #24493)
    • #24466: a copy of a partition input that the partition returns runs on the CPU (pick: #24478)
  • 1.6 rc4 (the final 1.6.0) changes nothing for OpenVINO. On 10 policy and backend pairs, rc3 and rc4 give the same checks. 7 pairs give the same program files, byte for byte, including all 4 OpenVINO pairs (act, diffusion, fineart_vla, eo1). The 3 Qualcomm pairs differ only in bytes that also differ between 2 exports with rc3 (build stamps and compiled Qualcomm code). This comparison ran on an earlier LeRobot version than the table.
  • OpenVINO 2026.4.1. torch 2.15.0.
  • LeRobot: the lerobot-export-stack branch of shoumikhin/lerobot (see #24353), as it was on Nov 3.
  • CPU: Intel Xeon Platinum 8339HC. Each row uses 16 CPUs (8 physical cores with both hyperthreads, one NUMA node) and 16 threads, for eager PyTorch and the bundle alike.

Known issues

"Fix" is an open pull request unless it says merged. "Needs a decision" is a design question for the maintainers. "Needs an owner" means nobody is fixing it yet. "Verified" means the policy was measured before and after the fix.

ExecuTorch
  • Review the 2 pull requests below. #24506 is merged into main but not into 1.6.
  • Pick up the rows marked "needs an owner".
Problem Policies Fix Status
The edge verifier refuses complex numbers (aten.mul.Tensor on complex128, from fastwam's rotary embedding). Only the CUDA backend rewrites them into real numbers (#24403). fastwam #24627 (merged into main, after 1.6) does the rewrite for every backend Verified: 1908 ms (2.3x), one delegate with nothing left on the CPU, check 3.91e-02 (limit 0.075)
Copies of constants stay in the partition, and inputs that a whole program returns are not copied. 1.6 has the first version of the copy rule without the guard, so a row that returns a copy of a channels-last, int64 or float64 constant may be wrong. none seen #24506 (merged into main, after 1.6) Not measured on LeRobot
Partition inputs and outputs ignore the dim order. The runtime gives OpenVINO only the sizes, so a channels-last tensor at a partition edge gets wrong values, with no error (repro D). Some ops hide it: x * 2 on a channels-last input cancels out. none so far: none of 99 OpenVINO programs (21 policies) has a non-default dim order at a partition edge needs an owner: tell OpenVINO the dim order (for example a Transpose at the edge) or reorder in the runtime. An OpenVINO maintainer offered a separate pull request. Repro
A failed run gives no reason. OpenvinoBackend::execute does not log OpenVINO's own message (ov_get_last_err_msg()), and some failures, such as setting an input or output tensor, return with no log at all. Before the 1.6 fixes, molmoact2 failed with only CALL_DELEGATE execute failed at instruction 0: 0x1. all needs an owner Measured
Importing the OpenVINO backend imports its quantizer, so nncf must be installed to export, even with nothing quantized. Running a program does not need it. all (export) needs an owner Measured

OpenVINO

These are in OpenVINO's PyTorch frontend or CPU plugin (2026.4.1 unless noted).

  • Triage the rows below. 4 are worked around in ExecuTorch 1.6, but the real fix is in OpenVINO.
Problem Policies Fix Status
An output that is just a model input (a Result fed by a Parameter) is not written when the caller sets output tensors after inputs. If that input feeds other nodes too, the plugin overwrites the caller's input buffer, so those results are wrong as well. The frontend makes this shape itself: it turns aten.clone and aten.alias_copy into nothing (repro A). eo1 (2.06, limit 0.2) needs an owner: write such an output in any order. An OpenVINO maintainer reproduced it and took it to the core team. Worked around in ExecuTorch 1.6: eo1 passes (0.063). Seen in 2025.2.0, 2025.4.1, 2026.2.1, 2026.4.1 and the 2026.5 nightly. Setting outputs first, or a same-type Convert, avoids it.
Boolean-mask indexing does not convert (Dynamic rank for aten::index input is not supported). Qwen3-VL's DeepStack code writes and then reads the same masked rows on several layers (repro E). TorchFXPythonDecoder.get_shape_for_value reads only node.meta["tensor_meta"], which is None after to_edge, and does not fall back to node.meta["val"] as get_found_shape does. vla_jepa, lawam needs an owner: fall back to meta["val"] (a small Qwen3-VL model then converts and matches PyTorch) Worked around in ExecuTorch 1.6: both pass
cumsum over a bool tensor keeps the narrow type, so the count wraps after 255 and prompts over 255 tokens get wrong position ids (torch.cumsum(pad_masks, dim=1) - 1). aten::cumsum over bool or integer inputs and aten::prod over integer inputs should follow PyTorch's type promotion. pi0, pi05, pi0_fast needs an owner Worked around in ExecuTorch 1.6 (int64 cast)
The CPU plugin computes int64 prod, cumsum and sum in 32 bits, with no error. prod of the int32 tensor [50000, 50000], cast to int64 first, returns -1,794,967,296 instead of 2,500,000,000. none so far needs an owner Repro
Integer cumsum over a constant comes back wrong on CPU from the second inference on: the mask keeps only what earlier calls allowed. Seen on 2025.4.1, 2026.0.0 and 2026.4.1, not on the nightly, and only when the graph is converted the way ExecuTorch does it (plain openvino.convert_model gives correct results). pi0 family needs an owner (released versions) Worked around in ExecuTorch 1.6 (computed during lowering): pi05 matches on 16 chunks in a row, and stopped matching after a few without it
A gripper flip written with ... (x[..., dims] = 1.0 - x[..., dims]) becomes an index_put with a leading None index and a 0-d value. AtenIndexPutReplacer transposes the value without broadcasting it first, so shape inference rejects it (repro C). Without ..., a list index (x[:, [7]]) fails the same way; only an integer index (x[:, 7]) converts. A bool mask after ... converts, but OpenVINO returns wrong values (off by 0.68), and ExecuTorch then fails at run time with no reason. Current master has the same code. flux3 needs an owner: broadcast the value before the transpose Repro
AtenIndexPutReplacer adds an int64 tensor to an int32 one for an index_put whose indices are a list of tensors, and the Add is rejected (repro B). flux3 (so101 checkpoint) needs an owner Measured on Oct 31
A partition that returns a value and a copy of it does not convert (Check 'is_number(name)' failed at src/frontends/pytorch/src/translate_session.cpp:434). With s = torch.sin(x): s, s.clone(), s.clone(), s.clone() and s, s[:, :] fail; s.clone(), torch.cos(s) and s, s work. none so far needs an owner: accept 2 outputs that become the same tensor, as for s, s (ExecuTorch could also keep such copies out of the partition) Repro
Lowering is slow for large models: wall_x about 8.3 hours (61,000 nodes, 12 partitions, shared CPU), lingbot_vla_v2 not done in 12 hours in either precision, flux3 about 2.6 hours (up to 134 GB) before the gripper flip row stops it. Part is ExecuTorch partitioning, fixed in 1.6 rc4 (see #24667). wall_x, lingbot_vla_v2, flux3 needs an owner: profile the conversion step to see the OpenVINO share Measured
HETERO:CPU lowering crashes in export_model(), and AUTO:CPU fails with Not Implemented. set_property on a HETERO: device string is rejected (2025.1.0 to 2026.4.1), and that is how the ExecuTorch runtime applies inference_precision. Setting it on each listed device works. none (API) needs an owner Measured
A bf16 constant can come back rounded differently from PyTorch. Small effect. not narrowed needs an owner (low priority) Seen
Repros

A. Output that is a model input (plain OpenVINO). On 2026.4.1, x and output 0 both hold -7, and output 1 is sin(-7) = -0.657 instead of sin(1) = 0.841. Only output 2 is right.

import numpy as np
import openvino as ov
import openvino.opset13 as ops

x = ops.parameter([2, 3], ov.Type.f32, name="x")
y = ops.parameter([2, 3], ov.Type.f32, name="y")
# Output 0 is input x itself. Output 1 is computed from x, output 2 from y.
model = ov.Model([ops.result(x), ops.result(ops.sin(x)), ops.result(ops.sin(y))], [x, y])
request = ov.Core().compile_model(model, "CPU").create_infer_request()

xv, yv = np.ones((2, 3), np.float32), np.ones((2, 3), np.float32)
outs = [np.full((2, 3), -7.0, np.float32) for _ in range(3)]
request.set_input_tensor(0, ov.Tensor(xv, shared_memory=True))
request.set_input_tensor(1, ov.Tensor(yv, shared_memory=True))
for i, out in enumerate(outs):
    request.set_output_tensor(i, ov.Tensor(out, shared_memory=True))
request.infer()
print(xv[0], outs[0][0], outs[1][0], outs[2][0])

B. Mixed int64 and int32 in AtenIndexPutReplacer:

While validating node 'opset1::Add Add_18585 (opset1::Add Add_18584[0]:i64[1,1,1,1],
  opset1::Unsqueeze aten_unsqueeze_copy_default_23/...[0]:i32[128,1]) -> (dynamic[...])'

C. Gripper flip with ... (plain OpenVINO):

import openvino as ov
import torch


class Flip(torch.nn.Module):
    def forward(self, x):
        x = x.clone()
        x[..., [7]] = 1.0 - x[..., [7]]
        return x


ov.convert_model(torch.export.export(Flip(), (torch.randn(1, 8),)))  # fails in AtenIndexPutReplacer

D. Dim order at a partition edge, lowered with OpenvinoPartitioner for CPU:

x = torch.randn(2, 3, 4, 5)


class Output(torch.nn.Module):  # run with x: max abs error 8.87
    def forward(self, x):
        return (x * 2).to(memory_format=torch.channels_last)


class Input(torch.nn.Module):  # run with x.to(memory_format=torch.channels_last): max abs error 0.29
    def forward(self, x):
        return x.mean(dim=(2, 3))

E. Qwen3-VL DeepStack (_deepstack_process in transformers' modeling_qwen3_vl.py). On the second layer the mask reaches aten.index with a dynamic rank:

local_this = hidden_states[visual_pos_masks, :] + visual_embeds
hidden_states[visual_pos_masks, :] = local_this

LeRobot
  • Review the open pull requests in the table.
  • Decide the rows marked "needs a decision".
Problem Policies Fix Status
transformers 5.22 drops the vision_model. level from SigLIP's weight names, and the pi0 and pi05 checkpoints keep the old names for 437 weights. The export's partial loader does not map them (the regular loader does), so the export stops before ExecuTorch. pi0, pi05 huggingface/lerobot#4775 Open. On an earlier build, OpenVINO passed pi0's check at 9.89e-06.
lawam's head cannot take bf16 on the CPU. lawam huggingface/lerobot#5214 Used for the lawam row
The published flux3 so101 checkpoint uses history conditioning, which the export does not serve yet, so the row uses the droid checkpoint. NATTEN needs the policy's backend passed through. flux3 huggingface/lerobot#5213, plus one more pull request queued behind GitHub's open pull request limit Then stops on the gripper flip row (OpenVINO table)
lerobot/fastwam_base has no action head, so the export adds its 6 action head tensors at their initial values. A fix for its CPU float32 layers is ready but waits behind GitHub's open pull request limit. fastwam ready, not opened yet Open
evo1's check uses 1 test frame where float32 flips the gripper: at action step 36, eager bf16 gives +0.70 and eager float32 gives -0.79. OpenVINO computes this policy in float32, so it lands on the float32 side: 1.45, against 0.075. evo1 needs a decision: check 16 frames (as in #24353) Measured
lingbot_va in bf16: another attention kernel or matmul setting alone moves its eager actions by up to 0.52, so this LeRobot version allows 1.0 on the whole test case and checks each program on its own. (Its rotary embedding now uses real numbers, so the edge verifier accepts it.) lingbot_va needs a decision: check per program (as in #24353) OpenVINO passes at 8.79e-02
There is no export recipe for lingbot_vla_v2 in this LeRobot version, so the trace stops before any backend sees the model (Exporting LingBot-VLA v2 needs a vision_grid). lingbot_vla_v2 in progress, no pull request yet Open

cc @suryasidd @daniil-lyakhov @anzr299

Vorherrschende Sprache
Python
Sterne
5.2k
Forks
1.2k
Ø Merge
1 T. 23 Std.
Gemergte PRs (30 T.)
700

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus pytorch/executorch

Alle Issues in pytorch/executorch

Ähnliche Issues

Weitere Issues zu Python

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.