A2VidPipelineTwoStage on DGX Spark (GB10): frequent Xid 31 / "unspecified launch failure" at the stage-2 handoff with the standard transformer loader; passes with CUDA_LAUNCH_BLOCKING=1 or --offload cpu

Start by comparing the standard and streaming transformer loader paths and the stage handoff around diffusion_stage.py line 549, ltx_core/tools.py line 159, and the cited copy sites in ltx_core/loader/state_loader.py and ltx_core/devices.py. Reproduce on DGX Spark with the standard loader, then compare against CUDA_LAUNCH_BLOCKING=1 and --offload cpu. Done means identifying and fixing the cause of the intermittent Xid 31 failure while preserving successful stage-2 inference.

On an NVIDIA DGX Spark (GB10), ltx_pipelines.a2vid_two_stage with LTX-2.5 faults in 8 of 10 runs with the default (standard) transformer loader. It dies right after stage 1, at the start of stage 2, with CUDA error: unspecified launch failure, and the kernel logs an MMU fault from the copy engine (Xid 31). The same command completes every time with CUDA_LAUNCH_BLOCKING=1 or with --offload cpu (which switches to the "streaming" loader), and the output of those two is bit-identical (PSNR inf over 185 frames, same seed).

I don't know whether the root cause is in LTX-2, PyTorch, or the GB10 driver stack. I'm filing it here because the failing call is in the stage handoff and because the loader choice changes the outcome. I also filed a driver-side report on NVIDIA's DGX Spark forum: [https://forums.developer.nvidia.com/t/repeated-xid-31-ce1-hubclient-ce0-fault-pte-virt-read-on-gb10-driver-580-178-04-reproduced-on-ngc-26-05-26-09-and-a-stock-upstream/388217].

Environment

• DGX Spark / GB10, 128 GB unified memory; DGX OS 7.6.0, kernel 7.0.0-1019-nvidia, host driver 580.178.04
• LTX-2 commit c1b4675863c07c59c92e502446cccdaedbe5889c (2026-11-03), installed with pip install -e packages/ltx-core -e packages/ltx-pipelines against the existing torch (not uv sync)
• Weights (Lightricks/LTX-2.5): ltx-2.5-22b-dev-transformer-bf16, gemma4-12b-with-proj-ltx-2.5-bf16, ltx-2.5-video-vae-conv-bf16, ltx-2.5-audio-vae-bf16, ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0, ltx-2.5-22b-distilled-lora-450-bf16
• TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas; no torch.compile; attention backend is the default SDPA order shown in the log

Tested in three software environments on the same machine:

Environment	torch	CUDA user-mode driver
NGC pytorch:26.09-py3 container	2.14.0a0 (NVIDIA build)	615.71.09 via forward compat (CUDA 13.4)
NGC pytorch:26.05-py3 container	2.12.0a0 (NVIDIA build)	595.58.03 via forward compat (CUDA 13.2)
host venv	2.14.1+cu130 (upstream wheel)	host 580.178.04, native (CUDA 13.0)

(On 26.05, torchaudio 2.11.1 had to run in pure-Python mode because its extension doesn't match that torch's ABI; the audio mel/resample output was bit-identical to the 26.09 container's, so the conditioning is unchanged.)

Reproduce

python -m ltx_pipelines.a2vid_two_stage \
  --transformer-path .../ltx-2.5-22b-dev-transformer-bf16.safetensors \
  --text-encoder-path .../gemma4-12b-with-proj-ltx-2.5-bf16.safetensors \
  --video-vae-path .../ltx-2.5-video-vae-conv-bf16.safetensors \
  --audio-vae-path .../ltx-2.5-audio-vae-bf16.safetensors \
  --distilled-lora .../ltx-2.5-22b-distilled-lora-450-bf16.safetensors 1.0 \
  --spatial-upsampler-path .../ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors \
  --audio-path speech_2s.wav --audio-max-duration 2 \
  --image photo.png 0 1.0 --height 1024 --width 576 --seed 42 \
  --num-inference-steps 8 --prompt "..." --output-path out.mp4

It also fails with 8 s of audio and 30 steps, so the failure doesn't depend on size.

What happens

Stage 1 finishes. The log shows the second Building transformer (standard), then Running denoising loop (3 steps ...), and then:

File ".../diffusion_stage.py", line 549, in _build_state -> create_noised_state
File ".../ltx_core/tools.py", line 159, in create_initial_state
    clean_latent = initial_latent.clone()
torch.AcceleratorError: CUDA error: unspecified launch failure
  Sticky error detected / Returning 719 (CUDA_ERROR_LAUNCH_FAILED) from cuMemcpyDtoDAsync_v2

The kernel log gets one line per failed run, with the same engine, fault type, and channel every time (only the address and pid differ):

NVRM: Xid (PCI:000f:01:00): 31, pid=..., name=python, channel 0x04000005, intr 00000000.
MMU Fault: ENGINE CE1 HUBCLIENT_CE0 faulted @ 0x... Fault is of type FAULT_PTE ACCESS_TYPE_VIRT_READ

In the first failure, the process lived about 14 s after stage 2's transformer finished loading before the error surfaced. Host memory in use peaked at about 53 GB of ~120 GB, so this is not an out-of-memory condition. The GPU context recovers after the process exits.

Results (default loader, blocking off, unless noted)

Environment	Runs	Faults
NGC 26.09	3	3
NGC 26.05	3	2
host venv, upstream torch + native driver	4	3
Total	10	8

Configuration	Outcome
CUDA_LAUNCH_BLOCKING=1	passes in every run (dozens in the 26.09 container, including a two-window 57 s run; 1 on the host)
--offload cpu (log says Building transformer (streaming))	passes in all runs so far (4+ in the 26.09 container, 1 on the host)

Two diagnostic changes to the default path also faulted (2 of 2): (a) changing the non_blocking=... host-to-device copies in ltx_core/loader/state_loader.py (lines 141, 295, 312), devices.py:82, and modality_tiling.py:384 to non_blocking=False, and (b) disabling torch.cuda.empty_cache() in ltx_core/devices.py. Neither alone is the cause.

Cost of the workarounds, 8 s clip, 30 steps, stage 1: 8.99 s/it (default, which crashes later in stage 2), 9.57 s/it (blocking, +6%), 10.20 s/it (streaming, +13%); stage 2: 10.1 s/it (blocking), 11.26 s/it (streaming).

Observations, which I have not confirmed

• The failing operation is a plain .clone() of the upsampled latent, a copy-engine read of an unmapped address, right after gpu_model (TRIM strategy) synced, disposed, and trimmed the stage-1 models.
• GB10 has unified memory, so I suspect the standard loader's memory handling around this handoff (pages mapped or released while a copy is in flight). This is a guess; the same fault with the stock driver and an upstream PyTorch build suggests it isn't specific to NVIDIA's container.
• CUDA_LAUNCH_BLOCKING=1 hiding the fault points at a timing problem.

Questions

1. Has the standard loader been exercised on unified-memory hardware such as GB10? Is --offload cpu the recommended path there?
2. Which part of the standard build path differs from streaming in how weights are mapped or released between stages?
3. Is there a supported way to keep the transformers resident across stages and chunks instead of rebuilding them per chunk? (On this machine, each 97-frame window spends about 41 s rebuilding and reloading models, more than half of its cost.)

I can attach full logs and the dmesg lines on request.