FastMetal-14B-QAD hangs before first denoise step on M4 Max (1.3B/5B fine; GPU command buffer never completes)

Abierto
  • ¿Algo no está bien?
#2,016 2 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 2 días

@Ishxn20 ya está trabajando en esto.

Desde el 6/11/2026.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
machine-learning, performance

Línea de trabajo

Start with examples/inference/basic/mlx_wan_prompt_to_video.py and reproduce using the documented command with the 14B checkpoint; compare against the working 1.3B and 5B runs. Read the MLX issue #3797 mentioned in the report and investigate where the 14B run stalls after weights load. Done means identifying the cause or a reliable diagnostic and confirming the 14B run reaches its first denoise step.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

installation performance platform scope: attention scope: docs scope: inference scope: kernel scope: model

Describe the bug

FastMetal-14B-QAD hangs indefinitely on Apple M4 Max (128 GB, macOS 27.0.1) when running the documented MLX text-to-video example. The process never reaches the first denoise step. Reproduced twice, including with OMP_NUM_THREADS=1 MKL_NUM_THREADS=1.

FastMetal-1.3B-QAD and FastMetal-5B-QAD work perfectly on the same machine, same script, same environment (1.3B: 3.87 GiB peak, matching the published baseline exactly; 5B 720p: 9.337 GiB peak / 122.1 s denoise, both matching the published numbers).

Command

.venv/bin/python examples/inference/basic/mlx_wan_prompt_to_video.py \
  --model-root /Users/yanyu/models/FastMetal-14B-QAD \
  --mlx-checkpoint /Users/yanyu/models/FastMetal-14B-QAD \
  --height 480 --width 832 --num-frames 81 \
  --output-path video14b.mp4 \
  --prompt "A fox runs through a misty pine forest, leaves kicking up behind it."

Last log line: Loading weights: 100%|██| 242/242 — then silence (no denoise step 1/3 ever).

Environment

  • MacBook Pro, Apple M4 Max (40-core GPU), 128 GB unified memory
  • macOS 27.0.1 (build 26A434)
  • Python 3.13, FastVideo installed via pip install -e '.[mlx]'
  • mlx 0.32.4 / mlx-lm 0.32.2 / mlx-metal 0.32.4
  • Model downloaded via hf download FastVideo/FastMetal-14B-QAD (39 GB on disk, all 8 safetensors open cleanly)

Evidence that the hang is in the 14B DiT load / first step (not text encoding)

  1. Text encoding completed on the first attempt: FastVideo wrote the prompt-embeds cache at ~/.cache/fastvideo/prompt_embeds/04cc776e*.npy (its .npy.json fingerprint points at the 14B text_encoder root) before the hang. Subsequent runs hit this cache instantly and still hang — so the hang is strictly after prompt encoding.
  2. sample <pid> stack (macOS): main thread parked in _PySemaphore_Wait; all libmlx ThreadPool workers idle in condition_variable::wait; one Metal CommandQueueDispatch thread is live inside IOGPUCommandQueueSubmitCommandBuffers — i.e. a GPU command buffer was submitted and never completes (kernel-level hang/livelock signature), while the CPU side is doing nothing.
  3. CPU time frozen: ps cpu_time stops advancing (~15 s total after 10+ min wall); process sits at 0.0–0.3% CPU.
  4. Partial residency: at hang, system-wide wired memory is only ~12.7 GiB (baseline ~7 GiB) while process RSS is ~14.8 GiB — the 14.4 GB DiT checkpoint is not fully resident on the GPU when it stalls.
  5. Not shader compilation: MTLCompilerService idle, Metal shader cache byte-identical over 5 minutes of waiting.

What we ruled out

  • Missing --mlx-checkpoint (the classic silent-online-fetch hang): both args passed on every attempt.
  • libomp/OpenMP deadlock: OMP_NUM_THREADS=1 changes nothing; 1.3B/5B run fine in the same env.
  • Network wait: zero TCP connections on the hung process.
  • Memory pressure: Metal budget 107.5 GiB; the published 14B peak is 21.68 GiB. System had >60 GiB free at hang.

Suspicion

Given mlx issue #3797 (NAX split-K GEMM NaN/livelock family on Apple GPUs for specific feed-forward shapes) already affected Qwen-Image-Flash at 4096-token sequences, the 14B Wan DiT may hit a related kernel path on M4 Max at this size. The 1.3B/5B models don't reach that regime.

Happy to run any diagnostics or test a fixed checkpoint/runtime build on this machine.

Full local dossier

We keep a Chinese dossier with raw sample output and a reproducible timeline at: yyc3-22:~/models/apple-silicon-stack/results/fastmetal-14b-hang-dossier.md.

Lenguaje dominante
Python
Estrellas
4.6k
Forks
486
Merge medio
3 d 15 h
PR fusionados (30 d)
74

Preparar el entorno

  • Sin Dockerfile ni archivo de Docker Compose
  • Tiene una plantilla de pull request
  • Sin guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de hao-ai-lab/FastVideo

Todos los issues de hao-ai-lab/FastVideo

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.