Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

Capture-time cuGraphUpload (#697) widens inter-kernel gaps in replayed decode graphs: ~1-2% slower decode

Abierto
  • ¿Algo no está bien?
#775 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
1-2 días
Aptitud para principiantes
48/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
backend, performance

Línea de trabajo

Start with python/freetoken/engine/graph.py and inspect _upload_graph() at lines 118–128 and its call in the capture loop at line 222. Compare replay timing with and without cuGraphUpload, and check how disk PLE wait-sync releases its fill around cudaGraphLaunch. Done means eliminating the reported inter-kernel gap regression while preserving the deadlock fix; verify with the same decode workload and Nsight Systems measurements.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

What happened

Since #697 (fix(engine): upload cuda graphs at capture …), decode steps replayed from CUDA graphs are about 1-2% slower. The extra time is idle time between kernels; the kernels themselves are not slower.

Measured on Qwen3.8-Flash-Next-NVFP4 (qwen4_exp, MoE offload, decode CUDA graphs at bs 1-2) with a like-for-like decode workload: the model copies a file verbatim from a 64K context, greedy, thinking off. The output hash is identical in every arm, so the tokens and expert misses match.

GPU ms per decode step tok/s
with the capture-time upload (current main) 20.97 47.56
same build, _upload_graph turned into a no-op 20.54 48.56
build before #697 20.51 48.62

nsys over 200 decode steps:

  • Kernel counts and API calls are identical.
  • Kernel busy time is not higher with the upload; it is 0.37 ms per step lower.
  • The median gap between consecutive kernels goes from 0.19 µs to 0.51 µs on all four streams the graph uses.

At about 1090 kernels per step, that adds 0.2-0.45 ms per step. The higher figure is with --moe-collect-stats, which adds kernels.

Cause (current main, cc56b6bd)

python/freetoken/engine/graph.py:118-128 _upload_graph() calls cuGraphUpload on every captured graph, then stream.synchronize(). It is called from the capture loop at :222. Graphs uploaded this way replay with the wider inter-kernel gaps above. Uploading after all graphs are captured instead of after each one gave the same result.

Proposed fix
  • Upload only where the deadlock #697 fixes can occur: a disk PLE backend in wait-sync mode whose fill is released after cudaGraphLaunch returns. Skip it otherwise, or put it behind a switch.
  • Or replace the upload with one warm replay at capture time. This recovered about 60% of the cost here. It needs the PLE wait released during that replay, for example by running the fill on a thread submitted before the launch.
Environment

RTX 5080 16 GB (sm_120, driver 615.78.08), torch 2.14.1+cu130, CUDA graphs for decode at bs 1 and 2, nvidia/Qwen3.8-Flash-Next-NVFP4, main cc56b6bd.

Lenguaje dominante
Python
Estrellas
14.2k
Forks
1.4k
Merge medio
21 h 53 min
PR fusionados (30 d)
100

Preparar el entorno

Este proyecto no incluye contenedor de desarrollo, Dockerfile ni guía de contribución, así que la configuración corre por tu cuenta: empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de FlashML-org/FreeToken

Todos los issues de FlashML-org/FreeToken

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.