Capture-time cuGraphUpload (#697) widens inter-kernel gaps in replayed decode graphs: ~1-2% slower decode

未关闭
  • 哪里不对？
#775 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
4/5
预计耗时
1-2 天
新手友好度
48/100
Issue 类型
缺陷
描述清晰度
基本清楚
活跃度
活跃
技术栈
python
领域
backend, performance

调研方向

Start with python/freetoken/engine/graph.py and inspect _upload_graph() at lines 118–128 and its call in the capture loop at line 222. Compare replay timing with and without cuGraphUpload, and check how disk PLE wait-sync releases its fill around cudaGraphLaunch. Done means eliminating the reported inter-kernel gap regression while preserving the deadlock fix; verify with the same decode workload and Nsight Systems measurements.

由索引模型根据 Issue 内容生成。

描述

What happened

Since #697 (fix(engine): upload cuda graphs at capture …), decode steps replayed from CUDA graphs are about 1-2% slower. The extra time is idle time between kernels; the kernels themselves are not slower.

Measured on Qwen3.8-Flash-Next-NVFP4 (qwen4_exp, MoE offload, decode CUDA graphs at bs 1-2) with a like-for-like decode workload: the model copies a file verbatim from a 64K context, greedy, thinking off. The output hash is identical in every arm, so the tokens and expert misses match.

GPU ms per decode step tok/s
with the capture-time upload (current main) 20.97 47.56
same build, _upload_graph turned into a no-op 20.54 48.56
build before #697 20.51 48.62

nsys over 200 decode steps:

  • Kernel counts and API calls are identical.
  • Kernel busy time is not higher with the upload; it is 0.37 ms per step lower.
  • The median gap between consecutive kernels goes from 0.19 µs to 0.51 µs on all four streams the graph uses.

At about 1090 kernels per step, that adds 0.2-0.45 ms per step. The higher figure is with --moe-collect-stats, which adds kernels.

Cause (current main, cc56b6bd)

python/freetoken/engine/graph.py:118-128 _upload_graph() calls cuGraphUpload on every captured graph, then stream.synchronize(). It is called from the capture loop at :222. Graphs uploaded this way replay with the wider inter-kernel gaps above. Uploading after all graphs are captured instead of after each one gave the same result.

Proposed fix
  • Upload only where the deadlock #697 fixes can occur: a disk PLE backend in wait-sync mode whose fill is released after cudaGraphLaunch returns. Skip it otherwise, or put it behind a switch.
  • Or replace the upload with one warm replay at capture time. This recovered about 60% of the cost here. It needs the PLE wait released during that replay, for example by running the fill on a thread submitted before the launch.
Environment

RTX 5080 16 GB (sm_120, driver 615.78.08), torch 2.14.1+cu130, CUDA graphs for decode at bs 1 and 2, nvidia/Qwen3.8-Flash-Next-NVFP4, main cc56b6bd.

主要语言
Python
星标
14.2k
派生
1.4k
平均合并
18 小时 46 分钟
30 天内合并 PR
98

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南，环境需要你自己搭建：先看它的 README，通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

FlashML-org/FreeToken 的其他 Issue

查看 FlashML-org/FreeToken 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。