Feature: k2_horizon_uno decode is about 20% slower than needed (no CUDA graph; LoRA branch computed on rows whose mask is 0)
维护者通常 1 天内回复
还没有人认领这个 Issue。
评估
- 难度
- 5/5
- 预计耗时
- 一周以上
- 新手友好度
- 35/100
- Issue 类型
- 功能
- 描述清晰度
- 基本清楚
- 活跃度
- 活跃
调研方向
Read families/k2_horizon_uno/README.md, especially section 7, then trace the Uno native runtime and engine profiles used by the k2-horizon-7b-uno manifest in linear_spec_lora mode. Investigate CUDA graph replay for each supported row count and how prompt and verify calls can use an engine without LoRA. Done means both changes are measured against the reported bf16 baselines and decoding behavior is checked for regressions.
由索引模型根据 Issue 内容生成。
描述
Request type: Improvement to existing behavior
Problem and use case
families/k2_horizon_uno runs speculative decoding with one dynamic engine that takes 1 to 8 rows per call and a per-row LoRA mask (draft rows mask 1; prompt and verify rows mask 0). Two costs on main, found during the FP8 study recorded in families/k2_horizon_uno/README.md (#2039, section 7):
- The Uno native runtime never enables CUDA graphs. A bf16 8-row call launches about 830 kernels; the host loop spends about 6 ms per 8-row call against 5.3 to 5.7 ms when the same call is replayed as a CUDA graph.
- The LoRA branch is computed for every row of every call, including prompt and verify calls whose mask is all 0. The branch (rank 128 on all 7 projections, in bf16) is 19% to 23% of a call: the bf16 engine without the branch is 1.21x to 1.24x faster.
Both are larger than what FP8 gives on this family (FP8 was measured and not shipped, see the README).
Proposed outcome
- Replay the decode calls as CUDA graphs in the Uno native runtime (per row count used).
- Skip the LoRA branch when the mask of a call is all 0, for example with a second engine profile (or engine) without LoRA for prompt and verify calls.
Model and target details
IFM/K2-Horizon-7B-Uno @ ec92bbd768f4a404319625204544782e3377bcd7 (the k2-horizon-7b-uno manifest, linear_spec_lora mode) on the K2-Horizon-7B base; bf16; TensorRT 11.4; B200 / B300; max_sequence_length 1024.
Alternatives considered
FP8 of the base projections (measured, not shipped): the 8-row call is 0.96x (B200) and 0.88x (B300) of bf16 without a CUDA graph, because the bf16 LoRA branch and the kernel count stay.
Additional context
The same study found that speculative decode equals the engine's own greedy autoregressive output on 13 of 14 prompts only (bf16): 8-row and 1-row calls differ in the last bits, which changes the argmax at near ties.
- 主要语言
- Python
- 星标
- 288
- 派生
- 73
- 平均合并
- 1 天 3 小时
- 30 天内合并 PR
- 273
环境准备
- 提供 Dockerfile 或 Docker Compose 文件
- 有 Pull Request 模板
- 阅读贡献指南
从这里开始
- 先读完整个 Issue，再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库，在一个分支上完成修改。
- 提交 Pull Request，并在描述里引用这个 Issue 编号。
NVIDIA/TensorRT-Model-Connect 的其他 Issue
-
Docs: ITrtModule::forward() output-buffer lifetime and dtype contract are undocumented可能重新可做 关联的 PR 已关闭且未合并。 未关闭
难度 2/5 1-3 小时 新手友好度 82/100
NVIDIA/TensorRT-Model-Connect#1126 · 1 条评论 ·
维护者通常 1 天内回复
-
enhancement
难度 5/5 一周以上 新手友好度 35/100
NVIDIA/TensorRT-Model-Connect#2060 ·
维护者通常 1 天内回复
-
enhancement
难度 5/5 一周以上 新手友好度 42/100
NVIDIA/TensorRT-Model-Connect#2053 ·
维护者通常 1 天内回复
-
Feature: fp16 builds for timesfm (3x to 12x faster than fp32) need an overflow guard for large normalized inputs可能已有人在做 关联的 PR 仍在进行中或已合并。 未关闭enhancement
难度 4/5 3-5 天 新手友好度 46/100
NVIDIA/TensorRT-Model-Connect#2050 ·
维护者通常 1 天内回复
-
bug
难度 4/5 3-5 天 新手友好度 48/100
NVIDIA/TensorRT-Model-Connect#2049 ·
维护者通常 1 天内回复
查看 NVIDIA/TensorRT-Model-Connect 的全部 Issue
相似的 Issue
-
bug
难度 2/5 1-3 小时 新手友好度 74/100
igormcsouza/tether#278 ·
维护者通常 1 天内回复
-
难度 1/5 1 小时以内 新手友好度 91/100
aws-samples/sample-per-model-bedrock#803 ·
维护者通常 1 天内回复
-
documentation p1
难度 2/5 1-3 小时 新手友好度 75/100
维护者通常 1 天内回复
-
难度 2/5 1-3 小时 新手友好度 78/100
huggingface/accelerate#4511 ·
维护者通常 1 天内回复
-
follow-up
难度 2/5 1-3 小时 新手友好度 78/100
维护者通常 1 天内回复