Feature: k2_horizon_uno decode is about 20% slower than needed (no CUDA graph; LoRA branch computed on rows whose mask is 0)

オープン
  • 何か問題がありますか？
#2,054 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
cpp, python, pytorch
領域
backend, machine-learning, performance

調査の方向性

Read families/k2_horizon_uno/README.md, especially section 7, then trace the Uno native runtime and engine profiles used by the k2-horizon-7b-uno manifest in linear_spec_lora mode. Investigate CUDA graph replay for each supported row count and how prompt and verify calls can use an engine without LoRA. Done means both changes are measured against the reported bf16 baselines and decoding behavior is checked for regressions.

索引モデルが issue の本文から書いたものです。

説明

enhancement

Request type: Improvement to existing behavior

Problem and use case

families/k2_horizon_uno runs speculative decoding with one dynamic engine that takes 1 to 8 rows per call and a per-row LoRA mask (draft rows mask 1; prompt and verify rows mask 0). Two costs on main, found during the FP8 study recorded in families/k2_horizon_uno/README.md (#2039, section 7):

  1. The Uno native runtime never enables CUDA graphs. A bf16 8-row call launches about 830 kernels; the host loop spends about 6 ms per 8-row call against 5.3 to 5.7 ms when the same call is replayed as a CUDA graph.
  2. The LoRA branch is computed for every row of every call, including prompt and verify calls whose mask is all 0. The branch (rank 128 on all 7 projections, in bf16) is 19% to 23% of a call: the bf16 engine without the branch is 1.21x to 1.24x faster.

Both are larger than what FP8 gives on this family (FP8 was measured and not shipped, see the README).

Proposed outcome

  1. Replay the decode calls as CUDA graphs in the Uno native runtime (per row count used).
  2. Skip the LoRA branch when the mask of a call is all 0, for example with a second engine profile (or engine) without LoRA for prompt and verify calls.

Model and target details

IFM/K2-Horizon-7B-Uno @ ec92bbd768f4a404319625204544782e3377bcd7 (the k2-horizon-7b-uno manifest, linear_spec_lora mode) on the K2-Horizon-7B base; bf16; TensorRT 11.4; B200 / B300; max_sequence_length 1024.

Alternatives considered

FP8 of the base projections (measured, not shipped): the 8-row call is 0.96x (B200) and 0.88x (B300) of bf16 without a CUDA graph, because the bf16 LoRA branch and the kernel count stay.

Additional context

The same study found that speculative decode equals the engine's own greedy autoregressive output on 13 of 14 prompts only (bf16): 8-row and 1-row calls differ in the last bits, which changes the argmax at near ties.

主要言語
Python
スター
288
フォーク
73
平均マージ
1日 3時間
マージ済み PR（30日）
273

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

NVIDIA/TensorRT-Model-Connect のほかの issue

NVIDIA/TensorRT-Model-Connect の issue をすべて見る

似ている issue

Python の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。