[Bug]: kv cache potential bugs
Maintainers usually reply within 1 day
@xin3he is already working on this.
Since Oct 20, 2026.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Active
- Tech stack
- python
- Domain
- machine-learning
Research direction
Start in kv_cache.py: the FP8 path in _quant_dequant (around lines 190-201) overwrites the per-layer scale on every forward, while _nvfp4_track_amax (lines 203-227) keeps a running max. Follow how calibrate_kv_cache_output_hook copies the scale into k_scale/v_scale, and where kvcache_quant_context wraps quantize() in base.py (around line 2188). Done means the FP8 scale accumulates a max across calibration like the NVFP4 path, with a test showing the exported scale reflects the largest observed KV amax.
Written by the indexing model from the issue text.
Description
Problem Description
1 诊断：FP8 static KV cache 精度偏低的根因
核心 bug：FP8 的 KV scale 没有在整个校准集上做累积（running max），只保留了"最后一次 forward"的 scale。
看 kv_cache.py 里 FP8 路径 _quant_dequant（第 190-201 行）：
qdq_tensor, scale = fp8_qdq(tensor, granularity=self.granularity) # tensor_max=None → 用"当前这一次" forward 的 amax
pad_and_append_at_idx(scales, layer_idx, scale.reshape(-1).detach()) # 直接覆盖该层的 scale
fp8_qdq 传入的 tensor_max=None，意味着每次 forward 都用当前这一个 batch/样本的 amax 现算一个 scale。
pad_and_append_at_idx 是覆盖写，不是取 max。
calibrate_kv_cache_output_hook 在每次 attention forward 后把这个 scale 拷进 k_scale/v_scale。
因为 kvcache_quant_context 包住了整个 self.quantize()（base.py 2188），attention 会被成百上千次 forward（所有校准样本 + 每个 block 的调优迭代）。最终导出的 scale 只反映最后一次 forward（即最后一个样本/最后一次调优 step）的统计量，而不是整个校准集的代表值。
对比一下 NVFP4 路径就是对的（第 203-227 行 _nvfp4_track_amax）——它用 amax_list[layer_idx] = max(amax_list[layer_idx], amax) 做了跨样本的 running max，只在最后由 output hook 换算成 scale。FP8 路径缺了这个累积步骤。
后果： 如果最后一次 forward 的 KV 幅度比其它样本小，scale 就偏小，推理时其它更大的 KV 值会被 clip 到 fp8 max（饱和）→ 精度下降；即便不饱和，scale 也是单样本噪声，不够鲁棒。这正是标准静态 KV 量化（llm-compressor 等）要用 observer 累积 min/max 的原因。
2. The reference output with FP8 KV cache enabled should be generated using BF16 weights and BF16 KV cache, currently it is generated by BF16 weights and FP8 KV cache
3. Tuning is not aware of NVFP4 KV cache.
4. torch.compile
block_forward methods use hooks generally cannot be compiled with torch.compile, as the hooks' stored values may be eliminated during compile
Reproduction Steps
~
Environment Information
~
Error Logs
Additional Context
No response
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 195
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 64
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from intel/auto-round
-
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
intel/auto-round#2525 ·
Maintainers usually reply within 1 day
-
[Bug]: ARK return_lse gives the same LSE for every 16 query rows (sdpa, sagev1, sageattn) on XPUPossibly taken @luoyu-intel claimed this 1 day ago. Openbug
Difficulty 4/5 3-5 days Newbie friendliness 55/100
intel/auto-round#2556 · 1 comment · 1 assignee ·
Maintainers usually reply within 1 day
-
[CI Flaky] test_other_bits 2-bit UT fails intermittently (Accuracy 0.18 < threshold 0.195)Possibly taken @luoyu-intel claimed this 2 days ago. Openbug CI-known-issue
intel/auto-round#2550 · 1 assignee ·
Maintainers usually reply within 1 day
-
[Feature]: support NVFP4_E5M3_GSPossibly taken @xin3he claimed this 2 days ago. Openenhancement
Difficulty 4/5 3-5 days Newbie friendliness 44/100
intel/auto-round#2532 · 2 comments · 1 assignee ·
Maintainers usually reply within 1 day
-
[Feature]: split special_model_handler to a folderPossibly taken @Cross2pro claimed this 3 days ago. Openenhancement
Difficulty 3/5 Half a day Newbie friendliness 42/100
intel/auto-round#2514 · 5 comments · 1 assignee ·
Maintainers usually reply within 1 day
All issues in intel/auto-round
Similar issues
-
getchaintips leaves out the active tip once a header is staged on it, where Core always lists itOpen
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
btclib-org/btclib-node#2158 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 85/100
dbt-msft/dbt-sqlserver#897 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
Gathin23/cleanyourtv#2770 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
vllm-project/vllm#64845 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
dekobon/big-code-analysis#1729 ·
Maintainers usually reply within 1 day