Hacktoberfest 2026：维护者为十月标记出来的 issue，仍然开放、适合新手。 浏览 Hacktoberfest issue

[Tracking Issue] Synthetic Acceptance for MTP Benchmarks / [Tracking Issue] MTP 基准测试的合成接受长度

未关闭
  • 哪里不对？
#1,651 7 条评论 0 个 reaction 已指派 1 人 在 GitHub 查看

维护者通常 1 天内回复

@xinli-sw 已经在做这个了。

开始于 2026年6月2日。

评估

这个 Issue 还没有评估数据。

描述

in progress p0

Proposed Change: Synthetic Acceptance

Recently, all major serving frameworks (vLLM, SGLang, TensorRT-LLM) have added support for synthetic acceptance length (AL) for benchmarking purposes: the server can be configured to accept specific number of draft tokens in order to match (on average) a target acceptance length. Using this feature, we can match expected generation speeds with far higher consistency, and have much more control over the fidelity of the acceptance rate.

Doing so will also ensure consistent benchmarking environments across FWs, avoid honest mistakes such as client side settings (e.g. chat-template), and provide uniform performance expectations when using random and pseudorandom text for benchmarking purposes.

For each model and number of speculated tokens, SemiAnalysis chooses a fixed AL based on the observed MTP AL calibrated on SpeedBench coding dataset using model-provider-recommended sampling parameters (M0). For example, running DSV4-Pro on SpeedBench coding with 3 MTP draft tokens gives a mean AL of ~2.75, so the settings would be roughly [MTP1: AL 1.85, MTP2: AL 2.45, MTP3: AL 2.75] (mock numbers).

We propose the following methodology:

M0: New Github Action for Auditable Reference AL Generation

We introduce a new GitHub Action that integrates SpeedBench to InferenceX. For each model that supports MTP, we use the workflow to generate reference AL values, with the coding dataset, for both thinking on and thinking off. All reference values will be generated using vLLM for consistency, for speculative tokens going from 1 to 7.

M1: Evaluation runs for correct implementation of MTP

During evaluation the synthetic acceptance feature is disabled and the same model-provider-recommended sampling parameters are used. The eval script now includes two pieces:

  1. Run SpeedBench, acceptance rate is automatically collected from the server at the end of the run. Then: we assert that the achieved acceptance length is above a threshold from the reference values.
  2. Run the existing GSM8k evals to ensure accuracy still pass.

M2: Apply Synthetic Acceptance During Benchmarks

During benchmarking synthetic acceptance is enabled with the target AL and uses the same prompt generation setup as non-MTP. We will roll this out with single node first, followed by multi-node tests. To make sure the benchmarks are fair, we ensure implementation for synthetic MTP acceptance rates to be open, auditable, and mathematically equivalent for all participating FWs.

  • M0: New Github Action for Auditable Reference AL Generation
  • M1: Evaluation runs for correct implementation of MTP
  • M2: Apply Synthetic Acceptance During Benchmarks

中文说明

提议在 MTP 基准测试中引入合成接受长度（Synthetic Acceptance Length）机制。所有主流推理框架（vLLM、SGLang、TensorRT-LLM）现已支持配置合成接受长度，可按目标接受率匹配预期生成速度，确保跨框架基准测试环境的一致性。分三个里程碑推进：M0——新增 GitHub Action，使用 SpeedBench 编码数据集为每个支持 MTP 的模型生成可审计的参考接受长度值；M1——评估运行，禁用合成接受功能并断言实际接受长度达到阈值，同时运行 GSM8K 准确性评估；M2——在基准测试中启用合成接受，先单节点后多节点，确保各框架的实现公开、可审计且数学等价。

主要语言
Python
星标
1.9k
派生
343
平均合并
2 天 40 分钟
30 天内合并 PR
431

环境准备

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

SemiAnalysisAI/InferenceX 的其他 Issue

查看 SemiAnalysisAI/InferenceX 的全部 Issue

相似的 Issue

更多 Python Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。