Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Tracking Issue] Synthetic Acceptance for MTP Benchmarks / [Tracking Issue] MTP 基准测试的合成接受长度

Abierto
  • ¿Algo no está bien?
#1,651 7 comentarios 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 1 día

@xinli-sw ya está trabajando en esto.

Desde el 2/6/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

in progress p0

Proposed Change: Synthetic Acceptance

Recently, all major serving frameworks (vLLM, SGLang, TensorRT-LLM) have added support for synthetic acceptance length (AL) for benchmarking purposes: the server can be configured to accept specific number of draft tokens in order to match (on average) a target acceptance length. Using this feature, we can match expected generation speeds with far higher consistency, and have much more control over the fidelity of the acceptance rate.

Doing so will also ensure consistent benchmarking environments across FWs, avoid honest mistakes such as client side settings (e.g. chat-template), and provide uniform performance expectations when using random and pseudorandom text for benchmarking purposes.

For each model and number of speculated tokens, SemiAnalysis chooses a fixed AL based on the observed MTP AL calibrated on SpeedBench coding dataset using model-provider-recommended sampling parameters (M0). For example, running DSV4-Pro on SpeedBench coding with 3 MTP draft tokens gives a mean AL of ~2.75, so the settings would be roughly [MTP1: AL 1.85, MTP2: AL 2.45, MTP3: AL 2.75] (mock numbers).

We propose the following methodology:

M0: New Github Action for Auditable Reference AL Generation

We introduce a new GitHub Action that integrates SpeedBench to InferenceX. For each model that supports MTP, we use the workflow to generate reference AL values, with the coding dataset, for both thinking on and thinking off. All reference values will be generated using vLLM for consistency, for speculative tokens going from 1 to 7.

M1: Evaluation runs for correct implementation of MTP

During evaluation the synthetic acceptance feature is disabled and the same model-provider-recommended sampling parameters are used. The eval script now includes two pieces:

  1. Run SpeedBench, acceptance rate is automatically collected from the server at the end of the run. Then: we assert that the achieved acceptance length is above a threshold from the reference values.
  2. Run the existing GSM8k evals to ensure accuracy still pass.

M2: Apply Synthetic Acceptance During Benchmarks

During benchmarking synthetic acceptance is enabled with the target AL and uses the same prompt generation setup as non-MTP. We will roll this out with single node first, followed by multi-node tests. To make sure the benchmarks are fair, we ensure implementation for synthetic MTP acceptance rates to be open, auditable, and mathematically equivalent for all participating FWs.

  • M0: New Github Action for Auditable Reference AL Generation
  • M1: Evaluation runs for correct implementation of MTP
  • M2: Apply Synthetic Acceptance During Benchmarks

中文说明

提议在 MTP 基准测试中引入合成接受长度（Synthetic Acceptance Length）机制。所有主流推理框架（vLLM、SGLang、TensorRT-LLM）现已支持配置合成接受长度，可按目标接受率匹配预期生成速度，确保跨框架基准测试环境的一致性。分三个里程碑推进：M0——新增 GitHub Action，使用 SpeedBench 编码数据集为每个支持 MTP 的模型生成可审计的参考接受长度值；M1——评估运行，禁用合成接受功能并断言实际接受长度达到阈值，同时运行 GSM8K 准确性评估；M2——在基准测试中启用合成接受，先单节点后多节点，确保各框架的实现公开、可审计且数学等价。

Lenguaje dominante
Python
Estrellas
1.9k
Forks
343
Merge medio
2 d 40 min
PR fusionados (30 d)
431

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de SemiAnalysisAI/InferenceX

Todos los issues de SemiAnalysisAI/InferenceX

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.