[Tracking Issue] Synthetic Acceptance for MTP Benchmarks / [Tracking Issue] MTP 基准测试的合成接受长度

Đang mở
  • Có gì không ổn?
#1,651 7 bình luận 0 reaction 1 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

@xinli-sw đang làm issue này rồi.

Từ ngày 2/6/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

in progress p0

Proposed Change: Synthetic Acceptance

Recently, all major serving frameworks (vLLM, SGLang, TensorRT-LLM) have added support for synthetic acceptance length (AL) for benchmarking purposes: the server can be configured to accept specific number of draft tokens in order to match (on average) a target acceptance length. Using this feature, we can match expected generation speeds with far higher consistency, and have much more control over the fidelity of the acceptance rate.

Doing so will also ensure consistent benchmarking environments across FWs, avoid honest mistakes such as client side settings (e.g. chat-template), and provide uniform performance expectations when using random and pseudorandom text for benchmarking purposes.

For each model and number of speculated tokens, SemiAnalysis chooses a fixed AL based on the observed MTP AL calibrated on SpeedBench coding dataset using model-provider-recommended sampling parameters (M0). For example, running DSV4-Pro on SpeedBench coding with 3 MTP draft tokens gives a mean AL of ~2.75, so the settings would be roughly [MTP1: AL 1.85, MTP2: AL 2.45, MTP3: AL 2.75] (mock numbers).

We propose the following methodology:

M0: New Github Action for Auditable Reference AL Generation

We introduce a new GitHub Action that integrates SpeedBench to InferenceX. For each model that supports MTP, we use the workflow to generate reference AL values, with the coding dataset, for both thinking on and thinking off. All reference values will be generated using vLLM for consistency, for speculative tokens going from 1 to 7.

M1: Evaluation runs for correct implementation of MTP

During evaluation the synthetic acceptance feature is disabled and the same model-provider-recommended sampling parameters are used. The eval script now includes two pieces:

  1. Run SpeedBench, acceptance rate is automatically collected from the server at the end of the run. Then: we assert that the achieved acceptance length is above a threshold from the reference values.
  2. Run the existing GSM8k evals to ensure accuracy still pass.

M2: Apply Synthetic Acceptance During Benchmarks

During benchmarking synthetic acceptance is enabled with the target AL and uses the same prompt generation setup as non-MTP. We will roll this out with single node first, followed by multi-node tests. To make sure the benchmarks are fair, we ensure implementation for synthetic MTP acceptance rates to be open, auditable, and mathematically equivalent for all participating FWs.

  • M0: New Github Action for Auditable Reference AL Generation
  • M1: Evaluation runs for correct implementation of MTP
  • M2: Apply Synthetic Acceptance During Benchmarks

中文说明

提议在 MTP 基准测试中引入合成接受长度（Synthetic Acceptance Length）机制。所有主流推理框架（vLLM、SGLang、TensorRT-LLM）现已支持配置合成接受长度，可按目标接受率匹配预期生成速度，确保跨框架基准测试环境的一致性。分三个里程碑推进：M0——新增 GitHub Action，使用 SpeedBench 编码数据集为每个支持 MTP 的模型生成可审计的参考接受长度值；M1——评估运行，禁用合成接受功能并断言实际接受长度达到阈值，同时运行 GSM8K 准确性评估；M2——在基准测试中启用合成接受，先单节点后多节点，确保各框架的实现公开、可审计且数学等价。

Ngôn ngữ chính
Python
Star
1.9k
Fork
343
Merge trung bình
2 ngày 40 phút
Pull request đã merge (30 ngày)
431

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của SemiAnalysisAI/InferenceX

Tất cả issue của SemiAnalysisAI/InferenceX

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.