help wantedstalewait-feedback
仓库指标
- 星标
- (2,637 个星标)
- PR 合并指标
- (平均合并 4天 5小时) (30 天内合并 559 个 PR)
描述
Anything you want to discuss about vllm on ascend.
Because of batch variance, We cannot guarantee that the same input will yield the same output in a multi-batch inference case. And this accuracy variance is more explicit in certain datasets.
| model | dataset | acc | acc variance |
|---|---|---|---|
| deepseek-v3.1 | GPQA | 74~82 | 8 |
| qwen3-235b | GPQA | 64~71 | 7 |
| qwen3-480b | GPQA | 60~67 | 7 |
We need to test these case on GPU or use other inference engine, such as sglang to check if this is an acc bug.
If GPU also has the same acc variance like us, we believe that this acc variance is reasonable for this dataset. Otherwise, we need to solve this acc bug.