help wantedwait-feedback
倉庫指標
- Star
- (2,180 star)
- PR 合併指標
- (平均合併 5天 16小時) (30 天內合併 419 個 PR)
描述
Anything you want to discuss about vllm on ascend.
Because of batch variance, We cannot guarantee that the same input will yield the same output in a multi-batch inference case. And this accuracy variance is more explicit in certain datasets.
| model | dataset | acc | acc variance |
|---|---|---|---|
| deepseek-v3.1 | GPQA | 74~82 | 8 |
| qwen3-235b | GPQA | 64~71 | 7 |
| qwen3-480b | GPQA | 60~67 | 7 |
We need to test these case on GPU or use other inference engine, such as sglang to check if this is an acc bug.
If GPU also has the same acc variance like us, we believe that this acc variance is reasonable for this dataset. Otherwise, we need to solve this acc bug.