[Bug]: MiniCPM-o 4.5 Daily-Omni accuracy ~66.75% vs paper/HF ~80.2%

Summary

When evaluating MiniCPM-o 4.5 with vLLM-Omni’s built-in Daily-Omni serve benchmark (vllm bench serve --dataset-name daily-omni), we get:

Source	Daily-Omni overall
Paper / model card (arXiv:2604.27393, HF openbmb/MiniCPM-o-4_5)	80.2% (MiniCPM-o 4.5-Instruct)
vLLM-Omni online bench (this run)	66.75% (799/1197)

That is an absolute gap of ~13.5 points, with 0 HTTP failures and 0 unparseable A–D answers, so this is not a transport / parsing artifact.

Environment

Item	Value
Model	openbmb/MiniCPM-o-4_5 (local checkpoint)
vLLM	0.25.0
vLLM-Omni	0.25.0rc2.dev47+g10d3a2591
Endpoint	POST /v1/chat/completions (openai-chat-omni)
Deploy	vllm_omni/deploy/minicpmo_4_5.yaml (stage0+stage1 co-located)
Serve flags	--omni --trust-remote-code
Dataset	local Daily-Omni qa.json + Videos/ (1197 items, input_mode=all)
Sampling	temperature=0, output_len=512, modalities=["text"], enable_thinking=false
Concurrency	--max-concurrency 10

Reproduce

1. Serve

vllm serve /path/to/MiniCPM-o-4_5 --omni --port 28889 \
  --trust-remote-code \
  --deploy-config vllm_omni/deploy/minicpmo_4_5.yaml

2. Bench

vllm bench serve \
  --omni \
  --port 28889 \
  --max-concurrency 10 \
  --dataset-name daily-omni \
  --daily-omni-inline-local-video \
  --num-prompts 2000 \
  --trust-remote-code \
  --no-oversample \
  --temperature 0 \
  --output-len 512 \
  --daily-omni-input-mode all \
  --daily-omni-video-dir /path/to/Daily-Omni/Videos \
  --daily-omni-qa-json /path/to/Daily-Omni/qa.json \
  --model /path/to/MiniCPM-o-4_5 \
  --endpoint /v1/chat/completions \
  --backend openai-chat-omni \
  --extra_body '{"modalities": ["text"], "chat_template_kwargs": {"enable_thinking": false}}'

3. Observe

=========== Daily-Omni accuracy (MCQ) ============
Overall Accuracy: 799/1197 = 66.75%
Submitted (gold present):                1197
Successful HTTP (GitHub denom.):         1197
Correct:                                 799
Parsed OK but no A–D found:              0
HTTP failed (excl. from GitHub acc.):    0

Breakdown (this run)

By QA type (weakest first):

QA type	Accuracy
Event Sequence	175/306 = 57.19%
AV Event Alignment	142/238 = 59.66%
Context understanding	123/193 = 63.73%
Comparative	100/131 = 76.34%
Reasoning	135/175 = 77.14%
Inference	124/154 = 80.52%

By duration:

Duration	Accuracy
30s	441/647 = 68.16%
60s	358/550 = 65.09%

Temporal / AV-alignment categories are clearly worse than overall, which often points at video+audio packing / time alignment / preprocess differences vs the official eval recipe—not random MCQ noise.

Actual vs Expected

	Result
Actual	Full Daily-Omni (1197 items) → 66.75% under vLLM-Omni serve
Expected	Align with reported ~80.2% (or document a known, justified gap if eval recipe differs)

Notes / prior context

• Earlier incomplete / misconfigured runs were much lower (~33%); after recent MiniCPM-o 4.5 audio / packing related fixes we are at 66.75%, but still far from 80.2%.
• Serve already uses --interleave-mm-strings (required to avoid front-padding all images then all audios and breaking AV time alignment).
• Bench uses the default Daily-Omni system prompt text currently wired in daily_omni_dataset.py (Qwen-style Daily-Omni system string) and input_mode=all.

Suspected areas (for investigation)

Not claiming root cause yet; likely candidates:

1. Eval recipe mismatch vs OpenBMB / paper setup
   (frame sampling, audio extraction, prompt template, answer extraction, thinking / system prompt, max frames, resolution, etc.).
2. AV packing / interleave in the OpenAI-chat path still not matching official MiniCPM-o interleaved media layout for Daily-Omni.
3. Video/audio preprocess differences (fps, duration clipping, Whisper chunking / pool_step, placeholder counts).
4. Serving-only path differences vs HF / official offline eval (chat template, multimodal processor, sampling defaults).

Suggested checks

• Diff vLLM-Omni Daily-Omni request construction against OpenBMB / Daily-Omni official MiniCPM-o eval scripts (prompt + media order + answer parse).
• A/B: HF / official offline Daily-Omni on the same checkpoint vs vLLM-Omni online on identical items.
• Ablations: input_mode=visual / audio / all; concurrency 1 vs 10; with/without --interleave-mm-strings.
• Dump a few failing AV Event Alignment / Event Sequence samples (gold vs pred + request payload) to see whether the model is missing audio, video, or temporal cues.

Impact

• Severity: High for claiming paper-level MiniCPM-o 4.5 omni understanding quality under vLLM-Omni.
• Scope: Observed on MiniCPM-o 4.5 + Daily-Omni online serve bench. Unclear how much is benchmark plumbing vs model/runtime regression.