[Bug] SenseNova-U1 E2E pixel-golden tests fail after think-mode generation drift

Summary

Weekly E2E (tests/e2e/offline_inference/test_sensenova_u1_*_expansion.py) fails at full_model / advanced_model when _validate_pixels() compares generated output against pinned REFERENCE_PIXELS.

Generation itself succeeds (correct resolution, non-empty image), but pixel pins no longer match.

vllm version: 0.23.0 vllm-omni version: 378359fc115a0bcd29a2f7bee93f6fa5afd49d3d

Reproduction

pytest -sv tests/e2e/offline_inference/test_sensenova_u1_text2img_expansion.py \
  --run-level full_model

Hardware: single H100

Sampling: seed=42, num_inference_steps=50, think=True, cfg_scale=4.0, cfg_norm=none, timestep_shift=3.0 (see test file).

Observed failure

AssertionError: Pixel mismatch at (768, 200): expected (176, 135, 97), got (106, 93, 68)

Image size is correct (1536x2720); only the golden-pixel assertion fails.

Likely causes

1. think=True path is not bit-exact: SenseNova-U1 runs autoregressive think decoding before diffusion. Even with a fixed diffusion seed, AR token sampling can vary across CUDA/driver/vLLM stack versions, changing downstream conditioning and the final image.
2. Stale REFERENCE_PIXELS: Golden values were captured from an older stack / commit. Recent pipeline changes (e.g. Cache-DiT adapter wiring on SenseNovaU1Model) may also shift numerics.
3. Mismatch between test intent and assertion: Weekly expansion jobs need a smoke check (model loads, image generated, expected resolution). Bit-exact pixel pins are closer to a determinism / accuracy regression test.

Affected tests

• tests/e2e/offline_inference/test_sensenova_u1_text2img_expansion.py::test_sensenova_u1_text2img
• tests/e2e/offline_inference/test_sensenova_u1_img2img_expansion.py::test_sensenova_u1_img2img (same _validate_pixels pattern; likely to flake similarly)

Proposed fix (pick one)

1. Regenerate REFERENCE_PIXELS on a pinned CI H100 image + document the capture commit/stack; re-run in merge/nightly only.
2. Split assertions: weekly smoke = generation + resolution only; move pixel pins to a dedicated accuracy job or advanced_model merge gate with refreshed goldens.
3. Improve determinism in think-mode decoding (greedy / fixed seed end-to-end) if bit-exact output is a product requirement.

Workaround

Skipping the pixel-validation cases in weekly CI until goldens are refreshed or the assertion strategy is updated.