e2e eval harness silently ignores `*_fp32_config.json` recipes (fp32 not in KNOWN_PRECISIONS) — affects #1084, #1093

Inizia leggendo KNOWN_PRECISIONS, split_task_precision() e discover_recipe_variants() in scripts/e2e_eval/utils/recipes.py, quindi esamina i file interessati in examples/recipes/. Esegui i test e2e pertinenti in tests/e2e/test_config_e2e.py e verifica che le ricette interessate scritte dagli autori vengano individuate senza modificarne il contenuto di quantizzazione.

Summary

The e2e eval harness discovers authored recipes by a precision token parsed from the filename. In scripts/e2e_eval/utils/recipes.py:

KNOWN_PRECISIONS: tuple[str, ...] = ("fp16", "w8a16", "w8a8")

discover_recipe_variants() → split_task_precision() only strips those suffixes; any other trailing token (e.g. fp32) yields precision=None, and the file is dropped in discover_recipe_variants (if cfg_task != task or precision is None: continue).

Consequence: a recipe named *_fp32_config.json is never recognized as a precision variant. If a model dir contains only fp32-named recipes, discover_recipe_variants returns empty and the harness silently falls back to winml config auto-generation — the authored recipe is never built or evaluated.

Why this is a naming trap

The catalog convention is that the unquantized bucket is named *_fp16_config.json with quant: null:

• 75 / 75 *_fp16_config.json recipes under examples/recipes/ have quant: null; 0 carry quant.mode=fp16.
• fp16 is realized per-EP at eval time via --precision fp16 (which resolves to quant.mode=fp16), not baked into the recipe. _run_recipe_build in run_eval.py builds authored recipes recipe-driven (winml build -c <recipe>), with no --precision/--device flag.
• On CPU the unquantized recipe deliberately stays fp32 — see tests/e2e/test_config_e2e.py: "device=cpu + precision=auto must NOT trigger FP16 conversion."

So a quant: null recipe named _fp16_ is correct and intended. The same content renamed _fp32_ falls outside the harness's recognized precisions and is dropped.

Impact (two open PRs)

Contrast (correct): usyd-community/vitpose-plus-base ships keypoint-detection_fp16_config.json + keypoint-detection_w8a8_config.json — both discovered.

Suggested fix

Rename the *_fp32_config.json recipes in #1084 and #1093 back to *_fp16_config.json. Their quant: null content already matches the fp16-bucket convention, so the rename makes them discoverable and consistent with the other 75 recipes — no content change needed.

Alternatively, if fp32-named recipes are meant to be first-class, add fp32 to KNOWN_PRECISIONS and the harness precision-resolution. The rename is the lower-risk, convention-aligned option.

References

• scripts/e2e_eval/utils/recipes.py — KNOWN_PRECISIONS, split_task_precision, discover_recipe_variants
• scripts/e2e_eval/run_eval.py — _run_recipe_build (recipe-driven build, no --precision)
• tests/e2e/test_config_e2e.py — CPU stays fp32 by design