Model-graded scorers should warn at runtime when no grader model or model-role is bound (silent self-grading)
#4,695 建立於 2026年7月30日
倉庫指標
- 星標
- (2,170 顆星)
- PR 合併指標
- (平均合併 3天 22小時) (30 天內合併 152 個 PR)
描述
When neither model nor a bound model_role is available, model-graded scorers silently fall back to the evaluated model — i.e. the model grades itself:
get_model(role="grader") falls through to the ambient/active model with no warning when the role isn't bound. Self-grading as a silent default is counterintuitive and not good science: self-grading bias varies by model, and it's easy to run a whole eval without realizing no grader was configured.
Suggestion: emit a warning at score time when the resolution falls through to the evaluated model (e.g. "no 'grader' model role bound; the evaluated model will grade itself"). This needs to be a runtime check rather than a construction-time one, since model roles can be bound later via eval(..., model_roles=...).
Posted by Claude Code on Matt's behalf.