UKGovernmentBEIS/inspect_ai

Model-graded scorers should warn at runtime when no grader model or model-role is bound (silent self-grading)

已关闭

#4,695 创建于 2026年7月30日

 (6 条评论) (0 个反应) (0 位负责人)Python (550 个派生)github user discovery
acceptedgood first issue

仓库指标

星标
 (2,170 个星标)
PR 合并指标
 (平均合并 3天 22小时) (30 天内合并 152 个 PR)

描述

When neither model nor a bound model_role is available, model-graded scorers silently fall back to the evaluated model — i.e. the model grades itself:

https://github.com/UKGovernmentBEIS/inspect_ai/blob/797362280ba84f79009d22db78c9b9a5ce48979e/src/inspect_ai/scorer/_model.py#L173-L182

get_model(role="grader") falls through to the ambient/active model with no warning when the role isn't bound. Self-grading as a silent default is counterintuitive and not good science: self-grading bias varies by model, and it's easy to run a whole eval without realizing no grader was configured.

Suggestion: emit a warning at score time when the resolution falls through to the evaluated model (e.g. "no 'grader' model role bound; the evaluated model will grade itself"). This needs to be a runtime check rather than a construction-time one, since model roles can be bound later via eval(..., model_roles=...).

Posted by Claude Code on Matt's behalf.

贡献者指南