LLM-as-judge default config triggers google-genai's AFC warning in every eval run
@surajksharma07 已经在做这个了。
开始于 2026年9月17日。
评估
- 难度
- 1/5
- 预计耗时
- 1-3 小时
- 新手友好度
- 86/100
- Issue 类型
- 缺陷
- 描述清晰度
- 描述清楚
- 活跃度
- 活跃
- 技术栈
- python
- 领域
- testing-qa
调研方向
从 src/google/adk/evaluation/llm_as_judge.py:224-225 开始,这里是 LlmAsJudge 为 judge 请求创建默认 GenerateContentConfig 的位置。验证默认设置会禁用自动 function calling,且不会更改用户提供的 judge_model_config,然后运行相关的评估检查,并确认 google-genai 的 AFC 警告不再出现。
由索引模型根据 Issue 内容生成。
描述
Summary
Every rubric-based eval run logs this google-genai warning once per process:
WARNING google_genai.models: Direct use of automatic function calling (AFC) in AsyncModels.generate_content is not recommended. Instead, we recommend to use AFC in AsyncChat.send_message. Similarly, direct use of AFC in AsyncModels.generate_content_stream is not recommended. ...
The agent under evaluation does not cause it. It comes from the LLM-as-judge request.
Where it comes from
LlmAsJudge builds the judge request with config=self._judge_model_options.judge_model_config or genai_types.GenerateContentConfig() (src/google/adk/evaluation/llm_as_judge.py:224-225 on main at ce53a36c0a, and the same in 2.9.0 and 2.9.1).
That default config sets neither tools nor automatic_function_calling. google-genai's AsyncModels.generate_content therefore takes its AFC branch and logs the warning:
_extra_utils.should_disable_afcreturnsFalsewhenautomatic_function_callingis unset.- With no tools there are no AFC-incompatible tool indexes, so the direct
_generate_contentpath is skipped.
An agent's own model calls do not hit this. ADK sends tools as function_declarations, which google-genai marks AFC-incompatible, so those calls go straight to _generate_content.
How it was observed
In an eval suite running rubric_based_*_quality_v1 metrics with gemini-3.5-flash as judge (google-adk 2.9.0, google-genai as resolved by it):
- The warning appears once per pytest process.
- It always appears right after the agent's inference for the case ends and right before the rubric verdicts.
- It never appears during the agent's own model calls.
- The deployed agent's logs, same code without the eval harness, contain no occurrence over 7 days.
Suggestion
The judge never needs automatic function calling. Its default config could disable it explicitly:
config=self._judge_model_options.judge_model_config
or genai_types.GenerateContentConfig(
automatic_function_calling=genai_types.AutomaticFunctionCallingConfig(disable=True)
),
This does not change the request sent to the model (the field is client-side only) and removes a warning that points users at their own agent. Users who pass judge_model_config are unaffected.
- 主要语言
- Python
- 星标
- 21.6k
- 派生
- 4k
- 平均合并
- 13 小时 49 分钟
- 30 天内合并 PR
- 10
贡献指南
从这里开始
- 先读完整个 Issue,再读项目的贡献指南。
- 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
- Fork 仓库,在一个分支上完成修改。
- 提交 Pull Request,并在描述里引用这个 Issue 编号。
google/adk-python 的其他 Issue
-
mcp
难度 2/5 1-3 小时 新手友好度 75/100
google/adk-python#7217 · 2 条评论 · 已指派 1 人 ·
-
tools
难度 2/5 1-3 小时 新手友好度 78/100
google/adk-python#7206 · 1 条评论 · 已指派 1 人 ·
-
tools
难度 2/5 1-3 小时 新手友好度 86/100
google/adk-python#7205 · 1 条评论 · 已指派 1 人 ·
-
mcp
难度 2/5 1-3 小时 新手友好度 78/100
google/adk-python#7196 · 1 条评论 · 已指派 1 人 ·
-
mcp
难度 2/5 1-3 小时 新手友好度 88/100
google/adk-python#7144 · 1 条评论 · 已指派 1 人 ·
查看 google/adk-python 的全部 Issue
相似的 Issue
-
难度 2/5 1-3 小时 新手友好度 88/100
-
难度 2/5 1-3 小时 新手友好度 82/100
-
难度 2/5 1-3 小时 新手友好度 78/100
-
enhancement
难度 2/5 1-3 小时 新手友好度 72/100
-
难度 2/5 1-3 小时 新手友好度 74/100