LLM-as-judge default config triggers google-genai's AFC warning in every eval run

Open Beginner friendly
#7,146 2 comments 0 reactions 1 assignee View on GitHub

@surajksharma07 is already working on this.

Since Sep 17, 2026.

Assessment

Difficulty
1/5
Estimated time
1-3 hours
Newbie friendliness
86/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
testing-qa

Research direction

Start at src/google/adk/evaluation/llm_as_judge.py:224-225, where LlmAsJudge creates the default GenerateContentConfig for judge requests. Verify the default disables automatic function calling without changing user-supplied judge_model_config, then run the relevant evaluation checks and confirm the google-genai AFC warning no longer appears.

Written by the indexing model from the issue text.

Description

eval request clarification

Summary

Every rubric-based eval run logs this google-genai warning once per process:

WARNING google_genai.models: Direct use of automatic function calling (AFC) in AsyncModels.generate_content is not recommended. Instead, we recommend to use AFC in AsyncChat.send_message. Similarly, direct use of AFC in AsyncModels.generate_content_stream is not recommended. ...

The agent under evaluation does not cause it. It comes from the LLM-as-judge request.

Where it comes from

LlmAsJudge builds the judge request with config=self._judge_model_options.judge_model_config or genai_types.GenerateContentConfig() (src/google/adk/evaluation/llm_as_judge.py:224-225 on main at ce53a36c0a, and the same in 2.9.0 and 2.9.1).

That default config sets neither tools nor automatic_function_calling. google-genai's AsyncModels.generate_content therefore takes its AFC branch and logs the warning:

  • _extra_utils.should_disable_afc returns False when automatic_function_calling is unset.
  • With no tools there are no AFC-incompatible tool indexes, so the direct _generate_content path is skipped.

An agent's own model calls do not hit this. ADK sends tools as function_declarations, which google-genai marks AFC-incompatible, so those calls go straight to _generate_content.

How it was observed

In an eval suite running rubric_based_*_quality_v1 metrics with gemini-3.5-flash as judge (google-adk 2.9.0, google-genai as resolved by it):

  • The warning appears once per pytest process.
  • It always appears right after the agent's inference for the case ends and right before the rubric verdicts.
  • It never appears during the agent's own model calls.
  • The deployed agent's logs, same code without the eval harness, contain no occurrence over 7 days.

Suggestion

The judge never needs automatic function calling. Its default config could disable it explicitly:

config=self._judge_model_options.judge_model_config
or genai_types.GenerateContentConfig(
    automatic_function_calling=genai_types.AutomaticFunctionCallingConfig(disable=True)
),

This does not change the request sent to the model (the field is client-side only) and removes a warning that points users at their own agent. Users who pass judge_model_config are unaffected.

Dominant language
Python
Stars
21.6k
Forks
4k
Avg merge
13h 49m
Merged PRs (30d)
10

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from google/adk-python

All issues in google/adk-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.