hallucinations_v1: sentence validator ignores grounding_metadata
@surajksharma07 ci sta già lavorando.
Dal 17/9/2026.
Valutazione
Questa issue non è ancora stata valutata.
Descrizione
🔴 Required Information
Describe the Bug:
HallucinationsV1Evaluator (google/adk/evaluation/hallucinations_v1.py) never includes model-supplied grounding metadata (InvocationEvent.grounding_metadata) in the context it builds for the sentence-validator judge. _create_context_for_step() only assembles developer instructions, user prompt, tool declarations, and per-event function_call/function_response/text parts — it never calls the existing get_grounding_metadata_as_json_str() helper (defined in llm_as_judge_utils.py) the way the sibling rubric_based_final_response_quality_v1.py metric does. For agents that use built-in/model-internal grounding tools (e.g. VertexAiSearchTool, google_search), search results only ever arrive as groundingMetadata on the event, never as a function_response part. As a result, sentences that are genuinely grounded by search have no supporting evidence in the judge's <context>, and the judge labels them unsupported, inflating the hallucination rate for any search-grounded agent.
Steps to Reproduce:
- Build an ADK agent whose only retrieval mechanism is a model-internal grounding tool, e.g.
VertexAiSearchTool(no explicitfunction_call/function_responsein the event stream, onlyevent.grounding_metadata). - Run an eval with
metrics_to_run: [hallucination](HallucinationsCriterion,hallucinations_v1.py) against a case where the final response correctly cites/uses retrieved document content. - Inspect the per-sentence validator output (or just the resulting score) for the invocation.
- Compare
hallucinations_v1.py::_create_context_for_stepagainstrubric_based_final_response_quality_v1.py::_build_auto_rater_prompt, which imports and injectsget_grounding_metadata_as_json_str(...)—hallucinations_v1.pyhas no equivalent import/usage.
Expected Behavior:
Sentences that are supported by the retrieved grounding chunks (search results attached via event.grounding_metadata) should be scored supported, since the information genuinely came from a trusted retrieval source.
Observed Behavior:
Sentences grounded only via grounding_metadata (no matching function_call/function_response part) are scored unsupported/hallucinated by the sentence validator, because the context string passed to the judge never contains the grounding chunks/search results — only explicit tool call/response JSON. This drags down the metric's Accuracy Score for any agent that relies on model-internal search grounding rather than explicit function-tool calls.
Environment Details:
- ADK Library Version (
pip show google-adk): 2.9.1 - Desktop OS: macOS (Darwin 24.6.0, arm64)
- Python Version (
python -V): 3.12.14
Model Information:
- Are you using LiteLLM: No
- Which model is being used:
gemini-3.1-flash-lite(agent model, viaGemini(model=GEMINI_MODEL)); judge model for thehallucinationcriterion uses the ADK default (not overridden in this project's eval config)
🟡 Optional Information
Regression:
N/A — this appears to be a design gap since the metric's introduction, not a regression. rubric_based_final_response_quality_v1.py already handles grounding metadata correctly, so the pattern to fix it exists in the same codebase.
Logs:
N/A — behavioral/code-path issue, not a crash. Confirmed via static review of
google/adk/evaluation/hallucinations_v1.py (context builder, lines ~309-494)
vs. google/adk/evaluation/rubric_based_final_response_quality_v1.py (lines ~28,
50, 308-347) and google/adk/evaluation/llm_as_judge_utils.py
(get_grounding_metadata_as_json_str, lines ~209-240).
Screenshots / Video:
N/A
Additional Context:
We independently worked around an analogous problem for a different metric in our own eval config (tests/eval/eval_config.yaml, search_evidence custom metric): VertexAiSearchTool only ever adds grounding_metadata, never a function_call event, so adaptive-rubric judges without grounding-metadata support routinely mark search/grounding criteria as unverifiable even when a search genuinely happened. The same root cause applies here to hallucinations_v1.py.
This is the same class of bug already fixed for a sibling metric: PR #5834 ("fix: include grounding metadata in rubric judge prompt", merged 2026-08-14, resolving #5831) added exactly this — grounding-metadata plumbing into the judge prompt — but only for rubric_based_final_response_quality_v1.py (touching eval_case.py, evaluation_generator.py, llm_as_judge_utils.py, and that file). hallucinations_v1.py was not updated in that PR and still has the gap described above. Suggested fix: apply the same pattern PR #5834 used — call get_grounding_metadata_as_json_str(...) in _create_context_for_step and add validator-prompt instructions analogous to the rubric-based metric's: "model-supplied grounding metadata is trusted evidence for model-internal tools such as google_search/VertexAiSearchTool whose raw search results may not appear as function tool responses." Note PR #5834's own stated limitation ("final answer text is still not treated as evidence") likely applies equally here and is worth calling out if filing upstream.
Minimal Reproduction Code:
# No standalone repro needed — the gap is visible by diffing the two files:
# google/adk/evaluation/hallucinations_v1.py::_create_context_for_step
# google/adk/evaluation/rubric_based_final_response_quality_v1.py (imports
# get_grounding_metadata_as_json_str and injects a <grounding_metadata> block;
# hallucinations_v1.py does neither).
How often has this issue occurred?:
- Always (100%) — for any invocation where a sentence's only support is grounding metadata rather than an explicit tool call/response.
- Lingua principale
- Python
- Stelle
- 21.6k
- Fork
- 4k
- Merge medio
- 13h 49m
- PR unite (30g)
- 10
Guida per i contributori
Apri la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di google/adk-python
-
mcp
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
google/adk-python#7217 · 2 commenti · 1 assegnatario ·
-
tools
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
google/adk-python#7206 · 1 commento · 1 assegnatario ·
-
tools
Difficoltà 2/5 1-3 ore Idoneità per principianti 86/100
google/adk-python#7205 · 1 commento · 1 assegnatario ·
-
mcp
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
google/adk-python#7196 · 1 commento · 1 assegnatario ·
-
eval request clarification
Difficoltà 1/5 1-3 ore Idoneità per principianti 86/100
google/adk-python#7146 · 2 commenti · 1 assegnatario ·
Tutte le issue di google/adk-python
Issue simili
-
bug
Difficoltà 2/5 1-3 ore Idoneità per principianti 82/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
use-agent-os/agent-os#3314 ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
BasedHardware/omi#15662 · 1 commento ·
-
documentation help wanted
Difficoltà 2/5 1-3 ore Idoneità per principianti 90/100
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 62/100
AiursoftWeb/AnduinOS-2#19 ·