[Bug]: LocalEvalSampler treats explicit empty eval case ID lists as all cases
@surajksharma07 ci sta già lavorando.
Dal 18/9/2026.
Valutazione
Questa issue non è ancora stata valutata.
Descrizione
🔴 Required Information
Describe the Bug:
LocalEvalSamplerConfig documents train_eval_case_ids and validation_eval_case_ids as optional lists where all cases are selected when a field is not provided. However, LocalEvalSampler.__init__ uses truthiness checks, so explicitly passing an empty list is indistinguishable from passing None.
As a result, train_eval_case_ids=[] expands to every training case, and validation_eval_case_ids=[] expands to every validation case (or inherits the training cases). This can unexpectedly turn a no-case selection into a full evaluation run.
Steps to Reproduce:
- Install the current
google-adkrepository in an isolated environment. - Construct
LocalEvalSamplerConfigwith explicit empty training and validation case ID lists. - Construct
LocalEvalSamplerwhile returning known IDs from_get_eval_case_ids. - Inspect
get_train_example_ids()andget_validation_example_ids().
Minimal code is included below.
Expected Behavior:
Only None / an omitted field should trigger the documented “all eval cases” fallback. An explicit empty list should either remain empty, consistent with sample_and_score(batch=[]), or be rejected with a clear validation error if empty datasets are unsupported. It should not silently expand to all cases.
Observed Behavior:
The explicit empty lists are replaced with every ID returned for their respective eval sets:
configured train: []
actual train: ['train-1', 'train-2']
configured validation: []
actual validation: ['validation-1']
lookup calls: [('train',), ('validation',)]
Environment Details:
- ADK Library Version: editable install from commit
3f24d2036a3434b755f6337026ac19a737041f85(google-adk 2.9.0) - Desktop OS: Windows 10.0.26200
- Python Version: Python 3.12 (64-bit)
Model Information:
- Are you using LiteLLM: No
- Which model is being used: N/A — the issue occurs during sampler construction before any model call
🟡 Optional Information
Regression:
Unknown. The truthiness behavior is present in the commit that originally introduced LocalEvalSampler.
Logs:
N/A — no external service or model call is required.
Screenshots / Video:
N/A.
Additional Context:
The constructor currently uses:
self._config.train_eval_case_ids or self._get_eval_case_ids(...)
and:
if self._config.validation_eval_case_ids:
Both checks collapse [] and None even though the Pydantic model and field descriptions distinguish them. The existing parameterized constructor test covers omitted and non-empty lists, but not explicit empty lists.
I searched the repository issues and pull requests for train_eval_case_ids and validation_eval_case_ids and found no existing report or implementation. I would be happy to contribute a focused fix and regression tests once the intended empty-list behavior is confirmed.
Minimal Reproduction Code:
from unittest.mock import MagicMock, patch
from google.adk.evaluation.eval_config import EvalConfig
from google.adk.evaluation.eval_sets_manager import EvalSetsManager
from google.adk.optimization.local_eval_sampler import LocalEvalSampler
from google.adk.optimization.local_eval_sampler import LocalEvalSamplerConfig
config = LocalEvalSamplerConfig(
eval_config=EvalConfig(),
app_name="app",
train_eval_set="train",
train_eval_case_ids=[],
validation_eval_set="validation",
validation_eval_case_ids=[],
)
with patch.object(
LocalEvalSampler,
"_get_eval_case_ids",
side_effect=lambda eval_set_id: (
["train-1", "train-2"]
if eval_set_id == "train"
else ["validation-1"]
),
):
sampler = LocalEvalSampler(config, MagicMock(spec=EvalSetsManager))
print("configured train:", config.train_eval_case_ids)
print("actual train:", sampler.get_train_example_ids())
print("configured validation:", config.validation_eval_case_ids)
print("actual validation:", sampler.get_validation_example_ids())
How often has this issue occurred?:
- Always (100%)
- Lingua principale
- Python
- Stelle
- 21.6k
- Fork
- 4k
- Merge medio
- 7h 10m
- PR unite (30g)
- 7
Guida per i contributori
Apri la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di google/adk-python
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
google/adk-python#7266 · 1 commento ·
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
google/adk-python#7265 · 1 commento ·
-
mcp
Difficoltà 2/5 1-3 ore Idoneità per principianti 75/100
google/adk-python#7217 · 3 commenti · 1 assegnatario ·
-
tools
Difficoltà 2/5 1-3 ore Idoneità per principianti 78/100
google/adk-python#7206 · 1 commento · 1 assegnatario ·
-
request clarification tools
Difficoltà 2/5 1-3 ore Idoneità per principianti 86/100
google/adk-python#7205 · 2 commenti · 1 assegnatario ·
Tutte le issue di google/adk-python
Issue simili
-
agent-ready documentation needs-triage
Difficoltà 1/5 1-3 ore Idoneità per principianti 88/100
-
documentation
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 91/100
-
workflow-status page template still says reusable workflows are "triggered only by workflow_call:" Aperta
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 92/100
-
instance instance add
Difficoltà 1/5 Meno di un'ora Idoneità per principianti 72/100
searxng/searx-instances#939 · 1 commento ·
-
area-deployment area-integrations triage:bot-seen
Difficoltà 2/5 Mezza giornata Idoneità per principianti 86/100