Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

[Bug]: LocalEvalSampler treats explicit empty eval case ID lists as all cases

Đang mở
#7,168 5 bình luận 0 reaction 1 người được giao Xem trên GitHub

@surajksharma07 đang làm issue này rồi.

Từ ngày 18/9/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

eval request clarification

🔴 Required Information

Describe the Bug:

LocalEvalSamplerConfig documents train_eval_case_ids and validation_eval_case_ids as optional lists where all cases are selected when a field is not provided. However, LocalEvalSampler.__init__ uses truthiness checks, so explicitly passing an empty list is indistinguishable from passing None.

As a result, train_eval_case_ids=[] expands to every training case, and validation_eval_case_ids=[] expands to every validation case (or inherits the training cases). This can unexpectedly turn a no-case selection into a full evaluation run.

Steps to Reproduce:

  1. Install the current google-adk repository in an isolated environment.
  2. Construct LocalEvalSamplerConfig with explicit empty training and validation case ID lists.
  3. Construct LocalEvalSampler while returning known IDs from _get_eval_case_ids.
  4. Inspect get_train_example_ids() and get_validation_example_ids().

Minimal code is included below.

Expected Behavior:

Only None / an omitted field should trigger the documented “all eval cases” fallback. An explicit empty list should either remain empty, consistent with sample_and_score(batch=[]), or be rejected with a clear validation error if empty datasets are unsupported. It should not silently expand to all cases.

Observed Behavior:

The explicit empty lists are replaced with every ID returned for their respective eval sets:

configured train: []
actual train: ['train-1', 'train-2']
configured validation: []
actual validation: ['validation-1']
lookup calls: [('train',), ('validation',)]

Environment Details:

  • ADK Library Version: editable install from commit 3f24d2036a3434b755f6337026ac19a737041f85 (google-adk 2.9.0)
  • Desktop OS: Windows 10.0.26200
  • Python Version: Python 3.12 (64-bit)

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: N/A — the issue occurs during sampler construction before any model call

🟡 Optional Information

Regression:

Unknown. The truthiness behavior is present in the commit that originally introduced LocalEvalSampler.

Logs:

N/A — no external service or model call is required.

Screenshots / Video:

N/A.

Additional Context:

The constructor currently uses:

self._config.train_eval_case_ids or self._get_eval_case_ids(...)

and:

if self._config.validation_eval_case_ids:

Both checks collapse [] and None even though the Pydantic model and field descriptions distinguish them. The existing parameterized constructor test covers omitted and non-empty lists, but not explicit empty lists.

I searched the repository issues and pull requests for train_eval_case_ids and validation_eval_case_ids and found no existing report or implementation. I would be happy to contribute a focused fix and regression tests once the intended empty-list behavior is confirmed.

Minimal Reproduction Code:

from unittest.mock import MagicMock, patch

from google.adk.evaluation.eval_config import EvalConfig
from google.adk.evaluation.eval_sets_manager import EvalSetsManager
from google.adk.optimization.local_eval_sampler import LocalEvalSampler
from google.adk.optimization.local_eval_sampler import LocalEvalSamplerConfig

config = LocalEvalSamplerConfig(
    eval_config=EvalConfig(),
    app_name="app",
    train_eval_set="train",
    train_eval_case_ids=[],
    validation_eval_set="validation",
    validation_eval_case_ids=[],
)

with patch.object(
    LocalEvalSampler,
    "_get_eval_case_ids",
    side_effect=lambda eval_set_id: (
        ["train-1", "train-2"]
        if eval_set_id == "train"
        else ["validation-1"]
    ),
):
  sampler = LocalEvalSampler(config, MagicMock(spec=EvalSetsManager))

print("configured train:", config.train_eval_case_ids)
print("actual train:", sampler.get_train_example_ids())
print("configured validation:", config.validation_eval_case_ids)
print("actual validation:", sampler.get_validation_example_ids())

How often has this issue occurred?:

  • Always (100%)
Ngôn ngữ chính
Python
Star
21.6k
Fork
4k
Merge trung bình
13 giờ 49 phút
Pull request đã merge (30 ngày)
10

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của google/adk-python

Tất cả issue của google/adk-python

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.