[Bug]: Updating an eval case moves it to the end in Local/GCS eval sets

Đang mở
#7,220 0 bình luận 0 reaction 1 người được giao Xem trên GitHub

@sanketpatil06 đang làm issue này rồi.

Từ ngày 21/9/2026.

Đánh giá

Issue này chưa được đánh giá.

Mô tả

eval

🔴 Required Information

Describe the Bug:

Updating an existing eval case through LocalEvalSetsManager changes its position in EvalSet.eval_cases. GcsEvalSetsManager uses the same update helper, while InMemoryEvalSetsManager replaces the case at its existing position. The difference is visible through the public get_eval_set() result. The list also supplies case IDs to LocalEvalSampler. I am reporting the unexpected list reorder and backend inconsistency, not claiming a guaranteed change in evaluation completion order.

Steps to Reproduce:

  1. Install google-adk from commit 3f24d2036a3434b755f6337026ac19a737041f85 or use a recent checkout.
  2. Run the minimal code below. It creates an eval set, adds cases a, b, and c, and updates only b through each manager's public API.
  3. Compare get_eval_set(...).eval_cases after the update.

Expected Behavior:

Updating case b replaces its contents without moving it: both managers return ['a', 'b', 'c'].

Observed Behavior:

InMemory: ['a', 'b', 'c']
Local:    ['a', 'c', 'b']

Environment Details:

  • ADK Library Version (pip show google-adk): editable checkout at 3f24d2036a3434b755f6337026ac19a737041f85, google-adk 2.9.0. The same remove-and-append implementation was still present on the GitHub default branch when checked on 2026-09-21.
  • Desktop OS: Windows 11, build 26200.
  • Python Version (python -V): Python 3.12.0.

Model Information:

  • Are you using LiteLLM: No.
  • Which model is being used: N/A. This occurs in eval-set storage without a model call.

🟡 Optional Information

Regression:

Unknown. I reproduced the behavior at the commit above and confirmed the same helper implementation on the current GitHub default branch. I have not bisected its introduction.

Logs:

No exception is raised. The output under Observed Behavior is the complete relevant result.

Screenshots / Video:

N/A.

Additional Context:

update_eval_case_in_eval_set() removes the existing case and appends the replacement. Both the local and GCS managers call this helper. The in-memory manager instead replaces the matching list element in place. The EvalSetsManager interface does not explicitly promise order stability, but the list order is observable, and the existing delete-case tests assert that other cases keep their order.

I searched repository Issues and PRs for update_eval_case with order, eval_cases with order, and eval case with position; I did not find a matching report or fix as of 2026-09-21. A focused local fix and regression test are ready if this behavior should be made consistent.

Minimal Reproduction Code:

from tempfile import TemporaryDirectory

from google.adk.evaluation.eval_case import EvalCase
from google.adk.evaluation.in_memory_eval_sets_manager import InMemoryEvalSetsManager
from google.adk.evaluation.local_eval_sets_manager import LocalEvalSetsManager


def run(manager):
    manager.create_eval_set("app", "set")
    for case_id in ("a", "b", "c"):
        manager.add_eval_case(
            "app", "set", EvalCase(eval_id=case_id, conversation=[])
        )
    manager.update_eval_case(
        "app", "set", EvalCase(eval_id="b", conversation=[], creation_timestamp=99)
    )
    return [case.eval_id for case in manager.get_eval_set("app", "set").eval_cases]


with TemporaryDirectory() as directory:
    print("InMemory:", run(InMemoryEvalSetsManager()))
    print("Local:   ", run(LocalEvalSetsManager(directory)))

How often has this issue occurred?:

  • Always (100%) when updating a non-final case in a local eval set. The GCS manager uses the same helper, but this reproducer does not contact GCS.
Ngôn ngữ chính
Python
Star
21.6k
Fork
4k
Merge trung bình
13 giờ 49 phút
Pull request đã merge (30 ngày)
10

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của google/adk-python

Tất cả issue của google/adk-python

Issue tương tự

Thêm issue về Python

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.