[Bug]: Updating an eval case moves it to the end in Local/GCS eval sets

Aperta
#7,220 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

@sanketpatil06 ci sta già lavorando.

Dal 21/9/2026.

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

eval

🔴 Required Information

Describe the Bug:

Updating an existing eval case through LocalEvalSetsManager changes its position in EvalSet.eval_cases. GcsEvalSetsManager uses the same update helper, while InMemoryEvalSetsManager replaces the case at its existing position. The difference is visible through the public get_eval_set() result. The list also supplies case IDs to LocalEvalSampler. I am reporting the unexpected list reorder and backend inconsistency, not claiming a guaranteed change in evaluation completion order.

Steps to Reproduce:

  1. Install google-adk from commit 3f24d2036a3434b755f6337026ac19a737041f85 or use a recent checkout.
  2. Run the minimal code below. It creates an eval set, adds cases a, b, and c, and updates only b through each manager's public API.
  3. Compare get_eval_set(...).eval_cases after the update.

Expected Behavior:

Updating case b replaces its contents without moving it: both managers return ['a', 'b', 'c'].

Observed Behavior:

InMemory: ['a', 'b', 'c']
Local:    ['a', 'c', 'b']

Environment Details:

  • ADK Library Version (pip show google-adk): editable checkout at 3f24d2036a3434b755f6337026ac19a737041f85, google-adk 2.9.0. The same remove-and-append implementation was still present on the GitHub default branch when checked on 2026-09-21.
  • Desktop OS: Windows 11, build 26200.
  • Python Version (python -V): Python 3.12.0.

Model Information:

  • Are you using LiteLLM: No.
  • Which model is being used: N/A. This occurs in eval-set storage without a model call.

🟡 Optional Information

Regression:

Unknown. I reproduced the behavior at the commit above and confirmed the same helper implementation on the current GitHub default branch. I have not bisected its introduction.

Logs:

No exception is raised. The output under Observed Behavior is the complete relevant result.

Screenshots / Video:

N/A.

Additional Context:

update_eval_case_in_eval_set() removes the existing case and appends the replacement. Both the local and GCS managers call this helper. The in-memory manager instead replaces the matching list element in place. The EvalSetsManager interface does not explicitly promise order stability, but the list order is observable, and the existing delete-case tests assert that other cases keep their order.

I searched repository Issues and PRs for update_eval_case with order, eval_cases with order, and eval case with position; I did not find a matching report or fix as of 2026-09-21. A focused local fix and regression test are ready if this behavior should be made consistent.

Minimal Reproduction Code:

from tempfile import TemporaryDirectory

from google.adk.evaluation.eval_case import EvalCase
from google.adk.evaluation.in_memory_eval_sets_manager import InMemoryEvalSetsManager
from google.adk.evaluation.local_eval_sets_manager import LocalEvalSetsManager


def run(manager):
    manager.create_eval_set("app", "set")
    for case_id in ("a", "b", "c"):
        manager.add_eval_case(
            "app", "set", EvalCase(eval_id=case_id, conversation=[])
        )
    manager.update_eval_case(
        "app", "set", EvalCase(eval_id="b", conversation=[], creation_timestamp=99)
    )
    return [case.eval_id for case in manager.get_eval_set("app", "set").eval_cases]


with TemporaryDirectory() as directory:
    print("InMemory:", run(InMemoryEvalSetsManager()))
    print("Local:   ", run(LocalEvalSetsManager(directory)))

How often has this issue occurred?:

  • Always (100%) when updating a non-final case in a local eval set. The GCS manager uses the same helper, but this reproducer does not contact GCS.
Lingua principale
Python
Stelle
21.6k
Fork
4k
Merge medio
13h 49m
PR unite (30g)
10

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di google/adk-python

Tutte le issue di google/adk-python

Issue simili

Altre issue su Python

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.