Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

[Bug]: Updating an eval case moves it to the end in Local/GCS eval sets

Cerrado
  • ¿Algo no está bien?
#7,220 1 comentario 0 reacciones 1 asignado Ver en GitHub

Los mantenedores suelen responder en 6 días

@sanketpatil06 ya está trabajando en esto.

Desde el 21/9/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

eval request clarification

🔴 Required Information

Describe the Bug:

Updating an existing eval case through LocalEvalSetsManager changes its position in EvalSet.eval_cases. GcsEvalSetsManager uses the same update helper, while InMemoryEvalSetsManager replaces the case at its existing position. The difference is visible through the public get_eval_set() result. The list also supplies case IDs to LocalEvalSampler. I am reporting the unexpected list reorder and backend inconsistency, not claiming a guaranteed change in evaluation completion order.

Steps to Reproduce:

  1. Install google-adk from commit 3f24d2036a3434b755f6337026ac19a737041f85 or use a recent checkout.
  2. Run the minimal code below. It creates an eval set, adds cases a, b, and c, and updates only b through each manager's public API.
  3. Compare get_eval_set(...).eval_cases after the update.

Expected Behavior:

Updating case b replaces its contents without moving it: both managers return ['a', 'b', 'c'].

Observed Behavior:

InMemory: ['a', 'b', 'c']
Local:    ['a', 'c', 'b']

Environment Details:

  • ADK Library Version (pip show google-adk): editable checkout at 3f24d2036a3434b755f6337026ac19a737041f85, google-adk 2.9.0. The same remove-and-append implementation was still present on the GitHub default branch when checked on 2026-09-21.
  • Desktop OS: Windows 11, build 26200.
  • Python Version (python -V): Python 3.12.0.

Model Information:

  • Are you using LiteLLM: No.
  • Which model is being used: N/A. This occurs in eval-set storage without a model call.

🟡 Optional Information

Regression:

Unknown. I reproduced the behavior at the commit above and confirmed the same helper implementation on the current GitHub default branch. I have not bisected its introduction.

Logs:

No exception is raised. The output under Observed Behavior is the complete relevant result.

Screenshots / Video:

N/A.

Additional Context:

update_eval_case_in_eval_set() removes the existing case and appends the replacement. Both the local and GCS managers call this helper. The in-memory manager instead replaces the matching list element in place. The EvalSetsManager interface does not explicitly promise order stability, but the list order is observable, and the existing delete-case tests assert that other cases keep their order.

I searched repository Issues and PRs for update_eval_case with order, eval_cases with order, and eval case with position; I did not find a matching report or fix as of 2026-09-21. A focused local fix and regression test are ready if this behavior should be made consistent.

Minimal Reproduction Code:

from tempfile import TemporaryDirectory

from google.adk.evaluation.eval_case import EvalCase
from google.adk.evaluation.in_memory_eval_sets_manager import InMemoryEvalSetsManager
from google.adk.evaluation.local_eval_sets_manager import LocalEvalSetsManager


def run(manager):
    manager.create_eval_set("app", "set")
    for case_id in ("a", "b", "c"):
        manager.add_eval_case(
            "app", "set", EvalCase(eval_id=case_id, conversation=[])
        )
    manager.update_eval_case(
        "app", "set", EvalCase(eval_id="b", conversation=[], creation_timestamp=99)
    )
    return [case.eval_id for case in manager.get_eval_set("app", "set").eval_cases]


with TemporaryDirectory() as directory:
    print("InMemory:", run(InMemoryEvalSetsManager()))
    print("Local:   ", run(LocalEvalSetsManager(directory)))

How often has this issue occurred?:

  • Always (100%) when updating a non-final case in a local eval set. The GCS manager uses the same helper, but this reproducer does not contact GCS.
Lenguaje dominante
Python
Estrellas
21.6k
Forks
4k
Merge medio
9 h 59 min
PR fusionados (30 d)
5

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de google/adk-python

Todos los issues de google/adk-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.