[Bug] Multi-trace files collapse into one eval case

Inizia da BaseEvaluator._build_eval_set_from_tracing_json() e leggi la copertura di regressione mirata in tests/test_evaluator.py. Riproduci prima il fallimento con più trace e span fuori ordine, poi verifica che ogni trace diventi una EvalCase isolata con span ordinati e che il timestamp di EvalSet utilizzi il timestamp del caso generato più vecchio. Al termine, esegui i test dell'evaluator, Ruff e git diff --check.

Bug description

BaseEvaluator._build_eval_set_from_tracing_json() groups spans by trace_id, but currently keeps conversation, metadata, and timestamps outside the per-trace loop and appends only one EvalCase after the loop.

For a tracing file containing multiple traces, this can collapse all traces into one case, attach metadata from the last trace, and produce an empty conversation when spans are not already ordered by start_time.

Minimal reproduction

On current main (43f958d), a tracing file with two traces and out-of-order call_llm spans produces:

• len(eval_set.eval_cases) == 1 instead of 2
• the only case uses the second trace's app_name and user_id
• conversation == []

The focused regression fails deterministically with:

assert len(eval_set.eval_cases) == 2
E assert 1 == 2

Expected behavior

Could you confirm whether the intended conversion semantics are:

1. each trace_id becomes one isolated EvalCase;
2. spans are ordered by start_time within that trace;
3. conversation, tool calls, and session metadata never cross trace boundaries;
4. the EvalSet timestamp is the earliest generated case timestamp?

Validated local fix

A local one-commit patch implements the behavior above and currently passes:

• tests/test_evaluator.py — 3 passed
• Ruff 0.11.12 check and format
• git diff --check

I have not opened a PR yet because this changes the public mapping between tracing files and evaluation cases. If the semantics above are intended, I can submit the tested patch.

AI assistance

The investigation and candidate patch were developed with AI assistance. I verified the failure on a clean origin/main worktree with an isolated Python bytecode cache and reviewed the trace-boundary semantics.