[Bug] Multi-trace files collapse into one eval case

BaseEvaluator._build_eval_set_from_tracing_json() から始め、tests/test_evaluator.py の対象を絞ったリグレッションカバレッジを読みます。まず複数の trace と順序が正しくない span による失敗を再現し、次に各 trace が順序付けられた span を持つ独立した EvalCase になり、EvalSet の timestamp が生成されたケースのうち最も早い timestamp を使用することを確認します。完了したら evaluator のテスト、Ruff、git diff --check を実行します。

Bug description

BaseEvaluator._build_eval_set_from_tracing_json() groups spans by trace_id, but currently keeps conversation, metadata, and timestamps outside the per-trace loop and appends only one EvalCase after the loop.

For a tracing file containing multiple traces, this can collapse all traces into one case, attach metadata from the last trace, and produce an empty conversation when spans are not already ordered by start_time.

Minimal reproduction

On current main (43f958d), a tracing file with two traces and out-of-order call_llm spans produces:

• len(eval_set.eval_cases) == 1 instead of 2
• the only case uses the second trace's app_name and user_id
• conversation == []

The focused regression fails deterministically with:

assert len(eval_set.eval_cases) == 2
E assert 1 == 2

Expected behavior

Could you confirm whether the intended conversion semantics are:

1. each trace_id becomes one isolated EvalCase;
2. spans are ordered by start_time within that trace;
3. conversation, tool calls, and session metadata never cross trace boundaries;
4. the EvalSet timestamp is the earliest generated case timestamp?

Validated local fix

A local one-commit patch implements the behavior above and currently passes:

• tests/test_evaluator.py — 3 passed
• Ruff 0.11.12 check and format
• git diff --check

I have not opened a PR yet because this changes the public mapping between tracing files and evaluation cases. If the semantics above are intended, I can submit the tested patch.

AI assistance

The investigation and candidate patch were developed with AI assistance. I verified the failure on a clean origin/main worktree with an isolated Python bytecode cache and reviewed the trace-boundary semantics.