Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Bug] Multi-trace files collapse into one eval case

Open
  • Is something wrong?
#1,021 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
78/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
testing-qa

Research direction

Start at BaseEvaluator._build_eval_set_from_tracing_json() and read the focused regression coverage in tests/test_evaluator.py. Reproduce the multi-trace, out-of-order-span failure first, then verify that each trace becomes an isolated EvalCase with ordered spans and that the EvalSet timestamp uses the earliest generated case timestamp. Run the evaluator tests, Ruff, and git diff --check when done.

Written by the indexing model from the issue text.

Description

Bug description

BaseEvaluator._build_eval_set_from_tracing_json() groups spans by trace_id, but currently keeps conversation, metadata, and timestamps outside the per-trace loop and appends only one EvalCase after the loop.

For a tracing file containing multiple traces, this can collapse all traces into one case, attach metadata from the last trace, and produce an empty conversation when spans are not already ordered by start_time.

Minimal reproduction

On current main (43f958d), a tracing file with two traces and out-of-order call_llm spans produces:

  • len(eval_set.eval_cases) == 1 instead of 2
  • the only case uses the second trace's app_name and user_id
  • conversation == []

The focused regression fails deterministically with:

assert len(eval_set.eval_cases) == 2
E assert 1 == 2

Expected behavior

Could you confirm whether the intended conversion semantics are:

  1. each trace_id becomes one isolated EvalCase;
  2. spans are ordered by start_time within that trace;
  3. conversation, tool calls, and session metadata never cross trace boundaries;
  4. the EvalSet timestamp is the earliest generated case timestamp?

Validated local fix

A local one-commit patch implements the behavior above and currently passes:

  • tests/test_evaluator.py — 3 passed
  • Ruff 0.11.12 check and format
  • git diff --check

I have not opened a PR yet because this changes the public mapping between tracing files and evaluation cases. If the semantics above are intended, I can submit the tested patch.

AI assistance

The investigation and candidate patch were developed with AI assistance. I verified the failure on a clean origin/main worktree with an isolated Python bytecode cache and reviewed the trace-boundary semantics.

Dominant language
Python
Stars
345
Forks
99
Avg merge
11h 38m
Merged PRs (30d)
89

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from volcengine/veadk-python

All issues in volcengine/veadk-python

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.