Selecting overlapping suites in one batch overwrites the same log and output identity

Open
  • Is something wrong?
#479 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend, testing

Research direction

Start with src/clawbench/runner/batch.py and trace how discover_cases, Job, log and run directory naming, batch-summary.json, and resume matching use job identity. The acceptance criteria define the expected behavior: selecting a V1 case and its V1-Lite counterpart must preserve distinct jobs and outputs, or emit an explicit duplicate warning. Add or update a test for that selection and check that resume treats the jobs consistently.

Written by the indexing model from the issue text.

Description

bug

Problem

A batch can select the same task twice through two suites and give both jobs one identity. Their logs overwrite each other, and their run directories differ only by timestamp.

At d8faa73:

  • discover_cases in src/clawbench/runner/batch.py deduplicates by Path. test-cases/v1/002-... and test-cases/v1-lite/002-... are different paths to the same task, so both survive.
  • Job carries only case_name; the batch log path is batch-logs/<case>-<model>.log with no suite or timestamp, so the second job's write_bytes replaces the first job's log.
  • The child run directory is <harness>-<case>-<model>-<timestamp> under the model directory, again with no suite, so same-second starts collide as in #433.
  • V1-Lite is twenty V1 tasks by construction, so a glob such as test-cases/*/002-* selects a duplicate pair by default.

Running a task under two suite labels could be legitimate. Losing one job's log and mixing its outputs is not.

Acceptance

  • Jobs are deduplicated by resolved task file, or the suite is part of the job identity used for logs, run directories, and the summary.
  • batch-summary.json records the suite or cases directory for every job.
  • A test that selects a V1 case and its V1-Lite counterpart produces two distinguishable jobs with two logs, or one job with an explicit duplicate warning.
  • Resume matching uses the same identity, so one suite's finished job does not skip the other's.

@Perry2004 please maintain this issue.

Dominant language
Python
Stars
1.2k
Forks
74
Avg merge
19h 1m
Merged PRs (30d)
5

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from TIGER-AI-Lab/ClawBench

All issues in TIGER-AI-Lab/ClawBench

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.