Stage-2 judges get no run date, so relative-date tasks are judged without a temporal anchor

Open
  • Is something wrong?
#476 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
56/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend-api-design, testing

Research direction

Start with src/clawbench/runner/judge.py and trace how judge_request and judge_answer build prompts; then inspect how clawbench-rescore reads run-meta.json and writes verdict files. Check existing fixtures and tests for those paths. Done means both judges receive and record the run date and timezone, rescoring uses the saved anchor (or marks it missing), and the relative-date fixture keeps its verdict when judged later.

Written by the indexing model from the issue text.

Description

bug

Problem

Tasks that say "next Saturday", "tomorrow", or "today" can only be checked against the date the run happened. Neither judge prompt includes that date, so the verdict depends on when the judge happens to be called — which, for a rescore, may be months later.

At d8faa73:

  • judge_request and judge_answer in src/clawbench/runner/judge.py build the user message from the instruction and the intercepted request or answer. No run timestamp, timezone, or current date is included.
  • clawbench-rescore re-judges from run-meta.json's instruction and the saved interception; the run's own timestamp is in that file but is not passed to the judge.
  • Thirteen shipping tasks use relative dates, including 047 / v2-047 TaskRabbit ("next Saturday 9am-1pm"), v2-503 Doodle ("tomorrow"), and v2-1113 Eat This Much ("today's meals").

#419 covers the browser clock being UTC. Even with a correct browser clock, the judge still needs the anchor date to compare against the request.

Acceptance

  • The run's start date and timezone are passed to both judges and shown in the prompt as the reference date for relative expressions.
  • Rescore reads that anchor from run-meta.json rather than using the rescore date; runs without one are marked as lacking an anchor instead of being judged against today.
  • A fixture with a request dated the Saturday after the run date passes, and the same request judged with a later anchor does not silently change verdict.
  • The anchor is recorded in the verdict file.

@Perry2004 please maintain this issue.

Dominant language
Python
Stars
1.2k
Forks
74
Avg merge
19h 1m
Merged PRs (30d)
5

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from TIGER-AI-Lab/ClawBench

All issues in TIGER-AI-Lab/ClawBench

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.