clawbench-prorl-submit --dry-run prints the judge and PurelyMail secrets

Open Beginner friendly
  • Is something wrong?
#457 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
82/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
backend, security

Research direction

Start with src/clawbench/prorl/submit.py and trace how the dry-run payload is printed after build_task_request and to_payload. Add a test that builds a request with canary credentials and checks they do not appear in printed output; verify submitted payloads remain unchanged. Done means dry-run output redacts sensitive values in both evaluator and runtime env.

Written by the indexing model from the issue text.

Description

bug

Problem

The dry run is the output a user pastes into an issue or a chat to ask why a submission looks wrong. It currently contains live credentials.

At d8faa73, main in src/clawbench/prorl/submit.py builds the full TaskRequest and, for --dry-run, prints json.dumps(payload, indent=2). build_task_request puts the judge configuration into EvaluatorSpec.env, which includes CLAWBENCH_JUDGE_API_KEY, and puts PurelyMail credentials into the runtime env when --purelymail-api-key is passed. Both are serialized by to_payload. The non-dry-run path does not print the payload, so the leak is specific to the debugging path.

Acceptance

  • Dry-run output redacts every env value whose name contains a key, token, secret, or password marker, in both evaluator and runtime env.
  • A test builds a request with canary credentials and asserts neither canary appears in the printed payload.
  • There is no flag that prints the unredacted payload by default; if one is added, it warns on stderr.
  • Submitted payloads are unchanged.

@Perry2004 please maintain this issue.

Dominant language
Python
Stars
1.2k
Forks
74
Avg merge
19h 1m
Merged PRs (30d)
5

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from TIGER-AI-Lab/ClawBench

All issues in TIGER-AI-Lab/ClawBench

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.