bench harness: a surplus TSV column is silently dropped, and stride = L // 6 duplicates questions

Open Beginner friendly
  • Is something wrong?
#492 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
python
Domain
testing-qa

Research direction

Start with load_instances() in bench/agentloop/grade_answers.py, where dict(zip(COLUMNS, cells)) silently truncates rows, and reuse the header width check that already refuses a 14-column header. Then read derive_questions in bench/headtohead/r4-2026-08-06/r4_worker.py, where stride = L // 6 becomes 0 when L is 3. Done means a 14-cell row is refused with an error, and a bank with L < 6 either fails closed or yields six distinct questions. Run the existing bench tests to confirm nothing else changes.

Written by the indexing model from the issue text.

Description

Field Value
Labels (suggested) bug, bench
Reproduced on commits 67b53a805fc3 (bench/agentloop/grade_answers.py) and 57d512684b60 (bench/headtohead/r4-2026-08-06/r4_worker.py)
Introduced by 4bd034665a24 ("feat(bench): the bank scores answers …") and dff64a2097af ("bench(headtohead): commit the round-4 harness …")

Summary

Two independent defects in the benchmark harness: the graded-TSV loader is fail-closed for headers but truncates
surplus row cells, and the question-derivation stride collapses to zero on small banks so the same question is
emitted six times. They can be filed separately.

(a) A row with more cells than the header is accepted and truncated

Expected behavior. A row whose cell count differs from the header should be refused, exactly as a header with
the wrong column count is refused ("fail-closed, no silent column re-mapping").

Actual behavior. load_instances() refuses a 14-column header but accepts a 14-cell row under the
13-column header, keeping only the first 13 cells via dict(zip(COLUMNS, cells)):

14-column header : refused
14-cell row      : accepted, no exception, len(rows)==1, 13 keys, notes='clean row'
                   the 14th cell ('T…' sentinel) is gone
control (width check added): refused, "row has 14 cells, schema has 13, refusing surplus columns";
                   a 13-cell row still loads

Suggested fix. Apply the same width check to rows as to headers (refuse, don't truncate).

(b) stride = L // 6 collapses to zero for small banks, producing duplicate questions

Expected behavior. For a bank that cannot supply six distinct candidates per commit, derive_questions
should either refuse to write the bank (fail closed) or emit distinct questions — never the same item six times.

Actual behavior. derive_questions picks six shapes per commit with stride = L // 6; for L = 3,
stride = 0 and all six picks index elig[0]:

case A (L=3): "S4 L=3 stride=0", "S3 L=3 stride=0"
             -> 30 rows / 20 distinct commits; S4 = 6 identical rows (same commit, same question), S3 likewise
case B (L>=6): 30 rows / 30 distinct, 6 per shape
case D (L=0): IndexError at s = elig[i*stride]
control (fail-closed `if L < 6: raise SystemExit`): refuses to write a duplicate bank for case A's corpus

A small bank therefore inflates that shape's weight with duplicates and makes the benchmark's numbers
meaningless; L = 0 crashes with IndexError instead.

Suggested fix. Fail closed when L < 6 (as the control does), or sample without replacement
(random.sample(elig, 6)) so a small bank either refuses or yields distinct items.

Dominant language
C++
Stars
2.5k
Forks
158
Avg merge
1d 9h
Merged PRs (30d)
68

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from redhat-et/ripwire

All issues in redhat-et/ripwire

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.