qa: recommendedAction rubric has no slot for silently wrong output — short correctness bugs get tagged `document`
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Newbie friendliness
- 76/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- typescript
- Domain
- tooling
Research direction
Read .claude/skills/qa/SKILL.md, especially the SEQUANT_QA_GAPS field table and recommendedAction definition. Update the rubric so silently wrong outputs are classified as fix_now, however small the fix; the optional /loop behavior is not required. Done when the definition includes this rule.
Written by the indexing model from the issue text.
Description
Summary
QA's recommendedAction rubric (.claude/skills/qa/SKILL.md, the SEQUANT_QA_GAPS field table, main @ 68446b02) defines fix_now as "blocks READY_FOR_MERGE, actionable by a code change" and document as "real but non-blocking — quality/polish, or explicitly deferred". It doesn't say where an output that is wrong without erroring belongs. In practice reviewers tag short correctness defects as document.
Observed (admarble/thenewwayai, sequant 2.19.0, 2026-10-07/08)
| Issue | QA finding | Tag | What it was | Runner fix |
|---|---|---|---|---|
| #398 | Algolia typo tolerance pulls unrelated stories (GPT matched "git": a C64 font, a PS5 port…) |
document (verdict AC_MET_BUT_NOT_A_PLUS) |
Silently wrong rows in a daily data file | 1 query param + a self-test row, < 10 lines |
| #401 | A 3-digit glob issue-[0-9][0-9][0-9].md would skip issue-1000.md and take the window from 999 |
document (verdict READY_FOR_MERGE) |
A silently wrong time window, latent | Numeric sort + a self-test row, < 10 lines |
| #216 (control) | A crashed dependency is scored as a 0% catch rate, exit 0 | fix_now, correctly |
Silently wrong metric | — |
Two of three silent-wrong-output findings were tagged as polish. Both were correctness bugs, each with a failing-first test under 10 lines. n=3, so this is a calibration signal, not a measurement.
Suggested change
Add one line to the recommendedAction definition: "An output that is wrong without erroring (a silently wrong result, a skipped input, a misleading score) is fix_now, however small the fix." Optionally, have /loop treat a document finding whose description names wrong output as fixable.
Related, but measuring something else: #1068 (detection recall on the #1067 seeded-defect fixture). This issue is about the tag a found defect gets.
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 1
- Avg merge
- 11h 3m
- Merged PRs (30d)
- 84
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from sequant-io/sequant
-
graph-2026-11-simplify
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
sequant-io/sequant#1650 ·
Maintainers usually reply within 1 day
-
run: --phases does not suppress the UI-label test phase (determinePhasesForIssue adds it to an explicit list)Possibly taken @admarble claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
sequant-io/sequant#1631 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
sequant-io/sequant#1601 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
sequant-io/sequant#1582 ·
Maintainers usually reply within 1 day
-
improve(skills): /loop and /fullsolve run affected tests, not the full npm test (#1349 follow-up)Openenhancement
Difficulty 2/5 1-3 hours Newbie friendliness 83/100
sequant-io/sequant#1360 ·
Maintainers usually reply within 1 day
All issues in sequant-io/sequant
Similar issues
-
content triage
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
tensorflow/tfjs#8754 ·
-
tool-submission
Difficulty 1/5 Under an hour Newbie friendliness 83/100
BraveOPotato/NoSignups#2348 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
code-yeongyu/oh-my-openagent#12784 ·
Maintainers usually reply within 1 day
-
external-issue to-triage
Difficulty 2/5 1-3 hours Newbie friendliness 71/100
Maintainers usually reply within 1 day