Parser fuzz validation targets: findings ledger and kill criterion (#1781 B2)
Nobody has claimed this yet.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 68/100
- Issue type
- Documentation
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- github-actions, typescript
- Domain
- ci-cd, documentation, testing
Research direction
Start with .github/workflows/replays-nightly.yml and scripts/fuzz/validation-arbitraries.test.ts to understand the Parser Fuzz Lane and its validation targets. Review nightly findings and append one ledger row per finding, identifying real defects versus phantoms. Done means the tracker records each finding and the stated phantom or six-month yield criteria can be applied at review.
Written by the indexing model from the issue text.
Description
The parser fuzz lane's validation targets (cli-validation, maestro-validation, added in #1866 for #1781 B2) ship with a kill criterion that nothing currently records, so this is their live tracker — the same treatment #1823 gave the subprocess-stub project.
What the lane costs
Nightly Parser Fuzz Lane in .github/workflows/replays-nightly.yml: 38,000 cases x 7 targets, measured at parity with the pre-B2 five-target/50k budget (16.9s vs 16.8s on a quiet host; the CI step was 21s before). No new job, no macOS occupancy. PR-time cost is scripts/fuzz/validation-arbitraries.test.ts (~0.4s, unit-core) plus the corpus replay that already existed.
Findings ledger (append one row per nightly finding)
| date | run | target · class | real defect or phantom? | outcome |
|---|---|---|---|---|
| (none yet) |
A phantom is a case whose expectation was wrong rather than the parser — a generator drift, not a bug. One was caught pre-merge (--scale=1.110000000000017, float modulo drifting past a fractional max) and fixed before landing; validation-arbitraries.test.ts exists to catch that class at PR time.
Kill criteria (either fires ⇒ delete the two validation targets, keep the classic five)
- Phantoms: a phantom finding reaches a nightly twice.
- Yield: no real defect found by either validation target in 6 months of nightlies — review on 2027-02-19.
The calibration behind the lane (#1781, B3 comment) measured reach, not yield: it proved the targets can rediscover seeded defects of the shape they aim at, and explicitly did not predict how many unknown defects they will find. Criterion 2 is what tests yield, which is why this ledger exists.
Notes for whoever reviews this
- Only two of the eight calibration rows were genuine generated new reach (#1433 via
cli-validation; a silently-accepted Maestro field viamaestro-validation); two more were new detection class but rediscovered by pinned seed cases, and four were already within the classic targets' reach. Judge yield against the first two. - Adding a mutation class is a few lines in
scripts/fuzz/validation-arbitraries.ts; classes whose whole input space is a handful of strings belong in the target's seed list instead.
Umbrella: #1781 (item B2/B3). Lane: #1414. Sibling tracker: #1823.
- Dominant language
- TypeScript
- Stars
- 4.7k
- Forks
- 304
- Avg merge
- 11h 6m
- Merged PRs (30d)
- 554
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from callstack/agent-device
-
needs-triage
Difficulty 5/5 Over a week Newbie friendliness 32/100
callstack/agent-device#2948 ·
-
Difficulty 4/5 3-5 days Newbie friendliness 55/100
callstack/agent-device#2946 ·
-
needs-triage
Difficulty 4/5 3-5 days Newbie friendliness 45/100
callstack/agent-device#2940 ·
-
bug ready-for-agent
Difficulty 5/5 Over a week Newbie friendliness 35/100
callstack/agent-device#2935 ·
-
needs-triage
Difficulty 5/5 Over a week Newbie friendliness 38/100
callstack/agent-device#2906 ·
All issues in callstack/agent-device
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
receptron/mulmoterminal#2264 ·
-
documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
components-web-app/docs#96 ·
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
simonsobs/tileviewer#114 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100