ci: test that the hosted proof aggregate runs candidate validation
Maintainers usually reply within 1 day
@schlessera is already working on this.
Since Oct 20, 2026.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 52/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- bun, github-actions, typescript
Research direction
Read scripts/ci-proof.ts around lines 78–93, then inspect the existing spawned aggregate tests and the generated-input and native-diagram qualification suites. Add the fast draft pull-request test described in the issue and refresh both producer digests in scripts/generated-build-inputs.ts and scripts/native-diagram-inputs.ts. Done means the mutation is caught by the expected refusal message and the stated suites and PR checks pass; the issue is blocked by #2365.
Written by the indexing model from the issue text.
Description
Blocked by #1229
Hosted module-settings recurrence — 2026-10-19
PR #2433 / #2400 ready run 39216998411, attempt 1, browser (2), job 117943499384, fails module settings: source-preserving edits and guarded navigation in dark at 320px at the unchanged 15,000-ms whole-case deadline (test, packages/ui-kit/tests/visual/module-settings.visual.tsx:77). The case artifact reports 15,937.5 ms; dark/1440 and light/320/1440 siblings pass at 10,996.6 / 10,140.4 / 8,848.8 ms. Shard totals: 3,786 passed / one failed / six skipped cases, 81 passed / one failed files. No specific edit/navigation assertion or screenshot mismatch is reported.
Actual checkout/resource and case artifacts agree on merge 7d202bc304fe14d2509ec8c14dcee30da65ca5b7, parents base 90a95123cfceb169bd2ecf7c213544db742eaed7 and head cc98b95e81a2ab2a1444f0073b18d2d49b264827, tree 2a641151375ba62ccce2b03d9b1e93fd060e664b; immutable identity was independently verified before requesting the run. Module-settings source SHA-256 is 79d4b53a5496cabdec62e7c087c2b298719f6e77091913a7f97541b3d345e702, unchanged from that base. The two-file contribution only reorders aggregate candidate/selection validation and adds its executable regression; it does not modify module-settings, the browser configuration or its capture commands.
Downloaded artifact browser-proof-2, ID 12170388255, matches API SHA-256 df78eeac440f03838621cf15a5dbe68305860ceeb3137f6a163d889d93f6626c. Its 303 entries contain no module-settings filename or the failure screenshot named by the log. The natural current-runner recurrence matches existing #1229, including the same-base main-push failure already recorded there. It does not locate the responsible waiting phase or establish an OOM, scheduler or product cause. Preserve #1229's needs: repro and existing acceptance criteria; no new duplicate, timeout increase, assertion change or unchanged retry.
The six skipped cases are the rail-fine keyboard/visual-viewport variants in working-sessions.pointer.tsx; their status is recorded separately from passing cases. Failed shard costs: 773 job-wall seconds, 728.358 proof seconds (101.519 build / 626.838 browser), 1,770.04 VM active CPU seconds, sampled VM used-memory peak 12,097,896,448 bytes. This is qualification cost, not measured savings.
Run completion: the aggregate correctly rejected the failed browser category; all other jobs passed. See the final handoff below. No merge or complete passing proof is claimed.
Context
Found while delivering #2363 (PR #2398). The hosted proof aggregate (if (process.argv[2] === "aggregate"), scripts/ci-proof.ts:78-93) calls validateSelections before requireCandidate. Nothing executes that entry point with GITHUB_ACTIONS=true for a pull-request cadence. If the await requireCandidate(...) call were removed, the hosted gate would skip candidate identity (confirmation) and live readiness (draft), and every existing test would still pass. This was confirmed by mutation during #2363.
A spawned-aggregate test was tried in #2398 and removed. validateSelections plans the whole repository, which takes about 8 s locally. Because the test must also compute matching outputs, it took 20 s locally and timed out at 30 s on a hosted unit shard (run 39083242416, unit (3)).
Scope
- Run
requireCandidatebeforevalidateSelectionsin the aggregate. Both are side-effect-free gates that must pass, so this only changes which refusal surfaces first. A run with no verifiable candidate then fails before planning. - Add a spawned-aggregate test with
GITHUB_ACTIONS=true, a draft pull-request event and no token. It asserts exit 1 and theDraft proof requires live read-only GitHub readiness verificationmessage, needs no plan computation, and runs in about a second. - Preserve the independently audited producer digests in
scripts/generated-build-inputs.tsandscripts/native-diagram-inputs.ts; verify source drift refuses binding and the independently frozen positive fixtures remain valid. Requalification of the complete current producer/source closure is outside this guard-order change.
Out of scope: any change to what either gate accepts.
Acceptance criteria
- The aggregate validates the candidate before selections; outputs are otherwise unchanged.
- The new spawned test passes within a few seconds and fails on the expected message when the aggregate's
requireCandidatecall is removed (mutation receipt in the PR). - Audited producer digests and complete source/installation guards remain unchanged, and native diagram qualification plus generated-input suites pass using the corrected independently frozen fixtures.
Verification
Focused tests plus the mutation receipt; bun run check:pr --base origin/main; complete candidate confirmation.
Contract impact
None; internal CI only.
Prerequisite #2365 is delivered through #2374; generated-fixture corrections #2373 / #2388 are also delivered.
2026-10-19 source-verified readiness correction: the original digest-refresh instruction predates the delivered fixture corrections. At base 90a95123cfceb169bd2ecf7c213544db742eaed7, scripts/ci-proof.ts still matches digest c58de06eba788b6b892416ddda3abb28fa1e8e908f25c0800482fbe05e0de678 in both binders and in each independently frozen owner (generated 498270fbdc4ef2788fdd3babb259b647d77f6c77; native 0d780605fe1ea9a281cce947b1f0074f3f3e7acc). Changing that production pin would invalidate those positive fixtures, while current source already differs elsewhere from the audited closure. The established docs/decisions/ci-utility.md policy requires fresh proof for producer/source drift. This task preserves the audit and verifies its refusals; it does not invent qualification by repinning a single caller. The aggregate citation remains accurate at lines 78–93. All candidate and selection acceptance rules remain unchanged.
Final blocked handoff
2026-10-19 — final blocked handoff for claim e1849-2400-20261019-r1.
Run 39216998411 attempt 1 completed with failure. Aggregate job 117949450544 emitted the correct candidate receipt for head cc98b95e81a2ab2a1444f0073b18d2d49b264827, base 90a95123cfceb169bd2ecf7c213544db742eaed7, actual checkout 7d202bc304fe14d2509ec8c14dcee30da65ca5b7, tree 2a641151375ba62ccce2b03d9b1e93fd060e664b. It then correctly refused browser: selected proof did not pass (failure). No new aggregate defect or identity mismatch was reported.
All 15 other jobs succeeded. Browser (2) failed the unchanged module-settings dark/320px whole-case deadline already recorded on #1229. There were no cancelled or skipped jobs. Summed job wall time, including setup and aggregate, was 9,552 seconds (159.2 job minutes); first-job start to aggregate completion was 1,104 seconds (18m24s). These are failed confirmation costs, not controlled savings or billed-minute estimates. The prior focused mutation, restored suites and preflight receipts remain valid for their recorded source; they do not replace complete passing hosted confirmation.
PR #2433 returns to draft. #2400 remains open and blocked by #1229, with milestone 0.44.0 preserved on both. Implementation is preserved on as/aggregate-candidate-guard at the head above; the clean worktree is retained. No test/assertion/deadline/guard change or unchanged rerun was made.
Release this invocation's claim e1849-2400-20261019-r1 and its assignment; other workers' claims remain untouched. To resume: resolve/qualify #1229, reclaim #2400, assess current base/source changes, run required focused preflight, verify live head/base and actual synthetic merge identity, request fresh selected confirmation, then merge and reconcile actual main only after all required proof passes. #2417 / #2434 remains independently watched for completion of its existing run.
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 0
- Avg merge
- 4h 40m
- Merged PRs (30d)
- 953
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from schlessera/brain-kit
-
agent-ready area: backends priority: p2 type: docs
Difficulty 1/5 1-3 hours Newbie friendliness 83/100
schlessera/brain-kit#2383 ·
Maintainers usually reply within 1 day
-
agent-ready area: ci area: docs priority: p2 type: docs
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
schlessera/brain-kit#2104 · 4 comments ·
Maintainers usually reply within 1 day
-
agent-ready area: backends priority: p2 type: test
Difficulty 2/5 1-3 hours Newbie friendliness 64/100
schlessera/brain-kit#1996 · 1 comment ·
Maintainers usually reply within 1 day
-
agent-ready area: docs priority: p2 type: docs
Difficulty 1/5 Under an hour Newbie friendliness 88/100
schlessera/brain-kit#1990 · 3 comments ·
Maintainers usually reply within 1 day
-
agent-ready area: core type: docs
Difficulty 2/5 1-3 hours Newbie friendliness 62/100
schlessera/brain-kit#1944 ·
Maintainers usually reply within 1 day
All issues in schlessera/brain-kit
Similar issues
-
frontend
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
ranxianglei/billion-context#4172 · 1 comment ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
awslabs/threat-composer#362 ·
Maintainers usually reply within 1 day
-
Area/Cloud Needs Triage Type/Bug
Difficulty 1/5 1-3 hours Newbie friendliness 86/100
wso2/product-integrator#2847 ·
Maintainers usually reply within 1 day