qa: verdict lands as NEEDS_VERIFICATION with 'CI pending' as the only gap because /qa runs before the PR's checks finish

Open
  • Is something wrong?
#1,630 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
22/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
git, github-actions, typescript
Domain
devtools

Research direction

The behaviour lives in the qa phase of the orchestrator, which decides the verdict from the gap JSON and the PR check state; start by finding where the verdict is emitted and where gh pr checks output is read. Either make that phase wait for checks (bounded by --auto-wait or phases.qa.waitForChecks) or add a CI_PENDING status, then confirm with a run where checks are still IN_PROGRESS. Done when the verdict no longer reads NEEDS_VERIFICATION for pending-only CI. Design choice needed first, so ask the maintainers which option they want.

Written by the indexing model from the issue text.

Description

What happens

/qa runs as soon as exec finishes, while the PR's CI checks are still queued or in progress. When every acceptance criterion is met locally, the verdict still lands as NEEDS_VERIFICATION with "CI pending" as the only gap, and the issue gets the needs-verification label. The coordinator then has to read the verdict to learn there is no code gap, wait for CI by hand, and either re-run /qa (another phase, about $1 to $2 and 2 to 3 minutes) or merge on the earlier review.

Where it was seen (sequant 2.20.0, admarble/matcha-maps, concurrency 1, --phases exec,qa)

Issue PR QA finished Verdict Only AC gap
#943 #1186 2026-10-16 NEEDS_VERIFICATION browser check + CI
#929 #1220 2026-11-02, 3 m 9 s after exec NEEDS_VERIFICATION "Run Tests and Lint & Type Check ... still pending"
#1101 #1221 2026-11-02, 70 s after exec NEEDS_VERIFICATION "Run Tests and Lint & Type Check were still running"
#928 #1222 2026-11-02, 2 m after exec NEEDS_VERIFICATION "Lint & Type Check and Run Tests IN_PROGRESS"

In the last three the QA gap JSON carried {"category":"execution_gap","evidence":"gh pr checks ...: ... IN_PROGRESS","recommendedAction":"pause_for_human"}. All three PRs went CLEAN within 15 to 30 minutes and merged without a code change.

Ask

One of:

  1. The qa phase waits for the PR's checks before emitting the verdict, bounded by --auto-wait or a phases.qa.waitForChecks setting, so the verdict reflects CI.
  2. Or the verdict distinguishes "CI pending, no code gap" from "needs a human", for example a CI_PENDING status or an execution_gap that the orchestrator resolves itself by polling mergeStateStatus and then upgrading the verdict without a second QA run.

Either saves a QA re-run per issue and keeps needs-verification for cases that need a person.

Dominant language
TypeScript
Stars
1
Forks
1
Avg merge
11h 3m
Merged PRs (30d)
84

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from sequant-io/sequant

All issues in sequant-io/sequant

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.