fix(review): resume requested reviews when checks become green

Open
#653 4 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
python
Domain
backend, ci-cd, tooling

Research direction

The issue involves extending the github-pr-reviewer automation in the OpenHands/extensions repository. Start by examining the existing automation code, likely in a directory like skills/github-pr-reviewer. Understand the current triggers (event-based) and the changes from PR #651. The new cron trigger must scan open non-draft PRs with outstanding all-hands-bot review requests. Check how current-head checks are evaluated and the waiting/blocked disposition. Tests need to cover pending-to-green, failed-to-green on same SHA, changed head, and duplicate notifications. 'Done' means the automation works with a cron trigger, meets all acceptance criteria, and has passing tests.

Written by the indexing model from the issue text.

Description

priority:low ready-for-dev

Problem

The live OSS PR reviewer on oss-agent-canvas (f36aba99-070d-40a9-9f77-627f9113347a) has only a GitHub review_requested / bot review.submitted event trigger. Extensions #651 correctly stops before launching an agent while current-head checks are pending or failing, but its message promises a scheduled retry that this deployment does not have. A request arriving during CI can therefore be left unreviewed after CI turns green unless a human removes and re-requests all-hands-bot. This undermines automatic coverage of non-draft PRs.

Desired behavior

Keep the cheap current-head eligibility gate from #651, then automatically revisit a pending requested review when its current-head checks reach a terminal green state. Reuse the existing reviewer automation and conversation-dispatch machinery. Choose the smallest reliable trigger mechanism supported by Canvas Automations; do not launch an agent or consume a slot while checks are pending or failing.

Acceptance criteria

  • A native all-hands-bot review request made before CI finishes results in one review after the exact requested head's checks are green, without a second manual review request.
  • A failing current-head check still prevents review, and a later fix on the same head (such as editing a PR description) or a new head can be reconsidered without a stale failure blocking it.
  • Repeated check-completion events or scans do not create duplicate conversations or reviews; an already completed review is not restarted.
  • The deployed OSS reviewer configuration uses the retry mechanism on all four repos, and the user-facing waiting comment names the actual retry condition instead of promising a nonexistent scan.
  • Tests cover pending-to-green, failed-to-green on the same SHA, changed head, and duplicate notifications. Live Canvas evidence shows a review requested while a check is pending and completed after it becomes green.

Related: #644 and #651. The currently live reviewer remains on the tested #652 bundle; do not merge or deploy #651 until its repeated-check-run policy and this retry path are resolved.


OpenHands AI triage

The following comments and acceptance criteria were added by the OpenHands AI agent.

Triage

The maintainer resolved the earlier design questions, so the scope is now bounded and does not need a further decision. The reusable reviewer behavior belongs in this registry, reusing the existing skills/github-pr-reviewer worker and conversation-dispatch machinery, and stacking on #651 (which must merge first). The OSS deployment gets its retry path by adding a five-minute cron trigger to the same reviewer automation record, scanning open non-draft PRs that hold an outstanding all-hands-bot review request, while the explicit-request event path is retained for other deployments.

Non-goals: no new GitHub webhook event types or check_run parsing (Canvas Automations does not parse that payload today, and adding it belongs to OpenHands/automation); no second automation record for the OSS deployment; no deployment-specific code in OpenHands/extensions. Updating the live oss-agent-canvas automation to the cron trigger on all four repos and syncing OpenHands/oss-agent-canvas is validation and deployment work performed after the PR is live-tested, not part of this change. Neither PR is merged or deployed automatically.

Acceptance Criteria
  • The github-pr-reviewer automation can run on a recurring cron trigger and, on each run, considers open non-draft PRs that carry an outstanding all-hands-bot review request, in addition to the existing trigger-label scan and the explicit-request event path.
  • A native all-hands-bot review request made before CI finishes results in exactly one review after the requested head's checks become green, with no second manual review request.
  • While the exact head has pending or failing checks, the run creates no review conversation and consumes no agent slot; it exits with a clear waiting/blocked disposition.
  • A completed failing or cancelled check on the exact head still prevents review, and the head is reconsidered once checks pass.
  • Current check state is determined from the latest run per logical check on the exact head, grouped by check name together with the reporting app identity and ordered deterministically by run start time and ID; a later successful run replaces an earlier failure on the same SHA, and a later pending rerun makes the head wait. Unrelated apps sharing a display name do not mask one another.
  • Checks attributed to any other head do not block the current head, and a changed head is evaluated on its own checks.
  • Repeated cron scans or duplicate notifications do not create duplicate conversations or reviews, and an already completed exact-head review is not restarted (existing keyed subject/delivery and completed-review dedupe reused).
  • The user-facing waiting comment names the actual retry condition (the scheduled scan, or a new review request) rather than promising a mechanism the deployment does not have, and a moved head updates or replaces the previous gate comment instead of stacking a duplicate.
  • Existing review verdict, maintainer-handoff, profile, and secret-handling behavior is unchanged, and no new secrets or elevated permissions are introduced.
  • Automated tests cover pending-to-green, failed-to-green on the same SHA (latest-run-per-check), changed head, and duplicate notifications/scans.
  • Live Canvas evidence demonstrates a review requested while a check is pending and completed after the check becomes green, and the documented SKILL.md/README.md retry contract matches the deployed cron behavior.
Dominant language
Python
Stars
184
Forks
137
Avg merge
1d 19h
Merged PRs (30d)
46

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from OpenHands/extensions

All issues in OpenHands/extensions

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.