Add DORA collector pipeline and time-window boundary guidance to scorecard AGENTS.md

オープン 初心者向け
  • 何か問題がありますか？
#4,733 コメント 3 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

@fullsend-ai-coder がすでに取り組んでいます。

2026年9月14日 から。

  • #4736 @fullsend-ai-coder による — オープン

評価

難易度
2/5
見積もり時間
1〜3時間
初心者へのやさしさ
76/100
issue の種類
ドキュメント
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
github, graphql
領域
documentation

調査の方向性

既存の「MetricProvider Architecture」セクションの後に workspaces/scorecard/AGENTS.md を更新します。DefaultScorecardCollectorsService.collect()、名前付き collector の出力スキーマ、getDeployments、getWorkflowRuns、および Jira リクエストにおけるページネーションの動作から始めてください。DORA データフロー、bounded な from/to 不変条件、ページネーションの制限、マージされた配列のフィルタリング、レビュー用チェックリストが文書化され、提示された検証基準と一致していれば完了です。

索引モデルが issue の本文から書いたものです。

説明

documentation ready-for-triage ready-to-code workspace/scorecard

What happened

The review agent approved PR #4474 with 2 low-severity stylistic findings (import ordering, commit-prefix mismatch), spending $7.80 on Claude Opus at high effort (run 32971799794).

Five days later, human reviewer gustavolira identified 7 substantive issues that the agent missed, including: an unbounded JQL query that can silently truncate incident results and report 0% CFR for failing services; an epoch-based lookback that pages through all in-window deployments before reaching a usable row (5+ extra GraphQL pages per metric per entity per run for repos with 500 deployments); an unbounded leading interval that makes CFR read 100% for quarterly deployers; and a missing time-range filter on merged deployments that allows custom collectors to introduce out-of-range rows.

All of these findings required understanding the DORA data collection pipeline: how collectors feed providers, how pagination interacts with time windows, and how the from/to boundaries affect both the number of API requests and the accuracy of metric calculations.

What could go better

The scorecard workspace AGENTS.md is thorough on metric IDs, naming conventions, and threshold resolution (lines 14–217), but contains no information about the data collection pipeline — the path from collector input schemas through API calls to provider metric calculations. Without this context, the review agent treated the PR as a straightforward feature addition and focused on surface-level style.

The human reviewer's findings all required reasoning about data flow across API boundaries: how from/to parameters map to pagination behavior, the difference between "rows returned" and "requests made" in GitHub's GraphQL API, how Jira's lack of ORDER BY in JQL affects which items survive truncation, and how time-window boundaries affect denominator counts in DORA metrics.

I am confident this is the root cause. The review agent had access to the AGENTS.md and used it to verify naming conventions (it correctly flagged the import ordering issue), but the file gave it no basis for reasoning about the collection pipeline. The human reviewer's domain expertise in these areas is what enabled the substantive findings.

Uncertainty: Even with pipeline context in AGENTS.md, the review agent may not catch all edge cases that require deep API-specific knowledge (e.g., Jira's pagination truncation behavior). However, documenting the key invariants — especially around time-window boundaries and pagination limits — would give the agent the conceptual framework to reason about them.

Proposed change

Add a new section to workspaces/scorecard/AGENTS.md after the existing "MetricProvider Architecture" section, titled "DORA Collector Pipeline" or similar, covering:

  1. Data flow overview: Collectors (GitHub deployments, GitHub workflow runs, Jira incidents) → DefaultScorecardCollectorsService.collect() → DORA providers (CFR, lead time, DF, MTTR) → metric calculation. Each collector has an input schema (deploymentsCollectorOutputSchema, etc.) and pagination limits.

  2. Time-window invariants: DORA metrics operate over a 30-day sliding window (windowFrom to windowTo). Key invariants to verify in reviews:

    • Every API query must bound both from and to to avoid unbounded history scans
    • The distinction between rows returned (controlled by fetchItemsLimit) and API requests made (controlled by pagination stopping conditions in getDeployments, getWorkflowRuns, etc.)
    • Merged arrays (e.g., in-window + pre-window deployments) must be filtered to the expected time range before calculation

  3. Pagination boundaries: Document that GitHub GraphQL pagination stops when reachedOlderThanWindow flips or deployments.length hits the limit, but rows newer than to are skipped without incrementing deployments.length. Jira's sendPaginatedRequest stops at DEFAULT_PAGINATED_FETCH_ITEMS_LIMIT (1000) with only a warning. Flag any query that sets from to epoch or unbounded as high-risk.

  4. Review checklist for DORA provider changes: When reviewing changes to DORA providers or collectors, verify: (a) all API queries have bounded from/to; (b) merged deployment arrays are filtered to the expected range; (c) pagination limits match the intended scope; (d) error handling distinguishes collection failures from data-insufficiency; (e) metric documentation matches the actual calculation window.

Validation criteria

On the next 3 DORA-related PRs in the scorecard workspace that modify collector or provider logic, the review agent should flag at least one of the following when applicable: unbounded time-range queries, pagination limit mismatches, or unfiltered merged arrays. The agent should not approve DORA pipeline changes without addressing time-window boundary conditions.

As a concrete test: if PR #4732 (the successor to #4474) still uses an epoch-based lookback or unbounded incident query, the review agent should flag it.

Generated by retro agent from https://github.com/redhat-developer/rhdh-plugins/pull/4474

主要言語
TypeScript
スター
51
フォーク
121
平均マージ
1日 17時間
マージ済み PR（30日）
270

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

redhat-developer/rhdh-plugins のほかの issue

redhat-developer/rhdh-plugins の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。