[agentrx-optimizer] Daily Workflow Optimization - 2026-11-05

クローズ 初心者向け
  • 何か問題がありますか？
#75,106 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
2/5
見積もり時間
1〜3時間
初心者へのやさしさ
74/100
issue の種類
機能追加
明瞭さ
明確に書かれている
活発さ
活発
技術スタック
github-actions, markdown
領域
ci-cd, devtools

調査の方向性

.github/workflows/daily-community-attribution.md と .github/workflows/code-scanning-fixer.md にあるエージェントプロンプトの本文を読み、edit ツールと tools.bash の許可リストがどのように設定されているかを確認してください。ファイルへの書き込みには edit を使うこと、許可されたコマンドの前に cd を付けないこと、python3 の代わりに jq を使うことを案内に追加し、make recompile を実行してロックファイルを再生成してください。両方のプロンプトに案内が含まれ、生成されたロックファイルが更新されたら完了です。

索引モデルが issue の本文から書いたものです。

説明

automation observability optimization traces
Executive Summary

AgentRx analyzed 10 sampled gh-aw agent runs from the last 24h (prioritizing failed/highest-event-volume trajectories) via the pre-downloaded log bundle. The IR normalization stage completed for all 10 trajectories; the static/dynamic/check/judge LLM-backed stages could not run in this environment (see Limitations). Direct session-log inspection of the largest failure cluster found a concrete, reproducible root cause: the Copilot CLI agent repeatedly issues bash commands (multi-line heredocs, cd ..., python3 ...) that the workflow's tools.bash allowlist rejects, and the resulting stream of denials crashes the CLI driver (exitCode 1). This single pattern accounts for 2 of 4 occurrences in the dataset's single largest failure-boundary cluster.

AgentRx Evidence

  • Critical step: agent / "Execute GitHub Copilot CLI" (job: agent)

  • Failure category: agent_logic (failure_boundary signal, cluster failure-8169433961c9451a)

  • Frequency / impact: 4 runs / 30 failed runs in the 24h window (13.3% share) — the single largest failure_boundary cluster observed; every other failure_boundary cluster in the window was a singleton (count=1). Within this cluster, 2 of 4 runs share an identical, reproducible signature (bash permission denial → sdk_driver crash); the other 2 have distinct causes (rate limit, query error — see details).

  • Representative run IDs:

Violation classification

AgentRx's automated check/judge stages did not produce output in this environment (see Limitations), so this table is derived directly from raw session telemetry (usage/aw_session.jsonl) for the 10 sampled trajectories.

violation evidence fix_type rationale
Multi-line cat > file << 'EOF' ... EOF heredoc bash commands denied despite a matching "cat *" allowlist entry §41955226606 events #368→#370, #325→#327 ("Tool invocation is not allowed by workflow tool permissions"); same pattern in §41944783819 prompt tightening to reduce invalid tool invocations The agent defaults to heredoc-style file writes even though the workflow enables the edit tool for exactly this purpose. Prefix-glob bash permission matching does not reliably match multi-line command bodies, so every heredoc attempt is denied and retried until the turn budget is exhausted and the CLI driver crashes (exitCode: 1, errorTypes: ["sdk_driver"]).
Compound cd <dir> && jq ... commands denied (cd absent from every observed bash allowlist) §41955226606 events #384, #467 prompt tightening to reduce invalid tool invocations The agent repeatedly prefixes otherwise-allowed commands (jq) with cd, which no sampled workflow allowlists, so the whole compound command is rejected each time.
python3 << 'EOF' ... EOF denied (python3 absent from allowlist) §41955226606 event #514 prompt tightening to reduce invalid tool invocations Agent falls back to inline Python for JSON/report generation instead of the allowlisted jq or the enabled edit tool.
Invocation cap exceeded / HTTP 429 after an unusually long trajectory §41925400471 (Avenger): categories: ["invocation_cap_exceeded"], errorCodes: [429], 1011 session substeps (3-10x any other sampled run) improving retry/backoff strategy By far the highest event volume among the 10 sampled trajectories, consistent with a tight retry loop burning through the rate-limit budget rather than backing off.
driver_exit failures in isolated infra/setup steps (repo checkout, TLA+ runtime prep, Cursor CLI, Codex CLI) §41940379704 (Smoke Drive), §41941812478 (Smoke Update Cross-Repo PR), §41972802666 (Daily Workflow Security Model), §41941619530 (Smoke Cursor) adding missing telemetry attributes for better triage Each is a singleton failure_boundary cluster with a distinct failed step name and no further session-level error detail available in the collected logs — these need step-level error capture before a shared fix can be identified.
AgentRx Artifacts

  • IR summary: trajectory_ir.json normalized all 10 sampled trajectories successfully (flash-domain converter, no LLM fallback needed) — substep counts per trajectory: 41955226606=687, 41944783819=664, 41925400471=1011, 41924480521=392, 41940379704=131, 41941812478=101, 41972802666=105, 41941619530=186, 41969455502=167, 41961870265=255.

  • Invariant/checker highlights: Not available — see limitations below.

  • Judge classification output: Not available — see limitations below.

  • Known limitations:

    • The static, dynamic, check, judge, and report pipeline stages all failed identically with SystemError: The installed pydantic-core version (2.50.2) is incompatible with the current pydantic version, which requires 2.50.1 while importing agentrx.llm_clients.azure → openai → pydantic. This is a pre-existing dependency mismatch in the installed AgentRx virtualenv, not something fixable here (no pip install/env-repair was performed, per instructions). Only the ir stage (no LLM dependency) completed.
    • The findings above are therefore grounded directly in raw aw_session.jsonl event inspection rather than AgentRx's automated invariant/judge output.
    • Log collection hit its count/storage limit for the 24h window (continuation cursor in summary.json pointed to an earlier start_date/before_run_id); this analysis covers the 25 of 30 unique failed runs with observed evidence that were already clustered, not the full 24h window. 7 of 15 eligible failure clusters remain unreviewed.
Recommended Optimization
  • Change: Add a short "File writes" guidance block to the agent prompt body of .github/workflows/daily-community-attribution.md and .github/workflows/code-scanning-fixer.md, instructing the agent to use the already-enabled edit tool for creating or appending file content instead of bash heredocs (cat > file << EOF), and to never prefix allowlisted commands with cd (use absolute paths) or invoke python3 (use jq, which is already allowlisted, for JSON transforms).
  • Why highest impact: This is the only repeating, reproducible signature in the largest failure cluster in the window — it directly causes a hard CLI driver crash (exitCode 1) after exhausting denied attempts, rather than a graceful degradation, in 2 of 4 occurrences. The fix is a markdown prompt change plus make recompile — no code change required — and the same tools.bash allowlist + edit tool combination is shared by other copilot-engine workflows in this repo, so the same guidance is worth propagating wherever that pattern recurs.
  • Where to implement: Prompt bodies of .github/workflows/daily-community-attribution.md and .github/workflows/code-scanning-fixer.md (the workflow source .md files, followed by make recompile to regenerate the .lock.yml files).
Validation Plan
  • Re-run both workflows and confirm usage/aw_session.jsonl no longer contains tool.execution_complete events with error.code: "denied" during the "Execute GitHub Copilot CLI" step.
  • Track cluster failure-8169433961c9451a (agent/"Execute GitHub Copilot CLI", failure_kind: agent_logic) in the next AgentRx run — expect its count to drop from 4 toward 2 (only the unrelated rate-limit and BYOK query-400 runs remaining).
  • Expected success-metric change: the built-in tool-success-rate grader for these two workflows should rise from the observed 0.76 toward ≥0.95, and tool-failure-count should drop from the observed 6–7 per run to 0–1.
References

Generated by ⚡ Daily AgentRx Trace Optimizer · claude · agent · ~131 AIC · ⌖ ~1 AIC · ⊞ 4.4K · ◷

  • expires on Nov 11, 2026, 9:45 PM UTC-08:00
主要言語
Go
スター
5.6k
フォーク
615
平均マージ
5時間 51分
マージ済み PR（30日）
793

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

github/gh-aw のほかの issue

github/gh-aw の issue をすべて見る

似ている issue

Go の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。