[agentrx-optimizer] Daily Workflow Optimization - 2026-11-05

未关闭 适合新手
  • 哪里不对？
#75,106 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

维护者通常 1 天内回复

还没有人认领这个 Issue。

评估

难度
2/5
预计耗时
1-3 小时
新手友好度
74/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
github-actions, markdown
领域
ci-cd, devtools

调研方向

阅读 .github/workflows/daily-community-attribution.md 和 .github/workflows/code-scanning-fixer.md 中的 agent 提示词正文，然后检查 edit 工具和 tools.bash 的允许列表是如何配置的。添加相关指导：使用 edit 写入文件，不要在允许的命令前加上 cd，并使用 jq 替代 python3；运行 make recompile 重新生成锁文件。两个提示词都包含这些指导且生成的锁文件已更新后，即为完成。

由索引模型根据 Issue 内容生成。

描述

automation observability optimization traces
Executive Summary

AgentRx analyzed 10 sampled gh-aw agent runs from the last 24h (prioritizing failed/highest-event-volume trajectories) via the pre-downloaded log bundle. The IR normalization stage completed for all 10 trajectories; the static/dynamic/check/judge LLM-backed stages could not run in this environment (see Limitations). Direct session-log inspection of the largest failure cluster found a concrete, reproducible root cause: the Copilot CLI agent repeatedly issues bash commands (multi-line heredocs, cd ..., python3 ...) that the workflow's tools.bash allowlist rejects, and the resulting stream of denials crashes the CLI driver (exitCode 1). This single pattern accounts for 2 of 4 occurrences in the dataset's single largest failure-boundary cluster.

AgentRx Evidence

  • Critical step: agent / "Execute GitHub Copilot CLI" (job: agent)

  • Failure category: agent_logic (failure_boundary signal, cluster failure-8169433961c9451a)

  • Frequency / impact: 4 runs / 30 failed runs in the 24h window (13.3% share) — the single largest failure_boundary cluster observed; every other failure_boundary cluster in the window was a singleton (count=1). Within this cluster, 2 of 4 runs share an identical, reproducible signature (bash permission denial → sdk_driver crash); the other 2 have distinct causes (rate limit, query error — see details).

  • Representative run IDs:

Violation classification

AgentRx's automated check/judge stages did not produce output in this environment (see Limitations), so this table is derived directly from raw session telemetry (usage/aw_session.jsonl) for the 10 sampled trajectories.

violation evidence fix_type rationale
Multi-line cat > file << 'EOF' ... EOF heredoc bash commands denied despite a matching "cat *" allowlist entry §41955226606 events #368→#370, #325→#327 ("Tool invocation is not allowed by workflow tool permissions"); same pattern in §41944783819 prompt tightening to reduce invalid tool invocations The agent defaults to heredoc-style file writes even though the workflow enables the edit tool for exactly this purpose. Prefix-glob bash permission matching does not reliably match multi-line command bodies, so every heredoc attempt is denied and retried until the turn budget is exhausted and the CLI driver crashes (exitCode: 1, errorTypes: ["sdk_driver"]).
Compound cd <dir> && jq ... commands denied (cd absent from every observed bash allowlist) §41955226606 events #384, #467 prompt tightening to reduce invalid tool invocations The agent repeatedly prefixes otherwise-allowed commands (jq) with cd, which no sampled workflow allowlists, so the whole compound command is rejected each time.
python3 << 'EOF' ... EOF denied (python3 absent from allowlist) §41955226606 event #514 prompt tightening to reduce invalid tool invocations Agent falls back to inline Python for JSON/report generation instead of the allowlisted jq or the enabled edit tool.
Invocation cap exceeded / HTTP 429 after an unusually long trajectory §41925400471 (Avenger): categories: ["invocation_cap_exceeded"], errorCodes: [429], 1011 session substeps (3-10x any other sampled run) improving retry/backoff strategy By far the highest event volume among the 10 sampled trajectories, consistent with a tight retry loop burning through the rate-limit budget rather than backing off.
driver_exit failures in isolated infra/setup steps (repo checkout, TLA+ runtime prep, Cursor CLI, Codex CLI) §41940379704 (Smoke Drive), §41941812478 (Smoke Update Cross-Repo PR), §41972802666 (Daily Workflow Security Model), §41941619530 (Smoke Cursor) adding missing telemetry attributes for better triage Each is a singleton failure_boundary cluster with a distinct failed step name and no further session-level error detail available in the collected logs — these need step-level error capture before a shared fix can be identified.
AgentRx Artifacts

  • IR summary: trajectory_ir.json normalized all 10 sampled trajectories successfully (flash-domain converter, no LLM fallback needed) — substep counts per trajectory: 41955226606=687, 41944783819=664, 41925400471=1011, 41924480521=392, 41940379704=131, 41941812478=101, 41972802666=105, 41941619530=186, 41969455502=167, 41961870265=255.

  • Invariant/checker highlights: Not available — see limitations below.

  • Judge classification output: Not available — see limitations below.

  • Known limitations:

    • The static, dynamic, check, judge, and report pipeline stages all failed identically with SystemError: The installed pydantic-core version (2.50.2) is incompatible with the current pydantic version, which requires 2.50.1 while importing agentrx.llm_clients.azure → openai → pydantic. This is a pre-existing dependency mismatch in the installed AgentRx virtualenv, not something fixable here (no pip install/env-repair was performed, per instructions). Only the ir stage (no LLM dependency) completed.
    • The findings above are therefore grounded directly in raw aw_session.jsonl event inspection rather than AgentRx's automated invariant/judge output.
    • Log collection hit its count/storage limit for the 24h window (continuation cursor in summary.json pointed to an earlier start_date/before_run_id); this analysis covers the 25 of 30 unique failed runs with observed evidence that were already clustered, not the full 24h window. 7 of 15 eligible failure clusters remain unreviewed.
Recommended Optimization
  • Change: Add a short "File writes" guidance block to the agent prompt body of .github/workflows/daily-community-attribution.md and .github/workflows/code-scanning-fixer.md, instructing the agent to use the already-enabled edit tool for creating or appending file content instead of bash heredocs (cat > file << EOF), and to never prefix allowlisted commands with cd (use absolute paths) or invoke python3 (use jq, which is already allowlisted, for JSON transforms).
  • Why highest impact: This is the only repeating, reproducible signature in the largest failure cluster in the window — it directly causes a hard CLI driver crash (exitCode 1) after exhausting denied attempts, rather than a graceful degradation, in 2 of 4 occurrences. The fix is a markdown prompt change plus make recompile — no code change required — and the same tools.bash allowlist + edit tool combination is shared by other copilot-engine workflows in this repo, so the same guidance is worth propagating wherever that pattern recurs.
  • Where to implement: Prompt bodies of .github/workflows/daily-community-attribution.md and .github/workflows/code-scanning-fixer.md (the workflow source .md files, followed by make recompile to regenerate the .lock.yml files).
Validation Plan
  • Re-run both workflows and confirm usage/aw_session.jsonl no longer contains tool.execution_complete events with error.code: "denied" during the "Execute GitHub Copilot CLI" step.
  • Track cluster failure-8169433961c9451a (agent/"Execute GitHub Copilot CLI", failure_kind: agent_logic) in the next AgentRx run — expect its count to drop from 4 toward 2 (only the unrelated rate-limit and BYOK query-400 runs remaining).
  • Expected success-metric change: the built-in tool-success-rate grader for these two workflows should rise from the observed 0.76 toward ≥0.95, and tool-failure-count should drop from the observed 6–7 per run to 0–1.
References

Generated by ⚡ Daily AgentRx Trace Optimizer · claude · agent · ~131 AIC · ⌖ ~1 AIC · ⊞ 4.4K · ◷

  • expires on Nov 11, 2026, 9:45 PM UTC-08:00
主要语言
Go
星标
5.6k
派生
615
平均合并
5 小时 36 分钟
30 天内合并 PR
805

环境准备

在 Codespaces 中打开

在浏览器里用你自己的 GitHub 账号启动这个项目的开发容器。

  • 提供 Dockerfile 或 Docker Compose 文件
  • 没有 Pull Request 模板
  • 阅读贡献指南

从这里开始

  1. 先读完整个 Issue，再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库，在一个分支上完成修改。
  4. 提交 Pull Request，并在描述里引用这个 Issue 编号。

github/gh-aw 的其他 Issue

查看 github/gh-aw 的全部 Issue

相似的 Issue

更多 Go Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。