[agentrx-optimizer] Daily Workflow Optimization - 2026-11-05

Đang mở Phù hợp với người mới
  • Có gì không ổn?
#75,106 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
2/5
Thời gian dự kiến
1-3 giờ
Mức phù hợp với người mới
74/100
Loại issue
Tính năng
Độ rõ ràng
Đặc tả rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
github-actions, markdown
Lĩnh vực
ci-cd, devtools

Hướng nghiên cứu

Đọc nội dung các prompt của agent trong .github/workflows/daily-community-attribution.md và .github/workflows/code-scanning-fixer.md, sau đó kiểm tra cách cấu hình danh sách cho phép của công cụ edit và tools.bash. Bổ sung hướng dẫn dùng edit để ghi tệp, tránh thêm cd trước các lệnh được cho phép và dùng jq thay cho python3; chạy make recompile để tạo lại các tệp khóa. Hoàn tất khi cả hai prompt đều có hướng dẫn này và các tệp khóa được tạo đã được cập nhật.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

automation observability optimization traces
Executive Summary

AgentRx analyzed 10 sampled gh-aw agent runs from the last 24h (prioritizing failed/highest-event-volume trajectories) via the pre-downloaded log bundle. The IR normalization stage completed for all 10 trajectories; the static/dynamic/check/judge LLM-backed stages could not run in this environment (see Limitations). Direct session-log inspection of the largest failure cluster found a concrete, reproducible root cause: the Copilot CLI agent repeatedly issues bash commands (multi-line heredocs, cd ..., python3 ...) that the workflow's tools.bash allowlist rejects, and the resulting stream of denials crashes the CLI driver (exitCode 1). This single pattern accounts for 2 of 4 occurrences in the dataset's single largest failure-boundary cluster.

AgentRx Evidence

  • Critical step: agent / "Execute GitHub Copilot CLI" (job: agent)

  • Failure category: agent_logic (failure_boundary signal, cluster failure-8169433961c9451a)

  • Frequency / impact: 4 runs / 30 failed runs in the 24h window (13.3% share) — the single largest failure_boundary cluster observed; every other failure_boundary cluster in the window was a singleton (count=1). Within this cluster, 2 of 4 runs share an identical, reproducible signature (bash permission denial → sdk_driver crash); the other 2 have distinct causes (rate limit, query error — see details).

  • Representative run IDs:

Violation classification

AgentRx's automated check/judge stages did not produce output in this environment (see Limitations), so this table is derived directly from raw session telemetry (usage/aw_session.jsonl) for the 10 sampled trajectories.

violation evidence fix_type rationale
Multi-line cat > file << 'EOF' ... EOF heredoc bash commands denied despite a matching "cat *" allowlist entry §41955226606 events #368→#370, #325→#327 ("Tool invocation is not allowed by workflow tool permissions"); same pattern in §41944783819 prompt tightening to reduce invalid tool invocations The agent defaults to heredoc-style file writes even though the workflow enables the edit tool for exactly this purpose. Prefix-glob bash permission matching does not reliably match multi-line command bodies, so every heredoc attempt is denied and retried until the turn budget is exhausted and the CLI driver crashes (exitCode: 1, errorTypes: ["sdk_driver"]).
Compound cd <dir> && jq ... commands denied (cd absent from every observed bash allowlist) §41955226606 events #384, #467 prompt tightening to reduce invalid tool invocations The agent repeatedly prefixes otherwise-allowed commands (jq) with cd, which no sampled workflow allowlists, so the whole compound command is rejected each time.
python3 << 'EOF' ... EOF denied (python3 absent from allowlist) §41955226606 event #514 prompt tightening to reduce invalid tool invocations Agent falls back to inline Python for JSON/report generation instead of the allowlisted jq or the enabled edit tool.
Invocation cap exceeded / HTTP 429 after an unusually long trajectory §41925400471 (Avenger): categories: ["invocation_cap_exceeded"], errorCodes: [429], 1011 session substeps (3-10x any other sampled run) improving retry/backoff strategy By far the highest event volume among the 10 sampled trajectories, consistent with a tight retry loop burning through the rate-limit budget rather than backing off.
driver_exit failures in isolated infra/setup steps (repo checkout, TLA+ runtime prep, Cursor CLI, Codex CLI) §41940379704 (Smoke Drive), §41941812478 (Smoke Update Cross-Repo PR), §41972802666 (Daily Workflow Security Model), §41941619530 (Smoke Cursor) adding missing telemetry attributes for better triage Each is a singleton failure_boundary cluster with a distinct failed step name and no further session-level error detail available in the collected logs — these need step-level error capture before a shared fix can be identified.
AgentRx Artifacts

  • IR summary: trajectory_ir.json normalized all 10 sampled trajectories successfully (flash-domain converter, no LLM fallback needed) — substep counts per trajectory: 41955226606=687, 41944783819=664, 41925400471=1011, 41924480521=392, 41940379704=131, 41941812478=101, 41972802666=105, 41941619530=186, 41969455502=167, 41961870265=255.

  • Invariant/checker highlights: Not available — see limitations below.

  • Judge classification output: Not available — see limitations below.

  • Known limitations:

    • The static, dynamic, check, judge, and report pipeline stages all failed identically with SystemError: The installed pydantic-core version (2.50.2) is incompatible with the current pydantic version, which requires 2.50.1 while importing agentrx.llm_clients.azure → openai → pydantic. This is a pre-existing dependency mismatch in the installed AgentRx virtualenv, not something fixable here (no pip install/env-repair was performed, per instructions). Only the ir stage (no LLM dependency) completed.
    • The findings above are therefore grounded directly in raw aw_session.jsonl event inspection rather than AgentRx's automated invariant/judge output.
    • Log collection hit its count/storage limit for the 24h window (continuation cursor in summary.json pointed to an earlier start_date/before_run_id); this analysis covers the 25 of 30 unique failed runs with observed evidence that were already clustered, not the full 24h window. 7 of 15 eligible failure clusters remain unreviewed.
Recommended Optimization
  • Change: Add a short "File writes" guidance block to the agent prompt body of .github/workflows/daily-community-attribution.md and .github/workflows/code-scanning-fixer.md, instructing the agent to use the already-enabled edit tool for creating or appending file content instead of bash heredocs (cat > file << EOF), and to never prefix allowlisted commands with cd (use absolute paths) or invoke python3 (use jq, which is already allowlisted, for JSON transforms).
  • Why highest impact: This is the only repeating, reproducible signature in the largest failure cluster in the window — it directly causes a hard CLI driver crash (exitCode 1) after exhausting denied attempts, rather than a graceful degradation, in 2 of 4 occurrences. The fix is a markdown prompt change plus make recompile — no code change required — and the same tools.bash allowlist + edit tool combination is shared by other copilot-engine workflows in this repo, so the same guidance is worth propagating wherever that pattern recurs.
  • Where to implement: Prompt bodies of .github/workflows/daily-community-attribution.md and .github/workflows/code-scanning-fixer.md (the workflow source .md files, followed by make recompile to regenerate the .lock.yml files).
Validation Plan
  • Re-run both workflows and confirm usage/aw_session.jsonl no longer contains tool.execution_complete events with error.code: "denied" during the "Execute GitHub Copilot CLI" step.
  • Track cluster failure-8169433961c9451a (agent/"Execute GitHub Copilot CLI", failure_kind: agent_logic) in the next AgentRx run — expect its count to drop from 4 toward 2 (only the unrelated rate-limit and BYOK query-400 runs remaining).
  • Expected success-metric change: the built-in tool-success-rate grader for these two workflows should rise from the observed 0.76 toward ≥0.95, and tool-failure-count should drop from the observed 6–7 per run to 0–1.
References

Generated by ⚡ Daily AgentRx Trace Optimizer · claude · agent · ~131 AIC · ⌖ ~1 AIC · ⊞ 4.4K · ◷

  • expires on Nov 11, 2026, 9:45 PM UTC-08:00
Ngôn ngữ chính
Go
Star
5.6k
Fork
615
Merge trung bình
5 giờ 36 phút
Pull request đã merge (30 ngày)
805

Chuẩn bị môi trường

Mở trong Codespaces

Khởi chạy dev container của dự án ngay trên trình duyệt, bằng tài khoản GitHub của bạn.

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của github/gh-aw

Tất cả issue của github/gh-aw

Issue tương tự

Thêm issue về Go

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.