[agentrx-optimizer] Daily Workflow Optimization - 2026-11-05

Open Beginner friendly
  • Is something wrong?
#75,106 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
74/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
github-actions, markdown
Domain
ci-cd, devtools

Research direction

Read the agent prompt bodies in .github/workflows/daily-community-attribution.md and .github/workflows/code-scanning-fixer.md, then check how their edit tool and tools.bash allowlists are configured. Add guidance to use edit for file writes, avoid prefixing allowed commands with cd, and use jq instead of python3; run make recompile to regenerate the lock files. Done when both prompts contain the guidance and the generated lock files are updated.

Written by the indexing model from the issue text.

Description

automation observability optimization traces
Executive Summary

AgentRx analyzed 10 sampled gh-aw agent runs from the last 24h (prioritizing failed/highest-event-volume trajectories) via the pre-downloaded log bundle. The IR normalization stage completed for all 10 trajectories; the static/dynamic/check/judge LLM-backed stages could not run in this environment (see Limitations). Direct session-log inspection of the largest failure cluster found a concrete, reproducible root cause: the Copilot CLI agent repeatedly issues bash commands (multi-line heredocs, cd ..., python3 ...) that the workflow's tools.bash allowlist rejects, and the resulting stream of denials crashes the CLI driver (exitCode 1). This single pattern accounts for 2 of 4 occurrences in the dataset's single largest failure-boundary cluster.

AgentRx Evidence

  • Critical step: agent / "Execute GitHub Copilot CLI" (job: agent)

  • Failure category: agent_logic (failure_boundary signal, cluster failure-8169433961c9451a)

  • Frequency / impact: 4 runs / 30 failed runs in the 24h window (13.3% share) — the single largest failure_boundary cluster observed; every other failure_boundary cluster in the window was a singleton (count=1). Within this cluster, 2 of 4 runs share an identical, reproducible signature (bash permission denial → sdk_driver crash); the other 2 have distinct causes (rate limit, query error — see details).

  • Representative run IDs:

Violation classification

AgentRx's automated check/judge stages did not produce output in this environment (see Limitations), so this table is derived directly from raw session telemetry (usage/aw_session.jsonl) for the 10 sampled trajectories.

violation evidence fix_type rationale
Multi-line cat > file << 'EOF' ... EOF heredoc bash commands denied despite a matching "cat *" allowlist entry §41955226606 events #368→#370, #325→#327 ("Tool invocation is not allowed by workflow tool permissions"); same pattern in §41944783819 prompt tightening to reduce invalid tool invocations The agent defaults to heredoc-style file writes even though the workflow enables the edit tool for exactly this purpose. Prefix-glob bash permission matching does not reliably match multi-line command bodies, so every heredoc attempt is denied and retried until the turn budget is exhausted and the CLI driver crashes (exitCode: 1, errorTypes: ["sdk_driver"]).
Compound cd <dir> && jq ... commands denied (cd absent from every observed bash allowlist) §41955226606 events #384, #467 prompt tightening to reduce invalid tool invocations The agent repeatedly prefixes otherwise-allowed commands (jq) with cd, which no sampled workflow allowlists, so the whole compound command is rejected each time.
python3 << 'EOF' ... EOF denied (python3 absent from allowlist) §41955226606 event #514 prompt tightening to reduce invalid tool invocations Agent falls back to inline Python for JSON/report generation instead of the allowlisted jq or the enabled edit tool.
Invocation cap exceeded / HTTP 429 after an unusually long trajectory §41925400471 (Avenger): categories: ["invocation_cap_exceeded"], errorCodes: [429], 1011 session substeps (3-10x any other sampled run) improving retry/backoff strategy By far the highest event volume among the 10 sampled trajectories, consistent with a tight retry loop burning through the rate-limit budget rather than backing off.
driver_exit failures in isolated infra/setup steps (repo checkout, TLA+ runtime prep, Cursor CLI, Codex CLI) §41940379704 (Smoke Drive), §41941812478 (Smoke Update Cross-Repo PR), §41972802666 (Daily Workflow Security Model), §41941619530 (Smoke Cursor) adding missing telemetry attributes for better triage Each is a singleton failure_boundary cluster with a distinct failed step name and no further session-level error detail available in the collected logs — these need step-level error capture before a shared fix can be identified.
AgentRx Artifacts

  • IR summary: trajectory_ir.json normalized all 10 sampled trajectories successfully (flash-domain converter, no LLM fallback needed) — substep counts per trajectory: 41955226606=687, 41944783819=664, 41925400471=1011, 41924480521=392, 41940379704=131, 41941812478=101, 41972802666=105, 41941619530=186, 41969455502=167, 41961870265=255.

  • Invariant/checker highlights: Not available — see limitations below.

  • Judge classification output: Not available — see limitations below.

  • Known limitations:

    • The static, dynamic, check, judge, and report pipeline stages all failed identically with SystemError: The installed pydantic-core version (2.50.2) is incompatible with the current pydantic version, which requires 2.50.1 while importing agentrx.llm_clients.azure → openai → pydantic. This is a pre-existing dependency mismatch in the installed AgentRx virtualenv, not something fixable here (no pip install/env-repair was performed, per instructions). Only the ir stage (no LLM dependency) completed.
    • The findings above are therefore grounded directly in raw aw_session.jsonl event inspection rather than AgentRx's automated invariant/judge output.
    • Log collection hit its count/storage limit for the 24h window (continuation cursor in summary.json pointed to an earlier start_date/before_run_id); this analysis covers the 25 of 30 unique failed runs with observed evidence that were already clustered, not the full 24h window. 7 of 15 eligible failure clusters remain unreviewed.
Recommended Optimization
  • Change: Add a short "File writes" guidance block to the agent prompt body of .github/workflows/daily-community-attribution.md and .github/workflows/code-scanning-fixer.md, instructing the agent to use the already-enabled edit tool for creating or appending file content instead of bash heredocs (cat > file << EOF), and to never prefix allowlisted commands with cd (use absolute paths) or invoke python3 (use jq, which is already allowlisted, for JSON transforms).
  • Why highest impact: This is the only repeating, reproducible signature in the largest failure cluster in the window — it directly causes a hard CLI driver crash (exitCode 1) after exhausting denied attempts, rather than a graceful degradation, in 2 of 4 occurrences. The fix is a markdown prompt change plus make recompile — no code change required — and the same tools.bash allowlist + edit tool combination is shared by other copilot-engine workflows in this repo, so the same guidance is worth propagating wherever that pattern recurs.
  • Where to implement: Prompt bodies of .github/workflows/daily-community-attribution.md and .github/workflows/code-scanning-fixer.md (the workflow source .md files, followed by make recompile to regenerate the .lock.yml files).
Validation Plan
  • Re-run both workflows and confirm usage/aw_session.jsonl no longer contains tool.execution_complete events with error.code: "denied" during the "Execute GitHub Copilot CLI" step.
  • Track cluster failure-8169433961c9451a (agent/"Execute GitHub Copilot CLI", failure_kind: agent_logic) in the next AgentRx run — expect its count to drop from 4 toward 2 (only the unrelated rate-limit and BYOK query-400 runs remaining).
  • Expected success-metric change: the built-in tool-success-rate grader for these two workflows should rise from the observed 0.76 toward ≥0.95, and tool-failure-count should drop from the observed 6–7 per run to 0–1.
References

Generated by ⚡ Daily AgentRx Trace Optimizer · claude · agent · ~131 AIC · ⌖ ~1 AIC · ⊞ 4.4K · ◷

  • expires on Nov 11, 2026, 9:45 PM UTC-08:00
Dominant language
Go
Stars
5.6k
Forks
615
Avg merge
5h 36m
Merged PRs (30d)
805

Getting set up

Open in Codespaces

Starts the project's dev container in your browser, under your own GitHub account.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from github/gh-aw

All issues in github/gh-aw

Similar issues

More Go issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.