[agentrx-optimizer] Daily Workflow Optimization - 2026-11-05

Aperta Adatta ai principianti
  • Qualcosa non va?
#75,106 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
2/5
Tempo stimato
1-3 ore
Idoneità per principianti
74/100
Tipo di issue
Funzionalità
Chiarezza
Specificata chiaramente
Stato di attività
Attiva
Stack tecnologico
github-actions, markdown
Ambito
ci-cd, devtools

Direzione di ricerca

Leggi i testi dei prompt dell’agente in .github/workflows/daily-community-attribution.md e .github/workflows/code-scanning-fixer.md, poi verifica come sono configurate le liste di autorizzazione dello strumento edit e di tools.bash. Aggiungi indicazioni per usare edit per scrivere nei file, evitare di anteporre cd ai comandi consentiti e usare jq invece di python3; esegui make recompile per rigenerare i file di lock. Il lavoro è concluso quando entrambi i prompt contengono le indicazioni e i file di lock generati sono aggiornati.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

automation observability optimization traces
Executive Summary

AgentRx analyzed 10 sampled gh-aw agent runs from the last 24h (prioritizing failed/highest-event-volume trajectories) via the pre-downloaded log bundle. The IR normalization stage completed for all 10 trajectories; the static/dynamic/check/judge LLM-backed stages could not run in this environment (see Limitations). Direct session-log inspection of the largest failure cluster found a concrete, reproducible root cause: the Copilot CLI agent repeatedly issues bash commands (multi-line heredocs, cd ..., python3 ...) that the workflow's tools.bash allowlist rejects, and the resulting stream of denials crashes the CLI driver (exitCode 1). This single pattern accounts for 2 of 4 occurrences in the dataset's single largest failure-boundary cluster.

AgentRx Evidence

  • Critical step: agent / "Execute GitHub Copilot CLI" (job: agent)

  • Failure category: agent_logic (failure_boundary signal, cluster failure-8169433961c9451a)

  • Frequency / impact: 4 runs / 30 failed runs in the 24h window (13.3% share) — the single largest failure_boundary cluster observed; every other failure_boundary cluster in the window was a singleton (count=1). Within this cluster, 2 of 4 runs share an identical, reproducible signature (bash permission denial → sdk_driver crash); the other 2 have distinct causes (rate limit, query error — see details).

  • Representative run IDs:

Violation classification

AgentRx's automated check/judge stages did not produce output in this environment (see Limitations), so this table is derived directly from raw session telemetry (usage/aw_session.jsonl) for the 10 sampled trajectories.

violation evidence fix_type rationale
Multi-line cat > file << 'EOF' ... EOF heredoc bash commands denied despite a matching "cat *" allowlist entry §41955226606 events #368→#370, #325→#327 ("Tool invocation is not allowed by workflow tool permissions"); same pattern in §41944783819 prompt tightening to reduce invalid tool invocations The agent defaults to heredoc-style file writes even though the workflow enables the edit tool for exactly this purpose. Prefix-glob bash permission matching does not reliably match multi-line command bodies, so every heredoc attempt is denied and retried until the turn budget is exhausted and the CLI driver crashes (exitCode: 1, errorTypes: ["sdk_driver"]).
Compound cd <dir> && jq ... commands denied (cd absent from every observed bash allowlist) §41955226606 events #384, #467 prompt tightening to reduce invalid tool invocations The agent repeatedly prefixes otherwise-allowed commands (jq) with cd, which no sampled workflow allowlists, so the whole compound command is rejected each time.
python3 << 'EOF' ... EOF denied (python3 absent from allowlist) §41955226606 event #514 prompt tightening to reduce invalid tool invocations Agent falls back to inline Python for JSON/report generation instead of the allowlisted jq or the enabled edit tool.
Invocation cap exceeded / HTTP 429 after an unusually long trajectory §41925400471 (Avenger): categories: ["invocation_cap_exceeded"], errorCodes: [429], 1011 session substeps (3-10x any other sampled run) improving retry/backoff strategy By far the highest event volume among the 10 sampled trajectories, consistent with a tight retry loop burning through the rate-limit budget rather than backing off.
driver_exit failures in isolated infra/setup steps (repo checkout, TLA+ runtime prep, Cursor CLI, Codex CLI) §41940379704 (Smoke Drive), §41941812478 (Smoke Update Cross-Repo PR), §41972802666 (Daily Workflow Security Model), §41941619530 (Smoke Cursor) adding missing telemetry attributes for better triage Each is a singleton failure_boundary cluster with a distinct failed step name and no further session-level error detail available in the collected logs — these need step-level error capture before a shared fix can be identified.
AgentRx Artifacts

  • IR summary: trajectory_ir.json normalized all 10 sampled trajectories successfully (flash-domain converter, no LLM fallback needed) — substep counts per trajectory: 41955226606=687, 41944783819=664, 41925400471=1011, 41924480521=392, 41940379704=131, 41941812478=101, 41972802666=105, 41941619530=186, 41969455502=167, 41961870265=255.

  • Invariant/checker highlights: Not available — see limitations below.

  • Judge classification output: Not available — see limitations below.

  • Known limitations:

    • The static, dynamic, check, judge, and report pipeline stages all failed identically with SystemError: The installed pydantic-core version (2.50.2) is incompatible with the current pydantic version, which requires 2.50.1 while importing agentrx.llm_clients.azure → openai → pydantic. This is a pre-existing dependency mismatch in the installed AgentRx virtualenv, not something fixable here (no pip install/env-repair was performed, per instructions). Only the ir stage (no LLM dependency) completed.
    • The findings above are therefore grounded directly in raw aw_session.jsonl event inspection rather than AgentRx's automated invariant/judge output.
    • Log collection hit its count/storage limit for the 24h window (continuation cursor in summary.json pointed to an earlier start_date/before_run_id); this analysis covers the 25 of 30 unique failed runs with observed evidence that were already clustered, not the full 24h window. 7 of 15 eligible failure clusters remain unreviewed.
Recommended Optimization
  • Change: Add a short "File writes" guidance block to the agent prompt body of .github/workflows/daily-community-attribution.md and .github/workflows/code-scanning-fixer.md, instructing the agent to use the already-enabled edit tool for creating or appending file content instead of bash heredocs (cat > file << EOF), and to never prefix allowlisted commands with cd (use absolute paths) or invoke python3 (use jq, which is already allowlisted, for JSON transforms).
  • Why highest impact: This is the only repeating, reproducible signature in the largest failure cluster in the window — it directly causes a hard CLI driver crash (exitCode 1) after exhausting denied attempts, rather than a graceful degradation, in 2 of 4 occurrences. The fix is a markdown prompt change plus make recompile — no code change required — and the same tools.bash allowlist + edit tool combination is shared by other copilot-engine workflows in this repo, so the same guidance is worth propagating wherever that pattern recurs.
  • Where to implement: Prompt bodies of .github/workflows/daily-community-attribution.md and .github/workflows/code-scanning-fixer.md (the workflow source .md files, followed by make recompile to regenerate the .lock.yml files).
Validation Plan
  • Re-run both workflows and confirm usage/aw_session.jsonl no longer contains tool.execution_complete events with error.code: "denied" during the "Execute GitHub Copilot CLI" step.
  • Track cluster failure-8169433961c9451a (agent/"Execute GitHub Copilot CLI", failure_kind: agent_logic) in the next AgentRx run — expect its count to drop from 4 toward 2 (only the unrelated rate-limit and BYOK query-400 runs remaining).
  • Expected success-metric change: the built-in tool-success-rate grader for these two workflows should rise from the observed 0.76 toward ≥0.95, and tool-failure-count should drop from the observed 6–7 per run to 0–1.
References

Generated by ⚡ Daily AgentRx Trace Optimizer · claude · agent · ~131 AIC · ⌖ ~1 AIC · ⊞ 4.4K · ◷

  • expires on Nov 11, 2026, 9:45 PM UTC-08:00
Lingua principale
Go
Stelle
5.6k
Fork
615
Merge medio
5h 36m
PR unite (30g)
805

Preparare l'ambiente

Apri in Codespaces

Avvia il container di sviluppo del progetto nel browser, con il tuo account GitHub.

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di github/gh-aw

Tutte le issue di github/gh-aw

Issue simili

Altre issue su Go

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.