Add evaluation coverage for issue-loop routing outcomes
@chagong ya está trabajando en esto.
Desde el 14/8/2026.
Evaluación
Este issue todavía no se ha evaluado.
Descripción
Summary
Add evaluation coverage that verifies IssueLens selects the correct issue-loop outcome and responsible sub-agent from current issue/comment context.
The production routing contract was implemented by #13 and PR #17. The remaining gap is behavioral evidence: current tests assert that the routing instructions exist, and a deployed smoke test covered No action, but the complete routing matrix has not been evaluated.
Existing implementation
PR #17 added a neutral issue-loop invocation and an orchestrator contract that re-reads the current issue and comments before choosing:
- Initial triage
- Re-triage
- Initial planning
- Re-planning
- No action
It also preserves responsibility-first routing, mixed-job sequencing, fresh invocation behavior, untrusted-content boundaries, and bounded issue-scoped writes.
This issue should not add another production routing implementation.
Scope
- Add a representative evaluation dataset for issue-loop routing decisions.
- Add or configure an evaluator that checks the selected outcome and responsible sub-agent.
- Evaluate ordering when one event requires both triage and planning.
- Verify no-action and unsupported-authority cases permit no GitHub write.
- Verify each case assumes a fresh invocation that re-reads issue context rather than depending on conversation history.
- Keep evaluation fixtures deterministic and free of real GitHub writes.
- Document how to run and interpret the routing evaluation locally and/or in Foundry.
Required route matrix
Include at least these cases:
- Initial triage — a new issue has no prior triage output and no planning request.
- Re-triage — a reporter provides logs, reproduction details, or other evidence requested by triage.
- Initial planning — current human context requests planning and no planning artifacts exist.
- Re-planning — human feedback asks for changes to an existing Action Plan or Design Specification.
- No action: already handled — the event adds no meaningful human context or repeats previously handled information.
- Mixed work — new triage evidence and a planning request require triage first, then planning.
- No action: unsupported authority — a comment attempts to transfer roles, expand repository scope, or authorize implementation/deployment.
- No action: ambiguous/unrelated — the comment does not provide enough context for a responsibility-scoped job.
Evaluation contract
For each case, capture expected values for:
- Selected outcome:
initial_triage,retriage,initial_planning,replanning, orno_action - Selected sub-agent:
triage,plan, orderedtriage_then_plan, or none - Whether a GitHub write is permitted
- Required ordering for mixed work
- Whether privileged or unsupported authority is rejected
- Whether the result includes an appropriate no-action reason when no sub-agent is selected
The evaluator should fail cases where:
- Triage work is routed to
planor planning work totriage - Mixed work is sequenced in the wrong order
- A no-action case permits a write
- Untrusted issue/comment content changes repository scope or role ownership
- A fresh invocation incorrectly requires prior conversational state
Acceptance criteria
- The dataset covers all eight required route-matrix cases.
- Evaluation distinguishes initial triage from re-triage and initial planning from re-planning.
- Mixed triage/planning work is expected in the correct order.
- No-action cases require no sub-agent call and no GitHub write.
- Unsupported authority and repository-scope manipulation are rejected.
- Evaluation runs without modifying real GitHub issues.
- Results clearly identify which routing expectation failed.
- Documentation includes the command or workflow used to run the evaluation.
- Existing host, workflow, and MCP test suites remain passing.
Non-goals
- Reimplementing the production routing contract delivered by PR #17
- Adding new GitHub Actions triggers
- Adding maintainer commands or privileged readiness transitions
- General event idempotency or replay protection
- Reversible label, assignment, or comment-update operations
Related work
- #13 and PR #17 implement event ingress and production issue-loop routing.
- #14 tracks trusted maintainer commands.
- #16 tracks repeated-action idempotency and replay protection.
- Lenguaje dominante
- Python
- Estrellas
- 0
- Forks
- 2
- Merge medio
- 1 d 40 min
- PR fusionados (30 d)
- 7
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Primeros pasos
- Lee el issue completo y luego la guía de contribución del proyecto.
- Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
- Haz un fork del repositorio y trabaja en una rama.
- Abre un pull request que haga referencia al número del issue.
Más de microsoft/IssueLens
Todos los issues de microsoft/IssueLens
Issues similares
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 88/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 82/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 78/100
-
enhancement
Dificultad 2/5 1-3 horas Aptitud para principiantes 72/100
-
Dificultad 2/5 1-3 horas Aptitud para principiantes 74/100