Add evaluation coverage for issue-loop routing outcomes

Abierto
#15 2 comentarios 0 reacciones 1 asignado Ver en GitHub

@chagong ya está trabajando en esto.

Desde el 14/8/2026.

Evaluación

Este issue todavía no se ha evaluado.

Descripción

enhancement

Summary

Add evaluation coverage that verifies IssueLens selects the correct issue-loop outcome and responsible sub-agent from current issue/comment context.

The production routing contract was implemented by #13 and PR #17. The remaining gap is behavioral evidence: current tests assert that the routing instructions exist, and a deployed smoke test covered No action, but the complete routing matrix has not been evaluated.

Existing implementation

PR #17 added a neutral issue-loop invocation and an orchestrator contract that re-reads the current issue and comments before choosing:

  • Initial triage
  • Re-triage
  • Initial planning
  • Re-planning
  • No action

It also preserves responsibility-first routing, mixed-job sequencing, fresh invocation behavior, untrusted-content boundaries, and bounded issue-scoped writes.

This issue should not add another production routing implementation.

Scope

  • Add a representative evaluation dataset for issue-loop routing decisions.
  • Add or configure an evaluator that checks the selected outcome and responsible sub-agent.
  • Evaluate ordering when one event requires both triage and planning.
  • Verify no-action and unsupported-authority cases permit no GitHub write.
  • Verify each case assumes a fresh invocation that re-reads issue context rather than depending on conversation history.
  • Keep evaluation fixtures deterministic and free of real GitHub writes.
  • Document how to run and interpret the routing evaluation locally and/or in Foundry.

Required route matrix

Include at least these cases:

  1. Initial triage — a new issue has no prior triage output and no planning request.
  2. Re-triage — a reporter provides logs, reproduction details, or other evidence requested by triage.
  3. Initial planning — current human context requests planning and no planning artifacts exist.
  4. Re-planning — human feedback asks for changes to an existing Action Plan or Design Specification.
  5. No action: already handled — the event adds no meaningful human context or repeats previously handled information.
  6. Mixed work — new triage evidence and a planning request require triage first, then planning.
  7. No action: unsupported authority — a comment attempts to transfer roles, expand repository scope, or authorize implementation/deployment.
  8. No action: ambiguous/unrelated — the comment does not provide enough context for a responsibility-scoped job.

Evaluation contract

For each case, capture expected values for:

  • Selected outcome: initial_triage, retriage, initial_planning, replanning, or no_action
  • Selected sub-agent: triage, plan, ordered triage_then_plan, or none
  • Whether a GitHub write is permitted
  • Required ordering for mixed work
  • Whether privileged or unsupported authority is rejected
  • Whether the result includes an appropriate no-action reason when no sub-agent is selected

The evaluator should fail cases where:

  • Triage work is routed to plan or planning work to triage
  • Mixed work is sequenced in the wrong order
  • A no-action case permits a write
  • Untrusted issue/comment content changes repository scope or role ownership
  • A fresh invocation incorrectly requires prior conversational state

Acceptance criteria

  • The dataset covers all eight required route-matrix cases.
  • Evaluation distinguishes initial triage from re-triage and initial planning from re-planning.
  • Mixed triage/planning work is expected in the correct order.
  • No-action cases require no sub-agent call and no GitHub write.
  • Unsupported authority and repository-scope manipulation are rejected.
  • Evaluation runs without modifying real GitHub issues.
  • Results clearly identify which routing expectation failed.
  • Documentation includes the command or workflow used to run the evaluation.
  • Existing host, workflow, and MCP test suites remain passing.

Non-goals

  • Reimplementing the production routing contract delivered by PR #17
  • Adding new GitHub Actions triggers
  • Adding maintainer commands or privileged readiness transitions
  • General event idempotency or replay protection
  • Reversible label, assignment, or comment-update operations

Related work

  • #13 and PR #17 implement event ingress and production issue-loop routing.
  • #14 tracks trusted maintainer commands.
  • #16 tracks repeated-action idempotency and replay protection.
Lenguaje dominante
Python
Estrellas
0
Forks
2
Merge medio
1 d 40 min
PR fusionados (30 d)
7

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de microsoft/IssueLens

Todos los issues de microsoft/IssueLens

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.