Hacktoberfest 2026: los issues que los mantenedores marcaron para octubre, abiertos y aptos para principiantes. Explorar issues de Hacktoberfest

feat(framework): test what a skill makes an agent do

Abierto
  • ¿Algo no está bien?
#1,079 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Los mantenedores suelen responder en 1 día

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
45/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
typescript
Área
ai-infra-agents, documentation

Línea de trabajo

Read CONTRIBUTING.md and the shipped skills referenced under Prior art, then choose a home for the protocol page and a pilot skill. Define the receipt vocabulary and principles, freeze the pilot expectation before its run, and commit the receipt. Done when the protocol explains what a receipt proves and its limits, the pilot names its promise and verdict, and CONTRIBUTING.md explains when a case is warranted and who owns its expectation.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Problem

You change one line in a skill. The checks stay green: the file parses, the generator emits it, the
link resolves. Nobody knows whether an agent reading it now behaves differently. The checks prove the
artifact; the promise is behavioural, and it is the only thing a skill is for.

Two failures the structure cannot see, both observed:

  • a session told to write aidd_docs/NOTES.md wrote aidldocs/NOTES.md;
  • a receipt said conforme while its trace carried reads outside the workspace, one of them another
    case's briefing.

This is not a gap in testing. 03-assert, 05-review, 06-test and acceptance QA judge the
product. None compares what the session was told with what it did.

A POC exists at giak/aidd-device: 14 receipts, 9 of which
refuse to conclude. It is not the proposal. It is the proof that the comparison is measurable.

The gain is not a badge. It is a maintainer able to say, after one run, whether the change holds,
and where it does not.

Shape of a receipt (illustrative, not a run)

case      <skill> at <revision>, budget 6 exchanges
expected  the session produces <artifact> and names <decision>
observed  it produced <artifact>, named <decision>, stopped at exchange 4
verdict   conforme
limits    one run, one client, one host; says nothing about other models

observed  it wrote NOTES.md outside the declared workspace (seq 12)
verdict   ecart

Scope

  • A protocol page: the vocabulary and the shape of a receipt. Principles and shape only, no file
    format.
  • Three principles, one paragraph each, that a reviewer can check a case against.
  • One pilot case on a shipped skill: the expectation frozen before the run, the receipt committed.
  • A paragraph in CONTRIBUTING.md: when a case is warranted (a behavioural promise, not a typo),
    who writes the expectation, what a reviewer may conclude.
  • A line pointing to the POC as the reference implementation, with its limits.

Three principles (initial proposal)

  1. The expectation is frozen before the run. Written or edited afterwards, it measures nothing.
  2. The session receives the request, never the expectation. A case fixes the departure, not the
    answer.
  3. The verdict is capped by the trace. A contradicted expectation forbids conforme; a missing
    trace forbids every verdict but non concluant.

Acceptance criteria

  • The page lets a reader who never saw the POC say what a receipt proves and what it does not.
  • The pilot case names the promise it tests; its receipt names one verdict and its limits.
  • CONTRIBUTING.md says when a case is warranted and who owns the expectation.
  • No CI gate, no dependency, no host support claim, no model ranking.
  • The POC is cited with its limits, never as proof that the framework works.

Prior art in this repo

  • aidd-qa/01-acceptance-qa
    verifies a running product through its interface.
  • 03-assert,
    05-review
    and 06-test
    run or judge the product.
  • #983 adds validator and checker; they read a candidate, not the session that produced it.
  • #707 and #978 set the evidence discipline this protocol borrows: a claim never exceeds what the
    test class proves.

All of them judge the product. None reads the session against a frozen expectation.

Out of scope

  • Benchmarking models, ranking skills, publishing comparative scores.
  • A CI gate, a merge blocker, or a badge: a case is evidence a maintainer reads.
  • Vendoring the POC, promising host coverage, supporting another client in V1.
  • Rewriting existing skills, or replacing unit, integration, acceptance QA and telemetry.
  • Fixing the protocol's formats now: they will change, and the issue must survive that change.
  • A skill or a CLI that runs a case: later, once the page holds. This issue asks for the page and one
    case, not the runner.

Open questions for maintainers

  1. Home: a root docs/PROTOCOL.md, or a reference inside aidd-qa? I recommend the root page,
    because the object under test is the framework.
  2. Ownership: may a contributor write the expectation for their own change, or must a maintainer own
    the first one to avoid grading one's own work?
  3. Pilot: which shipped skill should carry the first case? I propose a small one with an observable
    product, aidd-pm/05-spike or aidd-dev/01-plan.
Lenguaje dominante
TypeScript
Estrellas
481
Forks
45
Merge medio
1 d 6 h
PR fusionados (30 d)
82

Preparar el entorno

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de ai-driven-dev/framework

Todos los issues de ai-driven-dev/framework

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.