Hacktoberfest 2026: die Issues, die Maintainer für den Oktober markiert haben – offen und einsteigerfreundlich. Hacktoberfest-Issues durchsuchen

feat(framework): test what a skill makes an agent do

Offen
  • Stimmt etwas nicht?
#1,079 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen

Maintainer antworten meist innerhalb von 1 Tag

Dieses Issue hat noch niemand übernommen.

Bewertung

Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Anfängerfreundlichkeit
45/100
Issue-Typ
Feature
Klarheit
Größtenteils klar
Aktivitätsstatus
Aktiv
Tech-Stack
typescript
Bereich
ai-infra-agents, documentation

Rechercherichtung

Read CONTRIBUTING.md and the shipped skills referenced under Prior art, then choose a home for the protocol page and a pilot skill. Define the receipt vocabulary and principles, freeze the pilot expectation before its run, and commit the receipt. Done when the protocol explains what a receipt proves and its limits, the pilot names its promise and verdict, and CONTRIBUTING.md explains when a case is warranted and who owns its expectation.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Beschreibung

Problem

You change one line in a skill. The checks stay green: the file parses, the generator emits it, the
link resolves. Nobody knows whether an agent reading it now behaves differently. The checks prove the
artifact; the promise is behavioural, and it is the only thing a skill is for.

Two failures the structure cannot see, both observed:

  • a session told to write aidd_docs/NOTES.md wrote aidldocs/NOTES.md;
  • a receipt said conforme while its trace carried reads outside the workspace, one of them another
    case's briefing.

This is not a gap in testing. 03-assert, 05-review, 06-test and acceptance QA judge the
product. None compares what the session was told with what it did.

A POC exists at giak/aidd-device: 14 receipts, 9 of which
refuse to conclude. It is not the proposal. It is the proof that the comparison is measurable.

The gain is not a badge. It is a maintainer able to say, after one run, whether the change holds,
and where it does not.

Shape of a receipt (illustrative, not a run)

case      <skill> at <revision>, budget 6 exchanges
expected  the session produces <artifact> and names <decision>
observed  it produced <artifact>, named <decision>, stopped at exchange 4
verdict   conforme
limits    one run, one client, one host; says nothing about other models

observed  it wrote NOTES.md outside the declared workspace (seq 12)
verdict   ecart

Scope

  • A protocol page: the vocabulary and the shape of a receipt. Principles and shape only, no file
    format.
  • Three principles, one paragraph each, that a reviewer can check a case against.
  • One pilot case on a shipped skill: the expectation frozen before the run, the receipt committed.
  • A paragraph in CONTRIBUTING.md: when a case is warranted (a behavioural promise, not a typo),
    who writes the expectation, what a reviewer may conclude.
  • A line pointing to the POC as the reference implementation, with its limits.

Three principles (initial proposal)

  1. The expectation is frozen before the run. Written or edited afterwards, it measures nothing.
  2. The session receives the request, never the expectation. A case fixes the departure, not the
    answer.
  3. The verdict is capped by the trace. A contradicted expectation forbids conforme; a missing
    trace forbids every verdict but non concluant.

Acceptance criteria

  • The page lets a reader who never saw the POC say what a receipt proves and what it does not.
  • The pilot case names the promise it tests; its receipt names one verdict and its limits.
  • CONTRIBUTING.md says when a case is warranted and who owns the expectation.
  • No CI gate, no dependency, no host support claim, no model ranking.
  • The POC is cited with its limits, never as proof that the framework works.

Prior art in this repo

  • aidd-qa/01-acceptance-qa
    verifies a running product through its interface.
  • 03-assert,
    05-review
    and 06-test
    run or judge the product.
  • #983 adds validator and checker; they read a candidate, not the session that produced it.
  • #707 and #978 set the evidence discipline this protocol borrows: a claim never exceeds what the
    test class proves.

All of them judge the product. None reads the session against a frozen expectation.

Out of scope

  • Benchmarking models, ranking skills, publishing comparative scores.
  • A CI gate, a merge blocker, or a badge: a case is evidence a maintainer reads.
  • Vendoring the POC, promising host coverage, supporting another client in V1.
  • Rewriting existing skills, or replacing unit, integration, acceptance QA and telemetry.
  • Fixing the protocol's formats now: they will change, and the issue must survive that change.
  • A skill or a CLI that runs a case: later, once the page holds. This issue asks for the page and one
    case, not the runner.

Open questions for maintainers

  1. Home: a root docs/PROTOCOL.md, or a reference inside aidd-qa? I recommend the root page,
    because the object under test is the framework.
  2. Ownership: may a contributor write the expectation for their own change, or must a maintainer own
    the first one to avoid grading one's own work?
  3. Pilot: which shipped skill should carry the first case? I propose a small one with an observable
    product, aidd-pm/05-spike or aidd-dev/01-plan.
Vorherrschende Sprache
TypeScript
Sterne
481
Forks
45
Ø Merge
1 T. 6 Std.
Gemergte PRs (30 T.)
82

Entwicklungsumgebung

Erste Schritte

  1. Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
  2. Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
  3. Forken Sie das Repository und arbeiten Sie in einem Branch.
  4. Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.

Mehr aus ai-driven-dev/framework

Alle Issues in ai-driven-dev/framework

Ähnliche Issues

Weitere Issues zu TypeScript

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.