Hacktoberfest 2026：メンテナが10月に向けて印を付けた、オープンで初心者向けの issue。 Hacktoberfest の issue を見る

feat(framework): test what a skill makes an agent do

オープン
  • 何か問題がありますか？
#1,079 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
45/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
typescript
領域
ai-infra-agents, documentation

調査の方向性

Read CONTRIBUTING.md and the shipped skills referenced under Prior art, then choose a home for the protocol page and a pilot skill. Define the receipt vocabulary and principles, freeze the pilot expectation before its run, and commit the receipt. Done when the protocol explains what a receipt proves and its limits, the pilot names its promise and verdict, and CONTRIBUTING.md explains when a case is warranted and who owns its expectation.

索引モデルが issue の本文から書いたものです。

説明

Problem

You change one line in a skill. The checks stay green: the file parses, the generator emits it, the
link resolves. Nobody knows whether an agent reading it now behaves differently. The checks prove the
artifact; the promise is behavioural, and it is the only thing a skill is for.

Two failures the structure cannot see, both observed:

  • a session told to write aidd_docs/NOTES.md wrote aidldocs/NOTES.md;
  • a receipt said conforme while its trace carried reads outside the workspace, one of them another
    case's briefing.

This is not a gap in testing. 03-assert, 05-review, 06-test and acceptance QA judge the
product. None compares what the session was told with what it did.

A POC exists at giak/aidd-device: 14 receipts, 9 of which
refuse to conclude. It is not the proposal. It is the proof that the comparison is measurable.

The gain is not a badge. It is a maintainer able to say, after one run, whether the change holds,
and where it does not.

Shape of a receipt (illustrative, not a run)

case      <skill> at <revision>, budget 6 exchanges
expected  the session produces <artifact> and names <decision>
observed  it produced <artifact>, named <decision>, stopped at exchange 4
verdict   conforme
limits    one run, one client, one host; says nothing about other models

observed  it wrote NOTES.md outside the declared workspace (seq 12)
verdict   ecart

Scope

  • A protocol page: the vocabulary and the shape of a receipt. Principles and shape only, no file
    format.
  • Three principles, one paragraph each, that a reviewer can check a case against.
  • One pilot case on a shipped skill: the expectation frozen before the run, the receipt committed.
  • A paragraph in CONTRIBUTING.md: when a case is warranted (a behavioural promise, not a typo),
    who writes the expectation, what a reviewer may conclude.
  • A line pointing to the POC as the reference implementation, with its limits.

Three principles (initial proposal)

  1. The expectation is frozen before the run. Written or edited afterwards, it measures nothing.
  2. The session receives the request, never the expectation. A case fixes the departure, not the
    answer.
  3. The verdict is capped by the trace. A contradicted expectation forbids conforme; a missing
    trace forbids every verdict but non concluant.

Acceptance criteria

  • The page lets a reader who never saw the POC say what a receipt proves and what it does not.
  • The pilot case names the promise it tests; its receipt names one verdict and its limits.
  • CONTRIBUTING.md says when a case is warranted and who owns the expectation.
  • No CI gate, no dependency, no host support claim, no model ranking.
  • The POC is cited with its limits, never as proof that the framework works.

Prior art in this repo

  • aidd-qa/01-acceptance-qa
    verifies a running product through its interface.
  • 03-assert,
    05-review
    and 06-test
    run or judge the product.
  • #983 adds validator and checker; they read a candidate, not the session that produced it.
  • #707 and #978 set the evidence discipline this protocol borrows: a claim never exceeds what the
    test class proves.

All of them judge the product. None reads the session against a frozen expectation.

Out of scope

  • Benchmarking models, ranking skills, publishing comparative scores.
  • A CI gate, a merge blocker, or a badge: a case is evidence a maintainer reads.
  • Vendoring the POC, promising host coverage, supporting another client in V1.
  • Rewriting existing skills, or replacing unit, integration, acceptance QA and telemetry.
  • Fixing the protocol's formats now: they will change, and the issue must survive that change.
  • A skill or a CLI that runs a case: later, once the page holds. This issue asks for the page and one
    case, not the runner.

Open questions for maintainers

  1. Home: a root docs/PROTOCOL.md, or a reference inside aidd-qa? I recommend the root page,
    because the object under test is the framework.
  2. Ownership: may a contributor write the expectation for their own change, or must a maintainer own
    the first one to avoid grading one's own work?
  3. Pilot: which shipped skill should carry the first case? I propose a small one with an observable
    product, aidd-pm/05-spike or aidd-dev/01-plan.
主要言語
TypeScript
スター
481
フォーク
45
平均マージ
1日 6時間
マージ済み PR（30日）
82

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ai-driven-dev/framework のほかの issue

ai-driven-dev/framework の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。