Hacktoberfest 2026: những issue maintainer đã đánh dấu cho tháng Mười, đang mở và phù hợp người mới. Xem issue Hacktoberfest

feat(framework): test what a skill makes an agent do

Đang mở
  • Có gì không ổn?
#1,079 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
45/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
typescript
Lĩnh vực
ai-infra-agents, documentation

Hướng nghiên cứu

Read CONTRIBUTING.md and the shipped skills referenced under Prior art, then choose a home for the protocol page and a pilot skill. Define the receipt vocabulary and principles, freeze the pilot expectation before its run, and commit the receipt. Done when the protocol explains what a receipt proves and its limits, the pilot names its promise and verdict, and CONTRIBUTING.md explains when a case is warranted and who owns its expectation.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Problem

You change one line in a skill. The checks stay green: the file parses, the generator emits it, the
link resolves. Nobody knows whether an agent reading it now behaves differently. The checks prove the
artifact; the promise is behavioural, and it is the only thing a skill is for.

Two failures the structure cannot see, both observed:

  • a session told to write aidd_docs/NOTES.md wrote aidldocs/NOTES.md;
  • a receipt said conforme while its trace carried reads outside the workspace, one of them another
    case's briefing.

This is not a gap in testing. 03-assert, 05-review, 06-test and acceptance QA judge the
product. None compares what the session was told with what it did.

A POC exists at giak/aidd-device: 14 receipts, 9 of which
refuse to conclude. It is not the proposal. It is the proof that the comparison is measurable.

The gain is not a badge. It is a maintainer able to say, after one run, whether the change holds,
and where it does not.

Shape of a receipt (illustrative, not a run)

case      <skill> at <revision>, budget 6 exchanges
expected  the session produces <artifact> and names <decision>
observed  it produced <artifact>, named <decision>, stopped at exchange 4
verdict   conforme
limits    one run, one client, one host; says nothing about other models

observed  it wrote NOTES.md outside the declared workspace (seq 12)
verdict   ecart

Scope

  • A protocol page: the vocabulary and the shape of a receipt. Principles and shape only, no file
    format.
  • Three principles, one paragraph each, that a reviewer can check a case against.
  • One pilot case on a shipped skill: the expectation frozen before the run, the receipt committed.
  • A paragraph in CONTRIBUTING.md: when a case is warranted (a behavioural promise, not a typo),
    who writes the expectation, what a reviewer may conclude.
  • A line pointing to the POC as the reference implementation, with its limits.

Three principles (initial proposal)

  1. The expectation is frozen before the run. Written or edited afterwards, it measures nothing.
  2. The session receives the request, never the expectation. A case fixes the departure, not the
    answer.
  3. The verdict is capped by the trace. A contradicted expectation forbids conforme; a missing
    trace forbids every verdict but non concluant.

Acceptance criteria

  • The page lets a reader who never saw the POC say what a receipt proves and what it does not.
  • The pilot case names the promise it tests; its receipt names one verdict and its limits.
  • CONTRIBUTING.md says when a case is warranted and who owns the expectation.
  • No CI gate, no dependency, no host support claim, no model ranking.
  • The POC is cited with its limits, never as proof that the framework works.

Prior art in this repo

  • aidd-qa/01-acceptance-qa
    verifies a running product through its interface.
  • 03-assert,
    05-review
    and 06-test
    run or judge the product.
  • #983 adds validator and checker; they read a candidate, not the session that produced it.
  • #707 and #978 set the evidence discipline this protocol borrows: a claim never exceeds what the
    test class proves.

All of them judge the product. None reads the session against a frozen expectation.

Out of scope

  • Benchmarking models, ranking skills, publishing comparative scores.
  • A CI gate, a merge blocker, or a badge: a case is evidence a maintainer reads.
  • Vendoring the POC, promising host coverage, supporting another client in V1.
  • Rewriting existing skills, or replacing unit, integration, acceptance QA and telemetry.
  • Fixing the protocol's formats now: they will change, and the issue must survive that change.
  • A skill or a CLI that runs a case: later, once the page holds. This issue asks for the page and one
    case, not the runner.

Open questions for maintainers

  1. Home: a root docs/PROTOCOL.md, or a reference inside aidd-qa? I recommend the root page,
    because the object under test is the framework.
  2. Ownership: may a contributor write the expectation for their own change, or must a maintainer own
    the first one to avoid grading one's own work?
  3. Pilot: which shipped skill should carry the first case? I propose a small one with an observable
    product, aidd-pm/05-spike or aidd-dev/01-plan.
Ngôn ngữ chính
TypeScript
Star
481
Fork
45
Merge trung bình
1 ngày 6 giờ
Pull request đã merge (30 ngày)
82

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của ai-driven-dev/framework

Tất cả issue của ai-driven-dev/framework

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.