feat(framework): support portable multi-host skill evaluation

Đang mở
#1,237 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
5/5
Thời gian dự kiến
Hơn một tuần
Mức phù hợp với người mới
22/100
Loại issue
Tính năng
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
javascript
Lĩnh vực
testing-qa, tooling

Hướng nghiên cứu

Start with scripts/skill-eval.mjs, where the Claude-only launch, skill staging under .claude/skills/, judge handling and PASS/FAIL reporting live, and with scripts/skill-eval/cases.json and its README for the fixtures. Read the archived multi-host branch only for reference. Done means the default Claude invocation still passes its existing cases, --host=codex works with a deterministic fake-CLI test, and invalid --jobs or --repeat values fail before any host runs. The scope spans several subsystems and needs a design decision first, so it is not a good first task.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

Problem

The existing scripts/skill-eval.mjs runner is Claude Code-specific: it launches claude -p and stages skills under .claude/skills/. This prevents contributors using Codex CLI from running the same applicable behavioral evaluation cases through their native host.

The current harness also has no explicit INCOMPLETE outcome: skipped judge checks are marked ok: true, and case results are reported as PASS or FAIL. It parses --jobs and --repeat as numbers without validating positive integer values before launching hosts.

Earlier work in #992 added Codex execution, JSON assertions, filesystem snapshots, and explicit incomplete-result reporting; those unrelated changes were removed to keep #992 focused on Kilo-native generation following maintainer feedback. The historical implementation is preserved under a tag on the contributor's fork.

The need is distinct from #1079 (what constitutes trustworthy skill-behavior evidence) and #1236 (portable contributor AI resources). Without coordinated boundaries, these efforts could duplicate host launchers and skill-discovery test harnesses.

Scope

  • Extend the existing runner with --host=claude|codex, retaining Claude as the default and keeping existing cases compatible.
  • Introduce a minimal host-adapter boundary for native CLI execution, skill staging and discovery in isolated temporary projects, exit/error capture, and host/version/model metadata where available.
  • Keep portable cases and deterministic assertions independent of host adapters; explicitly report unsupported host features rather than silently skipping them.
  • Add a deterministic jsonValid assertion and optional before/after project-tree snapshots to detect unexpected mutations, without following symlinks outside the isolated workspace.
  • Define PASS, FAIL, and INCOMPLETE with distinct failure diagnostics. Skipped optional LLM-judge checks must not make otherwise complete deterministic evaluations incomplete; skipped required evidence must never produce a full PASS.
  • Validate positive integer --jobs and --repeat arguments before any host invocation.
  • Reuse skill-eval as the preferred behavioral execution infrastructure for the contributor skills investigated in #1236 where applicable; avoid a second independent host-execution or skill-discovery harness unless a concrete limitation is demonstrated. The runner must remain usable without aidd-contributor.
  • Maintain the boundary with #1079: this issue builds execution/assertion facilities, not an evidence protocol, grading methodology, or proof of complete skill compliance.
  • Start from current next; inspect the archived historical implementation (commit ccd32e3bd9075cce43be6bea4a7bb1c60a14e6dd) and port selectively, rather than cherry-picking unrelated changes from #992.
  • Keep model-backed evaluations local and opt-in. Deterministic runner tests may run in CI without authentication or paid-model calls. Design for later Kilo/OpenCode adapters without implementing them in this issue.

Acceptance criteria

  • Existing Claude evaluation cases and default invocation remain compatible, with regression tests.
  • One applicable evaluation fixture is exercised with both Claude Code and Codex CLI; report the observed host versions, commands, and outcomes (or explicitly document any unavailable runtime validation).
  • A deterministic fake-CLI test exercises each host adapter without model credentials and verifies skill staging, invocation, output, and error capture.
  • Unsupported host selection and capabilities produce explicit diagnostics; portable assertions do not contain host-specific launch logic.
  • Valid JSON passes jsonValid; malformed JSON fails it in isolated tests.
  • Snapshot tests detect creation, modification, and deletion while refusing to traverse external symlink targets.
  • Tested result classification distinguishes failed assertions, failed host invocation, unavailable required evidence, and omitted optional judge checks.
  • Invalid, zero, negative, and non-integer --jobs or --repeat values fail before host execution.
  • A documented contributor-skill example demonstrates how the #1236 POC could reuse the runner, without requiring an aidd-contributor installation or introducing a parallel harness.
  • Documentation explicitly separates this runner from the evidence methodology proposed in #1079.
  • No mandatory authenticated model-backed CI execution is introduced.

Prior art in this repo

Out of scope

  • Implementing an aidd-contributor plugin or distributing contributor instructions/rules/hooks (investigated by #1236).
  • Defining or replacing the behavioral evidence protocol and grading semantics of #1079.
  • Implementing Kilo Code, OpenCode, or every supported host in the initial change.
  • Benchmarking or ranking language models.
  • Introducing mandatory authenticated AI-backed CI gates.
  • Rewriting production skills, altering canonical plugin formats, or reintroducing unrelated Project Memory, hook, or generation changes from #992.
Ngôn ngữ chính
TypeScript
Star
481
Fork
45
Merge trung bình
1 ngày 13 giờ
Pull request đã merge (30 ngày)
91

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của ai-driven-dev/framework

Tất cả issue của ai-driven-dev/framework

Issue tương tự

Thêm issue về TypeScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.