feat(framework): support portable multi-host skill evaluation

オープン
#1,237 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 1 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
22/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
javascript

調査の方向性

Start with scripts/skill-eval.mjs, where the Claude-only launch, skill staging under .claude/skills/, judge handling and PASS/FAIL reporting live, and with scripts/skill-eval/cases.json and its README for the fixtures. Read the archived multi-host branch only for reference. Done means the default Claude invocation still passes its existing cases, --host=codex works with a deterministic fake-CLI test, and invalid --jobs or --repeat values fail before any host runs. The scope spans several subsystems and needs a design decision first, so it is not a good first task.

索引モデルが issue の本文から書いたものです。

説明

Problem

The existing scripts/skill-eval.mjs runner is Claude Code-specific: it launches claude -p and stages skills under .claude/skills/. This prevents contributors using Codex CLI from running the same applicable behavioral evaluation cases through their native host.

The current harness also has no explicit INCOMPLETE outcome: skipped judge checks are marked ok: true, and case results are reported as PASS or FAIL. It parses --jobs and --repeat as numbers without validating positive integer values before launching hosts.

Earlier work in #992 added Codex execution, JSON assertions, filesystem snapshots, and explicit incomplete-result reporting; those unrelated changes were removed to keep #992 focused on Kilo-native generation following maintainer feedback. The historical implementation is preserved under a tag on the contributor's fork.

The need is distinct from #1079 (what constitutes trustworthy skill-behavior evidence) and #1236 (portable contributor AI resources). Without coordinated boundaries, these efforts could duplicate host launchers and skill-discovery test harnesses.

Scope

  • Extend the existing runner with --host=claude|codex, retaining Claude as the default and keeping existing cases compatible.
  • Introduce a minimal host-adapter boundary for native CLI execution, skill staging and discovery in isolated temporary projects, exit/error capture, and host/version/model metadata where available.
  • Keep portable cases and deterministic assertions independent of host adapters; explicitly report unsupported host features rather than silently skipping them.
  • Add a deterministic jsonValid assertion and optional before/after project-tree snapshots to detect unexpected mutations, without following symlinks outside the isolated workspace.
  • Define PASS, FAIL, and INCOMPLETE with distinct failure diagnostics. Skipped optional LLM-judge checks must not make otherwise complete deterministic evaluations incomplete; skipped required evidence must never produce a full PASS.
  • Validate positive integer --jobs and --repeat arguments before any host invocation.
  • Reuse skill-eval as the preferred behavioral execution infrastructure for the contributor skills investigated in #1236 where applicable; avoid a second independent host-execution or skill-discovery harness unless a concrete limitation is demonstrated. The runner must remain usable without aidd-contributor.
  • Maintain the boundary with #1079: this issue builds execution/assertion facilities, not an evidence protocol, grading methodology, or proof of complete skill compliance.
  • Start from current next; inspect the archived historical implementation (commit ccd32e3bd9075cce43be6bea4a7bb1c60a14e6dd) and port selectively, rather than cherry-picking unrelated changes from #992.
  • Keep model-backed evaluations local and opt-in. Deterministic runner tests may run in CI without authentication or paid-model calls. Design for later Kilo/OpenCode adapters without implementing them in this issue.

Acceptance criteria

  • Existing Claude evaluation cases and default invocation remain compatible, with regression tests.
  • One applicable evaluation fixture is exercised with both Claude Code and Codex CLI; report the observed host versions, commands, and outcomes (or explicitly document any unavailable runtime validation).
  • A deterministic fake-CLI test exercises each host adapter without model credentials and verifies skill staging, invocation, output, and error capture.
  • Unsupported host selection and capabilities produce explicit diagnostics; portable assertions do not contain host-specific launch logic.
  • Valid JSON passes jsonValid; malformed JSON fails it in isolated tests.
  • Snapshot tests detect creation, modification, and deletion while refusing to traverse external symlink targets.
  • Tested result classification distinguishes failed assertions, failed host invocation, unavailable required evidence, and omitted optional judge checks.
  • Invalid, zero, negative, and non-integer --jobs or --repeat values fail before host execution.
  • A documented contributor-skill example demonstrates how the #1236 POC could reuse the runner, without requiring an aidd-contributor installation or introducing a parallel harness.
  • Documentation explicitly separates this runner from the evidence methodology proposed in #1079.
  • No mandatory authenticated model-backed CI execution is introduced.

Prior art in this repo

Out of scope

  • Implementing an aidd-contributor plugin or distributing contributor instructions/rules/hooks (investigated by #1236).
  • Defining or replacing the behavioral evidence protocol and grading semantics of #1079.
  • Implementing Kilo Code, OpenCode, or every supported host in the initial change.
  • Benchmarking or ranking language models.
  • Introducing mandatory authenticated AI-backed CI gates.
  • Rewriting production skills, altering canonical plugin formats, or reintroducing unrelated Project Memory, hook, or generation changes from #992.
主要言語
TypeScript
スター
481
フォーク
45
平均マージ
1日 13時間
マージ済み PR(30日)
91

環境構築

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

ai-driven-dev/framework のほかの issue

ai-driven-dev/framework の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。