feat(framework): support portable multi-host skill evaluation
Maintainer antworten meist innerhalb von 1 Tag
Dieses Issue hat noch niemand übernommen.
Bewertung
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Anfängerfreundlichkeit
- 22/100
- Issue-Typ
- Feature
- Klarheit
- Größtenteils klar
- Aktivitätsstatus
- Aktiv
- Tech-Stack
- javascript
- Bereich
- testing-qa, tooling
Rechercherichtung
Start with scripts/skill-eval.mjs, where the Claude-only launch, skill staging under .claude/skills/, judge handling and PASS/FAIL reporting live, and with scripts/skill-eval/cases.json and its README for the fixtures. Read the archived multi-host branch only for reference. Done means the default Claude invocation still passes its existing cases, --host=codex works with a deterministic fake-CLI test, and invalid --jobs or --repeat values fail before any host runs. The scope spans several subsystems and needs a design decision first, so it is not a good first task.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Beschreibung
Problem
The existing scripts/skill-eval.mjs runner is Claude Code-specific: it launches claude -p and stages skills under .claude/skills/. This prevents contributors using Codex CLI from running the same applicable behavioral evaluation cases through their native host.
The current harness also has no explicit INCOMPLETE outcome: skipped judge checks are marked ok: true, and case results are reported as PASS or FAIL. It parses --jobs and --repeat as numbers without validating positive integer values before launching hosts.
Earlier work in #992 added Codex execution, JSON assertions, filesystem snapshots, and explicit incomplete-result reporting; those unrelated changes were removed to keep #992 focused on Kilo-native generation following maintainer feedback. The historical implementation is preserved under a tag on the contributor's fork.
The need is distinct from #1079 (what constitutes trustworthy skill-behavior evidence) and #1236 (portable contributor AI resources). Without coordinated boundaries, these efforts could duplicate host launchers and skill-discovery test harnesses.
Scope
- Extend the existing runner with
--host=claude|codex, retaining Claude as the default and keeping existing cases compatible. - Introduce a minimal host-adapter boundary for native CLI execution, skill staging and discovery in isolated temporary projects, exit/error capture, and host/version/model metadata where available.
- Keep portable cases and deterministic assertions independent of host adapters; explicitly report unsupported host features rather than silently skipping them.
- Add a deterministic
jsonValidassertion and optional before/after project-tree snapshots to detect unexpected mutations, without following symlinks outside the isolated workspace. - Define
PASS,FAIL, andINCOMPLETEwith distinct failure diagnostics. Skipped optional LLM-judge checks must not make otherwise complete deterministic evaluations incomplete; skipped required evidence must never produce a full PASS. - Validate positive integer
--jobsand--repeatarguments before any host invocation. - Reuse
skill-evalas the preferred behavioral execution infrastructure for the contributor skills investigated in #1236 where applicable; avoid a second independent host-execution or skill-discovery harness unless a concrete limitation is demonstrated. The runner must remain usable withoutaidd-contributor. - Maintain the boundary with #1079: this issue builds execution/assertion facilities, not an evidence protocol, grading methodology, or proof of complete skill compliance.
- Start from current
next; inspect the archived historical implementation (commitccd32e3bd9075cce43be6bea4a7bb1c60a14e6dd) and port selectively, rather than cherry-picking unrelated changes from #992. - Keep model-backed evaluations local and opt-in. Deterministic runner tests may run in CI without authentication or paid-model calls. Design for later Kilo/OpenCode adapters without implementing them in this issue.
Acceptance criteria
- Existing Claude evaluation cases and default invocation remain compatible, with regression tests.
- One applicable evaluation fixture is exercised with both Claude Code and Codex CLI; report the observed host versions, commands, and outcomes (or explicitly document any unavailable runtime validation).
- A deterministic fake-CLI test exercises each host adapter without model credentials and verifies skill staging, invocation, output, and error capture.
- Unsupported host selection and capabilities produce explicit diagnostics; portable assertions do not contain host-specific launch logic.
- Valid JSON passes
jsonValid; malformed JSON fails it in isolated tests. - Snapshot tests detect creation, modification, and deletion while refusing to traverse external symlink targets.
- Tested result classification distinguishes failed assertions, failed host invocation, unavailable required evidence, and omitted optional judge checks.
- Invalid, zero, negative, and non-integer
--jobsor--repeatvalues fail before host execution. - A documented contributor-skill example demonstrates how the #1236 POC could reuse the runner, without requiring an
aidd-contributorinstallation or introducing a parallel harness. - Documentation explicitly separates this runner from the evidence methodology proposed in #1079.
- No mandatory authenticated model-backed CI execution is introduced.
Prior art in this repo
scripts/skill-eval.mjs, lines 37–52: Claude-only runner.scripts/skill-eval.mjs, lines 59–112: existing assertions and optional judge.scripts/skill-eval.mjs, lines 114–164: temporary project setup, skill staging, and result reporting.scripts/skill-eval/README.md, lines 3–30: current usage and prerequisites.scripts/skill-eval/cases.json: existing fixtures.- Historical multi-host implementation, archived tag, original commit
ccd32e3bd9075cce43be6bea4a7bb1c60a14e6dd. - #992 (original contribution); #1079 (behavioral evidence); #1236 (contributor portability); #929 and #978 (broader tool portability/compatibility).
Out of scope
- Implementing an
aidd-contributorplugin or distributing contributor instructions/rules/hooks (investigated by #1236). - Defining or replacing the behavioral evidence protocol and grading semantics of #1079.
- Implementing Kilo Code, OpenCode, or every supported host in the initial change.
- Benchmarking or ranking language models.
- Introducing mandatory authenticated AI-backed CI gates.
- Rewriting production skills, altering canonical plugin formats, or reintroducing unrelated Project Memory, hook, or generation changes from #992.
- Vorherrschende Sprache
- TypeScript
- Sterne
- 481
- Forks
- 45
- Ø Merge
- 1 T. 13 Std.
- Gemergte PRs (30 T.)
- 91
Entwicklungsumgebung
Erste Schritte
- Lesen Sie das ganze Issue und danach den Beitragsleitfaden des Projekts.
- Schreiben Sie ins Issue, dass Sie es übernehmen — das erspart doppelte Arbeit.
- Forken Sie das Repository und arbeiten Sie in einem Branch.
- Öffnen Sie einen Pull Request, der die Issue-Nummer nennt.
Mehr aus ai-driven-dev/framework
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 86/100
ai-driven-dev/framework#1311 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 79/100
ai-driven-dev/framework#1244 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 78/100
ai-driven-dev/framework#1197 ·
Maintainer antworten meist innerhalb von 1 Tag
-
refactor(aidd-context): apply the new skill template to learnEvtl. vergeben @blafourcade hat das vor 3 Tagen übernommen. Offen
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
ai-driven-dev/framework#1186 · 1 zugewiesene Person ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 83/100
ai-driven-dev/framework#1163 · 1 Kommentar ·
Maintainer antworten meist innerhalb von 1 Tag
Alle Issues in ai-driven-dev/framework
Ähnliche Issues
-
db-d1-sqlite: upsert is aliased to updateOne, so payload.kv.set never inserts new keysEvtl. vergeben @pcbeingused333 hat das heute übernommen. OffenBug db: d1-sqlite Payload OSS
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 76/100
payloadcms/payload#19766 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
device-management-toolkit/sample-web-ui#3711 ·
Maintainer antworten meist innerhalb von 1 Tag
-
bug impact:low Team: SecuritySolution Team:Defend Workflows v9.6.0
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 72/100
elastic/kibana#306401 · 3 Kommentare ·
Maintainer antworten meist innerhalb von 1 Tag
-
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 68/100
agentes-chat/agentes-chat#927 ·
Maintainer antworten meist innerhalb von 1 Tag
-
Closing a pinned tab jumps to the Tabs section instead of the next open pinned tabEvtl. vergeben @seenimohamed hat das heute übernommen. Offenbug
Schwierigkeit 2/5 1-3 Stunden Anfängerfreundlichkeit 78/100
ardia-browser/ardia#2769 ·
Maintainer antworten meist innerhalb von 1 Tag