SKILL.md / asset integrity is never checked: fault-injection findings and a proposal
I maintainer di solito rispondono entro 1 giorno
Valutazione
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Idoneità per principianti
- 48/100
Direzione di ricerca
Start with internal/cli/run.go and the existing skill-up validate behavior, then inspect internal/skill and the skill-upper SKILL.md workflow. Compare the proposed checks with the fixture behavior described in the issue and the linked stdlib-only Python reference implementation. Done means validation covers frontmatter, required name and description, and referenced paths without breaking existing eval.yaml validation.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Descrizione
Title: SKILL.md / asset integrity is never checked: fault-injection findings and a proposal
Summary
We ran a controlled fault-injection experiment against the skill-upper skill (truncated SKILL.md, deleted/truncated referenced attachments, blanked frontmatter) and measured whether host agents notice the damage while following the skill. Headline result: detection by the host is probabilistic and path-dependent — damage that blocks the executing instruction is reliably reported (14/14 across three hosts); damage in an optionally-read reference is not (down to 2/8). The toolchain itself performs no integrity checking at any layer.
Findings
1. The CLI never parses SKILL.md. internal/cli/run.go only requires it to be a regular file (isRegularFile); internal/skill installs by copying. Nothing validates frontmatter or verifies that files referenced from SKILL.md exist. Your own e2e fixtures demonstrate this — their SKILL.md files are frontmatter-less stubs (mock-engine, multiturn-session, custom-engine), which works because nothing reads them. Meanwhile skill-up validate does hard-fail on broken eval.yaml (missing case files, malformed YAML): the eval-config layer is defended, the skill-content layer is not.
2. The skill-upper workflow has no integrity step. Step 1 reads the target SKILL.md to extract behaviors, but nothing checks document completeness or reference liveness before cases are generated. In our wounded-examiner run, a target skill whose reference doc was truncated lost coverage of 5 format rules in the generated eval suite — silently.
3. Host vigilance is not a substitute. 30+ runs, three hosts (kimi-code, Claude Code, Codex CLI), same model across hosts:
| fault | host | detected |
|---|---|---|
| SKILL.md truncated at 60% (blocks main flow) | kimi-code | 8/8 |
| SKILL.md truncated at 60% | Claude Code / Codex CLI | 3/3 / 3/3 |
| SKILL.md truncated at 85% (only tail index lost) | kimi-code | 1/3 |
| attachment truncated at 50% (optionally read) | kimi-code | 2/8 |
| attachment truncated at 50% | Claude Code / Codex CLI | 3/3 / 2/3 |
| Step-2-mandated template deleted | kimi-code | 3/3 |
| frontmatter name/description blanked | kimi-code | 3/4 |
Detection tracks whether the damage contradicts the instruction the agent is executing right now — not how severe the damage is.
Proposal
Cheap, deterministic, in the spirit of the existing validate command:
- Extend
skill-up validate(or addvalidate --skill) to also check the skill itself: frontmatter parses,name/descriptionnon-empty, and everyreferences/,assets/,scripts/path mentioned in SKILL.md exists on disk. - Optionally one line in
skills/skill-upper/SKILL.mdStep 1 instructing the agent to run that check before scaffolding. - (Larger, future) a manifest/hash mode for tamper evidence against stealth cuts.
We have a working reference implementation of the checks (stdlib-only Python, ~150 LOC, fixture-tested) and a reusable fault-injection harness, both at https://github.com/BiBoyang/skill-quake (full experiment log, grading rubric, and raw per-cell summaries included). Happy to contribute the validate-side check as a Go PR if the direction fits.
Notes
- All experiments ran on local copies; no third-party skills were harmed. The evaluated "target" skill is a synthetic fixture.
- We deliberately did not test semantic tampering (plausible-wrong instructions) — a separate, scarier question.
- Lingua principale
- Go
- Stelle
- 1.5k
- Fork
- 134
- Merge medio
- 1g 22h
- PR unite (30g)
- 61
Preparare l'ambiente
- Nessun Dockerfile né file Docker Compose
- Ha un modello di pull request
- Leggi la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Altre issue di alibaba/skill-up
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 35/100
I maintainer di solito rispondono entro 1 giorno
-
OpenSandbox runtime: expect.files_exist and file_contains fail because WorkspacePath is remoteForse già presa @bababoom721 l’ha presa 1 giorno fa. Aperta
Difficoltà 4/5 3-5 giorni Idoneità per principianti 25/100
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 30/100
alibaba/skill-up#408 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 5/5 Più di una settimana Idoneità per principianti 15/100
alibaba/skill-up#407 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 4/5 3-5 giorni Idoneità per principianti 25/100
alibaba/skill-up#406 · 1 commento ·
I maintainer di solito rispondono entro 1 giorno
Tutte le issue di alibaba/skill-up
Issue simili
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 86/100
fluxcd/notification-controller#1441 ·
I maintainer di solito rispondono entro 1 giorno
-
Difficoltà 2/5 1-3 ore Idoneità per principianti 76/100
OneBusAway/maglev#2145 ·
I maintainer di solito rispondono entro 2 giorni
-
priority/medium status/ready type/chore
Difficoltà 2/5 1-3 ore Idoneità per principianti 84/100
discobox-ai/discobox#1357 ·
I maintainer di solito rispondono entro 1 giorno
-
TestPseudoLocaleRemote fails now and then: it types the code before the laptop asks for itForse già presa @fzn0x l’ha presa oggi. Apertabug priority: low
Difficoltà 2/5 1-3 ore Idoneità per principianti 88/100
I maintainer di solito rispondono entro 1 giorno
-
edit_file diff rendering panics on invalid UTF-8 textForse già presa @TDnorthgarden l’ha presa oggi. Apertaarea/tui
Difficoltà 2/5 1-3 ore Idoneità per principianti 85/100
docker/docker-agent#4955 ·
I maintainer di solito rispondono entro 1 giorno