[nolte-linux] Validate image declarations deterministically and fail closed without a validator

Open
#730 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
68/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
json, markdown, python, yaml

Research direction

Start by reading scripts/validate_schemas.py for the exit-code and install-hint convention, then schemas/image-declaration-v1.0.schema.yaml for the schema itself and plugins/nolte-linux/skills/linux-image-audit/SKILL.md step 3 for the three §F conditional rules. Write plugins/nolte-linux/skills/linux-image-audit/scripts/validate_declaration.py, mirroring the bundled-script layout under plugins/nolte-media/skills/image-generate/scripts/, and drive it with tests/test_validate_declaration.py using the seven mutation cases in the issue. Done means exit 0/1/2 as specified, both schema examples valid, and the fallback prose removed from linux-image-scanner.md:45 and linux-image-developer.md:41.

Written by the indexing model from the issue text.

Description

agents bug nolte-linux skills test

Summary

The image-declaration.yaml validation used by the linux-image-scanner and linux-image-developer agents is not deterministic and not fail-closed. When jsonschema/PyYAML are missing, both agents fall back to "validate by reading the schema", i.e. an LLM eyeballing the file, and still report the declaration as valid. Independently of that, the three §F conditional rules whose violation the audit classifies as Critical are not encoded in the schema at all and are enforced only by prose.

Context

spec/project/yaml-json-schema/ §Validation contract (spec/project/yaml-json-schema/en.md:110) says: "MUST NOT treat absence of a validator as a passing gate. If the chosen validator isn't installed, the gate fails with an install hint; silent skip is forbidden." The repository's own scripts/validate_schemas.py honours this (exit 2 when the validator is missing). The nolte-linux agents don't.

The declaration is the root of every later decision: update_model decides whether the §F mandatory pillar applies at all, so a mis-validated declaration silently changes the audit verdict.

Evidence (revision 605926fb)

  • plugins/nolte-linux/agents/linux-image-scanner.md:45 and :88: fallback to "a careful read (required fields, enums, schema_version as the string "1.0", no extra keys …)". The fallback list omits type, minLength, minimum, nested additionalProperties via $ref: "#/$defs/HostSwitch".

  • plugins/nolte-linux/agents/linux-image-developer.md:41: same fallback for declarations it writes.

  • schemas/image-declaration-v1.0.schema.yaml:10-17: the conditional rules (field-updated ⇒ update + health.failure_switch; reflash-only ⇒ no update) are deliberately not in the schema.

  • plugins/nolte-linux/skills/linux-image-audit/SKILL.md step 3: a violated conditional rule is a §F Critical.

  • Reproduced with jsonschema 4.26.0 against the schema's own first example, mutated:

    REJECTED  max_boot_attempts=0                 (fallback read list doesn't name `minimum`)
    REJECTED  empty image.name                    (fallback read list doesn't name `minLength`)
    REJECTED  extra key in $ref HostSwitch        (nested via $ref)
    REJECTED  max_boot_attempts as string
    ACCEPTED  field-updated WITHOUT update block  (Critical per audit, schema-valid)
    ACCEPTED  field-updated WITHOUT failure_switch
    ACCEPTED  reflash-only WITH update block
    
  • On the audit host, python3 -I -c "import yaml" fails (modules only in user site-packages), so the fallback path is realistic, not hypothetical.

Expected behavior

  • One deterministic validator decides both the JSON Schema and the three §F conditional rules, with stable exit codes.
  • When the validator dependencies are missing, the result is explicitly not-decidable (or a failed precondition), never valid.
  • Agents and skills consume the validator's output; no LLM-only schema judgement remains.

Proposed implementation

  • Add a bundled script, e.g. plugins/nolte-linux/skills/linux-image-audit/scripts/validate_declaration.py (bundled-script convention as in plugins/nolte-media/skills/image-generate/scripts/; invoked via ${CLAUDE_PLUGIN_ROOT}), that:
    • loads the declaration and the schema (path argument; repo copy first, installed nolte-shared copy second, same lookup the skills already describe),
    • validates with jsonschema Draft 2020-12,
    • checks the three §F conditional rules,
    • prints machine-readable JSON ({"result": "valid|invalid|not-decidable", "violations": [{"rule", "path", "message"}]}),
    • exits 0 valid, 1 invalid, 2 validator unavailable (mirrors scripts/validate_schemas.py).
  • Replace the "validate by reading" fallback in linux-image-scanner.md (line 45, Phase 2) and linux-image-developer.md (line 41) with: run the script; on exit 2 report the declaration schema check as not-decidable with the install hint, and never as valid.
  • Update linux-image-audit triage so a not-decidable declaration check is blocking (the §F applicability depends on it), and define its severity explicitly.
  • Keep the schema unchanged (v1.0); the conditional rules stay out of the schema per spec/project/yaml-json-schema/ §Document skeleton.

Acceptance criteria

  1. A script exists that validates an image-declaration.yaml against schemas/image-declaration-v1.0.schema.yaml and the three §F conditional rules, emitting JSON with result and violations.
  2. Exit codes: 0 valid, 1 invalid, 2 validator dependencies missing; exit 2 prints an install hint to stderr.
  3. Each of the seven mutation cases above is reported invalid with a rule identifier naming the violated constraint.
  4. Both schema examples validate as valid.
  5. linux-image-scanner.md and linux-image-developer.md no longer contain a "validate by reading the schema" fallback; on exit 2 they report not-decidable with the install hint.
  6. linux-image-audit/SKILL.md states the severity and blocking behaviour of a not-decidable declaration check, and never yields PASS for an image whose declaration couldn't be validated.
  7. task test and task lint pass; python3 scripts/validate_skills.py reports no new error for nolte-linux.

Tests

  • tests/test_validate_declaration.py (pytest): valid examples; one negative case per schema constraint class (required, enum, const, type, minLength, minimum, additionalProperties top-level and via $ref); the three conditional-rule negatives; unparseable YAML; missing file.
  • Negative dependency test: run the script with jsonschema import made unavailable (e.g. monkeypatch of sys.modules / subprocess with -I and empty PYTHONPATH) and assert exit 2 and no valid in stdout.

Dependencies

  • Prerequisite for the scanner tool-surface issue (removing Bash from linux-image-scanner), which moves this call into the skill.

Autonomy

A coding agent can implement this independently. No architectural decision needed beyond the severity choice in AC 6, for which "blocking, like resolvable" is the default consistent with spec §J.

Priority

P2. Real correctness gap with a reproducible false valid, but it needs either a missing dependency or an LLM misread to change a verdict; it does not by itself make the plugin unsafe.

Refs spec/project/linux-image-best-practices/, spec/project/yaml-json-schema/

Dominant language
Python
Stars
0
Forks
0
Avg merge
1d 10h
Merged PRs (30d)
25

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nolte/claude-shared

All issues in nolte/claude-shared

Similar issues

More Python issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.