[FEATURE] Socair directory attestation (socair.modeldir/v1) for multi-file Models, verified at download and prefetch
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 25/100
Research direction
Start with internal/controller/model_attestation.go and the existing attestation tests, then review the Model CRD rules and prefetch flow described in the issue. The change also depends on defilantech/socair-verify#1 and relates to #1968 and #2136. Done means multi-file models can be staged and verified against a Socair directory attestation, with mismatches and trust failures reported before the model becomes Ready.
Written by the indexing model from the issue text.
Description
Feature Description
Socair attestation for multi-file Models: let a Model staged with spec.files (a Hugging Face repo, a sharded safetensors checkpoint) pass the --model-attestation gate using Socair's model directory attestation. Verify the staged files against it wherever the model is downloaded, including the prefetch Job.
Problem Statement
Attestation (#1968) is single-artifact only, so any multi-file Model is refused under enforce and reported unattested under warn:
- The check needs
spec.sha256.internal/controller/model_attestation.go:137:the attestation policy requires spec.sha256, the digest the attestation must be for. The envelope is admitted withpolicy.Admit([]byte(envelope), model.Spec.SHA256)(line 180). spec.sha256can't be combined withspec.files. The Model CRD's CEL rule:sha256 verifies a single artifact: it cannot be combined with files or mmproj … (use fileSha256 for multi-file staging).spec.fileSha256(per-file digests) exists, but the attestation check never reads it.
Multi-file is how large models are served: for example source: nvidia/GLM-5.3-Flash-NVFP4 with files: [config.json, "*.json", "*.jinja", "*.safetensors"], which expands to 44 files (33 shards, about 190 GiB).
Socair already attests these as one artifact. socair scan <dir> attests a model directory. Its subject is the digest of a canonical manifest (socair.modeldir/v1\n, then <sha256> <size> <path>\n for each file, sorted by path), and the report lists every file with its hash (socair docs/dev.md). It's a single subject, so socair-verify's one-subject rule holds. What LLMKube lacks is a way to recompute that digest from the files it stages and admit against it.
Proposed Solution
Admit Socair directory attestations for multi-file Models:
apiVersion: inference.llmkube.dev/v1alpha1
kind: Model
spec:
source: nvidia/GLM-5.3-Flash-NVFP4
files: [config.json, "*.json", "*.jinja", "*.safetensors"]
attestation:
configMapKeyRef:
name: glm-5-3-flash-attestation
key: attestation.dsse.json # from `socair scan <dir>` + `socair sign`
- The digest admitted is the staged set's
socair.modeldir/v1manifest digest: every staged file, meaning the concretespec.filesentries plus every glob expansion (status.stagedFiles), each with its SHA-256 and size. It's computed with the helper proposed in defilantech/socair-verify#1 (ModelDirDigest), not re-implemented in LLMKube, so the format can't drift. - Where the per-file hashes come from, two modes:
- Known up front (
spec.fileSha256covers every staged file, or a later digest source): compute the manifest digest and runcheckAttestationbefore any download, as today. Underenforce, an unattested Model never starts a prefetch Job. - Not known up front (globs, no
fileSha256): check the envelope's signature and trust before downloading. The downloader (the prefetch Job, or the serving init containers) records each file's SHA-256 and size while writing it. The Model only becomesReadyonce the recomputed manifest digest equals the attested subject. On a mismatch,att.CompareFiles(staged)(also proposed in socair-verify#1) names the file added, removed or changed. This mirrors Socair's ownairlock promote, which renames a directory into place only when its recomputed manifest digest is the attested subject.
- Known up front (
- Prefetch is the natural place for the second mode: the files are hashed away from the GPU nodes, and the InferenceService already waits for
Ready(and, with #2136'smodelCache.prefetch, holds its workload until the prefetch completes). - The attested set must equal the staged set. The attestation has to be made over exactly the files LLMKube stages, so
spec.filesand the directory scanned withsocair scanhave to agree. A repo file outsidespec.files(a README,original/weights) that was in the scanned directory makes the digests differ. The condition should say so and name the files, using the attestation's file list. - Conditions name the failing file(s) on
AttestationVerified/Degraded. - Single-file Models are unchanged:
configMapKeyRef+spec.sha256as today.
Derive the staging plan from the attestation (filesFrom: Attestation). A Socair report already describes exactly what to stage, signed. Its artifact carries repo_url, commit_sha, and files[] with path, sha256, size_bytes and role (weights, config, tokenizer, chat_template, code, adapter, other), per socair's report schema v1. So the Model can point at the attestation and drop the hand-written file list:
apiVersion: inference.llmkube.dev/v1alpha1
kind: Model
spec:
source: nvidia/GLM-5.3-Flash-NVFP4 # must match the attestation's repo_url
attestation:
configMapKeyRef:
name: glm-5-3-flash-attestation
key: attestation.dsse.json
filesFrom: Attestation # stage exactly the attested files
- Verify first, then trust the list. The envelope must pass the gate (trusted signature, promotion policy,
maxAge) before its file list is used, sofilesFrom: Attestationrequires a trust policy (--model-attestation-trusted-keys) even inwarnmode. An unsigned or untrusted list is never staged. Underoffwith no keys, the field is rejected with a condition. - Stage exactly the attested set:
files[].pathatcommit_sha(the source pinned to that revision), each verified against its attestedsha256andsize_byteswhile downloading.spec.filesandspec.fileSha256are mutually exclusive withfilesFrom: Attestation, enforced in CEL.status.stagedFilesrecords the attested paths. - Check the source:
spec.sourcemust name the attestation'srepo_url(normalised:org/name,hf://org/nameandhttps://huggingface.co/org/nameare equivalent), or the Model is refused. A mirror endpoint (HF_ENDPOINT) doesn't change the repo identity. - Optional role policy, for example
spec.attestationPolicy.allowRolesorallowCodeFiles: false, refusing to stagerole: codefiles (Python a loader would import) unless allowed, sotrust_remote_codesurfaces as a decision at admission. - What it solves:
- no hand-maintained file list or globs: whoever ran
socair scan/socair airlock pull --file …decided what gets served; - the staged set equals the attested set by construction, so the scanned-set-versus-
spec.filesmismatch above can't happen; - every hash is known up front, so the gate runs before any download and prefetch gets a concrete file list (#2136's globs problem);
- the revision can't drift to a re-pushed branch.
- no hand-maintained file list or globs: whoever ran
- Needs from socair-verify:
Attestation.FileswithRole, plusRepoURLandCommitSHAonAttestation(defilantech/socair-verify#1).
Tests, fed real socair scan <dir> + socair sign output as the existing attestation suite does:
- a multi-file Model whose staged set matches the attested directory goes
Readyunderenforce; - a staged file with different bytes, or an extra or missing file, refuses the Model and names it;
- with
fileSha256covering every file, no prefetch Job starts for an unattested Model; filesFrom: Attestationstages exactly the attested files atcommit_shaand records them instatus.stagedFiles; aspec.sourcethat doesn't matchrepo_url, an untrusted envelope, or no trust policy refuses the Model before any download; arole: codefile is refused when the role policy disallows it;- without it, the prefetch Job's recorded hashes drive the digest comparison, and the Model stays out of
Readyuntil it matches.
Alternatives Considered
- One attestation per file (each single-subject, verified against
fileSha256): works withsocair-verifyv0.1.0 as is, but means one envelope per file (44 here) and doesn't match how Socair attests repositories. - Re-implement
socair.modeldir/v1in LLMKube: works today, but the format then lives in two codebases and a subtle difference refuses correctly attested models. socair-verify#1 avoids that. - Turn attestation off for multi-file Models: leaves the largest models outside the gate.
Additional Context
- Depends on defilantech/socair-verify#1: export
ModelDirDigest, addAttestation.Files(with roles),RepoURL,CommitSHAandCompareFiles. - Related: #1968 (Socair attestation gate) and #2136 (GitOps-safe prefetch), whose prefetch Job would hash and verify the staged files.
- Docs:
docs/security/model-attestation.mdhere; socairdocs/dev.md(directory manifest),docs/airlock.md,docs/kubernetes.md("Handing models to LLMKube").
- Dominant language
- Go
- Stars
- 232
- Forks
- 37
- Avg merge
- 21h 42m
- Merged PRs (30d)
- 244
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- Ships a Dockerfile or Docker Compose file
- Has a pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from defilantech/LLMKube
-
component/metal-agent documentation
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
defilantech/LLMKube#2581 ·
Maintainers usually reply within 1 day
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
defilantech/LLMKube#2578 ·
Maintainers usually reply within 1 day
-
documentation helm priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
defilantech/LLMKube#2420 ·
Maintainers usually reply within 1 day
-
component/metal-agent enhancement priority/low size/medium
Difficulty 4/5 3-5 days Newbie friendliness 48/100
defilantech/LLMKube#2597 ·
Maintainers usually reply within 1 day
-
[BUG] Docs and comments describe InsufficientMemory only as a start refusal, but a serving engine can now carry itPossibly taken @defilan-agent claimed this today. Openbug component/metal-agent documentation priority/low size/small
Difficulty 2/5 1-3 hours Newbie friendliness 25/100
defilantech/LLMKube#2596 ·
Maintainers usually reply within 1 day
All issues in defilantech/LLMKube
Similar issues
-
☢️ Bug v3
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Maintainers usually reply within 1 day
-
area/import-export area/projects concern/ux
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
go-vikunja/vikunja#4455 · 1 comment ·
Maintainers usually reply within 2 days
-
Dependabot PRs no longer merge automaticallyPossibly taken @dwisiswant0 claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
projectdiscovery/urlfinder#160 ·
Maintainers usually reply within 1 day
-
database
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
chnm/legal-modernism#450 ·
Maintainers usually reply within 1 day
-
tech-debt
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
Maintainers usually reply within 1 day