enhancement: migrate pre-existing GPUStack model files into the node cache

Start with pkg/modelmanager/store and the modelManager.rootPath chart value, then trace the existing ModelArtifact resolution and publish pipeline. Compare an offline import tool with plugin RPC, copy versus move, and shared-NFS handling; done means an agreed migration design that preserves hashing, manifest verification, and the store's single-writer invariant.

Background

The operator's node model cache is content-addressed and only admits bytes it has
verified itself. Hosts that already run GPUStack (server/worker product) commonly
hold large model caches downloaded by the legacy worker. Today there is no path
for those pre-existing files to enter the node cache: every model is re-downloaded
from the Hub (or pulled from a peer node) even when the same bytes already sit on
the same disk.

This issue collects the two designs as-found and sketches how a migration could
work.

Current GPUStack design (as-found)

Verified against the GPUStack repository (main branch) and a four-host GPU lab
running a mix of all-in-one and server/worker deployments.

• Default paths: data_dir=/var/lib/gpustack, cache_dir=<data_dir>/cache
  (gpustack/config/config.py), overridable with --data-dir / --cache-dir
  or GPUSTACK_DATA_DIR / GPUSTACK_CACHE_DIR.
• On-disk layout is flat: <cache>/huggingface/<org>/<name>/<repo files>
  via hf_hub_download(local_dir=...) (gpustack/worker/downloaders.py), with
  a <name>.lock sibling and partial files under .cache/huggingface/download/.
  ModelScope is the same shape: <cache>/model_scope/<org>/<name>/ with a
  ._____temp/ directory. This is not the models--<org>--<name> blobs/
  snapshots layout — that was the v0.x-era layout, and an alembic migration
  moved old installs to the flat one.
• The ModelFile table (gpustack/schemas/model_files.py) records source,
  local_dir, worker_id, resolved_paths (JSON), size, state,
  source_index, cleanup_on_delete — no per-file digest.
• Engines consume the path directly (vLLM positional argument, SGLang
  --model-path); there is no snapshot/symlink/copy indirection. Mirrored
  deployments bind-mount all of /var/lib/gpustack into inference containers
  (VOLUME /var/lib/gpustack in pack/Dockerfile). The GPUStack helm chart
  mounts worker.dataDir (default /var/lib/gpustack) as a hostPath.
• Model evaluation pulls config skeletons in the native models-- layout
  into the same cache/huggingface/ directory, so two layout generations
  coexist in one cache.
• Observed in the wild: caches ranging from empty to ~5.1 TB living on a shared
  NFS export consumed by two hosts at once, both layout generations mixed, many
  config-only skeletons, single-file GGUFs under ollama/, and several
  processes (pipx systemd service + docker containers) reading and writing the
  same tree concurrently.

Operator design (after the model-artifact work)

• rootPath defaults to /var/lib/gpustack/models (chart value
  modelManager.rootPath).
• Layout (pkg/modelmanager/store): a layout marker file, plus
  published/<hex>/{marker,manifest.json,tree/...} (read-only after publish),
  partial/<hex>/<attempt>/, trash/, and ledger/{refs,digests}/.
  Single-writer, atomic rename on publish, per-file SHA-256 in the manifest.
• store.Open() reads only the layout marker: foreign files anywhere under
  the root are not scanned, not adopted, and not deleted. A layout mismatch
  refuses to start. This is deliberate — the store only trusts bytes it
  verified during its own write pipeline.
• GC accounts capacity filesystem-wide (statfs), so foreign files consume
  capacity but are never evicted.
• The store forbids shared inodes, so a migration cannot hard-link files into
  the tree.

The two roots do not collide: the legacy product never writes under
/var/lib/gpustack/models, so old and new caches can coexist on one host.

The gap

There is no adoption path from cache/ into models/. Everything the node
cache serves is downloaded (or peer-pulled) and hashed by the plugin itself,
even when identical bytes are already local.

Options

A. Offline import tool (lazy, verify-then-publish). For each candidate
model on the host: enumerate the flat files, apply the artifact's file filters,
hash while copying into a partial/ attempt, verify the tree against the
resolved ModelArtifact's manifestDigest, then publish through the store's
own pipeline (same code path as a network download, source = local disk).
"Lazy" means only models actually referenced by a ModelArtifact are imported
— never bulk-copy a 5 TB cache.

B. Do nothing — re-download / peer-pull. Zero new code. Correct but wastes
bandwidth and time on hosts that already hold the bytes; on a shared NFS cache
serving several hosts the waste is multiplied per host.

C. Rename-in-place after retiring the legacy worker. Once the old
deployment is gone, move the flat tree into a store attempt and publish in
place — no copy, no double occupancy. Breaks the legacy install, so it only
fits decommissioning migrations, and still needs hashing + manifest
verification before publish.

Recommended shape: A with lazy scoping, keeping B as the always-available
baseline.

Blockers / dependencies

• No digests in the legacy DB. Every imported file must be hashed in full.
  Measurements on disk-bound nodes put a post-hoc re-read-and-hash pass at
  roughly 44–46% of the original download duration; hashing while copying
  avoids the second read but still costs one full read.
• Resolution needs the Hub. Verifying against manifestDigest requires a
  resolved ModelArtifact, which today means Hub reachability. Air-gapped
  hosts would need the deferred expectedDigest/pre-resolved artifact path.
• Single writer. The store has exactly one writer (the plugin). An import
  tool must either run while the plugin is stopped, or go through a new plugin
  RPC so the single-writer invariant holds.
• GC candidacy. An imported tree with zero references becomes a GC
  candidate after the grace period. That is fine — a deployment that needs it
  will claim it — but import tooling should not fight the collector for
  just-imported trees.
• Shared caches. Where several hosts share one NFS export, per-host import
  duplicates work; the tool needs a story for dedup/locking, or should simply
  declare shared-cache hosts out of scope and let peer sync propagate.

Questions to decide

1. One-shot CLI run by an operator, or a plugin RPC so migration is online?
2. Copy (safe, doubles peak usage) vs move (cheap, breaks the legacy install)?
3. Shared-NFS caches: dedup/lock, or explicitly out of scope?