Export fails with EINVAL when the package cache is on mergerfs (renameat2 RENAME_NOREPLACE unsupported)

Aperta
  • Qualcosa non va?
#54 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

@efegokdemir ci sta già lavorando.

Dal 25/10/2026.

  • #79 di @efegokdemir — aperta

Valutazione

Difficoltà
3/5
Tempo stimato
1-2 giorni
Idoneità per principianti
54/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
python, rust
Ambito
backend, cli

Direzione di ricerca

Start with heliopause/src/export/archive.rs, especially install_blob, and compare its blob commit with the hard-link approach used by install_file. Reproduce the export failure on a mergerfs package cache and check how EINVAL is handled. Done means export either commits blobs safely on that filesystem or rejects the unsupported operation before the lengthy preparation begins; the issue also suggests documenting the filesystem requirement in docs/cli-design.md.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

area:caching area:packaging bug model:gemma4 priority:P2 triage:review

ignition export fails at "Scan and copy reachable storage" when [paths].package_cache is on a mergerfs (FUSE) mount, and the error names neither the file nor the cause:

[    OK    ] Capture compiled modules and globals.  280053792 bytes, 542905 global bytes downloaded  8.3s
[  FAILED  ] Scan and copy reachable storage.  0.0 / 11.0 GiB  0.1s
RuntimeError: Invalid argument (os error 22)
[  FAILED  ] Export native package.  22.1s
error: native package export failed: Invalid argument (os error 22)

The failure lands after tuning, compilation and capture, so each attempt costs the full preparation pass, about 20 minutes for Gemma 4 E4B on a cold kernel cache.

Cause

strace -f -Z points at the blob commit in install_blob (heliopause/src/export/archive.rs:429, rechecked on bce032c0):

libc::renameat2(AT_FDCWD, source, AT_FDCWD, target, libc::RENAME_NOREPLACE) != 0
    => return Err(std::io::Error::last_os_error().into())

mergerfs returns EINVAL for RENAME_NOREPLACE, and last_os_error discards both paths, so the operator sees only Invalid argument (os error 22).

Minimal reproduction

Outside Flare, on the same mount:

import ctypes, os
libc = ctypes.CDLL(None, use_errno=True)
open("/models/x/.src", "w").close()
r = libc.renameat2(-100, b"/models/x/.src", -100, b"/models/x/.dst", 1)  # 1 = RENAME_NOREPLACE
print(r, os.strerror(ctypes.get_errno()))  # -1 Invalid argument on mergerfs; 0 on ZFS

Direct I/O and io_uring writes to the same mount work; only the no-replace rename fails. Moving --output alone does not help, because blobs are committed into the package cache and the output must share its filesystem.

The commit path used elsewhere

Hard-link publication is the usual commit in this tree, and link already gives the no-replace guarantee by failing with EEXIST when the destination exists:

  • helios/src/package/cache.rs:2581, :2606, :3029
  • flare/src/package_cache.rs:475, :496
  • heliopause/src/export/blob.rs:256
  • heliopause/src/export/package.rs:932, :1012
  • heliopause/src/export/transfer.rs:1021
  • heliopause/src/export/protection.rs:335

install_blob is the outlier that requires RENAME_NOREPLACE support from the filesystem.

Possible changes

  1. Fall back to link() followed by unlink() of the temporary file when renameat2 fails with EINVAL or ENOTSUP, keeping the existing EEXIST behaviour when the destination already exists. Both calls stay inside one filesystem, which export already requires.
  2. Check that the package cache supports the operation before preparation starts, and fail there. Record the requirement beside the same-filesystem rule at docs/multi-host.md:243, which requires output and package cache to share a filesystem but says nothing about rename support. docs/package-format-design.md:638 promises an explicit preflight/export error for filesystem file-size and space limits; nothing covers rename semantics, which is why this failure arrives after capture.

Workaround

Point [paths].package_cache and --output at a filesystem that supports RENAME_NOREPLACE, such as ZFS or ext4.

Environment

Proxmox LXC container, /models is a mergerfs pool over ZFS datasets, kernel 6.17.9-1-pve, upstream bce032c0.

Lingua principale
Rust
Stelle
800
Fork
106
Merge medio
1g 3h
PR unite (30g)
100

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di NVIDIA/hyperflare-inference

Tutte le issue di NVIDIA/hyperflare-inference

Issue simili

Altre issue su Rust

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.