Hacktoberfest 2026: le issue che i maintainer hanno segnato per ottobre, aperte e adatte ai principianti. Sfoglia le issue Hacktoberfest

Multi-file install: a single part error deletes the whole install tmpdir, including completed parts and resumable partials

Aperta
  • Qualcosa non va?
#9,481 0 commenti 0 reazioni 1 assegnatario Vedi su GitHub

I maintainer di solito rispondono entro 2 giorni

@lstein ci sta già lavorando.

Dal 24/8/2026.

  • #9488 di @lorenzozanee — chiusa senza merge
  • #9591 di @lorenzozanee — aperta

Valutazione

Questa issue non è ancora stata valutata.

Descrizione

Summary

When any single part of a multi-file install errors, _download_error_callback deletes the entire install tmpdir (model_install_default.py:1482-1489 → _safe_rmtree(install_job._install_tmpdir)) — including parts that finished completely and partials holding gigabytes of resumable progress. One transient failure (a 5xx on one file, a rename race) throws away everything the resume machinery exists to protect.

Found during the adversarial review of #9432; the rmtree behavior predates that PR, but #9432 added a new way to trip it (the sidecar rename race below).

Mechanism

Worker thread: any exception other than DownloadJobCancelledException in _do_download marks the part ERROR (download_default.py:327-330). For a part belonging to an install, _download_error_callback then pops the install job, sets it errored, cancels the multifile job, and rmtrees the whole _install_tmpdir.

Two concrete triggers:

  1. Transient server error on one file. A single HTTP 500 on file k of an n-file install (raised at download_default.py:456-459) deletes the completed files 1..k-1 and all partial progress. The next attempt starts the entire install from zero.
  2. Sidecar rename race (new surface from #9432). The 416-promotion path stats the sidecar (download_default.py:352) and renames it (download_default.py:446) a full network round-trip apart, with no existence guard. If the sidecar vanishes in between — user-triggered restart_file() runs clear_partials=True (model_install_default.py:665, :1314) against an in-flight request; manual deletion; AV/indexer interference on Windows — the rename raises FileNotFoundError → part ERROR → whole tmpdir deleted.

Reproduction (test sketch, deterministic)

Trigger 1 needs only two mounts — one good file, one 500 — through the install service, then assert the tmpdir is gone despite file 1 having completed.

Trigger 2 can be made deterministic at the download-queue level with an adapter that deletes the sidecar during the request, simulating the race:

class SidecarDeletingAdapter(TestAdapter):
    def __init__(self, *args, sidecar: Path, **kwargs):
        super().__init__(*args, **kwargs)
        self._sidecar = sidecar

    def send(self, request, **kwargs):
        self._sidecar.unlink()          # the race: sidecar vanishes mid-round-trip
        return super().send(request, **kwargs)

def test_sidecar_vanishing_during_416_roundtrip(tmp_path: Path) -> None:
    source = AnyHttpUrl("https://test.com/race.safetensors")
    content = b"complete"
    destination = tmp_path / "race.safetensors"
    sidecar = destination.with_name(destination.name + ".downloading")
    sidecar.write_bytes(content)

    session = TestSession()
    session.mount(
        str(source),
        SidecarDeletingAdapter(
            b"", status=416, headers={"Content-Range": f"bytes */{len(content)}"}, sidecar=sidecar
        ),
    )
    queue = DownloadQueueService(requests_session=session)
    queue.start()
    try:
        job = queue.download(source=source, dest=destination)
        queue.join()
    finally:
        queue.stop()

    # Today: FileNotFoundError escapes the rename -> job ERRORs; in an install context
    # _download_error_callback then rmtrees the entire tmpdir.
    assert job.status == DownloadJobStatus.ERROR

Suggested fix

  • In _download_error_callback, preserve the tmpdir when any part has resumable progress (a .downloading file on disk) or has already completed: write the install marker with an errored/paused status instead of rmtree, so the existing restore/restart_failed machinery can pick it up. Only rmtree when nothing on disk is worth keeping.
  • Guard the 416 promotion rename: if the sidecar is missing at rename time, fall through to the pause/restart path instead of letting FileNotFoundError escalate to a part ERROR.
Lingua principale
TypeScript
Stelle
28.6k
Fork
3k
Merge medio
3g 2h
PR unite (30g)
133

Preparare l'ambiente

  • Nessun Dockerfile né file Docker Compose
  • Ha un modello di pull request
  • Nessuna guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di invoke-ai/InvokeAI

Tutte le issue di invoke-ai/InvokeAI

Issue simili

Altre issue su TypeScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.