Render fails when removing a temporary directory hits a transient lock or network file system error

Aperta
  • Qualcosa non va?
#15,071 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

I maintainer di solito rispondono entro 1 giorno

Nessuno ha ancora preso questa issue.

Valutazione

Difficoltà
4/5
Tempo stimato
3-5 giorni
Idoneità per principianti
28/100
Tipo di issue
Bug
Chiarezza
Abbastanza chiara
Stato di attività
Attiva
Stack tecnologico
deno, typescript
Ambito
build-system, cli

Direzione di ricerca

Start in src/command/render/render-files.ts where the execute result is removed after the freezer copy, then read safeRemoveSync in src/deno_ral/fs.ts and removeIfEmptyDir in src/command/render/cleanup.ts. The maintainers have not chosen between a retry with delay, a warning, or writing directly to the freezer, so agree on the approach in the thread before coding. Done means a transient removal failure no longer aborts the render, reproduced by holding a file handle from another process on Windows or with a local NFS loopback mount on Linux.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Descrizione

bug file-systems

During render, Quarto removes several directories it has just written or copied. Each removal is a single Deno.removeSync(path, { recursive: true }) attempt, and any error aborts the render. On network file systems, cloud-synced folders, or with another process scanning the folder (editor, antivirus, sync client), this removal can fail transiently even though the directory is empty a moment later.

This happens after all code has executed, so the user loses the whole render for a cleanup step.

Reported cases

All with the same pattern: removal right after write or copy, one attempt, fatal error.

execute-results removal in project render (removeFreezeResults):

  • #15054: Linux NFS, Directory not empty (os error 39), Quarto 1.10.18
  • #8972: Windows mapped network drive, An unexpected network error occurred. (os error 59), Quarto 1.4.550
  • #2380 (comment from 2023-07-02): Google Drive, NotFound (os error 2)
  • Discussions #6143 and #10775

Render cleanup of _files, libs and mediabag (renderCleanup, removeIfEmptyDir):

  • #7993: Google Drive, os error 2 removing _files\libs with embed-resources
  • #6616: Windows, os error 303 then os error 32 removing mediabag, also on local disk with an editor open on the project
  • #4614 (closed by #7921)

Code path

In a project render, the execute result is written beside the document, copied to the freezer, then removed:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/command/render/render-files.ts#L242-L284

The removal goes through safeRemoveSync, which only ignores the error if the path is gone afterwards:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/deno_ral/fs.ts#L114-L125

This was added for denoland/deno#18996 (removal succeeds but reports an error) and covers the Google Drive os error 2 case when the directory ends up removed. It does not cover cases where the directory still exists at the time of the check: NFS ENOTEMPTY, SMB os error 59, sharing violations.

Render cleanup has the same pattern:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/command/render/cleanup.ts#L92-L109

Deno does not retry either. Deno.remove calls Rust std::fs::remove_dir_all, which on Linux returns ENOTEMPTY from the final rmdir directly, and on Windows only retries a few times without waiting, for ERROR_SHARING_VIOLATION (32) and ERROR_DIR_NOT_EMPTY (145).

Why the directory is empty right after

On NFS, unlinking a file that still has an open handle on the client renames it to a hidden .nfsXXXX file until the last handle closes (see "silly rename" in the Linux NFS FAQ). rmdir fails with ENOTEMPTY during that window. This matches what #15054 describes. On Windows, a sync client, antivirus, indexer or editor holding the file produces os error 32 / 145 / 303 the same way.

From my reading of the code, Quarto does not hold a handle on these files at removal time (all writes and copies use sync APIs that close the file). So the holder is another process, and we can't control it.

Prior work

Possible directions

We could:

  • Retry the removal with a short sleep, only after a failure, so there is no cost in the normal case. Node's fs.rm does this with maxRetries / retryDelay for EBUSY, EMFILE, ENFILE, ENOTEMPTY and EPERM; on Windows, 32, 145 and 303 would be the matching codes. os error 59 is not in any established retry list.
  • For execute-results specifically, report a warning instead of failing the render when the removal still fails, since the result is already in the freezer at this point. A leftover execute-results is not harmless (it could be picked on next thaw or end up in the output), so it should be cleaned up on the next execution.
  • Write the execute result directly into the freezer instead of beside the document. This removes the write-then-remove step entirely, but changes how the freezer mirrors <doc>_files.

Testing will need a way to reproduce without a network share: holding a file handle from another process (Windows sharing violation) or a local NFS loopback mount on Linux.

Lingua principale
JavaScript
Stelle
6.1k
Fork
478
Merge medio
17h 7m
PR unite (30g)
112

Preparare l'ambiente

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Altre issue di quarto-dev/quarto-cli

Tutte le issue di quarto-dev/quarto-cli

Issue simili

Altre issue su JavaScript

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.