Render fails when removing a temporary directory hits a transient lock or network file system error

Ouverte
  • Quelque chose ne va pas ?
#15,071 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub

Les mainteneurs répondent en général sous 1 jour

Personne n'a encore pris cette issue.

Évaluation

Difficulté
4/5
Temps estimé
3-5 jours
Accessibilité débutants
28/100
Type d'issue
Bug
Clarté
Plutôt claire
Activité
Active
Stack technique
deno, typescript
Domaine
build-system, cli

Piste de recherche

Start in src/command/render/render-files.ts where the execute result is removed after the freezer copy, then read safeRemoveSync in src/deno_ral/fs.ts and removeIfEmptyDir in src/command/render/cleanup.ts. The maintainers have not chosen between a retry with delay, a warning, or writing directly to the freezer, so agree on the approach in the thread before coding. Done means a transient removal failure no longer aborts the render, reproduced by holding a file handle from another process on Windows or with a local NFS loopback mount on Linux.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Description

bug file-systems

During render, Quarto removes several directories it has just written or copied. Each removal is a single Deno.removeSync(path, { recursive: true }) attempt, and any error aborts the render. On network file systems, cloud-synced folders, or with another process scanning the folder (editor, antivirus, sync client), this removal can fail transiently even though the directory is empty a moment later.

This happens after all code has executed, so the user loses the whole render for a cleanup step.

Reported cases

All with the same pattern: removal right after write or copy, one attempt, fatal error.

execute-results removal in project render (removeFreezeResults):

  • #15054: Linux NFS, Directory not empty (os error 39), Quarto 1.10.18
  • #8972: Windows mapped network drive, An unexpected network error occurred. (os error 59), Quarto 1.4.550
  • #2380 (comment from 2023-07-02): Google Drive, NotFound (os error 2)
  • Discussions #6143 and #10775

Render cleanup of _files, libs and mediabag (renderCleanup, removeIfEmptyDir):

  • #7993: Google Drive, os error 2 removing _files\libs with embed-resources
  • #6616: Windows, os error 303 then os error 32 removing mediabag, also on local disk with an editor open on the project
  • #4614 (closed by #7921)

Code path

In a project render, the execute result is written beside the document, copied to the freezer, then removed:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/command/render/render-files.ts#L242-L284

The removal goes through safeRemoveSync, which only ignores the error if the path is gone afterwards:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/deno_ral/fs.ts#L114-L125

This was added for denoland/deno#18996 (removal succeeds but reports an error) and covers the Google Drive os error 2 case when the directory ends up removed. It does not cover cases where the directory still exists at the time of the check: NFS ENOTEMPTY, SMB os error 59, sharing violations.

Render cleanup has the same pattern:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/command/render/cleanup.ts#L92-L109

Deno does not retry either. Deno.remove calls Rust std::fs::remove_dir_all, which on Linux returns ENOTEMPTY from the final rmdir directly, and on Windows only retries a few times without waiting, for ERROR_SHARING_VIOLATION (32) and ERROR_DIR_NOT_EMPTY (145).

Why the directory is empty right after

On NFS, unlinking a file that still has an open handle on the client renames it to a hidden .nfsXXXX file until the last handle closes (see "silly rename" in the Linux NFS FAQ). rmdir fails with ENOTEMPTY during that window. This matches what #15054 describes. On Windows, a sync client, antivirus, indexer or editor holding the file produces os error 32 / 145 / 303 the same way.

From my reading of the code, Quarto does not hold a handle on these files at removal time (all writes and copies use sync APIs that close the file). So the holder is another process, and we can't control it.

Prior work

Possible directions

We could:

  • Retry the removal with a short sleep, only after a failure, so there is no cost in the normal case. Node's fs.rm does this with maxRetries / retryDelay for EBUSY, EMFILE, ENFILE, ENOTEMPTY and EPERM; on Windows, 32, 145 and 303 would be the matching codes. os error 59 is not in any established retry list.
  • For execute-results specifically, report a warning instead of failing the render when the removal still fails, since the result is already in the freezer at this point. A leftover execute-results is not harmless (it could be picked on next thaw or end up in the output), so it should be cleaned up on the next execution.
  • Write the execute result directly into the freezer instead of beside the document. This removes the write-then-remove step entirely, but changes how the freezer mirrors <doc>_files.

Testing will need a way to reproduce without a network share: holding a file handle from another process (Windows sharing violation) or a local NFS loopback mount on Linux.

Langage dominant
JavaScript
Étoiles
6.1k
Forks
478
Merge moyen
17 h 36 min
PR mergées (30 j)
139

Préparer son environnement

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Autres issues de quarto-dev/quarto-cli

Toutes les issues de quarto-dev/quarto-cli

Issues similaires

Plus d'issues JavaScript

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.