Render fails when removing a temporary directory hits a transient lock or network file system error

Start in src/command/render/render-files.ts where the execute result is removed after the freezer copy, then read safeRemoveSync in src/deno_ral/fs.ts and removeIfEmptyDir in src/command/render/cleanup.ts. The maintainers have not chosen between a retry with delay, a warning, or writing directly to the freezer, so agree on the approach in the thread before coding. Done means a transient removal failure no longer aborts the render, reproduced by holding a file handle from another process on Windows or with a local NFS loopback mount on Linux.

During render, Quarto removes several directories it has just written or copied. Each removal is a single Deno.removeSync(path, { recursive: true }) attempt, and any error aborts the render. On network file systems, cloud-synced folders, or with another process scanning the folder (editor, antivirus, sync client), this removal can fail transiently even though the directory is empty a moment later.

This happens after all code has executed, so the user loses the whole render for a cleanup step.

Reported cases

All with the same pattern: removal right after write or copy, one attempt, fatal error.

execute-results removal in project render (removeFreezeResults):

• #15054: Linux NFS, Directory not empty (os error 39), Quarto 1.10.18
• #8972: Windows mapped network drive, An unexpected network error occurred. (os error 59), Quarto 1.4.550
• #2380 (comment from 2023-07-02): Google Drive, NotFound (os error 2)
• Discussions #6143 and #10775

Render cleanup of _files, libs and mediabag (renderCleanup, removeIfEmptyDir):

• #7993: Google Drive, os error 2 removing _files\libs with embed-resources
• #6616: Windows, os error 303 then os error 32 removing mediabag, also on local disk with an editor open on the project
• #4614 (closed by #7921)

Code path

In a project render, the execute result is written beside the document, copied to the freezer, then removed:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/command/render/render-files.ts#L242-L284

The removal goes through safeRemoveSync, which only ignores the error if the path is gone afterwards:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/deno_ral/fs.ts#L114-L125

This was added for denoland/deno#18996 (removal succeeds but reports an error) and covers the Google Drive os error 2 case when the directory ends up removed. It does not cover cases where the directory still exists at the time of the check: NFS ENOTEMPTY, SMB os error 59, sharing violations.

Render cleanup has the same pattern:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/command/render/cleanup.ts#L92-L109

Deno does not retry either. Deno.remove calls Rust std::fs::remove_dir_all, which on Linux returns ENOTEMPTY from the final rmdir directly, and on Windows only retries a few times without waiting, for ERROR_SHARING_VIOLATION (32) and ERROR_DIR_NOT_EMPTY (145).

Why the directory is empty right after

On NFS, unlinking a file that still has an open handle on the client renames it to a hidden .nfsXXXX file until the last handle closes (see "silly rename" in the Linux NFS FAQ). rmdir fails with ENOTEMPTY during that window. This matches what #15054 describes. On Windows, a sync client, antivirus, indexer or editor holding the file produces os error 32 / 145 / 303 the same way.

From my reading of the code, Quarto does not hold a handle on these files at removal time (all writes and copies use sync APIs that close the file). So the holder is another process, and we can't control it.

Prior work

• #1536 added safeRemoveIfExists (warn instead of fail). It is used for freezer pruning, with a comment about network drives, but not for execute-results:

  https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/command/render/freeze.ts#L227-L243

• #7921 and #10377 introduced and extended safeRemoveSync.

• #10629 tried a retry loop on Windows. Measured on a local Windows mediabag lock: 74 to 300 attempts, 18 to 70 ms before success. Discussion there: we would prefer a pause-and-retry with a short timeout, and giving up silently on removal should not be the default. Closed unmerged.

Possible directions

We could:

• Retry the removal with a short sleep, only after a failure, so there is no cost in the normal case. Node's fs.rm does this with maxRetries / retryDelay for EBUSY, EMFILE, ENFILE, ENOTEMPTY and EPERM; on Windows, 32, 145 and 303 would be the matching codes. os error 59 is not in any established retry list.
• For execute-results specifically, report a warning instead of failing the render when the removal still fails, since the result is already in the freezer at this point. A leftover execute-results is not harmless (it could be picked on next thaw or end up in the output), so it should be cleaned up on the next execution.
• Write the execute result directly into the freezer instead of beside the document. This removes the write-then-remove step entirely, but changes how the freezer mirrors <doc>_files.

Testing will need a way to reproduce without a network share: holding a file handle from another process (Windows sharing violation) or a local NFS loopback mount on Linux.