Render fails when removing a temporary directory hits a transient lock or network file system error

Đang mở
  • Có gì không ổn?
#15,071 0 bình luận 0 reaction 0 người được giao Xem trên GitHub

Maintainer thường phản hồi trong vòng 1 ngày

Chưa có ai nhận issue này.

Đánh giá

Độ khó
4/5
Thời gian dự kiến
3-5 ngày
Mức phù hợp với người mới
28/100
Loại issue
Lỗi
Độ rõ ràng
Khá rõ ràng
Mức độ hoạt động
Sôi nổi
Công nghệ
deno, typescript
Lĩnh vực
build-system, cli

Hướng nghiên cứu

Start in src/command/render/render-files.ts where the execute result is removed after the freezer copy, then read safeRemoveSync in src/deno_ral/fs.ts and removeIfEmptyDir in src/command/render/cleanup.ts. The maintainers have not chosen between a retry with delay, a warning, or writing directly to the freezer, so agree on the approach in the thread before coding. Done means a transient removal failure no longer aborts the render, reproduced by holding a file handle from another process on Windows or with a local NFS loopback mount on Linux.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Mô tả

bug file-systems

During render, Quarto removes several directories it has just written or copied. Each removal is a single Deno.removeSync(path, { recursive: true }) attempt, and any error aborts the render. On network file systems, cloud-synced folders, or with another process scanning the folder (editor, antivirus, sync client), this removal can fail transiently even though the directory is empty a moment later.

This happens after all code has executed, so the user loses the whole render for a cleanup step.

Reported cases

All with the same pattern: removal right after write or copy, one attempt, fatal error.

execute-results removal in project render (removeFreezeResults):

  • #15054: Linux NFS, Directory not empty (os error 39), Quarto 1.10.18
  • #8972: Windows mapped network drive, An unexpected network error occurred. (os error 59), Quarto 1.4.550
  • #2380 (comment from 2023-07-02): Google Drive, NotFound (os error 2)
  • Discussions #6143 and #10775

Render cleanup of _files, libs and mediabag (renderCleanup, removeIfEmptyDir):

  • #7993: Google Drive, os error 2 removing _files\libs with embed-resources
  • #6616: Windows, os error 303 then os error 32 removing mediabag, also on local disk with an editor open on the project
  • #4614 (closed by #7921)

Code path

In a project render, the execute result is written beside the document, copied to the freezer, then removed:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/command/render/render-files.ts#L242-L284

The removal goes through safeRemoveSync, which only ignores the error if the path is gone afterwards:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/deno_ral/fs.ts#L114-L125

This was added for denoland/deno#18996 (removal succeeds but reports an error) and covers the Google Drive os error 2 case when the directory ends up removed. It does not cover cases where the directory still exists at the time of the check: NFS ENOTEMPTY, SMB os error 59, sharing violations.

Render cleanup has the same pattern:

https://github.com/quarto-dev/quarto-cli/blob/183154fa2c026b3f684d9985856a3d555f4f457e/src/command/render/cleanup.ts#L92-L109

Deno does not retry either. Deno.remove calls Rust std::fs::remove_dir_all, which on Linux returns ENOTEMPTY from the final rmdir directly, and on Windows only retries a few times without waiting, for ERROR_SHARING_VIOLATION (32) and ERROR_DIR_NOT_EMPTY (145).

Why the directory is empty right after

On NFS, unlinking a file that still has an open handle on the client renames it to a hidden .nfsXXXX file until the last handle closes (see "silly rename" in the Linux NFS FAQ). rmdir fails with ENOTEMPTY during that window. This matches what #15054 describes. On Windows, a sync client, antivirus, indexer or editor holding the file produces os error 32 / 145 / 303 the same way.

From my reading of the code, Quarto does not hold a handle on these files at removal time (all writes and copies use sync APIs that close the file). So the holder is another process, and we can't control it.

Prior work

Possible directions

We could:

  • Retry the removal with a short sleep, only after a failure, so there is no cost in the normal case. Node's fs.rm does this with maxRetries / retryDelay for EBUSY, EMFILE, ENFILE, ENOTEMPTY and EPERM; on Windows, 32, 145 and 303 would be the matching codes. os error 59 is not in any established retry list.
  • For execute-results specifically, report a warning instead of failing the render when the removal still fails, since the result is already in the freezer at this point. A leftover execute-results is not harmless (it could be picked on next thaw or end up in the output), so it should be cleaned up on the next execution.
  • Write the execute result directly into the freezer instead of beside the document. This removes the write-then-remove step entirely, but changes how the freezer mirrors <doc>_files.

Testing will need a way to reproduce without a network share: holding a file handle from another process (Windows sharing violation) or a local NFS loopback mount on Linux.

Ngôn ngữ chính
JavaScript
Star
6.1k
Fork
478
Merge trung bình
17 giờ 7 phút
Pull request đã merge (30 ngày)
112

Chuẩn bị môi trường

Bắt đầu từ đâu

  1. Đọc hết issue, rồi đọc hướng dẫn đóng góp của dự án.
  2. Bình luận trên issue rằng bạn sẽ nhận — tránh hai người làm cùng một việc.
  3. Fork repository và làm thay đổi trên một nhánh.
  4. Mở pull request có tham chiếu số hiệu của issue.

Issue khác của quarto-dev/quarto-cli

Tất cả issue của quarto-dev/quarto-cli

Issue tương tự

Thêm issue về JavaScript

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.