flake: `stale NFS file handle`

オープン
#157 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る

メンテナーはふだん 2 日以内に返信

まだ誰も着手していません。

評価

難易度
5/5
見積もり時間
1週間以上
初心者へのやさしさ
35/100
issue の種類
バグ
明瞭さ
おおむね明確
活発さ
停滞
技術スタック
github-actions, linux, rust

調査の方向性

Start with qemu.rs:932 to understand how bcvk configures virtiofsd, then inspect the listed virtiofsd inode-store and file-open paths and the kernel FUSE staleness check. Reproduce or instrument the intermittent CI failures in the named integration tests; done means identifying a supported cause and validating a reliable fix or mitigation. The issue has had no recent discussion, so confirm the investigation is still active before starting.

索引モデルが issue の本文から書いたものです。

説明

I don't understand why we're getting ESTALE sometimes in CI; haven't seen it locally. My understanding of the situations where this can happen require mutation of the source, but that shouldn't be happening here.


Investigation Summary (2026-01-23)

Assisted-by: OpenCode (Opus 4.5)

Scope of Impact
  • Affected repos: Only bootc-dev/bcvk (checked bootc, infra, ocidir-rs, containers-image-proxy-rs)
  • Frequency: ~5% of recent failed runs (5 out of ~100 runs checked)
  • Tests affected: Various libvirt/to-disk integration tests:
    • test_to_disk_for_image_quay_io_fedora_fedora_bootc_42
    • test_libvirt_run_bind_storage_ro
    • test_libvirt_comprehensive_workflow
    • test_libvirt_port_forward_connectivity
Error Pattern

All failures occur in paths like:

/run/virtiofs-mnt-hoststorage/overlay/<layer-hash>/diff/sysroot/ostree/repo/objects/<xx>/<hash>.file

These are ostree content-addressed objects with many hardlinks (up to 970+ links for .file-xattrs-link files).

Architecture
Guest VM
  └── virtiofs mount at /run/virtiofs-mnt-hoststorage
        └── virtiofsd (v1.13.0, running unprivileged in container)
              └── Host container storage (~/.local/share/containers/storage/overlay/...)
                    └── ostree repo objects (read-only, hardlinked)
Root Cause Analysis
  1. bcvk passes --inode-file-handles=fallback to virtiofsd (see qemu.rs:932), which should enable file handles when possible.

  2. However, virtiofsd runs unprivileged (inside a podman container with --cap-add=all, but in a user namespace where CAP_DAC_READ_SEARCH doesn't grant real host filesystem capabilities).

  3. Without CAP_DAC_READ_SEARCH, file handles don't work. From virtiofsd README:

    "virtiofsd can't use file handles (--inode-file-handles requires CAP_DAC_READ_SEARCH)"

  4. Fallback to O_PATH FDs: virtiofsd falls back to using O_PATH file descriptors to track inodes. When the guest wants to open a file, virtiofsd reopens via /proc/self/fd/N.

  5. O_PATH FDs are vulnerable to ESTALE: According to virtiofsd docs, using O_PATH FDs instead of file handles "may also be important in cases where virtiofsd should only have file descriptors open for files that are open in the guest, e.g. to get around bad interactions with NFS's silly renaming."

The Puzzle

The container image layers are read-only - no files should be deleted or modified during the test. Yet we're seeing ESTALE. Possible explanations:

  1. Inode generation mismatch: The kernel FUSE code checks inode->i_generation != handle->generation and returns ESTALE on mismatch. virtiofsd always sets generation: 0, so this shouldn't trigger unless something is evicting/recreating inodes.

  2. Hardlink edge case: ostree objects have extremely high hardlink counts. There might be some edge case with how virtiofsd or the kernel handles lookups on highly-hardlinked files.

  3. Race in inode cache: Something in the kernel's inode cache management could be evicting and revalidating inodes during concurrent access, causing transient failures.

  4. Host filesystem peculiarity: Something specific to GitHub Actions runner filesystem configuration.

Relevant Code Paths
  • virtiofsd inode store: When --inode-file-handles=prefer/fallback but capabilities unavailable, falls back to O_PATH FDs (src/passthrough/inode_store.rs:405-417)
  • virtiofsd file open: Reopens via /proc/self/fd/N (src/passthrough/util.rs:41-53)
  • Kernel FUSE staleness check: fuse_stale_inode() in fs/fuse/fuse_i.h:1080-1085
Example Failing Runs
Potential Mitigations to Explore
  1. Run virtiofsd privileged: Grant CAP_DAC_READ_SEARCH to enable actual file handles
  2. Add debug logging: Capture virtiofsd debug output during failures
  3. Retry on ESTALE: Application-level workaround (not ideal)
  4. File upstream bug: With virtiofsd or kernel FUSE maintainers
主要言語
Rust
スター
142
フォーク
40
平均マージ
2日 22時間
マージ済み PR(30日)
10

環境構築

Codespaces で開く

このプロジェクトの開発コンテナを、あなたの GitHub アカウントでブラウザ上に起動します。

  • Dockerfile・Docker Compose ファイルなし
  • プルリクエストのテンプレートなし
  • コントリビューションガイドなし

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

bootc-dev/bcvk のほかの issue

bootc-dev/bcvk の issue をすべて見る

似ている issue

Rust の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。