openvmm: virtiofs: microVM readdirplus fails with ENOENT when directory entries are unlinked during pagination

Open
#1,622 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
62/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust

Research direction

Start in vm/devices/virtio/virtiofs/src/microvm/file.rs, comparing read_dir_microvm with VirtioFsFile::read_dir_with in vm/devices/virtio/virtiofs/src/file.rs; the latter shows the intended ENOENT/EACCES handling. Follow the plus branch around lookup_helper and the trailing EINVAL check, then reproduce with a two-page directory snapshot whose later entry is unlinked. Done means skipped entries are handled without aborting, buffered lookup references are not leaked, and end-of-directory does not return EINVAL merely because remaining entries were skipped.

Written by the indexing model from the issue text.

Description

area: openvmm

Summary

In microVM mode, VirtioFsFile::read_dir_microvm snapshots a directory's entries once when first read at offset == 0 and then calls fs.lookup_helper(&self.inode, name)? on each cached entry during FUSE_READDIRPLUS, propagating Err(lx::Error::ENOENT) directly to the guest instead of skipping unlinked entries as non-microVM VirtioFsFile::read_dir_with does. Unlinking or renaming an entry in a multi-page directory while paginating it (such as during rm -rf) causes FUSE_READDIRPLUS to fail with ENOENT on that page and on every subsequent read of that open directory handle until closed. Found by testing.

Root Cause

In vm/devices/virtio/virtiofs/src/microvm/file.rs, VirtioFsFile::read_dir_microvm materializes snapshot.entries once per open directory handle on the initial offset == 0 read and reuses that immutable snapshot across all continuation reads. In the plus branch (microvm/file.rs), read_dir_microvm uses ? on fs.lookup_helper(&self.inode, name):

                    // This check must happen before lookup_helper, because a
                    // readdirplus lookup creates a guest lookup reference.
                    if !buffer.check_dir_entry_plus(name) {
                        break;
                    }
                    let fuse_entry = fs.lookup_helper(&self.inode, name)?;
                    if fuse_entry.attr.ino != entry.guest_inode_id {
                        return Err(lx::Error::EIO);
                    }
                    if !buffer.dir_entry_plus(name, entry.next_cookie, fuse_entry) {
                        return Err(lx::Error::EIO);
                    }

By contrast, the non-microVM implementation in VirtioFsFile::read_dir_with explicitly ignores ENOENT and EACCES when an enumerated entry disappears or becomes inaccessible before lookup runs:

                    match lookup(&entry.name) {
                        Ok(e) => e,
                        Err(err) => {
                            // Ignore entries that are inaccessible to the
                            // user or deleted. ENOENT can occur if a file was
                            // deleted between enumeration and lookup.
                            if err.value() == lx::EACCES || err.value() == lx::ENOENT {
                                entry_count -= 1;
                                return Ok(true);
                            }
                            return Err(err);
                        }
                    }

Consequence

Because the microVM profile enables FUSE_DO_READDIRPLUS and FUSE_READDIRPLUS_AUTO, guest directory readers (getdents64) issue paginated FUSE_READDIRPLUS requests (typically 4 KiB per page, holding roughly 25–30 entries). Standard directory-removal and cleanup tools (rm -rf, find -delete, std::fs::remove_dir_all, Go os.RemoveAll, log rotation, and concurrent build/test runners) keep an open directory descriptor while unlinking entries — or encounter entries unlinked or renamed between pages by another guest process or on the host. Once any entry on an unread page of snapshot.entries is unlinked or renamed away, the continuation READDIRPLUS fails with ENOENT, discarding any entries already buffered in that call (and leaking their lookup references unless rolled back). Worse, because the guest's directory offset never advances past the missing entry and snapshot.entries is never rebuilt for that handle, every subsequent READDIRPLUS on that open directory handle fails with ENOENT until closed.

Reproduction

  1. Create a microVM VirtioFs share containing a directory with ~60 files (f00..f59), enough to span at least two 4 KiB FUSE_READDIRPLUS pages.
  2. Open the directory via FUSE_OPENDIR and issue FUSE_READDIRPLUS at offset = 0 with size = 4096, populating directory_snapshot and returning the first page of entries ending at cookie c.
  3. Unlink (via FUSE_UNLINK or on the host) one of the files positioned after cookie c in the snapshot.
  4. Issue FUSE_READDIRPLUS at offset = c with size = 4096. Observe Err(lx::Error::ENOENT) on that call and on every retry at offset = c for the lifetime of the open handle.

Suggested Fix

In VirtioFsFile::read_dir_microvm, align the plus path with VirtioFsFile::read_dir_with by skipping snapshot entries when fs.lookup_helper(&self.inode, name) returns lx::Error::ENOENT or lx::Error::EACCES (continue to the next snapshot entry instead of aborting). For any other lookup_helper error, stop the loop and return already-buffered entries if buffer is non-empty so lookup references acquired earlier in the batch are not leaked. In addition, adjust the trailing if !entries[start..].is_empty() && buffer.is_empty() { return Err(lx::Error::EINVAL); } check so it only returns EINVAL when size was too small to fit a live entry (!buffer.check_dir_entry_plus(name)), rather than returning EINVAL at end-of-directory when every remaining entry in entries[start..] was skipped with ENOENT or EACCES.

Related: #1611, #1605, #1634.

Dominant language
Rust
Stars
831
Forks
56
Avg merge
13h 37m
Merged PRs (30d)
571

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/nvx

All issues in microsoft/nvx

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.