openvmm: virtiofs: microVM readdirplus fails with ENOENT when directory entries are unlinked during pagination
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 62/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- operating-systems
Research direction
Start in vm/devices/virtio/virtiofs/src/microvm/file.rs, comparing read_dir_microvm with VirtioFsFile::read_dir_with in vm/devices/virtio/virtiofs/src/file.rs; the latter shows the intended ENOENT/EACCES handling. Follow the plus branch around lookup_helper and the trailing EINVAL check, then reproduce with a two-page directory snapshot whose later entry is unlinked. Done means skipped entries are handled without aborting, buffered lookup references are not leaked, and end-of-directory does not return EINVAL merely because remaining entries were skipped.
Written by the indexing model from the issue text.
Description
Summary
In microVM mode, VirtioFsFile::read_dir_microvm snapshots a directory's entries once when first read at offset == 0 and then calls fs.lookup_helper(&self.inode, name)? on each cached entry during FUSE_READDIRPLUS, propagating Err(lx::Error::ENOENT) directly to the guest instead of skipping unlinked entries as non-microVM VirtioFsFile::read_dir_with does. Unlinking or renaming an entry in a multi-page directory while paginating it (such as during rm -rf) causes FUSE_READDIRPLUS to fail with ENOENT on that page and on every subsequent read of that open directory handle until closed. Found by testing.
Root Cause
In vm/devices/virtio/virtiofs/src/microvm/file.rs, VirtioFsFile::read_dir_microvm materializes snapshot.entries once per open directory handle on the initial offset == 0 read and reuses that immutable snapshot across all continuation reads. In the plus branch (microvm/file.rs), read_dir_microvm uses ? on fs.lookup_helper(&self.inode, name):
// This check must happen before lookup_helper, because a
// readdirplus lookup creates a guest lookup reference.
if !buffer.check_dir_entry_plus(name) {
break;
}
let fuse_entry = fs.lookup_helper(&self.inode, name)?;
if fuse_entry.attr.ino != entry.guest_inode_id {
return Err(lx::Error::EIO);
}
if !buffer.dir_entry_plus(name, entry.next_cookie, fuse_entry) {
return Err(lx::Error::EIO);
}
By contrast, the non-microVM implementation in VirtioFsFile::read_dir_with explicitly ignores ENOENT and EACCES when an enumerated entry disappears or becomes inaccessible before lookup runs:
match lookup(&entry.name) {
Ok(e) => e,
Err(err) => {
// Ignore entries that are inaccessible to the
// user or deleted. ENOENT can occur if a file was
// deleted between enumeration and lookup.
if err.value() == lx::EACCES || err.value() == lx::ENOENT {
entry_count -= 1;
return Ok(true);
}
return Err(err);
}
}
Consequence
Because the microVM profile enables FUSE_DO_READDIRPLUS and FUSE_READDIRPLUS_AUTO, guest directory readers (getdents64) issue paginated FUSE_READDIRPLUS requests (typically 4 KiB per page, holding roughly 25–30 entries). Standard directory-removal and cleanup tools (rm -rf, find -delete, std::fs::remove_dir_all, Go os.RemoveAll, log rotation, and concurrent build/test runners) keep an open directory descriptor while unlinking entries — or encounter entries unlinked or renamed between pages by another guest process or on the host. Once any entry on an unread page of snapshot.entries is unlinked or renamed away, the continuation READDIRPLUS fails with ENOENT, discarding any entries already buffered in that call (and leaking their lookup references unless rolled back). Worse, because the guest's directory offset never advances past the missing entry and snapshot.entries is never rebuilt for that handle, every subsequent READDIRPLUS on that open directory handle fails with ENOENT until closed.
Reproduction
- Create a microVM
VirtioFsshare containing a directory with ~60 files (f00..f59), enough to span at least two 4 KiBFUSE_READDIRPLUSpages. - Open the directory via
FUSE_OPENDIRand issueFUSE_READDIRPLUSatoffset = 0withsize = 4096, populatingdirectory_snapshotand returning the first page of entries ending at cookiec. - Unlink (via
FUSE_UNLINKor on the host) one of the files positioned after cookiecin the snapshot. - Issue
FUSE_READDIRPLUSatoffset = cwithsize = 4096. ObserveErr(lx::Error::ENOENT)on that call and on every retry atoffset = cfor the lifetime of the open handle.
Suggested Fix
In VirtioFsFile::read_dir_microvm, align the plus path with VirtioFsFile::read_dir_with by skipping snapshot entries when fs.lookup_helper(&self.inode, name) returns lx::Error::ENOENT or lx::Error::EACCES (continue to the next snapshot entry instead of aborting). For any other lookup_helper error, stop the loop and return already-buffered entries if buffer is non-empty so lookup references acquired earlier in the batch are not leaked. In addition, adjust the trailing if !entries[start..].is_empty() && buffer.is_empty() { return Err(lx::Error::EINVAL); } check so it only returns EINVAL when size was too small to fit a live entry (!buffer.check_dir_entry_plus(name)), rather than returning EINVAL at end-of-directory when every remaining entry in entries[start..] was skipped with ENOENT or EACCES.
Related: #1611, #1605, #1634.
- Dominant language
- Rust
- Stars
- 831
- Forks
- 56
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 571
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/nvx
-
area: ci
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
area: ci
Difficulty 1/5 Under an hour Newbie friendliness 90/100
Maintainers usually reply within 1 day
-
openvmm: virtiofs: aggregate synthetic root st_nlink counts single-file shares as subdirectoriesOpenarea: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
microsoft/nvx#1632 · 1 comment ·
Maintainers usually reply within 1 day
-
area: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
microsoft/nvx#1628 · 1 comment ·
Maintainers usually reply within 1 day
-
area: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
microsoft/nvx#1629 · 1 comment ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
oxc-project/tsrs#1010 ·
Maintainers usually reply within 1 day
-
Link is deadOpen
Difficulty 1/5 Under an hour Newbie friendliness 63/100
nicholas-ochoa/OpenSC2K#104 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
libracoder/kiri#445 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
DioxusLabs/blitz#1614 ·
Maintainers usually reply within 1 day