openvmm: virtiofs: microVM readdirplus leaks lookup references on mid-listing errors
Maintainers usually reply within 1 day
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 48/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- rust
- Domain
- operating-systems
Research direction
Start in vm/devices/virtio/virtiofs/src/microvm/file.rs at read_dir_microvm and compare it with the non-microVM read_dir_with in src/file.rs, which returns a partially filled buffer before reporting an error. Then read the InodeMap lookup/forget accounting used by lookup_helper (including the caps in microvm/limits.rs) to see what must be released when the loop aborts. Reproduce with the three-file rename scenario, then re-run the microvm virtiofs read_dir_plus tests; done means every error path leaves no extra lookup references and earlier serialized entries are not silently dropped.
Written by the indexing model from the issue text.
Description
Summary
In microVM mode, VirtioFsFile::read_dir_microvm acquires a FUSE lookup reference via fs.lookup_helper for each non-dot FUSE_READDIRPLUS entry before checking whether the entry matches the handle's snapshot or fits in the reply buffer, and aborts the entire READDIRPLUS call with Err(...) whenever any entry fails. This leaks the lookup reference of the failing entry when lookup_helper already succeeded, and also leaks the lookup references of all earlier entries already serialized into buffer because the guest never receives those entries in an error reply and therefore never sends FUSE_FORGET for them. Found during formal specification (blocks formal verification).
Root Cause
In vm/devices/virtio/virtiofs/src/microvm/file.rs, read_dir_microvm returns Err(...) directly out of the entries loop in three places after one or more lookup references have already been incremented in InodeMap:
// This check must happen before lookup_helper, because a
// readdirplus lookup creates a guest lookup reference.
if !buffer.check_dir_entry_plus(name) {
break;
}
let fuse_entry = fs.lookup_helper(&self.inode, name)?;
if fuse_entry.attr.ino != entry.guest_inode_id {
return Err(lx::Error::EIO);
}
if !buffer.dir_entry_plus(name, entry.next_cookie, fuse_entry) {
return Err(lx::Error::EIO);
}
First, fs.lookup_helper(&self.inode, name)? on line 70 discards buffer if lookup_helper fails after earlier loop iterations already succeeded and appended fuse_direntplus records into buffer (unlike non-microVM VirtioFsFile::read_dir_with, which returns a non-empty buffer before reporting an error). Second, when fuse_entry.attr.ino != entry.guest_inode_id (line 71) or !buffer.dir_entry_plus(...) (line 74), lookup_helper has already inserted or incremented fuse_entry.nodeid in InodeMap, and returning Err(lx::Error::EIO) leaks both fuse_entry.nodeid's new lookup reference and every earlier entry's lookup reference in buffer.
Consequence
Every failed FUSE_READDIRPLUS call in microVM mode — triggered when a directory entry is unlinked (ENOENT), replaced with a new inode (attr.ino != entry.guest_inode_id), or denied by policy (EACCES) after the directory handle's snapshot was built — permanently leaks one lookup reference per previously buffered entry plus one lookup reference for any entry rejected after lookup_helper. Leaked inodes can never reach lookup_count == 0 via guest FUSE_FORGET requests, so they remain pinned in InodeMap (along with their recorded aliases and host paths) until FUSE_DESTROY, steadily consuming the microVM attachment's hard MAX_INODES, MAX_ALIASES, and MAX_ALIAS_BYTES caps in vm/devices/virtio/virtiofs/src/microvm/limits.rs until new lookups and file creations fail with ENOSPC or E2BIG, and inflating saved-state size across save/restore cycles.
Reproduction
- Create a microVM
VirtioFsshare containing three files ("first","second","third"), open the root directory, and callread_dironce to build the directory handle's cached snapshot and observe the entry order[first, second, third]. - Create a new file
"replacement"in the share while"second"is still linked (guaranteeing a distinct host inode number) and rename"replacement"over"second". - Call
fs.read_dir_plusatoffset = 0withsize = 4096. read_dir_pluslooks up"first"(lookup_count = 1) and"second"(lookup_count = 1), seesfuse_entry.attr.ino != entry.guest_inode_idon"second", and returnsErr(lx::Error::EIO). Because the guest received an error instead offirstorsecond, it never sendsFUSE_FORGETfor either node, leaving both inodes leaked infs.inner.inodes.
Related: #1605, #1634, #1622, #1626.
- Dominant language
- Rust
- Stars
- 848
- Forks
- 63
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 571
Getting set up
- No Dockerfile or Docker Compose file
- Has a pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/nvx
-
area: ci
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Maintainers usually reply within 1 day
-
area: ci
Difficulty 1/5 Under an hour Newbie friendliness 90/100
Maintainers usually reply within 1 day
-
openvmm: virtiofs: aggregate synthetic root st_nlink counts single-file shares as subdirectoriesOpenarea: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
microsoft/nvx#1632 · 1 comment ·
Maintainers usually reply within 1 day
-
area: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
microsoft/nvx#1628 · 1 comment ·
Maintainers usually reply within 1 day
-
area: openvmm
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
microsoft/nvx#1629 · 1 comment ·
Maintainers usually reply within 1 day
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
obi1kenobi/cargo-semver-checks#1814 ·
Maintainers usually reply within 1 day
-
community differential help wanted
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
hffgzktbcc-blip/ounce-core#489 · 1 comment ·
Maintainers usually reply within 1 day
-
needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
pbakaus/impeccable#1560 ·
Maintainers usually reply within 1 day
-
Difficulty 1/5 Under an hour Newbie friendliness 68/100
pact-foundation/pact-plugins#125 ·