openvmm: virtiofs: microVM readdirplus leaks lookup references on mid-listing errors

Open
#1,611 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
rust

Research direction

Start in vm/devices/virtio/virtiofs/src/microvm/file.rs at read_dir_microvm and compare it with the non-microVM read_dir_with in src/file.rs, which returns a partially filled buffer before reporting an error. Then read the InodeMap lookup/forget accounting used by lookup_helper (including the caps in microvm/limits.rs) to see what must be released when the loop aborts. Reproduce with the three-file rename scenario, then re-run the microvm virtiofs read_dir_plus tests; done means every error path leaves no extra lookup references and earlier serialized entries are not silently dropped.

Written by the indexing model from the issue text.

Description

Summary

In microVM mode, VirtioFsFile::read_dir_microvm acquires a FUSE lookup reference via fs.lookup_helper for each non-dot FUSE_READDIRPLUS entry before checking whether the entry matches the handle's snapshot or fits in the reply buffer, and aborts the entire READDIRPLUS call with Err(...) whenever any entry fails. This leaks the lookup reference of the failing entry when lookup_helper already succeeded, and also leaks the lookup references of all earlier entries already serialized into buffer because the guest never receives those entries in an error reply and therefore never sends FUSE_FORGET for them. Found during formal specification (blocks formal verification).

Root Cause

In vm/devices/virtio/virtiofs/src/microvm/file.rs, read_dir_microvm returns Err(...) directly out of the entries loop in three places after one or more lookup references have already been incremented in InodeMap:

                    // This check must happen before lookup_helper, because a
                    // readdirplus lookup creates a guest lookup reference.
                    if !buffer.check_dir_entry_plus(name) {
                        break;
                    }
                    let fuse_entry = fs.lookup_helper(&self.inode, name)?;
                    if fuse_entry.attr.ino != entry.guest_inode_id {
                        return Err(lx::Error::EIO);
                    }
                    if !buffer.dir_entry_plus(name, entry.next_cookie, fuse_entry) {
                        return Err(lx::Error::EIO);
                    }

First, fs.lookup_helper(&self.inode, name)? on line 70 discards buffer if lookup_helper fails after earlier loop iterations already succeeded and appended fuse_direntplus records into buffer (unlike non-microVM VirtioFsFile::read_dir_with, which returns a non-empty buffer before reporting an error). Second, when fuse_entry.attr.ino != entry.guest_inode_id (line 71) or !buffer.dir_entry_plus(...) (line 74), lookup_helper has already inserted or incremented fuse_entry.nodeid in InodeMap, and returning Err(lx::Error::EIO) leaks both fuse_entry.nodeid's new lookup reference and every earlier entry's lookup reference in buffer.

Consequence

Every failed FUSE_READDIRPLUS call in microVM mode — triggered when a directory entry is unlinked (ENOENT), replaced with a new inode (attr.ino != entry.guest_inode_id), or denied by policy (EACCES) after the directory handle's snapshot was built — permanently leaks one lookup reference per previously buffered entry plus one lookup reference for any entry rejected after lookup_helper. Leaked inodes can never reach lookup_count == 0 via guest FUSE_FORGET requests, so they remain pinned in InodeMap (along with their recorded aliases and host paths) until FUSE_DESTROY, steadily consuming the microVM attachment's hard MAX_INODES, MAX_ALIASES, and MAX_ALIAS_BYTES caps in vm/devices/virtio/virtiofs/src/microvm/limits.rs until new lookups and file creations fail with ENOSPC or E2BIG, and inflating saved-state size across save/restore cycles.

Reproduction

  1. Create a microVM VirtioFs share containing three files ("first", "second", "third"), open the root directory, and call read_dir once to build the directory handle's cached snapshot and observe the entry order [first, second, third].
  2. Create a new file "replacement" in the share while "second" is still linked (guaranteeing a distinct host inode number) and rename "replacement" over "second".
  3. Call fs.read_dir_plus at offset = 0 with size = 4096.
  4. read_dir_plus looks up "first" (lookup_count = 1) and "second" (lookup_count = 1), sees fuse_entry.attr.ino != entry.guest_inode_id on "second", and returns Err(lx::Error::EIO). Because the guest received an error instead of first or second, it never sends FUSE_FORGET for either node, leaving both inodes leaked in fs.inner.inodes.

Related: #1605, #1634, #1622, #1626.

Dominant language
Rust
Stars
848
Forks
63
Avg merge
13h 37m
Merged PRs (30d)
571

Getting set up

  • No Dockerfile or Docker Compose file
  • Has a pull request template
  • No contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from microsoft/nvx

All issues in microsoft/nvx

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.