[Bug]: Linux guests sometimes hang at boot: RCU nesting underflow in rtR0MemObjLinuxVirtToPage (fix included)

Open
#940 1 comment 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 6 days

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
45/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
c, linux

Research direction

Start in src/VBox/Runtime/r0drv/linux/memobj-r0drv-linux.c at rtR0MemObjLinuxVirtToPage(), where the issue identifies the PTE mapping and unmapping mismatch. Read the Linux version conditionals and compare the suggested fix with vmalloc_to_page() behavior. Done means the guest module no longer causes RCU nesting underflow or boot stalls on the described kernels; the issue gives a deterministic reproduction using vboxguest reload and repeated unshare -m true.

Written by the indexing model from the issue text.

Description

bug
Version

7.2.22

Host OS Type

Windows

Host OS name + version

Windows / Windows 11 Build 26300 / x86

Host Architecture

x86

Guest OS Type

Linux

Guest Architecture

x86

Guest OS name + version

Ubuntu 24.04, kernel 6.17.0-40-generic, Guest Additions 7.2.6

Component

Guest Additions/Other

What happened?

Linux guests with kernel 6.6+ sometimes hang at boot with rcu_preempt detected expedited stalls. The root cause is the RCU mismatch reported in #359, and it very likely also explains #861 and #227. The code is unchanged in 7.2.22 and on main.

Cause: Since Linux 6.6, pte_unmap() calls rcu_read_unlock(). On 6.5+ and RHEL 9.4+, rtR0MemObjLinuxVirtToPage() reads the PTE with __pte_map(), which takes no RCU lock, and then calls pte_unmap(). That leaves the caller's RCU nesting at −1. vboxguest hits this at module init, when VGDrvCommonProcessOptionsFromHost makes an HGCM call with a buffer on the (vmalloc'ed) stack. Usually this only causes the warning from #227. If an expedited RCU IPI reaches the loading task first, rcu_exp_handler() never reports a quiescent state, and the boot hangs:

WARNING: CPU: 3 PID: 443 at kernel/rcu/tree_exp.h:799 rcu_exp_handler
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-...D } ...

Fix: pv is a kernel address, so read the PTE with pte_offset_kernel() and skip pte_unmap(), as vmalloc_to_page() has done since 6.5. E.g.:

--- a/src/VBox/Runtime/r0drv/linux/memobj-r0drv-linux.c
+++ b/src/VBox/Runtime/r0drv/linux/memobj-r0drv-linux.c
@@ -1310,7 +1310,13 @@ RTDECL(struct page *) rtR0MemObjLinuxVirtToPage(void *pv)
 # endif
 
 # if RTLNX_VER_MIN(6,5,0) || RTLNX_RHEL_RANGE(9,4, 9,99)
-    pEntry = __pte_map(&u.Middle, ulAddr);
+    /* pv is a kernel address, so the page table is never in highmem and
+     * pte_offset_kernel() will do, like in vmalloc_to_page().  Don't pair
+     * __pte_map() with pte_unmap() here: since 6.6 (and in newer RHEL 9
+     * kernels) pte_unmap() does rcu_read_unlock() for the rcu_read_lock()
+     * taken by pte_offset_map().  __pte_map() doesn't take it, so that would
+     * leave the caller's RCU read-side nesting at -1. */
+    pEntry = pte_offset_kernel(&u.Middle, ulAddr);
 # elif RTLNX_VER_MIN(2,5,5) || defined(pte_offset_map) /* As usual, RHEL 3 had pte_offset_map earlier. */
     pEntry = pte_offset_map(&u.Middle, ulAddr);
 # else
@@ -1319,7 +1325,9 @@ RTDECL(struct page *) rtR0MemObjLinuxVirtToPage(void *pv)
     if (RT_UNLIKELY(!pEntry))
         return NULL;
     u.Entry = *pEntry;
-# if RTLNX_VER_MIN(2,5,5) || defined(pte_offset_map)
+# if RTLNX_VER_MIN(6,5,0) || RTLNX_RHEL_RANGE(9,4, 9,99)
+    /* pte_offset_kernel() mapped nothing, so there is nothing to unmap. */
+# elif RTLNX_VER_MIN(2,5,5) || defined(pte_offset_map)
     pte_unmap(pEntry);
 # endif
 
How can we reproduce this?

Guest kernel 6.6+ with CONFIG_PREEMPT_RCU=y and CONFIG_VMAP_STACK=y, e.g. Ubuntu 24.04 with the HWE kernel:

  • nm -u $(modinfo -n vboxguest) | grep rcu_read lists __rcu_read_unlock but not __rcu_read_lock.
  • Most boots log WARNING ... tree_plugin.h ... rcu_sched_clock_irq in (udev-worker) when vboxguest loads. Some boots hang as above.
  • Deterministic: unload vboxguest and load it again with finit_module() from a process that is pinned to one CPU and keeps busy-looping. Then run unshare -m true 60 times. Each mount-namespace teardown waits for an expedited grace period, and the loop stalls permanently (0 of 60 after 65 s; 0.09 s without the reload).
Did you upload all of your necessary log files, screenshots, etc.?
  • Yes, I've uploaded all pertinent files to this issue.
Dominant language
C
Stars
1.4k
Forks
219
PR merge metrics
No merged PRs in 30d

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from VirtualBox/virtualbox

All issues in VirtualBox/virtualbox

Similar issues

More C issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.