[Bug]: Linux guests sometimes hang at boot: RCU nesting underflow in rtR0MemObjLinuxVirtToPage (fix included)
Maintainers usually reply within 6 days
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 45/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Active
- Tech stack
- c, linux
- Domain
- backend, operating-systems
Research direction
Start in src/VBox/Runtime/r0drv/linux/memobj-r0drv-linux.c at rtR0MemObjLinuxVirtToPage(), where the issue identifies the PTE mapping and unmapping mismatch. Read the Linux version conditionals and compare the suggested fix with vmalloc_to_page() behavior. Done means the guest module no longer causes RCU nesting underflow or boot stalls on the described kernels; the issue gives a deterministic reproduction using vboxguest reload and repeated unshare -m true.
Written by the indexing model from the issue text.
Description
Version
7.2.22
Host OS Type
Windows
Host OS name + version
Windows / Windows 11 Build 26300 / x86
Host Architecture
x86
Guest OS Type
Linux
Guest Architecture
x86
Guest OS name + version
Ubuntu 24.04, kernel 6.17.0-40-generic, Guest Additions 7.2.6
Component
Guest Additions/Other
What happened?
Linux guests with kernel 6.6+ sometimes hang at boot with rcu_preempt detected expedited stalls. The root cause is the RCU mismatch reported in #359, and it very likely also explains #861 and #227. The code is unchanged in 7.2.22 and on main.
Cause: Since Linux 6.6, pte_unmap() calls rcu_read_unlock(). On 6.5+ and RHEL 9.4+, rtR0MemObjLinuxVirtToPage() reads the PTE with __pte_map(), which takes no RCU lock, and then calls pte_unmap(). That leaves the caller's RCU nesting at −1. vboxguest hits this at module init, when VGDrvCommonProcessOptionsFromHost makes an HGCM call with a buffer on the (vmalloc'ed) stack. Usually this only causes the warning from #227. If an expedited RCU IPI reaches the loading task first, rcu_exp_handler() never reports a quiescent state, and the boot hangs:
WARNING: CPU: 3 PID: 443 at kernel/rcu/tree_exp.h:799 rcu_exp_handler
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 3-...D } ...
Fix: pv is a kernel address, so read the PTE with pte_offset_kernel() and skip pte_unmap(), as vmalloc_to_page() has done since 6.5. E.g.:
--- a/src/VBox/Runtime/r0drv/linux/memobj-r0drv-linux.c
+++ b/src/VBox/Runtime/r0drv/linux/memobj-r0drv-linux.c
@@ -1310,7 +1310,13 @@ RTDECL(struct page *) rtR0MemObjLinuxVirtToPage(void *pv)
# endif
# if RTLNX_VER_MIN(6,5,0) || RTLNX_RHEL_RANGE(9,4, 9,99)
- pEntry = __pte_map(&u.Middle, ulAddr);
+ /* pv is a kernel address, so the page table is never in highmem and
+ * pte_offset_kernel() will do, like in vmalloc_to_page(). Don't pair
+ * __pte_map() with pte_unmap() here: since 6.6 (and in newer RHEL 9
+ * kernels) pte_unmap() does rcu_read_unlock() for the rcu_read_lock()
+ * taken by pte_offset_map(). __pte_map() doesn't take it, so that would
+ * leave the caller's RCU read-side nesting at -1. */
+ pEntry = pte_offset_kernel(&u.Middle, ulAddr);
# elif RTLNX_VER_MIN(2,5,5) || defined(pte_offset_map) /* As usual, RHEL 3 had pte_offset_map earlier. */
pEntry = pte_offset_map(&u.Middle, ulAddr);
# else
@@ -1319,7 +1325,9 @@ RTDECL(struct page *) rtR0MemObjLinuxVirtToPage(void *pv)
if (RT_UNLIKELY(!pEntry))
return NULL;
u.Entry = *pEntry;
-# if RTLNX_VER_MIN(2,5,5) || defined(pte_offset_map)
+# if RTLNX_VER_MIN(6,5,0) || RTLNX_RHEL_RANGE(9,4, 9,99)
+ /* pte_offset_kernel() mapped nothing, so there is nothing to unmap. */
+# elif RTLNX_VER_MIN(2,5,5) || defined(pte_offset_map)
pte_unmap(pEntry);
# endif
How can we reproduce this?
Guest kernel 6.6+ with CONFIG_PREEMPT_RCU=y and CONFIG_VMAP_STACK=y, e.g. Ubuntu 24.04 with the HWE kernel:
nm -u $(modinfo -n vboxguest) | grep rcu_readlists __rcu_read_unlock but not __rcu_read_lock.- Most boots log
WARNING ... tree_plugin.h ... rcu_sched_clock_irqin (udev-worker) when vboxguest loads. Some boots hang as above. - Deterministic: unload vboxguest and load it again with
finit_module()from a process that is pinned to one CPU and keeps busy-looping. Then rununshare -m true60 times. Each mount-namespace teardown waits for an expedited grace period, and the loop stalls permanently (0 of 60 after 65 s; 0.09 s without the reload).
Did you upload all of your necessary log files, screenshots, etc.?
- Yes, I've uploaded all pertinent files to this issue.
- Dominant language
- C
- Stars
- 1.4k
- Forks
- 219
- PR merge metrics
- No merged PRs in 30d
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from VirtualBox/virtualbox
-
bug
Difficulty 1/5 Under an hour Newbie friendliness 84/100
VirtualBox/virtualbox#996 ·
Maintainers usually reply within 6 days
-
enhancement
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
VirtualBox/virtualbox#979 ·
Maintainers usually reply within 6 days
-
[Bug]: Guest Additions 7.2.20 vboxvideo fails to build on Linux kernel 6.12.113 (drm_fb_helper_restore_fbdev_mode_unlocked: too few arguments)Possibly taken @Berbe claimed this 18 days ago. Openbug
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
VirtualBox/virtualbox#918 · 9 comments ·
Maintainers usually reply within 6 days
-
bug
Difficulty 4/5 3-5 days Newbie friendliness 25/100
VirtualBox/virtualbox#995 ·
Maintainers usually reply within 6 days
-
bug
Difficulty 3/5 1-2 days Newbie friendliness 45/100
VirtualBox/virtualbox#994 ·
Maintainers usually reply within 6 days
All issues in VirtualBox/virtualbox
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 80/100
ESPresense/ESPresense#2680 · 2 comments ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
ImageMagick/ImageMagick#9134 ·
Maintainers usually reply within 1 day
-
Builder: a structure opened without calculation data shows the Diffraction (fibre) view by defaultOpenbug
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
FriendsofECCE/ECCE#1245 · 1 comment ·
-
Arch: ARM/AArch64 Doc
Difficulty 1/5 Under an hour Newbie friendliness 90/100
RT-Thread/rt-thread#12082 · 1 comment ·
Maintainers usually reply within 2 days
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
apache/nuttx-apps#4108 · 4 comments ·
Maintainers usually reply within 1 day