[BUG] risc-v/SMP: Missing remote TLB shutdown allows access to a freed and reused page

Start with the shared-memory unmap path in mm/shm/shmdt.c and mm/shm/shmat.c, then trace riscv_unmap_pages() in arch/risc-v/src/common/riscv_addrenv_utils.c to the local SFENCE.VMA in riscv_mmu.h. Reproduce with the described rv-virt:ksmp64 configuration and tlbstale test; inspect SBI RFENCE definitions in riscv_sbi.h and the IPI fallback requirements. Done means remote invalidations complete before unmap returns and the cross-hart reproducer faults instead of reading a reused page.

Description / Steps to reproduce the issue

In RISC-V SMP kernel builds, unmapping user pages invalidates the TLB only on the hart performing the unmap. Other harts running threads of the same address environment can retain a valid translation to the unmapped physical page.

I reproduced this on rv-virt:ksmp64 under QEMU. After shmdt() and shmctl(IPC_RMID), a reader thread on another hart continued accessing the unmapped shared-memory page. When the physical page was reallocated to a writer process, the stale reader observed the writer's data through the old mapping.

In three first-run-after-boot tests, the reader observed the new writer's 0xb0b0b0b0 marker in all 1,024 words of the page. The same-hart control faults as expected, and a proof of concept using SBI remote SFENCE.VMA makes the cross-hart reader fault as expected.

This demonstrates cross-process data exposure through a stale translation in the tested configuration. The experiment has so far been conducted in QEMU, not on physical hardware.

Root cause

The shared-memory unmap path reaches a local-only TLB invalidation:

shmdt()                          mm/shm/shmdt.c
  munmap_shm()                   mm/shm/shmat.c
    up_shmdt()                   arch/risc-v/src/common/riscv_addrenv_shm.c
      riscv_unmap_pages()        arch/risc-v/src/common/riscv_addrenv_utils.c
        mmu_ln_clear()           arch/risc-v/src/common/riscv_mmu.c
          sfence.vma x0, va      riscv_mmu.h:284, local hart only

SFENCE.VMA invalidates translations on the executing hart; it does not invalidate the corresponding entries on other harts. The remote harts are not notified when the page table is changed.

The SBI RFENCE function IDs are defined in riscv_sbi.h, but the current RISC-V code does not use them for this path.

The same riscv_unmap_pages() function is also used by:

• User pgmap unmapping (riscv_addrenv_pgmap.c).
• Kernel kmap unmapping (up_addrenv_kunmap_pages()).

These paths therefore warrant investigation for the same local-only invalidation problem.

Steps to reproduce

Environment

• NuttX configuration: rv-virt:ksmp64
• Build options: CONFIG_BUILD_KERNEL=y, CONFIG_SMP=y
• QEMU: 8.2.2
• Firmware: OpenSBI v1.3
• Toolchain: riscv64-unknown-elf-gcc
• NuttX revisions tested: 5fda288a2a and 284d4d4ea4 (NuttX-13.1.0)
• Four RISC-V harts, running in S-mode

Reproducer

The test application is testing/mm/tlbstale, available on these branches:

• NuttX: https://github.com/PiyushPatle26/nuttx/tree/riscv-smp-stale-tlb
• NuttX apps: https://github.com/PiyushPatle26/nuttx-apps/tree/riscv-smp-stale-tlb
  The NuttX branch includes a change needed to link the test application in a kernel build; details are in the notes below.

To build and run it (with nuttx and apps checked out side by side):

cd nuttx
./tools/configure.sh -l rv-virt:ksmp64
kconfig-tweak --file .config --enable CONFIG_MM_SHM \
  --set-val CONFIG_ARCH_SHM_VBASE 0xC2000000 \
  --set-val CONFIG_ARCH_SHM_NPAGES 16 \
  --set-val CONFIG_ARCH_SHM_MAXREGIONS 4 \
  --enable CONFIG_TESTING_TLBSTALE
make olddefconfig
make -j$(nproc) && make export -j$(nproc)
cd ../apps
./tools/mkimport.sh -z -x ../nuttx/nuttx-export-*.tar.gz
make import -j$(nproc)
cd ../nuttx
qemu-system-riscv64 -semihosting -M virt,aclint=on -cpu rv64 -smp 4 -kernel nuttx -nographic
nsh> tlbstale -x     # as the first command after boot
nsh> tlbstale -l     # control

How the reproducer works

1. The main thread is pinned to CPU0. It creates a one-page shared-memory segment and fills it with 0x5a5aa5a5.
2. A reader thread in the same process is pinned to CPU1 and continuously reads the page.
3. With -x, three writer processes are created first and blocked on SIGUSR1.
4. The main thread calls shmdt() and shmctl(IPC_RMID), unmapping the page and returning it to the page pool. It then publishes a flag with release semantics and signals the writers.
5. Writer 0 performs shmget() and obtains the freed page. Its allocation zeroes the page, after which it fills the page with 0xb0b0b0b0.
6. The reader observes the flag with acquire semantics and continues scanning the virtual address through its stale translation. On a correctly invalidated mapping, its first post-unmap access should fault instead.

The -l option runs the reader on the same hart as the unmapper to provide a control case.

Results

Case	Result
tlbstale (reader on CPU1), 3 runs	stale: ~39.9M successful loads after shmdt() returned, every run
tlbstale -c 2, -c 3	stale: 39.99M / 39.17M loads
tlbstale -x, first run after a fresh boot, 3 of 3 boots (incl. the published branches)	leak: all 1024 words went 0x5a5aa5a5 → 0 → 0xb0b0b0b0, i.e. the reader saw writer 0's shmget() zero the reused page and then its marker, ~53M stale loads
tlbstale -x, later runs in the same boot	stale, but the freed page wasn't reallocated within the window (see notes)
tlbstale -l (control, same hart)	load page fault, SIGSEGV (correct)
PoC: SBI remote_sfence_vma added to riscv_unmap_pages()	reader on CPU1 / CPU3 faults like the control (correct)

nsh> tlbstale -x
tlbstale: reader on CPU1, main on CPU0 (cross-CPU unmap, cross-process reuse)
tlbstale -w0: filled 16 shm pages and 256 heap blocks with 0xb0b0b0b0
...
tlbstale: FAIL: the reader on CPU1 made 53014550 loads after shmdt() returned.
tlbstale: scanned the page 51772 times: 1024 of 1024 words changed after the free, 1024 held the writer's marker 0xb0b0b0b0
tlbstale:   load #27881 word 210 read 0x00000000 (unknown owner's data)
...
tlbstale: last view of the page: 0 words our old data, 1024 the writer's marker, 0 zero, 0 other
tlbstale: LEAK: the freed page was reallocated to a writer process, and this process read the writer's data

nsh> tlbstale -l
tlbstale: reader on CPU0, main on CPU0 (control: same CPU)
[CPU0] riscv_exception: EXCEPTION: Load page fault. MCAUSE: 000000000000000d, EPC: 00000000c000014e, MTVAL: 00000000c2000b90
[CPU0] riscv_fault_handler: Segmentation fault in tlbstale (PID 29: tlbstale)

Proof of concept

The diagnostic PoC adds an SBI remote SFENCE.VMA call to riscv_unmap_pages().

It broadcasts to all harts unconditionally, does not probe for RFENCE support, has no IPI fallback, and does not support NuttSBI. It confirms the suspected cause in the tested OpenSBI configuration but is not proposed as a production fix.

Proposed direction

I suggest addressing this in two stages.

1. Fix RISC-V SMP correctness first
   Ensure that riscv_unmap_pages() does not return until the necessary remote invalidations have completed. Use SBI RFENCE when the firmware advertises the required extension and provide a synchronous IPI-based fallback for configurations without RFENCE, including NuttSBI. The fallback must be reviewed for deadlocks involving critical sections and locks.

2. Discuss a generic TLB-maintenance contract separately
   A synchronous architecture-independent interface could allow common unmap paths to express the required completion semantics while architecture backends implement them appropriately. ARM/ARM64 could use their existing broadcast maintenance, while x86_64 would need to ensure remote invalidation has completed before returning. This interface would also provide a foundation for future RISC-V ASID support.

I would welcome feedback on whether this separation is appropriate and on the best place to establish the generic interface.

On which OS does this issue occur?

[OS: Linux]

What is the version of your OS?

Ubuntu 24

NuttX Version

13.0.1

Issue Architecture

[Arch: risc-v]

Issue Area

[Area: Memory Management]

Host information

No response

Verification

• I have verified before submitting the report.