[GSD-13703][Linux][BMG/B70] Dual-GPU TP=2: DEFER_BACKING BOs get host pages when dma-buf exported (xe_gem_prime_export -> ttm_bo_populate), +~25 GiB host RAM on 26.35; EnableDeferBacking=0 avoids it

The report traces the issue through the kernel path xe_gem_prime_export → ttm_bo_setup_export → ttm_bo_populate and identifies DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING BOs as the affected case. Start by reading the cited kernel commit 91494dee10 and the traces and suggested fix directions in this issue; there is no standalone reproducer yet. Done means determining how to avoid unnecessary host-page population for VRAM-placeable BOs exported before validation; the report suggests a kernel-side fix, but runtime options are also discussed.

Pre-submission Checklist

• I am using the latest GPU driver version (26.35.39758.10, the newest release)
• I have searched for similar issues. This is related to #953, #980 and #974, see "Relation to existing issues" below. Intel asked (in #953) for the dual-GPU vLLM tensor-parallel residual to be filed separately, with EnableDeferBacking=0 tested: this is that report.

Summary

On 2x Arc Pro B70 with vLLM tensor parallel (TP=2, both GPUs visible to each worker), the KV cache and other large device allocations made after VRAM use passes DeferBackingMemoryPressurePercent (default 50) are created by NEO with DRM_XE_GEM_CREATE_FLAG_DEFER_BACKING. The xe driver creates such a BO in system placement with an unpopulated TT. When NEO then dma-buf exports the BO (for multi-device peer access) before its first validation into VRAM, the kernel (xe_gem_prime_export -> ttm_bo_setup_export -> ttm_bo_populate, from kernel commit 91494dee10, v6.18+) allocates system pages for it and keeps them. Result: roughly 1:1 hidden host RAM for those allocations, about +24-28 GiB for this workload, which pushes a 123 GiB host into the OOM killer when a second GPU workload runs.

Whether the export hits a BO that still has a TT is timing dependent, so the effect is intermittent for the same binaries and parameters (details below). EnableDeferBacking=0 avoids it completely in every state tested, with no measurable performance change.

GPU Hardware

3x Intel Arc Pro B70 (Battlemage G31, 8086:e223, 32 GiB each). The two used for TP are 0000:43:00.0 and 0000:47:00.0; the third (0000:63:00.0) runs an unrelated single-GPU workload.

DRI Devices Information

$ ls -la /dev/dri/by-path
pci-0000:43:00.0-card -> ../card0   pci-0000:43:00.0-render -> ../renderD129
pci-0000:47:00.0-card -> ../card2   pci-0000:47:00.0-render -> ../renderD130
pci-0000:63:00.0-card -> ../card1   pci-0000:63:00.0-render -> ../renderD128

GPU Detailed Information (lspci output, one card)

43:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics] (prog-if 00 [VGA controller])
	Subsystem: Intel Corporation Device 1701
	IOMMU group: 31
	Region 0: Memory at 15800000000 (64-bit, prefetchable) [size=16M]
	Region 2: Memory at 14800000000 (64-bit, prefetchable) [size=32G]

Peer-to-peer works on this platform (AMD Ryzen Threadripper PRO 5955WX, 123.6 GiB usable RAM; the two GPUs sit behind separate root ports): every dma-buf attach seen in the traces below had attach->peer2peer == true (2440 of 2440).

Driver Version

26.35.39758.10 (libze-intel-gpu1, intel-opencl-icd), IGC 2.41.5, level-zero loader 1.32.0

Installed GPU Driver Packages (inside the container that runs the workload)

intel-igc-core-2 2.41.5
intel-igc-opencl-2 2.41.5
intel-opencl-icd 26.35.39758.10-0
libigdgmm12 22.10.1
libze-intel-gpu1 26.35.39758.10-0
libze1 1.32.0
torch 2.14.0+xpu, oneccl 2022.1.2, vllm 0.31.0+xpu, vllm-xpu-kernels 0.1.15.4

Linux Distribution / Kernel

Ubuntu 26.04.1 LTS, kernel 7.0.0-34-generic (#34-Ubuntu), stock xe. Boot parameters: root=... ro crashkernel=... (nothing GPU related; no iommu=/pci= options). iommu_group type is DMA-FQ.

Actual Behavior

vLLM TP=2 with a 27B model (FP8 weights, FP8 KV, 262144 context, --gpu-memory-utilization 0.94), ZE_AFFINITY_MASK=0,1, one worker process per GPU.

Host RAM (free, "used"), all stock kernel module, same container image, same command line:

setting	host RAM after boot	notes
default (defer backing on)	41-47 GiB in the bad case	steps up by ~20 GiB at KV-cache allocation
default, other runs of the same config	18-21 GiB	mirror absent (intermittent, see below)
EnableDeferBacking=0	18 GiB in every run	same KV size (527,266 tokens), same prefill latency, decode 61.0 vs 60.5 tok/s over 24 samples each

18 GiB is the processes' own resident memory (~13 GiB) plus other services.

BO creation (NEO PrintBOCreateDestroyResult=1) during one boot:

flags=0x1 (DEFER_BACKING) placement=0x2 (VRAM): 7493 BOs, 54.1 GiB   <- includes a single 9.55 GiB BO = the KV cache
flags=0x0                 placement=0x2 (VRAM): 3753 BOs, 32.3 GiB   <- mostly weights
system-only placement (0x1):                     694 BOs, ~1 GiB

No BO is created with a VRAM+SYSTEM placement.

fdinfo of a TP worker (/proc/<pid>/fdinfo, xe client of the peer device), identical in good and bad runs, so the accounting alone does not distinguish them:

local device fd:  drm-total-vram0: 27.2 GiB, drm-shared-vram0: 25.7 GiB   (nearly everything is exported)
peer  device fd:  drm-total-gtt:   25.7 GiB, drm-shared-gtt:   25.7 GiB

Kernel trace of a bad run (bpftrace kprobes on ttm_bo_setup_export / ttm_bo_populate, one boot of the workload):

ttm_bo_setup_export calls:                2400   (97,138 MiB of BOs exported)
  of which bo->ttm != NULL (TT present):  1224   (48,856 MiB)   <- these get populated
large (>=256 MiB) exports with a TT: 14, e.g. 1214 MiB, 1214 MiB, 608 MiB, 384 MiB, 256 MiB
populate stack:
  xe_ttm_tt_populate
  ttm_bo_populate
  ttm_bo_setup_export
  xe_gem_prime_export
  drm_gem_prime_handle_to_dmabuf
  drm_prime_handle_to_fd_ioctl

In good runs those exports find no TT (the BO was already validated into VRAM). xe_bo_validate-driven populates (first use) occur in both cases and are freed again.

Intermittency (same binaries, all boot-time module state; the only variable is a kernel module parameter I added that disables the populate in xe_gem_prime_export for BOs that have a VRAM placement, see the patch below): without the patch 3 of 7 runs showed the 41-48 GiB state; with the patch 0 of 5 did (host RAM 20.8-20.9 GiB every time). Small sample, but the mechanism is traced.

Expected Behavior

Exporting a device allocation for peer access should not allocate and retain an equal amount of system memory for a BO that lives in VRAM. Host RAM use of a dual-GPU tensor-parallel workload should stay at about the processes' own resident memory.

Reproduction Rate

Intermittent with default settings (timing dependent; the rate is not well characterised: 3 of 7 runs in the controlled comparison above, and every first boot-time run in two fresh boots). 0 of all runs with EnableDeferBacking=0.

Steps to Reproduce

There is no standalone reproducer yet (a bare torch.empty loop does not trigger it on 26.35, as already reported in #953). It reproduces with the real workload:

1. Two Arc Pro B70 visible to one process group (ZE_AFFINITY_MASK=0,1), kernel >= 6.18 (here 7.0.0-34), compute-runtime 26.35.39758.10.
2. Start vLLM (XPU, --tensor-parallel-size 2 --max-model-len 262144 --gpu-memory-utilization 0.94 --kv-cache-dtype fp8) with a ~27B model.
3. Watch free -m (used) while it loads and allocates the KV cache; compare with NEOReadDebugKeys=1 EnableDeferBacking=0.

I can try to build a smaller reproducer (torch + oneCCL, large allocation made after VRAM use > 50%, then a peer-visible export) if that helps.

Is this a regression?

Yes on the kernel side: the populate-on-export was added by 91494dee10 ("xe: populate buffers before exporting them", Dave Airlie, 2025-09-04, first in v6.18). A reporter in #953 bisected the same signature to the kernel (6.17 works, 7.0.0-29/-31 fail, userspace unchanged, trace xe_ttm_tt_populate <- xe_gem_prime_export). I have not tested 6.17 on this hardware.

API / system logs

Kernel populate stack and counters are above. dmesg shows OOM kills of the other GPU workload (and, once, a vLLM TP worker) when the mirrored state coincides with a second model loading, e.g.
Out of memory: Killed process ... (python3) ... anon-rss:4.4GB and Killed process ... (VLLM::Worker_TP).

Suggested fix directions

1. Kernel (patch to follow on dri-devel/intel-xe): xe_gem_prime_export() should not pre-populate BOs that can be placed in VRAM; the commit that added it states its scope as TTM_PL_SYSTEM objects ("VRAM doesn't use TT"), which is not true for BOs created with DEFER_BACKING (they start in system placement with a TT).
2. Runtime: avoid exporting a deferred-backing BO before its first validation, or do not create allocations that are going to be exported for peer access with DEFER_BACKING; or default EnableDeferBacking/DeferBackingMemoryPressurePercent accordingly for multi-device contexts.

Relation to existing issues

• #953: same kernel signature as the B60 reporter's TP report there, which Intel asked to be filed separately. 26.35 fixed the bare-allocation case but not this one.
• #980: shared multi-device context mirror, the underlying peer export.
• #974: IPC dma-buf fd leak, fix merged but unreleased. I have not tested master.

Command Line / Application Details

docker run ... -e ZE_AFFINITY_MASK=0,1 -e ONEAPI_DEVICE_SELECTOR=level_zero:gpu -e CCL_ATL_SHM=1 -e FI_PROVIDER=shm ...
  vllm serve <27B FP8 model> --tensor-parallel-size 2 --max-model-len 262144 --gpu-memory-utilization 0.94
  --kv-cache-dtype fp8 --max-num-seqs 4 --max-num-batched-tokens 8192 --enable-prefix-caching

Workaround that works for us: NEOReadDebugKeys=1 EnableDeferBacking=0.