[GSD-14047] [Linux][BMG/B70] zeMemAllocDevice accepts device allocations far beyond VRAM

The bug is in how the Level Zero driver's zeMemAllocDevice path accounts for total device memory, and the reproducer is ze_overcommit_repro.cpp, built against -lze_loader. Start by running it with NEOReadDebugKeys=1 PrintDebugMessages=1 to see the DeferBackingPressure lines, then find where the deferred-backing decision is made. Done would mean the agreed behavior (error return or intended overcommit) is settled by maintainers first, then implemented and verified against the reproducer.

Pre-submission Checklist

• I am using the latest GPU driver version (releases)
• I have searched for similar issues and found none

GPU Hardware

Intel Arc Pro B70 (BMG G31, 8086:e223), 2 cards, 32 GB each. Host: Intel Core Ultra 7 270K Plus, ASUS ROG MAXIMUS Z890 HERO (BIOS 3305), 30 GB system RAM.

DRI Devices Information

$ ls -ls /dev/dri/*
0 crw-rw---- 1 root video  226,   0 Oct 27 20:11 /dev/dri/card0
0 crw-rw---- 1 root video  226,   1 Oct 27 20:11 /dev/dri/card1
0 crw-rw---- 1 root video  226,   2 Oct 27 20:11 /dev/dri/card2
0 crw-rw---- 1 root render 226, 128 Oct 27 20:11 /dev/dri/renderD128
0 crw-rw---- 1 root render 226, 129 Oct 27 20:11 /dev/dri/renderD129
0 crw-rw---- 1 root render 226, 130 Oct 27 20:11 /dev/dri/renderD130

/dev/dri/by-path:
total 0
0 lrwxrwxrwx 1 root root  8 Oct 27 20:11 pci-0000:00:02.0-card -> ../card1
0 lrwxrwxrwx 1 root root 13 Oct 27 20:11 pci-0000:00:02.0-render -> ../renderD128
0 lrwxrwxrwx 1 root root  8 Oct 27 20:11 pci-0000:04:00.0-card -> ../card0
0 lrwxrwxrwx 1 root root  8 Oct 27 20:11 pci-0000:04:00.0-platform-simple-framebuffer.0-card -> ../card0
0 lrwxrwxrwx 1 root root 13 Oct 27 20:11 pci-0000:04:00.0-render -> ../renderD129
0 lrwxrwxrwx 1 root root  8 Oct 27 20:11 pci-0000:88:00.0-card -> ../card2
0 lrwxrwxrwx 1 root root 13 Oct 27 20:11 pci-0000:88:00.0-render -> ../renderD130

$ ls -la /dev/dri/by-path/
total 0
drwxr-xr-x 2 root root 180 Oct 27 20:11 .
drwxr-xr-x 3 root root 180 Oct 27 20:11 ..
lrwxrwxrwx 1 root root   8 Oct 27 20:11 pci-0000:00:02.0-card -> ../card1
lrwxrwxrwx 1 root root  13 Oct 27 20:11 pci-0000:00:02.0-render -> ../renderD128
lrwxrwxrwx 1 root root   8 Oct 27 20:11 pci-0000:04:00.0-card -> ../card0
lrwxrwxrwx 1 root root   8 Oct 27 20:11 pci-0000:04:00.0-platform-simple-framebuffer.0-card -> ../card0
lrwxrwxrwx 1 root root  13 Oct 27 20:11 pci-0000:04:00.0-render -> ../renderD129
lrwxrwxrwx 1 root root   8 Oct 27 20:11 pci-0000:88:00.0-card -> ../card2
lrwxrwxrwx 1 root root  13 Oct 27 20:11 pci-0000:88:00.0-render -> ../renderD130

0000:04:00.0 and 0000:88:00.0 are the two B70s (xe). 0000:00:02.0 is the CPU iGPU (i915). All tests below use 0000:04:00.0.

GPU Detailed Information (lspci output)

$ sudo lspci -vvv -k -s 0000:04:00.0
04:00.0 VGA compatible controller: Intel Corporation Battlemage G31 [Intel Graphics] (prog-if 00 [VGA controller])
	Subsystem: Intel Corporation Device 1701
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin ? routed to IRQ 211
	IOMMU group: 18
	Region 0: Memory at d000000000 (64-bit, prefetchable) [size=16M]
	Region 2: Memory at c000000000 (64-bit, prefetchable) [size=32G]
	Expansion ROM at 8d000000 [disabled] [size=2M]
	Capabilities: [40] Vendor Specific Information: Intel Capabilities v1
		CapA: Peg60Dis- Peg12Dis- Peg11Dis- Peg10Dis- PeLWUDis- DmiWidth=x4
		      EccDis- ForceEccEn- VTdDis- DmiG2Dis- PegG2Dis- DDRMaxSize=Unlimited
		      1NDis- CDDis- DDPCDis- X2APICEn- PDCDis- IGDis- CDID=0 CRID=0
		      DDROCCAP+ OCEn- DDRWrtVrefEn+ DDR3LEn+
		CapB: ImguDis- OCbySSKUCap- OCbySSKUEn- SMTCap- CacheSzCap 0x0
		      SoftBinCap- DDR3MaxFreqWithRef100=Disabled PegG3Dis-
		      PkgTyp- AddGfxEn- AddGfxCap- PegX16Dis- DmiG3Dis- GmmDis-
		      DDR3MaxFreq=2932MHz LPDDR3En-
	Capabilities: [70] Express (v2) Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
		LnkSta:	Speed 2.5GT/s, Width x1
			TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range B, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix-
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS- TPHComp- ExtTPHComp-
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: ReqEn-
			 IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
			 10BitTagReq+ OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5GT/s, Crosslink- Retimer- 2Retimers- DRS-
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
	Capabilities: [ac] MSI: Enable+ Count=1/1 Maskable+ 64bit+
		Address: 00000000fee00d98  Data: 0000
		Masking: 00000000  Pending: 00000000
	Capabilities: [d0] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [100 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 0
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [110 v1] Null
	Capabilities: [200 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [420 v1] Physical Resizable BAR
		BAR 2: current size: 32GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	Capabilities: [220 v1] Virtual Resizable BAR
		BAR 2: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB 32GB
	Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration- 10BitTagReq+ IntMsgNum 0
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy+ 10BitTagReq-
		IOVSta:	Migration-
		Initial VFs: 4, Total VFs: 4, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 1, stride: 1, Device ID: e223
		Supported Page Size: 00000553, System Page Size: 00000001
		Region 0: Memory at 000000d001000000 (64-bit, prefetchable)
		Region 2: Memory at 000000c800000000 (64-bit, prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [400 v1] Latency Tolerance Reporting
		Max snoop latency: 15728640ns
		Max no snoop latency: 15728640ns
	Kernel driver in use: xe
	Kernel modules: xe

Driver Version

26.35.39758.17 (Ubuntu PPA build of the 26.35.39758 release). Also reproduced on 26.31.39395.16.

Installed GPU Driver Packages

$ dpkg -l | grep -iE "igc|gmm|opencl|level-zero|level_zero|ocloc|libze"
ii clinfo                    3.0.25.02.14-1build1
ii cuda-opencl-13-4          13.4.92-1
ii cuda-opencl-dev-13-4      13.4.92-1
ii intel-opencl-icd          26.35.39758.17-1~26.04~ppa1
ii libclc-22-dev             1:22.1.2-1ubuntu1
ii libigc2                   2.41.15-1~26.04
ii libigdfcl2                2.41.15-1~26.04
ii libigdgmm12:amd64         22.10.2-1~26.04~ppa1
ii libze-intel-gpu1          26.35.39758.17-1~26.04~ppa1
ii libze1:amd64              1.32.0-1~26.04~ppa1
ii ocl-icd-libopencl1:amd64  2.3.4-1

Driver Installation Details

Distribution: Ubuntu 26.04.1 LTS

Packages from the Intel graphics PPA (intel-graphics), upgraded with apt from 26.31.39395.16 / IGC 2.40.16 to 26.35.39758.17 / IGC 2.41.15. oneAPI 2026.1 for the SYCL build of llama.cpp.

Linux Distribution

Other (please specify below)

Other Linux Distribution

Ubuntu 26.04.1 LTS

Kernel Version & Boot Parameters

$ uname -r
7.0.0-38-generic

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-7.0.0-38-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro crashkernel=2G-4G:320M,4G-32G:512M,32G-64G:1024M,64G-128G:2048M,128G-:4096M

$ lsmod | grep -E 'i915|xe'
xe                   4452352  5
drm_gpusvm_helper      57344  1 xe
i915                 5128192  1
gpu_sched              69632  1 xe
drm_gpuvm              57344  1 xe
drm_ttm_helper         20480  1 xe
drm_buddy              28672  2 xe,i915
drm_exec               12288  2 drm_gpuvm,xe
ttm                   139264  3 drm_ttm_helper,xe,i915
drm_suballoc_helper    24576  1 xe
drm_display_helper    307200  2 xe,i915
cec                   106496  3 drm_display_helper,xe,i915
intel_vsec             24576  3 intel_pmc_ssram_telemetry,pmt_telemetry,xe
i2c_algo_bit           16384  2 xe,i915
video                  77824  3 asus_wmi,xe,i915

Actual Behavior

This problem was first found using llama.cpp's SYCL backend (model buffers are allocated with zeMemAllocDevice) when trying to load a model only into VRAM bigger than available mem, occasionally causing kernel panics. For example, loading a 45 GB model onto one 32 GB B70 succeeds at allocation time. When the weights are uploaded, the GPU page faults.

On 26.35 the same llama.cpp load drained more than half of the host's 30.8 GB of RAM in under 2 seconds once the weights were uploaded; I stopped it before the host ran out of memory.

zeMemAllocDevice only checks the size of each allocation, not the total; past 50% of VRAM the driver uses deferred backing (no VRAM reserved until first use). One 36 GiB allocation is refused (ZE_RESULT_ERROR_UNSUPPORTED_SIZE), but 36 × 1 GiB all return ZE_RESULT_SUCCESS on a 30.30 GiB device. Sysman free memory (and sycl::ext::intel::info::device::free_memory) doesn't count the deferred allocations either: it only drops from 31.66 to 16.66 GiB, so an application can't see the overcommit.

With NEOReadDebugKeys=1 PrintDebugMessages=1 the driver shows that it knows it is past capacity and still accepts the allocation with deferred backing:

[DeferBackingPressure] used=15546MB + req=1024MB -> proj=16570MB / max=32656MB (50%) threshold=50% -> deferBacking=ON
[DeferBackingPressure] used=36026MB + req=1024MB -> proj=37050MB / max=32656MB (113%) threshold=50% -> deferBacking=ON

Turning deferred backing off with NEOReadDebugKeys=1 EnableDeferBacking=0 (immediate backing only) does not fix it:

• Reproducer: all 36 x 1 GiB still return ZE_RESULT_SUCCESS. VRAM is really filled (sysman free 0.84 GiB) and the overflow goes to system RAM at allocation time (MemAvailable 28.6 GB -> 23.4 GB).
• llama.cpp 120B on one GPU: MemAvailable 28.6 GB -> 11.7 GB during the model load, the same as with deferred backing.

Expected Behavior

From using llama.cpp, the load using Vulkan (Mesa ANV) on the same machine fails cleanly with unable to allocate Vulkan0 buffer after 18 s, with no kernel errors.

For SYCL, once the total of device allocations exceeds the device's memory, zeMemAllocDevice returns an error (e.g. ZE_RESULT_ERROR_OUT_OF_DEVICE_MEMORY), so applications can fail cleanly the way they do on Vulkan.

I'm not sure if overcommit is intended to work differently. Either way, if someone forces all layers onto the GPU with llama.cpp or any other application, there should be a way to signal OOM to the application.

Reproduction Rate

Always reproduces - 100%

Steps to Reproduce

1. Minimal reproducer (Code is in Source Code / Reproducer section):

g++ -O2 -o ze_overcommit_repro ze_overcommit_repro.cpp -lze_loader
./ze_overcommit_repro 36 1

Output on 26.35.39758.17:

Intel(R) Arc(TM) Pro B70 Graphics: totalSize 30.30 GiB, maxMemAllocSize 30.30 GiB, sysman free 31.66 GiB
allocated 36.0 GiB in 36 allocations (device has 30.30 GiB), all returned ZE_RESULT_SUCCESS
sysman free after allocating: 16.66 GiB

2. Reproduction with llama.cpp

llama.cpp SYCL build (upstream master d05706b4f or release b11827, oneAPI 2026.1), model unsloth/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-GGUF UD-IQ2_XXS (45 GB) on one 32 GB card:

llama-server -m NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-UD-IQ2_XXS-00001-of-00003.gguf \
  --device SYCL0 -ngl 999 --split-mode none -c 4096

common_param:   - SYCL0   : Intel(R) Arc(TM) Pro B70 Graphics (32656 MiB, 32571 MiB free)
common_fit_params: failed to fit params to free device memory: n_gpu_layers already set by user to 999, abort
load_tensors:        SYCL0 model buffer size = 45469.29 MiB

Is this a regression?

• Yes, this is a regression - functionality that previously worked is now broken

Last Known Working Driver Version

No response

First Known Failing Driver Version

26.31.39395.16 is the oldest version I tested. Still failing on 26.35.39758.17.

API Call Logs

No response

strace Logs

No response

System Logs / dmesg Output

From the previous driver (26.31.39395.16), where the same llama.cpp load was not stopped.

xe 0000:04:00.0: [drm] Tile0: GT0: Engine reset: engine_class=bcs, logical_mask: 0x1, guc_id=6, state=0x289
kworker/u97:9: page allocation failure: order:0, mode:0x104dc2(GFP_HIGHUSER|__GFP_ZERO|__GFP_RETRY_MAYFAIL), nodemask=(null),cpuset=/,mems_allowed=0
CPU: 0 UID: 0 PID: 8601 Comm: kworker/u97:9 Not tainted 7.0.0-38-generic #38-Ubuntu PREEMPT(lazy)
Hardware name: ASUS System Product Name/ROG MAXIMUS Z890 HERO, BIOS 3305 07/27/2026
Workqueue: xe_page_fault_work_queue xe_pagefault_queue_work [xe]
Call Trace:
 ttm_pool_alloc_page+0x7d/0x250 [ttm]
 __ttm_pool_alloc+0x1e0/0x470 [ttm]
 ttm_pool_alloc+0x75/0xb0 [ttm]
 xe_ttm_tt_populate+0x54/0xc0 [xe]
 ttm_tt_populate+0xa3/0x150 [ttm]
 ttm_bo_populate+0x38/0xb0 [ttm]
 ttm_bo_handle_move_mem+0x1a5/0x1e0 [ttm]
 ttm_bo_bounce_temp_buffer+0xa4/0xd0 [ttm]
 ttm_bo_evict+0xfd/0x180 [ttm]
 ttm_bo_evict_cb+0x94/0x110 [ttm]
 ttm_lru_walk_for_evict+0xc0/0x230 [ttm]
 ttm_bo_alloc_resource+0x1b9/0x5b0 [ttm]
 ttm_bo_validate+0x9a/0x190 [ttm]
 xe_bo_validate+0x95/0x140 [xe]
 xe_pagefault_handle_vma+0x302/0x410 [xe]
 xe_pagefault_queue_work+0x2ca/0x360 [xe]
 process_one_work+0x1ac/0x3d0
 worker_thread+0x1b8/0x360
 kthread+0xf7/0x130
 ret_from_fork+0x195/0x2a0
 ret_from_fork_asm+0x1a/0x30
[TTM] Buffer eviction failed

Backtrace (if crash or hang occurred)

See the kernel call trace above. No userspace crash; the host runs out of memory.

Source Code / Reproducer

ze_overcommit_repro.cpp (I wrote this using AI as a quick test):

// zeMemAllocDevice accepts device allocations whose sum exceeds device memory (BMG, xe KMD).
// Allocates <total> GiB in <chunk> GiB pieces on device 0, never touches them, then frees.
// Build: g++ -O2 -o ze_overcommit_repro ze_overcommit_repro.cpp -lze_loader
// Run:   ./ze_overcommit_repro 36 1
#include <level_zero/ze_api.h>
#include <level_zero/zes_api.h>
#include <cstdio>
#include <cstdlib>
#include <vector>

static double gib(uint64_t b) { return b / 1073741824.0; }

int main(int argc, char ** argv) {
    const double total = argc > 1 ? atof(argv[1]) : 36.0;
    const double chunk = argc > 2 ? atof(argv[2]) : 1.0;

    if (zesInit(0) != ZE_RESULT_SUCCESS || zeInit(ZE_INIT_FLAG_GPU_ONLY) != ZE_RESULT_SUCCESS) { puts("init failed"); return 1; }
    uint32_t n = 1; ze_driver_handle_t drv; zeDriverGet(&n, &drv);
    n = 1; ze_device_handle_t dev; zeDeviceGet(drv, &n, &dev);

    ze_device_properties_t dp = {ZE_STRUCTURE_TYPE_DEVICE_PROPERTIES};
    zeDeviceGetProperties(dev, &dp);
    uint32_t nm = 1; ze_device_memory_properties_t mp = {ZE_STRUCTURE_TYPE_DEVICE_MEMORY_PROPERTIES};
    zeDeviceGetMemoryProperties(dev, &nm, &mp);

    // free memory as reported by sysman
    auto sysman_free = [&]() -> double {
        uint32_t c = 0; zesDeviceEnumMemoryModules((zes_device_handle_t) dev, &c, nullptr);
        std::vector<zes_mem_handle_t> h(c); zesDeviceEnumMemoryModules((zes_device_handle_t) dev, &c, h.data());
        uint64_t f = 0;
        for (auto m : h) { zes_mem_state_t s = {ZES_STRUCTURE_TYPE_MEM_STATE}; if (zesMemoryGetState(m, &s) == ZE_RESULT_SUCCESS) f += s.free; }
        return gib(f);
    };

    printf("%s: totalSize %.2f GiB, maxMemAllocSize %.2f GiB, sysman free %.2f GiB\n",
           dp.name, gib(mp.totalSize), gib(dp.maxMemAllocSize), sysman_free());

    ze_context_desc_t cd = {ZE_STRUCTURE_TYPE_CONTEXT_DESC};
    ze_context_handle_t ctx; zeContextCreate(drv, &cd, &ctx);
    ze_device_mem_alloc_desc_t ad = {ZE_STRUCTURE_TYPE_DEVICE_MEM_ALLOC_DESC};
    std::vector<void *> ptrs;
    double got = 0;
    while (got + chunk <= total + 1e-9) {
        void * p = nullptr;
        ze_result_t r = zeMemAllocDevice(ctx, &ad, (size_t) (chunk * 1073741824.0), 64, dev, &p);
        if (r != ZE_RESULT_SUCCESS) { printf("zeMemAllocDevice failed after %.1f GiB: 0x%x\n", got, r); break; }
        ptrs.push_back(p);
        got += chunk;
    }
    printf("allocated %.1f GiB in %zu allocations (device has %.2f GiB)%s\n",
           got, ptrs.size(), gib(mp.totalSize),
           got + 1e-9 >= total ? ", all returned ZE_RESULT_SUCCESS" : "");
    printf("sysman free after allocating: %.2f GiB\n", sysman_free());

    for (void * p : ptrs) zeMemFree(ctx, p);
    zeContextDestroy(ctx);
    return 0;
}

Command Line / Application Details

llama.cpp SYCL backend, llama-server ... --device SYCL0 -ngl 999 --split-mode none -c 4096 (full command in Steps to Reproduce).

oneAPI Version (if applicable)

oneAPI DPC++/C++ Compiler 2026.1 (llama.cpp SYCL build).

Screenshots / Video

No response

Additional Notes

• My current open PR for a llama.cpp workaround (refuse SYCL buffers that would exceed VRAM): https://github.com/ggml-org/llama.cpp/pull/31285