[Bug]: AVC encode with nonzero LookAheadDepth passes NULL MemId to the external allocator

Start at _studio/mfx_lib/encode_hw/h264/src/mfx_h264_encode_hw.cpp in ImplementationAvc::AsyncRoutine, then read VideoCORE::GetFrameHDL, CommonCORE_VPL::GetFrameHDL, and the allocator wrapper cited in the report. Run the supplied FFmpeg command with nonzero lookahead to reproduce the crash. Done means the configuration no longer passes a null memory ID to the external allocator, either by completing encoding or rejecting unsupported parameters during initialization.

Which component impacted?

Encode

Is it regression? Good in old configuration?

No, this issue exist a long time

What happened?

I encountered a segfault using ffmpeg and lookahead on an Arc A310. I asked Opus to get to the bottom of it, it provided this analysis. I fact-checked it against Astra. Its analyses seem legit so I'm filing a bug report and a patch will follow. The answers make sense to me, and the PR coming up does fix the ffmpeg segfault, but I can't provide a real independent verification.

Note: The rest of this bug report has been written by Opus and fact-checked/edited by Astra.

Summary

With MFX_ENABLE_ENCTOOLS enabled, the AVC encoder calls GetNativeHandle() on an unpopulated surface when mfxExtCodingOption2::LookAheadDepth is nonzero:

#if defined(MFX_ENABLE_ENCTOOLS)
        if(extOpt2.LookAheadDepth)
            task->m_handleLpla.first = GetNativeHandle(*m_core,m_pLADataSurfaces);
#endif

m_pLADataSurfaces is declared as:

mfxFrameSurface1 m_pLADataSurfaces = {};

A repository-wide search finds only its declaration and the lookup above; nothing populates it. Consequently, both FrameInterface and Data.MemId remain null.

With CommonCORE_VPL and an application-supplied allocator, this lookup forwards a null memory ID to the application's mfxFrameAllocator::GetHDL callback. FFmpeg's QSV allocator dereferences that ID, causing a segmentation fault.

The lookup was introduced by d018e0db. It is present in intel-onevpl-26.2.4, intel-onevpl-26.3.5, and main at cab68d27b07df9efd574f0ac186047e2aeb1f33d.

Reproducer

On the environment below, this command initializes the encoder successfully but segfaults when processing the first frame:

ffmpeg -init_hw_device qsv=hw -filter_hw_device hw \
  -f lavfi -i testsrc2=s=1280x720:r=25:d=2 \
  -vf format=nv12,hwupload=extra_hw_frames=64 \
  -c:v h264_qsv -b:v 3000k -maxrate 4500k -bufsize 6000k \
  -look_ahead_depth 8 -f null -

FFmpeg's verbose log reports:

RateControlMethod: VBR
LookAheadDepth: 8
ExtBRC: OFF
LookAheadDS: 2x

Expected behavior: encoding completes, or unsupported parameters are rejected during initialization.

Actual behavior: the runtime passes mid == NULL to FFmpeg's allocator callback, which segfaults.

Observed variants

These results were observed on the reported environment; they are not claims about every platform or configuration.

Variant	Result
Nonzero lookahead depths tested within 1–40, across presets, with and without -extbrc 1	Segfault
-look_ahead_depth 0	Completes
Fixed QSV frame pool using VAAPI upload and hwmap=derive_device=qsv	Segfault with the same null ID
System-memory input, without hwupload	Completes
av1_qsv with -extbrc 1 -look_ahead_depth 40	Completes

Source trace

The failing path is:

ImplementationAvc::AsyncRoutine
  GetNativeHandle(*m_core, m_pLADataSurfaces)
    VideoCORE::GetFrameHDL(surface, handle)
      CommonCORE_VPL::GetFrameHDL(NULL, &handle.first, true)
        allocator wrapper GetHDL
          external allocator GetHDL(pthis, NULL, ...)
            FFmpeg qsv_frame_get_hdl

Specifically:

1. AsyncRoutine performs the lookup without checking whether the LA data surface exists.

2. VideoCORE::GetFrameHDL(surface, ...) uses surface.Data.MemId when FrameInterface is null.

3. CommonCORE_VPL::GetFrameHDL checks the output pointer but does not reject a null memory ID.

4. The allocator wrapper defaults an unknown memory ID to the external allocator. The null ID reaches its callback unchanged.

5. FFmpeg 8.1.2's qsv_frame_get_hdl dereferences the ID in both fixed-pool and dynamic-pool configurations.

This forwarding behavior is specific to the VPL core path. The older CommonCORE::GetFrameHDL explicitly rejects a null memory ID.

GetNativeHandle() asserts that the lookup succeeded, then returns null on failure in release builds. However, this cannot protect against a crash inside the allocator callback.

Backtrace

From the affected release build:

Thread 6 "ffmpeg" received signal SIGSEGV, Segmentation fault.
#0  libavcodec.so.62     +0x8c7f26   qsv_frame_get_hdl()
#1  libmfx-gen.so.1.2.17 +0x2f95af   MfxHwH264Encode::GetNativeHandle()
#2  libmfx-gen.so.1.2.17 +0x2d5699   ImplementationAvc::AsyncRoutine()
#3  libmfx-gen.so.1.2.17 +0x2d7301
#4  libmfx-gen.so.1.2.17 +0x15a17e   scheduler worker thread
#5  libstdc++.so.6
#6  libc.so.6

rdi (pthis) = 0x5555557cd848
rsi (mid)   = 0x0
rdx (hdl)   = 0x7fffeb980560

The faulting instruction in qsv_frame_get_hdl() is mov (%rsi),%rax, with rsi == 0.

Frames #1 and #2 were mapped to source using disassembly: the surf.Data.MemId load, the ExtendedSearch = true argument, the surface address at this + 0x178f0, and the result stored to task->m_handleLpla.first.

Debugger workaround

To isolate the failure, I placed a breakpoint at the offending call site (libmfx-gen +0x2d5694), skipped that call, and set its return value to zero.

With no other changes, encoding completed:

skipped GetNativeHandle(m_pLADataSurfaces) 50 times
[Inferior 1 exited normally]

ffprobe:
codec_name=h264
profile=High
width=1280
height=720
nb_read_frames=50

This supports the null lookup as the immediate cause of this reproducer's crash. It does not establish that bypassing the lookup implements all intended LA-data functionality.

Possible fix

An immediate defensive change would avoid looking up a handle when no LA data surface exists:

#if defined(MFX_ENABLE_ENCTOOLS)
        if (extOpt2.LookAheadDepth &&
            (m_pLADataSurfaces.Data.MemId ||
             m_pLADataSurfaces.FrameInterface))
        {
            task->m_handleLpla.first =
                GetNativeHandle(*m_core, m_pLADataSurfaces);
        }
#endif

This change is untested and addresses only the invalid lookup.

The intended ownership and initialization of the AVC LA data surface also need clarification:

• m_LADataFrameResponse appears only in its declaration and an Unlock() call; no allocation was found.
• The same introducing commit added a separate H264EncTools::m_laSurf and passes its address through m_EncToolCtrl.LADataSurfaces. It does not connect that surface to ImplementationAvc::m_pLADataSurfaces.
• HEVC's corresponding lookup checks whether Glob::LplaDataBuffer exists before accessing it. AVC has no equivalent existence check.

A platform check alone would not fix the uninitialized surface on platforms where that check passes. Rejecting null IDs in the VPL allocator path could provide additional protection, but would not resolve the missing AVC surface initialization.

Environment

• GPU: Intel Arc A310 (DG2), PCI 0000:01:00.0, i915 kernel driver
• Host: Ubuntu 24.04.4 LTS, kernel 6.8.0-142-generic
• Container: Debian 13 (trixie), Kubernetes with Intel GPU device plugin
• FFmpeg: jellyfin-ffmpeg8 8.1.2-4
• vpl-gpu-rt: intel-onevpl-26.2.4, libmfx-gen.so.1.2.17
  • MFX_ENABLE_KERNELS=ON
  • MFX_ENABLE_ENCTOOLS=ON
  • MFX_ENABLE_AENC=ON
• media-driver: intel-media-26.2.4, iHD revision 2aa8891
• gmmlib: intel-gmmlib-22.10.0
• libvpl dispatcher: v2.17.0, API 2.17
• libva: 2.24.1, VA-API 1.24

The source-level call chain and introducing commit were independently checked. The runtime observations above are from the reported environment; a second hardware reproduction has not been performed.

What's the usage scenario when you are seeing the problem?

Transcode for media delivery

What impacted?

H.264 lookahead rate control is unusable through FFmpeg's h264_qsv with GPU-memory input on runtimes from 25.3.2 onwards. Every -look_ahead_depth value crashes the process. Our workaround is to disable lookahead on the H.264 renditions.

Debug Information

ls /dev/dri (inside the container):

/dev/dri:
crw-rw---- 1 ingest fkupload 226,   0 card0
crw-rw---- 1 ingest fkupload 226, 128 renderD128
/dev/dri/by-path:
crw-rw---- 1 root video 226,   0 pci-0000:01:00.0-card
crw-rw---- 1 root   992 226, 128 pci-0000:01:00.0-render

vainfo (excerpt):

vainfo: VA-API version: 1.24 (libva 2.24.1)
vainfo: Driver version: Intel iHD driver for Intel(R) Gen Graphics - 26.2.4 (2aa8891)
      VAProfileH264Main               : VAEntrypointEncSliceLP
      VAProfileH264High               : VAEntrypointEncSliceLP
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSliceLP
      VAProfileHEVCMain               : VAEntrypointEncSliceLP
      VAProfileAV1Profile0            : VAEntrypointEncSliceLP
      ...

No libva trace is attached. The crash is a CPU-side NULL dereference in the allocator callback, before any VA call for the frame, and the GPU does not hang. Happy to provide a trace if it would still help.

Do you want to contribute a patch to fix the issue?

Yes, I'm glad to submit a patch to fix it