[macOS] Unconditional AV_CODEC_FLAG_LOW_DELAY (EnableLowLatencyRateControl) makes third-party VideoToolbox encoders unusable — falls back to libx264

Read src/video.cpp, especially make_avcodec_encode_session and the H.264 VideoToolbox encoder definition; compare where AV_CODEC_FLAG_LOW_DELAY is set with the fallback options that clear it. Reproduce the failure with a VideoToolbox encoder that rejects low latency rate control, then verify that hardware encoding succeeds when the flag is disabled or configurable. The issue also links #5814 for context.

Is there an existing issue for this?

• I have searched the existing issues

Is your issue described in the documentation?

• I have read the documentation

Is your issue present in the latest beta/pre-release?

Not verified as a binary. I ran the stable v2026.914.233613 build and checked the code path on master (src/video.cpp, make_avcodec_encode_session) — it is unchanged there, so a pre-release build should behave the same. Say the word if you want me to install the latest pre-release and retest; it is a disposable VM.

Describe the Bug

make_avcodec_encode_session sets AV_CODEC_FLAG_LOW_DELAY for every encoder, and on macOS FFmpeg's videotoolboxenc turns that into kVTVideoEncoderSpecification_EnableLowLatencyRateControl in the encoder specification. Apple's own VideoToolbox encoders accept that key, but a third-party VideoToolbox encoder that does not implement low-latency rate control is rejected outright: VTCompressionSessionCreate returns -12902 (kVTVideoEncoderNotFoundErr) and Sunshine falls back to libx264, even though the hardware encoder works fine without that one key.

Same root flag as #5814 (there it caps Apple Silicon at ~55 FPS at 4K; here it removes hardware encoding entirely), so if you add the vt_low_latency_rc knob proposed there, please consider it a fix for this too.

Environment where it happens

• macOS 26.7.1 (25G241), x86_64, QEMU/Proxmox VM with an RTX 5060 passed through
• Third-party VideoToolbox/Metal driver nullmoth/nvidia-macos-driver 2.5.2 (com.nvmtl.videoencoder.avc.nvenc, …hevc.nvenc); Metal 3 works, and the encoders encode real frames when LLRC is not requested
• Sunshine 2026.914.233613 (stable, macOS - dmg), capture = AVCaptureScreen, H.264, no config overrides beyond sw_preset/min_threads

What Sunshine logs

Info: Trying encoder [videotoolbox]
Info: Creating encoder [h264_videotoolbox]
Error: [h264_videotoolbox @ 0x…] Cannot create compression session: -12902
Error: [h264_videotoolbox @ 0x…] Try -allow_sw 1. The hardware encoder may be busy, or not supported.
Info: Retrying with fallback configuration options for [h264_videotoolbox] after error: Generic error in an external library
Error: [h264_videotoolbox @ 0x…] Error retrieving the supported property dictionary err=-12908
Encoder [videotoolbox] failed
Trying encoder [software] -> libx264

Evidence 1 — what Sunshine actually passes. Interposing VTCompressionSessionCreate inside the Sunshine process shows every attempt carries the same specification:

spec     = { EnableLowLatencyRateControl = 1; RequireHardwareAcceleratedVideoEncoder = 1; }
imgAttrs = (null)
-> status = -12902

Sunshine's fallback attempt (Retrying with fallback configuration options for [h264_videotoolbox]) fails the same way, and every create call captured in that process carried the key: the only place that clears it is the fallback options block {"flags"s, "-low_delay"}, while ctx->flags is reset and set to include AV_CODEC_FLAG_LOW_DELAY in make_avcodec_encode_session.

Evidence 2 — isolating the key. In a plain process where a Metal device has been created (so the third-party encoders are registered and appear in VTCopyVideoEncoderList), 1920x1080 H.264:

encoderSpecification	result
nil / {}	0 (session created)
EnableHardwareAcceleratedVideoEncoder	0
RequireHardwareAcceleratedVideoEncoder	0
EnableLowLatencyRateControl	-12902
RequireHardwareAcceleratedVideoEncoder + EnableLowLatencyRateControl (what Sunshine sends)	-12902
same + EncoderID = com.nvmtl.videoencoder.avc.nvenc	-12902

Removing only EnableLowLatencyRateControl makes the same create succeed and the encoder produces frames (VTCompressionSessionEncodeFrame callback status=0, one sample out).

Evidence 3 — the encoder never advertises the property. VTSessionCopySupportedPropertyDictionary on a session created without LLRC returns 65 keys including RealTime, ProfileLevel and MaxFrameDelayCount, but no EnableLowLatencyRateControl. So VideoToolbox filters that encoder out of the candidate list instead of ignoring the request.

Relevant source (master)

• src/video.cpp, make_avcodec_encode_session: ctx->flags |= AV_CODEC_FLAG_CLOSED_GOP | AV_CODEC_FLAG_LOW_DELAY;
• src/video.cpp, h264 VideoToolbox encoder definition: the only place that clears it is the fallback options block, {"flags"s, "-low_delay"}.

Expected Behavior

• A third-party VideoToolbox hardware encoder that does not support low-latency rate control should still be usable; the flag should not be applied unconditionally to *_videotoolbox.
• Or expose it, e.g. vt_low_latency_rc = enabled/disabled (as proposed in #5814), so users can turn it off for such encoders.

Additional Context

Minimal reproduction of the underlying VideoToolbox behaviour (needs any process that has created a Metal device first, so the third-party encoders are registered):

#import <Metal/Metal.h>
#import <VideoToolbox/VideoToolbox.h>

MTLCreateSystemDefaultDevice();   // registers com.nvmtl.videoencoder.*.nvenc

NSDictionary *bad  = @{ @"EnableLowLatencyRateControl": @YES,
                        @"RequireHardwareAcceleratedVideoEncoder": @YES };
NSDictionary *good = @{ @"RequireHardwareAcceleratedVideoEncoder": @YES };
VTCompressionSessionRef s = NULL;

VTCompressionSessionCreate(NULL, 1920, 1080, kCMVideoCodecType_H264,
                           (__bridge CFDictionaryRef)bad, NULL, NULL, NULL, NULL, &s);   // -12902
if (s) { CFRelease(s); s = NULL; }
VTCompressionSessionCreate(NULL, 1920, 1080, kCMVideoCodecType_H264,
                           (__bridge CFDictionaryRef)good, NULL, NULL, NULL, NULL, &s);  // 0

This host is a disposable VM, so I can test builds or gather any additional logs (min_log_level = 0, VT-level traces) you want. I did not attach a full log bundle because the above is the whole failure; say the word and I will paste one.

Host Operating System

macOS

Operating System Version

macOS 26.7.1 (25G241), x86_64, QEMU/Proxmox VE VM

Architecture

amd64/x86_64

Package

macOS - dmg

GPU Type

NVIDIA

GPU Model

NVIDIA GeForce RTX 5060 (GB206), passed through to the VM

GPU Driver/Mesa Version

nullmoth/nvidia-macos-driver 2.5.2 (third-party Metal + VideoToolbox plugin; kexts NVRM/NVAccel/NVRMFB/NVRMAGDC 2.5.2)

Capture Method

AVCaptureScreen (macOS)