Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

VAAPI "auto" quality sends compression_level 0 to the driver; Intel HD 620 rejects it for HEVC and every stream aborts

Open
  • Is something wrong?
#5,792 2 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
cpp, linux
Domain
audio-video-rtc

Research direction

Start in src/platform/linux/vaapi.cpp around lines 317-334 and trace how vaapi_quality maps to compression_level. Run the provided FFmpeg VAAPI reproduction on the affected Intel setup, then verify that auto leaves the quality unset and that the documented quality choices no longer cause the HEVC stream to abort.

Written by the indexing model from the issue text.

Description

Is there an existing issue for this?
  • I have searched the existing issues
Is your issue described in the documentation?
  • I have read the documentation
Is your issue present in the latest beta/pre-release?

This issue is present in the latest pre-release

Describe the Bug

Since #5388, src/platform/linux/vaapi.cpp sets ctx->compression_level = target_quality whenever the driver reports a quality range, and with vaapi_quality = auto (the default) target_quality is 0 (lines 317 to 334 at v2026.921.13701). libavcodec's vaapi_encode.c treats any compression_level >= 0 as set and passes it as VAEncMiscParameterBufferQualityLevel, so the driver receives quality level 0.

On my Intel HD Graphics 620 (Kaby Lake) with intel-media-driver 26.2.4, the driver accepts HEVC quality levels 1, 4 and 7 only. Level 0 fails at vaEndPicture with Failed to end picture encode issue: 24 (internal encoding error). Sunshine then aborts in the session::video thread (SIGABRT, stack below), systemd restarts it, Moonlight reports the stream ending, and the next connection does the same. Five core dumps in two minutes this morning. The encoder validation at startup passes and logs [VAAPI] Quality level set to 0 (fastest level: 7), so nothing looks wrong until a client connects.

The previous build I had (2026.516.143833, before #5388) did not set a quality level and streamed fine on the same machine, same driver.

Reproduction outside Sunshine, same GPU and driver, system FFmpeg 9.0.2:

ffmpeg -init_hw_device vaapi=va:/dev/dri/renderD128 -filter_hw_device va \
  -f lavfi -i testsrc2=size=1920x1080:rate=30 -t 1 -vf format=nv12,hwupload \
  -c:v hevc_vaapi -compression_level 0 -b:v 23M -maxrate 23M -bufsize 766k -f null -

Results by -compression_level value:

hevc_vaapi  0        fails: Failed to end picture encode issue: 24
hevc_vaapi  1, 4, 7  ok
hevc_vaapi  2, 3, 5, 6  fail (same error)
hevc_vaapi  unset    ok (libavcodec: "No quality level set; using default")
h264_vaapi  0 to 7   ok

Workaround that fixed it for me: vaapi_quality = speed (level 7). quality (level 1) also works. balanced computes 7 / 2 = 3, which this driver rejects as well, so two of the four documented choices crash on this hardware.

Expected Behavior

vaapi_quality = auto should leave ctx->compression_level at its libavcodec default (-1) so the driver picks its own default quality, which is what 2026.516 did. Something like: only assign compression_level in the speed, balanced and quality cases. For balanced, (1 + quality_attr.value) / 2 gives 4 on Intel, which is the driver's middle preset, instead of 3.

Additional Context

Not the same as #5471: that one is a null dereference (segfault at 28) on 2026.516, before any frame is encoded. This one is an abort after the driver rejects the first frame, and only exists since #5388.

I ran the CachyOS package of v2026.921.13701 (log reports commit 019b1ba661d8). I did not run the v2026.922.203725 binary; I checked that vaapi.cpp is unchanged there and on master apart from a designated initializer.

Moonlight client was the iOS app over Tailscale, requesting HEVC 8-bit at 30 fps. sunshine.conf was empty when it crashed.

Host Operating System

Linux

Operating System Version

CachyOS (Arch based), kernel 7.2.5-cachyos, niri 26.04 (Wayland), greetd

Architecture

amd64/x86_64

Package

other (not listed)

CachyOS repository package sunshine 2026.921.13701-1

GPU Type

Intel

GPU Model

Intel HD Graphics 620 (Kaby Lake-U GT2), Core i5-7200U

GPU Driver/Mesa Version

intel-media-driver (iHD) 26.2.4, libva 2.24.1, Mesa 26.2.3

Capture Method

wlroots (FreeBSD/Linux)

Config
# empty at the time of the crash; workaround now in place:
vaapi_quality = speed
Apps

Default Desktop entry, no commands.

Log output
[2026-09-24 09:15:35.175]: Info: CLIENT CONNECTED
[2026-09-24 09:15:35.397]: Info: Screencasting with Wayland's protocol
[2026-09-24 09:15:35.501]: Info: Creating encoder [hevc_vaapi]
[2026-09-24 09:15:35.501]: Info: Color depth: 8-bit
[2026-09-24 09:15:35.530]: Info: vaapi vendor: Intel iHD driver for Intel(R) Gen Graphics - 26.2.4 ()
[2026-09-24 09:15:35.530]: Info: Streaming bitrate is 22988000
[2026-09-24 09:15:35.530]: Info: Using normal encoding mode
[2026-09-24 09:15:35.530]: Info: [VAAPI] Quality level set to 0 (fastest level: 7)
[2026-09-24 09:15:35.530]: Info: [VAAPI] Using VBR rate control with single frame VBV size (whitelist override)
[2026-09-24 09:15:35.530]: Warning: [hevc_vaapi @ 0x7ffbb406ec40] Driver does not advertise encoder features, using guessed defaults.
[2026-09-24 09:15:35.530]: Warning: [hevc_vaapi @ 0x7ffbb406ec40] Driver does not advertise encoder block size, using guessed defaults.
[2026-09-24 09:15:35.607]: Info: Minimum FPS target set to ~15fps (66.6667ms)
[2026-09-24 09:15:35.645]: Error: [hevc_vaapi @ 0x7ffbb406ec40] Failed to end picture encode issue: 24 (internal encoding error).

systemd / coredumpctl:

systemd-coredump: Process 2462624 (sunshine) of user 1000 dumped core.
app-dev.lizardbyte.app.Sunshine.service: Main process exited, code=dumped, status=6/ABRT
Stack trace of thread 2541429 (session::video):
#0  pthread_kill (libc.so.6 + 0xaeceb)
#1  raise (libc.so.6 + 0x44d48)
#2  abort (libc.so.6 + 0x257c1)
#3  n/a (sunshine + 0x478cc4)
#4  n/a (sunshine + 0x4791a3)
#5  n/a (sunshine + 0x3fec61)
#6  n/a (sunshine + 0x3fee97)
#7  n/a (sunshine + 0x2629d0)
#8  n/a (sunshine + 0x25469c)

After vaapi_quality = speed and a restart: [VAAPI] Quality level set to 7 (fastest level: 7), stream and input work.

Online logs

No response

Dominant language
C++
Stars
41.4k
Forks
2.1k
Avg merge
19h 31m
Merged PRs (30d)
137

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from LizardByte/Sunshine

All issues in LizardByte/Sunshine

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.