Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Feature] turing h264 nvenc opens but sunshine fails with "engine already in use"

Open
  • Is something wrong?
#206 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
12/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
macos, rust
Domain
operating-systems

Research direction

The log shows the NVENC shim opening a second encoder session in the same process while the first is still open, and the engine refusing it with "engine already in use". Start by finding where the session is opened and closed in the nvenc-shim and nve264 code, and why the first session's teardown does not release the engine first. Done means a second StartSession succeeds, or the reuse path is explained, and H.264 encoding runs without falling back to x264. The reporter's Blackwell-check and T2D-layout patches are out-of-tree, so reproduce on a clean build first.

Written by the indexing model from the issue text.

Description

enhancement
What should it do, and why does it matter to you?

2070 super here. i got h264 nvenc to start on turing by patching out the blackwell check and using the t2d layout. the encoder actually opens (0xc4b7) but sunshine opens a second session before closing the first one and the lib refuses it with "engine already in use", so sunshine falls back to x264. log below

nvenc-shim: metal device created
nvenc-vt: CreateInstance 'avc1' pid 6856
nve264: classes: encoder 0xc4b7 channel 0xc46f
nve264: encoder engine NVENC0 (type 0x1b)
nve264: open: 1920x1080 pitch 1920 rows 1088, encoder class 0xc4b7
nvenc-vt: StartSession 1920x1080 pid 6856 -> 0
nvenc-vt: CreateInstance 'avc1' pid 6856
nve264: open: engine already in use in this process — refused
nvenc-vt: StartSession 1920x1080: engine refused
nvenc-vt: close 1920x1080: 0 frames emitted (0 IDR), 0 errors, 0.00 ms GPU/frame
nve264: teardown: 16 CPU mappings removed before any free
nvenc-vt: CreateInstance 'avc1' pid 6856
nve264: classes: encoder 0xc4b7 channel 0xc46f
nve264: encoder engine NVENC0 (type 0x1b)
nve264: open: 1920x1080 pitch 1920 rows 1088, encoder class 0xc4b7
nvenc-vt: StartSession 1920x1080 pid 6856 -> 0
nvenc-vt: CreateInstance 'avc1' pid 6856
nve264: open: engine already in use in this process — refused
nvenc-vt: StartSession 1920x1080: engine refused
nvenc-vt: close 1920x1080: 0 frames emitted (0 IDR), 0 errors, 0.00 ms GPU/frame
nve264: teardown: 16 CPU mappings removed before any free
nvenc-vt: CreateInstance 'avc1' pid 6856
nve264: classes: encoder 0xc4b7 channel 0xc46f
nve264: encoder engine NVENC0 (type 0x1b)
nve264: open: 1920x1080 pitch 1920 rows 1088, encoder class 0xc4b7
nvenc-vt: StartSession 1920x1080 pid 6856 -> 0
nvenc-vt: CreateInstance 'avc1' pid 6856
nve264: open: engine already in use in this process — refused
nvenc-vt: StartSession 1920x1080: engine refused
nvenc-vt: close 1920x1080: 0 frames emitted (0 IDR), 0 errors, 0.00 ms GPU/frame
nve264: teardown: 16 CPU mappings removed before any free
user@iMac-Pro ~ %```
Dominant language
Rust
Stars
2.5k
Forks
233
Avg merge
1h 37m
Merged PRs (30d)
1

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nullmoth/nvidia-macos-driver

All issues in nullmoth/nvidia-macos-driver

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.