Hacktoberfest 2026: the issues maintainers tagged for October, open and beginner-friendly. Browse Hacktoberfest issues

[Bug] RTX 5070 Ti: NVENC never opens — the encoder libraries hard-code NVENC0, which this card does not expose

Closed
  • Is something wrong?
#105 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
macos, rust
Domain
backend

Research direction

The report identifies libnvenc_h264.dylib and libnvenc_hevc.dylib as the failing libraries, but says their source is not in the repository. Start by checking how the project packages or builds these libraries and whether the engine list from RM is available to them. Done means encoder engine selection uses an engine the card reports and H.264 and HEVC hardware encoding succeeds.

Written by the indexing model from the issue text.

Description

bug
Send logs report ID

NM-DWRAR0TG, NM-3EQ0R0J4, NM-811KTXJ4, NM-A4HYNZ82, NM-SJDAJN86, NM-ZB48T7XT, NM-1T3RCX22

Where does it go wrong?

An app or game

What happens

On an RTX 5070 Ti, hardware H.264/HEVC encoding never starts: VideoToolbox returns -12908 and apps fall back to software. libnvenc_h264.dylib / libnvenc_hevc.dylib always ask for NVENC0, but RM on this card only exposes NVENC1 and NVENC2. Pointing the libraries at NVENC1 (3 bytes per library) makes hardware encoding work.

Environment: RTX 5070 Ti 16 GB (GB203, 10DE:2C05), macOS 15.8.1 (24H32). Driver 1.7.0; the two encoder libraries are byte-identical in 1.10.0 (libnvenc_h264 sha256 164f6dfa…, libnvenc_hevc 779ff79b…), so 1.10.0 is affected too.

Engine list RM returns to the encoder library (NV2080_ENGINE_TYPE_*) — 0x1c / 0x1d present, 0x1b (NVENC0) missing:

engine-query count=11 list: 1 9 d e 13 1c 1d 22 26 2c 33

The library still allocates its channel on NVENC0:

RMTRACE alloc ... class=ca6f ... status=57 (NV_ERR_OBJECT_NOT_FOUND)
nve264: open: engine bring-up failed for 1280x720
nvenc-vt: StartSession 1280x720: engine refused
codec=h264 encoder=com.nvmtl.videoencoder.avc.nvenc create_status=-12908

HEVC fails the same way. The validated RTX 5060 presumably has NVENC0, which would explain why this was not seen before.

Local fix (binary patch, for reference) — three bytes per library select NVENC1:

library offset old → new field
libnvenc_h264 0x3bff / 0x3cd5 0x1b → 0x1c channel / bind engineType
libnvenc_h264 0x3efd 0 → 1 encoder engineInstance
libnvenc_hevc 0x3d0f / 0x3de5 0x1b → 0x1c channel / bind engineType
libnvenc_hevc 0x400d 0 → 1 encoder engineInstance

With RequireHardwareAcceleratedVideoEncoder: H.264 and HEVC, 1080p and 4K 8-bit, 8 sessions, all hardware=true with the NullMoth encoder ID, every frame emitted, output decodes cleanly with Apple's software decoder.

Suggestion: hard-coding NVENC1 only fits this card. A general fix would pick the first NVENC engine RM actually reports (0x1b/0x1c/0x1d) and use the matching engine instance. The encoder libraries have no source in the repo, so this needs to be done on your side.

App or game (if it is one)

Any app that uses VideoToolbox hardware encoding (tested: ffmpeg -c:v h264_videotoolbox / hevc_videotoolbox, VideoProc Converter)

Did you change anything yourself?

Yes, so the logs will not match a stock install:

  • libnvenc_h264/hevc.dylib patched as in the table above (3 bytes each, re-signed ad hoc).
  • NVRMFB.kext built from 1.7.0 with PR #64 and #40 applied, plus a local patch that offers 4K modes.
    The failure above was captured on the unmodified 1.7.0 encoder libraries.
Dominant language
Rust
Stars
2.5k
Forks
233
Avg merge
21h 20m
Merged PRs (30d)
3

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nullmoth/nvidia-macos-driver

All issues in nullmoth/nvidia-macos-driver

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.