[Feature] Register NVMTL_NVENCVT decoders in VTDecoderXPCService (green H.264 in Safari on Ada and Blackwell)

Open Beginner friendly
  • Is something wrong?
#519 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 2 days

Nobody has claimed this yet.

Assessment

Difficulty
2/5
Estimated time
1-3 hours
Newbie friendliness
78/100
Issue type
Feature
Clarity
Clearly specified
Activity status
Active
Tech stack
macos, objective-c
Domain
backend

Research direction

Start in plugin/NVMTLDevice.m at nvmtl_nvdec_in_decode_service() around line 471 and read the dispatch block alongside the existing decoder registrations. Add the gated nvmtl_nvencvt_register() call there, then check the build or existing tests for that target. Done when the NVENCVT decoder is registered in VTDecoderXPCService under the existing whitelist and opt-out users remain unaffected.

Written by the indexing model from the issue text.

Description

What should it do, and why does it matter to you?

Register the opt-in NVMTL_NVENCVT codec library in VTDecoderXPCService too.

Concretely: add a nvmtl_nvencvt_register() call to the dispatch block in nvmtl_nvdec_in_decode_service() (plugin/NVMTLDevice.m:471), next to the two built-in registrations. It stays gated by the existing NVMTL_NVENCVT whitelist, so nothing changes for users who have not opted in.

Without it, Safari (and other sandboxed Apple clients that delegate decoding) can only reach the driver's built-in H.264 decoder, which produces an entirely green picture on both Ada and Blackwell. Chrome is fine because it decodes in its own GPU process, where the rating-1100 decoder is registered.

Why this happens
  1. nvmtl_decode_is_delegated() documents that Safari's WebKit GPU process, QuickTime and Preview delegate decoding to VTDecoderXPCService; in-process registration fails under their sandbox.
  2. GH #175 added nvmtl_nvdec_in_decode_service() to register the built-in decoders at load in that one process. Its comment says the XPC service "holds the bundle's mappings but never creates a Metal device". It calls nvmtl_nvdec_register() and nvmtl_nvdec_hevc_register() only.
  3. nvmtl_nvencvt_register() has exactly one call site, NVMTLDevice.m:718, inside Metal device initialisation. I checked both the v2.7.2 tag and current main: they are identical in this respect. So the external library that registers the rating-1100 decoders is never registered in VTDecoderXPCService.
  4. The built-in decoders hand out buffers with hard-coded attributes. On Blackwell, the third-party replacement (arrebel/nvcodec-blackwell v1.2.0, src/vtdec.c) puts it plainly: "What the decoder hands out, built the way the shipped plugin builds it: four keys, the pixel format as a CFNumber ('420v'), an empty IOSurface dictionary" — it never reads VTDecoderSessionGetDestinationPixelBufferAttributes. By contrast mickys/nvenc's set_output() in vt/nvencvt_dec.c inspects the client's destination attributes, optionally builds its own pool and exposes NVDEC_NOOWNPOOL / NVDEC_OWNATTRS / NVDEC_NOMETAL for exactly this variability.

A Safari session's attributes therefore do not match what the built-in decoder produces; VideoToolbox converts or transfers the picture and the chroma plane comes out empty (NV12 with UV=0 renders green). This is the most likely mechanism — I could not dump decoded planes from inside the sandboxed XPC service.

Measurements
  • VTCopyVideoDecoderList on 2.7.2 / RTX 4070 Ti SUPER with mickys/nvenc v0.9.5 shows two avc1 entries: com.nvmtl.videodecoder.avc.nvdec.avc1 (rating 1000, hardware) and org.nvenc.videodecoder.avc.avc1 (rating 1100, hardware).
  • lsof on VTDecoderXPCService shows it maps NVMTLDriver, libnvdec_h264_t2d.dylib and libnvencvt.dylib; com.apple.WebKit.GPU maps none of them.
  • Setting NVMTL_NO_NVDEC=1 removes the rating-1000 decoder from the XPC service and Safari falls back to AppleH264SW. That is the current workaround, but it gives up hardware decoding entirely — and on Blackwell it also disables the third-party replacement decoder, whose installer deliberately strips that very switch because it depends on the registration.
Scope: Ada and Blackwell
Ada (RTX 4070 Ti SUPER) Blackwell (RTX 5060 Ti / 5070 Ti)
rating 1000 in the XPC service built-in libnvdec_h264_t2d.dylib built-in / replaced libnvdec_h264.dylib
rating 1100 mickys/nvenc libnvencvt.dylib nvcodec-blackwell libnvencvt.dylib
H.264 in Safari green green

The two generations use completely different decoder binaries, so a chroma bug in one library does not explain both. The common factor is the XPC path, where only the rating-1000 decoder is reachable. This also fits #472 (green 1080p H.264 in Safari and Edge on an RTX 5060 Ti) and #414 (QuickTime still on AppleH264SW in the XPC service).

Suggested change
// in nvmtl_nvdec_in_decode_service(), inside the dispatch_async block
nvmtl_nvdec_register();
nvmtl_nvdec_hevc_register();
nvmtl_nvencvt_register();   // gated by NVMTL_NVENCVT; registers rating 1100 above ours

With the external decoders present in the XPC service, Safari should pick the rating-1100 decoder and get correct colour with hardware decoding on both generations, without needing NVMTL_NO_NVDEC=1.

Related

#472 (green video, RTX 5060 Ti), #414 (QuickTime H.264 falls back to AppleH264SW), #349 (delegate decoding to VTDecoderXPCService), #65 (hard-coded Blackwell video layout).

Environment

macOS 26.7.2, NullMoth 2.7.2, RTX 4070 Ti SUPER (AD103-A), mickys/nvenc v0.9.5, Safari 27.2 / Chrome 156. Blackwell observations are from #472 and the nvcodec-blackwell v1.2.0 README (RTX 5070 Ti, driver 2.7.1). I have no information about other generations.

Dominant language
Rust
Stars
2.5k
Forks
233
Avg merge
3d 14h
Merged PRs (30d)
14

Getting set up

This project ships no dev container, Dockerfile or contributing guide, so setting up is up to you: start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from nullmoth/nvidia-macos-driver

All issues in nullmoth/nvidia-macos-driver

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.