Direct Video Plane Access for WebNN
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 8/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Active
- Tech stack
- javascript
- Domain
- api, machine-learning
Research direction
This is a design proposal for the WebNN specification, not a code change, and the maintainers have not settled the API shape. Read the proposal and its API sketch, which adds a copyVideoFrameTo method that takes a VideoFrame and copies its NV12 planes into an MLTensor and a GPUTexture. Done would mean agreed spec text and an accepted format list, which needs maintainer discussion before any drafting.
Written by the indexing model from the issue text.
Description
Video processing workloads need the original NV12 luma and chroma values: the model processes luma only, and its output replaces the source luma before recombination with chroma.
WebGPU's importExternalTexture exposes color-converted RGB, not raw planes. Recovering the original values requires inverting that conversion in a shader. I verified byte-exact recovery against VideoFrame.copyTo (which does not copy directly into an MLTensor) only after matching the browser's BT.709 transfer handling and its matrix coefficients to the default sRGB destination. Exact recovery therefore depends on browser-specific color handling.
Even if WebGPU fully specified the conversion, a sampling API would remain the wrong abstraction for raw plane access. Moreover, even with raw-plane access, WebGPU couldn't copy luma directly into the model’s MLTensor.
Proposal: Since luma is model input, let WebNN copy a VideoFrame's planes without color conversion into caller-provided tensor or texture destinations. WebNN can then select the most efficient device-specific transfer path, giving video-inference workloads direct plane access without shader-based color conversion workarounds.
API sketch (NV12):
const uvTexture = gpuDevice.createTexture({
size: [W / 2, H / 2],
format: 'rg8unorm',
usage: GPUTextureUsage.COPY_DST | GPUTextureUsage.TEXTURE_BINDING,
});
// WebNN may use `gpuDevice` for per-plane GPU copies
context.copyVideoFrameTo(frame, {
y: yTensor, // uint8 [H, W]
uv: uvTexture,
});
Supported formats and API details are open for discussion. Feedback on the direction is welcome.
@huningxin @reillyeon @RafaelCintron @fdwr
- Dominant language
- Bikeshed
- Stars
- 566
- Forks
- 72
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 1
Getting set up
- No Dockerfile or Docker Compose file
- No pull request template
- Read the contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from webmachinelearning/webnn
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
webmachinelearning/webnn#967 ·
-
Difficulty 5/5 Over a week Newbie friendliness 25/100
webmachinelearning/webnn#966 · 1 comment ·
-
Difficulty 5/5 Over a week Newbie friendliness 35/100
webmachinelearning/webnn#965 · 1 comment ·
-
Difficulty 3/5 1-2 days Newbie friendliness 68/100
webmachinelearning/webnn#964 · 1 comment · 1 reaction ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
webmachinelearning/webnn#963 · 2 comments · 1 reaction ·
All issues in webmachinelearning/webnn
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
zach-blumenfeld/aip#13 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
endora-commerce/endora-commerce#397 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
NVIDIA-Omniverse/usd-brep#23 ·
-
documentation good first issue hacktoberfest python
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 69/100
pollen-robotics/reachy_mini#1493 ·
Maintainers usually reply within 1 day