Specify alpha association for VideoFrame buffer input and copyTo() output

Read the WebCodecs sections linked in the issue for the VideoFrame buffer and image source constructors, copyTo(), and Convert to RGB frame. Compare their alpha association rules with the cited HTML canvas and ImageBitmap requirements; the issue proposes defining observable behavior, conversion precision, zero-alpha pixels, and YUVA semantics. Done means the specification resolves those cases and adds distinguishing translucent regression tests.

The WebCodecs Editor's Draft checked on 2026-10-12 defines channel order and sample sizes for RGBA and BGRA, but does not explicitly say whether their RGB samples are premultiplied by alpha. The buffer constructor and copyTo() copy pixel data without specifying an alpha-association conversion. When an RGB output format is explicitly requested, copyTo() calls Convert to RGB frame, which specifies conversion of color space and pixel format, but does not explicitly define alpha association either.

This also matters for frames constructed from canvases and ImageBitmaps. The image-source constructor copies or references their bitmap data. HTML requires 2D canvas output bitmaps to use premultiplied alpha; ImageBitmap's premultiplyAlpha processing allows different representations. WebCodecs needs to define the observable buffer input/output semantics across these sources.

Measurements on Windows 11, rerun on 2026-10-12 with fresh profiles in headless mode:

• Chrome 154.0.8037.99, with default graphics flags and with --disable-gpu.
• Firefox Nightly 160.0a1, build ID 20261011213519.

The probe uses a 2x2 RGBA buffer whose pixels are (200, 100, 50, 128), and a 2x2 2D OffscreenCanvas filled with the corresponding CSS color. Each readback below is the first pixel. The Chrome columns identify launch settings; the measurements alone do not identify every internal graphics path.

(async () => {
  function readBack(source) {
    const canvas = new OffscreenCanvas(2, 2);
    const ctx = canvas.getContext("2d");
    ctx.drawImage(source, 0, 0);
    return Array.from(ctx.getImageData(0, 0, 1, 1).data);
  }

  async function copyBytes(frame, options) {
    const out = new Uint8Array(frame.allocationSize(options));
    await frame.copyTo(out, options);
    return Array.from(out.subarray(0, 4));
  }

  const data = new Uint8Array(16);
  for (let i = 0; i < data.length; i += 4) {
    data.set([200, 100, 50, 128], i);
  }
  const bufferFrame = new VideoFrame(data, {
    format: "RGBA", codedWidth: 2, codedHeight: 2, timestamp: 0,
  });
  const canvas = new OffscreenCanvas(2, 2);
  const ctx = canvas.getContext("2d");
  ctx.fillStyle = `rgba(200, 100, 50, ${128 / 255})`;
  ctx.fillRect(0, 0, 2, 2);
  const canvasFrame = new VideoFrame(canvas, {timestamp: 0});
  try {
    console.log({
      ua: navigator.userAgent,
      bufferFrame_drawImage: readBack(bufferFrame),
      bufferFrame_copyTo: await copyBytes(bufferFrame),
      canvas_getImageData: Array.from(ctx.getImageData(0, 0, 1, 1).data),
      canvasFrame_format: canvasFrame.format,
      canvasFrame_drawImage: readBack(canvasFrame),
      canvasFrame_copyTo: await copyBytes(canvasFrame),
      canvasFrame_copyTo_RGBA: await copyBytes(canvasFrame, {format: "RGBA"}),
    });
  } finally {
    bufferFrame.close();
    canvasFrame.close();
  }
})();

These observations are consistent with straight-alpha interpretation of the buffer in Chrome, and premultiplied-alpha interpretation when drawing the buffer in Firefox. For canvas frames, default-format copyTo() returns values consistent with straight alpha in the default Chrome run and premultiplied alpha in the other two runs. With --disable-gpu, Chrome also draws the canvas frame with roughly halved RGB values, consistent with interpreting premultiplied samples as straight. These are interpretations of this probe's output, not evidence that all frames follow the same internal path. Small rounding differences are separate from the much larger color errors.

The Firefox buffer-drawing issue is tracked in bug 2079836. D332178, currently awaiting review, changes RGBA/BGRA buffer handling to preserve straight-alpha colors when drawing. The existing local build tested during this audit (160.0a1, build ID 20261009205724) gives (201, 101, 51, 128) for the buffer drawing row and retains the canvas copyTo() values shown above. This local-build observation is not a controlled before/after test of that patch.

Proposal: explicitly define RGBA/BGRA buffer input and copyTo() output as non-premultiplied (straight) alpha, including output from Convert to RGB frame and from frames created with canvas or ImageBitmap sources. Specify the necessary conversions at observable API boundaries while allowing implementations to retain premultiplied resources internally. The clarification should address conversion precision and zero-alpha pixels. Alpha association should also be made explicit for the YUVA formats (I420A, I422A, I444A, and their higher-bit-depth variants); this RGB probe does not test them.

Relevant prior discussion and related API behavior:

• WebCodecs issue #508 explicitly argues that RGBA/BGRA copyTo() output should be unpremultiplied, while internal premultiplication should not affect consumers. It also discusses createImageBitmap() for callers requesting premultiplication. This supports the proposed semantics, but is discussion rather than a normative WebCodecs requirement.
• WebGPU's importExternalTexture() algorithm specifies conversion of the source image to the requested color space "with unpremultiplied alpha". This specifies the imported texture's values, not the representation returned by VideoFrame.copyTo().
• The naming discussion in issue #200 uses ImageData as the model for RGBA byte order. That comment does not itself specify alpha association.

The upstream webcodecs/videoFrame-alpha.any.js checked for this report tests opaque colors and transparent black. Those particular pixels do not distinguish the two alpha representations. D332178 adds translucent RGBA/BGRA buffer tests for drawing and related consumers, but does not add the translucent canvas-to-copyTo() checks illustrated here. Those cases would also be useful regression tests once the required semantics are agreed.