Deterministic Rendering Challenges

Kaleido silently renders text against the wrong font when a custom @font-face is injected, and headless Chromium's device-scale-factor is not guaranteed 1.0 - neither is documented

Summary

We render several thousand Plotly SVGs per CI run and compare them byte-for-byte against
golden references (a determinism requirement for our test suite, not a Kaleido use case
Kaleido needs to support directly - but two of its undocumented behaviours make any
approach to deterministic rendering fragile). Both issues below cost us real debugging time
(a build log, several developer machines, and days of diffing SVGs) because neither is
mentioned anywhere in Kaleido's README, docstrings, or (as far as we found) the issue tracker.

1. The rendered font is resolved from the host OS with no documented way to pin it, and
   injecting your own @font-face to work around that does not make rendering deterministic
   either. Kaleido bundles no fonts of its own, and neither does Chrome for Testing - text
   is resolved against whatever the OS's font stack happens to have installed (on Windows,
   via DirectWrite). Plotly's default font stack, 'Open Sans', verdana, arial, sans-serif,
   is not a Windows default and isn't installed by anything Kaleido or Chrome for Testing
   ship, so two machines - or two OSes - with different fonts present silently render the
   same figure with different text metrics (margins, legend height, plot-area height all
   derive from them). The only way we found to make this deterministic is to embed a font as
   a data: URI @font-face under a private family name, injected into the render page - but
   doing that surfaces the second half of this issue: Plotly.toImage is called with no
   font-readiness step, and @font-face loading is asynchronous even from a data: URI.
   Depending on how a figure is rendered (a fresh page per figure vs. a pool of reused
   tabs/pages), the SAME figure is sometimes measured against the just-injected font and
   sometimes against the CSS fallback stack - silently, with a valid SVG returned either way
   and nothing in Kaleido's return value, logs, or exit status indicating which happened.
   Neither the OS-dependency nor the workaround's own race is documented anywhere we could
   find.

2. Headless Chromium's device scale factor (DSF) is not reliably 1.0 by default. We
   confirmed this directly on a machine that had the correct pinned Chrome for Testing build
   and the correct embedded font, both verified, and it still produced text/legend
   measurements a fraction of a pixel off from a machine that measured 1.0 by default,
   independent of the OS's own display-scaling setting (both were at 100%) and independent of
   local vs. remote-desktop session (both were RDP). The only fix we found is subclassing
   choreographer.browsers.chromium.Chromium and appending
   --force-device-scale-factor=1 in an overridden get_cli() - there is no public Kaleido
   kwarg for this, and the flag itself is undocumented in Kaleido's context.

Filing this primarily as a documentation gap: neither behaviour is wrong, exactly, but
both are surprising, both silently change output pixel geometry, and neither has any pointer
in the docs telling an integrator they need to think about it. We'd be glad to send a PR
adding a "Determinism" section to the docs (font-readiness wrapping + DSF pinning) if that's
welcome - happy to discuss the right shape for it.

Reproduction 1: font-load race (fully reproducible, not flaky)

Attached: repro_font_race.py - self-contained, only needs pip install kaleido plotly.

It renders one simple figure with a custom @font-face injected into Kaleido's render page
(kopts={"page_generator": PageGenerator(others=[script_path])}), twice:

• (a) via ordinary, separate kaleido.write_fig_sync() calls, one figure per fresh tab -
  the natural way to render one-figure-at-a-time (e.g. a test suite rendering probes, or any
  caller not deliberately batching figures through one long-lived Kaleido instance);
• (b) identical in every respect, except the injected script additionally wraps
  Plotly.toImage in a promise that waits on document.fonts.ready - a workaround that isn't
  part of Kaleido's public API and isn't documented anywhere we could find; we found it by
  reading the CSS Font Loading Module spec, not Kaleido's docs.

On our machine (Windows 11, kaleido 1.0.0rc15, plotly 6.1.2, choreographer 1.0.7 - the same
root cause was also observed on kaleido 1.2.0 / plotly 6.9.0 / choreographer 1.2.1), (a)
measures the legend hit-box at 216.8125 on every single run - i.e. not flaky, just
silently and consistently wrong, always measured against the CSS fallback font, never the
injected one - while (b) measures 217.138671875 on every run, i.e. actually using the
injected font. Nothing about calling (a) signals that the font never applied; the SVG is
returned successfully.

$ python repro_font_race.py
Rendering N figures back-to-back, each in its own fresh Chromium tab, WITHOUT waiting for
the injected @font-face to finish loading (this is exactly what Plotly.toImage does by
default - no readiness step):

  render 0: legend hit-box width = 216.8125
  ...

Same figure, same font, same everything - except Plotly.toImage is now wrapped to wait for
document.fonts.ready before rasterising (NOT a Kaleido feature - hand-rolled, undocumented
workaround):

  render 0: legend hit-box width = 217.138671875
  ...

=> REPRODUCED: identical figure, identical injected font, identical process/environment -
216.8125 without the undocumented wait workaround vs 217.138671875 with it.

In our production use (many figures through a small pool of reused tabs, not one figure per
process), we additionally observed the same figure landing on either number depending on
scheduling - i.e. this can present as flakiness rather than a consistent miss, which is worse
to diagnose, not better.

Reproduction 2 (environmental, needs two machines): device scale factor

We don't have a single-machine repro for this half - by its nature it depends on ambient
Chromium/OS state we haven't fully characterised - but we observed it directly: two machines,
same pinned Chrome for Testing build, same embedded font (verified identical via reproduction
1's technique), same OS display-scaling setting (100% on both), one local and one over remote
desktop, produced different sub-pixel text/legend measurements for the same figure. The only
difference we could isolate was headless Chromium's internal device scale factor (DSF), which
is not guaranteed to default to 1.0.

Our workaround: subclassing choreographer.browsers.chromium.Chromium, overriding get_cli()
to append --force-device-scale-factor=1, and passing it as browser_cls= to
kaleido.Kaleido(...)/write_fig_sync(...).

Even without a portable single-machine repro, we think this is worth documenting: an
integrator who has already solved the font problem (issue 1 above) can still fail
byte-for-byte comparison purely from DSF, with no obvious next step and no mention of DSF
anywhere in Kaleido's docs.

What we're asking for

• A README/docs note (or a "Determinism" / "Reproducible rendering" section) covering:
  • Text is rendered using whatever font the host OS resolves; there's no documented,
    supported way to pin a specific font so output is identical across OSes/machines. The
    accepted-by-us workaround (embed the font as a data: URI @font-face under a private
    family name) deserves at least a documented pointer, even if Kaleido doesn't want to ship
    a font itself.
  • Plotly.toImage does not wait for injected/custom fonts to finish loading; anyone
    injecting a font-face needs to wrap it themselves (example above).
  • Headless Chromium's DSF is not guaranteed 1.0; --force-device-scale-factor=1 (via a
    browser_cls= override, since there's no direct kwarg) is the fix, if that remains the
    supported mechanism.
• If either of these has a supported, public way to configure it that we missed, a pointer
  would resolve this on its own - we would much rather use documented API than a
  Chromium subclass reaching into get_cli().

Happy to expand the repro script, run it against other Kaleido/choreographer versions, or
open a docs PR ourselves if that's the preferred next step.

repro_font_race.py