add runtime TTF fonts on ESP-IDF hosts — CJK text on e-ink

Open
#569 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
52/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Active
Tech stack
c, rust

Research direction

Start with docs/BACKENDS.md for the native-text contract (Ui::set_text_measure, TEXT_RUN op 9), then engine/backends/rgb565, where the raster loop falls into _ => return None on TEXT_RUN — add the skip-by-length branch (8 header words + ceil(bytes/4)) for parity with engine/core/src/raster.rs. Next trace hosts/esp-idf/native/ui-core and the generated native_ui.h for the missing set_text_measure export, plus feed_pak's unconditional ui:font.* registration. Done: an ESP-IDF host can install a measure provider and an rgb565 strip containing TEXT_RUN renders instead of aborting.

Written by the indexing model from the issue text.

Description

Summary

Proposal: runtime TTF text on the ESP-IDF hosts (esp32p4 / esp32s3), built on the engine's
existing native-text contract (Ui::set_text_measure + TEXT_RUN, docs/BACKENDS.md) plus a
host-side FreeType provider — a face mmap'd straight out of flash, a PSRAM glyph cache, and
measure/paint over TEXT_RUN.

Driving use case: Chinese UI text on e-ink panels — thousands of glyphs, several text
sizes, grayscale/MONO rendering targets, slow partial refresh. Baked bitmap atlases cost
chars × sizes; one subset TTF plus an on-device cache covers the same UI at a flat cost, and
cache hits make repaints cheap.

Related: draft PR #442 adds runtime TTF on native / WASM / PSP worker targets (HarfRust
shaping, worker-produced geometry, FreeType gray8). It doesn't cover MCU hosts. This is the
embedded tier of the same story, and it can be much smaller — CJK/UI text needs no complex
shaping. Also related: #518 (fallback glyphs scoped by size).

Why baked atlases don't scale to CJK

  • Chars × sizes. A CJK UI wants ~4–7k glyphs (GB2312 level-1 alone is 3,755 hanzi) across
    several sizes (atlas slots in our build: 12/14/16/19/25/55 logical px). Atlases bake per
    (family-weight, px) — every added size is another full bake. On our 16 MB-PSRAM P4 board
    the demo pack's ui:font.* payload was ~3.3 MB and the PAK load step consumed 3.42 MB of
    PSRAM — before any frame buffer existed.
  • One TTF replaces the cross product. Our font flash partition carries a 2.33 MB Source
    Han Sans SC subset (OFL; ASCII + symbols + GB2312 L1, ~4.5k glyphs); the device rasterizes
    any pixel size from it.
  • E-ink specifics. Grayscale AA (or 1-bit MONO) at size-appropriate crispness, and
    repeated paints that must be near-free — a glyph cache gives that; re-baking and
    re-uploading per size at build time doesn't.
  • Downstream data point: a commenter on #442 (jnhu76) reports CJK filenames rendering as
    tofu boxes in a Text node.

Already in-tree, and the four ESP-IDF gaps

Already present (verified against 223be88 / v0.13.0): the native-text contract
(Ui::set_text_measure, TEXT_RUN op 9 with the UTF-8 run packed into the draw words,
documented in docs/BACKENDS.md; gpui is the reference backend);
engine/core/src/raster.rs documents TEXT_RUN as a known op to skip ("Skipped, not a
stop"); enhances: ["text.layout.native"] as the app-side opt-in.

Each of these was a local patch in our fork to make the host path work end-to-end:

  1. set_text_measure isn't exported through the C shim. The engine has it, but
    hosts/esp-idf/native/ui-core and the generated native_ui.h don't surface it — a C host
    cannot install a provider at all. (We added FFI pocketjs_native_ui_set_text_measure + a
    C wrapper.)
  2. engine/backends/rgb565 aborts on TEXT_RUN. The raster loop has no skip-by-length
    branch; it falls into _ => return None and the whole strip render fails the first time a
    TEXT_RUN appears. Core's software rasterizer skips it by design — the backend should do
    the same (8 header words + ceil(bytes/4)).
  3. No way to skip ui:font.* atlas registration when a host provider is active.
    feed_pak() registers the payloads unconditionally; with a host provider they are dead
    weight (the 3.3 MB above). We added an explicit native_text field on the binding config
    that suppresses POCKETJS_UI_ASSET_FONT registration (STYLES/IMAGE/SPRITE untouched) and
    logs what was skipped.
  4. Tracked runs are excluded from the native path (upstream gate:
    native_active() && tracking == 0.0). Once atlases are skipped, tracking-wide text has
    neither atlas nor provider — measurement returns (0,0), the box collapses, and whole
    paragraphs vanish (our home screen's clock and header are tracking-wide). We pack
    tracking into TEXT_RUN word1 bits 16–31 as i16, 1/64 logical px (upstream: reserved 0),
    so measurement and painting keep coming from the same provider; transform subtrees still
    use the atlas pair.

Proof of concept (fork; ESP32-P4, 1024×600, raster density 2, 16 MB PSRAM)

Glyph service:

  • FreeType from the registry component espressif/freetype ^2.14.3, memory interface
    only
    : mmap the raw font partition → FT_New_Memory_Face — zero copy, the filesystem is
    never touched. (Our earlier LVGL attempt failed exactly on the prebuilt component's stdio
    ftsystem; this path doesn't use it.)
  • LRU cache keyed by (face, px, gid, load flags): hash buckets + LRU chain, bitmaps and
    tables in PSRAM, 1 MiB bitmap budget / 512 entries (either bound evicts). Hits make zero
    FreeType calls. Coverage is normalized to 8-bit gray on insert (MONO expanded) — callers
    see one format.

Provider over the contract:

  • measure = FreeType advances per line, same line-height/style math as the engine;
  • paint = walk TEXT_RUN (8-word header + UTF-8 payload) maintaining the SCISSOR stack,
    align/box_w line placement, 26.6 pen accumulation, baseline from FT ascender/descender,
    coverage alpha-blended into the RGB565 frame.

Measured: paragraph paint 14.9 ms → 2.4 ms with the cache; steady hit rate 99.7% (96% from
the first 10 s window) at 20–390 KiB of the 1 MiB budget; end-to-end probe 112k measures /
0 failures, 108k runs / 194k glyphs painted / 0 missing, ~1.7 µs/frame all-hit. FreeType
links +411 KiB.

Two notes for a reference provider:

  • esp_partition_mmap must run from a task with an internal-RAM stack (PSRAM-stack tasks
    trip the cache-disabled stack-sanity assert), so font mounting belongs on the boot path.
  • A FreeType-free floor exists: LVGL's TinyTTF memory API also works mmap'd and is ~400 KiB
    smaller, but it takes TrueType outlines only (we convert the CFF subset to quadratic,
    ×1.35 file size) and has a fixed 256-glyph cache.

Our PoC board is an LCD; e-ink panel plumbing (1-bit / 4-gray conversion, partial-refresh
policy) stays panel-specific and outside this contract. The contract-side needs — grayscale
coverage, MONO target, arbitrary sizes, cheap repaint — are exactly what the provider
supplies.

Proposal

  1. Export the native-text contract on the ESP-IDF host — pocketjs_ui_core_set_text_measure
    (FFI + generated header), or a small "host text provider" component interface
    (measure + optional paint) if that shape is preferred.
  2. Fix engine/backends/rgb565 to skip TEXT_RUN by length (parity with core raster).
  3. Native-text mode skips atlas registration: feed_pak doesn't register ui:font.*
    when the host declares it has a provider — ideally tied to the existing
    enhances: ["text.layout.native"] declaration (host admission already gates
    capabilities), so the packer can also stop baking.
  4. Tracked runs join the native path: tracking in TEXT_RUN word1 bits 16–31 (i16,
    1/64 logical px); transforms keep the atlas pair.
  5. A reference ESP-IDF host text provider beside the existing boot/runner components:
    FreeType, mmap face, PSRAM LRU cache, Kconfig for the cache budget and default face
    sizes; plus subsetting notes (we emit the Source Han Sans SC GB2312-L1 subset with a
    fontTools script; otf2ttf is only needed for the TinyTTF route).
  6. Font declaration for native-text apps: today fonts are the atlas assets. Options: a
    raw-TTF ui:fontfile.* asset in the pak (host admission already exists), and/or a
    host-provided default face per font-family; app-facing font-* classes and slots stay
    unchanged.

PoC patch set: a downstream private fork of this tree at 223be88; we can share the
patches and the deviation list on request.

Non-goals

  • Complex shaping (Arabic/Devanagari), bidi, variable fonts, word wrap (engine v1 breaks on
    \n only) — unchanged here.
  • Replacing #442's desktop/WASM/PSP worker geometry service. This is the embedded tier.

Open questions

  • Reuse #442's service abstractions (bitmap keys, leases, budget accounting) for one shared
    code path, or keep a smaller embedded tier (advance-based, no shaping)? Our numbers say
    the small tier suffices for CJK UI text.
  • Cache budget defaults, and the floor for PSRAM-less boards; which tables stay in internal
    RAM.
  • Render-target hint in the contract: 1-bit e-ink wants FT_LOAD_TARGET_MONO (no AA),
    grayscale wants gray8 — provider config or contract field?
  • Subsetting ownership: upstream tooling + charset presets (GB2312 L1 ≈ 3,755 hanzi ≈ 2.3 MB
    TTF; L2 ≈ 6,763), or app responsibility?
Dominant language
TypeScript
Stars
1.6k
Forks
88
PR merge metrics
No merged PRs in 30d

Getting set up

We have not checked this project's setup files yet. Start from its README, and see our first-contribution guide for the general steps.

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from pocket-stack/pocketjs

All issues in pocket-stack/pocketjs

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.