add runtime TTF fonts on ESP-IDF hosts — CJK text on e-ink

Abierto
#569 0 comentarios 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
52/100
Tipo de issue
Nueva funcionalidad
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
c, rust

Línea de trabajo

Start with docs/BACKENDS.md for the native-text contract (Ui::set_text_measure, TEXT_RUN op 9), then engine/backends/rgb565, where the raster loop falls into _ => return None on TEXT_RUN — add the skip-by-length branch (8 header words + ceil(bytes/4)) for parity with engine/core/src/raster.rs. Next trace hosts/esp-idf/native/ui-core and the generated native_ui.h for the missing set_text_measure export, plus feed_pak's unconditional ui:font.* registration. Done: an ESP-IDF host can install a measure provider and an rgb565 strip containing TEXT_RUN renders instead of aborting.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

Summary

Proposal: runtime TTF text on the ESP-IDF hosts (esp32p4 / esp32s3), built on the engine's
existing native-text contract (Ui::set_text_measure + TEXT_RUN, docs/BACKENDS.md) plus a
host-side FreeType provider — a face mmap'd straight out of flash, a PSRAM glyph cache, and
measure/paint over TEXT_RUN.

Driving use case: Chinese UI text on e-ink panels — thousands of glyphs, several text
sizes, grayscale/MONO rendering targets, slow partial refresh. Baked bitmap atlases cost
chars × sizes; one subset TTF plus an on-device cache covers the same UI at a flat cost, and
cache hits make repaints cheap.

Related: draft PR #442 adds runtime TTF on native / WASM / PSP worker targets (HarfRust
shaping, worker-produced geometry, FreeType gray8). It doesn't cover MCU hosts. This is the
embedded tier of the same story, and it can be much smaller — CJK/UI text needs no complex
shaping. Also related: #518 (fallback glyphs scoped by size).

Why baked atlases don't scale to CJK

  • Chars × sizes. A CJK UI wants ~4–7k glyphs (GB2312 level-1 alone is 3,755 hanzi) across
    several sizes (atlas slots in our build: 12/14/16/19/25/55 logical px). Atlases bake per
    (family-weight, px) — every added size is another full bake. On our 16 MB-PSRAM P4 board
    the demo pack's ui:font.* payload was ~3.3 MB and the PAK load step consumed 3.42 MB of
    PSRAM — before any frame buffer existed.
  • One TTF replaces the cross product. Our font flash partition carries a 2.33 MB Source
    Han Sans SC subset (OFL; ASCII + symbols + GB2312 L1, ~4.5k glyphs); the device rasterizes
    any pixel size from it.
  • E-ink specifics. Grayscale AA (or 1-bit MONO) at size-appropriate crispness, and
    repeated paints that must be near-free — a glyph cache gives that; re-baking and
    re-uploading per size at build time doesn't.
  • Downstream data point: a commenter on #442 (jnhu76) reports CJK filenames rendering as
    tofu boxes in a Text node.

Already in-tree, and the four ESP-IDF gaps

Already present (verified against 223be88 / v0.13.0): the native-text contract
(Ui::set_text_measure, TEXT_RUN op 9 with the UTF-8 run packed into the draw words,
documented in docs/BACKENDS.md; gpui is the reference backend);
engine/core/src/raster.rs documents TEXT_RUN as a known op to skip ("Skipped, not a
stop"); enhances: ["text.layout.native"] as the app-side opt-in.

Each of these was a local patch in our fork to make the host path work end-to-end:

  1. set_text_measure isn't exported through the C shim. The engine has it, but
    hosts/esp-idf/native/ui-core and the generated native_ui.h don't surface it — a C host
    cannot install a provider at all. (We added FFI pocketjs_native_ui_set_text_measure + a
    C wrapper.)
  2. engine/backends/rgb565 aborts on TEXT_RUN. The raster loop has no skip-by-length
    branch; it falls into _ => return None and the whole strip render fails the first time a
    TEXT_RUN appears. Core's software rasterizer skips it by design — the backend should do
    the same (8 header words + ceil(bytes/4)).
  3. No way to skip ui:font.* atlas registration when a host provider is active.
    feed_pak() registers the payloads unconditionally; with a host provider they are dead
    weight (the 3.3 MB above). We added an explicit native_text field on the binding config
    that suppresses POCKETJS_UI_ASSET_FONT registration (STYLES/IMAGE/SPRITE untouched) and
    logs what was skipped.
  4. Tracked runs are excluded from the native path (upstream gate:
    native_active() && tracking == 0.0). Once atlases are skipped, tracking-wide text has
    neither atlas nor provider — measurement returns (0,0), the box collapses, and whole
    paragraphs vanish (our home screen's clock and header are tracking-wide). We pack
    tracking into TEXT_RUN word1 bits 16–31 as i16, 1/64 logical px (upstream: reserved 0),
    so measurement and painting keep coming from the same provider; transform subtrees still
    use the atlas pair.

Proof of concept (fork; ESP32-P4, 1024×600, raster density 2, 16 MB PSRAM)

Glyph service:

  • FreeType from the registry component espressif/freetype ^2.14.3, memory interface
    only
    : mmap the raw font partition → FT_New_Memory_Face — zero copy, the filesystem is
    never touched. (Our earlier LVGL attempt failed exactly on the prebuilt component's stdio
    ftsystem; this path doesn't use it.)
  • LRU cache keyed by (face, px, gid, load flags): hash buckets + LRU chain, bitmaps and
    tables in PSRAM, 1 MiB bitmap budget / 512 entries (either bound evicts). Hits make zero
    FreeType calls. Coverage is normalized to 8-bit gray on insert (MONO expanded) — callers
    see one format.

Provider over the contract:

  • measure = FreeType advances per line, same line-height/style math as the engine;
  • paint = walk TEXT_RUN (8-word header + UTF-8 payload) maintaining the SCISSOR stack,
    align/box_w line placement, 26.6 pen accumulation, baseline from FT ascender/descender,
    coverage alpha-blended into the RGB565 frame.

Measured: paragraph paint 14.9 ms → 2.4 ms with the cache; steady hit rate 99.7% (96% from
the first 10 s window) at 20–390 KiB of the 1 MiB budget; end-to-end probe 112k measures /
0 failures, 108k runs / 194k glyphs painted / 0 missing, ~1.7 µs/frame all-hit. FreeType
links +411 KiB.

Two notes for a reference provider:

  • esp_partition_mmap must run from a task with an internal-RAM stack (PSRAM-stack tasks
    trip the cache-disabled stack-sanity assert), so font mounting belongs on the boot path.
  • A FreeType-free floor exists: LVGL's TinyTTF memory API also works mmap'd and is ~400 KiB
    smaller, but it takes TrueType outlines only (we convert the CFF subset to quadratic,
    ×1.35 file size) and has a fixed 256-glyph cache.

Our PoC board is an LCD; e-ink panel plumbing (1-bit / 4-gray conversion, partial-refresh
policy) stays panel-specific and outside this contract. The contract-side needs — grayscale
coverage, MONO target, arbitrary sizes, cheap repaint — are exactly what the provider
supplies.

Proposal

  1. Export the native-text contract on the ESP-IDF host — pocketjs_ui_core_set_text_measure
    (FFI + generated header), or a small "host text provider" component interface
    (measure + optional paint) if that shape is preferred.
  2. Fix engine/backends/rgb565 to skip TEXT_RUN by length (parity with core raster).
  3. Native-text mode skips atlas registration: feed_pak doesn't register ui:font.*
    when the host declares it has a provider — ideally tied to the existing
    enhances: ["text.layout.native"] declaration (host admission already gates
    capabilities), so the packer can also stop baking.
  4. Tracked runs join the native path: tracking in TEXT_RUN word1 bits 16–31 (i16,
    1/64 logical px); transforms keep the atlas pair.
  5. A reference ESP-IDF host text provider beside the existing boot/runner components:
    FreeType, mmap face, PSRAM LRU cache, Kconfig for the cache budget and default face
    sizes; plus subsetting notes (we emit the Source Han Sans SC GB2312-L1 subset with a
    fontTools script; otf2ttf is only needed for the TinyTTF route).
  6. Font declaration for native-text apps: today fonts are the atlas assets. Options: a
    raw-TTF ui:fontfile.* asset in the pak (host admission already exists), and/or a
    host-provided default face per font-family; app-facing font-* classes and slots stay
    unchanged.

PoC patch set: a downstream private fork of this tree at 223be88; we can share the
patches and the deviation list on request.

Non-goals

  • Complex shaping (Arabic/Devanagari), bidi, variable fonts, word wrap (engine v1 breaks on
    \n only) — unchanged here.
  • Replacing #442's desktop/WASM/PSP worker geometry service. This is the embedded tier.

Open questions

  • Reuse #442's service abstractions (bitmap keys, leases, budget accounting) for one shared
    code path, or keep a smaller embedded tier (advance-based, no shaping)? Our numbers say
    the small tier suffices for CJK UI text.
  • Cache budget defaults, and the floor for PSRAM-less boards; which tables stay in internal
    RAM.
  • Render-target hint in the contract: 1-bit e-ink wants FT_LOAD_TARGET_MONO (no AA),
    grayscale wants gray8 — provider config or contract field?
  • Subsetting ownership: upstream tooling + charset presets (GB2312 L1 ≈ 3,755 hanzi ≈ 2.3 MB
    TTF; L2 ≈ 6,763), or app responsibility?
Lenguaje dominante
TypeScript
Estrellas
1.6k
Forks
88
Métricas de merge de PR
Sin PR fusionados en 30 d

Preparar el entorno

Aún no hemos revisado los archivos de configuración de este proyecto. Empieza por su README y consulta nuestra guía para la primera contribución para los pasos generales.

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de pocket-stack/pocketjs

Todos los issues de pocket-stack/pocketjs

Issues similares

Más issues de TypeScript

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.