add runtime TTF fonts on ESP-IDF hosts — CJK text on e-ink

オープン
#569 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る

まだ誰も着手していません。

評価

難易度
4/5
見積もり時間
3〜5日
初心者へのやさしさ
52/100
issue の種類
機能追加
明瞭さ
おおむね明確
活発さ
活発
技術スタック
c, rust

調査の方向性

Start with docs/BACKENDS.md for the native-text contract (Ui::set_text_measure, TEXT_RUN op 9), then engine/backends/rgb565, where the raster loop falls into _ => return None on TEXT_RUN — add the skip-by-length branch (8 header words + ceil(bytes/4)) for parity with engine/core/src/raster.rs. Next trace hosts/esp-idf/native/ui-core and the generated native_ui.h for the missing set_text_measure export, plus feed_pak's unconditional ui:font.* registration. Done: an ESP-IDF host can install a measure provider and an rgb565 strip containing TEXT_RUN renders instead of aborting.

索引モデルが issue の本文から書いたものです。

説明

Summary

Proposal: runtime TTF text on the ESP-IDF hosts (esp32p4 / esp32s3), built on the engine's
existing native-text contract (Ui::set_text_measure + TEXT_RUN, docs/BACKENDS.md) plus a
host-side FreeType provider — a face mmap'd straight out of flash, a PSRAM glyph cache, and
measure/paint over TEXT_RUN.

Driving use case: Chinese UI text on e-ink panels — thousands of glyphs, several text
sizes, grayscale/MONO rendering targets, slow partial refresh. Baked bitmap atlases cost
chars × sizes; one subset TTF plus an on-device cache covers the same UI at a flat cost, and
cache hits make repaints cheap.

Related: draft PR #442 adds runtime TTF on native / WASM / PSP worker targets (HarfRust
shaping, worker-produced geometry, FreeType gray8). It doesn't cover MCU hosts. This is the
embedded tier of the same story, and it can be much smaller — CJK/UI text needs no complex
shaping. Also related: #518 (fallback glyphs scoped by size).

Why baked atlases don't scale to CJK

  • Chars × sizes. A CJK UI wants ~4–7k glyphs (GB2312 level-1 alone is 3,755 hanzi) across
    several sizes (atlas slots in our build: 12/14/16/19/25/55 logical px). Atlases bake per
    (family-weight, px) — every added size is another full bake. On our 16 MB-PSRAM P4 board
    the demo pack's ui:font.* payload was ~3.3 MB and the PAK load step consumed 3.42 MB of
    PSRAM — before any frame buffer existed.
  • One TTF replaces the cross product. Our font flash partition carries a 2.33 MB Source
    Han Sans SC subset (OFL; ASCII + symbols + GB2312 L1, ~4.5k glyphs); the device rasterizes
    any pixel size from it.
  • E-ink specifics. Grayscale AA (or 1-bit MONO) at size-appropriate crispness, and
    repeated paints that must be near-free — a glyph cache gives that; re-baking and
    re-uploading per size at build time doesn't.
  • Downstream data point: a commenter on #442 (jnhu76) reports CJK filenames rendering as
    tofu boxes in a Text node.

Already in-tree, and the four ESP-IDF gaps

Already present (verified against 223be88 / v0.13.0): the native-text contract
(Ui::set_text_measure, TEXT_RUN op 9 with the UTF-8 run packed into the draw words,
documented in docs/BACKENDS.md; gpui is the reference backend);
engine/core/src/raster.rs documents TEXT_RUN as a known op to skip ("Skipped, not a
stop"); enhances: ["text.layout.native"] as the app-side opt-in.

Each of these was a local patch in our fork to make the host path work end-to-end:

  1. set_text_measure isn't exported through the C shim. The engine has it, but
    hosts/esp-idf/native/ui-core and the generated native_ui.h don't surface it — a C host
    cannot install a provider at all. (We added FFI pocketjs_native_ui_set_text_measure + a
    C wrapper.)
  2. engine/backends/rgb565 aborts on TEXT_RUN. The raster loop has no skip-by-length
    branch; it falls into _ => return None and the whole strip render fails the first time a
    TEXT_RUN appears. Core's software rasterizer skips it by design — the backend should do
    the same (8 header words + ceil(bytes/4)).
  3. No way to skip ui:font.* atlas registration when a host provider is active.
    feed_pak() registers the payloads unconditionally; with a host provider they are dead
    weight (the 3.3 MB above). We added an explicit native_text field on the binding config
    that suppresses POCKETJS_UI_ASSET_FONT registration (STYLES/IMAGE/SPRITE untouched) and
    logs what was skipped.
  4. Tracked runs are excluded from the native path (upstream gate:
    native_active() && tracking == 0.0). Once atlases are skipped, tracking-wide text has
    neither atlas nor provider — measurement returns (0,0), the box collapses, and whole
    paragraphs vanish (our home screen's clock and header are tracking-wide). We pack
    tracking into TEXT_RUN word1 bits 16–31 as i16, 1/64 logical px (upstream: reserved 0),
    so measurement and painting keep coming from the same provider; transform subtrees still
    use the atlas pair.

Proof of concept (fork; ESP32-P4, 1024×600, raster density 2, 16 MB PSRAM)

Glyph service:

  • FreeType from the registry component espressif/freetype ^2.14.3, memory interface
    only
    : mmap the raw font partition → FT_New_Memory_Face — zero copy, the filesystem is
    never touched. (Our earlier LVGL attempt failed exactly on the prebuilt component's stdio
    ftsystem; this path doesn't use it.)
  • LRU cache keyed by (face, px, gid, load flags): hash buckets + LRU chain, bitmaps and
    tables in PSRAM, 1 MiB bitmap budget / 512 entries (either bound evicts). Hits make zero
    FreeType calls. Coverage is normalized to 8-bit gray on insert (MONO expanded) — callers
    see one format.

Provider over the contract:

  • measure = FreeType advances per line, same line-height/style math as the engine;
  • paint = walk TEXT_RUN (8-word header + UTF-8 payload) maintaining the SCISSOR stack,
    align/box_w line placement, 26.6 pen accumulation, baseline from FT ascender/descender,
    coverage alpha-blended into the RGB565 frame.

Measured: paragraph paint 14.9 ms → 2.4 ms with the cache; steady hit rate 99.7% (96% from
the first 10 s window) at 20–390 KiB of the 1 MiB budget; end-to-end probe 112k measures /
0 failures, 108k runs / 194k glyphs painted / 0 missing, ~1.7 µs/frame all-hit. FreeType
links +411 KiB.

Two notes for a reference provider:

  • esp_partition_mmap must run from a task with an internal-RAM stack (PSRAM-stack tasks
    trip the cache-disabled stack-sanity assert), so font mounting belongs on the boot path.
  • A FreeType-free floor exists: LVGL's TinyTTF memory API also works mmap'd and is ~400 KiB
    smaller, but it takes TrueType outlines only (we convert the CFF subset to quadratic,
    ×1.35 file size) and has a fixed 256-glyph cache.

Our PoC board is an LCD; e-ink panel plumbing (1-bit / 4-gray conversion, partial-refresh
policy) stays panel-specific and outside this contract. The contract-side needs — grayscale
coverage, MONO target, arbitrary sizes, cheap repaint — are exactly what the provider
supplies.

Proposal

  1. Export the native-text contract on the ESP-IDF host — pocketjs_ui_core_set_text_measure
    (FFI + generated header), or a small "host text provider" component interface
    (measure + optional paint) if that shape is preferred.
  2. Fix engine/backends/rgb565 to skip TEXT_RUN by length (parity with core raster).
  3. Native-text mode skips atlas registration: feed_pak doesn't register ui:font.*
    when the host declares it has a provider — ideally tied to the existing
    enhances: ["text.layout.native"] declaration (host admission already gates
    capabilities), so the packer can also stop baking.
  4. Tracked runs join the native path: tracking in TEXT_RUN word1 bits 16–31 (i16,
    1/64 logical px); transforms keep the atlas pair.
  5. A reference ESP-IDF host text provider beside the existing boot/runner components:
    FreeType, mmap face, PSRAM LRU cache, Kconfig for the cache budget and default face
    sizes; plus subsetting notes (we emit the Source Han Sans SC GB2312-L1 subset with a
    fontTools script; otf2ttf is only needed for the TinyTTF route).
  6. Font declaration for native-text apps: today fonts are the atlas assets. Options: a
    raw-TTF ui:fontfile.* asset in the pak (host admission already exists), and/or a
    host-provided default face per font-family; app-facing font-* classes and slots stay
    unchanged.

PoC patch set: a downstream private fork of this tree at 223be88; we can share the
patches and the deviation list on request.

Non-goals

  • Complex shaping (Arabic/Devanagari), bidi, variable fonts, word wrap (engine v1 breaks on
    \n only) — unchanged here.
  • Replacing #442's desktop/WASM/PSP worker geometry service. This is the embedded tier.

Open questions

  • Reuse #442's service abstractions (bitmap keys, leases, budget accounting) for one shared
    code path, or keep a smaller embedded tier (advance-based, no shaping)? Our numbers say
    the small tier suffices for CJK UI text.
  • Cache budget defaults, and the floor for PSRAM-less boards; which tables stay in internal
    RAM.
  • Render-target hint in the contract: 1-bit e-ink wants FT_LOAD_TARGET_MONO (no AA),
    grayscale wants gray8 — provider config or contract field?
  • Subsetting ownership: upstream tooling + charset presets (GB2312 L1 ≈ 3,755 hanzi ≈ 2.3 MB
    TTF; L2 ≈ 6,763), or app responsibility?
主要言語
TypeScript
スター
1.6k
フォーク
88
PR マージ指標
30日以内にマージされた PR はありません

環境構築

このプロジェクトの環境構築ファイルはまだ確認していません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。

はじめの一歩

  1. issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
  2. 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
  3. リポジトリをフォークし、ブランチを切って変更します。
  4. issue 番号を参照したプルリクエストを送ります。

pocket-stack/pocketjs のほかの issue

pocket-stack/pocketjs の issue をすべて見る

似ている issue

TypeScript の issue をもっと見る

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。