add runtime TTF fonts on ESP-IDF hosts — CJK text on e-ink
まだ誰も着手していません。
評価
- 難易度
- 4/5
- 見積もり時間
- 3〜5日
- 初心者へのやさしさ
- 52/100
- issue の種類
- 機能追加
- 明瞭さ
- おおむね明確
- 活発さ
- 活発
調査の方向性
Start with docs/BACKENDS.md for the native-text contract (Ui::set_text_measure, TEXT_RUN op 9), then engine/backends/rgb565, where the raster loop falls into _ => return None on TEXT_RUN — add the skip-by-length branch (8 header words + ceil(bytes/4)) for parity with engine/core/src/raster.rs. Next trace hosts/esp-idf/native/ui-core and the generated native_ui.h for the missing set_text_measure export, plus feed_pak's unconditional ui:font.* registration. Done: an ESP-IDF host can install a measure provider and an rgb565 strip containing TEXT_RUN renders instead of aborting.
索引モデルが issue の本文から書いたものです。
説明
Summary
Proposal: runtime TTF text on the ESP-IDF hosts (esp32p4 / esp32s3), built on the engine's
existing native-text contract (Ui::set_text_measure + TEXT_RUN, docs/BACKENDS.md) plus a
host-side FreeType provider — a face mmap'd straight out of flash, a PSRAM glyph cache, and
measure/paint over TEXT_RUN.
Driving use case: Chinese UI text on e-ink panels — thousands of glyphs, several text
sizes, grayscale/MONO rendering targets, slow partial refresh. Baked bitmap atlases cost
chars × sizes; one subset TTF plus an on-device cache covers the same UI at a flat cost, and
cache hits make repaints cheap.
Related: draft PR #442 adds runtime TTF on native / WASM / PSP worker targets (HarfRust
shaping, worker-produced geometry, FreeType gray8). It doesn't cover MCU hosts. This is the
embedded tier of the same story, and it can be much smaller — CJK/UI text needs no complex
shaping. Also related: #518 (fallback glyphs scoped by size).
Why baked atlases don't scale to CJK
- Chars × sizes. A CJK UI wants ~4–7k glyphs (GB2312 level-1 alone is 3,755 hanzi) across
several sizes (atlas slots in our build: 12/14/16/19/25/55 logical px). Atlases bake per
(family-weight, px) — every added size is another full bake. On our 16 MB-PSRAM P4 board
the demo pack'sui:font.*payload was ~3.3 MB and the PAK load step consumed 3.42 MB of
PSRAM — before any frame buffer existed. - One TTF replaces the cross product. Our
fontflash partition carries a 2.33 MB Source
Han Sans SC subset (OFL; ASCII + symbols + GB2312 L1, ~4.5k glyphs); the device rasterizes
any pixel size from it. - E-ink specifics. Grayscale AA (or 1-bit MONO) at size-appropriate crispness, and
repeated paints that must be near-free — a glyph cache gives that; re-baking and
re-uploading per size at build time doesn't. - Downstream data point: a commenter on #442 (jnhu76) reports CJK filenames rendering as
tofu boxes in a Text node.
Already in-tree, and the four ESP-IDF gaps
Already present (verified against 223be88 / v0.13.0): the native-text contract
(Ui::set_text_measure, TEXT_RUN op 9 with the UTF-8 run packed into the draw words,
documented in docs/BACKENDS.md; gpui is the reference backend);
engine/core/src/raster.rs documents TEXT_RUN as a known op to skip ("Skipped, not a
stop"); enhances: ["text.layout.native"] as the app-side opt-in.
Each of these was a local patch in our fork to make the host path work end-to-end:
set_text_measureisn't exported through the C shim. The engine has it, but
hosts/esp-idf/native/ui-coreand the generatednative_ui.hdon't surface it — a C host
cannot install a provider at all. (We added FFIpocketjs_native_ui_set_text_measure+ a
C wrapper.)engine/backends/rgb565aborts onTEXT_RUN. The raster loop has no skip-by-length
branch; it falls into_ => return Noneand the whole strip render fails the first time a
TEXT_RUNappears. Core's software rasterizer skips it by design — the backend should do
the same (8 header words + ceil(bytes/4)).- No way to skip
ui:font.*atlas registration when a host provider is active.
feed_pak()registers the payloads unconditionally; with a host provider they are dead
weight (the 3.3 MB above). We added an explicitnative_textfield on the binding config
that suppressesPOCKETJS_UI_ASSET_FONTregistration (STYLES/IMAGE/SPRITE untouched) and
logs what was skipped. - Tracked runs are excluded from the native path (upstream gate:
native_active() && tracking == 0.0). Once atlases are skipped,tracking-widetext has
neither atlas nor provider — measurement returns (0,0), the box collapses, and whole
paragraphs vanish (our home screen's clock and header aretracking-wide). We pack
tracking intoTEXT_RUNword1 bits 16–31 as i16, 1/64 logical px (upstream: reserved 0),
so measurement and painting keep coming from the same provider; transform subtrees still
use the atlas pair.
Proof of concept (fork; ESP32-P4, 1024×600, raster density 2, 16 MB PSRAM)
Glyph service:
- FreeType from the registry component
espressif/freetype^2.14.3, memory interface
only: mmap the rawfontpartition →FT_New_Memory_Face— zero copy, the filesystem is
never touched. (Our earlier LVGL attempt failed exactly on the prebuilt component's stdio
ftsystem; this path doesn't use it.) - LRU cache keyed by (face, px, gid, load flags): hash buckets + LRU chain, bitmaps and
tables in PSRAM, 1 MiB bitmap budget / 512 entries (either bound evicts). Hits make zero
FreeType calls. Coverage is normalized to 8-bit gray on insert (MONO expanded) — callers
see one format.
Provider over the contract:
- measure = FreeType advances per line, same line-height/style math as the engine;
- paint = walk
TEXT_RUN(8-word header + UTF-8 payload) maintaining the SCISSOR stack,
align/box_w line placement, 26.6 pen accumulation, baseline from FT ascender/descender,
coverage alpha-blended into the RGB565 frame.
Measured: paragraph paint 14.9 ms → 2.4 ms with the cache; steady hit rate 99.7% (96% from
the first 10 s window) at 20–390 KiB of the 1 MiB budget; end-to-end probe 112k measures /
0 failures, 108k runs / 194k glyphs painted / 0 missing, ~1.7 µs/frame all-hit. FreeType
links +411 KiB.
Two notes for a reference provider:
esp_partition_mmapmust run from a task with an internal-RAM stack (PSRAM-stack tasks
trip the cache-disabled stack-sanity assert), so font mounting belongs on the boot path.- A FreeType-free floor exists: LVGL's TinyTTF memory API also works mmap'd and is ~400 KiB
smaller, but it takes TrueType outlines only (we convert the CFF subset to quadratic,
×1.35 file size) and has a fixed 256-glyph cache.
Our PoC board is an LCD; e-ink panel plumbing (1-bit / 4-gray conversion, partial-refresh
policy) stays panel-specific and outside this contract. The contract-side needs — grayscale
coverage, MONO target, arbitrary sizes, cheap repaint — are exactly what the provider
supplies.
Proposal
- Export the native-text contract on the ESP-IDF host —
pocketjs_ui_core_set_text_measure
(FFI + generated header), or a small "host text provider" component interface
(measure + optional paint) if that shape is preferred. - Fix
engine/backends/rgb565to skipTEXT_RUNby length (parity with core raster). - Native-text mode skips atlas registration:
feed_pakdoesn't registerui:font.*
when the host declares it has a provider — ideally tied to the existing
enhances: ["text.layout.native"]declaration (host admission already gates
capabilities), so the packer can also stop baking. - Tracked runs join the native path: tracking in
TEXT_RUNword1 bits 16–31 (i16,
1/64 logical px); transforms keep the atlas pair. - A reference ESP-IDF host text provider beside the existing boot/runner components:
FreeType, mmap face, PSRAM LRU cache, Kconfig for the cache budget and default face
sizes; plus subsetting notes (we emit the Source Han Sans SC GB2312-L1 subset with a
fontTools script;otf2ttfis only needed for the TinyTTF route). - Font declaration for native-text apps: today fonts are the atlas assets. Options: a
raw-TTFui:fontfile.*asset in the pak (host admission already exists), and/or a
host-provided default face perfont-family; app-facingfont-*classes and slots stay
unchanged.
PoC patch set: a downstream private fork of this tree at 223be88; we can share the
patches and the deviation list on request.
Non-goals
- Complex shaping (Arabic/Devanagari), bidi, variable fonts, word wrap (engine v1 breaks on
\nonly) — unchanged here. - Replacing #442's desktop/WASM/PSP worker geometry service. This is the embedded tier.
Open questions
- Reuse #442's service abstractions (bitmap keys, leases, budget accounting) for one shared
code path, or keep a smaller embedded tier (advance-based, no shaping)? Our numbers say
the small tier suffices for CJK UI text. - Cache budget defaults, and the floor for PSRAM-less boards; which tables stay in internal
RAM. - Render-target hint in the contract: 1-bit e-ink wants
FT_LOAD_TARGET_MONO(no AA),
grayscale wants gray8 — provider config or contract field? - Subsetting ownership: upstream tooling + charset presets (GB2312 L1 ≈ 3,755 hanzi ≈ 2.3 MB
TTF; L2 ≈ 6,763), or app responsibility?
- 主要言語
- TypeScript
- スター
- 1.6k
- フォーク
- 88
- PR マージ指標
- 30日以内にマージされた PR はありません
環境構築
このプロジェクトの環境構築ファイルはまだ確認していません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
pocket-stack/pocketjs のほかの issue
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 74/100
pocket-stack/pocketjs#729 ·
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
pocket-stack/pocketjs#725 ·
-
enhancement
難易度 1/5 1時間未満 初心者へのやさしさ 85/100
pocket-stack/pocketjs#724 ·
-
enhancement
難易度 4/5 3〜5日 初心者へのやさしさ 62/100
pocket-stack/pocketjs#728 ·
-
enhancement
難易度 5/5 1週間以上 初心者へのやさしさ 30/100
pocket-stack/pocketjs#727 ·
pocket-stack/pocketjs の issue をすべて見る
似ている issue
-
kind:defect
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 66/100
mux-labs/mux-frontend#1240 ·
メンテナーはふだん 1 日以内に返信
-
area: agents-mcp priority: normal
難易度 2/5 1〜3時間 初心者へのやさしさ 64/100
メンテナーはふだん 1 日以内に返信
-
Wait for the IDS correction dialog to settle in the stale-audit test対応中かも @louistrue が今日担当しました。 オープンready
難易度 2/5 1〜3時間 初心者へのやさしさ 68/100
メンテナーはふだん 1 日以内に返信
-
Hono path parameters with a regex pattern keep the pattern in the OpenAPI path対応中かも @0z5a が今日担当しました。 オープン
難易度 2/5 1〜3時間 初心者へのやさしさ 82/100
cloudflare/chanfana#400 ·
メンテナーはふだん 1 日以内に返信