add runtime TTF fonts on ESP-IDF hosts — CJK text on e-ink
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 52/100
- Issue type
- Feature
- Clarity
- Mostly clear
- Activity status
- Active
- Domain
- computer-graphics, embedded-iot
Research direction
Start with docs/BACKENDS.md for the native-text contract (Ui::set_text_measure, TEXT_RUN op 9), then engine/backends/rgb565, where the raster loop falls into _ => return None on TEXT_RUN — add the skip-by-length branch (8 header words + ceil(bytes/4)) for parity with engine/core/src/raster.rs. Next trace hosts/esp-idf/native/ui-core and the generated native_ui.h for the missing set_text_measure export, plus feed_pak's unconditional ui:font.* registration. Done: an ESP-IDF host can install a measure provider and an rgb565 strip containing TEXT_RUN renders instead of aborting.
Written by the indexing model from the issue text.
Description
Summary
Proposal: runtime TTF text on the ESP-IDF hosts (esp32p4 / esp32s3), built on the engine's
existing native-text contract (Ui::set_text_measure + TEXT_RUN, docs/BACKENDS.md) plus a
host-side FreeType provider — a face mmap'd straight out of flash, a PSRAM glyph cache, and
measure/paint over TEXT_RUN.
Driving use case: Chinese UI text on e-ink panels — thousands of glyphs, several text
sizes, grayscale/MONO rendering targets, slow partial refresh. Baked bitmap atlases cost
chars × sizes; one subset TTF plus an on-device cache covers the same UI at a flat cost, and
cache hits make repaints cheap.
Related: draft PR #442 adds runtime TTF on native / WASM / PSP worker targets (HarfRust
shaping, worker-produced geometry, FreeType gray8). It doesn't cover MCU hosts. This is the
embedded tier of the same story, and it can be much smaller — CJK/UI text needs no complex
shaping. Also related: #518 (fallback glyphs scoped by size).
Why baked atlases don't scale to CJK
- Chars × sizes. A CJK UI wants ~4–7k glyphs (GB2312 level-1 alone is 3,755 hanzi) across
several sizes (atlas slots in our build: 12/14/16/19/25/55 logical px). Atlases bake per
(family-weight, px) — every added size is another full bake. On our 16 MB-PSRAM P4 board
the demo pack'sui:font.*payload was ~3.3 MB and the PAK load step consumed 3.42 MB of
PSRAM — before any frame buffer existed. - One TTF replaces the cross product. Our
fontflash partition carries a 2.33 MB Source
Han Sans SC subset (OFL; ASCII + symbols + GB2312 L1, ~4.5k glyphs); the device rasterizes
any pixel size from it. - E-ink specifics. Grayscale AA (or 1-bit MONO) at size-appropriate crispness, and
repeated paints that must be near-free — a glyph cache gives that; re-baking and
re-uploading per size at build time doesn't. - Downstream data point: a commenter on #442 (jnhu76) reports CJK filenames rendering as
tofu boxes in a Text node.
Already in-tree, and the four ESP-IDF gaps
Already present (verified against 223be88 / v0.13.0): the native-text contract
(Ui::set_text_measure, TEXT_RUN op 9 with the UTF-8 run packed into the draw words,
documented in docs/BACKENDS.md; gpui is the reference backend);
engine/core/src/raster.rs documents TEXT_RUN as a known op to skip ("Skipped, not a
stop"); enhances: ["text.layout.native"] as the app-side opt-in.
Each of these was a local patch in our fork to make the host path work end-to-end:
set_text_measureisn't exported through the C shim. The engine has it, but
hosts/esp-idf/native/ui-coreand the generatednative_ui.hdon't surface it — a C host
cannot install a provider at all. (We added FFIpocketjs_native_ui_set_text_measure+ a
C wrapper.)engine/backends/rgb565aborts onTEXT_RUN. The raster loop has no skip-by-length
branch; it falls into_ => return Noneand the whole strip render fails the first time a
TEXT_RUNappears. Core's software rasterizer skips it by design — the backend should do
the same (8 header words + ceil(bytes/4)).- No way to skip
ui:font.*atlas registration when a host provider is active.
feed_pak()registers the payloads unconditionally; with a host provider they are dead
weight (the 3.3 MB above). We added an explicitnative_textfield on the binding config
that suppressesPOCKETJS_UI_ASSET_FONTregistration (STYLES/IMAGE/SPRITE untouched) and
logs what was skipped. - Tracked runs are excluded from the native path (upstream gate:
native_active() && tracking == 0.0). Once atlases are skipped,tracking-widetext has
neither atlas nor provider — measurement returns (0,0), the box collapses, and whole
paragraphs vanish (our home screen's clock and header aretracking-wide). We pack
tracking intoTEXT_RUNword1 bits 16–31 as i16, 1/64 logical px (upstream: reserved 0),
so measurement and painting keep coming from the same provider; transform subtrees still
use the atlas pair.
Proof of concept (fork; ESP32-P4, 1024×600, raster density 2, 16 MB PSRAM)
Glyph service:
- FreeType from the registry component
espressif/freetype^2.14.3, memory interface
only: mmap the rawfontpartition →FT_New_Memory_Face— zero copy, the filesystem is
never touched. (Our earlier LVGL attempt failed exactly on the prebuilt component's stdio
ftsystem; this path doesn't use it.) - LRU cache keyed by (face, px, gid, load flags): hash buckets + LRU chain, bitmaps and
tables in PSRAM, 1 MiB bitmap budget / 512 entries (either bound evicts). Hits make zero
FreeType calls. Coverage is normalized to 8-bit gray on insert (MONO expanded) — callers
see one format.
Provider over the contract:
- measure = FreeType advances per line, same line-height/style math as the engine;
- paint = walk
TEXT_RUN(8-word header + UTF-8 payload) maintaining the SCISSOR stack,
align/box_w line placement, 26.6 pen accumulation, baseline from FT ascender/descender,
coverage alpha-blended into the RGB565 frame.
Measured: paragraph paint 14.9 ms → 2.4 ms with the cache; steady hit rate 99.7% (96% from
the first 10 s window) at 20–390 KiB of the 1 MiB budget; end-to-end probe 112k measures /
0 failures, 108k runs / 194k glyphs painted / 0 missing, ~1.7 µs/frame all-hit. FreeType
links +411 KiB.
Two notes for a reference provider:
esp_partition_mmapmust run from a task with an internal-RAM stack (PSRAM-stack tasks
trip the cache-disabled stack-sanity assert), so font mounting belongs on the boot path.- A FreeType-free floor exists: LVGL's TinyTTF memory API also works mmap'd and is ~400 KiB
smaller, but it takes TrueType outlines only (we convert the CFF subset to quadratic,
×1.35 file size) and has a fixed 256-glyph cache.
Our PoC board is an LCD; e-ink panel plumbing (1-bit / 4-gray conversion, partial-refresh
policy) stays panel-specific and outside this contract. The contract-side needs — grayscale
coverage, MONO target, arbitrary sizes, cheap repaint — are exactly what the provider
supplies.
Proposal
- Export the native-text contract on the ESP-IDF host —
pocketjs_ui_core_set_text_measure
(FFI + generated header), or a small "host text provider" component interface
(measure + optional paint) if that shape is preferred. - Fix
engine/backends/rgb565to skipTEXT_RUNby length (parity with core raster). - Native-text mode skips atlas registration:
feed_pakdoesn't registerui:font.*
when the host declares it has a provider — ideally tied to the existing
enhances: ["text.layout.native"]declaration (host admission already gates
capabilities), so the packer can also stop baking. - Tracked runs join the native path: tracking in
TEXT_RUNword1 bits 16–31 (i16,
1/64 logical px); transforms keep the atlas pair. - A reference ESP-IDF host text provider beside the existing boot/runner components:
FreeType, mmap face, PSRAM LRU cache, Kconfig for the cache budget and default face
sizes; plus subsetting notes (we emit the Source Han Sans SC GB2312-L1 subset with a
fontTools script;otf2ttfis only needed for the TinyTTF route). - Font declaration for native-text apps: today fonts are the atlas assets. Options: a
raw-TTFui:fontfile.*asset in the pak (host admission already exists), and/or a
host-provided default face perfont-family; app-facingfont-*classes and slots stay
unchanged.
PoC patch set: a downstream private fork of this tree at 223be88; we can share the
patches and the deviation list on request.
Non-goals
- Complex shaping (Arabic/Devanagari), bidi, variable fonts, word wrap (engine v1 breaks on
\nonly) — unchanged here. - Replacing #442's desktop/WASM/PSP worker geometry service. This is the embedded tier.
Open questions
- Reuse #442's service abstractions (bitmap keys, leases, budget accounting) for one shared
code path, or keep a smaller embedded tier (advance-based, no shaping)? Our numbers say
the small tier suffices for CJK UI text. - Cache budget defaults, and the floor for PSRAM-less boards; which tables stay in internal
RAM. - Render-target hint in the contract: 1-bit e-ink wants
FT_LOAD_TARGET_MONO(no AA),
grayscale wants gray8 — provider config or contract field? - Subsetting ownership: upstream tooling + charset presets (GB2312 L1 ≈ 3,755 hanzi ≈ 2.3 MB
TTF; L2 ≈ 6,763), or app responsibility?
- Dominant language
- TypeScript
- Stars
- 1.6k
- Forks
- 88
- PR merge metrics
- No merged PRs in 30d
Getting set up
We have not checked this project's setup files yet. Start from its README, and see our first-contribution guide for the general steps.
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from pocket-stack/pocketjs
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
pocket-stack/pocketjs#729 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
pocket-stack/pocketjs#725 ·
-
enhancement
Difficulty 1/5 Under an hour Newbie friendliness 85/100
pocket-stack/pocketjs#724 ·
-
enhancement
Difficulty 4/5 3-5 days Newbie friendliness 62/100
pocket-stack/pocketjs#728 ·
-
enhancement
Difficulty 5/5 Over a week Newbie friendliness 30/100
pocket-stack/pocketjs#727 ·
All issues in pocket-stack/pocketjs
Similar issues
-
dream report: normalizeMutation discards ok/status, consumers can only string-match the summaryOpen
Difficulty 2/5 1-3 hours Newbie friendliness 66/100
thecodacus/understory#61 ·
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
Termix-SSH/Plugin-Hosts-Config#14 · 1 comment ·
Maintainers usually reply within 4 days
-
ai-generated priority: p2 size: s type: bug
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
JonathanTSilva/jts-website#92 ·
Maintainers usually reply within 1 day
-
Toast announcer reads a doubled period when a toast's title ends in punctuationPossibly taken A pull request linked to this issue is open or already merged. Open
Difficulty 1/5 Under an hour Newbie friendliness 88/100
vercel-labs/ui#999 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 65/100
ehmpathy/rhachet-roles-ehmpathy#1443 ·
Maintainers usually reply within 1 day