Map: decouple speech synthesis from synchronization units

未关闭
#320 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看

还没有人认领这个 Issue。

评估

难度
5/5
预计耗时
一周以上
新手友好度
25/100
Issue 类型
功能
描述清晰度
描述清楚
活跃度
活跃
技术栈
typescript

调研方向

Read docs/architecture/conversion.md and the glossary first, then examine narration synthesis in libs/audiobook-production, synchronization units in libs/narration-document-creation, and player position mapping in apps/web-app/src/app/player/playback-session.ts. Compare larger narration chunks with the overlap-and-cut baseline, evaluating continuity by listening on a fixed page and investigating placement, language coverage, cost, and domain-model impact. Done means a reasoned go/no-go decision; if go, leave build design to a follow-up map or spec.

由索引模型根据 Issue 内容生成。

描述

wayfinder:map

Destination

A go/no-go decision on decoupling what Cup sends to speech synthesis from the synchronization units a listener navigates: narration chunks sized for speech synthesis (larger, spanning several synchronization units, as large as listening shows one request stays free of a fresh start), each chunk's audio transcribed by speech recognition with word timing, and the recognized words mapped back onto the narration document by a model, an AI step rather than deterministic alignment code, to place every synchronization unit inside the chunk's audio. The map ends when that decision is made with its reasons (cost against speech policy 6, placement accuracy, narration language coverage, and what the change would cost in the domain model); a "go" hands off to a new map or spec that designs the build.

Notes

  • Origin: the Out of scope section of Map: narration flows across segment boundaries, which ruled this idea out of that effort and chose overlap and cut. No Linear issue tracks it yet.
  • Baseline: speech policy 6, one request per narration chunk with a fresh start at every segment boundary. Overlap and cut (speech policy 7, #336) was deployed on 2026-11-13 and withdrawn the same day in #352 (ADR 0028); Linear CUP-58's "Attempt 1" section records what it learned. What still holds: speaking the previous sentence in front removed the fresh start at 9 of 10 boundaries, against 4 of 10 for policy 6 (#291), which is the continuity this map's chunks must reach without a cut. What failed is the cut point found by deterministic alignment of Whisper's word timing (#316, #317).
  • Operator's direction (2026-11-13): decoupled synthesis replaces overlap and cut rather than building on it, and a model maps the transcription back onto the synchronization units instead of deterministic code.
  • Goals, in the operator's words: lower synthesis cost (fewer requests and fewer segment boundaries), and decoupling synthesis requests from navigation granularity. Seeking or highlighting by sentence or word is a later increment, out of scope here.
  • Concerns to face early:
    • Whisper's word timing is least reliable at a unit's first word. On 2026-11-13 Whisper often started a chunk's first word early, stretching it back over the pause in front of it, so a unit start taken from Whisper's word start lands inside the previous sentence. That is what withdrew overlap and cut: 11 live re-runs of one production boundary gave 2 cut-check fallbacks and 3 accepted cuts inside the overlap. Judge placement at unit starts against the audio's real pauses. The 14 recorded Gemini takes with verified pauses at 424d7b2e libs/audiobook-production/test/evals/cases/ are ready-made fixtures, and recognize-speech.ts at that commit is a working Whisper call through the Workers AI binding.
    • Gemini 3.8 TTS reportedly has fresh starts inside one long request, tied to internal pieces of about 290 words; one probe of 1,003 words showed speech-level plateaus about every 300 words but was never listened to (How do others make long-form Gemini narration flow across segment boundaries?). The chunk size must come from listening.
    • Today each synchronization unit has its own audio segment and starts in the narration track at the sum of the preceding segments' durations; the audio segment is also the unit of storage, settlement and duration reservation (ADR 0022). Decoupling changes all of that, plus listening positions and failed-segment marking.
    • Whisper covers 68 of Cup's 99 narration languages (71 together with Gemini 3.5 Transcribe); 28 are in neither list. The operator wants a fallback placement explored and other speech recognition models researched: one may cover enough that the intersection of Gemini TTS languages and its languages is good enough.
    • The whole document is synthesized and the track is playable only once every segment has audio (ADR 0008, ADR 0011; ADR 0006 is superseded), so larger chunks lengthen the slowest request and widen a failed segment's retry.
  • Domain: narration synthesis in libs/audiobook-production; the narration document and synchronization units in libs/narration-document-creation; the player's position mapping in apps/web-app/src/app/player/playback-session.ts. Read Conversion and the glossary first.
  • Vocabulary: use the glossary terms (narration block, narration chunk, audio segment, synchronization unit, segment, narration track, speech policy, segment boundary, fresh start). Say speech recognition, cut point and cut check, never "aligner" or "spoken-text check"; spoken text is only what the provider reports. Under the glossary a narration chunk is already "a single narration-synthesis input", so a larger synthesis input is a larger narration chunk, not a new concept; whether that definition holds is part of the domain-model ticket.
  • Skills: grilling and domain-modeling for HITL tickets, prototype for prototype tickets, research for research tickets. Judge continuity by ear on a fixed listening page, as the earlier map did, not by an automated metric.
  • Paid probes are fine at the scale of earlier research (a few US dollars); report the spend.

Decisions so far

  • Which speech recognition models with word timing cover Cup's narration languages?: by documented lists hosted APIs reach at most 78 of 99 narration languages (Chirp 2 alone 74, with Whisper 76) and self-hosted Omnilingual ASR 95 without upstream timestamps; but Whisper matched to the narration text by characters placed 36 of 36 sentence starts in 9 unlisted languages, so test that on a harder sample before adding a provider.
  • How can synchronization units be placed in audio without speech recognition?: forced alignment of the known text with Omnilingual ASR CTC 300M (Apache-2.0, 25 of the 28 uncovered languages; MMS is CC-BY-NC) matched Whisper on 114/116 cuts and agreed at 38/39 unit starts in 7 uncovered languages, so run it in a Cloudflare Container with a cut check; pause matching alone fails in real pauses; per-unit synthesis stays the fallback for Kongo, Santali, South Azerbaijani and failed checks.

Not yet specified

  • Where a larger narration chunk may end: only between narration blocks, also inside an oversized block, and whether headings or the source title and source subtitle may share a chunk with the blocks around them.
  • What happens when placement fails for one synchronization unit inside a chunk (a unit start that speech recognition cannot place, or places outside a pause): retry the whole chunk, split it, or fall back to per-unit synthesis. The fallback rules of What happens when a cut point or its cut check fails? belonged to the withdrawn overlap and cut and no longer constrain this.
  • Whether a shared seed keeps a retried large chunk's take, and whether its unit placement still matches after a retry.
  • Whether the narration track, the listener API's manifest or the mobile apps need anything beyond new unit start times.
  • If the decision is "go": how the build is staged (a new speech policy, which conversions get it, rollout) — for the follow-up map or spec, not this one.

Out of scope

  • Seeking or highlighting by sentence or word: a later increment; this map may only note what recognized word timing would have to be kept for it.
  • Designing the build itself: the destination is a go/no-go decision; a "go" opens its own map or spec.
  • The ElevenLabs speech configuration, and re-synthesizing audiobooks that already exist.
主要语言
TypeScript
星标
1
派生
0
PR 合并指标
30 天内没有已合并 PR

环境准备

这个项目没有提供开发容器、Dockerfile 或贡献指南,环境需要你自己搭建:先看它的 README,通用步骤见我们的新手贡献指南。

从这里开始

  1. 先读完整个 Issue,再读项目的贡献指南。
  2. 在 Issue 下留言说明你要接手 —— 这能避免两个人做同样的事。
  3. Fork 仓库,在一个分支上完成修改。
  4. 提交 Pull Request,并在描述里引用这个 Issue 编号。

patricktree/cup 的其他 Issue

查看 patricktree/cup 的全部 Issue

相似的 Issue

更多 TypeScript Issue

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。