Map: decouple speech synthesis from synchronization units
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 25/100
- issue の種類
- 機能追加
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- typescript
調査の方向性
Read docs/architecture/conversion.md and the glossary first, then examine narration synthesis in libs/audiobook-production, synchronization units in libs/narration-document-creation, and player position mapping in apps/web-app/src/app/player/playback-session.ts. Compare larger narration chunks with the overlap-and-cut baseline, evaluating continuity by listening on a fixed page and investigating placement, language coverage, cost, and domain-model impact. Done means a reasoned go/no-go decision; if go, leave build design to a follow-up map or spec.
索引モデルが issue の本文から書いたものです。
説明
Destination
A go/no-go decision on decoupling what Cup sends to speech synthesis from the synchronization units a listener navigates: narration chunks sized for speech synthesis (larger, spanning several synchronization units, as large as listening shows one request stays free of a fresh start), each chunk's audio transcribed by speech recognition with word timing, and the recognized words mapped back onto the narration document by a model, an AI step rather than deterministic alignment code, to place every synchronization unit inside the chunk's audio. The map ends when that decision is made with its reasons (cost against speech policy 6, placement accuracy, narration language coverage, and what the change would cost in the domain model); a "go" hands off to a new map or spec that designs the build.
Notes
- Origin: the Out of scope section of Map: narration flows across segment boundaries, which ruled this idea out of that effort and chose overlap and cut. No Linear issue tracks it yet.
- Baseline: speech policy 6, one request per narration chunk with a fresh start at every segment boundary. Overlap and cut (speech policy 7, #336) was deployed on 2026-11-13 and withdrawn the same day in #352 (ADR 0028); Linear CUP-58's "Attempt 1" section records what it learned. What still holds: speaking the previous sentence in front removed the fresh start at 9 of 10 boundaries, against 4 of 10 for policy 6 (#291), which is the continuity this map's chunks must reach without a cut. What failed is the cut point found by deterministic alignment of Whisper's word timing (#316, #317).
- Operator's direction (2026-11-13): decoupled synthesis replaces overlap and cut rather than building on it, and a model maps the transcription back onto the synchronization units instead of deterministic code.
- Goals, in the operator's words: lower synthesis cost (fewer requests and fewer segment boundaries), and decoupling synthesis requests from navigation granularity. Seeking or highlighting by sentence or word is a later increment, out of scope here.
- Concerns to face early:
- Whisper's word timing is least reliable at a unit's first word. On 2026-11-13 Whisper often started a chunk's first word early, stretching it back over the pause in front of it, so a unit start taken from Whisper's word start lands inside the previous sentence. That is what withdrew overlap and cut: 11 live re-runs of one production boundary gave 2 cut-check fallbacks and 3 accepted cuts inside the overlap. Judge placement at unit starts against the audio's real pauses. The 14 recorded Gemini takes with verified pauses at
424d7b2elibs/audiobook-production/test/evals/cases/are ready-made fixtures, andrecognize-speech.tsat that commit is a working Whisper call through the Workers AI binding. - Gemini 3.8 TTS reportedly has fresh starts inside one long request, tied to internal pieces of about 290 words; one probe of 1,003 words showed speech-level plateaus about every 300 words but was never listened to (How do others make long-form Gemini narration flow across segment boundaries?). The chunk size must come from listening.
- Today each synchronization unit has its own audio segment and starts in the narration track at the sum of the preceding segments' durations; the audio segment is also the unit of storage, settlement and duration reservation (ADR 0022). Decoupling changes all of that, plus listening positions and failed-segment marking.
- Whisper covers 68 of Cup's 99 narration languages (71 together with Gemini 3.5 Transcribe); 28 are in neither list. The operator wants a fallback placement explored and other speech recognition models researched: one may cover enough that the intersection of Gemini TTS languages and its languages is good enough.
- The whole document is synthesized and the track is playable only once every segment has audio (ADR 0008, ADR 0011; ADR 0006 is superseded), so larger chunks lengthen the slowest request and widen a failed segment's retry.
- Whisper's word timing is least reliable at a unit's first word. On 2026-11-13 Whisper often started a chunk's first word early, stretching it back over the pause in front of it, so a unit start taken from Whisper's word start lands inside the previous sentence. That is what withdrew overlap and cut: 11 live re-runs of one production boundary gave 2 cut-check fallbacks and 3 accepted cuts inside the overlap. Judge placement at unit starts against the audio's real pauses. The 14 recorded Gemini takes with verified pauses at
- Domain: narration synthesis in
libs/audiobook-production; the narration document and synchronization units inlibs/narration-document-creation; the player's position mapping inapps/web-app/src/app/player/playback-session.ts. Read Conversion and the glossary first. - Vocabulary: use the glossary terms (narration block, narration chunk, audio segment, synchronization unit, segment, narration track, speech policy, segment boundary, fresh start). Say speech recognition, cut point and cut check, never "aligner" or "spoken-text check"; spoken text is only what the provider reports. Under the glossary a narration chunk is already "a single narration-synthesis input", so a larger synthesis input is a larger narration chunk, not a new concept; whether that definition holds is part of the domain-model ticket.
- Skills:
grillinganddomain-modelingfor HITL tickets,prototypefor prototype tickets,researchfor research tickets. Judge continuity by ear on a fixed listening page, as the earlier map did, not by an automated metric. - Paid probes are fine at the scale of earlier research (a few US dollars); report the spend.
Decisions so far
- Which speech recognition models with word timing cover Cup's narration languages?: by documented lists hosted APIs reach at most 78 of 99 narration languages (Chirp 2 alone 74, with Whisper 76) and self-hosted Omnilingual ASR 95 without upstream timestamps; but Whisper matched to the narration text by characters placed 36 of 36 sentence starts in 9 unlisted languages, so test that on a harder sample before adding a provider.
- How can synchronization units be placed in audio without speech recognition?: forced alignment of the known text with Omnilingual ASR CTC 300M (Apache-2.0, 25 of the 28 uncovered languages; MMS is CC-BY-NC) matched Whisper on 114/116 cuts and agreed at 38/39 unit starts in 7 uncovered languages, so run it in a Cloudflare Container with a cut check; pause matching alone fails in real pauses; per-unit synthesis stays the fallback for Kongo, Santali, South Azerbaijani and failed checks.
Not yet specified
- Where a larger narration chunk may end: only between narration blocks, also inside an oversized block, and whether headings or the source title and source subtitle may share a chunk with the blocks around them.
- What happens when placement fails for one synchronization unit inside a chunk (a unit start that speech recognition cannot place, or places outside a pause): retry the whole chunk, split it, or fall back to per-unit synthesis. The fallback rules of What happens when a cut point or its cut check fails? belonged to the withdrawn overlap and cut and no longer constrain this.
- Whether a shared seed keeps a retried large chunk's take, and whether its unit placement still matches after a retry.
- Whether the narration track, the listener API's manifest or the mobile apps need anything beyond new unit start times.
- If the decision is "go": how the build is staged (a new speech policy, which conversions get it, rollout) — for the follow-up map or spec, not this one.
Out of scope
- Seeking or highlighting by sentence or word: a later increment; this map may only note what recognized word timing would have to be kept for it.
- Designing the build itself: the destination is a go/no-go decision; a "go" opens its own map or spec.
- The ElevenLabs speech configuration, and re-synthesizing audiobooks that already exist.
- 主要言語
- TypeScript
- スター
- 1
- フォーク
- 0
- PR マージ指標
- 30日以内にマージされた PR はありません
環境構築
このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
patricktree/cup のほかの issue
-
wayfinder:grilling
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
patricktree/cup#484 ·
-
wayfinder:grilling
難易度 2/5 1〜3時間 初心者へのやさしさ 48/100
patricktree/cup#487 ·
-
wayfinder:grilling
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
patricktree/cup#486 ·
-
wayfinder:grilling
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
patricktree/cup#485 ·
-
wayfinder:grilling
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
patricktree/cup#483 ·
patricktree/cup の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
vercel-labs/ui#865 ·
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 63/100
outline/outline#14638 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
bot:ai-assisted component:platform-js status:untriaged
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
midnightntwrk/platform-js#69 ·
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 67/100
メンテナーはふだん 1 日以内に返信