Map: decouple speech synthesis from synchronization units
Ninguém assumiu esta issue ainda.
Avaliação
- Dificuldade
- 5/5
- Tempo estimado
- Mais de uma semana
- Facilidade para iniciantes
- 25/100
- Tipo de issue
- Funcionalidade
- Clareza
- Claramente especificada
- Status de atividade
- Ativa
- Stack de tecnologia
- typescript
- Domínio
- audio-video-rtc, backend
Direção de pesquisa
Read docs/architecture/conversion.md and the glossary first, then examine narration synthesis in libs/audiobook-production, synchronization units in libs/narration-document-creation, and player position mapping in apps/web-app/src/app/player/playback-session.ts. Compare larger narration chunks with the overlap-and-cut baseline, evaluating continuity by listening on a fixed page and investigating placement, language coverage, cost, and domain-model impact. Done means a reasoned go/no-go decision; if go, leave build design to a follow-up map or spec.
Escrita pelo modelo de indexação a partir do texto da issue.
Descrição
Destination
A go/no-go decision on decoupling what Cup sends to speech synthesis from the synchronization units a listener navigates: narration chunks sized for speech synthesis (larger, spanning several synchronization units, as large as listening shows one request stays free of a fresh start), each chunk's audio transcribed by speech recognition with word timing, and the recognized words mapped back onto the narration document by a model, an AI step rather than deterministic alignment code, to place every synchronization unit inside the chunk's audio. The map ends when that decision is made with its reasons (cost against speech policy 6, placement accuracy, narration language coverage, and what the change would cost in the domain model); a "go" hands off to a new map or spec that designs the build.
Notes
- Origin: the Out of scope section of Map: narration flows across segment boundaries, which ruled this idea out of that effort and chose overlap and cut. No Linear issue tracks it yet.
- Baseline: speech policy 6, one request per narration chunk with a fresh start at every segment boundary. Overlap and cut (speech policy 7, #336) was deployed on 2026-11-13 and withdrawn the same day in #352 (ADR 0028); Linear CUP-58's "Attempt 1" section records what it learned. What still holds: speaking the previous sentence in front removed the fresh start at 9 of 10 boundaries, against 4 of 10 for policy 6 (#291), which is the continuity this map's chunks must reach without a cut. What failed is the cut point found by deterministic alignment of Whisper's word timing (#316, #317).
- Operator's direction (2026-11-13): decoupled synthesis replaces overlap and cut rather than building on it, and a model maps the transcription back onto the synchronization units instead of deterministic code.
- Goals, in the operator's words: lower synthesis cost (fewer requests and fewer segment boundaries), and decoupling synthesis requests from navigation granularity. Seeking or highlighting by sentence or word is a later increment, out of scope here.
- Concerns to face early:
- Whisper's word timing is least reliable at a unit's first word. On 2026-11-13 Whisper often started a chunk's first word early, stretching it back over the pause in front of it, so a unit start taken from Whisper's word start lands inside the previous sentence. That is what withdrew overlap and cut: 11 live re-runs of one production boundary gave 2 cut-check fallbacks and 3 accepted cuts inside the overlap. Judge placement at unit starts against the audio's real pauses. The 14 recorded Gemini takes with verified pauses at
424d7b2elibs/audiobook-production/test/evals/cases/are ready-made fixtures, andrecognize-speech.tsat that commit is a working Whisper call through the Workers AI binding. - Gemini 3.8 TTS reportedly has fresh starts inside one long request, tied to internal pieces of about 290 words; one probe of 1,003 words showed speech-level plateaus about every 300 words but was never listened to (How do others make long-form Gemini narration flow across segment boundaries?). The chunk size must come from listening.
- Today each synchronization unit has its own audio segment and starts in the narration track at the sum of the preceding segments' durations; the audio segment is also the unit of storage, settlement and duration reservation (ADR 0022). Decoupling changes all of that, plus listening positions and failed-segment marking.
- Whisper covers 68 of Cup's 99 narration languages (71 together with Gemini 3.5 Transcribe); 28 are in neither list. The operator wants a fallback placement explored and other speech recognition models researched: one may cover enough that the intersection of Gemini TTS languages and its languages is good enough.
- The whole document is synthesized and the track is playable only once every segment has audio (ADR 0008, ADR 0011; ADR 0006 is superseded), so larger chunks lengthen the slowest request and widen a failed segment's retry.
- Whisper's word timing is least reliable at a unit's first word. On 2026-11-13 Whisper often started a chunk's first word early, stretching it back over the pause in front of it, so a unit start taken from Whisper's word start lands inside the previous sentence. That is what withdrew overlap and cut: 11 live re-runs of one production boundary gave 2 cut-check fallbacks and 3 accepted cuts inside the overlap. Judge placement at unit starts against the audio's real pauses. The 14 recorded Gemini takes with verified pauses at
- Domain: narration synthesis in
libs/audiobook-production; the narration document and synchronization units inlibs/narration-document-creation; the player's position mapping inapps/web-app/src/app/player/playback-session.ts. Read Conversion and the glossary first. - Vocabulary: use the glossary terms (narration block, narration chunk, audio segment, synchronization unit, segment, narration track, speech policy, segment boundary, fresh start). Say speech recognition, cut point and cut check, never "aligner" or "spoken-text check"; spoken text is only what the provider reports. Under the glossary a narration chunk is already "a single narration-synthesis input", so a larger synthesis input is a larger narration chunk, not a new concept; whether that definition holds is part of the domain-model ticket.
- Skills:
grillinganddomain-modelingfor HITL tickets,prototypefor prototype tickets,researchfor research tickets. Judge continuity by ear on a fixed listening page, as the earlier map did, not by an automated metric. - Paid probes are fine at the scale of earlier research (a few US dollars); report the spend.
Decisions so far
- Which speech recognition models with word timing cover Cup's narration languages?: by documented lists hosted APIs reach at most 78 of 99 narration languages (Chirp 2 alone 74, with Whisper 76) and self-hosted Omnilingual ASR 95 without upstream timestamps; but Whisper matched to the narration text by characters placed 36 of 36 sentence starts in 9 unlisted languages, so test that on a harder sample before adding a provider.
- How can synchronization units be placed in audio without speech recognition?: forced alignment of the known text with Omnilingual ASR CTC 300M (Apache-2.0, 25 of the 28 uncovered languages; MMS is CC-BY-NC) matched Whisper on 114/116 cuts and agreed at 38/39 unit starts in 7 uncovered languages, so run it in a Cloudflare Container with a cut check; pause matching alone fails in real pauses; per-unit synthesis stays the fallback for Kongo, Santali, South Azerbaijani and failed checks.
Not yet specified
- Where a larger narration chunk may end: only between narration blocks, also inside an oversized block, and whether headings or the source title and source subtitle may share a chunk with the blocks around them.
- What happens when placement fails for one synchronization unit inside a chunk (a unit start that speech recognition cannot place, or places outside a pause): retry the whole chunk, split it, or fall back to per-unit synthesis. The fallback rules of What happens when a cut point or its cut check fails? belonged to the withdrawn overlap and cut and no longer constrain this.
- Whether a shared seed keeps a retried large chunk's take, and whether its unit placement still matches after a retry.
- Whether the narration track, the listener API's manifest or the mobile apps need anything beyond new unit start times.
- If the decision is "go": how the build is staged (a new speech policy, which conversions get it, rollout) — for the follow-up map or spec, not this one.
Out of scope
- Seeking or highlighting by sentence or word: a later increment; this map may only note what recognized word timing would have to be kept for it.
- Designing the build itself: the destination is a go/no-go decision; a "go" opens its own map or spec.
- The ElevenLabs speech configuration, and re-synthesizing audiobooks that already exist.
- Linguagem predominante
- TypeScript
- Estrelas
- 1
- Forks
- 0
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Preparar o ambiente
Este projeto não oferece contêiner de desenvolvimento, Dockerfile nem guia de contribuição, então a configuração fica por sua conta: comece pelo README e veja nosso guia da primeira contribuição para os passos gerais.
Primeiros passos
- Leia a issue inteira e depois o guia de contribuição do projeto.
- Comente na issue dizendo que vai assumir — evita que duas pessoas façam o mesmo trabalho.
- Faça um fork do repositório e trabalhe em uma branch.
- Abra um pull request que referencie o número da issue.
Mais de patricktree/cup
-
wayfinder:grilling
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 72/100
patricktree/cup#484 ·
-
wayfinder:grilling
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 48/100
patricktree/cup#487 ·
-
wayfinder:grilling
Dificuldade 5/5 Mais de uma semana Facilidade para iniciantes 25/100
patricktree/cup#486 ·
-
wayfinder:grilling
Dificuldade 5/5 Mais de uma semana Facilidade para iniciantes 25/100
patricktree/cup#485 ·
-
wayfinder:grilling
Dificuldade 5/5 Mais de uma semana Facilidade para iniciantes 25/100
patricktree/cup#483 ·
Todas as issues de patricktree/cup
Issues semelhantes
-
bug
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 68/100
Mantenedores costumam responder em até 2 dias
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 78/100
Mantenedores costumam responder em até 1 dia
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 78/100
inabagumi/shinju-date#7406 ·
Mantenedores costumam responder em até 1 dia
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 76/100
umami-software/umami#4787 ·
Mantenedores costumam responder em até 3 dias
-
Dificuldade 2/5 1-3 horas Facilidade para iniciantes 82/100
Mantenedores costumam responder em até 1 dia