How accurately does speech recognition place synchronization units inside a larger narration chunk?
まだ誰も着手していません。
評価
- 難易度
- 5/5
- 見積もり時間
- 1週間以上
- 初心者へのやさしさ
- 35/100
- issue の種類
- 機能追加
- 明瞭さ
- 明確に書かれている
- 活発さ
- 活発
- 技術スタック
- typescript
調査の方向性
Start with the prototype's word alignment of recognized words to the narration document's words, then review issues #316, #322, #323, and #324 for prior measurements and the chunk size. Test Whisper large v3 turbo and, if available, the other candidate across the specified languages and fixture articles; measure placement accuracy, chunk splitting effects, cost, and latency, settling disagreements by ear. Report measured versus unverified claims and propose an acceptance rule, including whether one method covers every language.
索引モデルが issue の本文から書いたものです。
説明
Question
How accurately does speech recognition with word timing place each synchronization unit's start inside a larger narration chunk's audio?
Overlap and cut needed one cut point per audio segment, and Whisper large v3 turbo placed 116 of 116 researched cuts (Which aligner finds overlap-and-cut points reliably?). In production that did not hold, and overlap and cut was withdrawn on 2026-11-13 (ADR 0028): Whisper often started a chunk's first word early, stretching it back over the pause in front of it, so the deterministic search never saw the real pause. In 11 live re-runs of one production boundary, 2 cut checks failed and 3 accepted a cut inside the previous sentence. A unit start is exactly that spot, so this question must not take Whisper's word starts as the unit starts. Decoupling needs one placement per synchronization unit inside a chunk of the size How long can one Gemini 3.8 TTS request get before a fresh start inside it? settles, so errors add up per unit, and recognition drift over several minutes of audio may grow. Using that chunk size, the fixture articles in English, German and French, compare two placements: the prototype's deterministic word alignment of recognized words to the narration document's words, and a model-based placement, an AI step that maps the recognized words, their timing and the audio's pauses onto the synchronization units, which is the operator's preferred direction (2026-11-13). For each, measure:
- how often a unit start lands in the real pause before its first word, as the audio's silence shows it rather than Whisper's word start, versus inside a word or in the wrong pause, including at headings, the source title and source subtitle, list items, table cells, numbers, abbreviations and names;
- whether long chunks need the audio split before recognition, and what that does to word timing;
- speech recognition cost and latency per chunk against synthesis.
Beyond English, German and French, take a harder sample (headings, numbers, table cells, short units) in several of the 28 narration languages Whisper does not list, and compare two placements there: Whisper matched to the narration text letter by letter, which placed 36 of 36 sentence starts on an easy sample (Which speech recognition models with word timing cover Cup's narration languages?), and forced alignment with Omnilingual ASR CTC 300M, which agreed with Whisper at 38 of 39 unit starts and missed only at numbers (How can synchronization units be placed in audio without speech recognition?). Where the two disagree, settle it by ear. The answer should say whether one method can serve every narration language it covers, so the uncovered languages need no second pipeline.
Use Whisper large v3 turbo, plus the best other candidate from Which speech recognition models with word timing cover Cup's narration languages? if it is resolved by then. Propose an acceptance rule for a placement that also catches a placement inside the previous sentence, which overlap and cut's cut check accepted. The 14 recorded Gemini takes with verified pauses at 424d7b2e libs/audiobook-production/test/evals/cases/ are ready-made fixtures for unit starts at headings, bylines, numbers, table cells and quotes, and recognize-speech.ts at that commit is a working Whisper call through the Workers AI binding. Mark measured and unverified claims; light paid probes are fine.
Part of Map: decouple speech synthesis from synchronization units.
- 主要言語
- TypeScript
- スター
- 1
- フォーク
- 0
- PR マージ指標
- 30日以内にマージされた PR はありません
環境構築
このプロジェクトには開発コンテナ、Dockerfile、コントリビューションガイドがありません。まず README を読み、一般的な手順ははじめてのコントリビューションガイドを参照してください。
はじめの一歩
- issue を最後まで読み、次にプロジェクトのコントリビューションガイドを読みます。
- 着手することを issue にコメントします — 二人が同じ作業をするのを防げます。
- リポジトリをフォークし、ブランチを切って変更します。
- issue 番号を参照したプルリクエストを送ります。
patricktree/cup のほかの issue
-
wayfinder:grilling
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
patricktree/cup#484 ·
-
wayfinder:grilling
難易度 2/5 1〜3時間 初心者へのやさしさ 48/100
patricktree/cup#487 ·
-
wayfinder:grilling
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
patricktree/cup#486 ·
-
wayfinder:grilling
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
patricktree/cup#485 ·
-
wayfinder:grilling
難易度 5/5 1週間以上 初心者へのやさしさ 25/100
patricktree/cup#483 ·
patricktree/cup の issue をすべて見る
似ている issue
-
難易度 2/5 1〜3時間 初心者へのやさしさ 72/100
メンテナーはふだん 1 日以内に返信
-
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
vercel-labs/ui#865 ·
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 63/100
outline/outline#14638 · コメント 1 件 ·
メンテナーはふだん 1 日以内に返信
-
bot:ai-assisted component:platform-js status:untriaged
難易度 2/5 1〜3時間 初心者へのやさしさ 78/100
midnightntwrk/platform-js#69 ·
メンテナーはふだん 1 日以内に返信
-
bug
難易度 2/5 1〜3時間 初心者へのやさしさ 67/100
メンテナーはふだん 1 日以内に返信