Make the Video Translator honest about voice cloning, lip-sync and non-English TTS

Open
  • Is something wrong?
#32 0 comments 0 reactions 0 assignees View on GitHub

Maintainers usually reply within 1 day

Nobody has claimed this yet.

Assessment

Difficulty
3/5
Estimated time
1-2 days
Newbie friendliness
72/100
Issue type
Bug
Clarity
Clearly specified
Activity status
Active
Tech stack
nextjs, playwright, typescript
Domain
audio-video-rtc, frontend, testing-qa

Research direction

Start with frontend/src/components/translate/VideoTranslator.tsx, frontend/src/engine/translation-pipeline.ts, and frontend/src/engine/f5tts-engine.ts, then follow CONTRIBUTING.md for setup. Run the relevant Playwright and unit tests to confirm the clone toggle, language selection, and TTS voice-selection behavior. Done means unsupported cloning is disabled with a reason, non-English text never reaches an English voice, Wav2Lip failures are surfaced, and an English-to-English run still produces an ffprobe-valid MP4.

Written by the indexing model from the issue text.

Description

engine help wanted ui
Context

The translator UI promises things the engines cannot do today. The "Clone original speaker voice" toggle says it uses F5-TTS, but cloneAndSpeak always throws, because F5 is quarantined under L4 (LICENSES.md). Non-English targets are voiced with English Kokoro voices (af_sky, bf_emma), even though the Kokoro front-end only supports en-US/en-GB (voice-presets.ts:24), so Spanish or Hindi text gets English pronunciation. Wav2Lip is quarantined too, and its errors are swallowed by a bare catch {}. The language list also uses emoji flags, which breaks the no-emoji brand rule.

Where
  • frontend/src/components/translate/VideoTranslator.tsx:11-21 (emoji flags, duplicate language list), :176, :186, :335-349
  • frontend/src/engine/translation-pipeline.ts:15-26, :241, :255-260 (voiceMap), :278-287 (swallowed Wav2Lip error)
  • frontend/src/engine/f5tts-engine.ts:116-124
What to do
  1. Disable the clone toggle with the real reason whenever f5tts.isModelDownloaded() is false.
  2. For targets Kokoro cannot voice, either block TTS with an explicit message or produce a translated-caption-only output. Never pass non-English text to an English voice.
  3. Reuse TRANSLATION_LANGUAGES in the component, use text or Lucide icons instead of emoji, and log or surface the Wav2Lip failure reason.
Acceptance criteria
  • An English → English run still produces an ffprobe-valid MP4 (attach it)
  • A Playwright spec asserts that the clone toggle is disabled and shows the reason
  • A non-English target never calls ttsManager.synthesize with an English voice (unit test)
Getting started

Read CONTRIBUTING.md for dev setup and the quality bar (no facades, evidence before done). Comment here to claim the issue.

Dominant language
TypeScript
Stars
0
Forks
3
Avg merge
1d 4h
Merged PRs (30d)
5

Getting set up

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from BeyondITL/contentine-create

All issues in BeyondITL/contentine-create

Similar issues

More TypeScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.