facebookresearch/fairseq

Evaluating TTS using STT

Open

#5,629 opened on Jul 19, 2025

View on GitHub
 (0 comments) (0 reactions) (0 assignees)Python (29,107 stars) (6,224 forks)batch import
enhancementhelp wantedneeds triage

Description

I tried to get a text back from the speech I generated using this model, but the text was a bit of. So maybe having some type of test could help while improving the model.

Basically, "some text" -> speech output -> speech input -> text output -> check if text output is "some text"

Contributor guide

Tech stack
python
Domain
machine learningaitesting
Issue type
test
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
1-3 hours
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
fresh
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
unclear
Prerequisites
basic knowledge of fairseqfamiliarity with TTS and STT models
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
30
Research direction
The issue proposes a cyclical evaluation for TTS models: convert text to speech using the model, then transcribe the speech back to text using an STT model, and compare the output text to the input. To implement this, a contributor should first explore if fairseq already has any evaluation scripts or example notebooks. They could also look into integrating an off the shelf STT model like wav2vec 2.0 or Whisper. The main steps would be: (1) generate speech from a set of test texts using the existing TTS model, (2) pass the audio through an STT model, (3) compute metrics like WER or CER. The contributor should also check existing issues or PRs for similar work.