RVC-Project/Retrieval-based-Voice-Conversion-WebUI

African Language Phonemes

Open

#2057 opened on May 14, 2024

View on GitHub
 (3 comments) (0 reactions) (0 assignees)Python (18,427 stars) (2,849 forks)batch import
bugdocumentationfollowing uphelp wanted

Description

Hi!

I'm trying to train RVC models for a range of dialects in my country. I live in South Africa.

I'm struggling with click-consonant phonemes.

These African language sounds are not understood as phonemic by the model, so transforming voice-to-voice using recordings of African language speakers results in dropped phonemes or incorrect phonemes.

Click consonants are not reliably preserved in the output and are even occasionally replaced by other consonants ("c", "ck", "k").

Is there anyway for me to overcome this roadblock? Do I need to include more of the click-consonants in my training data?

Click Consonant Transformation.zip

Contributor guide

Tech stack
pythonpytorch
Domain
machine learningai
Issue type
bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
needs investigation
Prerequisites
phoneticsmachine learning basicsaudio processing
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
20
Research direction
The issue reports that click consonant phonemes are dropped or replaced during voice conversion. Examine the model's phoneme dictionary and acoustic model implementation, likely in the model/ or voice conversion/ directories. Review the attached zip file with example audio to understand the failure modes. Check if the training data includes click consonants; if not, consider augmenting the dataset. Engage with maintainers on potential modifications to the phoneme set or preprocessing pipeline. See community comments for any suggested workarounds.