BENYO-S2ST-Corpus-1: English-Yorùbá Speech Translation

• BENYO-S2ST-Corpus-1 is a bilingual English-to-Yorùbá speech-to-speech corpus constructed via a hybrid pipeline that combines real Yorùbá recordings and synthetic English audio.
• It employs the AcoustAug framework to augment acoustic features, expanding 1,504 original samples to over 24,000 diverse audio pairs for scalable low-resource translation research.
• The corpus underpins reproducible end-to-end S2ST and TTS modeling, bridging resource gaps for African languages through cost-efficient, deterministic processing.

BENYO-S2ST-Corpus-1 is a bilingual English-to-Yorùbá speech-to-speech translation (S2ST) corpus designed to address the lack of S2ST datasets for high-resource to low-resource language pairs, specifically for English and Yorùbá. Leveraging a hybrid corpus creation pipeline that combines real-time Yorùbá speech recordings, synthetic English speech, and an acoustic data augmentation framework (AcoustAug), BENYO-S2ST-Corpus-1 enables scalable direct S2ST research and pretraining for low-resource African languages. The corpus provides over 24,000 audio-text pairs, facilitating end-to-end S2ST and text-to-speech (TTS) modeling, with datasets and proof-of-concept models released for reproducibility and further development (Adetiba et al., 12 Jul 2025).

1. Corpus Construction and Workflow

BENYO-S2ST-Corpus-1 employs a hybrid large-scale corpus construction pipeline, explicitly designed to reduce cost and manual effort for high-to-low resource S2ST pairs. The primary foundation is the YORULECT corpus, which contains 1,504 samples of real-time Standard Yorùbá (SY) speech and transcripts across religion, TED talks, and news domains. Since YORULECT does not include paired English audio, aligned English textual translations are converted to audio using a pre-trained neural TTS model, specifically Facebook MMS.

The workflow comprises sequential stages:

1. Data Acquisition: Download YORULECT SY wav files and corresponding English text transcripts.
2. Semi-Automated Preprocessing: Clean garbled Yorùbá transcripts using the Davlan/m2m-100_418M translation model in a human-in-the-loop pipeline, and resample all waveforms to 22.05 kHz.
3. Automated Speech Synthesis: Generate Standard English speech using Facebook MMS TTS at 16 kHz, then upsample to 22.05 kHz.
4. Audio Augmentation (AcoustAug): Apply augmentation based on controlled manipulations of duration, pitch, and amplitude.
5. Pairing & Archiving: Maintain one-to-one sample correspondence via consistent filenames, constructing a metadata CSV for downstream processing:

   audio_eng_filename, transcript_eng, audio_yor_filename, transcript_yor

6. Distribution: Publish the finalized corpus on HuggingFace.

Filtration and alignment are script-driven and deterministic due to the preserved filename schema inherited from YORULECT (e.g., recorder_2023-04-10_10-43-10_631976_raw.wav).

2. Audio Augmentation and Expansion: AcoustAug

The AcoustAug algorithm systematically augments each original sample, expanding dataset breadth by varying three latent acoustic features independently: duration (time-stretch), fundamental frequency (F₀/pitch), and amplitude (volume). For each augmentation dimension:

• Duration utilizes time-stretch factors α ∈ {0.9, 1.1}, modifying the waveform according to $x_\alpha(t) = x(t/\alpha)$.
• Pitch shifts fundamental frequency by β ∈ {0.95, 1.05}, $F_{0,\textrm{out}} = F_{0,\textrm{in}} \cdot \beta$.
• Amplitude applies gain γ per $$A_{\textrm{out}}(n) = A_{\textrm{in}}(n) \cdot 10^{\gamma/20}$$ with γ ∈ {−5, 5, 10} dB.

Augmentation is performed as:

• 1 original + 2 for duration + 2 for pitch + 3 for amplitude = 8 versions per sample.
• Each language yields 12,032 augmented examples from the 1,504 original samples, resulting in 24,064 total corpus samples (41.20 hours).

Pseudocode for AcoustAug:

function AugmentAudio(input_folder, output_folder, aug_type, sample_rate=22050): ensure output_folder exists for each file.wav in input_folder: audio, sr = load_wav(file, sr=None) audio_22k = resample(audio, orig_sr=sr, new_sr=sample_rate) if aug_type == "speed": for α in [0.9, 1.1]: y = time_stretch(audio_22k, rate=α) save_wav(y, sample_rate, output_folder/name+"_speed_"+α+".wav") elif aug_type == "pitch": for β in [0.95, 1.05]: y = pitch_shift(audio_22k, n_steps=log2(β)*12) save_wav(y, sample_rate, output_folder/name+"_pitch_"+β+".wav") elif aug_type == "volume": for γ in [-5, 5, 10]: y = audio_22k * (10**(γ/20)) save_wav(y, sample_rate, output_folder/name+"_vol_"+γ+".wav")

(Adetiba et al., 12 Jul 2025)

3. Corpus Statistics and Characteristics

BENYO-S2ST-Corpus-1 contains the following per-language statistics:

Language	Samples	Total Duration (h)	Min/Max/Average Length (s)
English	12,032	17.81	1.12 / 14.85 / 5.33
Yorùbá	12,032	23.39	1.13 / 16.62 / 7.00

Grand total: 24,064 samples, 41.20 hours.

Domains include religion, TED, and news, reflecting the YORULECT diversity. Each language is single-speaker; the dataset's base (YORULECT) manages available accent and dialect variations. (Adetiba et al., 12 Jul 2025)

4. TTS Model Implementation and Evaluation: YoruTTS-0.5

A proof-of-concept Yoruba TTS system, YoruTTS-0.5, was constructed using the Coqui TTS VITS (Variational Inference Text-to-Speech) architecture:

• Pipeline: Text normalization → G2P conversion → phoneme embedding (UTF-8 Yorùbá phonemes) → VITS model (conditional VAE, normalizing flows, HiFi-GAN vocoder) → waveform output.
• Core Loss: Evidence Lower Bound (ELBO):

$$\log p(x|c) \geq \mathbb{E}_{q(z|x)}[\log p(x|z)] - D_{KL}(q(z|x) \| p(z|c))$$

• Key hyperparameters:
  • batch_size=32, eval_batch_size=16
  • epochs=1,000, mixed_precision=True
  • sample_rate=22,050Hz, n_mels=80, fft_size=1024, hop_length=256
• Objective Evaluation (F₀ RMSE):

$F₀_{RMSE} = \sqrt{\frac{1}{N} \sum_{i=1}^N (F₀_{pred}(i) - F₀_{ref}(i))^2}$

Achieved 63.54 Hz after 1,000 epochs (down from 72.85 Hz at 540 epochs), indicating moderate similarity in pitch with reference audio. All major loss components (generator, mel-reconstruction, duration, KL divergence, feature matching) converged within 200–300 epochs with minor improvements up to 1,000 epochs. Subjective listening suggested moderate naturalness and intelligibility. (Adetiba et al., 12 Jul 2025)

5. Reproducibility, Extensibility, and Application to Other Language Pairs

All scripts, code, and pretrained models are openly accessible:

• Download corpus: git clone https://huggingface.co/aspmirlab/BENYO-S2ST-Corpus-1
• Preprocessing (e.g., resampling, transcript cleaning) uses standard Python libraries such as torchaudio, soundfile, and HuggingFace transformers for the m2m-100_418M model.

Generalization for other language pairs entails:

1. Sourcing or recording ~1,000–2,000 samples in the target low-resource language.
2. Cleaning and correcting text; synthesizing high-resource language audio via a robust multilingual TTS.
3. Uniform resampling (e.g., 22.05 kHz).
4. Applying AcoustAug, with language-appropriate factors for pitch and duration.
5. Pairwise filename alignment and metadata archiving.
6. Adapting the Coqui-VITS pipeline with appropriate phoneme and audio configs; training for 1,000–2,000 epochs using mixed precision.

This methodology enables scalable dataset creation for morphologically complex, low-resource African language S2ST tasks. (Adetiba et al., 12 Jul 2025)

6. Relation to Other S2ST Corpora and Broader Implications

BENYO-S2ST-Corpus-1 complements larger-scale massively multilingual resources such as the CVSS corpus (Jia et al., 2022), which targets 21 language-to-English S2ST directions using synthesized English targets in both canonical and speaker-cloned voices. However, CVSS is not designed for high-to-low resource African pairs and employs multivoice alignment using Common Voice and CoVoST 2 data. By contrast, BENYO-S2ST-Corpus-1 explicitly addresses digital divides for African language S2ST via a cost-efficient augmentation-based expansion from limited recordings. A plausible implication is that this architecture can be leveraged to bootstrap parallel S2ST datasets in other high-to-low resource scenarios with restricted accessible speech data but available text translations.

BENYO-S2ST-Corpus-1 and the YoruTTS-0.5 model provide a baseline for research on direct S2ST between English and Yorùbá, with open resources enabling reproducibility and transfer to further African languages, thereby addressing a critical gap in S2ST research infrastructure (Adetiba et al., 12 Jul 2025).