Telugu–English Speech Translation Benchmark

• The paper introduces a benchmark for robust Telugu–English ASR/ST that leverages large multilingual models and parameter-efficient adaptation techniques.
• Telugu–English is defined by a structured dataset with 5 hours of segmented audio, high speaker variability, and a focus on low-resource challenges.
• Empirical results demonstrate significant WER reductions using methods like length adapter tuning, text-only adaptation, and cross-lingual transfer.

A Telugu--English speech translation benchmark characterizes a testbed and methodology for automatic speech recognition (ASR) and speech translation (ST) assessment between Telugu (a low-resource, Dravidian language spoken by >80M) and English. Building robust systems for this pair is challenging due to sparse labeled speech data, a large speaker community, and substantial linguistic divergence. Foundational research now leverages large multilingual models, parameter-efficient adaptation, text-only transfer, and cross-lingual pipelines to evaluate and optimize Telugu--English ASR/ST with empirical rigor (Gupta et al., 17 Oct 2024).

1. Benchmark Rationale and Dataset Structure

The core motivation for Telugu--English benchmarks arises from low-resource constraints: the scarcity of human-annotated paired speech and text for Telugu, coupled with high OOV (out-of-vocabulary) rates and dialectal diversity. Standard datasets for this evaluation comprise:

• Labeled Speech: ∼5 hours of segmented audio in Telugu, each with time-aligned transcripts and (typically machine-translated) English reference translations. Dataset splits: ∼5 h train, ∼1 h dev, ∼1 h test.
• Speaker Diversity: Train set covers ∼336 unique speakers; test OOV speaker ratio is ≈44%.
• Text Data: Full corpus sizes range from 0.12–0.83 M tokens per language (mean ≈0.49 M); subsets aligned with limited speech yield 30–43 K Telugu text tokens.

Speech recordings are sampled at 16 kHz. Transcripts and references use Unicode tokenization, compatible with end-to-end speech pipelines.

2. Model Architectures and Task Formulations

Recent Telugu--English benchmarks utilize SeamlessM4T, a large multilingual, multimodal Transformer system (Communication et al., 2023). Its ASR/ST pathway for this benchmark is as follows:

• Speech Encoder: 12 Conformer blocks (311 M params), initialized from Wav2Vec-BERT 2.0 (self-supervised, multilingual speech representations).
• Length Adapter: 46 M parameters; compresses input sequence $X \in \mathbb{R}^{L \times D}$ via 1-D convolution to a shorter sequence $\tilde{X} \in \mathbb{R}^{L' \times D}$, where $L' = \lfloor \frac{L+2p-k}{s} \rfloor + 1$.
• Text Decoder: 12 Transformer blocks, dimension 1024 (201 M params), initialized from NLLB text-to-text translation model.
• ASR/ST Inference Path: Telugu waveform ⇒ speech encoder ⇒ length adapter ⇒ text decoder ⇒ English tokens.

The total backbone for Telugu--English is ≈558 M parameters (SeamlessM4T-medium (Gupta et al., 17 Oct 2024)).

3. Parameter-Efficient Adaptation Protocols

Benchmarks emphasize adaptation strategies for low-resource settings where updating all model parameters is infeasible. Techniques used:

• Feed-forward Adapters: Inserted after every Conformer and Transformer block; bottleneck size $D_2$ chosen as 256 (“small” adapters, 6.3 M params per encoder/decoder side) or 2048 (“large” adapters, 50 M per side). Adapter operation: $$h_\text{out} = h_\text{in} + U \cdot \sigma(V \cdot h_\text{in})$$, $V \in \mathbb{R}^{D_2 \times D_1}$, $U \in \mathbb{R}^{D_1 \times D_2}$, $\sigma = \text{GeLU}$.
• Text-only Adaptation: Fine-tune only the decoder adapters on Telugu–English parallel text. Loss: $$L_{MT} = - \sum_{t=1}^{T} \log p(y_t | y_{<t}, x ; \theta_\text{frozen}, \theta_\text{adapters})$$. This strategy leverages ∼30–43 K tokens for preadaptation, yielding multi-point WER drops on subsequent ASR evaluation.
• Length Adapter Tuning: Fine-tune only the length adapter (46 M params), offering ∼92% parameter savings.

No LoRA, prefix-tuning, or other low-rank protocols were directly used for Telugu--English in these experiments, but their efficiency rationale ($W_\text{new} = W + \Delta W$, $\Delta W$ small) is conceptually similar (Gupta et al., 17 Oct 2024).

4. Cross-Lingual Transfer and Zero-Shot Benchmarks

To compensate for minimal Telugu data, cross-lingual transfer is leveraged:

• Pivot Language Transfer: Fine-tune adapters/length-adapter on a related high-resource language (e.g., Bengali, 400 h labeled) and transfer learning to Telugu in a zero- or few-shot regime. Linguistic proximity is validated via lang2vec metrics; transfer efficacy is measured by relative WER improvement (Gupta et al., 17 Oct 2024).
• Combined Pipelines: After pivot adaptation, text-only Telugu decoder adapters are optionally added, further optimizing performance.

For downstream evaluation, word error rate (WER) is computed: $WER = \frac{S + D + I}{N}$, with $S =$ substitutions, $D =$ deletions, $I =$ insertions, $N =$ reference wordcount.

5. Empirical Performance and Trade-offs

The Telugu--English benchmark reports:

• Baseline WER: Odia (close genetic analog to Telugu/Bengali), WER$_\text{base}$ = 42.81%.
• Length Adapter Fine-Tuned via Bengali: WER$_\text{new}$ = 35.45%, yielding a relative reduction $$= \frac{42.81 - 35.45}{42.81} \times 100\% \approx 17.2\%$$.
• Further Telugu Text-only Tuning: WER drops to 34.0% (additional ≈3% relative).
• Trade-offs: Tuning only length + encoder adapters (52 M params, ≈9.3%) attains within 2–3% absolute WER of full fine-tuning (558 M). Text-only decoder adapters alone give substantial WER reductions provided they are pretrained.

Table: Adaptation Strategies and Relative WER on Telugu–English Benchmark

Method	Params Fine-tuned	WER Reduction
Full ASR Backbone	558 M	Baseline
Length Adapter Only	46 M	–17%
Encoder+Length Adapter	52 M	–17%
Text-only Decoder Adapters	6 M	–3–10%

These results establish parameter-efficient adaptation and cross-lingual transfer as requisite for resource-limited speech translation, especially for Telugu (Gupta et al., 17 Oct 2024).

6. Benchmark Standardization, Metrics, and Best Practices

• Data Partitioning: Strict train/dev/test splits, with non-overlapping speakers to maximize OOV generalization.
• Metrics: Use WER for ASR; BLEU/chRF for ST, with reference translations. Hyperparameters: batch size = 16, learning rate $5 \times 10^{-6}$, 40 ASR epochs, 200 text-only epochs.
• Robustness Considerations: High OOV speaker rates and variable transcript lengths/quality require normalization and error analysis.
• Scaling: Text adaptation is effective with only 30 K tokens, while speech adaptation benefits from cross-lingual pivot strategies.

7. Significance and Outlook for Speech Translation Benchmarks

The Telugu--English benchmark exemplifies scalable evaluation protocols for low-resource language speech translation. Parameter-efficient adaptation (feed-forward adapters, length adapter-only tuning), text-only transfer, and cross-lingual pipelines are the recommended strategy. SeamlessM4T’s multimodal, multilingual architecture sets a robust baseline, and similar protocols are applicable across Indic and Dravidian language pairs.

Future research directions include:

• Further compression of adapters for ultra-low compute
• Incorporation of unsupervised speech and in-domain code-switched text
• Improved tokenization and better leveraging linguistic typology in transfer learning
• Systematic bias, toxicity, and safety auditing in benchmarks (Communication et al., 2023, Communication et al., 2023)

Benchmarks following these principles enable rigorous, replicable development of speech translation systems for languages such as Telugu, with best practices and metrics that generalize across the multilingual modeling domain.