Triple X ASR: Multilingual Speech Recognition

• Triple X ASR is a multilingual speech recognition system that combines a Transformer-based encoder, bottleneck adapter, and LoRA fine-tuned LLM to convert speech to text.
• It employs a multi-stage training strategy with encoder pre-fine-tuning, adapter alignment, and LLM LoRA fine-tuning to optimize performance across 11 languages.
• Robust data augmentation and efficient parameter updates enable the system to achieve a competitive 9.67% word error rate, outperforming baseline models.

The Triple X Automatic Speech Recognition (ASR) System is a LLM-based multilingual speech recognition pipeline developed for the INTERSPEECH 2025 Multi-Lingual Conversational Speech Language Modeling (MLC-SLM) Challenge. Optimizing for multilingual conversational scenarios, the system integrates a Transformer-based acoustic encoder, a bottleneck adapter module, and a powerful LLM (Qwen-3B) via a multi-stage training strategy, employing parameter-efficient fine-tuning and extensive data augmentation to achieve a competitive word error rate (WER) across 11 languages (Gao et al., 23 Jul 2025).

1. System Architecture

The Triple X system utilizes an encoder–adapter–LLM architecture to bridge continuous acoustic representations and discrete text transcriptions.

• Encoder: The backbone is Whisper-large-v3, which accepts 128-dimensional log-Mel frames computed with a 25 ms window and 10 ms hop size. The output is a sequence of hidden states

$$\mathbf{E} = (\mathbf{e}_1, \mathbf{e}_2, \dots, \mathbf{e}_T), \quad \mathbf{e}_t \in \mathbb{R}^{d_e}.$$

• Adapter: Reduces temporal resolution ($T$ to $T'$) via simple frame-splicing (e.g., group every $s$ frames). Projects encoder features into the LLM embedding space through a two-layer MLP:

$$\mathbf{h} = \mathrm{ReLU}(W_1 \mathbf{e}' + \mathbf{b}_1), \quad \mathbf{z} = W_2\,\mathbf{h} + \mathbf{b}_2, \quad \mathbf{z}\in\mathbb{R}^{d_\text{LLM}}.$$

• LLM: The Qwen-3B model consumes the adapter’s sequence $(\mathbf{z}_1, \dots, \mathbf{z}_{T'})$ appended to an optional text prompt. The LLM predicts token probabilities

$P_{\theta}(y_i\mid y_{<i},\mathbf{z}_{1:T'}).$

A schematic overview (ASCII block diagram from (Gao et al., 23 Jul 2025)):

┌───────────┐ ┌────────────┐ ┌───────────────┐ │ Audio in │──▶ │ Whisper │──▶ │ Down-sample │ │ (waveform)│ │ Encoder │ │ + Adapter MLP │──▶ features to LLM └───────────┘ └────────────┘ └───────────────┘ │ ▼ ┌────────┐ │ Qwen-3B│ │ LLM │ └────────┘ │ ▼ Transcription

2. Adapter Integration and Efficient Training

Adapter integration and low-rank adaptation (LoRA) ensure efficient transfer of multimodal features and effective tuning.

• The adapter is inserted after Whisper encoder fine-tuning but before Qwen-3B’s first embedding layer; its outputs serve as "soft tokens" prepended to the LLM prompt.
• Only adapter parameters ($W_1, b_1, W_2, b_2$) and low-rank LLM updates are updated during later training stages.
• LoRA: For each large LLM weight matrix $W$, low-rank matrices ($A, B$) with $r \ll d$ are trained such that

$\Delta W = A\,B, \quad W = W_0 + \alpha\,\Delta W,$

helping to adapt the LLM without full-parameter updates, thereby mitigating catastrophic forgetting.

3. Multi-Stage Training Procedure

A sequential training strategy maximizes data efficiency and component specialization.

• Stage I: Encoder Pre-Fine-Tuning
  • Trained on 30K hours of multilingual public corpora (GigaSpeech2, KsponSpeech, ReasonSpeech, MLS).
  • Objective: cross-entropy minimization to robustly initialize the speech encoder.
• Stage II: Adapter Alignment
  • Freeze encoder; train MLP adapter on 1.5K hours of official conversational data in 11 languages.
  • Adapter learns to project acoustic representations into the LLM’s semantic space.
• Stage III: LLM LoRA Fine-Tuning
  • Freeze both encoder and adapter; update only LoRA parameters in Qwen-3B on the official conversational dataset at a reduced learning rate.

The loss function is

$$\mathcal{L}_{\text{CE}} = -\sum_{i=1}^N \log P_{\theta}(y_i\mid y_{<i},\,\mathbf{z}_{1:T'}).$$

AdamW optimizer is used with learning rates decreasing by stage. Backpropagation targets only stage-relevant parameters.

4. Multilingual Data Processing

The system leverages extensive and diverse datasets.

• Official Set: 1,500 hours, 11 languages, including dialect/broad-accent splits for English (~500 hours), and ~100 hours each for French, German, Italian, Portuguese, Spanish, Japanese, Korean, Russian, Thai, Vietnamese.
• Pre-Fine-Tuning Set: 30,000 hours from public sources for encoder robustness.

Data adaptation uses oracle segmentation, speaker labels, SpecAugment, and speed perturbation (0.9×, 1.0×, 1.1×). There are no explicit language tags; Qwen-3B infers the linguistic context from input embeddings and prompt, accommodating code-switching directly.

5. Experimental Performance

5.1 Word Error Rates

Triple X exhibits strong recognition performance:

• Development set: 9.73%
• Test set: 9.67%

5.2 Beam Search Ablation

Impact of LLM backbone and beam size, as reported:

Backbone	Beam	WER (%)
Qwen3-8B	2	10.83
	3	10.75
	5	10.71
	8	10.70
Qwen3-8B-Base	2	10.56
	3	10.47
	5	10.43
	8	10.41
	10	10.42

5.3 Benchmark Comparison

System	WER (%)
MLC-SLM Baseline	20.17
Triple X (ours)	9.67

This WER achieved second place in the INTERSPEECH 2025 MLC-SLM Challenge (Gao et al., 23 Jul 2025).

6. Technical Impact and Insights

Key drivers of the observed performance include:

• Encoder Pre-Fine-Tuning: Large-scale, heterogeneous datasets significantly improved acoustic modeling.
• Adapter Module: Sequence length reduction and high-dimensional projection facilitated effective alignment between encoder output and LLM token space at minimal computational cost.
• LoRA Updates: Parameter-efficient LLM adaptation enabled effective specialization for multilingual transcription without loss of generalization.
• Data Augmentation: Robust augmentations and prompt-contextualization mitigated challenges due to conversational noise and code-switching.

Simple frame-splicing for down-sampling was shown to be as effective as more complex alternatives in bridging temporal granularity between speech and text tokens.

7. Challenges and Limitations

Challenges managed in the development of Triple X included:

• Multilingual Noise and Code-Switching: Addressed through robust data augmentation and prompt-based context incorporation.
• Sequence Length Mismatch: Resolved via adapter-moderated down-sampling, which also facilitated efficient mapping to the LLM input space.
• LLM Generalization Maintenance: Core LLM weights were frozen during fine-tuning; only low-rank adapters were trained, preserving general language understanding capabilities while allowing task-specific adaptation.

A plausible implication is the demonstrated effectiveness of lightweight adapters and LoRA-based updates for efficient scaling of LLM-based ASR in multilingual and conversationally complex contexts (Gao et al., 23 Jul 2025).