Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervision

Abstract: Speech LLMs that understand and follow instructions in many languages are useful for real-world interaction, but are difficult to train with supervised fine-tuning, requiring large, task-specific speech corpora. While recent distillation-based approaches train performant English-only Speech LLMs using only annotated ASR data by aligning text and speech using only a lightweight projector, these models under-perform when scaled to multilingual settings due to language interference in the shared projector. We address this by introducing language-aware distillation using a query bank and a gating network that selects or mixes query tokens using a Q-Former projector. Our approach shows gains of 14% over matched multilingual distillation baselines on instruction following. We further synthesize Audio-MLQA, a multilingual spoken QA benchmark built on MLQA with high-quality TTS questions. Our best model improves over existing Speech LLM baselines by 32% on Audio-MLQA.

• The paper introduces gated, per-language Q-Former queries with soft or hard routing, enabling a frozen Whisper-Llama Speech LLM to learn from 5,870 hours of ASR-only data across six languages.
• Hard language gating improves open-ended instruction-following scores by 14% over multilingual DiVA, including a 22% relative gain for Indonesian, while reaching 3.96 on Audio-MLQA.
• The approach uses modest compute and frozen backbones but remains untested on natural, noisy, accented, or code-switched speech, and Chinese performance still trails the text-only reference.

Motivation and problem setting

This paper addresses a specific bottleneck in training multilingual Speech LLMs: alignment-based distillation methods such as DiVA can train capable English-only speech assistants using only annotated ASR data, but a naive extension to multiple languages degrades performance because a single static Q-Former query sequence becomes a shared representation space in which dominant languages interfere with lower-represented ones. The authors observe this interference empirically and connect it to prior findings that explicit language conditioning benefits multilingual ASR. The proposed solution keeps the speech encoder (Whisper-large-v3) and the text backbone (Llama-SEA-LION-v3-8B-IT) frozen, trains only a lightweight projector plus gating module, and requires just 5,870 hours of ASR-only data across six languages (EN, VI, ID, ZH, ES, DE). Freezing both backbones avoids catastrophic forgetting and eliminates the need for task-specific speech instruction corpora.

Method

The architecture follows the DiVA recipe: speech embeddings from the frozen Whisper encoder are projected by a Q-Former into $L$ soft-prefix tokens for the frozen LLM, trained with input distillation (regressing projected speech embeddings onto transcript-derived LLM input embeddings over an "audio tail" alignment) and output distillation (matching last-layer hidden states under speech versus transcript conditioning), supplemented by a language identification cross-entropy loss on the gate.

The central contribution is language-aware distillation. A bank of per-language learnable query sequences $\{\mathbf{Q}^{(k)}\}_{k=1}^{K}$ replaces the single shared query set. A gating network — either a convolutional down-sampling-and-pooling network or an attention-pooling MLP — produces language logits $\mathbf{g}$ from the speech embeddings, which are used for either:

• Soft query mixing: queries are combined as $\tilde{\mathbf{Q}} = \sum_k \pi_k \mathbf{Q}^{(k)}$ with $\boldsymbol{\pi} = \mathrm{softmax}(\mathbf{g})$, allowing sharing across related languages.
• Hard query selection: the argmax language's query is used, with a straight-through estimator backpropagating through the soft mixture; scheduled teacher forcing anneals reliance on ground-truth language labels to zero at 50% of training steps.

Training uses 4 H100 GPUs with DeepSpeed ZeRO-2, BF16 precision, and roughly 80 hours for 30K steps — a modest compute budget relative to SFT-based pipelines.

Evaluation setup

Because established multilingual spoken-instruction benchmarks are scarce, the paper contributes two synthetic evaluation sets generated with high-quality 44.1 kHz TTS: an open-ended instruction-following suite covering ZH (UROBench AlpacaEval-zh), EN (AudioBench ALPACA-Audio/OpenHermes-Audio), and ID (native-speaker-translated prompts); and Audio-MLQA, a close-ended spoken QA benchmark built from MLQA with 250 items per language across five languages and ten voices per language to reduce speaker bias. All evaluation uses GPT-4.1 Model-as-Judge scoring on a 0–5 scale, following evidence that strong judge models align closely with human judgment. The authors compare against matched re-trained baselines (EN-DiVA, ML-DiVA) and external zero-shot end-to-end systems (GLM-4-Voice, MERaLiON-2-10B, SeaLLMs-Audio, Qwen2-Audio).

Results

The headline numbers are substantial:

Model	A-MLQA avg	Open-ended ZH	Open-ended EN avg	Open-ended ID avg
Text-only SEA-LION-v3-8B-IT (upper bound)	4.14	4.21	4.52	4.02
Whisper + SEA-LION (cascaded)	3.99	3.95	4.08	3.65
ML-DiVA baseline	3.85	2.87	4.16	3.04
Ours (soft-gating)	3.88	3.21	4.44	3.62
Ours (hard-gating)	3.96	3.33	4.41	3.71

On open-ended instruction following, hard-gating yields a 14% average gain over ML-DiVA, with the largest improvement on Indonesian (3.04 → 3.71, a 22% relative gain) — direct evidence that language-aware routing protects lower-represented languages from interference. On Audio-MLQA, the best model improves over SeaLLMs-Audio and Qwen2-Audio by 32% and 31% respectively, reaching 3.96 against a text-only reference of 4.14. Notably, the distilled model outperforms its own cascaded Whisper+LLM counterpart, which is a nontrivial result given the cascaded system's access to explicit transcription. The paper also reports that simply scaling ML-DiVA to more languages improved English open-ended performance by 21%, attributed to shared linguistic structure and data diversity — a useful observation independent of the proposed method.

Ablations show that increasing query length $L$ from 64 to 256 reduces input distillation loss by 89% ($8.63 \rightarrow 0.97$), indicating that projector capacity is a first-order factor in capturing phonetic–semantic mappings. Both gating architectures achieve >94.9% LID accuracy, and hard selection consistently beats soft mixing downstream, which the authors attribute to stronger decoupling of language-specific information and avoidance of an "averaging" effect favoring dominant languages.

Limitations and open questions

Several caveats bear directly on the results. First, all evaluation audio is TTS-synthesized; performance on spontaneous, noisy, or accented real-world speech is not measured, and Model-as-Judge scoring with GPT-4.1 introduces judge-dependent bias despite cited evidence of human alignment. Second, the method assumes loosely annotated language labels for LID supervision and teacher forcing; behavior under code-switching or unknown-language inputs is untested. Third, gains on Chinese — the least similar language to English in the training pool — remain below the text-only reference and behind some baselines on open-ended tasks, suggesting the approach does not fully resolve interference for typologically distant pairs. Fourth, the query bank scales linearly with the number of languages, leaving open how the mechanism behaves when $K$ grows large or when languages are added after initial training. Finally, whether hidden-state regression remains a faithful proxy for logit-level KL distillation in more diverse multilingual regimes is asserted from prior work rather than verified here.

Conclusion

The paper demonstrates that replacing a static Q-Former query sequence with a gated per-language query bank resolves the language-interference bottleneck in ASR-supervised distillation, yielding a six-language Speech LLM trained on 5.8K hours of ASR data with frozen backbones, competitive with or superior to substantially larger SFT-trained systems. The released Audio-MLQA benchmark and multilingual open-ended evaluation sets provide useful infrastructure for future work, though validation on naturalistic speech and extension beyond labeled-language settings remain unresolved questions.