Speech-LLM: Unified Multimodal Speech Model
- Speech-LLM is a unified neural architecture that directly maps continuous acoustic features into a language model’s embedding space, enabling end-to-end speech processing.
- It leverages pre-trained speech encoders with modality adapters—such as projection MLPs and cross-modal attention—to integrate tasks like ASR, SLU, and speech translation.
- Innovative training objectives and efficient multi-modal alignment enable competitive performance and fine-grained control over both linguistic and paralinguistic features.
A Speech-LLM is a unified neural architecture that directly maps speech input into the embedding space of a LLM, enabling the LLM to condition its generation on acoustic features for tasks spanning automatic speech recognition (ASR), spoken language understanding (SLU), speech translation, dialogue modeling, and more. In contrast to cascaded ASR+LLM pipelines, Speech-LLMs natively handle continuous acoustic features by integrating foundation-model speech encoders, specialized modality adapters or projection layers, and—optionally—cross-modal attention, enabling direct multimodal reasoning and end-to-end instruction following (Li et al., 2024, Ma et al., 16 May 2025, Ghazal et al., 10 Oct 2025, Deng et al., 22 Apr 2025).
1. Foundational Architecture and Integration Strategies
Speech-LLMs share a canonical structure: a pre-trained speech encoder (e.g., Whisper, w2v-BERT, mHuBERT) transforms a waveform into frame-level representations, which are then mapped into the LLM’s input space by a neural adapter—typically a projection MLP, compression module, or CTC-posterior-based reconstructor. For example:
- LegoSLM: Projects CTC posterior matrices onto the LLM vocabulary and reconstructs pseudo-embeddings as framewise weighted sums of LLM token embeddings, enabling modular encoder-LLM swapping and zero-shot adaptation (Ma et al., 16 May 2025).
- Ideal-LLM: Fuses dual encoder outputs (e.g., Whisper and MMS) with language-adaptive scalar weighting, learned per-language, prior to projection into the LLM embedding space (Xue et al., 2024).
- SpeechMapper: Employs a two-stage strategy, pre-training a deep projector to match speech embeddings to LLM token embeddings using only ASR data, then rapidly attaching to any LLM via lightweight instruction tuning (Mohapatra et al., 28 Jan 2026).
- End-to-End Systems: Models such as (Ghazal et al., 10 Oct 2025) use a direct connector (e.g., 1-layer Transformer) to project encoder frames to the LLM space, supporting full speech context input.
Adapters range from linear/MLP projections combined with temporal downsampling (Fong et al., 7 Aug 2025, Li et al., 2024), to more advanced alignment modules such as Q-Formers or dynamic window cross-attention (AlignFormer), and CTC-posterior-to-embedding projections (LegoSLM).
A table of representative strategies:
| System | Speech Encoder(s) | Modality Adapter | LLM Backbone |
|---|---|---|---|
| WHISMA | Whisper | CNN + bottleneck + LoRA | Llama-3 |
| LegoSLM | USM-CTC | CTC posterior reconstructor | Gemma 2B |
| SpeechMapper | SeamlessM4T | Deep conv+Transformer MLP | Llama3 8B, others |
| Ideal-LLM | Whisper + MMS | Language-weighted fusion, proj | LLM (not specified) |
| UTI-LLM | HuBERT + CLIP (video) | Linear to Qwen2.5-7B space | Qwen2.5-7B |
| PROST-LLM | mHuBERT (tokens) | Shared embedding space | LLaMA 3.2-3B |
2. Training Objectives, Supervision, and Alignment
Speech-LLMs generally freeze their LLM backbones to preserve instruction-following and text understanding; only modality adapters and sometimes LoRA adapters are tuned (Li et al., 2024, Ma et al., 16 May 2025, Ghazal et al., 10 Oct 2025). Supervised learning proceeds with task-appropriate targets:
- ASR: Cross-entropy on generated transcript conditioned on acoustic embeddings (Mohapatra et al., 28 Jan 2026, Fong et al., 7 Aug 2025).
- SLU: Multi-task cross-entropy for slot filling, intent classification, and QA answers (Li et al., 2024).
- Speech Translation: Cross-entropy over target language tokens or semantic speech tokens (Deng et al., 22 Apr 2025, Xu et al., 23 Jan 2026, Arya et al., 22 Jan 2026).
- Multi-Aspect Assessment: Regression/classification losses for pronunciation, quality, or speaker-aware scores (Parikh et al., 20 Jan 2026, Monjur et al., 9 Dec 2025).
- Alignment: Some methods add Connectionist Temporal Classification (CTC) loss or auxiliary MSE/cosine embedding losses to bridge speech-text representations (Ma et al., 16 May 2025, Mohapatra et al., 28 Jan 2026).
Advanced objectives include:
- Chain-of-Modality: PROST-LLM first generates text and then speech in a sequence, stabilizing S2ST by forcing text intermediates (Xu et al., 23 Jan 2026).
- Direct Preference Optimization (DPO): Trains LLMs with preference pairs (from back-translation or self-sampling) to optimize S2ST outputs without human labels (Xu et al., 23 Jan 2026).
- Dynamic window alignment (CTC+QFormer): Addressing speech-text length mismatch with CTC-guided Q-Former adapters (AlignFormer; see abstract (Fan et al., 2024)).
Zero-shot performance and generalization are enhanced by large-scale multi-task datasets and multi-stage pipelines: pre-train the modality adapter/projector on ASR, followed by brief tuning for instruction-following or downstream tasks (Mohapatra et al., 28 Jan 2026, Fong et al., 7 Aug 2025).
3. Multimodal Context Handling and Speech-LLM Dynamics
Speech-LLMs uniquely allow direct manipulation and reasoning across entire spoken contexts:
- Full Spoken History: Feeding all previous turn embeddings (not just transcripts) enables dramatic gains in Spoken Dialogue State Tracking over text-history or current-turn-only inputs (JGA: 39.32% full speech vs. 32.06% with multimodal context; (Ghazal et al., 10 Oct 2025)).
- Context Compression: Attention-pooling (query-based compression) reduces context size with modest accuracy loss (e.g., JGA 36.49% with 10 queries), supporting longer contexts in low-memory environments (Ghazal et al., 10 Oct 2025).
- Multilingual Fusion: Language-aware weighting in dual-encoder systems exploits complementary strengths (e.g., Whisper for English, MMS for under-represented languages), yielding tight language-specific clusters in LLM embedding spaces (Xue et al., 2024, Mei et al., 4 Jan 2026).
- Speaker, Prosody, Articulatory Cues: Integration with paralinguistic and sensor modalities (e.g., speaker identity, emotion, ultrasound tongue imaging) via parallel encoders and fusion modules enables richer representations for tasks like therapy feedback and empathetic response (Yang et al., 16 Sep 2025, Chen et al., 11 Feb 2026, Xu et al., 23 Jan 2026, Wu et al., 2024).
4. Evaluation Benchmarks and Quantitative Results
Speech-LLMs are benchmarked on ASR, SLU, ST, S2ST, pronunciation assessment, and dialogue tracking. Key quantitative findings include:
| Task | System(s) | Topline Metric / Gain | Notes |
|---|---|---|---|
| ASR | LegoSLM | MLS-en WER 6.1% | 31.5% WERR vs. baseline |
| SpeechMapper | CV IT WER 6.4%* | Beats Whisper-only (7.1%) | |
| Ideal-LLM (Large) | WER 7.81% | 32.6% reduction (Xue et al., 2024) | |
| Dialogue State Tracking | OLMo2-1B+LoRA | JGA 39.32% | +7.3 pts over prior SOTA (Ghazal et al., 10 Oct 2025) |
| S2ST | DS²ST-LM | BLEU up to 14.71 (zh–en) | Outperforms ST+TTS (Arya et al., 22 Jan 2026) |
| S2ST | PROST-LLM | S2ST BLEU 25.1 (CoM+DPO) | <4 BLEU gap to cascade |
| SLU | WHISMA | Slot F1 63.3 | +26.6% rel. over SOTA (Li et al., 2024) |
| Pron. Assess. | Phi-4 LoRA-MLLM | PER 0.114 / PCC 0.73 | LoRA ≃ full fine-tune (Ahn et al., 3 Sep 2025) |
| Speech QA/Emotion | RE-LLM | Ex 1.206 (ESD), UA 98.3% | +14.7% ER rel. (ESD) |
*CV IT: CommonVoice Italian; WER values cross check with (Fong et al., 7 Aug 2025).
Speech-LLMs can rival or exceed end-to-end Whisper baselines in low-resource settings by leveraging cross-lingual projector pretraining and rapid fine-tuning (Fong et al., 7 Aug 2025). Multi-task and chain-of-modality learning close the gap with cascaded S2ST quality (Xu et al., 23 Jan 2026), and direct speech conditioning enables fine-grained outputs like articulatory feedback (Yang et al., 16 Sep 2025).
5. Specialized Capabilities and Applications
Recent architectures demonstrate expansion beyond standard ASR/SLU:
- Empathetic Dialogue and Emotion: RE-LLM fuses speech and emotion embeddings, with auxiliary valence/arousal/dominance objectives, achieving significant boosts in empathy and emotion recognition (Chen et al., 11 Feb 2026).
- Speech Quality Judgment: SpeechQualityLLM estimates MOS and dimension-wise perceptual scores from audio, matching or exceeding classical metrics (r = 0.86 for MOS; (Monjur et al., 9 Dec 2025)).
- Therapy Feedback: UTI-LLM merges ultrasound and speech signals, decoding synchronized articulatory and acoustic features for personalized clinical feedback (BLEU/METEOR/ROUGE/Accuracy; (Yang et al., 16 Sep 2025)).
- Zero-shot Pronunciation Assessment: Speech LLMs like Qwen2-Audio-7B-Instruct produce rubric-aligned fluency/prosody/completeness ratings within ±2 points of human raters for ~90% of high-quality utterances (Parikh et al., 20 Jan 2026).
- Data Generation: Systems such as SpeechDialogueFactory automate large-scale, quality-filtered, paralinguistically enriched dialogue data generation, supporting efficient model pretraining, fine-tuning, and evaluation using composable metadata, scripted control, and voice-cloned synthesis (Wang et al., 31 Mar 2025).
6. Theoretical and Practical Challenges
Speech-LLMs must overcome several technical limitations:
- Modality Alignment: Speech–text length mismatch and sequencing are nontrivial; advanced adapters (AlignFormer: CTC + dynamic window QFormers) or sequence-level alignment objectives are critical for preserving instruction-following fidelity (Fan et al., 2024).
- Generalization vs. Specialization: Overfitting to small instruction-tuning datasets can harm cross-task transfer; SpeechMapper's two-stage decoupling strategy with robust MSE/cosine losses mitigates this (Mohapatra et al., 28 Jan 2026).
- Low-Resource Language Adaptation: Projector pretraining on high-resource languages and rapid adaptation is required to match Whisper ASR performance with 10–20 h of target language data (Fong et al., 7 Aug 2025).
- Emotion and Speaker Identity: Default architectures show limited improvement in speaker identification and emotion extraction unless explicitly encoded (e.g., WavLM, wav2vec auxiliary models), indicating that text transcript reasoning dominates unless training and benchmarks specifically sharpen acoustic feature use (Wu et al., 2024, Chen et al., 11 Feb 2026).
- Clinical Robustness: Incorporating highly specialized, sensor-based modalities (e.g., UTI) demands dedicated annotated data and careful fusion, but yields high-fidelity, context-aware outputs for healthcare (Yang et al., 16 Sep 2025).
- Resource Efficiency: State-of-the-art Speech-LLMs reach competitive performance by tuning only <1% of total parameters (LoRA/projector weights) and training on commodity hardware (Fong et al., 7 Aug 2025, Mohapatra et al., 28 Jan 2026). Fully end-to-end fine-tuning can result in minimal gains or decreased robustness due to over-specialization (Mei et al., 4 Jan 2026).
7. Future Directions and Recommendations
Key directions, drawn from recent empirical findings and ablations, include:
- Cross-modal PEFT (LoRA/Q-Former/compression) and linear projectors are preferred for robust, generalizable alignment (Li et al., 2024, Ma et al., 16 May 2025, Mohapatra et al., 28 Jan 2026).
- Multi-Encoder/Multi-Modal Fusion: Language-adapted weighting and sensor integration will expand the range of applications and improve under-resourced language and clinical task performance (Xue et al., 2024, Yang et al., 16 Sep 2025).
- Comprehensive Spoken Context Modeling: Direct context feeding and strategic compression balance accuracy with tractable context length (Ghazal et al., 10 Oct 2025).
- Benchmark Design: Evaluations must distinguish between content understanding and acoustic/paralinguistic capabilities (e.g., speaker ID, emotion, slot filling); synthetic dialogue generators (SpeechDialogueFactory) enable controlled, rich evaluation sets (Wang et al., 31 Mar 2025).
- Zero-shot and Few-shot Transfer: Training strategies should exploit multi-task and multi-modal data, automatic preference optimization, and frequent rapid adapter tuning to support rapidly developing languages, domains, and user groups (Mohapatra et al., 28 Jan 2026, Xu et al., 23 Jan 2026, Fong et al., 7 Aug 2025).
- Non-Speech-Training (TESU-LLM): Unified encoders trained solely on text can yield competitive speech-LLMs for core tasks, though paralinguistic and fine acoustic feature performance is reduced (Kim et al., 1 Jun 2025).
Speech-LLMs are converging toward a paradigm where content, paralinguistic, and domain-specific expertise can be natively and efficiently integrated within a single large-model backend, supporting both generic and specialized spoken language applications with minimal retraining overhead.
