SpeechLLM: Unified Speech & Language Modeling

• SpeechLLM is a multimodal architecture that unifies neural speech processing and language models to perform tasks like ASR, SLU, and speech generation in a single framework.
• It employs a dedicated speech encoder, modality alignment modules, and an LLM backbone to effectively bridge audio and text representations, enabling end-to-end, multitask performance.
• Empirical studies demonstrate improved performance in WER and SLU metrics, while also highlighting challenges such as bias mitigation, streaming latency, and efficient low-resource adaptation.

A Speech LLM (SpeechLLM) is a multimodal architecture that integrates LLMs with neural speech processing modules, unifying speech and text modalities to support automatic speech recognition (ASR), spoken language understanding (SLU), natural language generation from speech, and speech-to-speech tasks. Distinguished from cascaded pipelines, SpeechLLMs directly bridge audio representations and LLMs, enabling end-to-end, multitask, and instruction-following capabilities across diverse speech applications.

1. Architectural Principles and Core Components

SpeechLLMs implement an architectural split between a frozen or trainable speech encoder—typically a high-capacity model such as Whisper, W2v-BERT, or a Conformer—and a backbone LLM (e.g., T5-XXL, Llama-3, mT5, Qwen2.5, Gemma, or OLMo), connected via a modality alignment module (adapter, projector, or fusion layer).

• Speech Encoder: Extracts frame-level hidden representations from audio. Architectures vary from CTC-pretrained models (Wang et al., 2023), large-scale multilingual encoders (Nguyen et al., 16 Jun 2025), to streaming-capable Transformers (Jia et al., 2 Oct 2024, Deng et al., 22 Apr 2025).
• Adapter/Projector: Compresses, aligns, and transforms speech features into the embedding space of the LLM using linear projections, MLPs, self-attention adapters, or attention pooling (Wang et al., 2023, Nguyen et al., 16 Jun 2025, Ma et al., 20 Dec 2025).
• LLM Integration: Either a decoder-only or encoder-decoder architecture receives the adapted speech sequence (possibly interleaved or as a prompt prefix) and jointly models sequential prediction for tasks such as ASR, DST, translation, and generation (Nguyen et al., 16 Jun 2025, Chen et al., 28 Jun 2024).
• Retriever/Augmentation: Optional dual-encoder retrievers augment the context with symbolic knowledge (e.g., entities) to support rare entity recovery and domain adaptation (Wang et al., 2023).
• Compression and Pooling: Context compression via attention-pooling or fixed temporal pooling is used to manage memory and computational costs, especially in dialogue or streaming setups (Ghazal et al., 10 Oct 2025, Nguyen et al., 16 Jun 2025).
• Fine-tuning: Parameter-efficient tuning via LoRA or other adapters is typical, freezing most model weights for scalability and robustness (Li et al., 29 Aug 2024, Ghazal et al., 10 Oct 2025).

2. Methodological Innovations and Training Schemes

• Adapter and Retriever-Based Alignment: The use of CTC blank-filtering to reduce speech frame lengths, followed by self-attention and projection adapters, enables direct mapping of speech embeddings to text-token spaces, supporting unified ASR and SLU objectives on a shared LLM backbone (Wang et al., 2023).
• Contrastive and Multitask Learning: Several systems employ contrastive alignment, multi-task objectives (ASR, slot-filling, AST, SQA), and joint speech-text contrastive loss to align modalities at both local and global levels (Wang et al., 2023, Li et al., 29 Aug 2024, Chen et al., 28 Jun 2024, Züfle et al., 20 Dec 2024).
• Streaming and Chunked Decoding: Architectures such as SpeechLLM-XL and SimulS2S-LLM process audio in fixed-duration chunks with limited attention windows, maintaining linear compute complexity and bounded streaming latency (Jia et al., 2 Oct 2024, Deng et al., 22 Apr 2025). CTC forced alignment and chunk-based segmentation are used to synchronize audio and transcript chunks.
• Instruction/Fusion Layers: Hybrid GPT- and T5-style fusion layers, as in BESTOW, insert causal self-attention and speech cross-attention either once (up-front) or per-layer, achieving a balance between computation, knowledge transfer, and flexibility for streaming or multitask deployment (Chen et al., 28 Jun 2024).
• LoRA and Parameter-Efficient Fine-Tuning: Low-rank adaptation modules (LoRA) allow efficient, scalable adaptation to downstream speech tasks without updating the entire LLM or speech encoder (Li et al., 29 Aug 2024, Ghazal et al., 10 Oct 2025).
• Unified Modality Encoders Without Speech Data: TESU-LLM demonstrates that a properly aligned text-speech encoder, trained only on text with a lightweight MLP projector, can enable a frozen LLM to generalize to speech inputs during inference, even in zero-speech-data scenarios (Kim et al., 1 Jun 2025).

3. SpeechLLM Applications and Task Domains

• Automatic Speech Recognition (ASR): SpeechLLMs have matched or exceeded state-of-the-art WER on LibriSpeech and other benchmarks using streaming and chunked decoding strategies, with explicit comparison to CTC, Transducer, and cascaded LLM systems (Jia et al., 2 Oct 2024, Nguyen et al., 16 Jun 2025, Ma et al., 20 Dec 2025).
• Spoken Language Understanding (SLU): End-to-end systems support slot-filling, intent detection, dialogue state tracking (DST), and question answering (SQA), often surpassing traditional pipelined approaches and achieving strong zero-shot generalization (Wang et al., 2023, Li et al., 29 Aug 2024, Ghazal et al., 10 Oct 2025, Hacioglu et al., 22 Oct 2025).
• Pseudo-Labeling for Semi-Supervised Learning: Multi-ASR fusion with LLM- or SpeechLLM-guided error correction yields high-accuracy pseudo-labels, improving downstream ASR models in low-resource domains (Prakash et al., 5 Jun 2025).
• Speech Quality and Proficiency Assessment: SpeechLLMs act as graders for L2 oral proficiency, outperforming cascaded and direct regression baselines. They are extended to natural-language, aspect-aware speech quality evaluation using chain-of-thought and reward-optimized LLMs (e.g., SQ-LLM, SpeechQualityLLM) (Ma et al., 27 May 2025, Wang et al., 16 Oct 2025, Monjur et al., 9 Dec 2025).
• Role-Playing and Persona-Driven Spoken Dialogue: Unified speech–LLMs with speech token decoding and speed-optimized TTS can create role-consistent, low-latency conversational agents (OmniCharacter) (Zhang et al., 26 May 2025).
• Visual Speech Generation: SpeechLLMs underlie VisualTTS models (VSpeechLM) that integrate fine-grained phoneme-lip alignment to generate lip-synchronized, high-quality speech from video and text (Wang et al., 27 Nov 2025).
• Low-Resource and Multilingual Scenarios: SLAM-ASR and similar frameworks adapt SpeechLLMs with lightweight projectors for robust ASR in low-resource settings and cross-lingual transfer (Fong et al., 7 Aug 2025).

4. Empirical Performance, Limitations, and Robustness

• ASR and S2ST Metrics: State-of-the-art WERs (e.g., 2.7% test-clean, 6.7% test-other on LibriSpeech with SpeechLLM-XL; 12.5% WER, 78.9% speaker similarity for VisualTTS tasks) have been reported (Jia et al., 2 Oct 2024, Wang et al., 27 Nov 2025).
• DST and SLU Gains: Adapter and retriever-augmented SpeechLLMs show absolute joint goal accuracy gains of 3–6 pp and nontrivial reductions in WER on challenging dialogue datasets (Wang et al., 2023, Ghazal et al., 10 Oct 2025).
• Pseudo-Labeling: SpeechLLM-based pseudo-labels reduce WER by 10–15% relative over strong ASR ensembles and textual LLM correction (Prakash et al., 5 Jun 2025).
• Speaker Awareness and Paralinguistic Cues: Studies reveal that current SpeechLLMs show little to no speaker-discriminative ability in SQA tasks unless explicit speaker tags are provided in the prompt, indicating gaps in paralinguistic reasoning (Wu et al., 7 Sep 2024).
• Bias and Fairness: Token-level analysis demonstrates position and gender bias in MCQA settings, with female-voice inputs yielding more pronounced slot-avoidance effects. Standard MCQA benchmarks may mask such biases (Satish et al., 1 Oct 2025).
• Limitations: SpeechLLMs may lag modular phoneme-based decoders (e.g., SKM-driven LLM-P2G) in ASR accuracy (Ma et al., 20 Dec 2025), and resource requirements for matching Whisper-only ASR in low-resource scenarios remain high (∼200 h of labeled speech). Speechless LLMs relying solely on semantic encoders offer reduced performance on paralinguistic and ASR tasks (Kim et al., 1 Jun 2025).

5. Design Trade-Offs and System Variants

Variant	Speech Adaptation	LLM Integration	Typical Use Case
Adapter/Prefix	Self-attention, MLP	Decoder-only or Enc-Dec	Unified ASR/SLU, Multitask pipelines
Cross-Attention	Per-layer fusion	GPT/T5 hybrid	Streaming ASR, SQA, multitask S2ST
Retriever-Aided	Dual-encoder	Prefix injection	DST, rare-entity recovery
Pool+Projector	Avg/Attn pooling	Compact embedding	Large-context dialogue, efficient ASR
Chain-of-Thought	CoT prompting	Reasoning-LLMs	Slot-filling, structured QA, logic tasks

Adapters and pooling modules enable scalable bridging between continuous audio and discrete tokens, but trade off between compressive efficiency and fine-grained alignment. Per-layer fusion (T5-style) offers stronger context integration at higher computational cost. Hybrid architectures (BESTOW) combine up-front fusion for efficiency with deep LLM stacks for rich reasoning and multitask capability (Chen et al., 28 Jun 2024).

6. Future Directions and Open Challenges

• Improved Modality Alignment: Research continues on contrastive and generative pretraining to align speech and text layers at scale and with minimal paired data (Züfle et al., 20 Dec 2024).
• Speaker/Style Conditioning: Architectures for robust "who said what" reasoning and speaker-aware dialogue remain open problems (Wu et al., 7 Sep 2024).
• Streaming and Low-Latency: New models target ultra-low-latency streaming, chunk-based decoding, and real-time speech-to-speech interaction for dialogue and translation (Jia et al., 2 Oct 2024, Deng et al., 22 Apr 2025).
• Bias Mitigation: Systematic benchmark design and evaluation protocols are needed to diagnose and offset positional, gender, and paralinguistic bias in SpeechLLM outputs (Satish et al., 1 Oct 2025).
• Speechless and Data-Efficient Models: Frameworks like TESU-LLM, which enable speech understanding with zero speech data in training via aligned unified encoders, offer scalable paths for low-resource environments, though coverage of paralinguistic phenomena remains limited (Kim et al., 1 Jun 2025).
• Joint Generation and Understanding: Extensions to joint speech-text or speech-speech agents, including multimodal visual input (VSpeechLM), persona-driven dialogue (OmniCharacter), and explanatory quality assessment (SQ-LLM), illustrate the breadth of current SpeechLLM research (Wang et al., 27 Nov 2025, Zhang et al., 26 May 2025, Wang et al., 16 Oct 2025).

7. Representative References

• Adapter and retriever-augmented LLMs: (Wang et al., 2023)
• Comparative studies of projectors and LLM backbones: (Nguyen et al., 16 Jun 2025, Ma et al., 20 Dec 2025)
• SpeechLLM-XL streaming recognition: (Jia et al., 2 Oct 2024)
• End-to-end dialogue state tracking: (Ghazal et al., 10 Oct 2025)
• Multitask and streaming fusion (BESTOW): (Chen et al., 28 Jun 2024)
• Low-resource adaptation and projector pretraining: (Fong et al., 7 Aug 2025)
• Zero-shot SLU and instruction-following: (Li et al., 29 Aug 2024)
• Chain-of-thought slot-filling: (Hacioglu et al., 22 Oct 2025)
• Unified text-speech alignment for speechless training: (Kim et al., 1 Jun 2025)
• Role-playing speech agents: (Zhang et al., 26 May 2025)
• Visual Text-to-Speech and lip synchronization: (Wang et al., 27 Nov 2025)
• Pseudo-captioning and error correction: (Prakash et al., 5 Jun 2025)
• Speech quality assessment: (Monjur et al., 9 Dec 2025, Wang et al., 16 Oct 2025)

SpeechLLM research thus encompasses foundation model adaptation, cross-modal alignment, low-resource robustness, rich reasoning, streaming, and evaluation—defining a rapidly maturing paradigm for integrated speech and language modeling.