SpeechLLM: Unified Speech and Language Model

• SpeechLLM is a unified neural framework that integrates speech and language modalities using frozen encoders and LLM backbones for end-to-end processing.
• It employs three integration paradigms—text-based, latent-representation, and audio-token approaches—to robustly support tasks from ASR and translation to dialogue generation.
• Advanced techniques like SpeechXL, iterative fusion, and LoRA adapters optimize long-form speech processing while reducing computational costs.

A Speech LLM (SpeechLLM) is a neural architecture that directly integrates speech and language modalities within a large-scale LLM framework, enabling end-to-end modeling of both speech understanding and generation. These models leverage frozen or pre-trained components—speech encoders, specialized adapters, and LLM backbones—to process and generate speech or text, supporting tasks from ASR and translation to dialogue and evaluation. The paradigm extends beyond traditional ASR cascades by treating speech as a first-class input or output modality, encoded at either the token, latent, or raw waveform level, and optimized jointly with language comprehension or generation objectives.

1. Integration Paradigms and Model Architectures

There are three canonical integration paradigms for SpeechLLM design, each with distinct dataflow and optimization characteristics (Yang et al., 26 Feb 2025):

• Text-based Integration: Utilizes external ASR/TTS modules to transcribe or synthesize speech, with the LLM operating purely in the text domain. Typical examples include cascaded recognition/generation pipelines, LLM-based rescoring, or generative error correction.
• Latent-Representation-based Integration: Speech encoders produce frame-level continuous embeddings, which are downsampled or sparsified (via adapters, CTC compression, or Q-Formers) and then projected into the LLM’s token embedding space. The LLM backbone (usually decoder-only) is then conditioned directly on these aligned acoustic features (Li et al., 2024, Wang et al., 2023, Sun et al., 5 Feb 2026).
• Audio-token-based Integration: Discretizes speech via learned semantic and acoustic codebooks; tokens are then modeled autoregressively by the LLM alongside text tokens. Two-stage neural vocoders or acoustic LMs may be used for waveform synthesis (Shen et al., 2024, Hao et al., 2023).

Many state-of-the-art systems adopt a hybrid approach, combining speech encoders (e.g., Whisper, WavLM, HuBERT) with LLMs such as Qwen2.5, Llama-3, or GPT-3/4, bridged by lightweight adapters and LoRA modules for parameter-efficient tuning (Li et al., 2024, Tian et al., 21 Feb 2025, Guo et al., 20 Jul 2025). Table 1 summarizes principal architectural choices:

Integration Paradigm	Input to LLM	Adapter Type
Text-based	Transcript	None / Prompting
Latent-representation	Acoustic embeddings	CNNs, Q-Former, Conv1D+MLP
Audio-token (discrete)	Semantic/acoustic tok.	Token embedding table

2. Core Algorithms and Compression for Long-Form Speech

Handling multi-minute or long-form audio is central to advanced SpeechLLMs. Key innovations address the quadratic memory and compute cost of Transformer self-attention:

• SpeechXL and SST Mechanism: SpeechXL (Sun et al., 5 Feb 2026) introduces Speech Summarization Tokens (SSTs) as interval-wise KV proxies. For an input $X = \{x_1,...,x_N\}$ partitioned into intervals $I_i$, and target compression $\alpha$, each interval is condensed into $k = \lceil w/\alpha \rceil$ SSTs. Within each Transformer layer $\ell$, SSTs pool the KV states of their local window, after which original tokens’ KV pairs may be discarded. This reduces complexity from $O(N^2)$ to $O((N/\alpha)^2)$.
• Iterative Fusion (FastLongSpeech): FastLongSpeech (Guo et al., 20 Jul 2025) compresses a sequence of $J$ frames to length $L \ll J$ via an iterative density-aware fusion, guided by CTC non-blank probabilities and frame similarity. Dynamic compression training randomly varies the frame target during fine-tuning for robust adaptation.
• CTC-based Blank Filtering (Speech2Text Adapter): Adapter architectures decrease frame rate by retaining only high-confidence frames determined by CTC decoding, minimizing sequence length mismatch between speech and text (Wang et al., 2023).

These mechanisms enable practical end-to-end LSLMs to operate on long-form content while managing resource constraints and preserving semantic and paralinguistic content.

3. Multi-Task and Downstream Application Scenarios

Contemporary SpeechLLMs support multitask and modular capabilities via instruction or prompt conditioning, parameter-efficient tuning, and joint optimization:

• ASR, ST, and SQA: Multi-task instruction-tuning with synthetic or human-provided data supports joint ASR, speech translation (ST), and spoken QA tasks (Huang et al., 2023, Li et al., 2024, Chen et al., 2024).
• Dialogue and Speech Synthesis: SpeechLLMs can autoregressively emit both dialogue text and detailed prosodic annotations—or even discrete speech tokens for TTS or S2S generation (Zhou et al., 2023, Zhang et al., 26 May 2025, Shen et al., 2024).
• Zero-shot Spoken Language Understanding: Models like WHISMA leverage instruction-tuning and modality aligners to generalize robustly to new SLU domains and slot-filling tasks, including with internal ASR chain-of-thought or multi-round prompting (Li et al., 2024).
• Speech Retrieval-Augmented Generation: SEAL aligns speech and text in a shared semantic embedding space for end-to-end speech-to-document retrieval, bypassing ASR and minimizing cross-modal error (Sun et al., 26 Jan 2025).
• Assessment and Rescoring: SpeechLLMs have achieved state-of-the-art L2 oral proficiency grading and strong ASR rescoring, leveraging both semantic and acoustic cues unavailable to cascade or text-only systems (Ma et al., 27 May 2025, Shivakumar et al., 2024). Discriminative fine-tuning (MWER) and multi-modal token streams lead to further improvements in WER and ranking accuracy.

4. Training Methods, Optimization, and Evaluation

SpeechLLMs employ a range of training and fine-tuning techniques tailored for multimodal adaptation and efficiency:

• Instruction-based and Chain-of-Thought Tuning: Multi-task objectives combine standard cross-entropy, chain-of-thought prompting, and curriculum learning for complex tasks and progressive compression (Huang et al., 2023, Sun et al., 5 Feb 2026).
• Parameter-efficient Fine-Tuning: LoRA adapters, bottleneck projections, and two-stage (alignment then contrastive) optimization enable scalable adaptation with minimal backbone modification (Li et al., 2024, Sun et al., 26 Jan 2025, Shen et al., 2024).
• Evaluation and Benchmarks: Standard metrics include WER, BLEU/COMET (ST), F1 (timestamp accuracy), and task-specific metrics (SLU-F1, slot-filling, preference agreement). Benchmarks such as LongSpeech-Eval, SLU-GLUE, and OmniCharacter-10K support comprehensive comparison (Guo et al., 20 Jul 2025, Li et al., 2024, Zhang et al., 26 May 2025).

Summary tables below highlight key system-level and performance comparisons:

System	Paradigm	ASR WER (Libri/etc.)	Long-form QA	S2T BLEU	SLU-F1 / Avg Acc	Latency/Speed	Key Features
Speech-XL	Latent/KV-spars	11.4 (LongSpeech)	72.84 (CS)	—	—	~60% TFLOPs	SSTs, multi-min inf.
FastLongSpeech	Latent/Fusion	3.87 (L=200, SQA)	3.55 LS-QA	—	—	1.47s (LS-QA)	Iterative fusion
WHISMA	Latent/Aligner	—	—	—	63.3 (SF:SLU-F1)	—	Llama-3+Whisper, LoRA
ReSLM	Latent+Retriever	8.5 (DSTC11)	—	—	34.6 (DST JGA)	—	Entity prefix, contrast.
SageLM	End-to-end	—	—	—	—	—	Explainable S2S judge
OmniCharacter	Token-based	3.26 (LS)	—	—	—	289 ms	Role/personality S2S
TTS-Llama	Token-based	—	—	—	—	—	LoRA, speech gen., QA

5. Broader Implications and Future Research Directions

SpeechLLMs have advanced the boundary of end-to-end spoken language modeling, but several open challenges remain:

• Compression Limits: Aggressive interval compression (e.g., $\alpha \geq 16$ SST/interval) degrades fidelity in content-sensitive tasks (Sun et al., 5 Feb 2026, Guo et al., 20 Jul 2025).
• Streaming and Real-time Processing: Efficient and low-latency streaming inference is an active area, with recent streamable architectures (BESTOW) making progress toward multitask and simultaneous speech-to-text (Chen et al., 2024).
• Evaluation and Explainability: Multi-aspect explainable evaluation models (SageLM) provide fine-grained, rationale-based judgments over both semantic and acoustic axes, advancing benchmarking (Ge et al., 28 Aug 2025).
• Extension to Multimodal Fusion: There is a clear trajectory toward speech-language-vision models and joint paralinguistic, prosodic, and semantic understanding (Yang et al., 26 Feb 2025).
• Personalization and Role-Conditioned Generation: Integration of persistent voice embeddings and context-aware conditioning supports immersive, personality-driven applications (RPAs, dialogue agents) (Zhang et al., 26 May 2025).

Current limitations include English-centric training data, sensitivity to pooling and adapter design, and domain adaptation to spontaneous, accented, or code-switched input. Ongoing research targets multi-lingual, zero-shot, and scenario-specific generalization, as well as hardware-efficient model scaling and inference (Guo et al., 20 Jul 2025, Sun et al., 5 Feb 2026).

6. Representative Toolkits and Benchmarks

The open-source community supports SpeechLLM development with reproducible toolkits and large benchmarks:

• ESPnet-SpeechLM: An integrated platform for sequence modeling, tokenization, data preprocessing, multi-stream (codec+SSL) fusion, and evaluation across ASR, TTS, and downstream metrics (Tian et al., 21 Feb 2025).
• LongSpeech-Eval, SLU-GLUE, OmniCharacter-10K: Datasets for rigorous longitudinal, zero-shot, and character/personality-conditioned evaluation (Guo et al., 20 Jul 2025, Li et al., 2024, Zhang et al., 26 May 2025).

These frameworks enable rapid deployment and empirical comparison, driving field progress on transparent, multitask speech-language modeling.