SpeechLLM: Unified Speech and Text Models

• SpeechLLMs are deep neural architectures that integrate speech processing with language modeling, enabling multimodal understanding and generation.
• They employ methods ranging from cascaded ASR pipelines to latent representation adapters and audio-token quantization for diverse speech tasks.
• Empirical results highlight low word-error rates, efficient fine-tuning, and robust performance across ASR, SLU, and speech translation tasks.

Speech LLMs (SpeechLLM) constitute a broad class of architectures that extend text-only LLMs to the speech modality, enabling unified or tightly integrated speech understanding and generation capabilities within the LLM framework. These models leverage recent advances in Transformer architectures, architectural adaptivity (such as parameter-efficient fine-tuning), modality bridging with adapters or tokenization, and large-scale multi-task pretraining to achieve robust performance across diverse spoken language processing tasks.

1. Definition, Scope, and Motivation

SpeechLLMs are systems that adapt or extend pretrained LLMs (typically decoder-only Transformers with ≥1B parameters) to ingest, understand, or generate spoken language alongside text. They may support speech input, speech output, or both, and unify modalities at either the level of sequence modeling or latent representation. The motivation lies in:

• Linguistic completeness: Natural human communication is inherently multimodal—spoken language carries lexical, prosodic, and paralinguistic information (e.g., prosody, speaker identity, emotion) that text-only LLMs cannot model (Yang et al., 26 Feb 2025, Peng et al., 24 Oct 2024).
• Pipeline simplicity and robustness: Classical cascades (ASR→NLP→TTS) suffer error propagation and lack end-to-end optimization (Lakomkin et al., 2023).
• Research momentum: The success of vision-language multimodal LLMs motivates similar integration for speech, which brings new challenges in temporal sequencing, data compression, and multimodal reasoning.

Formally, speech understanding in this context is the process of perceptually and cognitively transforming raw acoustic signals into textual or structured output that may reflect both linguistic and paralinguistic content (Peng et al., 24 Oct 2024).

2. Methodological Taxonomy of Integration Strategies

Methodologies for integrating speech into LLMs fall into three principal categories, each with representative architectures and associated trade-offs (Yang et al., 26 Feb 2025).

• Text-Based Integration: Cascaded architectures use an external ASR system to transform speech into text before passing it to a frozen LLM. Optional TTS can be used for speech output generation. Extensions include LLM-based N-best rescoring (Shivakumar et al., 25 Sep 2024) and generative error correction (LLM corrects errors in ASR hypotheses via instruction prompting) (Prakash et al., 5 Jun 2025). These methods retain interpretability but propagate upstream ASR errors and lose paralinguistic nuance.
• Latent-Representation-Based Integration: Speech is encoded via pretrained neural encoders (e.g., Whisper, Conformer, WavLM) into continuous or compressed frame-level representations. Dedicated adapters—possibly with blank-filtering or convolutional downsampling—bridge these representations to the LLM’s embedding space (Wang et al., 2023, Peng et al., 24 Oct 2024, Cuervo et al., 31 Mar 2024). The unified model is then fine-tuned for sequence-to-sequence tasks (e.g., ASR, SLU, SQA). These approaches capture deeper cross-modal alignment and achieve strong ASR/S2TT results at the expense of higher computation and model complexity.
• Audio-Token-Based Integration: The audio signal is quantized into discrete tokens (semantic and/or acoustic, via vector quantization, EnCodec, etc.) (Hao et al., 2023). The LLM is trained to model joint sequences of audio and text tokens, enabling direct speech generation, speech-to-speech translation, or “audio-in, audio-out” tasks (Wang et al., 5 Apr 2025, Shen et al., 27 Oct 2024). These methods support prosody and speaker style transfer, but the choice of audio token vocabulary and sequence compression presents challenges.

A subset of recent architectures combines these strategies (e.g., BESTOW employs both adapter-based fusion and cross-attention for streaming multitask integration (Chen et al., 28 Jun 2024)).

3. Representative Model Architectures and Training Protocols

Modern SpeechLLM systems adopt highly modular architectures, characterized by the following principal components and training strategies:

• Speech Encoder: Most models utilize large, often frozen, self-supervised speech encoders (Whisper, WavLM, Conformer) for modality extraction (Meng et al., 13 Sep 2024, Peng et al., 24 Oct 2024). For discrete-token systems, codebooks (e.g., EnCodec, CosyVoice) segment intermediate representations.
• Adapter Layers and Modality Alignment:
  • Linear or Transformer-based adapters project speech encoder outputs to the LLM’s input dimension. LoRA or similar low-rank adapters enable parameter-efficient adaptation while freezing most LLM weights (Lakomkin et al., 2023, Meng et al., 13 Sep 2024).
  • CTC pretraining and blank filtering are used to compress sequence length and align speech tokens with text tokens (Wang et al., 2023).
  • Cross-attention or early fusion (e.g., T5-style cross-modal attention blocks) facilitates information mixing.
• Instruction and Multi-Task Fine-Tuning:
  • SpeechLLMs are typically trained via autoregressive next-token prediction over concatenated or interleaved sequences of speech embeddings/tokens, text context, and output targets (Tian et al., 21 Feb 2025, Huang et al., 2023).
  • Instruction fine-tuning is crucial for multitask generalization (ASR, SLU, summarization, QA). Multi-task and chain-of-thought (CoT) training improve compositionality and reasoning (Huang et al., 2023, Li et al., 29 Aug 2024).
  • Losses include cross-entropy over output spans, minimum WER for discriminative fine-tuning (Shivakumar et al., 25 Sep 2024), and auxiliary contrastive or CTC objectives during pretraining (Cuervo et al., 31 Mar 2024).
• Parameter-Efficient Modality Adaptation:
  • LoRA tuning in select layers scales efficiently; only 30–100M adapter parameters are required to adapt 7B+ LLMs for speech while retaining base text abilities (Lakomkin et al., 2023, Meng et al., 13 Sep 2024).
  • Mixture-of-experts and late fusion (e.g., MoLE-Llama) mix specialized PEFT modules to balance performance across modalities without catastrophic forgetting (Shen et al., 27 Oct 2024).
• Efficient Inference and Streaming:
  • Streaming integration, as in BESTOW and VocalNet, uses policy-driven read–write strategies and multi-token prediction for real-time, low-latency speech interaction (Chen et al., 28 Jun 2024, Wang et al., 5 Apr 2025).

4. Empirical Capabilities and Key Benchmarks

SpeechLLMs now reach or exceed prior state-of-the-art in the following tasks:

Task / Benchmark	Model/Method	Key Results
ASR (LibriSpeech)	Speech LLaMA, BESTOW, Qwen2-Audio	WER <3.5% on test-clean (Lakomkin et al., 2023, Chen et al., 28 Jun 2024)
Contextualized ASR	Speech LLaMA w/ context prompt	7.5% WER rel. reduction vs strong RNN-T (Lakomkin et al., 2023)
SLU (SLURP/Speech MASSIVE)	WHISMA, Qwen2-Audio	WHISMA: +26.6% rel F₁ over SOTA (Li et al., 29 Aug 2024, Choi et al., 18 Sep 2025)
Spoken QA (OpenAudioBench, LongSpeech-Eval)	VocalNet, FastLongSpeech	Latency ↓ > 2x at similar QA/BLEU (Wang et al., 5 Apr 2025, Guo et al., 20 Jul 2025)
Speech Generation (TTS Synthesis, Role-Playing)	MoLE-Llama, OmniCharacter	MOS ~3.0–4.2, speaker ID preserved (Shen et al., 27 Oct 2024, Zhang et al., 26 May 2025)
Speech Translation (LLM-ST)	LLM-ST (13B), CoT prompting	BLEU 36.4 (En→Zh), CER <8% (Huang et al., 2023)
L2 Proficiency Grading	Qwen2Audio-7B	RMSE 0.32, r >0.95 vs BERT/wav2vec2 (Ma et al., 27 May 2025)

Further, SpeechLLMs support:

• End-to-end contextual biasing (e.g., video title prompt).
• Multi-talker and instruction-following ASR in cocktail-party scenarios (Meng et al., 13 Sep 2024).
• Speaker and emotion conditioning for immersive, personality-aware agents (Zhang et al., 26 May 2025).
• Pseudo-label generation for semi-supervised ASR, yielding pseudo-label quality rivaling or surpassing human annotators (Prakash et al., 5 Jun 2025).
• Efficient long-speech handling via adaptive frame compression (Guo et al., 20 Jul 2025).

5. Scaling Laws, Data, and Resource Requirements

Systematic scaling studies reveal critical distinctions between speech- and text-based LLMs:

• Scaling Laws: Linguistic proficiency in speech-based models lags text models by three orders of magnitude in compute efficiency (to reach BLIMP 80% accuracy: text LLM needs 3×10²³ FLOP, speech SLMs need ≈10²⁶ FLOP) (Cuervo et al., 31 Mar 2024).
• Compute–Model–Data Allocation: Optimal scaling balances parameter increase and data size, although speech models benefit slightly more from model scale (α=β=~0.25 vs 0.35 for text).
• Synthetic Data: Massive TTS-generated corpora (e.g., 72k h sTinyStories) boost semantic scaling and downstream reasoning capacity.
• Tokenization: Fine-grained unit tokenization (e.g., HuBERT K=500) is superior for downstream reasoning compared to aggressive unigram compression.
• Cross-modal pretraining: Leveraging text-only LLM initialization or multitask text–speech alignment accelerates convergence and generalization (Cuervo et al., 31 Mar 2024, Choi et al., 18 Sep 2025).

6. Practical Limitations, Challenges, and Future Directions

The current landscape presents both open problems and methodological advances:

• Instruction Sensitivity and Robustness: SpeechLLMs exhibit significant variation in performance across semantically identical prompts due to instruction-following brittleness. Addressing this demands diverse prompt augmentation and meta-prompt tuning during training (Peng et al., 24 Oct 2024).
• Semantic Reasoning vs Acoustic Alignment: Joint adaptation to speech can degrade original deep text-reasoning capabilities. Proposed remedies include two-branch architectures and modular reasoning controllers (Peng et al., 24 Oct 2024).
• Long-Form Speech and Temporal Modeling: Frame-level audio yields prohibitively long sequences; recent advances such as iterative fusion and dynamic compression (FastLongSpeech) enable efficient context reduction with minimal accuracy loss (Guo et al., 20 Jul 2025).
• Label Scarcity and Cross-Modal Transfer: LALMs can attain strong SLU performance with text-only fine-tuning, and just 2–10% speech data suffices to close most of the gap, especially with curriculum or few-shot learning (Choi et al., 18 Sep 2025).
• Streaming and Real-Time Applications: Efficient architectures allow streaming multitask inference with manageable latency and resource profile (e.g., BESTOW, VocalNet MTP). However, streaming performance still exceeds human-level simultaneity (Chen et al., 28 Jun 2024).
• Benchmarking and Evaluation: Diverse and relevant benchmarks for understanding, reasoning, and generation—spanning perception, shallow and deep cognition, and free-form tasks—remain an active area, with tools like SLU-GLUE (Li et al., 29 Aug 2024), LongSpeech-Eval (Guo et al., 20 Jul 2025), and CharacterEval (Zhang et al., 26 May 2025).
• Future Research Directions: Anticipated advances include learned compression and attention in long-speech extractors, adaptive tokenization, further scaling of backbone LLMs, truly unified multi-modal LLMs (vision, text, speech), and fine-grained, human-aligned RLHF for spoken outputs (Peng et al., 24 Oct 2024, Yang et al., 26 Feb 2025).

7. Impact and Research Significance

SpeechLLMs have transformed paradigms for spoken language understanding and generation. They facilitate general-purpose, instruction-following models that support rich, context-dependent, and personality-aware interaction; enable zero-shot and cross-lingual transfer; and allow unified benchmarks and evaluation across modalities. As models scale and methods mature, SpeechLLMs are likely to drive future progress in conversational AI, multimodal reasoning, language education, accessibility, and real-time speech-driven applications (Tian et al., 21 Feb 2025, Ma et al., 27 May 2025, Zhang et al., 26 May 2025).

By systematically integrating advances in model architecture, training regimes, adaptive tokenization, and multitask evaluation, the field is converging toward robust, generalizable, and human-aligned SpeechLLMs. Open challenges remain in efficiency, multilinguality, deep reasoning, and multimodal scaling; addressing these will be central to the continued evolution of spoken language intelligence (Peng et al., 24 Oct 2024, Guo et al., 20 Jul 2025, Yang et al., 26 Feb 2025, Chen et al., 28 Jun 2024).