Speech-Based Large Language Models

• Speech-based LLMs are AI systems that unify speech and language processing via text-based, latent-representation, and audio-token integration, enabling tasks like transcription and translation.
• They employ decoder-only Transformer architectures with techniques such as LoRA for parameter-efficient fine-tuning, achieving reduced model size and enhanced performance.
• Ongoing research addresses multimodal alignment, adversarial robustness, and improved training regimes to advance state-of-the-art speech understanding and synthesis.

Speech-based LLMs are a class of AI systems that perform unified processing of speech and language, enabling models to accept, generate, and reason over spoken language in a variety of tasks including transcription, translation, understanding, synthesis, and dialogue. This field synthesizes approaches from automatic speech recognition (ASR), speech synthesis, and natural language understanding, empowering LLMs with direct multimodal speech capabilities and, in advanced systems, supporting seamless speech-to-speech and mixed-modality interaction.

1. Speech-Language Integration Paradigms

Integration of speech with LLMs falls along three principal paradigms: text-based, latent-representation-based, and audio-token-based integration (Yang et al., 26 Feb 2025). In text-based approaches, speech is transcribed to text using an ASR front-end, after which the LLM processes the resulting textual stream (e.g., cascaded ASR→LLM, or LLM rescoring via n-best list interpolation). While simple and widely used, text-based integration risks information loss—prosody, emotion, and paralinguistic information present in the speech signal are not preserved in the textual stream, and the pipeline suffers from error propagation and higher latency.

Latent-representation-based methods introduce a speech encoder that yields continuous representations. A modality adaptation module (typically convolutional downsampling, CTC compression, or Q-Former) aligns high-dimensional, long speech sequences to the coarser semantic space of the LLM. These representations are concatenated or projected and consumed by either a partially or fully frozen LLM as seen in models such as Speech-LLaMA (Wu et al., 2023), LLM-ST (Huang et al., 2023), and LLaMA-Omni (Fang et al., 10 Sep 2024). This approach enables deep integration and supports end-to-end training; however, it requires careful balancing to avoid overshadowing text tokens in the LLM attention mechanism (Peng et al., 24 Oct 2024).

Audio-token-based approaches discretize the speech waveform into tokens (semantic and/or acoustic) with neural codecs or clustering. The LLM processes these tokens analogously to text, enabling unified spoken LLMing, cross-modal tasks, and speech synthesis by re-synthesizing waveforms from LLM-generated tokens (Hao et al., 2023).

Paradigm	Step	Representative Models
Text-based	ASR/TTS + LLM, n-best rescoring, GER (H2T)	AudioGPT, HuggingGPT
Latent-representation	Speech encoder + modality adapter + LLM	Speech-LLaMA, LLM-ST, LLaMA-Omni
Audio-token-based	Speech→tokens→LLM→tokens→TTS/vocoder	SpeechGPT, AudioPaLM, VALL-E

2. Decoding Architectures and Training

The dominant architectural choice in modern speech-based LLMs is the decoder-only Transformer, enabling unified autoregressive modeling for both speech (as latent or token embeddings) and text (Wu et al., 2023, Huang et al., 2023, Fang et al., 10 Sep 2024). Unlike encoder-decoder models, this offers parameter efficiency by fusing representations and generation in a single stack, resulting in up to 40% parameter reduction in Speech-LLaMA compared to classical seq2seq baselines (Wu et al., 2023).

LoRA (Low-Rank Adaptation) is widely used for parameter-efficient fine-tuning—inserting a minimal number of trainable matrices within specific linear modules (Wu et al., 2023, Meng et al., 13 Sep 2024, Peng et al., 23 Oct 2024). LoRA adapts the LLM for cross-modal fusion without overfitting or catastrophic forgetting, a challenge addressed by joint speech-text supervised fine-tuning (VoiceTextBlender (Peng et al., 23 Oct 2024)) and multi-task behavior imitation with speech-text interleaving (MTBI (Xie et al., 24 May 2025)).

Training regimes include self-supervised pretraining on large-scale unlabeled speech, supervised fine-tuning (SFT) on labeled speech-text pairs for ASR/ST/QA, and reinforcement learning (e.g., Reinforced Behavior Alignment RBA), often employing self-synthesized multimodal data aligned to a teacher LLM.

Key objectives formulated include masked modeling, next-token prediction (with cross-entropy), minimum word error rate (MWER) loss for discriminative rescoring (Shivakumar et al., 25 Sep 2024), and explicit reward-based RL criteria (Liu et al., 25 Aug 2025). Modality adapters handle sequence length and embedding dimension mismatches. Advanced designs use chunking, upsampling (for streaming), and non-autoregressive decoding to achieve low latency in speech-to-speech systems (Fang et al., 10 Sep 2024).

3. Core Applications and Benchmarks

Speech-based LLMs span a growing set of applications:

• Automatic Speech Recognition (ASR): Models like LLM-ST, Speech-LLaMA, and MT-LLM deliver competitive or state-of-the-art WER in multilingual, long-form, and multi-talker transcription (Wu et al., 2023, Huang et al., 2023, Meng et al., 13 Sep 2024).
• Speech-to-Text and Speech Translation (ST): Joint modeling supports direct speech translation, document-level refinement, and hybrid correction workflows (Huang et al., 2023, Dou et al., 25 Jan 2025).
• Speech Synthesis (TTS): Architectural coupling of LLMs with TTS models such as VALL-E achieves substantial improvements in naturalness and speaker similarity (Hao et al., 2023). Speech-informed dialogue generation produces linguistically and paralinguistically rich outputs (Zhou et al., 2023).
• Speech Understanding and Spoken QA: Unified models tackle slot filling under noisy ASR (Sun et al., 2023), spoken language comprehension for education (Peng et al., 2023), and joint spoken question answering (Liu et al., 25 Aug 2025).
• Multimodal Reasoning and Language Learning: Benchmarks (SAGI (Bu et al., 17 Oct 2024)) and surveys (Peng et al., 24 Oct 2024) delineate intent, semantic reasoning, and the ability to incorporate paralinguistic and non-semantic cues (prosody, emotion, context) into inference.

Standard evaluation metrics include WER (for ASR), BLEU/COMET (for ST), SLU-F1 (for slot filling), and task-specific scores. Emerging benchmarks like SAGI introduce hierarchical evaluation—from basic ASR to "Speech AGI" (integrating abstract and non-semantic knowledge).

4. Generalization, Robustness, and Systemic Challenges

Significant challenges persist in instruction-following and prompt generalization. Models are often found to be instruction-sensitive—LLM "dormancy" occurs when audio embeddings dominate attention and suppress the effect of text prompts, undermining the model's reasoning and ability to follow external instructions (Peng et al., 24 Oct 2024). Catastrophic forgetting, where speech-aligned LLMs degrade on text-only tasks, is mitigated by joint, single-stage SFT and multi-task imitation (Peng et al., 23 Oct 2024, Xie et al., 24 May 2025).

Robustness to adversarial or manipulative input (e.g., gaslighting attacks (Wu et al., 24 Sep 2025)) is a novel area of concern. Five manipulation strategies—Anger, Cognitive Disruption, Sarcasm, Implicit, and Professional Negation—can induce behavioral and performance failures, with average accuracy drops of 24.3% across diverse tasks and systems. Added acoustic noise amplifies these vulnerabilities, highlighting the need for robust, multi-modal instruction resilience.

Data scarcity and modality gaps also hinder generalization, especially for zero-shot reasoning or tasks with minimal annotated speech data. Self-synthesis and RL-based alignment (RBA (Liu et al., 25 Aug 2025)), behavior imitation, and speech-text interleaving (Xie et al., 24 May 2025) offer viable solutions that depend on large-scale, teacher-aligned or automatically generated multimodal corpora.

5. Future Directions and Roadmap

Current research delineates a roadmap toward superhuman speech understanding (Bu et al., 17 Oct 2024). The five-level hierarchy—from basic ASR, through paralinguistic and non-semantic comprehension, to expert and generalist models—summarizes the evolution from recognition pipelines to speech AGI capable of integrated semantic and acoustic reasoning. Advancements in acoustic feature modeling (e.g., via Q-Former, CTC-compression, or adapter innovations), curated large-scale benchmarks (SAGI), and unified training paradigms (instruction tuning, RLHF/DPO) are expected to close the gap between modality-specialized and generalist foundation models.

Open research topics include:

• Improved modality alignment via better adapters and fusion strategies (Peng et al., 24 Oct 2024).
• End-to-end joint architectures that preserve both semantic and non-semantic cues (Bu et al., 17 Oct 2024).
• Broader support for prompt variability, cross-lingual transfer, and multi-turn, mixed-modal human-computer dialogue (Peng et al., 23 Oct 2024).
• Defensive strategies against adversarial input, leveraging behavioral regularization and multi-modal prompt control (Wu et al., 24 Sep 2025).
• Scaling speech-LMs with continual, parameter-efficient pretraining to new and underrepresented languages, dialects, and domains.

6. Broader Impacts and Significance

Speech-based LLMs have established state-of-the-art results across multiple domains—multilingual translation, robust ASR, slot filling under high ASR error, and natural TTS generation. By directly fusing acoustic and linguistic knowledge and supporting context- and instruction-aware inference, these systems pave the way for seamless, efficient human-computer interaction across modalities. Industrial-scale models now match or surpass cascaded ASR+LLM+TTS systems in both content fidelity and style, while achieving low-latency end-to-end speech pipelines (Fang et al., 10 Sep 2024).

At the same time, persistent vulnerabilities—particularly under adversarial or ambiguous prompts—demonstrate that reliability, alignment, and comprehensive human-like understanding remain open problems. The field continues to move toward unified, robust, and general-purpose foundation models that integrate all forms of language and communication.