Speech LLM: Architectures & Applications
- Speech LLM is an architectural paradigm combining large language models with speech processing modules to handle transcription, translation, and dialogue tasks.
- It integrates speech encoders/decoders with modality adapters or unified encoders to bridge speech-text representation gaps and enable real-time, end-to-end processing.
- It supports a wide range of tasks from ASR and spoken language understanding to speech synthesis and direct speech-to-speech translation, emphasizing latency reduction and contextual reasoning.
Speech LLM designates a family of systems that extend LLMs to speech tasks by coupling an LLM backbone with speech encoders, speech decoders, or both, so that the model can consume speech, generate speech, or operate end-to-end across speech recognition, spoken dialogue modeling, spoken language understanding, speech translation, and speech generation. In the recent literature, the term is used both for decoder-only architectures used for automatic speech recognition and for broader multimodal systems that integrate speech and text-based foundation models in a unified framework (Seide et al., 2024, Li et al., 2024). This suggests that “Speech LLM” is best understood as an architectural paradigm rather than a single task definition.
1. Scope and conceptual variants
One recurring definition treats a Speech LLM as a speech-capable, decoder-only LLM used for automatic speech recognition, where a transformer decoder of the kind used in Llama-2 is coupled with an acoustic encoder and trained to transcribe speech in real time without changing the core LLM architecture (Seide et al., 2024). A second definition emphasizes a unified framework that integrates speech and text-based foundation models so that the model can perform spoken language understanding tasks in a prompt-driven way rather than through task-specific heads (Li et al., 2024). A third usage places the LLM at the center of a direct speech-to-speech pipeline, where the model generates speech-derived semantic tokens rather than only text (Arya et al., 22 Jan 2026).
The field therefore spans at least three broad configurations. In speech-input systems, the LLM consumes projected acoustic representations and generates text for ASR, SLU, dialogue state tracking, or translation (Li et al., 2024, Ghazal et al., 10 Oct 2025). In speech-output systems, the LLM remains text-centric while a streaming speech module vocalizes its output with low latency (Dekel et al., 2023, Shikhar et al., 6 Mar 2025). In speech-to-speech systems, the LLM acts as a semantic engine between source speech and target speech, while a timbre-controlled vocoder restores speaker identity in the output waveform (Arya et al., 22 Jan 2026).
A related distinction separates tightly integrated models from modular ones. Some work argues for modifying neither the base LLM nor its linguistic behavior, instead wrapping it with a lightweight streaming TTS system that is “LLM-agnostic” and preserves the capabilities of the base model (Shikhar et al., 6 Mar 2025). Other work argues that stronger performance comes from explicitly aligning speech and text representations inside the model, either through neural adapters, optimal transport losses, or unified encoders (Liu et al., 13 Mar 2025, Kim et al., 1 Jun 2025).
2. Architectural patterns and interfaces
A mainstream solution is to connect a well-trained speech encoder and LLM with a neural adapter (Fan et al., 2024). In practice, recent systems implement that interface in markedly different ways. WHISMA combines the speech encoder from Whisper with the Llama-3 LLM, uses a modality aligner with two 1D CNN layers, a bottleneck adaptor, and a linear projection, freezes Whisper and the base Llama-3 weights, and trains only the aligner and LoRA parameters (Li et al., 2024). TESU-LLM uses SeamlessM4T-v2-large as a unified text-speech encoder, a two-layer MLP projector of about 13M parameters, and a frozen LLaMA-3.1-8B-Instruct backbone; during training it uses only text, while at inference it replaces the text pathway with the speech pathway of the same unified encoder (Kim et al., 1 Jun 2025).
Another pattern uses the LLM’s own vocabulary and embedding matrix as the interface. LegoSLM trains the speech encoder to generate CTC posteriors over the LLM vocabulary and reconstructs pseudo-audio embeddings by computing a weighted sum of the LLM input embeddings, which are then concatenated with text embeddings in the LLM input space (Ma et al., 16 May 2025). CoT-ASR introduces a related CTC-guided Modality Adapter that uses CTC non-blank token probabilities to weight LLM embeddings, while a gated residual branch adds transformed encoder features (Deng et al., 1 Apr 2026).
A third pattern emphasizes internal alignment rather than only input mapping. AI-STA explicitly aligns speech and text representations at selected layers within LLMs, uses optimal transport to quantify fine-grained representation discrepancies, and uses cross-modal retrieval to identify the layers that are best suited for alignment (Liu et al., 13 Mar 2025). In audio-visual settings, MMS-LLaMA uses an early AV-fusion module, an audio-visual speech Q-Former that dynamically allocates tokens based on input duration, and a speech rate predictor that adjusts token allocation according to speaking speed (Yeo et al., 14 Mar 2025). These designs all target the same underlying problem: speech and text do not naturally occupy the same sequence length, timing structure, or latent geometry.
3. Major task families
The task surface of Speech LLMs is now wide enough that the term no longer implies a single benchmark or output format. Representative systems in the literature cover the following families.
| Task family | Representative systems | Reported capability |
|---|---|---|
| Streaming ASR and transcription | Speech ReaLLM (Seide et al., 2024), CoT-ASR (Deng et al., 1 Apr 2026) | Real-time continuous ASR without explicit end-pointing; contextual reasoning plus transcription in a single pass |
| Audio-visual and spoken understanding | MMS-LLaMA (Yeo et al., 14 Mar 2025), WHISMA (Li et al., 2024), Spoken DST with full spoken history (Ghazal et al., 10 Oct 2025) | Efficient AVSR with 3.5 tokens per second; zero-shot SLU; end-to-end spoken dialogue state tracking |
| Speech output and dialogue generation | LLM2Speech (Dekel et al., 2023), LLMVoX (Shikhar et al., 6 Mar 2025), SLIDE (Lu et al., 1 Jan 2025) | Speech synthesis while text is being generated; autoregressive streaming TTS for any LLM; spontaneous spoken dialogue generation |
| Direct speech-to-speech translation | DS2ST-LM (Arya et al., 22 Jan 2026) | Single-stage direct S2ST with timbre-aware synthesis, extensible to multiple language pairs |
| Diarization and speaker reasoning | Unified diarization+ASR Speech LLM (Saengthong et al., 26 Jun 2025), SpeakerLLM (Nam et al., 14 May 2026), “Just ASR + LLM?” (Wu et al., 2024) | Joint diarization and ASR; speaker profiling and verification reasoning; analysis of speaker-awareness limits |
| Training and data infrastructure | TESU-LLM (Kim et al., 1 Jun 2025), SpeechDialogueFactory (Wang et al., 31 Mar 2025) | Speech-capable LLM training with only text; large-scale generation of natural speech dialogue data |
Across these task families, the generative interface remains central. Speech ReaLLM interleaves speech embeddings and word tokens in one sequence and uses a BLANK mechanism inspired by RNN-T so that a decoder-only multimodal LLM can learn the flow of time (Seide et al., 2024). WHISMA uses the same LLM backbone to perform ASR, intent classification, slot filling, spoken QA, and speech-based instruction following through prompt variation rather than task-specific heads (Li et al., 2024). The SpokenWOZ work goes further by showing that a Speech-LLM can take the full spoken conversation as input and directly generate the dialogue state as JSON, without textual history (Ghazal et al., 10 Oct 2025).
Speech generation systems extend the concept in the opposite direction. LLM2Speech synthesizes speech while text is being generated by an LLM, exploits hidden embeddings of the LLM as informative semantic context, and maintains the teacher’s quality while reducing latency (Dekel et al., 2023). SLIDE divides labor explicitly: an LLM generates the textual content of a spoken dialogue, a duration predictor produces spoken phoneme sequences, and a speech LLM vocalizes the dialogue while retaining naturalistic speech patterns such as turn-taking and non-verbal vocalizations (Lu et al., 1 Jan 2025).
4. Streaming, latency, and the flow of time
A defining frontier for Speech LLMs is real-time behavior. Speech ReaLLM is presented as the first decoder-only ASR architecture designed to handle continuous audio without explicit end-pointing, and an 80M model achieves WERs of 3.0% and 7.4% on LibriSpeech “test” in real time, without an external LM or auxiliary loss (Seide et al., 2024). Its core idea is to generate after every input token received in real time, often emitting a BLANK symbol rather than a word, so that the LLM learns to represent and reproduce the flow of time (Seide et al., 2024).
Speech output has an analogous latency problem. LLM2Speech addresses the gap between token-by-token text generation and non-streaming text-to-speech by synthesizing speech while text is being generated, limiting exposure to future context and exploiting the hidden embeddings of the LLM (Dekel et al., 2023). LLMVoX pushes the same idea into a deployment-oriented architecture: a 30M-parameter, LLM-agnostic, autoregressive streaming TTS system with a multi-queue token streaming design, support for seamless, infinite-length dialogues, and end-to-end latency of about 475 ms with LLaMA-3.1-8B, Whisper small, and chunk size 40 (Shikhar et al., 6 Mar 2025).
Not all low-latency work is purely streaming. CoT-ASR reframes LLM-based ASR as contextual reasoning followed by transcription in a single pass, and reports a relative reduction of 8.7% in WER and 16.9% in entity error rate compared to standard LLM-based ASR (Deng et al., 1 Apr 2026). This result is not a streaming claim, but it highlights a related design principle: once speech is injected into an LLM, the model need not be limited to direct speech-to-text mapping and can instead exploit the generative capacity of the LLM before producing the final transcript (Deng et al., 1 Apr 2026).
Taken together, these systems show two distinct but related temporal strategies. One teaches the model when to wait and when to emit, as in real-time ASR (Seide et al., 2024). The other reduces the boundary between reasoning and speaking, as in streaming TTS and speech synthesis during text generation (Dekel et al., 2023, Shikhar et al., 6 Mar 2025).
5. Evaluation ecosystems and empirical lessons
Evaluation in this area is fragmented because different Speech LLMs solve different mixtures of recognition, understanding, generation, and conversation. WHISMA emphasizes zero-shot spoken language understanding and reports a relative gain of 26.6% on zero-shot slot filling over the current state-of-the-art model on SLURP, while on the task-agnostic SLU-GLUE benchmark it outperforms Qwen-Audio with a relative gain of 33.0% (Li et al., 2024). In spoken dialogue state tracking, feeding the full spoken conversation into a Speech-LLM yields the highest performance among similar-size models on SpokenWOZ, with 39.32% Joint Goal Accuracy after post-processing (Ghazal et al., 10 Oct 2025).
Other evaluation suites focus on realistic conversational structure. The MLC-SLM Challenge measures ASR with oracle segmentation and joint diarization-plus-recognition without oracle information. For Task II, the proposed unified speech LLM reports a 54.87% relative improvement in tcpWER/tcpCER over the baseline while using a smaller LLM backbone (Saengthong et al., 26 Jun 2025). In AVSR, MMS-LLaMA reports state-of-the-art performance on LRS3 with a WER of 0.72% while using only 3.5 tokens per second, and also reports an 86% reduction in token usage and a 35.7% reduction in FLOPs relative to the previous multimodal speech LLM framework (Yeo et al., 14 Mar 2025).
A recurring misconception is that strong spoken dialogue QA performance automatically demonstrates speaker awareness. The study “Just ASR + LLM?” shows a different pattern: on Gaokao and the “What Do You Like?” dataset, current SpeechLLMs obtain much higher accuracy on context-based questions than on identity-critical questions, which can only be answered reliably with correct speaker identification (Wu et al., 2024). The same work reports that a text-only Llama3 system with explicit speaker gender labels reaches 91.8% on identity-critical questions, which indicates that the missing ingredient is not general reasoning but reliable access to speaker identity (Wu et al., 2024). This suggests that benchmark choice can overstate progress if content comprehension is easier than genuine speaker-conditioned understanding.
6. Data, limitations, and research directions
Speech LLM development remains strongly constrained by data. SpeechDialogueFactory begins from the observation that high-quality speech dialogue datasets are crucial, while human recordings incur high costs and privacy concerns and synthetic approaches often lack conversational authenticity (Wang et al., 31 Mar 2025). Its pipeline combines metadata generation, dialogue scripting, paralinguistic-enriched utterance simulation, natural speech synthesis with voice cloning, content evaluation, and speech evaluation, and its generated dialogues are reported to achieve quality comparable to human recordings while significantly reducing production costs (Wang et al., 31 Mar 2025). This is important because many downstream capabilities of Speech LLMs—turn-taking, prosody, emotional progression, dialogue coherence—depend on the properties of the training corpus rather than on the adapter alone.
At the same time, recent work weakens the assumption that speech-capable LLMs must always be trained on paired speech-text data. TESU-LLM trains only a lightweight projector while keeping a unified encoder and an 8B instruction-tuned LLM frozen, and despite being trained exclusively on text it achieves strong performance on speech-related benchmarks and competitive VoiceBench results (Kim et al., 1 Jun 2025). A plausible implication is that future Speech LLM stacks may separate speech representation learning, semantic alignment, and instruction following more aggressively than current end-to-end systems.
Several open problems remain explicit in the literature. Direct S2ST systems still face instability in semantic-acoustic alignment when parallel speech data is scarce, difficulty in preserving speaker identity, and limited multilingual scalability (Arya et al., 22 Jan 2026). Speaker-sensitive reasoning is still weaker than content-based reasoning in many dialogue settings (Wu et al., 2024). Even when speaker information is the main target, general audio-LLMs lag behind specialized architectures; SpeakerLLM addresses this with a hierarchical speaker tokenizer, profile-level evidence, recording-condition understanding, and structured verification reasoning (Nam et al., 14 May 2026). This suggests that the next phase of Speech LLM research may be less about merely attaching an encoder to an LLM and more about factorizing speech into semantic, speaker, temporal, and environmental components that can be reasoned about separately but generated jointly.
In that sense, the contemporary Speech LLM is not a single model type but a design space. Some systems treat the LLM as a decoder for continuous audio streams, some as a semantic planner for speech synthesis, some as a multilingual semantic engine for direct speech-to-speech translation, and some as a reasoning layer over speaker identity or dialogue state. What unifies them is the attempt to make LLMs operate over speech with the same generality they already exhibit over text, while preserving the temporal, paralinguistic, and interactive structure that speech contributes.
