Speech Language Models: Advances & Challenges

• SLMs are speech-domain models that integrate linguistic semantics with acoustic cues like prosody and speaker traits for authentic spoken interaction.
• A refined taxonomy categorizes SLMs into pure, joint speech–text, and speech-aware text models, each addressing different modalities and integration needs.
• Recent advances in tokenization, alignment strategies, and evaluation benchmarks drive improvements in ASR, empathetic dialogue, and cross-modal reasoning.

Searching arXiv for recent and foundational papers on Speech LLMs to ground the encyclopedia entry. arXiv search query: "speech LLMs spoken LLMs survey 2025" Speech LLMs (SLMs) are the speech-domain analogue of modern LLMs, but the term is used broadly across several research traditions. In the narrowest sense, an SLM is an autoregressive model over tokenized speech sequences; in a broader and now common sense, it is a speech-aware LLM that accepts spoken input, reasons over audio-conditioned representations, and generates text and/or speech. Across these formulations, SLMs are designed to move beyond cascaded ASR$\rightarrow$LLM$\rightarrow$TTS pipelines by modeling not only linguistic content but also prosody, timbre, speaker traits, non-speech vocalizations, and environmental signals that materially affect spoken interaction (Arora et al., 11 Apr 2025, Zhou et al., 26 Oct 2025, Chen et al., 24 Jul 2025).

1. Taxonomic scope and defining properties

The contemporary literature treats SLMs as a family rather than a single architecture. A widely used taxonomy distinguishes pure speech LLMs, joint speech–text LLMs, and speech-aware text LLMs. Pure SLMs model distributions over discretized speech units directly; joint speech–text models learn over both modalities in a unified generative framework; speech-aware text LMs retain a text LLM backbone and add speech encoders plus modality adapters so that speech can condition textual reasoning and response generation (Arora et al., 11 Apr 2025).

Family	Modeling target	Representative systems
Pure speech LLMs	$p(\text{speech})$ over tokenized speech	GSLM, AudioLM, SoundStorm, TWIST
Speech+text LLMs	Joint modeling of speech and text	SpiRit-LM, Moshi, SpeechGPT, GLM-4-Voice
Speech-aware text LLMs	$p(\text{text}\mid \text{speech}, \text{text})$ or related conditioning	SALMONN, Qwen-Audio, Qwen2-Audio, DeSTA, DeSTA2

This taxonomy also clarifies what SLMs are not. Text-only LLMs process symbolic text and therefore cannot condition on speaking style; identical transcripts are treated identically. Cascaded speech systems can be strong, but they may discard paralinguistic nuance when speech is collapsed into text or metadata. General audio-LLMs benchmark broad audio comprehension, including music and environmental sounds, yet often do not target spoken conversational intelligence, empathy, or speech-conditioned dialogue generation. By contrast, SLMs are explicitly concerned with spoken interaction as a first-class modality, including instruction following from audio, speech-native response planning, and in many cases speech-to-speech generation (Zhou et al., 26 Oct 2025, Lu et al., 2024).

A second defining property is the need to represent two tightly coupled layers of information: linguistic semantics and acoustic realization. GOAT-SLM makes this distinction explicit by separating semantic reasoning from acoustic realization, while EchoMind frames the practical consequence directly: identical words delivered calmly, angrily, breathlessly, or in a noisy street can warrant different responses (Chen et al., 24 Jul 2025, Zhou et al., 26 Oct 2025). This suggests that the core problem in SLM design is not merely audio ingestion, but controlled integration of lexical, prosodic, paralinguistic, and contextual evidence into a coherent conversational policy.

2. Representations, tokenization, and architectural patterns

A central design choice in SLMs is the representation of speech. The surveyed literature spans raw waveforms, mel-spectrograms, continuous self-supervised features, supervised encoders such as Whisper and USM, and neural audio codec latents. Discrete unit modeling remains especially important because it converts speech into sequences compatible with decoder-only language modeling, but the choice of units strongly constrains semantic density, controllability, bitrate, and downstream synthesis quality (Arora et al., 11 Apr 2025).

Recent work on LLM-centric SLMs shows that speech tokenization is not a minor implementation detail. "Speech-LLMs with Decoupled Tokenizers and Multi-Token Prediction" compares coupled, semi-decoupled, and fully decoupled tokenizers and reports that fully decoupled tokenization substantially improves cross-modal alignment, ASR intelligibility, and speaker consistency. In that framework, speech tokens may be factorized into semantic/content tokens $\bar{s}$ and acoustic-detail tokens $\tilde{s}$, with separate heads

$$P(\bar{s}_{i+1}\mid X_{\le i})=\mathrm{softmax}(W^{\bar{}}_{\text{speech}}\cdot h_i),\qquad P(\tilde{s}_{i+1}\mid X_{\le i})=\mathrm{softmax}(W^{\tilde{}}_{\text{speech}}\cdot h_i),$$

and multi-token prediction compresses the speech stream by predicting several speech tokens from one hidden state, yielding up to $12\times$ faster decoding and reducing WER from $6.07$ to $3.01$ in the reported setup (Fan et al., 14 Jun 2025).

Textless SLMs expose a related but distinct tokenizer problem. DC-Spin introduces Double-Codebook Speaker-invariant Clustering, with a small primary codebook for downstream tokens and a large auxiliary codebook used during training to improve phonetic granularity. The paper argues that tokens that are easily modeled by an n-gram LM or strongly aligned with phonemes provide strong downstream performance, and it also presents a chunk-wise streaming variant "without retraining and degradation" (Chang et al., 2024). Flow-SLM departs from discrete-only formulations by jointly predicting semantic tokens and a continuous acoustic frame embedding with conditional flow matching, thereby giving the LLM access to acoustic context rather than delegating all acoustic detail to a separate vocoder (Chou et al., 12 Aug 2025).

Architecturally, many speech-aware SLMs reuse a pretrained LLM as a shared semantic core and add speech-specific modules around it. GOAT-SLM organizes this as Listen, Think, Write, Speak, and Flow-Matching; the top language layers are reused with different heads for text and speech, yielding a dual-modality head architecture that decouples linguistic modeling from acoustic realization (Chen et al., 24 Jul 2025). Other systems employ heterogeneous adapters rather than speech-token decoders. The dual-information SLM for emotional conversations uses a paralinguistic adapter that produces utterance-level soft prompts and a linguistic adapter that produces content-aligned embeddings at 10 Hz, explicitly separating stable utterance-level affect from time-varying linguistic content (Wang et al., 11 Aug 2025). Still other variants extend a TTS-pretrained SLM to singing voice synthesis through multi-stream token prediction over score conditions and audio token streams at 50 frames per second, followed by conditional flow matching in mel space (Zhao et al., 16 Dec 2025).

3. Objectives, alignment strategies, and scaling behavior

At the objective level, pure speech language modeling largely follows the autoregressive paradigm familiar from text:

$\rightarrow$0

The survey presents this as the canonical pretraining objective for tokenized speech, and the scaling study "Scaling Properties of Speech LLMs" applies the same general scaling-law machinery used for text LLMs to speech-only models trained on HuBERT-derived units (Arora et al., 11 Apr 2025, Cuervo et al., 2024).

Speech-aware SLMs, however, rarely rely on this objective alone. Their central difficulty is speech–text alignment: speech embeddings are longer, more variable, and richer in paralinguistic structure than LLM text embeddings. DeSTA addresses this gap through descriptive speech-text alignment: speech is paired with rich captions describing transcription, emotion, gender, pitch, volume, and speaking speed, and the model is trained to generate these captions from adapted Whisper features plus transcript context (Lu et al., 2024). DeSTA2 simplifies the alignment regime further by constructing seed transcripts with metadata and asking the frozen text LLM a single question—“What can you hear from the audio?”—then training only a small modality adapter so that the end-to-end SLM reproduces the LLM’s own descriptive style. In the reported implementation, Whisper-small and Llama3-8B-Instruct are frozen and only a 22.3M-parameter adapter is trained (Lu et al., 2024).

Other works attack the modality gap more explicitly. OTReg formulates speech–text alignment as entropic regularized optimal transport between transformed speech embeddings and transcript embeddings, adding a transport-derived regularizer and an OT-based compression step during stage-two training. The method introduces no additional learnable parameters and is designed to make speech embeddings more text-like in the LLM embedding space (Xu et al., 11 Aug 2025). The dual-information emotional-conversation model instead treats disentanglement as the primary challenge: Equivalence Replacement Regularization and controlled randomness are used so that linguistic and paralinguistic adapters learn complementary representations without collapsing into task-specific vectors (Wang et al., 11 Aug 2025).

Generative speech modeling has also adopted flow-based objectives. Flow-SLM defines a joint loss

$\rightarrow$1

combining multi-token prediction of semantic units with conditional flow matching for continuous acoustic embeddings (Chou et al., 12 Aug 2025). In singing adaptation, conditional flow matching is used after LM token prediction to refine mel-spectrograms, explicitly compensating for artifacts that arise when a speech-pretrained codec decoder is used directly for singing (Zhao et al., 16 Dec 2025).

Scaling results qualify the feasibility of speech-only learning. The scaling paper reports that linguistic performance in speech-only SLMs scales predictably with pretraining loss, but up to three orders of magnitude more slowly than in text LLMs for comparable downstream improvements before saturation. It further reports that coarser unigram compression improves upstream loss scaling while degrading downstream semantics, and that synthetic speech data with short, semantically self-contained narratives improves semantic scaling (Cuervo et al., 2024). A plausible implication is that textless speech modeling remains viable, but current methods are far less compute-efficient than text-based or aligned multimodal alternatives.

4. Functional capabilities and application domains

The application space of SLMs now extends well beyond transcription. The survey and task-specific papers collectively position SLMs as general systems for ASR, speech translation, spoken question answering, dialogue, TTS, speech enhancement, speaker-aware generation, emotional conversation, and full-duplex spoken interaction (Arora et al., 11 Apr 2025). Several recent works also show that SLMs are increasingly evaluated on whether they can respond to speech in ways that are socially and pragmatically appropriate, rather than merely correct at the transcript level.

Empathy and paralinguistic awareness are a prominent example. EchoMind defines SLMs as end-to-end systems that “listen to speech, understand it, reason over both what is said and how it is said, and then speak back,” and constructs a benchmark in which semantically neutral scripts are rendered in controlled vocal styles so that response differences can be attributed to delivery rather than transcript content (Zhou et al., 26 Oct 2025). The benchmark covers speaker information, physiological state, emotion, volume, speed, non-verbal expressions, and environmental information. This framing treats empathetic response generation as the endpoint of a pipeline comprising content understanding, vocal-cue perception, integrated reasoning, and speech-conditioned response generation.

Style- and speaker-aware generation form a second major application cluster. GOAT-SLM targets proactive adaptation to emotion, dialect, age, and non-speech vocalizations, and evaluates these capacities in both text and speech outputs (Chen et al., 24 Jul 2025). "Speech-LLMs with Decoupled Tokenizers and Multi-Token Prediction" introduces a speaker-aware generation paradigm and RoleTriviaQA, a role-playing knowledge QA benchmark that measures both textual answer quality and speaker consistency in speech responses (Fan et al., 14 Jun 2025). StyleBench, in turn, focuses on conversational speaking style control and asks whether SLMs can modulate emotion, speed, volume, and pitch intensity across multi-turn dialogue while preserving semantic content (Zhao et al., 8 Mar 2026).

A third application class concerns broader generative transfer. "Adapting Speech LLM to Singing Voice Synthesis" shows that a 1.7B-parameter TTS-pretrained SLM can be adapted to singing voice synthesis using a 135-hour synthetic singing corpus, with a pipeline of score tokenization, multi-stream token prediction, conditional flow matching, and a HiFi-GAN vocoder (Zhao et al., 16 Dec 2025). Although singing is not conversation, the paper is significant because it demonstrates that discrete-token SLM backbones can transfer to temporally structured, pitch-controlled generation outside ordinary speech.

The literature also increasingly treats explicit reasoning as an SLM capability in its own right. Speech World Model proposes a graph-based modular system in which speech understanding is factorized into World Model Activation, Theory of Mind, Speech Act, and Pragmatic Intent, with causal edges among these modules. Posterior traces from this graph are then fed to an instruction-tuned LLM to produce both a causal analysis and a user-facing response (Zhou et al., 5 Dec 2025). This suggests a shift from monolithic audio-to-text behavior toward more transparent, state-based reasoning over speech.

5. Evaluation regimes and the empirical state of the field

The benchmark landscape reflects the breadth of current SLM research. Dynamic-SUPERB and AIR-Bench-Chat target general speech understanding and open-ended instruction following; EchoMind targets empathetic dialogue under controlled vocal manipulations; TELEVAL measures semantic intelligence plus paralinguistic and speaker characteristic awareness; StyleBench focuses on multi-turn style intensity control; EMIS evaluates emotion recognition under semantic–prosodic incongruence; and layer-wise minimal-pair probing examines what grammatical and conceptual information is linearly decodable from speech encoders (Lu et al., 2024, Zhou et al., 26 Oct 2025, Chen et al., 24 Jul 2025, Zhao et al., 8 Mar 2026, Corrêa et al., 29 Oct 2025, He et al., 19 Sep 2025).

Benchmark or evaluation	Primary focus	Representative measures
Dynamic-SUPERB, AIR-Bench-Chat	General speech understanding, instruction following	Accuracy, GPT-4 agreement
EchoMind	Empathetic dialogue from content + vocal cues	WER, ACC, BLEU/ROUGE/METEOR/BERTScore, C1–C4, EmoAlign, VES
TELEVAL	Semantic intelligence and paralinguistic awareness	Accuracy, CER, DNSMOS, emotion correctness
StyleBench	Multi-turn style intensity control	SRD, MRD, VSP, SVD
EMIS	Emotion recognition on incongruent speech	Target accuracy, proxy accuracy, $\rightarrow$2, Cramér’s $\rightarrow$3
Minimal-pair probing	Grammatical vs conceptual encoding in speech representations	Accuracy, selection score, confidence score

Several empirical regularities recur across these evaluations. DeSTA2 reports Dynamic-SUPERB ALL accuracy of 56.78, exceeding Qwen2-Audio’s 51.69 in the reported comparison, and attributes much of its strength to broad paralinguistic and degradation understanding acquired without speech instruction-tuning data (Lu et al., 2024). EchoMind finds that even state-of-the-art models struggle with high-expressive vocal cues: GPT-4o-Audio leads voice understanding at 66.25% ACC and reasoning at 68.04% ACC, yet none of the evaluated systems exceeds 4.0 on C4, the speech-information-relevance dimension of conversational response quality (Zhou et al., 26 Oct 2025). TELEVAL shows that GOAT-SLM is particularly strong on age-aware and non-speech-vocalization-aware tasks, reaching 72.13% on Age-zh and 40.91% on Para_mix300-zh, while also achieving the best reported CER on ESD-zh at 1.57% (Chen et al., 24 Jul 2025). StyleBench reports that only Qwen2.5-omni, GLM-4-Voice, and Kimi-Audio exceed 60% MRD, the multi-turn relevance threshold associated with reliable style-control evaluation, and it identifies strong initial but unstable follow-up adjustments as a common failure mode (Zhao et al., 8 Mar 2026).

Two lines of work expose deeper structural limitations. First, the EMIS study shows that four evaluated SLMs rely predominantly on textual semantics rather than vocal emotion under emotionally incongruent speech. The reported association between model predictions and the acoustic target label is very small ($\rightarrow$4), whereas the association with the semantic proxy label is considerable ($\rightarrow$5), despite a prompt that explicitly instructs the model to use tone of voice only (Corrêa et al., 29 Oct 2025). Second, layer-wise minimal-pair probing finds that across 16 speech encoders, grammatical features are encoded much more robustly than conceptual features, with an average gap of about 20 percentage points and conceptual knowledge peaking only around 60–65% (He et al., 19 Sep 2025).

The speech-only scaling results point in the same direction. The best 823M-parameter speech-only model reported in the scaling study reaches BLIMP 61.3, TopicCloze 78.0, and StoryCloze 56.7, while Pythia-6.9B reaches 80.0, 97.5, and 76.2 respectively, and the fitted exponents imply that speech-only SLMs improve far more slowly with compute than text LLMs (Cuervo et al., 2024). Taken together, these evaluations suggest that current SLMs are strongest at transcription-adjacent reasoning, moderately strong at structured paralinguistics under favorable supervision, and still fragile when lexical semantics must be balanced against prosody, context, or world knowledge.

6. Persistent limitations, security issues, and likely directions

A consistent concern across the literature is that many SLMs remain text-dominant despite their speech interface. OTReg diagnoses this as a modality gap: speech embeddings are long and highly variable, and models can exploit unintended acoustic variation for in-domain performance without truly aligning speech to text-like linguistic structure (Xu et al., 11 Aug 2025). EMIS reaches a complementary conclusion from a behavioral angle, showing that instruction-following SLMs can still act as though they are reading the words rather than listening to the prosody (Corrêa et al., 29 Oct 2025). EchoMind sharpens the same point for dialogue, reporting that many systems produce contextually correct but voice-insensitive replies and underuse vocal information precisely where empathy depends on it (Zhou et al., 26 Oct 2025).

These problems intersect with data and evaluation limitations. EchoMind’s human recordings use two professional speakers; GOAT-SLM notes uneven dialect coverage and lower subjective performance for Northeastern Mandarin; StyleBench relies on synthetic control procedures for several prosodic dimensions; and the probing work is English-only and based on single-voice TTS minimal pairs (Zhou et al., 26 Oct 2025, Chen et al., 24 Jul 2025, Zhao et al., 8 Mar 2026, He et al., 19 Sep 2025). This suggests that generalization across demographics, cultures, accents, and recording conditions remains an open problem rather than a solved benchmark issue.

Security introduces an additional axis of difficulty. "SPIRIT: Patching Speech LLMs against Jailbreak Attacks" argues that SLMs are substantially more vulnerable to jailbreaks than text-only LMs because continuous audio provides a high-dimensional, differentiable attack surface. The paper reports attack success rates up to 100% in some categories and introduces a post-hoc activation-patching defense that improves robustness up to 99% with negligible impact on utility and without retraining (Djanibekov et al., 18 May 2025). For deployed SLMs, robust spoken interaction therefore depends not only on speech understanding and generation, but also on architecture-level defenses against adversarial audio.

The forward-looking agenda in the literature is relatively coherent. Several papers call for prosody-aware encoders, stronger emotion and paralinguistic representations, richer multimodal fusion, and more expressive tokenization schemes (Zhou et al., 26 Oct 2025, Fan et al., 14 Jun 2025). Others emphasize structural solutions: explicit disentanglement of linguistic and paralinguistic channels (Wang et al., 11 Aug 2025), causal or world-model reasoning over speech states (Zhou et al., 5 Dec 2025), and cross-lingual interleaving for textless multilingual competence (Moumen et al., 1 Dec 2025). The scaling study, meanwhile, argues that transfer from strong text LLMs or multimodal alignment is more practical in the near term than relying on speech-only scaling alone (Cuervo et al., 2024).

The resulting picture is neither that SLMs are merely audio front-ends for text LLMs nor that they have already achieved speech-native intelligence. Rather, they are becoming a heterogeneous class of models in which the core research problem is how to preserve the reasoning and instruction-following capacity of LLMs while making prosody, speaker variation, non-verbal vocal signals, and contextual acoustics first-class variables in representation, inference, and generation.