Speech Language Model (SLM)

• Speech Language Models (SLMs) are a family of models that extend language modeling from text to raw speech, processing both semantic and paralinguistic cues.
• They utilize diverse tokenization strategies—from phonetic units to codec tokens—to balance acoustic detail with linguistic structure.
• SLMs combine explicit alignment techniques for speech and text to enhance synthesis quality, paralinguistic understanding, and downstream performance.

Speech LLM (SLM), also termed Spoken LLM in part of the literature, denotes a family of models that extend language modeling from text to speech. In current usage, the term covers at least three related formulations: pure speech LMs that model tokenized speech sequences autoregressively, speech+text LMs that jointly model interleaved speech and text, and speech-aware text LMs that condition a text LLM on speech representations (Arora et al., 11 Apr 2025). In the LLM-centric setting, an SLM ingests raw audio, infers both linguistic content and paralinguistic cues, and produces text, speech tokens, or both through a shared backbone; in the textless setting, it models speech directly without intermediate text supervision (Cui et al., 7 May 2026).

1. Formal scope and model families

A unifying probabilistic formulation treats speech as a sequence of tokens $x=(x_1,\dots,x_T)$, with the simplest pure SLM defining

$$P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.$$

More general systems model either a joint speech-text distribution or a conditional distribution such as $$P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})$$ (Arora et al., 11 Apr 2025). A distinct but compatible encoder-centric definition describes an SLM as a neural encoder $f$ mapping raw audio or spectrograms into layerwise hidden states $\mathbf{H}^{(l)}=f^{(l)}(x)$, which are then probed for acoustic, syntactic, or semantic content (He et al., 19 Sep 2025). This suggests that the term now spans both generative and representational traditions.

Family	Formulation	Representative examples
Pure Speech LMs	$P(x_t\mid x_{<t})$ over tokenized speech	GSLM, AudioLM, TWIST
Speech + Text LMs	Joint modeling of speech and text	Moshi, Mini-Omni2, SpiRit-LM
Speech-Aware Text LMs	$$P(\mathrm{text}\mid \mathrm{speech},\mathrm{text\ prompt})$$	SALMONN, Qwen-Audio, SLM

Architecturally, these families differ along several axes: multilingual versus monolingual pre-training, end-to-end versus modular pipelined design, and fully autoregressive versus hybrid non-autoregressive decoding (Arora et al., 11 Apr 2025). Early bridge-style systems made this heterogeneity explicit. The model named “SLM: Bridge the thin gap between speech and text foundation models” combines a frozen speech encoder, a frozen multilingual LLM, and a trainable adapter of 156 M parameters, or approximately $1\%$ of the total parameters, to align speech representations with the LLM embedding space (Wang et al., 2023). The adapter-depth ablation in that work found a large jump from $1\rightarrow2$ layers and near-saturation thereafter, motivating the description of a “thin” speech-text representational gap.

2. Speech representations and tokenization

Tokenization is a central design choice because it determines the information density, tractability, and modality alignment properties of the model. The survey literature distinguishes two principal discrete regimes. The first uses phonetic or self-supervised units, typically obtained by quantizing SSL features such as wav2vec 2.0 or HuBERT via k-means or VQ-VAE into $50$–$$P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.$$0 symbols, sometimes followed by deduplication or BPE. The second uses neural audio codec tokens, often produced by models such as SoundStream, EnCodec, or TS3-Codec with multi-codebook quantization (Arora et al., 11 Apr 2025). Continuous inputs also remain common in speech-aware text LMs, including log-mel features, MFCCs, Whisper embeddings, HuBERT embeddings, and codec latents.

Recent work emphasizes that the tokenizer is not merely a front-end detail. In an LLM-centric framework, coupled tokenizers jointly quantize semantic and acoustic information into one token stream, semi-decoupled tokenizers partially separate high-level and residual information, and fully decoupled tokenizers explicitly split each frame into semantic and acoustic subspaces (Fan et al., 14 Jun 2025). Under a controlled comparison, coupled tokenizers showed synthesis success rates below $$P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.$$1, elevated word error rates above $$P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.$$2, and speaker similarity below $$P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.$$3, whereas the fully decoupled FACodec achieved $$P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.$$4 synthesis success, UTMOS near $$P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.$$5, WER $$P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.$$6, and SIM $$P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.$$7 on LibriTTS (Fan et al., 14 Jun 2025). This supports the broader claim that decoupling linguistic and acoustic structure can improve both alignment and generation.

A complementary line of work studies speaker-invariant units for SLMs. DC-Spin introduces a dual-codebook tokenizer with a small primary codebook for SLM tokens and a large auxiliary codebook that shapes phonetic sensitivity through swapped-prediction training (Chang et al., 2024). In zero-shot SLM benchmarks, DC-Spin improved phonetic and grammatical proxies relative to several baselines; the authors further report that tokens easily modeled by an n-gram LM or aligned with phonemes offer strong downstream performance (Chang et al., 2024). This suggests that token predictability, phonetic invariance, and speaker disentanglement are jointly important for scalable SLM design.

3. Modality alignment and the speech–text gap

A recurrent problem in LLM-centric SLMs is the modality gap: the performance drop between a text LLM on text input and the corresponding SLM on speech input. This gap persists even when the SLM is initialized from the same text checkpoint (Cui et al., 7 May 2026). Earlier work largely treated alignment as an output-side problem. Systems generated raw speech tokens directly from the LLM, adopted thinker-talker or interleaved text-and-speech architectures, or applied knowledge distillation and cross-modal distillation losses at the output layers (Cui et al., 7 May 2026). Those strategies improved spoken generation, but did not eliminate the gap.

Input-side alignment has therefore become a distinct research direction. TextPro-SLM argues that the remaining bottleneck lies on the input side and introduces WhisperPro, a unified speech encoder built on Whisper-large-v3 with a mel-reconstructor that preserves prosodic detail while producing synchronized text tokens and token-aligned prosody embeddings (Cui et al., 7 May 2026). From the $$P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.$$8th Whisper decoder layer, the model extracts $$P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.$$9 aligned to the decoded text sequence $$P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})$$0, and reconstructs the log-mel spectrogram with a joint ASR-plus-mel objective. The resulting prosody-aware LLM backbone can either prepend a global prosody summary or interleave grouped prosody summaries with re-tokenized LLM text tokens. The training is two-stage: first the LLM is frozen and only the prosody projector is trained; then projector and LLM are jointly optimized.

Other alignment mechanisms use small bridges rather than synchronized prosody-aware inputs. The Speech2Text adapter approach maps CTC-encoder speech features into the T5 embedding space and uses CTC blank-filtering to reduce the speech sequence length by approximately $$P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})$$1, improving dialog state tracking from $$P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})$$2 to $$P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})$$3 accuracy; adding a Speech2Entity retriever raises this to $$P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})$$4, while ASR WER improves from $$P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})$$5 to $$P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})$$6 (Wang et al., 2023). Descriptive alignment takes a different route: DeSTA trains a speech captioning module so that a frozen Whisper encoder and Llama2-7B-chat can generate richly descriptive captions of speech, improving Dynamic-SUPERB generalization, while DeSTA2 constructs speech-text pairs automatically from speech metadata and LLM-generated captions, explicitly targeting preservation of the frozen LLM’s original instruction-following behavior (Lu et al., 2024, Lu et al., 2024). Collectively, these results suggest that “alignment” in SLMs includes representational geometry, sequence compression, descriptive grounding, and preservation of LLM operating regimes.

4. Training objectives and optimization regimes

The dominant objective in SLM pre-training remains next-token cross-entropy. For generative pre-training, the standard form is

$$P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})$$7

with masked prediction, contrastive InfoNCE, CTC, instruction-tuning losses, duration and pitch regression, and preference optimization used as supplements or alternatives depending on the architecture (Arora et al., 11 Apr 2025). In pure speech LMs, the model typically learns over discrete units discovered from speech. In speech-aware text LMs, the objective is usually cast as text generation conditioned on speech features or aligned adapters.

TextPro-SLM explicitly combines semantic preservation with paralinguistic learning through

$$P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})$$8

where $$P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})$$9 is a knowledge-distillation or cross-entropy loss aligning speech-conditioned next-token distributions with a teacher TLM, and $f$0 is a cross-entropy loss on paralinguistic labels such as emotion, speaker age, gender, and accent; the reported balance is $f$1 (Cui et al., 7 May 2026). This multi-task design reflects an increasingly common requirement that an SLM preserve text-LM semantics while adding non-textual speech understanding.

Hybrid systems often decouple semantic planning and acoustic realization. SLIDE first uses an LLM to generate spoken-dialogue text, converts that text into phoneme sequences, predicts phoneme durations with a two-tower duration predictor, and then conditions a dGSLM-style unit LM on the resulting spoken phoneme sequence (Lu et al., 1 Jan 2025). Its SLM objective is

$f$2

and the full system combines this with duration and edge losses. In a different low-compute regime, Slamming uses TWIST initialization from a text LM, synthetic speech data, and Direct Preference Optimization on synthetic SpokenSWAG, reporting that a one-day single-GPU recipe can train a competitive SLM and that the empirical results outperform scaling-law predictions from prior work (Maimon et al., 19 Feb 2025). Such results do not invalidate scaling constraints, but they do indicate that initialization, synthetic data, and objective design materially affect the efficiency frontier.

5. Empirical behavior, scaling laws, and evaluation

Evaluation protocols for SLMs span likelihood-style probes, generation quality, downstream tasks, and safety diagnostics. Common metrics include token perplexity, sWUGGY, sBLIMP, StoryCloze variants, WER, BLEU, MOS or MOSNet, speaker and prosody consistency, and broad benchmark suites such as Dynamic-SUPERB, AIR-Bench, MMAU, and AudioBench (Arora et al., 11 Apr 2025). LLM-centric spoken QA settings also use model-judged scoring or task accuracy, while speech generation papers frequently report objective speech metrics and transcriber-based WER.

A central empirical finding is that textless SLMs scale, but much less favorably than text LLMs. “Scaling Properties of Speech LLMs” reports a strong correlation between pre-training loss and downstream syntactic and semantic performance, yet finds that the linguistic performance of SLMs scales up to three orders of magnitude more slowly than that of text-based LLMs (Cuervo et al., 2024). The paper also shows that adding the synthetic sTinyStories corpus yields consistent $f$3–$f$4 percentage-point gains in Topic and Story Cloze at all scales, whereas coarser unigram tokenization improves upstream loss scaling but causes semantic degradation and syntax lag (Cuervo et al., 2024). A plausible implication is that compression which reduces sequence length without preserving semantically useful structure is not sufficient.

The “modality evolving” analysis isolates three factors in the failure of semantically coherent speech generation: Factor A, the low semantic density of speech tokens; Factor B, the much greater sequence length; and Factor C, paralinguistic complexity (Wang et al., 2024). The reported conclusion is that Factor A has a relatively minor impact, Factor B more obviously harms syntax and semantics, and Factor C exerts the most significant impact, particularly in basic lexical modeling (Wang et al., 2024). This aligns with probing evidence from layer-wise minimal-pair evaluation: across S3Ms, ASR encoders, codec encoders, and AudioLLM encoders, grammatical categories achieve peak accuracies of $f$5–$f$6, conceptual categories top out near $f$7, syntax is linearly decodable from mid-layers, morphology emerges mainly in upper layers, and conceptual knowledge remains weak (He et al., 19 Sep 2025). The study summarizes this as “Form $f$8 Meaning.”

For LLM-centric models, the modality-gap metric is defined as

$f$9

Using this definition, TextPro-SLM reports the lowest average modality gap among leading SLMs at both $\mathbf{H}^{(l)}=f^{(l)}(x)$0B and $\mathbf{H}^{(l)}=f^{(l)}(x)$1B scales (Cui et al., 7 May 2026). At $\mathbf{H}^{(l)}=f^{(l)}(x)$2B, the average gap is approximately $\mathbf{H}^{(l)}=f^{(l)}(x)$3, compared to $\mathbf{H}^{(l)}=f^{(l)}(x)$4 for SALAD and $\mathbf{H}^{(l)}=f^{(l)}(x)$5 for Qwen2.5-Omni; on reasoning-heavy VoxEval, the average gap is reduced to $\mathbf{H}^{(l)}=f^{(l)}(x)$6, compared with $\mathbf{H}^{(l)}=f^{(l)}(x)$7 for Qwen2.5-Omni and $\mathbf{H}^{(l)}=f^{(l)}(x)$8 for Kimi-Audio; the same model also reports zero-shot paralinguistic accuracies of $\mathbf{H}^{(l)}=f^{(l)}(x)$9 on emotion, $P(x_t\mid x_{<t})$0 on gender, $P(x_t\mid x_{<t})$1 on age, and $P(x_t\mid x_{<t})$2 on accent, with only approximately $P(x_t\mid x_{<t})$3 hours of audio for LLM adaptation (Cui et al., 7 May 2026). These numbers are specific to one architecture, but they exemplify a broader empirical shift toward jointly evaluating semantic robustness and paralinguistic competence.

6. Applications, misconceptions, and emerging directions

The application space of SLMs includes voice assistants, spoken question answering, accessibility tools requiring robust reasoning from speech input, spoken dialogue generation, emotional or speaker-aware conversational agents, speech enhancement, and speech generation (Cui et al., 7 May 2026, Wang et al., 2023). However, several common misconceptions are contradicted by the literature. One is that better codec reconstruction necessarily yields better SLM generation. A systematic codec study shows the opposite: better speech reconstruction in codec systems does not guarantee improved speech generation in SLMs; naturalness depends strongly on decoder quality, while intelligibility depends more on quantization behavior and code utilization (Li et al., 2024). Another is that naturalistic speech generation implies semantic coherence. SLIDE explicitly addresses this by offloading semantics to an LLM and using the SLM for vocalization, because unconstrained textless SLMs can generate fluent but semantically vacuous speech (Lu et al., 1 Jan 2025).

Recent work pushes the field in several directions. Flow-SLM jointly models discrete semantic tokens and continuous acoustic embeddings with a flow-matching head, reporting competitive linguistic benchmark performance together with improved speaker preservation and acoustic detail in prompted generation (Chou et al., 12 Aug 2025). GOAT-SLM adopts a dual-modality head and a staged training strategy to handle emotion, dialect, age, and non-speech vocalizations, framing paralinguistic awareness as a first-class modeling target rather than an auxiliary attribute (Chen et al., 24 Jul 2025). Speech World Model factors speech understanding into four causal modules connected by a graph and uses posterior traces to prompt an instruction-tuned LLM, thereby making explicit reasoning and counterfactual intervention central design goals under partial supervision (Zhou et al., 5 Dec 2025). FlexiSLM extends the design space further by introducing dynamic and controllable frame rates on both input and output, verifying steering down to $P(x_t\mid x_{<t})$4 Hz and reporting that at $P(x_t\mid x_{<t})$5 Hz it roughly halves inference time relative to $P(x_t\mid x_{<t})$6 Hz while retaining strong speech-to-speech quality (Li et al., 30 Jun 2026).

Taken together, these developments indicate that SLM research has moved beyond the initial question of whether speech can simply be treated as another token sequence. The contemporary field studies how speech should be tokenized, how acoustic and linguistic information should be separated or fused, how LLM semantics can be preserved under speech conditioning, how paralinguistic structure should be represented, and how efficiency, controllability, and explicit reasoning can be integrated into a single spoken modeling framework.