---
title: Speech Language Model (SLM)
url: https://www.emergentmind.com/topics/speech-language-model-slm
type: topic
---

# Speech Language Model (SLM)

Speech Language Model (SLM), also termed Spoken Language Model in part of the literature, denotes a family of models that extend language modeling from text to speech. In current usage, the term covers at least three related formulations: pure speech LMs that model tokenized speech sequences autoregressively, speech+text LMs that jointly model interleaved speech and text, and speech-aware text LMs that condition a text LLM on speech representations [2504.08528]. In the LLM-centric setting, an SLM ingests raw audio, infers both linguistic content and paralinguistic cues, and produces text, speech tokens, or both through a shared backbone; in the textless setting, it models speech directly without intermediate text supervision [2605.05927].

## 1. Formal scope and model families

A unifying probabilistic formulation treats speech as a sequence of tokens \(x=(x_1,\dots,x_T)\), with the simplest pure SLM defining
\[
P(x_1,\dots,x_T)=\prod_{t=1}^T P(x_t\mid x_{<t})\,.
\]
More general systems model either a joint speech-text distribution or a conditional distribution such as \(P(y^{\mathrm{txt}}\mid x^{\mathrm{sp}},x^{\mathrm{txt}})\) [2504.08528]. A distinct but compatible encoder-centric definition describes an SLM as a neural encoder \(f\) mapping raw audio or spectrograms into layerwise hidden states \(\mathbf{H}^{(l)}=f^{(l)}(x)\), which are then probed for acoustic, syntactic, or semantic content [2509.15655]. This suggests that the term now spans both generative and representational traditions.

| Family | Formulation | Representative examples |
|---|---|---|
| Pure Speech LMs | \(P(x_t\mid x_{<t})\) over tokenized speech | GSLM, AudioLM, TWIST |
| Speech + Text LMs | Joint modeling of speech and text | Moshi, Mini-Omni2, SpiRit-LM |
| Speech-Aware Text LMs | \(P(\mathrm{text}\mid \mathrm{speech},\mathrm{text\ prompt})\) | SALMONN, Qwen-Audio, SLM |

Architecturally, these families differ along several axes: multilingual versus monolingual pre-training, end-to-end versus modular pipelined design, and fully autoregressive versus hybrid non-autoregressive decoding [2504.08528]. Early bridge-style systems made this heterogeneity explicit. The model named “SLM: Bridge the thin gap between speech and text foundation models” combines a frozen speech encoder, a frozen multilingual LLM, and a trainable adapter of 156 M parameters, or approximately \(1\%\) of the total parameters, to align speech representations with the LLM embedding space [2310.00230]. The adapter-depth ablation in that work found a large jump from \(1\rightarrow2\) layers and near-saturation thereafter, motivating the description of a “thin” speech-text representational gap.

## 2. Speech representations and tokenization

Tokenization is a central design choice because it determines the information density, tractability, and modality alignment properties of the model. The survey literature distinguishes two principal discrete regimes. The first uses phonetic or self-supervised units, typically obtained by quantizing SSL features such as wav2vec 2.0 or HuBERT via k-means or VQ-VAE into \(50\)–\(1\text{ K}\) symbols, sometimes followed by deduplication or BPE. The second uses neural audio codec tokens, often produced by models such as SoundStream, EnCodec, or TS3-Codec with multi-codebook quantization [2504.08528]. Continuous inputs also remain common in speech-aware text LMs, including log-mel features, MFCCs, Whisper embeddings, HuBERT embeddings, and codec latents.

Recent work emphasizes that the tokenizer is not merely a front-end detail. In an LLM-centric framework, coupled tokenizers jointly quantize semantic and acoustic information into one token stream, semi-decoupled tokenizers partially separate high-level and residual information, and fully decoupled tokenizers explicitly split each frame into semantic and acoustic subspaces [2506.12537]. Under a controlled comparison, coupled tokenizers showed synthesis success rates below \(70\%\), elevated word error rates above \(13\%\), and speaker similarity below \(0.18\), whereas the fully decoupled FACodec achieved \(87\%\) synthesis success, UTMOS near \(3.93\), WER \(=6.07\), and SIM \(=0.49\) on LibriTTS [2506.12537]. This supports the broader claim that decoupling linguistic and acoustic structure can improve both alignment and generation.

A complementary line of work studies speaker-invariant units for SLMs. DC-Spin introduces a dual-codebook tokenizer with a small primary codebook for SLM tokens and a large auxiliary codebook that shapes phonetic sensitivity through swapped-prediction training [2410.24177]. In zero-shot SLM benchmarks, DC-Spin improved phonetic and grammatical proxies relative to several baselines; the authors further report that tokens easily modeled by an n-gram LM or aligned with phonemes offer strong downstream performance [2410.24177]. This suggests that token predictability, phonetic invariance, and speaker disentanglement are jointly important for scalable SLM design.

## 3. Modality alignment and the speech–text gap

A recurrent problem in LLM-centric SLMs is the modality gap: the performance drop between a text LLM on text input and the corresponding SLM on speech input. This gap persists even when the SLM is initialized from the same text checkpoint [2605.05927]. Earlier work largely treated alignment as an output-side problem. Systems generated raw speech tokens directly from the LLM, adopted thinker-talker or interleaved text-and-speech architectures, or applied knowledge distillation and cross-modal distillation losses at the output layers [2605.05927]. Those strategies improved spoken generation, but did not eliminate the gap.

Input-side alignment has therefore become a distinct research direction. TextPro-SLM argues that the remaining bottleneck lies on the input side and introduces WhisperPro, a unified speech encoder built on Whisper-large-v3 with a mel-reconstructor that preserves prosodic detail while producing synchronized text tokens and token-aligned prosody embeddings [2605.05927]. From the \(16\)th Whisper decoder layer, the model extracts \(P=(p_1,\dots,p_N)\) aligned to the decoded text sequence \(y=(y_1,\dots,y_N)\), and reconstructs the log-mel spectrogram with a joint ASR-plus-mel objective. The resulting prosody-aware LLM backbone can either prepend a global prosody summary or interleave grouped prosody summaries with re-tokenized LLM text tokens. The training is two-stage: first the LLM is frozen and only the prosody projector is trained; then projector and LLM are jointly optimized.

Other alignment mechanisms use small bridges rather than synchronized prosody-aware inputs. The Speech2Text adapter approach maps CTC-encoder speech features into the T5 embedding space and uses CTC blank-filtering to reduce the speech sequence length by approximately \(4\times\), improving dialog state tracking from \(24.7\%\) to \(28.4\%\) accuracy; adding a Speech2Entity retriever raises this to \(34.6\%\), while ASR WER improves from \(9.4\%\) to \(8.5\%\) [2306.07944]. Descriptive alignment takes a different route: DeSTA trains a speech captioning module so that a frozen Whisper encoder and Llama2-7B-chat can generate richly descriptive captions of speech, improving Dynamic-SUPERB generalization, while DeSTA2 constructs speech-text pairs automatically from speech metadata and LLM-generated captions, explicitly targeting preservation of the frozen LLM’s original instruction-following behavior [2406.18871] [2409.20007]. Collectively, these results suggest that “alignment” in SLMs includes representational geometry, sequence compression, descriptive grounding, and preservation of LLM operating regimes.

## 4. Training objectives and optimization regimes

The dominant objective in SLM pre-training remains next-token cross-entropy. For generative pre-training, the standard form is
\[
\mathcal{L}_{\mathrm{CE}}=-\sum_{t=1}^{T}\log P(x_t\mid x_{<t})\,,
\]
with masked prediction, contrastive InfoNCE, CTC, instruction-tuning losses, duration and pitch regression, and preference optimization used as supplements or alternatives depending on the architecture [2504.08528]. In pure speech LMs, the model typically learns over discrete units discovered from speech. In speech-aware text LMs, the objective is usually cast as text generation conditioned on speech features or aligned adapters.

TextPro-SLM explicitly combines semantic preservation with paralinguistic learning through
\[
L=L_{\mathrm{semantic}}(y_{\mathrm{pred}},y_{\mathrm{gt}})+\lambda_p L_{\mathrm{para}}(z_{\mathrm{pred}},z_{\mathrm{gt}})\,,
\]
where \(L_{\mathrm{semantic}}\) is a knowledge-distillation or cross-entropy loss aligning speech-conditioned next-token distributions with a teacher TLM, and \(L_{\mathrm{para}}\) is a cross-entropy loss on paralinguistic labels such as emotion, speaker age, gender, and accent; the reported balance is \(\lambda_p\approx1.0\) [2605.05927]. This multi-task design reflects an increasingly common requirement that an SLM preserve text-LM semantics while adding non-textual speech understanding.

Hybrid systems often decouple semantic planning and acoustic realization. SLIDE first uses an LLM to generate spoken-dialogue text, converts that text into phoneme sequences, predicts phoneme durations with a two-tower duration predictor, and then conditions a dGSLM-style unit LM on the resulting spoken phoneme sequence [2501.00805]. Its SLM objective is
\[
L_{SLM}=-\sum_{t=1}^{T}\log p(u_t\mid u_{<t},\mathrm{phon}_{\le T_1})\,,
\]
and the full system combines this with duration and edge losses. In a different low-compute regime, Slamming uses TWIST initialization from a text LM, synthetic speech data, and Direct Preference Optimization on synthetic SpokenSWAG, reporting that a one-day single-GPU recipe can train a competitive SLM and that the empirical results outperform scaling-law predictions from prior work [2502.15814]. Such results do not invalidate scaling constraints, but they do indicate that initialization, synthetic data, and objective design materially affect the efficiency frontier.

## 5. Empirical behavior, scaling laws, and evaluation

Evaluation protocols for SLMs span likelihood-style probes, generation quality, downstream tasks, and safety diagnostics. Common metrics include token perplexity, sWUGGY, sBLIMP, StoryCloze variants, WER, BLEU, MOS or MOSNet, speaker and prosody consistency, and broad benchmark suites such as Dynamic-SUPERB, AIR-Bench, MMAU, and AudioBench [2504.08528]. LLM-centric spoken QA settings also use model-judged scoring or task accuracy, while speech generation papers frequently report objective speech metrics and transcriber-based WER.

A central empirical finding is that textless SLMs scale, but much less favorably than text LLMs. “Scaling Properties of Speech Language Models” reports a strong correlation between pre-training loss and downstream syntactic and semantic performance, yet finds that the linguistic performance of SLMs scales up to three orders of magnitude more slowly than that of text-based LLMs [2404.00685]. The paper also shows that adding the synthetic sTinyStories corpus yields consistent \(+5\)–\(10\) percentage-point gains in Topic and Story Cloze at all scales, whereas coarser unigram tokenization improves upstream loss scaling but causes semantic degradation and syntax lag [2404.00685]. A plausible implication is that compression which reduces sequence length without preserving semantically useful structure is not sufficient.

The “modality evolving” analysis isolates three factors in the failure of semantically coherent speech generation: Factor A, the low semantic density of speech tokens; Factor B, the much greater sequence length; and Factor C, paralinguistic complexity [2412.17048]. The reported conclusion is that Factor A has a relatively minor impact, Factor B more obviously harms syntax and semantics, and Factor C exerts the most significant impact, particularly in basic lexical modeling [2412.17048]. This aligns with probing evidence from layer-wise minimal-pair evaluation: across S3Ms, ASR encoders, codec encoders, and AudioLLM encoders, grammatical categories achieve peak accuracies of \(85\)–\(95\%\), conceptual categories top out near \(60\%\), syntax is linearly decodable from mid-layers, morphology emerges mainly in upper layers, and conceptual knowledge remains weak [2509.15655]. The study summarizes this as “Form \(\gg\) Meaning.”

For LLM-centric models, the modality-gap metric is defined as
\[
\mathrm{Gap}=\mathrm{Accuracy}_{\text{text-LLM on text input}}-\mathrm{Accuracy}_{\text{SLM on speech input}}\,.
\]
Using this definition, TextPro-SLM reports the lowest average modality gap among leading SLMs at both \(3\)B and \(7\)B scales [2605.05927]. At \(7\)B, the average gap is approximately \(0.7\%\), compared to \(7.1\%\) for SALAD and \(3.1\%\) for Qwen2.5-Omni; on reasoning-heavy VoxEval, the average gap is reduced to \(5.3\%\), compared with \(11.5\%\) for Qwen2.5-Omni and \(23.9\%\) for Kimi-Audio; the same model also reports zero-shot paralinguistic accuracies of \(60.5\%\) on emotion, \(88.6\%\) on gender, \(64.9\%\) on age, and \(45.1\%\) on accent, with only approximately \(1{,}000\) hours of audio for LLM adaptation [2605.05927]. These numbers are specific to one architecture, but they exemplify a broader empirical shift toward jointly evaluating semantic robustness and paralinguistic competence.

## 6. Applications, misconceptions, and emerging directions

The application space of SLMs includes voice assistants, spoken question answering, accessibility tools requiring robust reasoning from speech input, spoken dialogue generation, emotional or speaker-aware conversational agents, speech enhancement, and speech generation [2605.05927] [2312.09747]. However, several common misconceptions are contradicted by the literature. One is that better codec reconstruction necessarily yields better SLM generation. A systematic codec study shows the opposite: better speech reconstruction in codec systems does not guarantee improved speech generation in SLMs; naturalness depends strongly on decoder quality, while intelligibility depends more on quantization behavior and code utilization [2409.04016]. Another is that naturalistic speech generation implies semantic coherence. SLIDE explicitly addresses this by offloading semantics to an LLM and using the SLM for vocalization, because unconstrained textless SLMs can generate fluent but semantically vacuous speech [2501.00805].

Recent work pushes the field in several directions. Flow-SLM jointly models discrete semantic tokens and continuous acoustic embeddings with a flow-matching head, reporting competitive linguistic benchmark performance together with improved speaker preservation and acoustic detail in prompted generation [2508.09350]. GOAT-SLM adopts a dual-modality head and a staged training strategy to handle emotion, dialect, age, and non-speech vocalizations, framing paralinguistic awareness as a first-class modeling target rather than an auxiliary attribute [2507.18119]. Speech World Model factors speech understanding into four causal modules connected by a graph and uses posterior traces to prompt an instruction-tuned LLM, thereby making explicit reasoning and counterfactual intervention central design goals under partial supervision [2512.05933]. FlexiSLM extends the design space further by introducing dynamic and controllable frame rates on both input and output, verifying steering down to \(4.0\) Hz and reporting that at \(6.25\) Hz it roughly halves inference time relative to \(12.5\) Hz while retaining strong speech-to-speech quality [2606.31247].

Taken together, these developments indicate that SLM research has moved beyond the initial question of whether speech can simply be treated as another token sequence. The contemporary field studies how speech should be tokenized, how acoustic and linguistic information should be separated or fused, how LLM semantics can be preserved under speech conditioning, how paralinguistic structure should be represented, and how efficiency, controllability, and explicit reasoning can be integrated into a single spoken modeling framework.

Source: https://www.emergentmind.com/topics/speech-language-model-slm