SpeechLMs: Unified Speech and Text Modeling

• SpeechLMs are unified sequential models that tokenize speech and text to reduce information loss and cumulative errors seen in traditional ASR, LLM, and TTS pipelines.
• They employ specialized tokenizers, modality connectors, and adapters to bridge acoustic and linguistic representations, enhancing efficiency and performance.
• These models support applications like ASR, TTS, spoken QA, and translation while addressing challenges such as catastrophic forgetting and acoustic biases.

Speech-aware LLMs (SpeechLMs) are models that extend LLMs to the speech modality, or more generally cast speech and text within a unified sequential modeling framework. Relative to the classical “ASR + LLM + TTS” pipeline, the SpeechLM literature motivates tighter integration by emphasizing information loss during modality conversion, significant latency, and cumulative error propagation across stages. Across recent work, the architectural core typically consists of a speech tokenizer or acoustic encoder, a language-model backbone or modality connector, and a speech generator or vocoder, with evaluation spanning ASR, TTS, spoken question answering, instruction following, speaker-related tasks, and paralinguistic understanding (Cui et al., 2024).

1. Conceptual scope and task space

A common formalization treats SpeechLM development as universal sequential modeling: given input conditions $\mathbf{x}$ and target outputs $\mathbf{y}$, the model maximizes $P(\mathbf{y}\mid \mathbf{x})$. In this view, text, speech, and other modalities are tokenized into discrete sequences and spliced into a single training sequence, so ASR, TTS, AudioLM, TextLM, translation, multimodal interaction, and related tasks differ mainly by task template and tokenization scheme rather than by entirely separate model classes (Tian et al., 21 Feb 2025).

The task space described in the literature is broad. Surveyed capabilities include semantic tasks such as spoken dialogue, speech translation, ASR, keyword spotting, TTS, and spoken language understanding; speaker-related tasks such as speaker identification, verification, diarization, voice-conditioned generation, voice cloning, and voice conversion; and paralinguistic tasks such as emotion recognition and speech separation (Cui et al., 2024). Some systems are explicitly unified across speech and text. OpusLM is trained on speech-only, text-only, ASR, and TTS task types within one decoder-only family (Tian et al., 21 Jun 2025). Granite-speech operates in a speech mode for ASR and AST and in a text mode that directly calls the underlying instruction-tuned LLM, thereby preserving the base model’s text capabilities and safety (Saon et al., 13 May 2025).

Taken together, these works suggest that “SpeechLM” is best understood functionally rather than as a single fixed architecture. In current usage, the term covers both speech-aware LLMs that process speech and emit text and broader end-to-end systems that directly model and generate speech. This suggests that the central unifying property is not a particular tokenizer, connector, or decoder, but the attempt to place speech within the modeling regime of large-scale language systems.

2. Representations and tokenization

Speech tokenization is repeatedly treated as foundational. The surveyed design space distinguishes tokenizers optimized for semantic understanding, tokenizers optimized for acoustic generation, and mixed-objective tokenizers that combine semantic and acoustic properties (Cui et al., 2024). The technical problem is not only discretization, but also whether the resulting units align with the sequential structure that a LLM can exploit.

Several recent lines of work pursue explicitly language-model-aware representations. LAST trains a speech tokenizer by leveraging objectives from a frozen pre-trained textual LM, backpropagating the LM loss through adapter and quantization modules while keeping the speech encoder and text LM frozen. The reported result is that the proposed tokenization outperforms evaluated baselines on both spoken language modeling and speech-to-text, and, unlike prior work, allows a single pre-trained LM to process both speech and text inputs (Turetzky et al., 2024). SyllableLM attacks the token-rate problem from a different angle: it learns coarse semantic units through LossPred and SylBoost, produces controllable-rate units at as low as 5Hz and 60bps, achieves state of the art in syllabic segmentation and clustering, and reports a 30x reduction in training compute together with a 4x wall-clock inference speedup (Baade et al., 2024).

A parallel line studies the internal factorization of speech tokens. Under a fair LLM-centric framework, fully decoupled tokenization is reported to improve alignment and synthesis quality relative to coupled and semi-decoupled alternatives. The same work introduces multi-token prediction, in which one hidden state decodes multiple speech tokens, yielding up to 12$\times$ faster decoding and reducing word error rate from 6.07 to 3.01 (Fan et al., 14 Jun 2025). This places tokenization and decoding efficiency in direct relation: improved factorization of semantics, prosody, and timbre can benefit both controllability and throughput.

A recurrent misconception is that coarser tokenization is uniformly beneficial because it compresses sequence length. The scaling study complicates that view. It reports that unigram tokenization slightly improves upstream loss, but downstream linguistic performance degrades, and for Story Cloze there is no improvement with more compute (Cuervo et al., 2024). This suggests that compression alone is insufficient; token granularity must be evaluated against the downstream linguistic abstractions that SpeechLMs are expected to acquire.

3. Architectural patterns and modality fusion

One dominant architectural pattern is warm-starting from text LMs. TWIST replaces the original text-token embedding layer with a randomly initialized embedding table for speech tokens, copies the Transformer body from a pretrained textual LM, and then fine-tunes on large-scale speech token datasets. Reported findings are that textual pretraining consistently outperforms random initialization, that TWIST models trained with just 10% of data match or outperform randomly initialized models trained with 100% of the data, and that convergence to a given validation perplexity is about 4x faster (Hassid et al., 2023).

A second pattern is open unified decoder-only design. OpusLM builds decoder-only SpeechLMs up to 7B parameters from pretrained text LMs and continuously pre-trains them on 213K hours of speech-text pairs and 292B text-only tokens. Its multi-stream design uses discrete semantic and acoustic tokens and a delay-interleave architecture that preserves frame-wise autoregression while keeping time complexity $\mathcal{O}(T)$, independent of the number of streams. The reported family is competitive in ASR, TTS, and text-only capabilities, and is released together with code, data, checkpoints, and training logs (Tian et al., 21 Jun 2025).

A third pattern centers on modality connectors and adapters. Granite-speech uses a conformer acoustic encoder, a windowed Q-former speech modality adapter for temporal downsampling and projection into the LLM embedding space, and LoRA adapters for parameter-efficient speech alignment. Its two-mode design is notable because speech-specific components are activated only in speech mode, whereas text mode directly uses the underlying text LLM (Saon et al., 13 May 2025). SSR-Connector addresses a different bottleneck: long-form speech and catastrophic forgetting. By exploiting speech-text alignments, it segments and compresses speech features to match text-token granularity, then uses a two-stage distillation and fine-tuning pipeline. The reported outcome is better speech understanding, including +10 accuracy on StoryCloze and +20 on Speech-MMLU, while preserving pre-trained text ability (Tan et al., 2024).

Single-stage joint speech-text supervised fine-tuning provides yet another architectural compromise. VoiceTextBlender combines a pre-trained speech encoder, a modality adapter, an instruction-tuned Gemma 2.5B LLM, and LoRA adapters, and trains with text-only SFT data together with ASR/AST, speech-based QA, and mixed-modal data. The paper reports that its 3B model outperforms previous 7B and 13B SpeechLMs across various speech benchmarks while preserving the original capabilities on text-only tasks (Peng et al., 2024).

4. Training, alignment, and catastrophic forgetting

A central training problem is catastrophic forgetting: speech-centric training can erode the instruction-following and general text behavior inherited from the base LLM. Speech-IFEval was introduced specifically to disentangle instruction-following from speech perception and to quantify forgetting relative to the original text-only LLM. The reported findings are that most SLMs struggle with even basic instructions, are highly sensitive to prompt variations, and often perform far worse than text-based LLMs. The same benchmark reports IFrate 74.07 for Vicuna-13B versus 36.89 for SALMONN, and 92.53 for Llama3-8B-Instruct versus 89.23 for DeSTA2, with early SLMs showing large negative forgetting rates (Lu et al., 25 May 2025).

Joint optimization with text data is one response to this problem. VoiceTextBlender argues that only joint, single-stage SFT with LoRA adapters and both text and speech data reliably preserves original text ability while enabling new speech skills. Its ablations report that speech-only SFT causes catastrophic loss on text benchmarks, while freezing the LLM preserves text but impairs speech performance (Peng et al., 2024). Speech-IFEval adds a complementary observation: LoRA scaling at inference can recover some instruction-following in SALMONN, improving IFrate from 36.89 to 68.06, but at the expense of lower raw speech task performance (Lu et al., 25 May 2025). This exposes a persistent trade-off between speech-task specialization and preservation of general instruction behavior.

Recent work also pushes beyond supervised fine-tuning toward reward-based alignment. For open-format spoken question answering and automatic speech translation, GRPO with BLEU as the reward signal is reported to surpass standard SFT across several metrics, improving LibriSQA and CoVoST2 results and avoiding the value model required by PPO-style methods (Elmakies et al., 21 Sep 2025). Reinforced Behavior Alignment adopts a different RL formulation: it synthesizes large-scale speech-text alignment data with a teacher LLM and zero-shot TTS, then aligns the SpeechLM to teacher behavior through a reinforcement-learning-based objective. The reported outcome is improved instruction-following relative to distillation baselines and state-of-the-art performance on open benchmarks for spoken question answering and speech-to-text translation using only self-generated data (Liu et al., 25 Aug 2025).

5. Evaluation and multilingual benchmarking

Evaluation has become a first-order research problem rather than a purely downstream reporting step. A major theme is that existing benchmarks are heavily centered on English. The Korean benchmark paper argues that straightforward transfer through ASR, translation, normalization, and TTS can corrupt language-specific instructions, answer constraints, and spoken forms, while transferring source-language audio fails to preserve target-language speaker attributes, accents, and paralinguistic properties. It therefore proposes two human-agent benchmark-construction frameworks and publicly releases KVoiceBench, KOpenAudioBench, and KMMAU, comprising 12,345 samples in total (Kim et al., 27 May 2026).

The empirical finding from those Korean benchmarks is not merely that non-English performance is lower, but that model rankings differ by task family. English–Korean performance gaps vary substantially across models and task families, and SpokenQA and audio-understanding rankings diverge. The paper reports, for example, that Raon-Speech leads on KVoiceBench and KOpenAudioBench, whereas Fun-Audio-Chat and Step-Audio 2 Mini perform well on KMMAU. This reveals complementary weaknesses that English-only evaluation does not expose (Kim et al., 27 May 2026).

Low-resource multilingual ASR evaluation raises a related but distinct issue. LoASR-Bench evaluates the latest SpeechLMs on 25 languages from 9 language families and includes both Latin and non-Latin scripts. Reported results show lower error rates for Latin-script languages than for non-Latin-script families, substantial difficulty for languages with less than 1 hour of supervised data, and meaningful gains from language-aware prompting, especially for some Indo-Aryan and Uralic languages. The same benchmark argues that simply scaling up models is not enough, because improvement remains modest and cross-family generalization is limited (Chen et al., 20 Mar 2026).

Beyond multilinguality, benchmark design is increasingly decomposed by capability. Speech-IFEval isolates instruction-following from speech perception and introduces instruction-following rate and forgetting rate as explicit metrics (Lu et al., 25 May 2025). More broadly, the survey catalogs representation evaluation, linguistic evaluation, generation quality and diversity, real-time and full-duplex benchmarks, and downstream suites such as SUPERB, VoiceBench, and related speech/audio metrics (Cui et al., 2024). This suggests that SpeechLM evaluation is moving away from single-task scoreboards toward multidimensional auditing of linguistic, acoustic, safety, and interactional behavior.

6. Applications, limitations, and active controversies

The application surface of SpeechLMs now includes general multimodal assistance, spoken QA, speech translation, and domain-specific agents. VoiceTextBlender reports emergent handling of unseen prompts and tasks, including multi-turn, mixed-modal inputs and arbitrary output styles such as JSON slot-filling (Peng et al., 2024). A data-centric perspective is provided by SpeLangy: controlled ablations on pretraining data processing, synthetic dataset construction, and interleaving strategy are used to pretrain a 3.8B-parameter SpeechLM that outperforms models up to 3x larger by 10.2% absolute performance on spoken question answering (Udandarao et al., 22 Oct 2025). In a vertical domain, SpeechMedAssist proposes a two-stage adaptation recipe—knowledge and capability injection via text, followed by modality re-alignment with limited speech data—and reports that only 10k synthesized medical speech samples are needed for effective speech-based multi-turn consultation (Chen et al., 8 Jan 2026).

At the same time, several limitations recur across the literature. Scaling studies show that pre-training loss is strongly correlated with downstream syntactic and semantic performance, but that the linguistic performance of SLMs scales up to three orders of magnitude more slowly than that of text-based LLMs. The same work reports that synthetic spoken TinyStories improves semantic performance, indicating that data design can matter as much as scale (Cuervo et al., 2024). This suggests that raw compute scaling, by itself, is unlikely to close the gap between speech-native and text-native language modeling.

Controllability in speech generation is also less mature than the term “prompt-conditioned synthesis” can imply. An empirical study of autoregressive and non-autoregressive speech LMs reports that heterogeneous and nonstationary prompts hurt audio quality, contrary to the previous finding that longer prompts always lead to better synthesis. It further reports that semantic units carry pitch, tempo, volume, and speech emphasis, so speaker style is affected by content in addition to the prompt. The paper concludes that zero-shot, prompt-conditioned, fully controllable speech synthesis is not yet attainable with current SpeechLMs (Peng et al., 2024).

Fairness and contextual appropriateness remain contested. A study of acoustic-based gender differentiation introduces a dataset with 9,208 speech samples across Gender-Independent, Gender-Stereotypical, and Gender-Dependent categories, and reports a paradoxical pattern: models give male-oriented responses in Gender-Stereotypical questions, while failing to differentiate in Gender-Dependent questions where gender would be contextually appropriate. The authors further report that these patterns primarily stem from Whisper speech encoders, which generate male-oriented acoustic tokens (Choi et al., 25 Sep 2025). This suggests that fairness in SpeechLMs cannot be reduced to text-side debiasing alone; acoustic representations themselves can mediate systematic behavioral distortions.

SpeechLMs therefore occupy a technically heterogeneous and still unsettled area of multimodal language modeling. The field has established workable patterns for tokenization, modality fusion, multilingual benchmarking, and post-training alignment, and it has produced open systems that are competitive in ASR, TTS, and speech-conditioned generation. Yet the same literature shows that representation choice, prompt sensitivity, catastrophic forgetting, multilingual transfer, and acoustic bias are structural issues rather than peripheral defects. A plausible implication is that future progress will depend less on treating speech as merely another input channel and more on developing representations, training objectives, and evaluations that are explicitly aware of the linguistic and paralinguistic structure unique to spoken interaction.