Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning

Abstract: Instruction tuning for speech LLMs (SLMs) is substantially more challenging than for text-based LLMs, as it requires learning a new modality and a wide range of speech-specific instructions in addition to those supported by text LLMs. Existing SLM training approaches largely replicate the text LLM training paradigm by synthesizing large-scale speech pre-training and instruction-tuning datasets. However, this strategy is difficult to scale, since speech sequences are significantly longer than text sequences. In this paper, we propose SpeechCombine, an instruction-following speech LLM trained without any instruction tuning, using only a single round of speech pre-training on 30k hours of data. Starting from a text LLM base model, we perform continuous pre-training on speech utterances to obtain a speech-adapted model, and then directly combine its weights with the weight difference between the instruction-tuned and base versions of the text LLM. Our results show that this simple combination strategy not only preserves the knowledge and capabilities of the original text LLM, but also effectively transfers them to the speech domain. These findings suggest a new direction for SLM training that avoids reliance on massive speech data.

• The paper demonstrates that linear composition of instruction-following and speech adaptation weights can efficiently integrate text and speech skills with reduced data requirements.
• It employs a dual-axis parameter-space method that merges pre-trained text instructions with speech pre-training to address challenges like data inflation and modality variance.
• Empirical results reveal state-of-the-art performance on both text-oriented and speech-centric tasks, underscoring effective cross-modal transfer and reasoning.

Instruction-Following Speech LLMs via Parameter-Space Composition: An Analysis of SpeechCombine

Introduction and Motivation

Instruction-following capabilities have become central in the development of Speech LLMs (SLMs), mirroring similar breakthroughs in text-based LLMs. However, speech introduces additional challenges: SLMs must process, generate, and reason over longer sequences and handle a broader spectrum of multimodal and paralinguistic instructions, including tasks involving prosody and expressivity.

Traditional instruction tuning for SLMs typically involves (1) pre-training on large speech corpora and (2) supervised fine-tuning or RL on speech instructions. This approach is bottlenecked by "data inflation": a short sentence mapped to speech spans an order of magnitude more tokens than its text equivalent, leading to limited data efficiency and catastrophic forgetting of base LLM knowledge. Furthermore, while adapter-based methods partially mitigate forgetting by keeping the LLM frozen and learning modality bridges, these models are restricted in skill composition and generally cannot address the full spectrum of SLM instructions.

The "SpeechCombine" model introduces a novel paradigm: rather than conventional multi-stage speech instruction tuning, it employs a single round of speech pre-training and explicitly composes instruction-following and speech adaptation in weight space, inspired by recent advances in model merging (2607.02214).

Figure 1: Illustration of SpeechCombine in weight space, showing the composition of instruction-following and speech adaptation directions.

The SpeechCombine Framework

SpeechCombine operates by combining two axes in parameter space. Starting from a base text LLM, three models are involved:

• $\bm{\theta}_{\text{base}}$: Base text LLM parameters
• $\bm{\theta}_{\text{inst}}$: Parameters after instruction tuning on text
• $\bm{\theta}_{\text{speech}}$: Parameters after continuous pre-training on speech

Let $$\Delta \bm{\theta}_{\text{inst}} = \bm{\theta}_{\text{inst}} - \bm{\theta}_{\text{base}}$$ denote the instruction-following direction, which encodes how instruction tuning perturbs the base model to induce compositional/interactive behavior; and $$\Delta \bm{\theta}_{\text{speech}} = \bm{\theta}_{\text{speech}} - \bm{\theta}_{\text{base}}$$, capturing the speech-modality adaptation. The final SpeechCombine model parameters are constructed as:

$$\bm{\theta}_{\text{SC}} = \bm{\theta}_{\text{base}} + \lambda \Delta \bm{\theta}_{\text{speech}} + \Delta \bm{\theta}_{\text{inst}}$$

where $\lambda \in (0, 1]$ trades off speech adaptation against instruction-following, and is optimized via ablation to balance content preservation and speech capability transfer.

Figure 2: The SpeechCombine architecture and inference pipeline demonstrating the interface between speech tokenization, LLM reasoning, and speech decoding.

This approach consciously avoids further SFT or RL on speech instructions, relying solely on a standard next-token prediction objective over a well-structured pre-training corpus (approx. 30k hours). The input format during both pre-training and inference is a sequence of interleaved text, speech, and natural-language speech captions describing content, prosody, and paralinguistic features.

Speech Tokenization and Data Construction

SpeechCombine leverages discrete prosody tokens, following [ProsodyLM, (Qian et al., 27 Jul 2025)]. For each word in a speech utterance, five quantized values are produced representing median pitch, pitch range, pitch slope, duration, and energy. This tokenization dramatically reduces speech sequence length compared to waveform-based codecs and better aligns with the compositional structure of text LLMs, with delimiters marking speech boundaries.

The pre-training input is structured as:

$\bm{\theta}_{\text{inst}}$2

where "[cap]" is a natural language description of the speech segment’s attributes (e.g., pitch, emotion, emphasis), and "[speech]" encodes prosody tokens. This is designed to ensure not only familiarity with speech tokens but also to induce representation for the information hierarchy within speech signals (content, prosody, partial paralinguistics).

Empirical Results

Text-Oriented Tasks (Shallow Composition)

SpeechCombine achieves performance on text-oriented evaluation benchmarks—such as QA (OpenbookQA, MMSU), logical reasoning (GSM8k, MLCpro)—that matches or exceeds both ASR+TextLLM pipelines and instruction-finely-tuned SLM baselines. Notably, it remains competitive or outperforms advanced SLM architectures trained with $>100\times$ more data, like Fun-Audio-Chat or Kimi-Audio, despite employing less than 1% of their data scale.

Speech Understanding and Generation (Deep Composition)

On speech-centric tasks—emotion understanding, emphasis detection, expressive/emphatic speech generation—SpeechCombine demonstrates highly non-trivial skill transfer:

• Achieves state-of-the-art or second-best F1 on emphasis detection and speech generation tasks relative to similarly sized SLMs.
• Outperforms all baselines except those trained with far larger resources on speech-specific reasoning, supporting the claim that compositional integration in weight space induces emergent cross-modal performance, even on tasks "zero-shot" to both pre-training axes.

A critical finding: for prosody-based reasoning (e.g., emphasis detection), the model tends to answer based on text content alone unless explicitly prompted to attend to prosodic features (e.g., by prepending “Based on the prosody” to instructions), highlighting the boundary of weight-space compositionality in controlling modality-specific attention.

Figure 3: Excerpts of SpeechCombine’s thinking processes, showing natural integration of reasoning and prosody manipulation for both speech understanding and expressive generation.

Figure 4: Representative QA samples contextualizing multimodal inference and answer mapping.

Ablation and Analysis

• $\lambda$ sensitivity: Performance on text-oriented tasks (QA) deteriorates for $\lambda \to 1$ as speech direction dominates, while speech-focused tasks improve up to an optimal $\bm{\theta}_{\text{inst}}$0 in [0.8, 0.95], demonstrating the compositional tradeoff.
• Section removal: Removing speech captions or text anchor sections in pre-training data severely degrades speech understanding and speech generation performance, confirming their critical role as knowledge-bridging anchors.
• Instruction direction: Eliminating $\bm{\theta}_{\text{inst}}$1 (i.e., relying on speech-only adaptation plus in-context learning) results in significant loss in reasoning and compositionality, underlining the necessity of direct instruction-following transfer.
• Long-thinking transfer: By matching the prompt structure of the base-instruct LLM (e.g., adding a > section), SpeechCombine can inherit long-form reasoning and multi-step planning structures originally learned in text, with no further adaptation.

  Implications and Limitations

  The evidence presented repositions instruction-following SLM training as a weight-space composition problem, drastically reducing data and computational requirements, while preserving and integrating skills across domains. This paradigm offers a scalable pathway for rapid modality expansion and future-proof inheritance of emergent LLM capabilities, assuming the base/instruct/speech weights remain sufficiently compatible for linear combination.

  However, several limitations emerge:

  • Output formatting instability: Without format-enforcing decoding constraints, generative outputs can degrade, particularly in extended “thinking” sections.

  • Limited speech parameterization: Current prosody-only tokenization does not encode timbre or accent, restricting the model’s expressivity and generalization over regional/dialectical variation.
  • ASR dependence: Current usage requires external recognition to anchor text and speech segment pairing, introducing additional latency and error propagation. Full end-to-end modeling remains an open direction.

  Conclusion

  SpeechCombine demonstrates that direct, linear composition of instruction-following and modality adaptation vectors in parameter space can induce robust, multimodal SLMs with high data efficiency, strong skill transfer, and emergent compositional behaviors across instruction types and modalities. This approach points to a general, scalable recipe for future SLM development, particularly as LLMs evolve and new modalities emerge. The theoretical and practical value of weight-space composition for cross-modal learning is substantiated, although further work is needed to address formatting robustness and to extend the approach to richer speech representations including timbre, accent, and multi-party dialog.