StepAudio 2.5 Technical Report

Abstract: Unified audio-language modeling has emerged as a prominent trend in modern speech systems, promising to bring the reasoning capabilities of LLMs to auditory tasks. However, existing unified foundations often struggle to match the depth of specialized systems across automatic speech recognition (ASR), text-to-speech synthesis (TTS), and realtime spoken interaction. Bridging this gap remains an open challenge. This report presents StepAudio 2.5, a unified audio-language foundation model that matches or exceeds specialized systems across all three capabilities. Rather than treating these tasks as architecturally distinct, we operate on the premise that once text and audio share a multimodal representational space, task specialization becomes a matter of operational regimes: data construction, optimization targets, and decoding constraints. Guided by this insight, we advance the post-training paradigm from standard supervised learning to task-tailored Reinforcement Learning from Human Feedback (RLHF), using it as the primary mechanism to define complex optimization targets. We leverage this RLHF-centric alignment, alongside specialized decoding, to shape a shared backbone into three distinct operational modes. Concretely, the ASR branch advances transcription efficiency via verifiable multi-token decoding; the TTS branch achieves controllable, expressive synthesis through preference-based RLHF and context-rich supervision; and the Realtime branch realizes low-latency, persona-consistent dialogue via generative reward modeling within an RLHF framework. On standard benchmarks, StepAudio 2.5 achieves state-of-the-art results across ASR, TTS, and Realtime, demonstrating that a singular audio-language foundation can successfully internalize the distinct deployment objectives of speech understanding, generation, and live interaction.

• The paper introduces a unified audio-language model that leverages a shared encoder-adaptor-decoder backbone to perform ASR, TTS, and realtime interaction.
• It employs Multi-Token Proposal and RLHF techniques to enhance efficiency and quality, achieving strong numerical results in CER, WER, and human subjective evaluations.
• The architecture demonstrates practical deployment benefits by unifying modalities and enabling fine-tuned, low-latency performance across diverse speech tasks.

StepAudio 2.5: A Unified Audio-Language Foundation Model for Speech Recognition, Synthesis, and Realtime Interaction

Unified Audio-Language Foundation: Architectural Overview

StepAudio 2.5 is introduced as a unified foundation audio-LLM targeting three canonical speech system paradigms: automatic speech recognition (ASR), text-to-speech (TTS), and realtime spoken interaction. The architecture is centered around a shared backbone, following an asymmetric encoder-adapter-decoder pattern. A frozen audio encoder generates acoustic embeddings, which are projected into the hidden space of a text LLM decoder through an adaptor. This unified stack is the foundation for all three tasks, leveraging a multimodal token space such that both text and audio tokens coexist as representational primitives.

Figure 1: A unified view of the StepAudio 2.5 model family. The shared audio-language stack provides the common architectural basis used to organize ASR, TTS, and Realtime, while the three systems serve different deployment goals.

The architectural philosophy is that task differentiation emerges primarily from operational regimes—data construction, optimization objectives, and decoding strategies—rather than from isolated networks. Once text and audio inhabit a well-shaped joint latent space, ASR, TTS, and realtime interaction become directional queries against the same underlying memory.

Foundation Pretraining and Data Pipeline

Pretraining is structured and multimodal, combining 2.2T tokens across text and audio with a sequence length up to 32K. The curriculum begins with aligning the adaptor via ASR data, followed by a vocabulary expansion and a unified mixture of text and speech modalities encompassing ASR, TTS, speech-to-text translation, and interleaved multimodal sequences. Initial training phases focus on feature alignment and vocabulary stabilization with expert modulation via the MoE router, followed by extensive main phase pretraining and high-quality cooldown that includes Audio Caption and Instruct TTS data. This curriculum facilitates cross-modal grounding and context retention through extended-range modeling.

ASR Specialization: Efficient Decoding with MTP

StepAudio 2.5 ASR augments the foundational encoder-adaptor-decoder stack with a Multi-Token Proposal (MTP-5) head, where, at every decoding step, multiple future tokens are predicted. These proposals are subject to autoregressive verification: tokens are only accepted if they match the main decoding path, guaranteeing that acceleration through MTP does not compromise transcription correctness.

Figure 2: ASR architecture in StepAudio 2.5. The shared encoder-adaptor-decoder backbone is augmented with parallel future-token branches, making decoding substantially more efficient while preserving autoregressive verification.

The ASR pipeline employs a staged optimization strategy. First, frozen-branch alignment calibrates new MTP branches without perturbing the converged autoregressive recognizer; subsequent joint calibration harmonizes the backbone and lookahead proposals. The final loss combines standard next-token cross-entropy with an exponentially decayed aggregation of MTP losses. The ASR training corpus consists of 100K hours of short-form transcribed data and a 50K-hour curated long-form set constructed by multi-hypothesis verification and LLM-based normalization.

Figure 3: Long-form ASR data construction pipeline. The process transitions from individual clip transcription to global session-level refinement to ensure both accuracy and consistency.

Strong Numerical Results

• Chinese CER: 2.97% average; AISHELL-1 CER of 0.71%
• English WER: 3.68% average; LibriSpeech clean WER of 1.38%
• Long-form WER: 3.70% average, beating all compared baselines

The MTP-5 configuration achieves an acceptance rate decaying at approximately 0.9 per lookahead position, with an optimal average accepted length determined by branch utility. The Real-Time Factor reaches an unprecedented 0.0053 on 30s clips, demonstrating that scalable decoder capacity does not intrinsically increase latency under this regime.

Implications

StepAudio 2.5 demonstrates that externally grounded generation tasks, such as ASR, are significantly accelerable by exploiting modality-induced determinism, shifting the scalability-efficiency trade-off for large decoder models.

TTS Specialization: Semantic-to-Audio Alignment with RLHF

In TTS mode, StepAudio 2.5 bypasses the audio encoder and adapter, formulating speech synthesis purely as next-token prediction using audio tokens as a "language." Alignment between text and audio representations is strengthened by a two-stage SFT pipeline composed of large-scale zero-shot voice cloning and high-quality global/inline control supervision. Human-preference RLHF is conducted via a Generative Reward Model (GRM) that provides scalar preference feedback for policy gradient optimization.

Evaluation: Arena-Style Pairwise Preference

To address the inadequacy of conventional objective metrics in TTS, StepAudio 2.5 TTS is evaluated via a robust arena-style framework with standardized human preference ratings.

Figure 4: Arena Win Rates of StepAudio-2.5-TTS.

StepAudio-2.5-TTS achieves a 67.6% overall win rate across strong baselines (MiniMax-2.8-HD, Elevenlabs-v3, Gemini-3.1-Flash-TTS) on 774 prompts, with consistent improvements observed in global and inline control regimes.

Implications

The capacity for fine-grained instruction-following and controllable expressivity under RLHF-induced alignment forces a paradigm shift in audio-language modeling, enabling robust zero-shot voice cloning, emotional alignment, and fine-grained hierarchical expressive control.

Realtime Specialization: Low-Latency Spoken Dialogue

Realtime specialization leverages the foundation architecture without topology changes but introduces a three-phase training regime: audio-centric mid-training, progressive SFT for dialogue/perona/paralinguistic control, and RLHF with a generative reward model and explicit interaction rubrics. Data construction is grounded in large-scale persona-conditioned dialog, algorithmic attribute fission, and detailed paralinguistic annotation.

Evaluation encompasses subjective human rating through mobile-app scenarios and objective API-based performance across dialog, in-car dialogue, audio-question answering, and acoustic attribute inference.

Figure 5: Realtime interaction evaluation. Higher is better. Best results are in bold.

Numerical Results

• Human subjective evaluation: Top system with +10.0 point lead
• Objective metrics: Top position with +16.6 margin on Step-SPQA and strong performance across all evaluated suites

Implications

The joint improvement in conversational quality and acoustic understanding validates the claim that RLHF and persona/atmosphere conditioning can be integrated with foundational reasoning ability without catastrophic forgetting—a key outcome for open-domain, low-latency dialogue systems.

Theoretical Implications and Prospects

The central claim of StepAudio 2.5 is that once modalities are unified in representation, architectural separation is unnecessary; task specialization is a matter of operational regime engineering. The capacity to meet or exceed the domain-specific state of the art for ASR, TTS, and realtime dialogue using a shared backbone substantiates this position, providing empirical evidence against the necessity for distinct models in modern speech AI.

For theory, this points to the utility of joint latent spaces in multimodal modeling, the universality of large decoder LLMs for sequential acoustic-textual processing, and the algorithmic dividends from exploiting external modality determinism (e.g., in ASR MTP).

Practically, there are profound deployment advantages: shared model weights, data, and infrastructure smooth the path for multi-capability conversational agents, voice interfaces, and transcription tools.

Future developments may include further generalization to additional modalities, domain transfer with minimal post-training, and specialized reward shaping for fine-tuning persona, emotion, or application safety constraints. Integration with streaming and full-duplex interaction regimes will also likely benefit from the operational regime conceptual framework advanced in this work.

Conclusion

StepAudio 2.5 reifies a unified audio-language modeling paradigm in which the needs of ASR, TTS, and realtime spoken interaction are met via shared architecture and separation achieved through supervised and RLHF-driven regimen differentiation. It achieves strong performance on recognition, synthesis, and interactive benchmarks and demonstrates the feasibility and effectiveness of multimodal unification for end-to-end speech and language applications (2605.23463).