Step-Audio-R1.5 Technical Report

Abstract: Recent advancements in large audio LLMs have extended Chain-of-Thought (CoT) reasoning into the auditory domain, enabling models to tackle increasingly complex acoustic and spoken tasks. To elicit and sustain these extended reasoning chains, the prevailing paradigm -- driven by the success of text-based reasoning models -- overwhelmingly relies on Reinforcement Learning with Verified Rewards (RLVR). However, as models are strictly optimized to distill rich, continuous auditory contexts into isolated, verifiable text labels, a fundamental question arises: are we fostering true audio intelligence, or merely reducing a continuous sensory medium into a discrete puzzle? We identify this as the "verifiable reward trap." While RLVR yields remarkable scores on standardized objective benchmarks, it systematically degrades the real-world conversational feel of audio models. By prioritizing isolated correctness over acoustic nuance, RLVR reduces dynamic interactions to mechanical "answering machines," severely compromising prosodic naturalness, emotional continuity, and user immersion, particularly in long-turn dialogues. To bridge the gap between mechanical objective verification and genuine sensory empathy, we introduce Step-Audio-R1.5, marking a paradigm shift toward Reinforcement Learning from Human Feedback (RLHF) in audio reasoning. Comprehensive evaluations demonstrate that Step-Audio-R1.5 not only maintains robust analytical reasoning but profoundly transforms the interactive experience, redefining the boundaries of deeply immersive long-turn spoken dialogue.

• The paper's main contribution is demonstrating that RLHF with rubric-based reward modeling significantly enhances both conversational naturalness and analytical reasoning in audio language models.
• The model architecture integrates a Qwen2 audio encoder with a Qwen2.5-32B LLM decoder, leveraging temporal downsampling and chain-of-thought reasoning to maintain long-context dialogue.
• Empirical results show a marked performance boost, surpassing previous iterations and rivaling commercial models on multi-turn conversational and paralinguistic benchmarks.

Step-Audio-R1.5: RLHF-Driven Audio Language Modeling for Conversational Naturalness

Introduction

Step-Audio-R1.5 presents a significant methodological advancement for audio LLMs by systematically integrating Reinforcement Learning from Human Feedback (RLHF) into the audio reasoning paradigm, specifically targeting the limitations imposed by Reinforcement Learning with Verified Rewards (RLVR). The work rigorously defines the "verifiable reward trap," arguing that prevailing RLVR strategies—optimized exclusively for discrete, checkable text labels—are fundamentally misaligned with the perceptual and interactive demands of real-world auditory dialogue systems. The central thesis is that, while RLVR ensures semantic correctness, it leads to conversational degradation, reducing models to "answering machines" devoid of prosodic nuance, emotional continuity, and immersive interactivity.

Theoretical Framework: The "Verifiable Reward Trap" and its Resolution

The core critique levied by the report is the RLVR-induced tunnel vision toward factual correctness at the expense of conversational quality. When models are trained solely to maximize objective metrics derived from discrete answer verification, the consequential loss is in their incapacity to maintain context, attend to paralinguistic cues, and exhibit natural language variation during long-turn spoken interactions. This is empirically observed as progressively mechanical outputs as RLVR training proceeds.

Step-Audio-R1.5 circumvents this trap through a unified RLHF framework. Preference-based reward modeling is introduced, utilizing both holistic human judgment and rubric-based scoring. This moves the optimization target from strictly what to say towards how to say it, realigning policy gradients to favor outputs that blend analytical rigor with experiential naturalness.

Model Architecture

Step-Audio-R1.5 extends the Step-Audio-R1 architecture, introducing several key modifications:

• Audio Encoder: Utilizes the Qwen2 audio encoder pretrained for broad speech perception, frozen during all alignment stages to avoid catastrophic forgetting.
• Audio Adaptor: Applies temporal downsampling to compress sequence length, maintaining tractability during long-context compositions.
• LLM Decoder: Initializes from Qwen2.5-32B, ingesting downsampled representations and producing exclusively textual outputs. Importantly, the decoding process is structurally partitioned: intermediate Chain-of-Thought reasoning traces are synthesized before final reply generation. This architectural decoupling is essential for seamlessly integrating RLHF, allowing for separate yet unified optimization of analytical and interactive aspects.

Training Regimen

The training pipeline comprises three stages:

1. Audio-Centric Mid-Training: Jointly optimizes on audio-grounded and text-only reasoning tasks to strengthen both perceptual and analytical capacities across modalities.
2. Cold-start Supervised Fine-tuning: Aligns the model towards interaction-oriented behavior, emphasizing multi-turn continuity, instruction retention, response naturalness, and robustness to conversational events (e.g., interruptions or clarifications).
3. RLHF with Rubric-based Generated Reward Model: Introduces a reward model supporting both explicit rubric-gated evaluation and ordinary pairwise preference comparison. This enables a nuanced, fine-grained reward signal, optimized via a PPO-style objective, to align both instruction-sensitive and preference-sensitive behaviors in joint training. The preference model outputs relative quality judgments rather than absolute scores, reflecting the comparative nature of human conversational assessment.

Empirical Results

Comprehensive evaluation is performed across eight reasoning and perception benchmarks, including AudioMultiChallenge (Audio MC), Step-Caption, Step-Dialogue-Understanding (Step-DU), and StepEval-Audio-Paralinguistic (Step-SPQA).

Step-Audio-R1.5 attains a mean aggregate score of 77.97 across all benchmarks, surpassing its direct predecessor by 5.47 points and demonstrating competitive performance relative to commercial state-of-the-art models such as Gemini 3 Pro, despite a smaller parameter count (32B vs. larger proprietary systems). The largest gains are observed on tasks emphasizing multi-turn, long-context understanding, notably achieving 41.15 on Audio MC, trailing only the Gemini architectures. There is also clear improvement on perceptual and paralinguistic tasks, with considerable advances on Step-DU (+18.39) and Step-SPQA (+5.04).

Figure 1: Aggregate performance of Step-Audio-R1.5 across eight speech-to-text reasoning and perception benchmarks, illustrating its competitiveness with state-of-the-art commercial models.

The aggregate evaluation substantiates that RLHF-oriented training does not compromise analytic reasoning, but instead systematically enhances interaction quality, manifesting as richer, more coherent, and more emotionally resonant long-turn dialogues.

Practical and Theoretical Implications

The findings emphatically demonstrate that the previously observed conversational flatness in audio LLMs is not a necessary byproduct of Chain-of-Thought (CoT) architectures or the auditory modality itself, but rather a reversible consequence of reward signal impoverishment. By incorporating RLHF and rubric-driven preference modeling, Step-Audio-R1.5 establishes that alignment with human evaluative preferences is both feasible and essential for closing the experience gap in audio AI.

Practically, this paradigm enables the development of interactive spoken agents able to maintain contextual fidelity, respond naturally to user instructions, and navigate real-world dialogue phenomena—properties critical for deployment in assistive technologies, education, and open-domain conversational agents.

Theoretically, the work recontextualizes reward modeling in multimodal domains: discrete, verifiable rewards are insufficient for capturing the high-dimensional, subjective, and often weakly specifiable qualities that determine human satisfaction in AI-mediated interaction. RLHF thus emerges as a necessary and generalizable corrective, not only in text but across sensory modalities.

Future Prospects

The modular, joint-optimization structure of Step-Audio-R1.5 suggests clear directions for future research:

• Scaling RLHF Supervisory Data: Expanded and more diverse forms of human feedback—including prosodic, emotional, and multimodal preference signals—could further enhance naturalness and user experience.
• Cross-modal Compositionality: Incorporating video or other sensory inputs may extend the scope of holistic interactive AI.
• Robustness to Noisy Feedback: Research into reward modeling robust to annotator disagreement or adversarial preference signals could ensure reliable alignment without overfitting to particular annotator idiosyncrasies.

Conclusion

Step-Audio-R1.5 makes a compelling case for RLHF as an indispensable ingredient in the evolution of audio reasoning models, resolving the reward misalignment responsible for prior conversational deficiencies. By realigning optimization objectives from factual correctness to holistic interaction quality, the model achieves robust analytical performance without sacrificing the prosodic and experiential dimensions critical to human-machine communication. These results reorient future efforts away from narrow benchmark maximization toward comprehensive sensory empathy as the defining challenge for next-generation audio AI.