AdaSR: Adaptive Streaming Reasoning with Hierarchical Relative Policy Optimization

Abstract: Large reasoning models typically follow a read-then-think paradigm: they observe the complete input, reason over a static context, and then produce the answer. Yet many real-world scenarios are inherently dynamic, such as audio and video stream, where information arrives as a continuous stream and models must reason, update, and respond under partial observations. Recent streaming reasoning methods allow models to think while reading, but they largely rely on supervised imitation of pre-constructed trajectories, which limits their flexibility. In this paper, we propose AdaSR, an adaptive streaming reasoning framework that enables models to reason during input streaming and perform final deliberation once the stream is complete, learning when to think, and how much computation to allocate across different stages. To optimize this hierarchical reasoning process, we introduce Hierarchical Relative Policy Optimization (HRPO), which decomposes policy optimization into streaming reasoning and deep reasoning phases, providing more fine-grained advantage assignment instead of uniformly distributing a single sequence-level advantage over all tokens. HRPO integrates format, accuracy, and adaptive thinking rewards to enforce valid reasoning protocols, preserve final task performance, and encourage latency-aware computation allocation. Experiments show that AdaSR achieves a better balance among reasoning accuracy, computational efficiency, and streaming latency compared with supervised fine-tuning baseline. We release our code at https://github.com/EIT-NLP/StreamingLLM/tree/main/AdaSR.

• The paper introduces AdaSR, which decouples streaming from deep reasoning to enable online, incremental inference under partial observations.
• It pioneers Hierarchical Relative Policy Optimization (HRPO) to decompose credit assignment across streaming, deep, and global phases for enhanced decision-making.
• Experimental results demonstrate substantial accuracy gains and reduced latency across benchmarks like GSM-Symbolic and MetaMathQA.

Adaptive Streaming Reasoning with Hierarchical Relative Policy Optimization: A Technical Analysis

Introduction and Motivation

The traditional chain-of-thought (CoT) paradigm in LLM reasoning assumes full input visibility prior to deliberation. This read-then-think protocol incurs suboptimal latency in domains where inputs arrive as streams, such as speech or real-time sensory data. Streaming reasoning seeks to interleave reasoning with input perception—permitting online, incremental inference under partial observations. Prior approaches to streaming CoT have predominantly employed supervised imitation of expert-constructed trajectories, restricting adaptability and generality.

"AdaSR: Adaptive Streaming Reasoning with Hierarchical Relative Policy Optimization" (2606.14694) introduces a reinforcement learning (RL)-based framework to autonomously learn computation allocation policies in streaming contexts. By formulating an explicit separation between streaming and deep (post-stream) reasoning, AdaSR enables LLMs to dynamically decide when to think, when to skip, and how to balance computation for minimal latency and maximal task performance. At its core, AdaSR develops Hierarchical Relative Policy Optimization (HRPO), which decomposes trajectory-level credit assignment into phase-specific (streaming, deep, and global) advantages—a theoretically motivated refinement of the standard Group Relative Policy Optimization (GRPO) technique.

Figure 1: AdaSR enables online reasoning over streaming inputs, with HRPO providing fine-grained temporal credit assignment aligning with the underlying hierarchical structure of streamed reasoning.

AdaSR Framework and Methodological Innovations

AdaSR couples a hierarchical RL objective with a streaming rollout engine, establishing a foundation for phase-structured credit assignment within reasoning trajectories.

Streaming Reasoning Paradigm

AdaSR's reasoning protocol is instantiated as follows:

• For each revealed input segment, the model emits a streaming "thought" under current partial context, or may opt to skip.
• After the input stream completes, a final deep deliberation integrates all prior context and thoughts, generating the ultimate answer.

Contrastingly, in read-then-think paradigms, reasoning and output are strictly deferred until full context arrival, resulting in increased response time.

Figure 2: Streaming thinking enables concurrent reasoning and input perception, unlike read-then-think which accumulates input before any computation, causing delays.

Hierarchical Relative Policy Optimization (HRPO)

GRPO assigns a single normalized advantage to all tokens in a rollout. This sequence-level allocation fails for streaming settings: reward signals cannot distinguish between the utility of tokens generated under incomplete versus complete observations, creating a cross-phase credit paradox.

HRPO addresses this by decomposing the advantage into three temporally aligned levels:

1. Streaming-local advantage: Assigned to tokens generated in the streaming phase, capturing the value of reasoning under partial context.
2. Deep-local advantage: Assigned to tokens generated during the final deliberation phase, crediting synthesis over complete context.
3. Trajectory-global advantage: Assigned across the full rollout, reflecting end-to-end task outcome.

HRPO's phase-structured surrogate objective supports both phase-specific and global policy updates. The mixing coefficient $\lambda$ controls the balance between local and global signals, reducing to GRPO when set to zero.

Figure 3: AdaSR training pipeline: streaming rollouts are constructed with position encoding and segmental caching, enabling consistent advantage computation for streaming-aware RL updates.

Reward Decomposition

AdaSR's reward function incorporates three tightly coupled components:

• Format reward: Enforces syntactic correctness for intermediate and final outputs, ensuring parseability for phase assignments.
• Accuracy reward: Anchors the global objective with the final answer correctness signal.
• Adaptive thinking reward: Implements a length-based shaping penalty and latency discounts, incentivizing concise yet sufficient computation in both streaming and deep phases. Latency-aware discounting further prioritizes reasoning performed during the input stream, as this computation overlaps with real-time data arrival.

Experimental Results and Empirical Claims

AdaSR is evaluated on mathematical (GSM-Symbolic, MetaMathQA, GSM-Infinite), biomedical (PubMedQA), and logic (LogicNLI) datasets using the Qwen3 LLM series. Key benchmarks involve problems naturally segmentable to simulate streamed input.

AdaSR with HRPO achieves consistent and often substantial accuracy improvements versus both SFT-based StreamingThinker and GRPO baselines, with stronger gains on more challenging, longer-form benchmarks:

• On GSM-symbolic P2 with Qwen3-1.7B, accuracy improves from 0.642 (SFT) and 0.758 (GRPO) to 0.788 (HRPO), while reducing deep-stage reasoning length by an order of magnitude compared to read-then-think.
• For Qwen3-4B on MetaMathQA, accuracy rises from 0.860 (SFT) to 0.924 (HRPO) with a 13.4% reduction in total generated length relative to SFT.

AdaSR's streaming-deep computation division yields a more favorable accuracy/efficiency frontier. Compared to sequence-level RL, hierarchical advantage assignment enables AdaSR to suppress unnecessary token emissions—reducing redundancy—without sacrificing global performance. Notably, token-level advantage assignment (overly fine granularity) degrades performance versus coarser, phase-aligned divisions, highlighting the necessity of granularity aligned with the semantic structure of streaming reasoning.

Figure 4: HRPO delivers superior training dynamics with higher reward curves, better accuracy, and confident policy entropy declining steadily during training, as compared to GRPO and fine-grained variants.

AdaSR generalizes effectively in out-of-distribution (GSM-Infinite) and out-of-task (LogicNLI) scenarios, maintaining or improving accuracy while keeping total reasoning length compact.

Practical Implications and Theoretical Perspectives

The implications of AdaSR are multifaceted:

• Practical serving: AdaSR's computation reallocation to streaming stages provides more than an $8\times$ reduction in post-input latency on real servers (e.g., vLLM), critical for interactive real-time applications.
• Adaptive computation and RL-over-temporal structure: This framework formalizes RLHF/VRL objective decomposition across temporal/structural boundaries, offering a blueprint for adaptive, context-sensitive credit assignment in any phase-structured generative process.
• Beyond text: While the present work focuses on segmented text streams with verifiable outputs, the architectural principles of AdaSR—hierarchical advantage assignment, streaming rollouts, and adaptive reward shaping—are extensible to continuous audio, video, and multimodal settings.
• Open challenges: Extending AdaSR to continuous domains or temporally unaligned modalities will require reward designs that blend format, delayed accuracy, and richer phase-aware signals, as well as scalable rollout infrastructure for dynamic trajectory segmentation.

Comparative qualitative analysis further supports the capability of RL-trained AdaSR models to mitigate propagation of streaming-phase errors into the final answer, a limitation frequently observed in SFT-trained baselines.

Figure 5: RL-trained AdaSR corrects intermediate computational steps during streaming, avoiding error propagation found in SFT-based reasoning.

Conclusion

AdaSR establishes that effective streaming reasoning in LLMs demands RL objectives tailored to the unique temporal structure of streamed input and output. Its HRPO method overcomes fundamental credit assignment failures of sequence-level RL in this context, yielding models that are more accurate, efficient, and responsive under streaming input dynamics. The theoretical and practical insights presented by AdaSR provide a foundation for future developments in adaptive LLM computation—potentially spanning real-time multimodal reasoning and agentic interaction in dynamically evolving environments.