Hierarchical Relative Policy Optimization (HRPO)

• The paper introduces HRPO, a method that decomposes credit assignment into streaming-local, deep-local, and trajectory-global signals for reinforcement learning.
• HRPO redefines relative policy updates by structuring token generation into phases, mitigating credit-assignment mismatches found in flat GRPO approaches.
• Empirical results show that HRPO improves accuracy and reduces generation length compared to standard methods, highlighting its efficiency in streaming reasoning.

Searching arXiv for papers directly relevant to Hierarchical Relative Policy Optimization and closely related hierarchical/relative policy optimization methods. Hierarchical Relative Policy Optimization (HRPO) is a reinforcement-learning formulation in which relative policy updates are coupled to an explicitly hierarchical credit-assignment scheme. The term is used most explicitly in AdaSR, where HRPO is introduced for adaptive streaming reasoning and decomposes optimization into a streaming reasoning phase and a deep reasoning phase, rather than assigning a single sequence-level advantage uniformly to all generated tokens (Tong et al., 12 Jun 2026). In current arXiv usage, however, the phrase is not fully standardized. Closely related papers use hierarchical PPO-style ratios, group-relative objectives, tree-structured rollouts, or pairwise hierarchical preferences, but they do not all define the same algorithmic object. The resulting literature is best read as a set of neighboring formulations for combining hierarchy with relative policy improvement.

1. Terminology and conceptual scope

In AdaSR, HRPO is motivated by a mismatch between standard sequence-level RL and temporally structured reasoning. A streaming trajectory contains at least two qualitatively different stages: a streaming reasoning phase under partial observations and a deep reasoning phase after the full context is available. Standard GRPO collapses such a rollout to one scalar reward and assigns the same normalized advantage to every token, which the paper characterizes as a temporal credit-assignment mismatch and a cross-phase credit paradox: deep-phase success can reinforce poor streaming thoughts, while deep-phase failure can punish useful streaming reasoning (Tong et al., 12 Jun 2026).

This usage makes “hierarchical” refer first to a hierarchy of temporal phases and credit signals, not to options, macro-actions, or separate high-level and low-level neural policies. The same paper is also explicit that the method remains critic-free and GRPO-like in its use of clipped relative policy updates with group-normalized rewards. Current terminology therefore suggests a narrow, explicit sense of HRPO in AdaSR and a broader, looser sense across adjacent work.

2. Streaming reasoning as the canonical HRPO setting

AdaSR contrasts standard read-then-think reasoning with streaming reasoning by factorizing the trajectory differently. For standard reasoning, the paper writes

$$\mathcal{P}_{\mathrm{standard}} = \prod_{t=1}^{T} P(R_t \mid Q, C_{\leq T}, R_{\leq t-1}),$$

whereas streaming reasoning is

$$\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).$$

Here $Q$ is the question, $C=\{C_1,\ldots,C_T\}$ is the streamed context, $R_t$ is the local reasoning segment produced after observing $C_{\le t}$, and $R$ is the final deep reasoning once the stream is complete. The phase boundary is denoted $t_{\mathrm{s}}$ (Tong et al., 12 Jun 2026).

A sampled rollout is written as

$o_i=[R_1,\ldots,R_T,R].$

The action space is not cast as a classical finite-action MDP. Instead, the policy is an autoregressive LLM whose actions are tokens emitted under phase-specific visibility constraints. During streaming, each segment must terminate with <EOT>, and the content before <EOT> is either a reasoning thought or <skip>. During deep reasoning, the model produces a non-empty deep reasoning field ending with <EOR>. This means the same policy $\pi_\theta$ implicitly learns both whether to think and how much computation to spend, but does so through token generation under a structured rollout protocol rather than through an external scheduler (Tong et al., 12 Jun 2026).

3. Objective: from flat GRPO to hierarchical relative optimization

AdaSR begins from the GRPO baseline

$$\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).$$0

with

$$\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).$$1

HRPO replaces the single trajectory-level advantage with three signals: streaming-local $$\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).$$2, deep-local $$\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).$$3, and trajectory-global $$\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).$$4. These are attached to token ranges

$$\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).$$5

Thus,

$$\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).$$6

The resulting objective is

$$\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).$$7

When $$\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).$$8, the method reduces to purely trajectory-level optimization. The same tokenwise ratio is used across levels; the superscripts indicate which advantage branch the ratio is paired with, not distinct probability models. The time-grouped form makes the semantics explicit: streaming tokens receive streaming-local and global credit, while deep tokens receive deep-local and global credit (Tong et al., 12 Jun 2026).

This decomposition changes the effective advantage seen by each token. Near $$\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).$$9, the streaming tokens receive

$Q$0

whereas deep tokens receive

$Q$1

A plausible implication is that HRPO preserves the relative-policy character of GRPO while making the attribution phase-aware rather than sequence-flat.

4. Reward decomposition, advantage construction, and rollout mechanics

The local and global advantages in AdaSR are built from decomposed rewards rather than from a single terminal scalar. The reward components are:

• Format rewards: $Q$2 for streaming-format validity and $Q$3 for deep-format validity. Streaming segments must terminate with <EOT>, and the deep reasoning field must be non-empty and end with <EOR>.
• Accuracy reward:

$Q$4

• Local length penalties:

$Q$5

• Global efficiency reward:

$Q$6

Here $Q$7 is the total streaming reasoning length, $Q$8 is the deep reasoning length, $Q$9 discounts streaming latency relative to deep latency, and $C=\{C_1,\ldots,C_T\}$0 is a scale parameter (Tong et al., 12 Jun 2026).

Advantages are then composed by separately normalizing each reward component within the rollout group: $C=\{C_1,\ldots,C_T\}$1 The paper defines

$C=\{C_1,\ldots,C_T\}$2

$C=\{C_1,\ldots,C_T\}$3

$C=\{C_1,\ldots,C_T\}$4

The notation is overloaded: $C=\{C_1,\ldots,C_T\}$5 denotes the adaptive-thinking coefficient in the advantage composition, while the formal objective also uses $C=\{C_1,\ldots,C_T\}$6 as the KL coefficient. The appendix further notes that weighting should be applied after normalization; otherwise, reward-level coefficients can cancel under group normalization (Tong et al., 12 Jun 2026).

The rollout and update pipeline is similarly hierarchical. AdaSR initializes $C=\{C_1,\ldots,C_T\}$7 from a streaming SFT checkpoint, splits context into sentences, samples $C=\{C_1,\ldots,C_T\}$8 trajectories from $C=\{C_1,\ldots,C_T\}$9, decodes each streaming segment until <EOT>, decodes the final deep segment, computes reward components, composes $R_t$0, assigns token-range advantages, and updates $R_t$1 by gradient ascent. The implementation extends vLLM with a round-based state machine, grouped position IDs, and streaming visibility masks so that old-policy, reference-policy, and current-policy log-probabilities are all computed under the same partial-observation structure (Tong et al., 12 Jun 2026).

5. Related formulations and competing interpretations

The literature surrounding HRPO is heterogeneous. Several nearby methods are substantively related, but they instantiate “hierarchical” and “relative” in different ways.

Method	Relation to HRPO	Core distinction
AdaSR HRPO (Tong et al., 12 Jun 2026)	Explicit use of “Hierarchical Relative Policy Optimization”	Phase-structured GRPO for streaming and deep reasoning
HiFPO (Liu et al., 18 Jun 2026)	Close variant	Hierarchical feedback pipeline wrapped around hint-contextualized step-level GRPO
T-GRPO (Cao et al., 7 Oct 2025)	Tree-structured relative optimization	GRPO-style updates over video-event-frame trees with node- and tree-level rewards
HiPPO (Li et al., 2019)	PPO-style hierarchical relative-ratio optimization	Separate clipped ratios for manager and worker policies
BrHPO (Luo et al., 2024)	Hierarchical policy optimization, but not PPO-style relative optimization	Reachability-aware SAC with bidirectional subgoal reachability
HPO (Singh et al., 2024)	Hierarchical relative/preference neighbor	Primitive-regularized token-level DPO for feasible subgoal prediction

HiFPO is introduced in MobileForge as “Hierarchical Feedback-Guided Policy Optimization,” and the paper is explicit that it does not replace GRPO with a new optimizer. Its hierarchy is primarily a feedback hierarchy over trajectory outcomes, step-level process labels, and corrective hints, while relative optimization still occurs through a per-step GRPO objective with group-normalized advantages (Liu et al., 18 Jun 2026). T-GRPO in VideoMiner is similarly not named HRPO, but it adapts GRPO to a hierarchical video tree and decomposes reward into node-level and tree-level terms (Cao et al., 7 Oct 2025). HiPPO, by contrast, is a conventional two-level HRL algorithm with per-level PPO ratios,

$R_t$2

and is therefore very close to a classical hierarchical relative-ratio optimizer (Li et al., 2019).

Other neighboring papers delimit the concept by contrast. BrHPO is a hierarchical SAC method whose only ratio is a subgoal reachability ratio, so it is explicitly not PPO/TRPO-style relative policy optimization (Luo et al., 2024). HPO is a hierarchical preference optimization method in which pairwise trajectory preferences and primitive regularization replace clipped policy-ratio objectives (Singh et al., 2024). Current usage therefore suggests that “HRPO” can denote at least four distinct structures: phase hierarchy, feedback hierarchy, tree hierarchy, or manager-worker policy hierarchy. It also suggests that “relative” can refer to group-normalized rollout comparisons, old/new policy ratios, pairwise preference margins, or geometric progress ratios, depending on the paper.

6. Empirical behavior, implementation profile, and limitations

In AdaSR, HRPO improves the accuracy-efficiency frontier relative to both streaming SFT baselines and flat GRPO. The paper states that, compared with GRPO, HRPO improves accuracy and reduces total generation length across all eight settings in the main table. On GSM-symbolic P2 with Qwen3-1.7B, StreamingThinker SFT reaches $R_t$3 accuracy, AdaSR-GRPO reaches $R_t$4, and AdaSR-HRPO reaches $R_t$5; HRPO also reduces streaming length to $R_t$6 from $R_t$7 and total length to $R_t$8 from $R_t$9. On MetaMathQA, the same model improves from $C_{\le t}$0 under SFT to $C_{\le t}$1 under HRPO. On Qwen3-4B MetaMathQA, SFT reaches $C_{\le t}$2, GRPO $C_{\le t}$3, and HRPO $C_{\le t}$4, with HRPO also shortening total length from $C_{\le t}$5 to $C_{\le t}$6 (Tong et al., 12 Jun 2026).

The latency evidence is equally central. All streaming methods in the paper achieve over $C_{\le t}$7 latency reduction relative to read-then-think. HRPO preserves that advantage while shifting computation toward the streaming phase and sharply reducing deep-phase reasoning. On GSM-symbolic P2 with Qwen3-1.7B, deep-stage reasoning drops from $C_{\le t}$8 tokens in read-then-think to $C_{\le t}$9 under HRPO, while accuracy rises from $R$0 to $R$1. Ablations show that stage-level HRPO outperforms GRPO, HRPO-sentence, and HRPO-token, which supports the specific claim that phase-level rather than sentence-level or token-level decomposition best matches the task structure (Tong et al., 12 Jun 2026).

The main training configuration uses $R$2, adaptive reward weight $R$3, latency discount coefficient $R$4, train batch size $R$5, PPO mini-batch size $R$6, learning rate $R$7, temperature $R$8, top-$R$9, top-$t_{\mathrm{s}}$0, and $t_{\mathrm{s}}$1 samples per prompt. Maximum prompt length is $t_{\mathrm{s}}$2, maximum response length is $t_{\mathrm{s}}$3, and the method is initialized from a streaming SFT checkpoint rather than trained from scratch (Tong et al., 12 Jun 2026).

The limitations in AdaSR are explicit. The method is evaluated on text streams with verifiable answers; extending it to audio, video, or open-ended interactive streams would require richer reward design and different rollout infrastructure. The phase decomposition assumes a meaningful split between streaming and deep reasoning. Reward construction is delicate, especially for local length penalties under group normalization. The current rollout mechanism is sentence-level rather than continuous-time. Finally, the notation overload on $t_{\mathrm{s}}$4 and the implementation table entry “KL coefficient: False” indicate some ambiguity between the formal objective and the practical training configuration (Tong et al., 12 Jun 2026).

Taken together, the literature places HRPO at the intersection of hierarchical credit assignment and relative policy improvement, but not yet as a single settled recipe. In its explicit AdaSR form, HRPO is a phase-structured, critic-free, clipped relative objective for streaming reasoning. In the broader ecosystem, it names a family of designs that refine where relative signals are attached inside a hierarchy, whether that hierarchy is temporal, tree-structured, feedback-mediated, or manager-worker.