---
title: Hierarchical Relative Policy Optimization (HRPO)
url: https://www.emergentmind.com/topics/hierarchical-relative-policy-optimization-hrpo
type: topic
---

# Hierarchical Relative Policy Optimization (HRPO)

Searching arXiv for recent papers directly relevant to Hierarchical Relative Policy Optimization and closely related hierarchical/relative policy optimization methods.
Hierarchical Relative Policy Optimization (HRPO) is a reinforcement-learning formulation in which relative policy updates are coupled to an explicitly hierarchical credit-assignment scheme. The term is used most explicitly in AdaSR, where HRPO is introduced for adaptive streaming reasoning and decomposes optimization into a streaming reasoning phase and a deep reasoning phase, rather than assigning a single sequence-level advantage uniformly to all generated tokens [2606.14694]. In current arXiv usage, however, the phrase is not fully standardized. Closely related papers use hierarchical PPO-style ratios, group-relative objectives, tree-structured rollouts, or pairwise hierarchical preferences, but they do not all define the same algorithmic object. The resulting literature is best read as a set of neighboring formulations for combining hierarchy with relative policy improvement.

## 1. Terminology and conceptual scope

In AdaSR, HRPO is motivated by a mismatch between standard sequence-level RL and temporally structured reasoning. A streaming trajectory contains at least two qualitatively different stages: a streaming reasoning phase under partial observations and a deep reasoning phase after the full context is available. Standard GRPO collapses such a rollout to one scalar reward and assigns the same normalized advantage to every token, which the paper characterizes as a temporal credit-assignment mismatch and a cross-phase credit paradox: deep-phase success can reinforce poor streaming thoughts, while deep-phase failure can punish useful streaming reasoning [2606.14694].

This usage makes “hierarchical” refer first to a hierarchy of temporal phases and credit signals, not to options, macro-actions, or separate high-level and low-level neural policies. The same paper is also explicit that the method remains critic-free and GRPO-like in its use of clipped relative policy updates with group-normalized rewards. Current terminology therefore suggests a narrow, explicit sense of HRPO in AdaSR and a broader, looser sense across adjacent work.

## 2. Streaming reasoning as the canonical HRPO setting

AdaSR contrasts standard read-then-think reasoning with streaming reasoning by factorizing the trajectory differently. For standard reasoning, the paper writes
\[
\mathcal{P}_{\mathrm{standard}} = \prod_{t=1}^{T} P(R_t \mid Q, C_{\leq T}, R_{\leq t-1}),
\]
whereas streaming reasoning is
\[
\mathcal{P}_{\mathrm{streaming}} = P(R_q \mid Q)\prod_{t=1}^{T} P(R_t \mid Q, C_{\leq t}, R_{\leq t-1}) \cdot P(R \mid Q, C_{\leq T}, R_{\leq T}).
\]
Here \(Q\) is the question, \(C=\{C_1,\ldots,C_T\}\) is the streamed context, \(R_t\) is the local reasoning segment produced after observing \(C_{\le t}\), and \(R\) is the final deep reasoning once the stream is complete. The phase boundary is denoted \(t_{\mathrm{s}}\) [2606.14694].

A sampled rollout is written as
\[
o_i=[R_1,\ldots,R_T,R].
\]
The action space is not cast as a classical finite-action MDP. Instead, the policy is an autoregressive language model whose actions are tokens emitted under phase-specific visibility constraints. During streaming, each segment must terminate with `<EOT>`, and the content before `<EOT>` is either a reasoning thought or `<skip>`. During deep reasoning, the model produces a non-empty deep reasoning field ending with `<EOR>`. This means the same policy \(\pi_\theta\) implicitly learns both whether to think and how much computation to spend, but does so through token generation under a structured rollout protocol rather than through an external scheduler [2606.14694].

## 3. Objective: from flat GRPO to hierarchical relative optimization

AdaSR begins from the GRPO baseline
\[
J_{\mathrm{GRPO}}(\theta)=\mathbb{E}\Bigg\{ \frac{1}{G}\sum_{i=1}^{G} \Bigg[ \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \mathcal{C}\!\left(r_{i,t}(\theta), \hat{A}_i\right) - \beta\, \operatorname{KL}\!\left( \pi_{\theta} \,\|\, \pi_{\mathrm{ref}} \right) \Bigg] \Bigg\},
\]
with
\[
r_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}\mid q,o_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(o_{i,t}\mid q,o_{i,<t})},
\qquad
\hat{A}_{i}=\frac{R_i-\mu(\{R_j\}_{j=1}^G)}{\sigma(\{R_j\}_{j=1}^G)}.
\]
HRPO replaces the single trajectory-level advantage with three signals: streaming-local \(A_i^s\), deep-local \(A_i^d\), and trajectory-global \(A_i^g\). These are attached to token ranges
\[
\mathcal{T}_i^s=[1,|t_{\mathrm{s}}|],\qquad
\mathcal{T}_i^d=(|t_{\mathrm{s}}|,|o_i|],\qquad
\mathcal{T}_i^g=[1,|o_i|].
\]
Thus,
\[
\hat{A}_{i,t}^{\ell}=A_i^{\ell},\quad t\in\mathcal{T}_i^\ell,\quad \ell\in\{s,d,g\}.
\]

The resulting objective is
\[
\begin{aligned}
J_{\mathrm{HRPO}}(\theta)=\mathbb{E}\Bigg\{ \frac{1}{G}\sum_{i=1}^{G} \Bigg[
&\frac{\lambda}{|t_{\mathrm{s}}|}\sum_{t=1}^{|t_{\mathrm{s}}|}\mathcal{C}(r_{i,t}^{s},\hat{A}_{i,t}^{s})
+\frac{\lambda}{|o_i|-|t_{\mathrm{s}}|}\sum_{t=|t_{\mathrm{s}}|+1}^{|o_i|}\mathcal{C}(r_{i,t}^{d},\hat{A}_{i,t}^{d})\\
&+\frac{1}{|o_i|}\sum_{t=1}^{|o_i|}\Big(\mathcal{C}(r_{i,t}^{g},\hat{A}_{i,t}^{g})-\beta\operatorname{KL}(\pi_\theta\|\pi_{\mathrm{ref}})\Big)
\Bigg]\Bigg\}.
\end{aligned}
\]
When \(\lambda=0\), the method reduces to purely trajectory-level optimization. The same tokenwise ratio is used across levels; the superscripts indicate which advantage branch the ratio is paired with, not distinct probability models. The time-grouped form makes the semantics explicit: streaming tokens receive streaming-local and global credit, while deep tokens receive deep-local and global credit [2606.14694].

This decomposition changes the effective advantage seen by each token. Near \(\theta=\theta_{\mathrm{old}}\), the streaming tokens receive
\[
\frac{\lambda}{|t_{\mathrm{s}}|}\hat{A}_{i,t}^{s}+\frac{1}{|o_i|}\hat{A}_{i,t}^{g},
\]
whereas deep tokens receive
\[
\frac{\lambda}{|o_i|-|t_{\mathrm{s}}|}\hat{A}_{i,t}^{d}+\frac{1}{|o_i|}\hat{A}_{i,t}^{g}.
\]
A plausible implication is that HRPO preserves the relative-policy character of GRPO while making the attribution phase-aware rather than sequence-flat.

## 4. Reward decomposition, advantage construction, and rollout mechanics

The local and global advantages in AdaSR are built from decomposed rewards rather than from a single terminal scalar. The reward components are:

- **Format rewards**: \(R_{i,\mathrm{fmt}^s}\) for streaming-format validity and \(R_{i,\mathrm{fmt}^d}\) for deep-format validity. Streaming segments must terminate with `<EOT>`, and the deep reasoning field must be non-empty and end with `<EOR>`.
- **Accuracy reward**:
  \[
  R_i^{\mathrm{acc}}=\mathbbm{1}[\hat{a}_i=a_i].
  \]
- **Local length penalties**:
  \[
  R_i^{L_s}=\frac{1}{N}\sum_{n=1}^{N} -\log(1+|s_{i,n}|),\qquad
  R_i^{L_d}=-\log(1+L_i^D).
  \]
- **Global efficiency reward**:
  \[
  \begin{aligned}
  R_i^{\mathrm{eff}} &=
  R_i^{\mathrm{acc}} R_{i,\mathrm{fmt}^s} R_{i,\mathrm{fmt}^d}
  \left(1-\exp\!\left(-\frac{L_i^D}{\tau}\right)\right)
  \left(1-\exp\!\left(-\frac{L_i^S}{\tau}\right)\right)
  \exp\!\left(-\frac{\alpha L_i^S+L_i^D}{\tau}\right).
  \end{aligned}
  \]
Here \(L_i^S\) is the total streaming reasoning length, \(L_i^D\) is the deep reasoning length, \(0<\alpha<1\) discounts streaming latency relative to deep latency, and \(\tau>0\) is a scale parameter [2606.14694].

Advantages are then composed by separately normalizing each reward component within the rollout group:
\[
\mathcal{N}_{\mathcal{G}}(X_i)=\frac{X_i-\mu(\{X_j\}_{j=1}^G)}{\sigma(\{X_j\}_{j=1}^G)}.
\]
The paper defines
\[
A_i^s=\mathcal{N}_{\mathcal{G}}(R_{i,\mathrm{fmt}^s})+\beta\mathcal{N}_{\mathcal{G}}(R_i^{L_s}),
\]
\[
A_i^d=\mathcal{N}_{\mathcal{G}}(R_{i,\mathrm{fmt}^d})+\beta\mathcal{N}_{\mathcal{G}}(R_i^{L_d}),
\]
\[
A_i^g=\mathcal{N}_{\mathcal{G}}(R_i^{\mathrm{acc}})+\beta\mathcal{N}_{\mathcal{G}}(R_i^{\mathrm{eff}}).
\]
The notation is overloaded: \(\beta\) denotes the adaptive-thinking coefficient in the advantage composition, while the formal objective also uses \(\beta\) as the KL coefficient. The appendix further notes that weighting should be applied after normalization; otherwise, reward-level coefficients can cancel under group normalization [2606.14694].

The rollout and update pipeline is similarly hierarchical. AdaSR initializes \(\pi_\theta\) from a streaming SFT checkpoint, splits context into sentences, samples \(G\) trajectories from \(\pi_{\theta_{\mathrm{old}}}\), decodes each streaming segment until `<EOT>`, decodes the final deep segment, computes reward components, composes \(A_i^s,A_i^d,A_i^g\), assigns token-range advantages, and updates \(\theta\) by gradient ascent. The implementation extends vLLM with a round-based state machine, grouped position IDs, and streaming visibility masks so that old-policy, reference-policy, and current-policy log-probabilities are all computed under the same partial-observation structure [2606.14694].

## 5. Related formulations and competing interpretations

The literature surrounding HRPO is heterogeneous. Several nearby methods are substantively related, but they instantiate “hierarchical” and “relative” in different ways.

| Method | Relation to HRPO | Core distinction |
|---|---|---|
| AdaSR HRPO [2606.14694] | Explicit use of “Hierarchical Relative Policy Optimization” | Phase-structured GRPO for streaming and deep reasoning |
| HiFPO [2606.19930] | Close variant | Hierarchical feedback pipeline wrapped around hint-contextualized step-level GRPO |
| T-GRPO [2510.06040] | Tree-structured relative optimization | GRPO-style updates over video-event-frame trees with node- and tree-level rewards |
| HiPPO [1906.05862] | PPO-style hierarchical relative-ratio optimization | Separate clipped ratios for manager and worker policies |
| BrHPO [2406.18053] | Hierarchical policy optimization, but not PPO-style relative optimization | Reachability-aware SAC with bidirectional subgoal reachability |
| HPO [2411.00361] | Hierarchical relative/preference neighbor | Primitive-regularized token-level DPO for feasible subgoal prediction |

HiFPO is introduced in MobileForge as “Hierarchical Feedback-Guided Policy Optimization,” and the paper is explicit that it does not replace GRPO with a new optimizer. Its hierarchy is primarily a feedback hierarchy over trajectory outcomes, step-level process labels, and corrective hints, while relative optimization still occurs through a per-step GRPO objective with group-normalized advantages [2606.19930]. T-GRPO in VideoMiner is similarly not named HRPO, but it adapts GRPO to a hierarchical video tree and decomposes reward into node-level and tree-level terms [2510.06040]. HiPPO, by contrast, is a conventional two-level HRL algorithm with per-level PPO ratios,
\[
w_{h,kp}(\theta)=\frac{\pi_{\theta_h}(z_{kp}\mid s_{kp})}{\pi_{\theta_{h,\mathrm{old}}}(z_{kp}\mid s_{kp})},
\qquad
w_{l,t}(\theta)=\frac{\pi_{\theta_l}(a_t\mid s_t,z_{kp})}{\pi_{\theta_{l,\mathrm{old}}}(a_t\mid s_t,z_{kp})},
\]
and is therefore very close to a classical hierarchical relative-ratio optimizer [1906.05862].

Other neighboring papers delimit the concept by contrast. BrHPO is a hierarchical SAC method whose only ratio is a subgoal reachability ratio, so it is explicitly not PPO/TRPO-style relative policy optimization [2406.18053]. HPO is a hierarchical preference optimization method in which pairwise trajectory preferences and primitive regularization replace clipped policy-ratio objectives [2411.00361]. Current usage therefore suggests that “HRPO” can denote at least four distinct structures: phase hierarchy, feedback hierarchy, tree hierarchy, or manager-worker policy hierarchy. It also suggests that “relative” can refer to group-normalized rollout comparisons, old/new policy ratios, pairwise preference margins, or geometric progress ratios, depending on the paper.

## 6. Empirical behavior, implementation profile, and limitations

In AdaSR, HRPO improves the accuracy-efficiency frontier relative to both streaming SFT baselines and flat GRPO. The paper states that, compared with GRPO, HRPO improves accuracy and reduces total generation length across all eight settings in the main table. On GSM-symbolic P2 with Qwen3-1.7B, StreamingThinker SFT reaches \(0.642\) accuracy, AdaSR-GRPO reaches \(0.758\), and AdaSR-HRPO reaches \(0.788\); HRPO also reduces streaming length to \(210.046\) from \(235.784\) and total length to \(370.256\) from \(384.488\). On MetaMathQA, the same model improves from \(0.688\) under SFT to \(0.826\) under HRPO. On Qwen3-4B MetaMathQA, SFT reaches \(0.860\), GRPO \(0.909\), and HRPO \(0.924\), with HRPO also shortening total length from \(253.555\) to \(237.358\) [2606.14694].

The latency evidence is equally central. All streaming methods in the paper achieve over \(8\times\) latency reduction relative to read-then-think. HRPO preserves that advantage while shifting computation toward the streaming phase and sharply reducing deep-phase reasoning. On GSM-symbolic P2 with Qwen3-1.7B, deep-stage reasoning drops from \(1866.474\) tokens in read-then-think to \(160.210\) under HRPO, while accuracy rises from \(0.424\) to \(0.788\). Ablations show that stage-level HRPO outperforms GRPO, HRPO-sentence, and HRPO-token, which supports the specific claim that phase-level rather than sentence-level or token-level decomposition best matches the task structure [2606.14694].

The main training configuration uses \(\lambda=0.05\), adaptive reward weight \(\beta=0.05\), latency discount coefficient \(\alpha=0.5\), train batch size \(16\), PPO mini-batch size \(16\), learning rate \(2\times10^{-6}\), temperature \(0.6\), top-\(p=0.95\), top-\(k=20\), and \(12\) samples per prompt. Maximum prompt length is \(8192\), maximum response length is \(2048\), and the method is initialized from a streaming SFT checkpoint rather than trained from scratch [2606.14694].

The limitations in AdaSR are explicit. The method is evaluated on text streams with verifiable answers; extending it to audio, video, or open-ended interactive streams would require richer reward design and different rollout infrastructure. The phase decomposition assumes a meaningful split between streaming and deep reasoning. Reward construction is delicate, especially for local length penalties under group normalization. The current rollout mechanism is sentence-level rather than continuous-time. Finally, the notation overload on \(\beta\) and the implementation table entry “KL coefficient: False” indicate some ambiguity between the formal objective and the practical training configuration [2606.14694].

Taken together, the literature places HRPO at the intersection of hierarchical credit assignment and relative policy improvement, but not yet as a single settled recipe. In its explicit AdaSR form, HRPO is a phase-structured, critic-free, clipped relative objective for streaming reasoning. In the broader ecosystem, it names a family of designs that refine where relative signals are attached inside a hierarchy, whether that hierarchy is temporal, tree-structured, feedback-mediated, or manager-worker.

Source: https://www.emergentmind.com/topics/hierarchical-relative-policy-optimization-hrpo