Hierarchical Feedback-Guided Policy Optimization

• HiFPO is a hierarchical policy learning framework that organizes feedback by authority, subgoal, or step-level granularity to guide optimization.
• It formalizes multi-level optimization mechanisms—using constraint enforcement in HIPO, reachability regularization in BrHPO, and human-guided reward shaping in MENTOR—to separate objectives and feasibility conditions.
• The framework has been applied to diverse domains, demonstrating competitive performance in tasks like mobile GUI adaptation and LLM alignment through structured, dynamic credit assignment.

Hierarchical Feedback-Guided Policy Optimization (HiFPO) denotes a class of policy-learning formulations in which feedback is organized by hierarchy—either by authority, temporal control level, or evaluation granularity—and optimization uses that structure rather than collapsing all signals into a single scalar objective. In the literature considered here, HiFPO appears both as an explicit algorithmic component for mobile GUI agents and as a broader interpretive framework for constrained LLM alignment and hierarchical reinforcement learning: higher-priority instruction feedback can be encoded as constraints, subgoal reachability can couple manager and worker policies, and human preference signals can guide high-level subgoal selection under dynamic feasibility constraints (Liu et al., 18 Jun 2026, Chen et al., 17 Mar 2026, Luo et al., 2024, Zhou et al., 2024).

1. Conceptual scope

HiFPO is defined by the way it treats feedback, not by a single fixed backbone optimizer. In the instruction-following setting of HIPO, the relevant hierarchy is a priority-ordered stack of instructions, typically system $>$ user, in which the system prompt defines a hard boundary within which user instructions can be satisfied. In BrHPO and MENTOR, the hierarchy is the standard two-level HRL decomposition in which a high-level policy proposes subgoals and a low-level policy executes them. In MobileForge, the hierarchy lies in the feedback itself: trajectory-level outcome, step-level process labels, and corrective hints are used for different optimization roles rather than merged into one undifferentiated reward (Chen et al., 17 Mar 2026, Luo et al., 2024, Zhou et al., 2024, Liu et al., 18 Jun 2026).

A central unifying idea is that higher-level signals are treated as feasibility conditions, regularizers, or context modifiers, while lower-level signals drive utility maximization or local action improvement. HIPO states this explicitly: system prompt compliance is encoded as a constraint, and user utility is encoded as the primary objective. BrHPO uses a shared reachability signal to penalize high-level subgoal choices that are hard to reach and to shape low-level reward. MENTOR uses human feedback to optimize the high-level policy, while a Dynamic Distance Constraint (DDC) restricts subgoals to those matching low-level learning progress. MobileForge uses hierarchical feedback to decide which tasks, trajectories, and steps become training signal, then performs hint-contextualized step-level GRPO updates (Chen et al., 17 Mar 2026, Luo et al., 2024, Zhou et al., 2024, Liu et al., 18 Jun 2026).

This suggests that HiFPO is best understood as a family of hierarchical credit-assignment schemes. What varies across instantiations is the semantics of hierarchy: instruction authority, manager-worker decomposition, or evaluator granularity. What remains stable is the refusal to reduce all feedback to a single scalar without structural distinctions.

2. Formal structures

The main formalisms associated with HiFPO-like methods can be summarized concisely.

Setting	Hierarchy	Feedback role
HIPO	system $>$ user	system as constraint, user as objective
BrHPO	manager / worker	reachability regularizer and reward shaping
MENTOR	subgoal policy / dual low-level policies	human-feedback reward plus distance constraint
MobileForge	task / attempt / step	outcome filtering, step selection, hint conditioning

In HIPO, Hierarchical Instruction Following is formulated as a CMDP with prompt $x=[x_{\text{sys}},x_{\text{user}}]$, policy $\pi_\theta(y\mid x)$, user reward $r_{\text{user}}(x,y)\in[0,1]$, and system reward $r_{\text{sys}}(x,y)\in[0,1]$. The objective is

$$\max_\theta \; J_{\text{user}}(\theta)=\mathbb{E}[r_{\text{user}}(x,y)]-\beta\,\mathbb{D}_{KL}(\pi_\theta\parallel \pi_{\text{ref}})$$

subject to

$$J_{\text{sys}}(\theta)=\mathbb{E}[r_{\text{sys}}(x,y)]\ge \tau.$$

Introducing $\lambda\ge 0$ yields the Lagrangian

$$\mathcal{L}(\theta,\lambda)=J_{\text{user}}(\theta)+\lambda(J_{\text{sys}}(\theta)-\tau),$$

so system instructions become algorithmic feasibility requirements rather than extra reward terms (Chen et al., 17 Mar 2026).

BrHPO adopts the standard two-level subgoal HRL setting with high-level policy $>$0, low-level policy $>$1, subtask length $>$2, and bidirectional subgoal reachability

$>$3

Smaller $>$4 means better reachability. This scalar is then inserted into the high-level objective as a regularizer and into the low-level reward as an additive penalty, producing explicitly bidirectional coupling between hierarchical levels (Luo et al., 2024).

MENTOR also uses a two-level HRL formulation, but with a learned human-feedback reward model $>$5 and a learned distance model $>$6. Its high-level problem is

$>$7

with

$>$8

Under Lagrangian relaxation, the high-level objective becomes

$>$9

where $x=[x_{\text{sys}},x_{\text{user}}]$0 is a dual variable, and $x=[x_{\text{sys}},x_{\text{user}}]$1 is dynamically adjusted according to low-level success (Zhou et al., 2024).

MobileForge’s HiFPO models a task $x=[x_{\text{sys}},x_{\text{user}}]$2, attempt $x=[x_{\text{sys}},x_{\text{user}}]$3, and step $x=[x_{\text{sys}},x_{\text{user}}]$4 with decision state

$x=[x_{\text{sys}},x_{\text{user}}]$5

where $x=[x_{\text{sys}},x_{\text{user}}]$6 is corrective hint context aggregated from earlier attempts. MobileGym-Critic returns

$x=[x_{\text{sys}},x_{\text{user}}]$7

consisting of trajectory outcome $x=[x_{\text{sys}},x_{\text{user}}]$8, step-level labels $x=[x_{\text{sys}},x_{\text{user}}]$9, and corrective hint $\pi_\theta(y\mid x)$0. These objects then determine task filtering, attempt selection, step extraction, and the final hint-contextualized GRPO loss (Liu et al., 18 Jun 2026).

3. Optimization mechanisms

A recurring optimization pattern in HiFPO-like methods is explicit separation of feedback channels followed by dynamic recombination during policy improvement. HIPO makes this most explicit. For each prompt $\pi_\theta(y\mid x)$1, it samples a group of $\pi_\theta(y\mid x)$2 responses, obtains two judge-derived rewards $\pi_\theta(y\mid x)$3 and $\pi_\theta(y\mid x)$4, standardizes them into group-relative advantages $\pi_\theta(y\mid x)$5 and $\pi_\theta(y\mid x)$6, and forms a combined advantage

$\pi_\theta(y\mid x)$7

The policy update is PPO-style with clipping and KL regularization, while the dual update

$\pi_\theta(y\mid x)$8

increases system weight when compliance falls below threshold and decreases it when the constraint is satisfied. In practice, an EMA of system reward is used to smooth the update (Chen et al., 17 Mar 2026).

BrHPO also uses a shared scalar to coordinate levels, but in a different manner. The same reachability measure $\pi_\theta(y\mid x)$9 appears in the high-level policy objective as $r_{\text{user}}(x,y)\in[0,1]$0 and in the low-level reward as

$r_{\text{user}}(x,y)\in[0,1]$1

Both levels are optimized with SAC-style actor-critic updates, separate critics, and separate replay buffers, while the coupling is only through the shared feedback term. Reachability is estimated online from two scalar low-level rewards per subtask, with complexity $r_{\text{user}}(x,y)\in[0,1]$2 per subtask (Luo et al., 2024).

MENTOR combines reward learning, constrained optimization, and exploration-exploitation decoupling. Human feedback is pairwise preference comparison over $r_{\text{user}}(x,y)\in[0,1]$3 tuples, trained with a modified Bradley–Terry objective. The high-level policy then maximizes a human-feedback reward with entropy augmentation and DDC penalty, while $r_{\text{user}}(x,y)\in[0,1]$4 is updated by gradient descent on the relaxed dual objective. At the low level, MENTOR trains a base policy $r_{\text{user}}(x,y)\in[0,1]$5 on subgoal achievement and a separate exploration policy $r_{\text{user}}(x,y)\in[0,1]$6 on subgoal achievement plus RND reward. HER is applied in low-level training, and $r_{\text{user}}(x,y)\in[0,1]$7 is increased or decreased based on measured low-level success thresholds 0.6 and 0.3 (Zhou et al., 2024).

MobileForge preserves GRPO’s clipped ratio and KL penalty but changes the substrate of optimization. HiFPO first runs multi-attempt rollouts, aggregates prior hints into $r_{\text{user}}(x,y)\in[0,1]$8, then filters tasks by empirical success rate

$r_{\text{user}}(x,y)\in[0,1]$9

Among non-mastered tasks, it selects one informative attempt using the reasonable-step fraction

$r_{\text{sys}}(x,y)\in[0,1]$0

and constructs the training set from only reasonable steps. For each selected step, GRPO samples $r_{\text{sys}}(x,y)\in[0,1]$1 candidate actions, scores them with a rule-based GUI reward

$r_{\text{sys}}(x,y)\in[0,1]$2

normalizes rewards within the group, and optimizes

$r_{\text{sys}}(x,y)\in[0,1]$3

Hints are not additional reward terms; they modify the conditioning state itself (Liu et al., 18 Jun 2026).

4. Major instantiations

HIPO is the most direct formulation of HiFPO for LLM alignment. It targets Hierarchical Instruction Following, especially the standard two-level stack $r_{\text{sys}}(x,y)\in[0,1]$4, though the formalism naturally extends to deeper hierarchies such as $r_{\text{sys}}(x,y)\in[0,1]$5. It uses two LLM-judge prompts to compute system compliance and user utility, implements group-relative PPO-style updates, and enforces a target compliance threshold $r_{\text{sys}}(x,y)\in[0,1]$6. The method is implemented with TRL, PyTorch, and FlashAttention-2; uses full-parameter fine-tuning in bf16; keeps the reference model in 4-bit NF4; and reports experiments on Qwen3-1.7B/4B/8B, Phi-3-3.8B, and Llama3.2-3B over the SystemCheck dataset of 2,000 hierarchical prompt pairs with a 1:1 aligned/conflicting split and an 1,800/200 train/test split (Chen et al., 17 Mar 2026).

MobileForge introduces HiFPO as the policy-learning component of an annotation-free adaptation system for mobile GUI agents. MobileGym supplies real-app exploration, curriculum generation, and evaluator feedback, while HiFPO turns trajectory outcomes, step-level process feedback, and corrective hints into hint-contextualized step-level GRPO updates. The decision state includes screenshot, interaction history, task instruction, and accumulated hints; the policy outputs structured GUI actions $r_{\text{sys}}(x,y)\in[0,1]$7; and only automatically generated adaptation data are used. The reported main 900-task runs use 8 $r_{\text{sys}}(x,y)\in[0,1]$8 80GB GPUs for roughly 80 hours, max prompt and response length 2048, group size $r_{\text{sys}}(x,y)\in[0,1]$9, 4 epochs, global batch size 128, rollout batch size 512, AdamW with learning rate $$\max_\theta \; J_{\text{user}}(\theta)=\mathbb{E}[r_{\text{user}}(x,y)]-\beta\,\mathbb{D}_{KL}(\pi_\theta\parallel \pi_{\text{ref}})$$0, KL coefficient $$\max_\theta \; J_{\text{user}}(\theta)=\mathbb{E}[r_{\text{user}}(x,y)]-\beta\,\mathbb{D}_{KL}(\pi_\theta\parallel \pi_{\text{ref}})$$1, and reward weights $$\max_\theta \; J_{\text{user}}(\theta)=\mathbb{E}[r_{\text{user}}(x,y)]-\beta\,\mathbb{D}_{KL}(\pi_\theta\parallel \pi_{\text{ref}})$$2, $$\max_\theta \; J_{\text{user}}(\theta)=\mathbb{E}[r_{\text{user}}(x,y)]-\beta\,\mathbb{D}_{KL}(\pi_\theta\parallel \pi_{\text{ref}})$$3 (Liu et al., 18 Jun 2026).

BrHPO is not presented under the HiFPO name, but the paper explicitly abstracts it into a more general Hierarchical Feedback-Guided Policy Optimization framework. Its core innovation is mutual response: the same reachability statistic regularizes the high-level manager and shapes the low-level worker’s intrinsic reward. The method is implemented on top of SAC for both hierarchies, uses two replay buffers plus a temporary per-subtask buffer, and avoids adjacency matrices, large graphs, and expensive candidate-subgoal searches. It is evaluated on long-horizon MuJoCo tasks including AntMaze, AntBigMaze, AntPush, AntFall, Reacher3D, Pusher, and HumanoidMaze (Luo et al., 2024).

MENTOR is likewise characterized in the provided material as, in essence, a HiFPO method. Its distinctive feature is the placement of human feedback at the high level only: pairwise preference comparisons guide subgoal selection, while DDC and the learned distance model keep those subgoals aligned with low-level competence. The low level is split into exploitation and exploration policies, the latter using RND to expand coverage without destabilizing subgoal achievement learning. Benchmarks include FetchPush, FetchPickAndPlace, FetchDraw, FetchObsPush, Pusher, and Four Rooms (Zhou et al., 2024).

5. Empirical behavior and mechanistic findings

The empirical record associated with HiFPO-like methods is heterogeneous because the domains differ, but several recurrent effects are reported. In HIPO, on conflicting prompts, system compliance rises to approximately $$\max_\theta \; J_{\text{user}}(\theta)=\mathbb{E}[r_{\text{user}}(x,y)]-\beta\,\mathbb{D}_{KL}(\pi_\theta\parallel \pi_{\text{ref}})$$4, matching the chosen threshold $$\max_\theta \; J_{\text{user}}(\theta)=\mathbb{E}[r_{\text{user}}(x,y)]-\beta\,\mathbb{D}_{KL}(\pi_\theta\parallel \pi_{\text{ref}})$$5, while user utility remains higher than under SFT, DPO, or attention-based methods at that compliance level. On aligned prompts, HIPO improves both system and user scores; one cited example is Qwen3-4B, where the score pair is $$\max_\theta \; J_{\text{user}}(\theta)=\mathbb{E}[r_{\text{user}}(x,y)]-\beta\,\mathbb{D}_{KL}(\pi_\theta\parallel \pi_{\text{ref}})$$6 versus Base $$\max_\theta \; J_{\text{user}}(\theta)=\mathbb{E}[r_{\text{user}}(x,y)]-\beta\,\mathbb{D}_{KL}(\pi_\theta\parallel \pi_{\text{ref}})$$7. On Qwen3-1.7B, MMLU-Redux remains almost unchanged at $$\max_\theta \; J_{\text{user}}(\theta)=\mathbb{E}[r_{\text{user}}(x,y)]-\beta\,\mathbb{D}_{KL}(\pi_\theta\parallel \pi_{\text{ref}})$$8 versus $$\max_\theta \; J_{\text{user}}(\theta)=\mathbb{E}[r_{\text{user}}(x,y)]-\beta\,\mathbb{D}_{KL}(\pi_\theta\parallel \pi_{\text{ref}})$$9, while Attack Success Rate on WildJailbreak and HarmBench decreases when a safety system prompt is present, with modest over-refusal compared to SFT. HIPO’s mechanistic analysis further reports FarMass $$J_{\text{sys}}(\theta)=\mathbb{E}[r_{\text{sys}}(x,y)]\ge \tau.$$0, NearMass $$J_{\text{sys}}(\theta)=\mathbb{E}[r_{\text{sys}}(x,y)]\ge \tau.$$1, SysMass $$J_{\text{sys}}(\theta)=\mathbb{E}[r_{\text{sys}}(x,y)]\ge \tau.$$2, UserMass $$J_{\text{sys}}(\theta)=\mathbb{E}[r_{\text{sys}}(x,y)]\ge \tau.$$3, and SysUserRatio $$J_{\text{sys}}(\theta)=\mathbb{E}[r_{\text{sys}}(x,y)]\ge \tau.$$4, consistent with increased long-range attention to system tokens at response onset (Chen et al., 17 Mar 2026).

MobileForge reports that annotation-free adaptation with HiFPO is competitive with, and in some cases exceeds, strong GUI baselines. Using only automatically generated annotation-free adaptation data, MobileForge adapts Qwen3-VL-8B to $$J_{\text{sys}}(\theta)=\mathbb{E}[r_{\text{sys}}(x,y)]\ge \tau.$$5 Pass@3 on AndroidWorld, close to the closed-data GUI-specialized GUI-Owl-1.5-8B base model at $$J_{\text{sys}}(\theta)=\mathbb{E}[r_{\text{sys}}(x,y)]\ge \tau.$$6. The MobileForge-adapted ForgeOwl-8B reaches $$J_{\text{sys}}(\theta)=\mathbb{E}[r_{\text{sys}}(x,y)]\ge \tau.$$7 Pass@3 on AndroidWorld and $$J_{\text{sys}}(\theta)=\mathbb{E}[r_{\text{sys}}(x,y)]\ge \tau.$$8 success on the out-of-domain MobileWorld GUI-only split. On a 200-task Qwen3 rollout ablation, corrective hints improve overall success from $$J_{\text{sys}}(\theta)=\mathbb{E}[r_{\text{sys}}(x,y)]\ge \tau.$$9 to $\lambda\ge 0$0, Pass@3 from $\lambda\ge 0$1 to $\lambda\ge 0$2, and reduce average steps per attempt from 18.4 to 17.2. On AndroidWorld Pass@1, hint-contextualized GRPO outperforms SFT on the same data: for 900 tasks, Base is $\lambda\ge 0$3, Hint SFT is $\lambda\ge 0$4, and Hint-contextualized GRPO reaches $\lambda\ge 0$5 (Liu et al., 18 Jun 2026).

BrHPO reports higher success rates, better sample efficiency, and more stable training than HIRO, HIGL, RIS, CHER, and flat SAC across navigation, manipulation, and HumanoidMaze tasks. The ablations are particularly important for HiFPO interpretation: Full BrHPO outperforms Vanilla, NoReg, and NoBonus, indicating that both directions of feedback matter, and NoBonus exceeds NoReg, indicating that high-level awareness of reachability is more critical than low-level bonus alone. The method is also reported to remain effective under different distance metrics $\lambda\ge 0$6, different subtask lengths $\lambda\ge 0$7, and Gaussian observation noise (Luo et al., 2024).

MENTOR reports significantly higher success rates and faster learning than HER, HhP, PEBBLE, and DDL across sparse-reward robotic and navigation tasks. The DDC mechanism produces a curriculum in which subgoal distances start small and grow as low-level success increases, and automatic dual-variable tuning of $\lambda\ge 0$8 outperforms fixed $\lambda\ge 0$9. The feedback-efficiency result is also notable: even 10 labels per 25–100 episodes dramatically reduce the number of episodes needed to reach 100% success compared to no feedback. Human-collected labels are reported to yield performance similar to scripted labels, with minor variability due to noise (Zhou et al., 2024).

6. Limitations, misconceptions, and extensions

A common misconception is that HiFPO denotes one canonical optimizer. The available literature does not support that interpretation. HIPO is described as the special case of HiFPO with one constraint level and one primary objective; BrHPO is presented as a concrete instantiation in which the feedback is subgoal reachability; MENTOR is described as already being, in essence, a HiFPO method; and MobileForge uses HiFPO as an explicit GRPO-based adaptation procedure. This suggests that the term functions more as a structural template than as a single architecture or loss function (Chen et al., 17 Mar 2026, Luo et al., 2024, Zhou et al., 2024, Liu et al., 18 Jun 2026).

Another misconception is that HiFPO necessarily depends on human feedback. MENTOR does use human pairwise preferences, but HIPO relies on two LLM judges, BrHPO uses an online reachability statistic derived from intrinsic rewards, and MobileForge uses MobileGym-Critic to produce outcome labels, process labels, and hints without human-authored rewards or demonstrations. The broader pattern is hierarchical feedback, not exclusively human supervision (Zhou et al., 2024, Chen et al., 17 Mar 2026, Luo et al., 2024, Liu et al., 18 Jun 2026).

The limitations are domain-specific but structurally related. HIPO enforces compliance only in expectation and provides no per-query worst-case guarantee; it is also computationally heavy because of LLM-as-a-judge and suggests distilling learned rewards into smaller models. BrHPO assumes a fixed two-level hierarchy, fixed subtask length $$\mathcal{L}(\theta,\lambda)=J_{\text{user}}(\theta)+\lambda(J_{\text{sys}}(\theta)-\tau),$$0, a meaningful mapping $$\mathcal{L}(\theta,\lambda)=J_{\text{user}}(\theta)+\lambda(J_{\text{sys}}(\theta)-\tau),$$1, and a dense low-level distance-based intrinsic reward, and the paper notes difficulty in purely sparse low-level reward scenarios. MENTOR assumes $$\mathcal{L}(\theta,\lambda)=J_{\text{user}}(\theta)+\lambda(J_{\text{sys}}(\theta)-\tau),$$2, requires a learnable distance model, and provides no formal convergence guarantees. MobileForge depends on evaluator quality, explored app coverage, and substantial compute for multi-attempt rollouts and VLM-based evaluation (Chen et al., 17 Mar 2026, Luo et al., 2024, Zhou et al., 2024, Liu et al., 18 Jun 2026).

The main extensions proposed across the papers are mutually compatible. HIPO explicitly proposes multiple constraint layers, multiple feedback modalities, and off-policy or inference-time HiFPO, including constraint-aware decoding or scoring such as SITAlign-style satisficing. BrHPO suggests learned reachability models, multi-level HiFPO with adjacent-pair feedback metrics, integration with model-based RL, and variable-horizon subtasks. MENTOR suggests a general template in which human feedback shapes high-level reward, feasibility models define dynamic constraints, dual variables balance reward against feasibility, and low-level exploration is decoupled from exploitation. MobileForge suggests application beyond mobile GUIs to desktop agents, web agents, and broader computer-use settings, as well as richer step-level credit assignment. Taken together, these proposals indicate that HiFPO is evolving toward a general policy-optimization paradigm in which hierarchical structure is imposed directly on both feedback representation and optimization dynamics (Chen et al., 17 Mar 2026, Luo et al., 2024, Zhou et al., 2024, Liu et al., 18 Jun 2026).