Hop-Anchored Policy Optimization (HaPO)

• HaPO is a reinforcement learning algorithm that leverages gold entity sequences to perform structured hop-level credit assignment.
• It enhances policy optimization by decomposing sparse trajectory rewards into hop-aware advantages within a GRPO-style framework, boosting multi-hop search accuracy.
• The method integrates smooth asymmetric gating, fatal-aware masking, and observation-token masking to stabilize training and reduce reward variance.

Searching arXiv for the cited HaPO paper and closely related similarly named methods to ground the article in current preprints. Hop-Anchored Policy Optimization (HaPO) is the reinforcement-learning training algorithm introduced in SearchEyes for multimodal search agents that perform multi-hop reasoning over a simulated search world built from a typed knowledge graph (Jiao et al., 7 Jul 2026). It addresses the credit-assignment problem in long-horizon search by reusing hop-level structural metadata that SearchEyes retains from Perception-Knowledge Chains (PKC): each training instance carries a gold entity sequence, and each successful retrieval of a gold entity becomes a hop anchor for step-level credit assignment. Rather than training a separate process reward model, HaPO decomposes sparse trajectory-level supervision into hop-aware, group-relative advantages inside a GRPO-style optimization framework, while also applying smooth asymmetric gating, fatal-aware masking, and observation-token masking (Jiao et al., 7 Jul 2026). The term should be distinguished from similarly abbreviated methods such as Anchored Policy Optimization (APO) (Wang et al., 5 Feb 2026) and Hindsight-Anchored Policy Optimization (HAPO) (Wu et al., 11 Mar 2026), which address different RL settings.

1. Origin and problem setting

HaPO was proposed as part of SearchEyes, a system designed to train multimodal search agents for multi-hop reasoning in a simulated search world derived from a typed knowledge graph (Jiao et al., 7 Jul 2026). The underlying difficulty is structural: in multi-hop search, the agent must make many decisions across tool calls, observations, and reasoning turns, while standard RL commonly supplies only a single trajectory-level reward indicating whether the final answer is correct. As the number of hops grows, the probability of a fully correct chain drops sharply, making the reward extremely sparse and high-variance (Jiao et al., 7 Jul 2026).

SearchEyes frames this problem through PKC, which samples constrained multi-hop paths over the visual-knowledge intersection of Wikidata5M and retains the associated gold entity sequence

$\mathbf{e}^*=(v_1, v_2, \ldots, v_{K+1}).$

This sequence is preserved as metadata through data synthesis and supervised fine-tuning, and it specifies the entities that the agent is expected to visit across hops (Jiao et al., 7 Jul 2026). Because the simulated environment maps every document to exactly one entity via a Wikidata ID, SearchEyes can determine precisely which entity each retrieval step accessed. HaPO exploits this alignment to define hop anchors and to construct step-level RL signals without any separately trained process reward model (Jiao et al., 7 Jul 2026).

This design situates HaPO between two established but imperfect approaches. On one side are trajectory-level policy-gradient methods such as GRPO, DAPO, SAPO, GiGPO, and ARPO, which leave the reward sparse at the episode level. On the other side are process reward models, which provide step-level feedback but require additional models and annotations. HaPO uses the typed KG, the PKC entity chain, and the deterministic environment to transform final-answer supervision into structured hop-level credit assignment (Jiao et al., 7 Jul 2026).

2. Formal RL formulation and hop-anchored objective

In SearchEyes, the RL state at turn $t$ is

$$s_t = (I_0, q, z_1, a_1, o_1, \ldots, z_{t-1}, a_{t-1}, o_{t-1}),$$

where $I_0$ is the anchor image, $q$ is the question, $z_t$ is a natural-language thought, $a_t$ is a tool call or final answer, and $o_t$ is the environment observation (Jiao et al., 7 Jul 2026). The policy outputs $y_t=(z_t,a_t)$ autoregressively, and a trajectory is

$\tau = (z_1, a_1, o_1, \ldots, z_T, a_T).$

Reward remains binary at the episode level,

$t$0

indicating final-answer correctness, and only agent-generated tokens receive gradient; observation tokens are masked (Jiao et al., 7 Jul 2026).

HaPO is built on top of GRPO. For a given question $t$1, SearchEyes samples $t$2 trajectories from the current policy. The GRPO baseline uses a single trajectory-level advantage for all tokens in a trajectory:

$t$3

This is group-normalized reward within the $t$4 trajectories for a given question (Jiao et al., 7 Jul 2026).

HaPO replaces the uniform trajectory-level advantage with a token-level advantage that depends on which hop the trajectory has reached. For each gold entity $t$5, the earliest step at which trajectory $t$6 retrieves it is

$t$7

The corresponding hop anchor group is

$t$8

Within each group $t$9 with at least two members and non-zero outcome variance, HaPO defines the hop-level advantage

$$s_t = (I_0, q, z_1, a_1, o_1, \ldots, z_{t-1}, a_{t-1}, o_{t-1}),$$0

where $$s_t = (I_0, q, z_1, a_1, o_1, \ldots, z_{t-1}, a_{t-1}, o_{t-1}),$$1 and $$s_t = (I_0, q, z_1, a_1, o_1, \ldots, z_{t-1}, a_{t-1}, o_{t-1}),$$2 are the mean and standard deviation of the final outcomes over that hop group (Jiao et al., 7 Jul 2026).

This hop-level advantage is then assigned to all tokens after the anchor step in that trajectory. If multiple anchors apply, the latest anchor wins. If no anchor covers a token, the hop-level advantage defaults to the trajectory-level advantage. HaPO then interpolates the two:

$$s_t = (I_0, q, z_1, a_1, o_1, \ldots, z_{t-1}, a_{t-1}, o_{t-1}),$$3

with $$s_t = (I_0, q, z_1, a_1, o_1, \ldots, z_{t-1}, a_{t-1}, o_{t-1}),$$4; SearchEyes uses $$s_t = (I_0, q, z_1, a_1, o_1, \ldots, z_{t-1}, a_{t-1}, o_{t-1}),$$5, so hop signals dominate (Jiao et al., 7 Jul 2026).

The full HaPO loss is

$$s_t = (I_0, q, z_1, a_1, o_1, \ldots, z_{t-1}, a_{t-1}, o_{t-1}),$$6

where $$s_t = (I_0, q, z_1, a_1, o_1, \ldots, z_{t-1}, a_{t-1}, o_{t-1}),$$7 masks out non-agent tokens, $$s_t = (I_0, q, z_1, a_1, o_1, \ldots, z_{t-1}, a_{t-1}, o_{t-1}),$$8 is the per-token importance ratio, and $$s_t = (I_0, q, z_1, a_1, o_1, \ldots, z_{t-1}, a_{t-1}, o_{t-1}),$$9 is a smooth asymmetric gate (Jiao et al., 7 Jul 2026).

3. Hop anchors, dense credit assignment, and reward decomposition

The core of HaPO is the reinterpretation of sparse reward through semantic anchors. In PKC, each synthesized question is derived from a constrained path on the typed KG,

$I_0$0

subject to constraints such as P–K alternation, a disambiguating constraint increasing treewidth, domain diversity, and anti-shortcut filtering (Jiao et al., 7 Jul 2026). The gold entity sequence $I_0$1 is stored as persistent metadata and later reused during RL.

Each $I_0$2 acts as a semantic milestone. When a rollout retrieves $I_0$3, that event defines a hop anchor. Trajectories that share the same anchor are grouped and compared using final outcomes, and the resulting group-relative statistic is propagated from that point onward in the trajectory (Jiao et al., 7 Jul 2026). The reward itself remains binary per trajectory; there is no explicit dense shaped reward such as graph distance or a potential function. The densification arises entirely from where the advantage is computed: among the trajectories that reached entity $I_0$4, which ones eventually succeeded (Jiao et al., 7 Jul 2026).

This is important conceptually. HaPO does not claim that visiting a gold entity is itself a reward. Instead, it uses anchor-conditioned outcome normalization to shift part of the credit assignment burden from the final step back to earlier retrieval events. This makes the training more sample-efficient and more stable because group-normalized hop advantages reduce variance relative to a single episode-level score (Jiao et al., 7 Jul 2026).

A plausible implication is that HaPO is particularly well suited to environments in which semantic states are instrumented and can be unambiguously detected. The paper makes this explicit by relying on the deterministic mapping between documents and entity IDs in the simulated search world. Without that alignment, the hop groups $I_0$5 would not be directly observable, and the method’s central mechanism would be weakened (Jiao et al., 7 Jul 2026).

4. Optimization mechanics: gating, masking, and training procedure

HaPO differs from standard GRPO not only in its advantage construction but also in how it stabilizes token-level updates. Instead of GRPO’s hard clipping, SearchEyes uses a smooth asymmetric sigmoid gate:

$I_0$6

with $I_0$7, so negative-advantage updates are more strongly damped (Jiao et al., 7 Jul 2026). This replaces the hard clipping used in the GRPO baseline and yields a smooth, asymmetric update rule.

HaPO also applies fatal-aware masking and one-sided clamping. If a trajectory has $I_0$8 or more consecutive tool errors starting at step $I_0$9, all tokens from steps $q$0 are masked by setting their final advantage to zero. For valid prefix tokens before $q$1, HaPO applies

$q$2

so only positive signals are kept, avoiding punishment of actions that occurred before exogenous tool failures (Jiao et al., 7 Jul 2026). Observation tokens are always masked out, matching the masking convention used in SearchEyes supervised fine-tuning.

The two-stage training pipeline is as follows. Stage 1 is supervised fine-tuning on expert trajectories generated with a retrieval-boosted environment and observation denoising, using

$q$3

Each example carries the gold sequence $q$4 for later reuse. Stage 2 performs online rollouts in an unboosted deterministic environment and optimizes the policy with HaPO (Jiao et al., 7 Jul 2026).

The paper specifies the following implementation choices for HaPO in SearchEyes: $q$5, $q$6, $q$7, fatal threshold $q$8 consecutive tool errors, group size $q$9 rollouts per question, KL coefficient $z_t$0, RL learning rate $z_t$1, maximum trajectory length of 12 tool calls, and observation truncation to $z_t$2 characters (Jiao et al., 7 Jul 2026). RL is performed on top of SFT-initialized Qwen3.5-9B or 27B models.

5. Empirical performance and ablation findings

SearchEyes reports that HaPO substantially improves multi-hop multimodal search performance. On a four-benchmark subset consisting of SimpleVQA, VDR, MMSearch, and FVQA, standard GRPO attains an average of 57.8, whereas full HaPO attains 61.8, a gain of 4.0 points (Jiao et al., 7 Jul 2026). The paper attributes a substantial portion of SearchEyes’ overall strength to HaPO, while also noting that the full system combines PKC data synthesis, the simulated search world, and RL.

The HaPO ablation study isolates the effect of its individual components. Removing hop-anchored advantages reduces the average to 59.1. Removing smooth asymmetric gating yields 60.5. Removing fatal-aware masking yields 59.9. Removing one-sided clamping yields 60.7. Removing observation token masking yields 60.2. Setting $z_t$3, which corresponds to episode-only credit, yields 59.0; setting $z_t$4, which corresponds to hop-only credit, yields 61.1 (Jiao et al., 7 Jul 2026). The strongest single contribution is therefore the hop-anchored advantage itself, while the additional mechanisms contribute incremental gains on the order of roughly 2–3 points.

The following table summarizes the reported ablation averages.

Variant	Average
Standard GRPO	57.8
Full HaPO	61.8
w/o hop-anchored advantage	59.1
w/o smooth asymmetric gating	60.5
w/o fatal-aware masking	59.9
w/o one-sided clamping	60.7
w/o observation token masking	60.2
$z_t$5	59.0
$z_t$6	61.1

These results support two specific claims made in the paper. First, hop-level credit assignment contributes most of the gain over GRPO. Second, mixing hop-level and trajectory-level signals with $z_t$7 performs best, even though hop-only already substantially improves over the baseline (Jiao et al., 7 Jul 2026).

At the system level, SearchEyes-9B achieves an average of 59.3 over six benchmarks, compared with 44.7 for Qwen3.5-9B with a generic agentic workflow and 54.3 for OpenSearch-VL-8B. SearchEyes-27B achieves 68.1, compared with 61.9 for OpenSearch-VL-32B. SearchEyes-27B improves over the strongest open-source baseline by 6.2 points on average (Jiao et al., 7 Jul 2026). These gains are not attributable solely to HaPO, but the ablations show that replacing HaPO with plain GRPO on the same framework degrades performance by about four points on the four-benchmark subset (Jiao et al., 7 Jul 2026).

6. Relation to adjacent methods, misconceptions, and limitations

A recurrent source of confusion is nomenclature. HaPO in SearchEyes denotes Hop-Anchored Policy Optimization (Jiao et al., 7 Jul 2026). It is distinct from Anchored Policy Optimization (APO), which addresses Recursive Space Contraction in RLVR for LLM reasoning by defining a Safe Manifold and rectifying negative-advantage updates through a push–pull ratio inside PPO/GRPO (Wang et al., 5 Feb 2026). It is also distinct from Hindsight-Anchored Policy Optimization (HAPO), which addresses sparse reward in RLVR by injecting teacher demonstrations into low-confidence groups using Synthetic Success Injection and a Thompson sampling-inspired gate (Wu et al., 11 Mar 2026). These methods share an “anchored” vocabulary but operate in different domains and with different anchoring objects: support over reference tokens in APO, teacher demonstrations in HAPO, and gold entity hops in HaPO.

Relative to standard trajectory-level sparse-reward optimization, HaPO still uses binary final outcomes but conditions them on which gold anchors were reached. Relative to process reward models such as PRIME and Agent-RRM, HaPO does not train a separate reward model; instead, it derives step-level structure from environment–metadata alignment (Jiao et al., 7 Jul 2026). Relative to generic turn- or step-level credit-assignment methods such as GiGPO, ARPO, and AT²PO, HaPO’s defining feature is semantic grouping by shared gold entities rather than grouping by arbitrary turns or learned signals (Jiao et al., 7 Jul 2026).

The paper does not explicitly list limitations of HaPO itself, but it identifies several implicit constraints. Hop anchors rely on matching gold entities in retrieval results; if retrieval fails to surface these entities, hop-level signals become weaker. HaPO uses binary final outcomes, so partial success beyond anchor-conditioned normalization is not explicitly rewarded. The anchor sequence also assumes that visiting each gold entity is necessary and sufficient; alternative valid reasoning chains involving different entities or relations are not represented in the anchor structure (Jiao et al., 7 Jul 2026). This suggests that HaPO is most natural in domains where gold semantic trajectories are well specified and where the environment can precisely log whether those states were reached.

The paper also points toward extensions. A plausible implication is that HaPO-style mechanisms can be transferred to other multi-step domains if three ingredients are available: a structured environment with semantically meaningful anchors, gold anchor sequences for training tasks, and reliable instrumentation that records when anchors are visited. The SearchEyes paper explicitly suggests settings such as web or computer-use environments, database workflows, and other tool-based systems, and it notes possible extensions including more complex hop-level shaping, branched reasoning graphs, soft anchors, combinations with process reward models, and adaptive anchor weighting (Jiao et al., 7 Jul 2026).