Heterogeneous Adaptive Policy Optimization (HAPO)

• HAPO is a token-aware reinforcement learning framework that uses token entropy to dynamically balance exploration and exploitation in large language models.
• It integrates adaptive temperature sampling, token-level normalization, advantage redistribution, and asymmetric adaptive clipping to tailor policy updates at the token level.
• Empirical results show HAPO outperforms uniform PPO-style methods on reasoning-intensive RLHF tasks, achieving measurable improvements in accuracy across model scales.

Searching arXiv for papers on HAPO and closely related terminology to ground the article. arxiv_search(query="Heterogeneous Adaptive Policy Optimization HAPO OR HAPPO OR HARPO", max_results=10, sort_by="relevance") Heterogeneous Adaptive Policy Optimization (HAPO) is a token-aware reinforcement learning framework for LLMs that embeds entropy-conditioned decisions into every stage of training—sampling, advantage normalization, advantage redistribution, and clipping—and uses token-level entropy as a continuous signal to modulate exploration versus exploitation (Liu et al., 20 Sep 2025). In its exact named form, HAPO addresses a central limitation of PPO-style methods such as PPO, GRPO, and DAPO: the uniform treatment of all tokens regardless of their role during reasoning. The framework is designed for reasoning-intensive RLHF and RLVR settings in which long sequences, branching decisions, and heterogeneous token roles make uniform optimization suboptimal (Liu et al., 20 Sep 2025).

1. Terminology, scope, and acronym ambiguity

The acronym HAPO is not stable across recent arXiv literature. In the exact sense of the paper titled "From Uniform to Heterogeneous: Tailoring Policy Optimization to Every Token's Nature," HAPO denotes Heterogeneous Adaptive Policy Optimization, a token-aware RL algorithm for LLM post-training (Liu et al., 20 Sep 2025). That paper is the primary source for the exact title phrase.

Closely related literatures use adjacent or conflicting names. In cooperative MARL, "Heterogeneous-Agent Reinforcement Learning" introduces HAPPO, not HAPO, and explicitly notes that “Heterogeneous Adaptive Policy Optimization” is a misnomer for HAPPO; the official acronym there is Heterogeneous-Agent Proximal Policy Optimisation (Zhong et al., 2023). In another MARL paper, "Multi-Agent Trust Region Policy Optimisation: A Joint Constraint Approach," the phrase HAPO in spirit is used for adaptive allocation of a global KL budget across heterogeneous agents, instantiated by HATRPO-W and HATRPO-G rather than by an algorithm literally named HAPO (Shek et al., 14 Aug 2025). "OmniSapiens" similarly states that HARPO is the paper’s terminology and that HAPO can be read only as an informal synonym in that context (Ong et al., 11 Feb 2026).

The same acronym is also used for unrelated methods. "Hindsight-Anchored Policy Optimization" defines HAPO as a sparse-reward RLVR method with Synthetic Success Injection and a Thompson sampling-inspired gate (Wu et al., 11 Mar 2026). "Robotic Policy Learning via Human-assisted Action Preference Optimization" defines HAPO as a human-in-the-loop preference optimization method for Vision-Language-Action models (Xia et al., 8 Jun 2025). By contrast, "HAEPO" explicitly states that it is not HAPO and expands to History-Aggregated Exploratory Policy Optimization (Trivedi et al., 26 Aug 2025).

A practical implication is that the exact referent of “HAPO” depends on subfield. In LLM RL, the exact-titled HAPO of (Liu et al., 20 Sep 2025) denotes token-entropy-conditioned policy optimization; in heterogeneous MARL, the nearest exact methods are usually HAPPO, HATRPO, or HARPO rather than a canonical algorithm named HAPO (Zhong et al., 2023).

2. Design principle: from uniform token treatment to heterogeneous token treatment

The central motivation of HAPO is the claim that standard PPO/GRPO/DAPO mechanisms apply fixed sampling temperatures, sequence-level or uniform advantages, and fixed clipping bounds to all tokens, even though tokens play different roles in reasoning (Liu et al., 20 Sep 2025). The paper identifies several consequences. Rare, high-entropy “critical” tokens often correspond to branching decisions in reasoning and are under-sampled during standard rollout because entropy is inversely related to sampling probability. Low-entropy tokens often encode routine computations, formatting, or syntactic glue, and these require stability or coherence rather than aggressive exploration. Sequence-level advantage attribution combined with DAPO’s token-mean loss lets longer negative samples dominate gradients, and uniform clipping can invert the desired priorities for exploration and noise suppression (Liu et al., 20 Sep 2025).

HAPO uses token entropy as its organizing signal. For token position $t$, entropy is defined as

$$H_t = -\sum_{v \in \mathcal{V}} \pi_\theta(v \mid q, o_{<t}) \log \pi_\theta(v \mid q, o_{<t}).$$

High entropy is treated as a proxy for uncertainty, where exploration is valuable; low entropy is treated as a confident regime, where exploitation and coherence are preferred (Liu et al., 20 Sep 2025). The paper’s “Dual-Entropy Phenomenon” further argues that tokens with similar stems can occur at very different entropy levels across contexts, which motivates continuous rather than binary treatment (Liu et al., 20 Sep 2025).

This design places HAPO in contrast with several nearby baselines. Relative to PPO, HAPO retains the clipped-surrogate structure but makes clip bounds entropy-adaptive. Relative to GRPO, it remains critic-free but normalizes advantages at token level rather than by sequence-level group normalization. Relative to DAPO, it preserves token-mean normalization while augmenting it with entropy-aware sampling, redistribution, and clipping (Liu et al., 20 Sep 2025).

3. Core algorithmic components

HAPO is organized around four named components, each attached to a different stage of the RL loop (Liu et al., 20 Sep 2025).

Component	Stage	Function
Adaptive Temperature Sampling (ATS)	Rollout sampling	Raises temperature at high-entropy positions and lowers it at low-entropy positions
Token-Level Group Average (TLGA)	Advantage normalization	Normalizes advantages across tokens in the sampled group
Differential Advantage Redistribution (DAR)	Advantage modulation	Uses entropy and importance ratios to amplify or temper token updates
Asymmetric Adaptive Clipping (AAC)	Clipped loss	Uses entropy-conditioned left and right clipping bounds

Adaptive Temperature Sampling

ATS updates temperature in real time during decoding according to the token’s current entropy. In the continuous formulation,

$$T_{i,t} = T_{\text{base}} \cdot \left(1 + \frac{\log(H_{i,t}) - \rho_{\log(H)}^{\text{init}}}{\sigma_{\log(H)}^{\text{init}}} \cdot \tau \right).$$

In the experiments, $T_{\text{base}} = 1.0$ and $\tau = 0.05$ (Liu et al., 20 Sep 2025). The paper also evaluates a binary ATS baseline with threshold $\theta = 0.5$, $T_{\text{high}} = 1.1$, and $T_{\text{low}} = 0.8$ (Liu et al., 20 Sep 2025). The intended effect is explicit balancing of exploration and coherence. The reported ablation attributes a sizeable gain to ATS alone, with Avg accuracy +1.88 points on Qwen2.5-Math-7B (Liu et al., 20 Sep 2025).

Token-Level Group Average

TLGA replaces sequence-level normalization with token-level normalization over the entire sampled group. If the per-token reward is inherited from the sequence reward via $a_{i,t} = r_i \in \{0,1\}$, then the group statistics are

$$\mu_{\text{tok}} = \frac{1}{|\mathcal{T}|} \sum_{(i,t) \in \mathcal{T}} a_{i,t}, \qquad \sigma_{\text{tok}} = \sqrt{\frac{1}{|\mathcal{T}|} \sum_{(i,t) \in \mathcal{T}} (a_{i,t} - \mu_{\text{tok}})^2},$$

and the normalized advantage is

$$H_t = -\sum_{v \in \mathcal{V}} \pi_\theta(v \mid q, o_{<t}) \log \pi_\theta(v \mid q, o_{<t}).$$0

The paper states that TLGA enforces $$H_t = -\sum_{v \in \mathcal{V}} \pi_\theta(v \mid q, o_{<t}) \log \pi_\theta(v \mid q, o_{<t}).$$1, removing aggregate bias while preserving DAPO’s length-aware property within reward classes (Liu et al., 20 Sep 2025). In the component study, TLGA contributes +1.59 Avg points relative to the baseline (Liu et al., 20 Sep 2025).

Differential Advantage Redistribution

DAR modulates token advantages using both entropy and the importance ratio

$$H_t = -\sum_{v \in \mathcal{V}} \pi_\theta(v \mid q, o_{<t}) \log \pi_\theta(v \mid q, o_{<t}).$$2

Entropy is batch-normalized through $$H_t = -\sum_{v \in \mathcal{V}} \pi_\theta(v \mid q, o_{<t}) \log \pi_\theta(v \mid q, o_{<t}).$$3 statistics and then asymmetrically rescaled to $$H_t = -\sum_{v \in \mathcal{V}} \pi_\theta(v \mid q, o_{<t}) \log \pi_\theta(v \mid q, o_{<t}).$$4 (Liu et al., 20 Sep 2025). The redistributed advantage is

$$H_t = -\sum_{v \in \mathcal{V}} \pi_\theta(v \mid q, o_{<t}) \log \pi_\theta(v \mid q, o_{<t}).$$5

with

$$H_t = -\sum_{v \in \mathcal{V}} \pi_\theta(v \mid q, o_{<t}) \log \pi_\theta(v \mid q, o_{<t}).$$6

The condition $$H_t = -\sum_{v \in \mathcal{V}} \pi_\theta(v \mid q, o_{<t}) \log \pi_\theta(v \mid q, o_{<t}).$$7 depends on whether the token is high-entropy or low-entropy and whether the ratio lies inside or outside a neutral zone tied to the clipping bounds (Liu et al., 20 Sep 2025). High-entropy tokens are amplified only when the model already shows a clear update preference; low-entropy tokens are suppressed when their ratios indicate neutrality. DAR alone yields +1.31 Avg points, and DAR jointly with AAC yields +1.77 points (Liu et al., 20 Sep 2025).

Asymmetric Adaptive Clipping

AAC generalizes asymmetric clipping from DAPO to token-specific, entropy-conditioned bounds. The clipped objective is

$$H_t = -\sum_{v \in \mathcal{V}} \pi_\theta(v \mid q, o_{<t}) \log \pi_\theta(v \mid q, o_{<t}).$$8

The bounds are

$$H_t = -\sum_{v \in \mathcal{V}} \pi_\theta(v \mid q, o_{<t}) \log \pi_\theta(v \mid q, o_{<t}).$$9

and

$$T_{i,t} = T_{\text{base}} \cdot \left(1 + \frac{\log(H_{i,t}) - \rho_{\log(H)}^{\text{init}}}{\sigma_{\log(H)}^{\text{init}}} \cdot \tau \right).$$0

In the main experiments, $$T_{i,t} = T_{\text{base}} \cdot \left(1 + \frac{\log(H_{i,t}) - \rho_{\log(H)}^{\text{init}}}{\sigma_{\log(H)}^{\text{init}}} \cdot \tau \right).$$1 and $$T_{i,t} = T_{\text{base}} \cdot \left(1 + \frac{\log(H_{i,t}) - \rho_{\log(H)}^{\text{init}}}{\sigma_{\log(H)}^{\text{init}}} \cdot \tau \right).$$2 (Liu et al., 20 Sep 2025). AAC alone yields +1.05 Avg points and, according to the paper, is especially important for allowing aggressive reduction for noisy low-entropy tokens while preserving exploration room for high-entropy ones (Liu et al., 20 Sep 2025).

4. Objective, training loop, and implementation

HAPO keeps DAPO’s token-mean normalization and replaces its surrogate with AAC while using TLGA and DAR to produce the final advantages. The full objective is

$$T_{i,t} = T_{\text{base}} \cdot \left(1 + \frac{\log(H_{i,t}) - \rho_{\log(H)}^{\text{init}}}{\sigma_{\log(H)}^{\text{init}}} \cdot \tau \right).$$3

where

$$T_{i,t} = T_{\text{base}} \cdot \left(1 + \frac{\log(H_{i,t}) - \rho_{\log(H)}^{\text{init}}}{\sigma_{\log(H)}^{\text{init}}} \cdot \tau \right).$$4

Consistent with the DAPO settings used in the paper’s experiments, no KL penalty or entropy bonus is used (Liu et al., 20 Sep 2025).

The training pipeline is explicitly staged. Rollouts are first generated with continuous ATS; sequence rewards are then broadcast to tokens; TLGA computes $$T_{i,t} = T_{\text{base}} \cdot \left(1 + \frac{\log(H_{i,t}) - \rho_{\log(H)}^{\text{init}}}{\sigma_{\log(H)}^{\text{init}}} \cdot \tau \right).$$5; batch entropy statistics define $$T_{i,t} = T_{\text{base}} \cdot \left(1 + \frac{\log(H_{i,t}) - \rho_{\log(H)}^{\text{init}}}{\sigma_{\log(H)}^{\text{init}}} \cdot \tau \right).$$6; AAC sets token-specific clipping bounds; DAR produces $$T_{i,t} = T_{\text{base}} \cdot \left(1 + \frac{\log(H_{i,t}) - \rho_{\log(H)}^{\text{init}}}{\sigma_{\log(H)}^{\text{init}}} \cdot \tau \right).$$7; and the HAPO loss is optimized by gradient ascent (Liu et al., 20 Sep 2025). The paper emphasizes that TLGA should be applied before DAR and that the neutral zone for DAR should be tied to the current clipping bounds $$T_{i,t} = T_{\text{base}} \cdot \left(1 + \frac{\log(H_{i,t}) - \rho_{\log(H)}^{\text{init}}}{\sigma_{\log(H)}^{\text{init}}} \cdot \tau \right).$$8 (Liu et al., 20 Sep 2025).

The reported RLHF setup uses Qwen2.5-Math-1.5B, Qwen2.5-Math-7B, and Qwen3-8B on DAPO-Math-17K (RLVR), with 16 responses per prompt, maximum generation length 10240 tokens, 4096-token cache, batch size 512, mini-batch size 32, learning rate $$T_{i,t} = T_{\text{base}} \cdot \left(1 + \frac{\log(H_{i,t}) - \rho_{\log(H)}^{\text{init}}}{\sigma_{\log(H)}^{\text{init}}} \cdot \tau \right).$$9, 10-step warmup, and no KL divergence or entropy regularization losses (Liu et al., 20 Sep 2025). The entropy quantile is $T_{\text{base}} = 1.0$0, ATS uses $T_{\text{base}} = 1.0$1 and $T_{\text{base}} = 1.0$2, and the experiments were run on 4 nodes, 32 A100 GPUs; analysis runs were also reported with 8 A100 GPUs (Liu et al., 20 Sep 2025). The paper states that entropy computation reuses quantities already available in the verl/DAPO pipeline and that the computational overhead is negligible relative to DAPO (Liu et al., 20 Sep 2025).

5. Empirical results and ablation evidence

The main evaluation reports zero-shot performance at $T_{\text{base}} = 1.0$3, with pass@8 responses per question, on AIME24, AIME25, AMC, MATH500, OlympiadBench, and Minerva (Liu et al., 20 Sep 2025). Across all three model scales, HAPO outperforms vanilla DAPO in average accuracy.

Model	Vanilla DAPO Avg	HAPO Avg
Qwen2.5-Math-1.5B	38.34	40.62
Qwen2.5-Math-7B	46.97	50.04
Qwen3-8B	50.00	51.97

At Qwen2.5-Math-1.5B scale, HAPO improves over vanilla DAPO on AIME24 (25.33 vs 21.73), AIME25 (20.12 vs 17.11), AMC (75.69 vs 72.85), OlympiadBench (33.86 vs 31.47), and Minerva (21.91 vs 19.52) (Liu et al., 20 Sep 2025). At Qwen2.5-Math-7B, the paper highlights AIME24 41.31 vs 37.24, AIME25 24.34 vs 20.24, AMC 85.47 vs 81.99, and MATH500 78.45 vs 76.72 (Liu et al., 20 Sep 2025). At Qwen3-8B, highlighted gains include AIME24 39.01 vs 35.84, AIME25 26.83 vs 25.24, AMC 81.77 vs 78.37, and Minerva 34.74 vs 32.13 (Liu et al., 20 Sep 2025).

The paper also compares HAPO against DAPO w/ Forking Tokens, Archer, and EDGE-GRPO, and reports that HAPO consistently outperforms DAPO across multiple model scales (Liu et al., 20 Sep 2025). This empirical pattern is reinforced by a component ablation on Qwen2.5-Math-7B over 1560 eval items. The reported averages are: Baseline DAPO 46.97, ATS only 48.85, TLGA only 48.56, DAR only 48.28, AAC only 48.02, DAR + AAC 48.74, TLGA + DAR + AAC 49.42, and HAPO 50.04 (Liu et al., 20 Sep 2025). The ablation supports the paper’s claim that all four components contribute and that they work synergistically.

The training-dynamics analysis reports that HAPO maintains longer response lengths and higher overall entropy than DAPO throughout training, which the paper interprets as better exploration while improving correctness (Liu et al., 20 Sep 2025). It also reports that relaxing left clipping without TLGA can cause entropy collapse, which is presented as evidence that clipping asymmetry and advantage normalization are tightly coupled (Liu et al., 20 Sep 2025).

6. Broader context, misconceptions, and related formulations

A recurring misconception is to conflate HAPO with heterogeneous-agent PPO or TRPO in MARL. That conflation is historically understandable, because HAPPO and HATRPO are prominent heterogeneous-agent policy-optimization methods, and some later papers use “HAPO” only informally to describe adaptive heterogeneous allocation schemes. However, the exact HAPO of (Liu et al., 20 Sep 2025) is not a sequential CTDE MARL algorithm; it is a token-aware RL objective for LLM reasoning (Zhong et al., 2023).

The broader literature nonetheless exhibits a family resemblance around adaptive optimization under heterogeneity. In "Multi-Agent Trust Region Policy Optimisation: A Joint Constraint Approach," adaptive per-agent KL allocation under a global budget is explicitly described as HAPO in spirit, with HATRPO-W using a KKT-based allocation and HATRPO-G using greedy scheduling (Shek et al., 14 Aug 2025). In "Prescribe-then-Select," HAPO is used as a conceptual label for adaptive policy selection across heterogeneous covariate regions in contextual stochastic optimization, operationalized through cross-validated policy libraries and Optimal Policy Trees (Iglesias et al., 9 Sep 2025). "Trust the Batch, On- or Off-Policy" describes a batch-adaptive objective driven by normalized effective sample size and presents it as an instance of Heterogeneous Adaptive Policy Optimization in perspective, even though the exact term is not the paper’s formal method name (Fakoor et al., 12 May 2026).

This suggests that “HAPO” now denotes at least two layers of meaning. At the narrow level, it names the token-entropy-conditioned LLM RL algorithm of (Liu et al., 20 Sep 2025). At the broader level, it functions as a recurring design motif for adapting policy updates to structured heterogeneity—across tokens, agents, tasks, or behavior distributions—rather than enforcing a single uniform update rule (Shek et al., 14 Aug 2025). A plausible implication is that future usage will continue to require explicit disambiguation, especially in discussions that span LLM RL, heterogeneous MARL, RLVR, and constrained decision-making.

Within that broader landscape, the exact HAPO paper occupies a specific niche: it transforms entropy from a diagnostic statistic into a control variable that affects rollout sampling, advantage construction, and trust-region geometry at token resolution (Liu et al., 20 Sep 2025). That combination, rather than the acronym alone, is what distinguishes HAPO from the other neighboring methods that share similar names but optimize different objects under different assumptions.