---
title: History-Aware Policy Optimization (HAPO)
url: https://www.emergentmind.com/topics/history-aware-policy-optimization-hapo
type: topic
---

# History-Aware Policy Optimization (HAPO)

History-Aware Policy Optimization (HAPO) refers to a class of algorithms and frameworks in reinforcement learning (RL) and imitation learning that explicitly leverage historical data—either as trajectory-level statistics, problem-specific histories, or temporally aggregated representations—to shape policy updates, improve exploration, and induce efficient behavior. Unlike memoryless or strictly per-timestep policy optimization schemes, HAPO methods encode, compress, or reward the model’s use of history (e.g., observation-action sequences, answer length records, or moment-token memories) to address the credit assignment, exploration, and efficiency challenges that arise in long-horizon, partially-observable, or sample-inefficient regimes [2505.11225, 2508.18884, 2510.00695, 2512.01763].

## 1. Conceptual Foundations and Motivation

HAPO frameworks are motivated by several shortcomings of conventional policy optimization methods:

- **Sparse reward environments**: In long-horizon tasks, signal vanishes when updates are distributed per token or timestep instead of via the full trajectory. HAPO uses trajectory-level or aggregated statistics to address this issue [2508.18884].
- **Exploration under trajectory correlation**: Standard approaches (e.g., PPO, token-level RLHF) can collapse exploration, particularly for listwise structured outputs or RL environments where return is jointly determined by entire action sequences, not individual steps.
- **Task-specific efficiency goals**: In settings like LLM reasoning, where concise yet correct outputs are preferred (e.g., math problem-solving), history-based policies can reward incremental improvements over model’s own previously discovered solutions [2505.11225].
- **Partial observability and long-term dependency**: In robotic control, GUI navigation, or vision-language-action (VLA) models, optimal policies must condition on varying lengths of the recent context, which HAPO retrieves and compresses dynamically [2510.00695, 2512.01763].

Thus, HAPO provides a unified framework to integrate historical information into the optimization protocol—either as an optimization target (e.g., per-query best-so-far), as dynamic memory (moment tokens/anchors), or via exploration/exploitation tradeoffs aggregated over entire trajectories.

## 2. Methodological Templates and Key Algorithms

The term “History-Aware Policy Optimization” encompasses diverse implementations, characterized primarily by their mechanism of encoding and leveraging history. The following table summarizes representative approaches:

| Approach        | History Encoding     | Optimization Target                 |
|-----------------|---------------------|-------------------------------------|
| HAPO [2505.11225]    | Per-problem minimum solution length | Reward-shaping: reward for producing solutions shorter than the best prior correct |
| HAEPO [2508.18884]   | Trajectory log likelihood + returns | Weighting and exploration via Plackett–Luce softmax over entire trajectories |
| HAMLET [2510.00695]  | Learnable moment tokens + memory network | Action prediction conditioned on compressed cross-timestep features |
| HCPO [2512.01763]    | Variable-length truncated history + anchor tokens | Adaptive sampling, dual-branch compression, and alignment losses |

**HAPO (Language Model Compression):**
- Maintains a scalar “history state” $h_i$ for each training problem, which records the minimum length of any previous correct solution.
- Defines a combined reward: binary correctness plus a per-trace length reward that is positive if and only if the new solution is both correct and shorter than $h_i$.
- Updates $h_i$ after each batch, so the policy incrementally “beats its own best,” compressing reasoning over training [2505.11225].

**HAEPO (Exploratory RL and LLM Tuning):**
- For each batch, compresses every trajectory into its summed log-likelihood $L_k$.
- Computes a Plackett–Luce softmax over these, with per-trajectory weights $w_k$ encouraging preference for diverse, high-likelihood rollouts.
- Objective is a weighted return minus an entropy bonus (to prevent collapse) and a soft KL penalty to a frozen previous policy (for stability). Normalization of returns is batch-based (sum-normalization or z-score) [2508.18884].

**HAMLET (Vision-Language-Action Policies):**
- At each timestep, represents state history using moment tokens (compact learnable embeddings) extracted from the visual input via a frozen backbone.
- Aggregates these with a causal Transformer memory module, forming a history-augmented feature used for action selection.
- Initial moment tokens are pre-trained with a time-contrastive loss to ensure temporal discrimination [2510.00695].

**HCPO (GUI Agent Optimization):**
- Employs Dynamic Context Sampling (DCS), providing variable-length observation/action histories at each step, driven by a schedule that starts uniform and increasingly favors longer contexts.
- During policy updates, utilizes Anchor-guided History Compression (AHC): branches with full vs. action-only compressed histories are both optimized, and a KL alignment term ensures consistency and efficiency [2512.01763].

## 3. Loss Functions, Regularization, and Pseudocode

HAPO-style methods typically deviate from classical per-timestep or per-token RL objectives. Some notable design elements:

- **Plackett–Luce softmax (HAEPO):** Batch-level weights $w_k = \exp(L_k) / \sum_j \exp(L_j)$ distribute credit across entire trajectories.
- **Entropy regularization:** HAEPO adds $H(w) = -\sum_k w_k \log w_k$ with a tunable coefficient $\beta_{ent}$ to prevent all weight concentrating on a single history, sustaining exploration.
- **KL trust-region penalty:** Soft regularization to a reference policy ensures update stability, with coefficient $\lambda$ [2508.18884].
- **History state update (HAPO for LLMs):** For each problem, upon generating a correct solution (matches gold answer), update $h_i$ if the new trace is even shorter. Length reward is shaped by $f(\ell, h) = \cos(\min(\frac{\pi}{2} \frac{\ell}{h}, \pi))$ with careful clipping to avoid over-penalizing short but incorrect attempts [2505.11225].
- **Dual-branch and compressed-history alignment:** HCPO’s $\mathcal{L}_{\mathrm{HCPO}}$ combines GRPO-style policy-losses on both full-history and compressed-history branches plus a KL alignment objective [2512.01763].

Pseudocode provided in each source paper (see [2508.18884] for HAEPO, [2505.11225] for HAPO, [2512.01763] for HCPO) demonstrates the centrality of batch-level history statistics and joint optimization over trajectory-level history as opposed to standard on-policy or off-policy gradient methods.

## 4. Empirical Evaluation and Applications

HAPO frameworks have been empirically tested across a range of domains, each exploiting different facets of the history-aware paradigm:

- **Language model reasoning compression:** HAPO achieves 33–59% reduction in reasoning chain length with only 2–5% accuracy degradation relative to base LLMs on math problems spanning GSM8K, MATH500, and AIME2024 [2505.11225].
- **RL benchmarks:** In multi-armed bandit settings, HAEPO matches or exceeds PPO and DPO in regret minimization, sustaining higher entropy and lower variance on large $K$; on 1D random walks, it attains near-optimal success rates faster than PPO and DPO while exhibiting reduced variance [2508.18884].
- **LLM reward modeling:** On TL;DR summarization (LLaMA, Qwen), HAEPO’s listwise, trajectory-level reward yields the best human preference scores and fastest training throughput compared to DPO and GRPO [2508.18884].
- **Robotic manipulation:** HAMLET, built on GR00T N1.5, lifts success rates by up to 47.2 percentage points on real-world long-horizon tasks compared to VLA baselines, with only marginal computational overhead [2510.00695].
- **GUI navigation:** HCPO enables a 3B-parameter agent to outperform much larger (7B) models on out-of-distribution GUI benchmarks (e.g., GUI-Odyssey, AndroidInTheWild) while reducing FLOPs by 60% [2512.01763].

These results collectively suggest that HAPO-style architectures—by leveraging explicit history—excel in domains where long-horizon dependency, sparse rewards, or efficiency constraints are bottlenecks.

## 5. Limitations and Open Challenges

Despite encouraging results, multiple limitations are documented in the literature:

- **Hyperparameter sensitivity:** Regularization coefficients (entropy, KL), normalization schemes, and memory module sizes typically require nontrivial tuning between tasks [2508.18884, 2512.01763].
- **Compute and memory overhead:** Although techniques such as moment tokens or anchor pruning reduce incremental cost, storing, maintaining, and reprocessing batch- or task-level history can be expensive, especially for hierarchical or continual learning extensions [2508.18884, 2510.00695].
- **Episodes and generalization:** HAPO methods with per-query history (HAPO for LLMs) maintain state per training example, potentially limiting sample-efficiency when generalization across problems is desired [2505.11225].
- **Scale and multi-agent compatibility:** Demonstrations are currently limited to tasks with $\lesssim 10^3$ steps or single-agent scenarios. Large-scale public LLM benchmarks and multi-agent settings are untested, and generalization to cross-episode or lifelong learning is an open problem [2508.18884].
- **History encoding compression:** Optimal strategies for history compression, anchoring, or gating (e.g., moment token number, Transformer memory depth) remain empirical and domain-specific [2510.00695].

## 6. Future Directions and Potential Extensions

Several directions are identified for extending HAPO:

- **Automated regularization schedules:** Meta-gradient or annealing approaches for auto-tuning entropy and KL penalties, potentially improving stability and transfer across domains [2508.18884].
- **Hierarchical and continual memory:** Hierarchical versions of HAEPO are proposed for ultra-long-horizon or continual settings, with truncated backpropagation or memory buffers [2508.18884, 2510.00695].
- **Adaptive memory allocation:** Learning to represent and select only the most temporally informative features (e.g., via gating or attention) for history encoding [2510.00695].
- **Integration with other RL paradigms:** Joint application of history-aware weighting and on-policy RL objectives (e.g., PPO, offline RL) or expanded to joint trajectory ranking in multi-agent contexts [2508.18884].
- **Broader benchmarks:** Future work will likely apply HAPO to larger LLM alignment corpora (AlpacaEval, MT-Bench), real-world robotics, and life-long reasoning benchmarks when computational resources allow [2508.18884, 2505.11225].
- **Richer history representations:** For problem-specific history, tracking more than a scalar (e.g., distributional length statistics, histograms) could potentially drive even more efficient behavioral compression [2505.11225].

## 7. Related Frameworks and Distinctions

HAPO subsumes and extends several earlier approaches:

- **Trajectory-level and listwise RL:** Unlike pairwise (DPO) or per-timestep (GRPO), HAEPO and similar listwise-weighted frameworks use trajectory-level normalization and ranking to amplify high-quality, diverse trajectories [2508.18884].
- **Contrastive and memory-based architectures:** Models like HAMLET demonstrate that temporally-aware contrastive pretraining and lightweight memory transformers efficiently transmit historical perceptual cues, achieving substantial improvements even with frozen vision-language backbones [2510.00695].
- **History-enhanced policy optimization (HCPO):** For domains where context size must be dynamically tuned for each decision (e.g., GUI agents), HCPO provides evidence that dual-branch, anchor-guided compression can trade off context fidelity and efficiency, outperforming fixed-window or fully compressed baselines [2512.01763].

In sum, History-Aware Policy Optimization provides a rigorously tested, flexible, and increasingly adopted paradigm for addressing the challenges of sparse-reward, long-horizon, and history-dependent environments in RL, LLM alignment, and sequential decision-making [2505.11225, 2508.18884, 2510.00695, 2512.01763].

Source: https://www.emergentmind.com/topics/history-aware-policy-optimization-hapo