HAEPO: History-Aggregated Exploratory Policy Optimization

• The paper introduces a trajectory-level optimization technique that aggregates entire rollouts into a single score to address credit assignment dilution.
• It employs listwise Plackett–Luce weighting to rank trajectories, preserving exploration signals in environments with sparse or delayed rewards.
• Empirical evaluations show that HAEPO achieves competitive stability and alignment quality compared to PPO, GRPO, and DPO across diverse tasks.

Searching arXiv for the HAEPO paper and closely related methods to ground the article with citations. arxiv_search(query="HAEPO History-Aggregated Exploratory Policy Optimization (Trivedi et al., 26 Aug 2025)") History-Aggregated Exploratory Policy Optimization (HAEPO) is a trajectory-level policy optimization method designed to improve exploration and stability in long-horizon reinforcement learning and LLM alignment settings. Its defining move is to compress an entire rollout into one history-aware score and then apply a listwise Plackett–Luce weighting over trajectories to determine which episodes should drive learning. The method was proposed in response to the claim that standard per-step policy gradients, PPO-style clipping, and several trajectory-level or preference-optimization methods still struggle to preserve exploration signals over long horizons, especially when rewards are sparse or delayed (Trivedi et al., 26 Aug 2025).

1. Motivation and problem setting

HAEPO was introduced to address a specific failure mode in long-horizon learning: credit assignment dilution. The paper argues that if optimization remains local to actions or tokens, the signal from a rare success at the end of a long episode can vanish. In that framing, the entire history of a trajectory becomes the unit of optimization, so that long-range dependencies and sparse rewards are reflected directly in the learning signal (Trivedi et al., 26 Aug 2025).

The method is positioned against several established baselines with complementary strengths and limitations. PPO is described as stable due to clipping or trust-region ideas, but its per-step update can still dilute long-horizon credit assignment and restrict exploration. GRPO removes the critic and uses group-relative advantages, but still largely relies on token- or sequence-level treatment of the output and does not explicitly use full-trajectory history as a ranking signal. DPO is described as elegant and simple for preference alignment, but fundamentally pairwise rather than trajectory-exploration-oriented. The paper further states that other trajectory-level approaches often either collapse rich trajectory distinctions too aggressively or aggregate in ways that weaken the exploration signal across long horizons (Trivedi et al., 26 Aug 2025).

This motivation yields three stated design goals. First, the update should “see” the full history rather than local actions alone. Second, the method should compare rollouts relatively at the batch level rather than by isolated tokenwise factors. Third, exploration should be broadened without losing stability, particularly in sparse-reward and delayed-reward regimes.

2. History aggregation and listwise weighting

The first step in HAEPO is cumulative trajectory scoring. For each trajectory $k$ of length $T_k$, the method computes the cumulative log-likelihood

$$L_k \;=\; \sum_{t=1}^{T_k} \log \pi_{\theta}\bigl(a^{(k)}_t \mid s^{(k)}_t\bigr), \qquad k=1,\dots,M.$$

This is the paper’s “history aggregation” step: the entire sequence of state-action choices is compressed into a single scalar score (Trivedi et al., 26 Aug 2025).

The paper attributes three functions to this aggregation. It preserves full-trajectory history, provides a single object for ranking rollouts, and strengthens discrimination between successful and unsuccessful episodes in long-horizon tasks. It also notes that because individual log-probabilities are usually negative, summing them can sharpen differences among trajectories. This suggests that HAEPO treats trajectory identity as an intrinsically sequential object rather than as a bag of local decisions.

HAEPO then normalizes these scores across the batch using a Plackett–Luce-style softmax,

$$w_k \;=\; \frac{\exp(L_k)}{\sum_{j=1}^{M} \exp(L_j)}.$$

In the appendix, this is emphasized as not merely a softmax but as interpretable as the first-choice probability under the Plackett–Luce model over the set of trajectories (Trivedi et al., 26 Aug 2025).

That listwise normalization is central to the method’s exploratory character. The weights are relative to the whole batch, so learning is listwise rather than pairwise. Low-weight rollouts are downweighted rather than discarded, which the paper presents as a mechanism for preserving diversity better than hard selection. The abstract further states that HAEPO “applies a Plackett-Luce softmax across trajectories to obtain normalized weights proportional to their returns,” while the objective couples those weights to returns explicitly (Trivedi et al., 26 Aug 2025).

3. Objective, regularization, and gradient decomposition

The base HAEPO objective rewards trajectories in proportion to both their return and their Plackett–Luce weight: $\mathcal{L}_{\mathrm{HAEPO}^{\mathrm{orig}(\theta) \;=\; - \sum_{k=1}^{M} R_k\,w_k(\theta).$ The discounted return is defined as

$R_k = \sum_{t=1}^{T_k} \gamma^{t-1} r^{(k)}_t.$

In the paper’s interpretation, high-return trajectories are rewarded, but only to the extent that they also have substantial listwise weight, yielding a trajectory-level credit assignment rule (Trivedi et al., 26 Aug 2025).

The score-function form is

$$\nabla_\theta \log w_k = \nabla_\theta L_k - \sum_{j=1}^M w_j(\theta)\,\nabla_\theta L_j.$$

The paper states that each trajectory’s gradient is thereby centered by the batch-weighted average, which reduces variance and makes the update more stable. This centered form is one of the method’s main conceptual claims: reward-seeking is expressed through trajectory scores, but the update is always relative to the current batch structure (Trivedi et al., 26 Aug 2025).

To prevent early collapse onto a small set of trajectories, HAEPO adds entropy regularization over the trajectory-weight distribution rather than only over action distributions. The entropy term is

$\beta \sum_{k=1}^{M} w_k \log w_k.$

Because $w_k \log w_k \le 0$, the paper states that adding this term with $\beta > 0$ rewards broader, higher-entropy trajectory distributions. The final objective also introduces a soft KL penalty relative to a frozen reference policy, described as a soft trust region rather than a hard clipping rule or hard constraint (Trivedi et al., 26 Aug 2025).

The paper gives the full gradient decomposition as

$T_k$0

The paper explicitly separates this into reward-seeking, entropy spreading, and KL-based stabilization. It also presents the compact form

$T_k$1

A plausible implication is that HAEPO’s regularizers are not auxiliary heuristics but direct modulators of the effective trajectory-level advantage.

4. Reward normalization, workflow, and implementation features

The paper uses two reward-normalization schemes depending on task structure. Sum-normalization is

$T_k$2

and is used when reward magnitudes are naturally comparable and absolute scale is less important, especially in short-horizon or bandit-like tasks. Z-score normalization is

$T_k$3

It is used for long-horizon or sparse or high-variance environments; the paper argues that this lowers variance while preserving unbiasedness (Trivedi et al., 26 Aug 2025).

The appendix reports a task-dependent preference between the two normalization schemes: z-score works better on sparse chain-like tasks, whereas sum-normalization works better on one-step dense-reward tasks like newsvendor. This is presented as a practical lesson rather than a universal rule. The broader implication drawn in the paper is that HAEPO is not only a new loss but a framework for trajectory-level optimization that can be tuned to reward geometry.

Although no compact standalone pseudocode block is presented, the workflow is stated explicitly:

1. Sample a batch of full trajectories.
2. Compute cumulative log-likelihood $T_k$4 for each trajectory.
3. Normalize returns using sum-normalization or z-score normalization.
4. Compute Plackett–Luce weights $T_k$5.
5. Optionally compute reference weights $T_k$6 from a frozen policy.
6. Apply the reward-weighted trajectory objective, entropy bonus, and KL penalty.
7. Backpropagate the final loss.

The appendix additionally states that the gradient implementation was sanity-checked via finite differences, with maximum absolute errors around $T_k$7, which the paper presents as evidence that the derivation and implementation are consistent (Trivedi et al., 26 Aug 2025).

5. Comparative position relative to PPO, GRPO, DPO, and related trajectory methods

The paper consistently describes HAEPO as more trajectory-centric than PPO. PPO is characterized as a per-step policy-gradient method with clipped updates: stable, but local over actions, susceptible to diluted trajectory-level credit assignment, and conservative via clipping. HAEPO differs by using full-trajectory cumulative log-likelihoods, ranking trajectories listwise via Plackett–Luce weights, and adding explicit entropy and KL terms at the trajectory level (Trivedi et al., 26 Aug 2025).

Relative to GRPO, the paper emphasizes that HAEPO aggregates the entire history into one cumulative log-likelihood, uses a Plackett–Luce normalization directly over trajectories, and emphasizes batch-wise trajectory ranking rather than tokenwise ratios. The stated claim is that GRPO can still dilute long-horizon exploration signals, while HAEPO preserves them more cleanly.

Relative to DPO, the distinction is conceptual as much as formal. DPO is presented as pairwise preference optimization: effective for alignment, but fundamentally a comparison objective between preferred and dispreferred outputs rather than a mechanism for broad rollout exploration. HAEPO instead treats feedback as scalar trajectory returns, uses listwise normalization rather than pairwise preference classification, and explicitly encourages exploration through entropy on trajectory weights (Trivedi et al., 26 Aug 2025).

The paper also places HAEPO among trajectory optimization, listwise preference optimization, and ranking-based approaches more broadly. Its claimed differentiator is that it does not merely optimize a static ranking or pairwise comparison. Instead, it aggregates the full rollout history into a score, normalizes across the batch, and regulates the resulting distribution with entropy and KL. The paper presents this combination as making the method more stable and interpretable than approaches that either collapse distinctions too aggressively or use per-token weighting that fragments long-horizon credit assignment.

6. Empirical results, ablations, and terminological clarification

The empirical study spans stochastic Gaussian bandits, a long-horizon sparse-reward random walk, Gymnasium CartPole-v1, and TL;DR summarization with LLaMA 3.2 (1B) and Qwen 2.5 (1.5B) (Trivedi et al., 26 Aug 2025).

Setting	Reported findings	Reported takeaway
Multi-armed bandit, $T_k$8	Faster than or comparably to PPO and DPO; higher exploration entropy as arms grow; regret fluctuations shrink with larger $T_k$9	Exploration breadth with stability
Random walk, $$L_k \;=\; \sum_{t=1}^{T_k} \log \pi_{\theta}\bigl(a^{(k)}_t \mid s^{(k)}_t\bigr), \qquad k=1,\dots,M.$$0	Near-optimal in about 12 updates; $$L_k \;=\; \sum_{t=1}^{T_k} \log \pi_{\theta}\bigl(a^{(k)}_t \mid s^{(k)}_t\bigr), \qquad k=1,\dots,M.$$1 faster than PPO and $$L_k \;=\; \sum_{t=1}^{T_k} \log \pi_{\theta}\bigl(a^{(k)}_t \mid s^{(k)}_t\bigr), \qquad k=1,\dots,M.$$2 faster than DPO per 100 updates	Strong in sparse, delayed-reward settings
CartPole-v1	PPO solves around 40 seconds; HAEPO around 130 seconds; unregularized variant around 140 seconds and more oscillatory	Smoother but slower convergence; regularization matters
TL;DR fine-tuning	Best human preference ratings; faster than GRPO and DPO on Qwen 2.5; lower peak VRAM than GRPO	Competitive or superior alignment quality with good efficiency

In the bandit setting, the paper reports that HAEPO converges faster than or comparably to PPO and DPO, maintains higher exploration entropy as the number of arms grows, and exhibits shrinking regret fluctuations with larger $$L_k \;=\; \sum_{t=1}^{T_k} \log \pi_{\theta}\bigl(a^{(k)}_t \mid s^{(k)}_t\bigr), \qquad k=1,\dots,M.$$3. In the long-horizon random walk with $$L_k \;=\; \sum_{t=1}^{T_k} \log \pi_{\theta}\bigl(a^{(k)}_t \mid s^{(k)}_t\bigr), \qquad k=1,\dots,M.$$4, HAEPO reaches near-optimal performance in about 12 updates, matches PPO and DPO in speed, is approximately $$L_k \;=\; \sum_{t=1}^{T_k} \log \pi_{\theta}\bigl(a^{(k)}_t \mid s^{(k)}_t\bigr), \qquad k=1,\dots,M.$$5 faster than PPO and $$L_k \;=\; \sum_{t=1}^{T_k} \log \pi_{\theta}\bigl(a^{(k)}_t \mid s^{(k)}_t\bigr), \qquad k=1,\dots,M.$$6 faster than DPO per 100 updates, and preserves strong early exploration (Trivedi et al., 26 Aug 2025).

On CartPole-v1, the paper reports a different tradeoff: PPO solves the task faster in wall-clock time, around 40 seconds, whereas HAEPO takes about 130 seconds. However, its learning curve is described as much smoother. The unregularized HAEPO variant is reported as much more oscillatory and slower, solving around 140 seconds. This is one of the clearest ablations in the paper: entropy and KL regularization are important, and without them learning becomes oscillatory and less stable.

In TL;DR summarization, HAEPO is reported to achieve the best human preference ratings: $$L_k \;=\; \sum_{t=1}^{T_k} \log \pi_{\theta}\bigl(a^{(k)}_t \mid s^{(k)}_t\bigr), \qquad k=1,\dots,M.$$7 for LLaMA 3.2 and $$L_k \;=\; \sum_{t=1}^{T_k} \log \pi_{\theta}\bigl(a^{(k)}_t \mid s^{(k)}_t\bigr), \qquad k=1,\dots,M.$$8 for Qwen 2.5, compared with GRPO at $$L_k \;=\; \sum_{t=1}^{T_k} \log \pi_{\theta}\bigl(a^{(k)}_t \mid s^{(k)}_t\bigr), \qquad k=1,\dots,M.$$9 and $$w_k \;=\; \frac{\exp(L_k)}{\sum_{j=1}^{M} \exp(L_j)}.$$0, and DPO at $$w_k \;=\; \frac{\exp(L_k)}{\sum_{j=1}^{M} \exp(L_j)}.$$1 and $$w_k \;=\; \frac{\exp(L_k)}{\sum_{j=1}^{M} \exp(L_j)}.$$2. For Qwen 2.5, training time is reported as 20 min for HAEPO, 25 min for GRPO, and 28 min for DPO; peak VRAM is reported as 28 GB for HAEPO, 38 GB for GRPO, and 26 GB for DPO. Qualitative feedback in the paper states that HAEPO summaries were concise, profanity-free, and aligned well with intent; DPO sometimes produced richer but overly long summaries; GRPO occasionally produced jarring profanity (Trivedi et al., 26 Aug 2025).

A recurrent source of confusion is nomenclature. A distinct later paper, “HiconAgent: History Context-aware Policy Optimization for GUI Agents” (Zhou et al., 1 Dec 2025), defines HCPO, not HAEPO. That paper explicitly states that it does not define a method named History-Aggregated Exploratory Policy Optimization, and that HCPO is the actual method name in the GUI-agent setting. Accordingly, HAEPO and HCPO should not be treated as the same officially named algorithm. The relationship is conceptual rather than terminological: both are history-aware policy-optimization frameworks, but the paper on HCPO states that “HAEPO” is not a paper-defined term in that work (Zhou et al., 1 Dec 2025).