Entropy-Based Advantage Reshaping (EAR)

• EAR is a reinforcement learning framework that integrates entropy signals into policy gradients rather than relying solely on reward, enabling refined exploration.
• It employs mechanisms like additive entropy advantages, baseline modifications, and multiplicative reweighting to stabilize and enhance policy improvement.
• Applied in continuous control, LLM reasoning, and test-time RL, EAR demonstrates improved generalization and balanced exploration–exploitation dynamics.

Entropy-based Advantage Reshaping (EAR) denotes a family of reinforcement-learning techniques in which the effective advantage used by policy optimization is modified by entropy-derived structure rather than by reward alone. In the cited literature, this modification appears as an explicit additive entropy advantage in on-policy actor-critic, as return- or baseline-level transformations whose induced policy signal is entropy-aware, and as multiplicative response- or token-level reweighting driven by entropy, surprisal, or entropy polarity. Across continuous-control RL, RLVR for reasoning LLMs, and test-time RL, the common purpose is to align policy updates with maximum-entropy objectives or to regulate entropy dynamics so that exploration is preserved without destabilizing exploitation (Choe et al., 2024, Petrenko et al., 12 Mar 2026, Zhang et al., 12 May 2026, Luo et al., 17 Jun 2026, Wu et al., 26 Sep 2025).

1. Conceptual scope and design space

EAR is not a single canonical method name in the literature. Closely related formulations include Entropy Advantage Estimation (EAE), Entropy Augmented Reinforcement Learning (EARL), REPO, AEM, PAPO, STARE, QAE, and LESS. These methods differ mainly in where entropy enters the update: inside the advantage itself, through an entropy-augmented return or baseline, or through token- and response-level reweighting rules. In each case, the central object is still the policy-gradient multiplier attached to $\nabla_\theta \log \pi$, but that multiplier is no longer purely reward-derived (Ma, 2022, Zhao et al., 1 May 2026, Chen et al., 30 Nov 2025).

A core distinction in this literature is between entropy bonus methods and advantage reshaping methods. The former add entropy directly to the actor loss, whereas the latter make entropy affect the same advantage-weighted policy-gradient structure used for reward optimization. A second distinction is between EAR and architectural entropy-control methods. "Entropy Regularizing Activation" is explicitly framed as an output-parameter projection or activation that enforces an entropy floor, rather than as an explicit entropy bonus or an advantage reshaping method in continuous control (Kang et al., 9 Oct 2025).

Formulation family	Representative papers	Mechanism
Explicit soft advantage decomposition	(Choe et al., 2024)	Separate entropy critic and additive advantage correction
Entropy-augmented return or baseline shaping	(Ma, 2022, Wu et al., 26 Sep 2025, Petrenko et al., 12 Mar 2026)	Modified Bellman target, baseline, or log-probability correction
Response- or token-level multiplicative reshaping	(Liu et al., 15 Aug 2025, Zhao et al., 1 May 2026, Zhang et al., 12 May 2026, Luo et al., 17 Jun 2026, Chen et al., 30 Nov 2025)	Entropy-, surprisal-, polarity-, or segment-conditioned weighting

2. Explicit EAR in on-policy actor-critic

The clearest explicit EAR formulation in standard actor-critic appears in "Maximum Entropy On-Policy Actor-Critic via Entropy Advantage Estimation" (Choe et al., 2024). The paper begins from the distinction between adding an entropy regularizer to PPO/TRPO and optimizing the discounted maximum-entropy objective

$$J_{ME}(\pi)=\mathbb E_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t\left(r_t+\alpha \mathcal H(\pi(\cdot|s_t))\right)\right].$$

Its central claim is that standard on-policy entropy bonuses are not faithful estimators of the policy gradient of this discounted objective, because entropy is usually injected only as a local regularizer: it is not bootstrapped by a value function, not accumulated into a return estimate, and therefore not included in the advantage estimator in the same way as reward.

The paper’s solution is Entropy Advantage Estimation (EAE), which introduces entropy-side analogues of reward return, value, action-value, and advantage. With

$$A_{\text{soft}}^\pi(s,a)=A_r^\pi(s,a)+\alpha A_{\mathcal H}^\pi(s,a),$$

the actor update uses a combined soft advantage rather than a reward-only advantage plus a separate entropy bonus. The entropy value satisfies a Bellman recursion,

$$V_{\mathcal H}^{\pi}(s_t)=\mathcal H(\pi(\cdot|s_t))+\gamma \mathbb E[V_{\mathcal H}^{\pi}(s_{t+1})],$$

with entropy TD residual

$$\delta_t^{\mathcal H}=\mathcal H(\pi(\cdot|s_t))+\gamma V_{\mathcal H}(s_{t+1})-V_{\mathcal H}(s_t),$$

and an entropy GAE

$$\hat A_t^{\mathcal H,\text{GAE}(\gamma,\lambda)}=\sum_{l=0}^{T-t-1}(\gamma\lambda)^l \delta_{t+l}^{\mathcal H}.$$

The final estimator is

$$\hat A_t^{\text{soft}}=\hat A_t^r+\alpha \hat A_t^{\mathcal H},$$

which is then inserted directly into PPO or TRPO surrogates in place of the ordinary reward advantage. In this formulation, EAR is not merely entropy-flavored regularization; it is an explicit decomposition of the policy-improvement signal into reward and entropy branches (Choe et al., 2024).

3. Return shaping, baseline shaping, and implicit EAR

A second family of EAR formulations begins not by defining a new advantage estimator, but by altering the return or the baseline so that the induced advantage becomes entropy-aware. "Entropy Augmented Reinforcement Learning" defines an entropy-augmented return

$$\eta(\pi)=\mathbb E_{\tau}\Biggl[\sum_{t=0}^{\infty}\gamma^t\bigl(r(s_t,a_t)+\gamma\alpha \mathbb E_{s_{t+1}} \mathcal H(s_{t+1})\bigr)\Biggr],$$

equivalently using the augmented reward $\hat r(s,a)=r(s,a)+\gamma\alpha \mathcal H(s')$. In the on-policy algorithm, the practical actor-side signal becomes

$A_t+\gamma\alpha \mathcal H^{\pi_\theta}(s_{t+1}),$

while critic targets are trained on the entropy-augmented reward. The paper presents this primarily as entropy-augmented Bellman consistency, but operationally it is an EAR-style reshaping of the advantage-like signal used in PPO/TRPO updates (Ma, 2022).

"Entropy-preserving reinforcement learning" makes the EAR mechanism even more explicit. Its REPO family modifies the advantage as

$$J_{ME}(\pi)=\mathbb E_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t\left(r_t+\alpha \mathcal H(\pi(\cdot|s_t))\right)\right].$$0

where

$$J_{ME}(\pi)=\mathbb E_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t\left(r_t+\alpha \mathcal H(\pi(\cdot|s_t))\right)\right].$$1

The practical REPO-R approximation uses sign-specific multiplicative forms such as

$$J_{ME}(\pi)=\mathbb E_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t\left(r_t+\alpha \mathcal H(\pi(\cdot|s_t))\right)\right].$$2

for positive advantages and

$$J_{ME}(\pi)=\mathbb E_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t\left(r_t+\alpha \mathcal H(\pi(\cdot|s_t))\right)\right].$$3

for negative advantages. The stated intuition is that increasing policy entropy requires upweighting rare correct solutions while reducing, on average, the penalty assigned to rare incorrect solutions (Petrenko et al., 12 Mar 2026).

"Quantile Advantage Estimation for Entropy-Safe Reasoning" reshapes the advantage through the baseline itself. For binary rewards, the right-continuous $$J_{ME}(\pi)=\mathbb E_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t\left(r_t+\alpha \mathcal H(\pi(\cdot|s_t))\right)\right].$$4-quantile baseline reduces to

$$J_{ME}(\pi)=\mathbb E_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t\left(r_t+\alpha \mathcal H(\pi(\cdot|s_t))\right)\right].$$5

with

$$J_{ME}(\pi)=\mathbb E_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t\left(r_t+\alpha \mathcal H(\pi(\cdot|s_t))\right)\right].$$6

This yields a two-regime gate: on hard queries only successful responses receive positive advantage, while on easy queries only failures receive negative advantage. The method is therefore a baseline-based EAR mechanism in which entropy control is realized by changing which responses receive nonzero credit at all (Wu et al., 26 Sep 2025).

4. Response- and token-level EAR in LLM RL

In LLM RL, EAR often takes the form of multiplicative reweighting rather than additive soft-value decomposition. The most literal use of the term appears in "ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanism," where response entropy

$$J_{ME}(\pi)=\mathbb E_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t\left(r_t+\alpha \mathcal H(\pi(\cdot|s_t))\right)\right].$$7

is used to rescale the GRPO advantage by

$$J_{ME}(\pi)=\mathbb E_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t\left(r_t+\alpha \mathcal H(\pi(\cdot|s_t))\right)\right].$$8

With the practical clip range stated as $$J_{ME}(\pi)=\mathbb E_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^t\left(r_t+\alpha \mathcal H(\pi(\cdot|s_t))\right)\right].$$9, high-entropy responses are downweighted and low-entropy responses are upweighted. The purpose is to prevent low-confidence majority-vote pseudo-rewards from producing disproportionately large GRPO updates during test-time RL (Liu et al., 15 Aug 2025).

"AEM: Adaptive Entropy Modulation for Multi-Turn Agentic Reinforcement Learning" defines a response-level uncertainty proxy

$$A_{\text{soft}}^\pi(s,a)=A_r^\pi(s,a)+\alpha A_{\mathcal H}^\pi(s,a),$$0

normalizes it within a prompt group, and forms a multiplicative coefficient

$$A_{\text{soft}}^\pi(s,a)=A_r^\pi(s,a)+\alpha A_{\mathcal H}^\pi(s,a),$$1

so that

$$A_{\text{soft}}^\pi(s,a)=A_r^\pi(s,a)+\alpha A_{\mathcal H}^\pi(s,a),$$2

This rule is sign-agnostic at the level of the coefficient, but because it multiplies signed advantages it induces different entropy effects on positive and negative responses, allowing the aggregate training dynamics to shift from exploration to exploitation without an explicit entropy schedule (Zhao et al., 1 May 2026).

"Entropy Polarity in Reinforcement Fine-Tuning: Direction, Asymmetry, and Control" and "STARE: Surprisal-Guided Token-Level Advantage Reweighting for Policy Entropy Stability" move EAR to token granularity. PAPO defines entropy polarity

$$A_{\text{soft}}^\pi(s,a)=A_r^\pi(s,a)+\alpha A_{\mathcal H}^\pi(s,a),$$3

and applies branchwise reweighting,

$$A_{\text{soft}}^\pi(s,a)=A_r^\pi(s,a)+\alpha A_{\mathcal H}^\pi(s,a),$$4

STARE instead derives

$$A_{\text{soft}}^\pi(s,a)=A_r^\pi(s,a)+\alpha A_{\mathcal H}^\pi(s,a),$$5

identifies entropy-critical tokens by batch-internal surprisal quantiles, and applies

$$A_{\text{soft}}^\pi(s,a)=A_r^\pi(s,a)+\alpha A_{\mathcal H}^\pi(s,a),$$6

under a target-entropy closed-loop gate. In both cases, token updates are partitioned by their predicted entropy effect and then reweighted rather than discarded (Zhang et al., 12 May 2026, Luo et al., 17 Jun 2026).

"Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs" extends EAR in a different direction. LESS leaves high-entropy tokens unchanged, but reshapes low-entropy segments according to whether they recur only in correct responses, only in incorrect responses, or in both. Shared low-entropy segments receive zero token advantage; correct-only and incorrect-only segments are scaled by normalized recurrence factors. The method therefore treats entropy not merely as a monotone exploration signal, but as a segmentation signal that must be interpreted jointly with correctness statistics (Chen et al., 30 Nov 2025).

5. Theoretical foundations

The theoretical basis of EAR is the claim that entropy dynamics are governed by the same multiplier that drives policy improvement, so entropy cannot be treated as a wholly separate add-on. In the on-policy maximum-entropy actor-critic setting, this appears as the decomposition

$$A_{\text{soft}}^\pi(s,a)=A_r^\pi(s,a)+\alpha A_{\mathcal H}^\pi(s,a),$$7

which makes the discounted maximum-entropy objective induce an additive entropy advantage inside the policy gradient itself (Choe et al., 2024).

In REPO, the central theorem states that under a first-order update,

$$A_{\text{soft}}^\pi(s,a)=A_r^\pi(s,a)+\alpha A_{\mathcal H}^\pi(s,a),$$8

under the orthogonality approximation, so entropy collapse arises when reward advantage and centered log-probability become positively aligned. The REPO correction then directly alters that correlation structure through the transformed advantage (Petrenko et al., 12 Mar 2026).

In AEM, the key drift identity is

$$A_{\text{soft}}^\pi(s,a)=A_r^\pi(s,a)+\alpha A_{\mathcal H}^\pi(s,a),$$9

which says that the sign of entropy change depends on the interaction between sampled-response advantage and relative surprisal. PAPO makes a closely related point at token level with

$$V_{\mathcal H}^{\pi}(s_t)=\mathcal H(\pi(\cdot|s_t))+\gamma \mathbb E[V_{\mathcal H}^{\pi}(s_{t+1})],$$0

leading to entropy polarity $$V_{\mathcal H}^{\pi}(s_t)=\mathcal H(\pi(\cdot|s_t))+\gamma \mathbb E[V_{\mathcal H}^{\pi}(s_{t+1})],$$1; STARE obtains a four-quadrant structure from

$$V_{\mathcal H}^{\pi}(s_t)=\mathcal H(\pi(\cdot|s_t))+\gamma \mathbb E[V_{\mathcal H}^{\pi}(s_{t+1})],$$2

showing that the same response-level advantage can produce opposite entropy effects at different token positions depending on local surprisal (Zhao et al., 1 May 2026, Zhang et al., 12 May 2026, Luo et al., 17 Jun 2026).

QAE provides a baseline-level entropy theory. With the first-order approximation

$$V_{\mathcal H}^{\pi}(s_t)=\mathcal H(\pi(\cdot|s_t))+\gamma \mathbb E[V_{\mathcal H}^{\pi}(s_{t+1})],$$3

the paper proves that $$V_{\mathcal H}^{\pi}(s_t)=\mathcal H(\pi(\cdot|s_t))+\gamma \mathbb E[V_{\mathcal H}^{\pi}(s_{t+1})],$$4 is strictly increasing in the baseline $$V_{\mathcal H}^{\pi}(s_t)=\mathcal H(\pi(\cdot|s_t))+\gamma \mathbb E[V_{\mathcal H}^{\pi}(s_{t+1})],$$5. Choosing the quantile baseline therefore yields two-sided entropy safety: when success probability is low, $$V_{\mathcal H}^{\pi}(s_t)=\mathcal H(\pi(\cdot|s_t))+\gamma \mathbb E[V_{\mathcal H}^{\pi}(s_{t+1})],$$6 gives the smallest entropy increase among all $$V_{\mathcal H}^{\pi}(s_t)=\mathcal H(\pi(\cdot|s_t))+\gamma \mathbb E[V_{\mathcal H}^{\pi}(s_{t+1})],$$7; when success probability is high, $$V_{\mathcal H}^{\pi}(s_t)=\mathcal H(\pi(\cdot|s_t))+\gamma \mathbb E[V_{\mathcal H}^{\pi}(s_{t+1})],$$8 gives the largest entropy change and therefore resists collapse (Wu et al., 26 Sep 2025).

A plausible unifying interpretation is that EAR methods intervene on the factor multiplying $$V_{\mathcal H}^{\pi}(s_t)=\mathcal H(\pi(\cdot|s_t))+\gamma \mathbb E[V_{\mathcal H}^{\pi}(s_{t+1})],$$9: they either add an entropy advantage, subtract a log-probability correction, or rescale the reward-derived advantage by uncertainty proxies such as response entropy, surprisal, polarity, or regime-dependent gates. This suggests that EAR is best viewed not as one algorithm but as a theory-guided class of policy-gradient modifications.

6. Empirical profile, limitations, and related distinctions

In continuous-control and classical actor-critic settings, EAR-style methods are reported to improve policy optimization relative to naive entropy regularization. EAE extends PPO and TRPO within the maximum-entropy framework and improves policy optimization performance in both MuJoCo and Procgen tasks, while also highlighting MaxEnt RL’s capacity to enhance generalisation. EARL augments TRPO and PPO on MuJoCo and verifies its exploration bonus on custom environments; it also emphasizes that large initial temperature is worse than small initial temperature and that decay is better than no decay (Choe et al., 2024, Ma, 2022).

In LLM RL and RLVR, the empirical picture is broader and more heterogeneous. ETTRL reports a 68 percent relative improvement in Pass@1 on AIME 2024 while consuming only 60 percent of the rollout tokens budget. AEM improves strong RL baselines on ALFWorld, WebShop, and SWE-bench-Verified, including a +1.4\% gain when integrated into a state-of-the-art software-engineering RL training framework. PAPO consistently outperforms competitive baselines on mathematical reasoning and agentic benchmarks and is reported to reach mature DAPO-level reward with about half the training budget. STARE sustains stable RL training over thousands of steps, maintains policy entropy within a target band, and reports 4%-8% average accuracy gains on AIME24 and AIME25. LESS improves average accuracy and the performance floor across six math benchmarks, and QAE stabilizes entropy while making roughly 80% of responses zero-advantage under tuned $$\delta_t^{\mathcal H}=\mathcal H(\pi(\cdot|s_t))+\gamma V_{\mathcal H}(s_{t+1})-V_{\mathcal H}(s_t),$$0 (Liu et al., 15 Aug 2025, Zhao et al., 1 May 2026, Zhang et al., 12 May 2026, Luo et al., 17 Jun 2026, Chen et al., 30 Nov 2025, Wu et al., 26 Sep 2025).

The limitations are correspondingly diverse. EAE adds an entropy value function and inherits the usual critic-dependent bias-variance tradeoffs. EARL stresses temperature control and annealing because its shaping term is not potential-based shaping, so policy invariance is not guaranteed for arbitrary $$\delta_t^{\mathcal H}=\mathcal H(\pi(\cdot|s_t))+\gamma V_{\mathcal H}(s_{t+1})-V_{\mathcal H}(s_t),$$1. AEM relies on a heuristic response-level proxy and prompt-group normalization; PAPO and STARE rely on first-order local analyses, entropy controllers, and proxy approximations rather than exact global entropy computation; LESS depends on entropy thresholding, segment length, and recurrence structure; QAE turns $$\delta_t^{\mathcal H}=\mathcal H(\pi(\cdot|s_t))+\gamma V_{\mathcal H}(s_{t+1})-V_{\mathcal H}(s_t),$$2 into a direct entropy-control knob; ETTRL’s relative-entropy scaling and clipping bounds are described as model- and dataset-dependent. A final conceptual boundary is provided by ERA: it pursues entropy control through the actor’s output activation and removes entropy terms from actor and critic losses in continuous control, so it belongs to architectural entropy-constrained policy parameterization rather than to EAR proper (Kang et al., 9 Oct 2025).

A recurrent misconception is that EAR is equivalent to adding an entropy bonus to the loss. The cited literature does not support that equivalence. In these papers, the essential move is that entropy changes the effective policy-improvement signal itself—through additive soft advantages, return or baseline shaping, or multiplicative reweighting—not merely through a separate regularizer. Another misconception is that entropy-aware reshaping must always favor high-entropy tokens. LESS explicitly argues against that monotone view by treating low-entropy segments as reusable structure that can be reinforced, suppressed, or neutralized depending on correctness-conditioned overlap.