Entropy Trend Reward (ETR)

• ETR is a reward-shaping principle that guides reasoning by incentivizing a downward trend in uncertainty across multi-step inference.
• It employs momentum accumulation to weight early entropy reductions more heavily, promoting concise and decisive chains of thought.
• Empirical results show that ETR improves model accuracy and efficiency by balancing exploration with correctness gating during reasoning.

Searching arXiv for the cited ETR and related entropy-reward papers to ground the article with current metadata. arxiv_search(query="(Xiong et al., 7 Apr 2026) OR (Huang et al., 9 Oct 2025) OR (Tan et al., 6 Aug 2025) OR (Cao et al., 28 Mar 2025) OR (Rahman, 23 May 2025)", max_results=10) Entropy Trend Reward (ETR) is a reward-shaping principle for reasoning models in which the optimization target is not merely the absolute level of predictive uncertainty, nor the final response length, but the trajectory of uncertainty across a chain of thought. In the form explicitly introduced for large reasoning models, ETR rewards reasoning traces whose entropy declines progressively over the course of multi-step inference, while preserving correctness as the primary criterion; closely related work operationalizes the same general idea at coarser phase granularity by penalizing disproportionately high entropy in the “thinking” segment relative to the final answer segment (Xiong et al., 7 Apr 2026).

1. Definition and conceptual scope

In the ETR formulation for chain-of-thought reasoning, a generated response is treated as a trajectory whose informational dynamics matter. The central empirical claim is that effective and efficient reasoning is associated not with uniformly low entropy, but with a downward trend in entropy as reasoning proceeds: earlier steps may remain exploratory, whereas later steps should become increasingly decisive. The paper introducing ETR states that “CoTs with dominant downward entropy trends are substantially shorter,” and operationalizes “dominant downward entropy trends” via a negative Spearman rank correlation between reasoning-step index and step-wise entropy (Xiong et al., 7 Apr 2026).

This makes ETR distinct from two common alternatives. First, it is not a pure length penalty: length is treated as an outcome correlated with uncertainty dynamics rather than as a direct target. Second, it is not global entropy minimization: the ETR paper argues that penalizing uncertainty everywhere suppresses useful early exploration and self-reflection. A closely related precursor is Phase Entropy Aware Reward (PEAR), which reports that average token-level entropy is positively correlated with response length and that the thinking phase delimited by > ... </think> systematically exhibits higher entropy than the final answer phase; PEAR uses this phase asymmetry as a control signal for conciseness without explicit length targets (Huang et al., 9 Oct 2025).

In this sense, “Entropy Trend Reward” names both a specific reward introduced in 2026 and a broader family of entropy-trajectory objectives. This broader usage is visible in adjacent papers that discuss ETR as a conceptual category, even when their default method relies on entropy magnitude rather than entropy trend (Tan et al., 6 Aug 2025).

2. Mathematical formulation

The canonical ETR formulation begins by segmenting a chain of thought into sentence-level steps using the delimiter \n\n, yielding

$o = \{C_1, C_2, \dots, C_T\}.$

For each step $t$, the step-wise uncertainty is defined as the Shannon entropy of the model’s next-token predictive distribution after completing $C_{1:t}$:

$H_t = H\big(p_\theta(\cdot \mid C_{1:t})\big).$

The paper further writes the step-wise entropy change as

$\Delta_t = H_{t-1} - H_t, \qquad t = 2, \dots, T.$

Positive $\Delta_t$ indicates uncertainty reduction, while negative $\Delta_t$ indicates an increase in uncertainty (Xiong et al., 7 Apr 2026).

A naïve objective would sum these changes:

$$R_{\text{naive}}(o) = \sum_{t=2}^{T} \Delta_t = H_1 - H_T.$$

The ETR paper rejects this telescoping form because it ignores the internal shape of the trajectory. Instead, it introduces a momentum accumulator

$S_t = \gamma S_{t-1} + \Delta_t, \qquad S_1 = 0,$

with $\gamma \in (0,1)$, and defines the entropy-trend reward as

$t$0

The same quantity can be unrolled as

$t$1

where

$t$2

Because $t$3 strictly decreases with $t$4, earlier entropy reductions are weighted more heavily than later ones. The intended effect is to encourage early convergence and to penalize oscillatory or repeatedly rising-uncertainty reasoning (Xiong et al., 7 Apr 2026).

Correctness remains the hard gate. The final reward is

$t$5

where $t$6 is the entropy-trend weight. The paper describes this as a two-stage reward structure in which entropy trend differentiates among correct trajectories rather than compensating for incorrectness (Xiong et al., 7 Apr 2026).

A related but phase-level instantiation appears in PEAR. Let $t$7 and $t$8 denote average entropies over the <think> and post-</think> segments, computed under the old policy and excluding the closing `` token. PEAR defines a phase penalty

$t$9

and then uses

$C_{1:t}$0

The paper explicitly interprets this as an entropy trend reward instantiation: high entropy in thinking is penalized, while some answer-phase entropy offsets the penalty (Huang et al., 9 Oct 2025).

3. Optimization and implementation

ETR is integrated into Group Relative Policy Optimization (GRPO). For each question $C_{1:t}$1, a group of $C_{1:t}$2 candidate responses is sampled from $C_{1:t}$3, each response is assigned a return $C_{1:t}$4, and a group-normalized advantage is computed from the relative position of each reward within its sampled group. The GRPO objective then applies PPO-style clipping to likelihood ratios together with a KL penalty for stabilization (Xiong et al., 7 Apr 2026).

The training configuration reported for ETR uses group size $C_{1:t}$5, rollout temperature $C_{1:t}$6, maximum response length $C_{1:t}$7 tokens, KL penalty $C_{1:t}$8, PPO clip $C_{1:t}$9, entropy-trend weight $H_t = H\big(p_\theta(\cdot \mid C_{1:t})\big).$0, optimizer learning rate $H_t = H\big(p_\theta(\cdot \mid C_{1:t})\big).$1, LoRA rank $H_t = H\big(p_\theta(\cdot \mid C_{1:t})\big).$2, LoRA alpha $H_t = H\big(p_\theta(\cdot \mid C_{1:t})\big).$3, batch size $H_t = H\big(p_\theta(\cdot \mid C_{1:t})\big).$4, mini-batch $H_t = H\big(p_\theta(\cdot \mid C_{1:t})\big).$5, micro-batch $H_t = H\big(p_\theta(\cdot \mid C_{1:t})\big).$6, one epoch, and $H_t = H\big(p_\theta(\cdot \mid C_{1:t})\big).$7 training steps on $H_t = H\big(p_\theta(\cdot \mid C_{1:t})\big).$8 H100 GPUs in VeRL. The default momentum coefficient is $H_t = H\big(p_\theta(\cdot \mid C_{1:t})\big).$9 (Xiong et al., 7 Apr 2026).

Implementation is intentionally lightweight. Step entropies are computed from rollout logits at sentence boundaries, with no moving-average or window smoothing beyond the momentum recurrence itself. The paper states that entropy calculation reuses rollout logits and adds negligible overhead relative to sequence generation. The recommended segmentation is sentence-level via \n\n; token-level segmentation is explicitly reported to over-compress outputs and harm accuracy (Xiong et al., 7 Apr 2026).

PEAR uses the same general RL setting—GRPO with KL regularization—but replaces binary correctness rewards with a phase-aware entropy penalty. Its main experiments use DeepSeek-R1-Distill-Qwen-1.5B, Qwen3-4B, and Qwen3-8B, trained in verl on GSM8K with batch size $\Delta_t = H_{t-1} - H_t, \qquad t = 2, \dots, T.$0, learning rate $\Delta_t = H_{t-1} - H_t, \qquad t = 2, \dots, T.$1, GRPO rollouts $\Delta_t = H_{t-1} - H_t, \qquad t = 2, \dots, T.$2, KL coefficient $\Delta_t = H_{t-1} - H_t, \qquad t = 2, \dots, T.$3, and default $\Delta_t = H_{t-1} - H_t, \qquad t = 2, \dots, T.$4. Evaluation uses temperature $\Delta_t = H_{t-1} - H_t, \qquad t = 2, \dots, T.$5, top-$\Delta_t = H_{t-1} - H_t, \qquad t = 2, \dots, T.$6, and a maximum generation length of $\Delta_t = H_{t-1} - H_t, \qquad t = 2, \dots, T.$7 tokens (Huang et al., 9 Oct 2025).

4. Empirical behavior and performance

The ETR paper evaluates on AIME24, AMC23, MATH500, and GPQA Diamond using DeepSeek-R1-Distill-Qwen-7B, Qwen3-4B, and Qwen3-8B, with additional scale studies on DeepSeek-R1-Distill-14B and DeepSeek-R1-Distill-Qwen-1.5B. Reported metrics are accuracy, response length, compression rate, and Accuracy–Efficiency Trade-off Score (AES), where

$\Delta_t = H_{t-1} - H_t, \qquad t = 2, \dots, T.$8

Its headline result is that ETR improves DeepSeek-R1-Distill-7B by $\Delta_t = H_{t-1} - H_t, \qquad t = 2, \dots, T.$9 in accuracy while reducing CoT length by $\Delta_t$0 across the four benchmarks. For that model, the overall metrics move from Acc $\Delta_t$1, Len $\Delta_t$2k, AES $\Delta_t$3 to Acc $\Delta_t$4, Len $\Delta_t$5k, AES $\Delta_t$6 under ETR (Xiong et al., 7 Apr 2026).

Ablation studies are central to the interpretation. Instant entropy minimization (“Min. H”) shortens CoTs but yields weaker AES than full ETR; maximizing entropy collapses reasoning; removing momentum degrades length reduction; and removing correctness gating produces short outputs with a large accuracy drop. The paper therefore attributes ETR’s behavior specifically to the combination of trajectory-level entropy shaping, momentum, and correctness gating rather than to entropy manipulation in isolation (Xiong et al., 7 Apr 2026).

PEAR exhibits a different but related empirical pattern. Across GSM8K, MATH500, AIME24, and AMC23, it reports average token reductions of $\Delta_t$7 for DeepSeek-R1-Distill-Qwen-1.5B, $\Delta_t$8 for Qwen3-4B, and $\Delta_t$9 for Qwen3-8B, while keeping average accuracy changes small. The paper further reports out-of-distribution robustness: although trained only on GSM8K, PEAR reduces length and sustains accuracy on MATH500, AIME24, and AMC23. It also finds that after PEAR training, entropy decreases appreciably in the thinking phase and slightly increases in the final-answer phase, which the authors interpret as suppressed exploratory redundancy together with preserved flexibility where answer formatting and commitment occur (Huang et al., 9 Oct 2025).

5. Relation to adjacent entropy-based methods

ETR sits within a larger line of work that uses entropy as a training signal, but its defining characteristic is that it rewards the shape of entropy over time rather than entropy magnitude alone.

Method	Entropy signal	Core mechanism
ETR	Step-wise entropy trend	Rewards progressive uncertainty reduction
PEAR	Thinking/answer phase averages	Penalizes high thinking entropy relative to answer entropy
GTPO / GRPO-S	Token or sequence entropy magnitude	Upweights high-entropy correct tokens or sequences
PPO-BR	Policy entropy level plus reward progression	Adapts PPO clipping threshold

GTPO and GRPO-S explicitly distinguish their dynamic entropy weighting from ETR. Their token-level and sequence-level weights are based on entropy magnitude in correct responses, not on temporal entropy trend, although the paper states that explicit ETR-style trend features such as $\Delta_t$0 or correlation with time can be added naturally (Tan et al., 6 Aug 2025). PPO-BR similarly does not define an entropy trend reward; instead, it uses normalized policy entropy and smoothed reward progression to modulate PPO’s trust-region clipping threshold, with bounded guarantees and reported gains in convergence speed and variance reduction (Rahman, 23 May 2025).

EDU-PRM is adjacent in a different way. It uses token-level entropy to detect dynamic step boundaries during decoding and then assigns step rewards through Monte Carlo-estimated correctness probabilities. The paper presents ETR as a direct extension: the entropy signal already used for partitioning can also be used to weight or modulate step rewards based on entropy level or trend, thereby turning uncertainty-guided process supervision into an entropy-trend reward scheme (Cao et al., 28 Mar 2025).

6. Terminological ambiguity, limitations, and broader usage

The term “Entropy Trend Reward” is not fully standardized across the literature. In some papers it denotes a precise reward function over reasoning trajectories, as in the 2026 ETR paper. In others it denotes a conceptual family of trend-aware entropy shaping, or even a proposed extension rather than the default method. An unrelated example appears in Ethereum decentralization analysis, where the study does not introduce ETR explicitly but notes that one can define an “ETR” as a first difference or regression slope of Shannon entropy over time to track the temporal trend of validator reward decentralization (Yan et al., 2024). This suggests that the phrase is semantically broader than any single implementation.

Within reasoning-model training, several limitations recur across the literature. Over-penalizing uncertainty can induce under-exploration on hard multi-step problems. The ETR paper mitigates this by allowing limited local increases in entropy and by keeping the entropy coefficient small; PEAR mitigates it by setting $\Delta_t$1 so that answer-phase entropy offsets excessive thinking-phase penalty. Both approaches therefore reject the assumption that low uncertainty is desirable everywhere (Xiong et al., 7 Apr 2026). A related failure mode is reward gaming through mechanically deterministic but uninformative text. PEAR addresses this with correctness gating, format checks, KL regularization, and diversity monitoring, while ETR uses correctness as a hard gate so that entropy shaping only ranks correct samples (Huang et al., 9 Oct 2025).

A further misconception is that entropy-aware conciseness control is equivalent to rigid truncation. Both ETR and PEAR oppose that interpretation. ETR leaves maximum response length large and relies on trajectory reward to induce earlier convergence. PEAR likewise emphasizes adaptive control without explicit length targets or truncation rules, and reports that redundancy is concentrated in the thinking phase rather than the final answer segment (Huang et al., 9 Oct 2025).

Taken together, the literature presents ETR as a trajectory-sensitive alternative to length penalties and global entropy suppression. Its central claim is not that reasoning should be uniformly low-entropy, but that efficient reasoning should exhibit a disciplined transition from exploratory uncertainty to committed resolution.