---
title: 'TRE-K/TRE-P: Restricting Entropy in LLMs'
url: https://www.emergentmind.com/topics/restricting-entropy-maximization-to-plausible-actions-tre-k-tre-p
type: topic
---

# TRE-K/TRE-P: Restricting Entropy in LLMs

Restricting entropy maximization to plausible actions in reinforcement learning with large language models (LLMs) addresses the failure modes observed when applying naïve entropy regularization in vast action spaces. Standard entropy bonuses—formulated as global Shannon entropy over the entire vocabulary—are detrimental in LLMs because the action space (often 10^5 tokens) and long generation horizons lead to the accumulation of probability mass on semantically invalid tokens. Over many decoding steps, this “tail noise” severely degrades coherent reasoning by injecting unstructured randomness at every step. Trust Region Entropy (TRE), and specifically its instantiations TRE-K (“top-K”) and TRE-P (“top-p”), restrict the entropy maximization process to a dynamically selected set of plausible actions, thereby directing exploration to high-confidence regions of the token distribution while preserving stability and reasoning fidelity [2602.03635].

## 1. Motivation: Limitations of Global Entropy Regularization

Naïve entropy regularization in RL is conventionally used to encourage policy exploration via a global entropy term added to the standard surrogate (e.g., vanilla PPO loss). For LLMs, this is written as:
$$
L_{\text{total}} = L_{\text{surr}} + \beta(-H(\pi(\cdot|s))),
$$
where $\pi(a|s)$ outputs probabilities for actions $a$ given state $s$, and $H(\cdot)$ denotes Shannon entropy over the full token set $A$. Because $|A|$ can be $\gtrsim 10^5$, even a minuscule allocation of mass to the tail can result in substantial aggregate leakage, $\epsilon = \sum_{a\notin \text{valid}} \Delta \pi(a)$. Over a sequence of length $T$, the probability of not sampling an invalid token decays as $(1-\epsilon)^T$, rapidly diminishing to zero for long horizons. This compounds incoherence and degrades performance on long reasoning tasks [2602.03635].

## 2. Trust Region Entropy: Core Principle and Formalism

Trust Region Entropy (TRE) is designed to remedy the global tail risk by maximizing entropy only within a carefully chosen subset of plausible tokens—the “trust region”—at each decision point. Let $A_{TR}(s) \subseteq A$ denote this subset for state $s$. The policy is re-normalized over $A_{TR}$:
$$
\pi_{TR}(a|s) = \frac{\exp(z_a)}{\sum_{b \in A_{TR}(s)} \exp(z_b)} \quad \text{for } a \in A_{TR}(s),
$$
and $0$ otherwise. The local entropy within this trust region is
$$
H(\pi_{TR}(\cdot|s)) = -\sum_{a\in A_{TR}(s)} \pi_{TR}(a|s) \log \pi_{TR}(a|s).
$$
To ensure comparability in scale with global entropy, the local entropy is rescaled by $\log|A_{TR}(s)|/\log|A|$:
$$
L_{TRE}(s) = -\lambda \cdot \frac{\log|A_{TR}(s)|}{\log|A|} \cdot H(\pi_{TR}(\cdot|s)).
$$
If $|A_{TR}(s)| = 1$, the entropy bonus is omitted ($H = 0$).

The overall RL objective at step $t$ becomes
$$
L_{\text{total},t} = L_{\text{surr},t} + L_{TRE}(s_t),
$$
preserving the PPO surrogate and implicit KL-regularization properties but constraining entropy-driven exploration to trusted actions. This formulation ensures exploration without mass drifting into implausible or invalid regions [2602.03635].

## 3. TRE-K and TRE-P: Instantiations of the Trust Region

Two practical variants are introduced:

- **TRE-K (“top-K”):** Here, $A_{TR}(s)$ comprises the $K$ tokens with highest logits at step $s$. The entropy bonus is computed as
  $$
  L_{TRE-K}(s) = -\lambda \cdot \frac{\log K}{\log |A|} \cdot H_K(s),
  $$
  where $H_K(s)$ is the entropy of the top-$K$ re-normalized policy. The algorithm selects the $K$ largest-logit tokens, re-normalizes, computes the entropy, and applies the scaled bonus.

- **TRE-P (“top-p” or nucleus):** Here, $A_{TR}(s)$ is the smallest subset of $A$ whose cumulative softmax probability exceeds a threshold $P$. The corresponding bonus is
  $$
  L_{TRE-P}(s) = -\lambda \cdot \frac{\log |I|}{\log |A|} \cdot H_P(s),
  $$
  with $I$ the selected prefix tokens. This variant adapts the region size to the model’s confidence—expanding when uncertain (large $|I|$), shrinking when confident (small $|I|$)—yielding smoother policy confidence dynamics and more effective regulation of exploration [2602.03635].

## 4. Empirical Performance and Comparative Analysis

Empirical evaluation demonstrates the superiority of both TRE-K and TRE-P over global entropy regularization, vanilla PPO, Forking-Tokens, and covariance-based KL penalties (KL-Cov) across mathematical reasoning (MATH), combinatorial (Countdown), and preference alignment (HH) tasks. Key results from Table 1 in [2602.03635]:

| Method           | MATH Pass@1 ↑ | Countdown Pass@1 ↑ | HH Reward ↑  |
|------------------|---------------|--------------------|--------------|
| PPO (vanilla)    |     57.04%    |      64.12%        |    3.24      |
| Entropy (global) |     56.64%    |      63.20%        |    3.19      |
| Forking-Tokens   |     57.16%    |      62.82%        |    3.32      |
| KL-Cov           |     58.23%    |      66.50%        |    3.39      |
| TRE-K ($K=2$)    | **58.26%**    |      66.28%        |    3.82      |
| TRE-P ($P=0.99$) | **58.28%**    |  **66.96%**        | **3.88**     |

TRE-P specifically led to more pronounced improvements on larger models (Qwen2.5-7B), with smoother entropy regularization and better maintenance of exploratory capacity over training, as indicated by non-saturated top-token probabilities and resilience against premature convergence [2602.03635].

## 5. Algorithmic Implementation and Complexity

Integration of TRE-K or TRE-P into standard RL fine-tuning loops for LLMs is direct:

- Compute actor logits for each step in the rollout.
- Apply TRE-K (select top-$K$ logits) or TRE-P (greedily accumulate tokens until cumulative softmax mass $\geq P$).
- Re-normalize the selected logits and compute the corresponding entropy.
- Scale and add the TRE loss to the PPO surrogate.
- Backpropagate through the combined objective.

The additional computational cost is $O(|A| \log K)$ for top-$K$ and $O(|A| \log |A|)$ for top-$p$, but can be mitigated via partial-sorting and prefix-sum optimizations. The wall-clock overhead remains modest relative to model forward-pass costs [2602.03635].

## 6. Connections to Related Entropy Regularization Approaches

Restricting entropy maximization to plausible actions has convergent motivation with methods such as AEnt (“clamped entropy”) [2509.03493] and SIREN (“selective entropy regularization”) [2509.25133]. All adopt the principle of restricting the entropy bonus to a reduced, high-confidence subset of the action space:

- **AEnt** evaluates clamped entropy on a subset of top $(1-p)$-percent tokens, adaptively adjusting its entropy coefficient to maintain bounded entropy within this plausible region [2509.03493].
- **SIREN** applies a two-step masking strategy: top-p masking for output token selection and a peak-entropy mask (selecting only positions with top entropies), in combination with a self-anchored entropy penalty that stabilizes entropy drift while targeting exploration [2509.25133].

These convergent approaches highlight the consensus that global entropy regularization is fundamentally unsuited to large-scale, sparse-reward reasoning tasks, and that targeted, trust region-style entropy is required for stable and effective LLM RL fine-tuning.

## 7. Practical Considerations and Hyperparameterization

Best empirical results using TRE observed with $(K=2,\,P=0.99,\,\lambda=0.001)$. Increasing $K$ or $P$ reintroduces tail noise, while $K=1$ (i.e., minimum entropy) collapses exploration and underperforms vanilla PPO. Proper hyperparameter tuning over a small validation set is critical.

Both TRE-K and TRE-P exhibit stability compatible with PPO’s clipped surrogate, and their locality ensures bounded per-step KL divergence. This design maintains trust-region goals of controlling policy divergence while enabling high-quality exploration strictly within high-confidence neighborhoods of the prior network’s generative distribution [2602.03635].

A plausible implication is that as LLMs scale further, explicit restriction of entropy maximization to trusted action subsets is increasingly essential for effective and stable policy improvement in RL finetuning—an insight reflected by several distinct research programs converging on similar methodology.

Source: https://www.emergentmind.com/topics/restricting-entropy-maximization-to-plausible-actions-tre-k-tre-p