Top-H Decoding for Language Models

• Top-H decoding is a truncated sampling method that controls the trade-off between creativity and coherence by bounding the Shannon entropy of the next-token distribution.
• It formulates decoding as an entropy-constrained optimization problem, equating maximum retained probability mass with minimized divergence from the full model distribution.
• A greedy algorithm approximates the NP-hard optimal solution, achieving significant empirical gains in creative writing and reasoning benchmarks.

Top-H decoding is a truncated sampling method for LLMs that controls the trade-off between creativity and coherence by bounding the Shannon entropy of the truncated-and-renormalized distribution used for sampling. It was introduced as a method that incorporates model confidence through the full next-token distribution rather than through a single statistic such as the top-1 probability, with the stated goal of preserving coherence when the model is confident and allowing more diversity when it is uncertain (Potraghloo et al., 2 Sep 2025).

1. Motivation and defining idea

Open-ended text generation exposes a persistent tension between diversity and logical coherence. If decoding permits many low-probability tokens, outputs can become more varied and creative, but they can also become incoherent, off-topic, or logically inconsistent. If decoding is restricted too aggressively to the highest-probability tokens, outputs may become more consistent and factual but also dull, repetitive, or brittle (Potraghloo et al., 2 Sep 2025).

Top-H decoding addresses this trade-off by constraining the entropy of the actual sampling distribution after truncation and renormalization. Let the model’s next-token distribution be $p = (p_i)_{i \in \mathcal{V}}$ over vocabulary $\mathcal{V}$, and let $S \subseteq \mathcal{V}$ be the selected support. The retained mass is

$\Gamma_S := \sum_{i \in S} p_i,$

and the truncated-and-renormalized distribution is

$$q_i = \begin{cases} p_i / \sum_{j \in S} p_j, & i \in S,\ 0, & i \notin S. \end{cases}$$

Its Shannon entropy is

$H(q) = -\sum_{i \in S} q_i \log q_i.$

The central design choice is to require

$H(q) \le \alpha H(p),$

where $\alpha \in (0,1)$ is a tunable coefficient and $H(p)$ is the entropy of the model distribution itself (Potraghloo et al., 2 Sep 2025).

This formulation makes the allowed randomness proportional to the model’s uncertainty. When $H(p)$ is low, the entropy budget is tight and the decoding step tends to admit few tokens. When $\mathcal{V}$0 is high, the budget relaxes and more tokens can be retained. The paper presents this as a direct mechanism for adapting creativity to model confidence (Potraghloo et al., 2 Sep 2025).

A related but distinct context appears in "Foundations of Top-$\mathcal{V}$1 Decoding For LLMs" (Noarov et al., 25 May 2025). That work does not define Top-H decoding explicitly, but it suggests entropy-guided truncations as an extrapolation from a sparse Bregman decoding framework. This contrast is useful because the 2025 Top-H paper gives an explicit entropy-bounded construction, whereas the top-$\mathcal{V}$2 foundations paper treats entropy-thresholded variants as a natural extension rather than the primary object of study (Noarov et al., 25 May 2025).

2. Optimization formulation and theoretical basis

The Top-H paper formulates truncated decoding as an entropy-constrained minimum divergence problem. Using Jensen–Shannon divergence rather than KL divergence, it defines

$\mathcal{V}$3

Here the constraint limits the randomness of the truncated distribution $\mathcal{V}$4 as a fraction of the model’s current uncertainty $\mathcal{V}$5 (Potraghloo et al., 2 Sep 2025).

The central theoretical result is an equivalence between this entropy-constrained minimum divergence problem and an entropy-constrained mass maximization problem. Under truncation and renormalization, the paper proves that $\mathcal{V}$6 depends only on the retained mass $\mathcal{V}$7 and is strictly decreasing in $\mathcal{V}$8. Consequently,

$\mathcal{V}$9

is equivalent to

$S \subseteq \mathcal{V}$0

The intuition given in the paper is that, once truncation and renormalization are fixed, any support retaining more original probability mass yields a renormalized distribution closer to $S \subseteq \mathcal{V}$1 in Jensen–Shannon divergence (Potraghloo et al., 2 Sep 2025).

The paper further gives the closed-form dependence

$S \subseteq \mathcal{V}$2

with derivative

$S \subseteq \mathcal{V}$3

This makes the monotonic relationship explicit (Potraghloo et al., 2 Sep 2025).

The exact entropy-constrained mass maximization problem is NP-hard, and its decision version is NP-complete. The reduction is from Cardinality-Constrained Subset Sum. The construction uses “heavy” items and many tiny “booster” items, with the entropy budget set so that any feasible subset including boosters violates the entropy bound, forcing feasible solutions to satisfy the cardinality and target-sum conditions of the source problem (Potraghloo et al., 2 Sep 2025).

This hardness result is significant because it rules out a simple exact polynomial-time optimizer in the general case. It also motivates the decoding algorithm actually used in practice: a greedy approximation designed to retain as much mass as possible while respecting the entropy budget (Potraghloo et al., 2 Sep 2025).

3. Greedy Top-H algorithm

Because exact ECMM is NP-hard, Top-H decoding is implemented via a computationally efficient greedy procedure. At each decoding step, the method sorts tokens in descending order of $S \subseteq \mathcal{V}$4 and adds them one by one while maintaining the entropy constraint on the renormalized distribution over the running support (Potraghloo et al., 2 Sep 2025).

The algorithm uses the identity

$S \subseteq \mathcal{V}$5

where

$S \subseteq \mathcal{V}$6

This permits $S \subseteq \mathcal{V}$7 incremental entropy updates when a candidate token is tentatively added (Potraghloo et al., 2 Sep 2025).

At a single decoding step, the procedure is:

1. Compute $S \subseteq \mathcal{V}$8 and set $S \subseteq \mathcal{V}$9.
2. Sort tokens by descending probability.
3. Initialize $\Gamma_S := \sum_{i \in S} p_i,$0, $\Gamma_S := \sum_{i \in S} p_i,$1, $\Gamma_S := \sum_{i \in S} p_i,$2.
4. For each token in sorted order, compute

$\Gamma_S := \sum_{i \in S} p_i,$3

1. If $\Gamma_S := \sum_{i \in S} p_i,$4, accept the token and continue; otherwise stop.
2. Sample from the renormalized distribution on the final support $\Gamma_S := \sum_{i \in S} p_i,$5 (Potraghloo et al., 2 Sep 2025).

The complexity is dominated by sorting, giving $\Gamma_S := \sum_{i \in S} p_i,$6 time per decoding step, with the incremental pass taking $\Gamma_S := \sum_{i \in S} p_i,$7. The paper reports negligible per-token runtime overhead relative to min-$\Gamma_S := \sum_{i \in S} p_i,$8 and top-$\Gamma_S := \sum_{i \in S} p_i,$9 in tested setups, with less than $$q_i = \begin{cases} p_i / \sum_{j \in S} p_j, & i \in S,\ 0, & i \notin S. \end{cases}$$0 difference (Potraghloo et al., 2 Sep 2025).

The paper also gives a termination guarantee. Under greedy selection in descending $$q_i = \begin{cases} p_i / \sum_{j \in S} p_j, & i \in S,\ 0, & i \notin S. \end{cases}$$1, adding a token strictly increases $$q_i = \begin{cases} p_i / \sum_{j \in S} p_j, & i \in S,\ 0, & i \notin S. \end{cases}$$2 for the renormalized truncated distribution. Since the budget satisfies $$q_i = \begin{cases} p_i / \sum_{j \in S} p_j, & i \in S,\ 0, & i \notin S. \end{cases}$$3 with $$q_i = \begin{cases} p_i / \sum_{j \in S} p_j, & i \in S,\ 0, & i \notin S. \end{cases}$$4, the procedure must stop before including all tokens (Potraghloo et al., 2 Sep 2025).

An illustrative example in the paper uses

$$q_i = \begin{cases} p_i / \sum_{j \in S} p_j, & i \in S,\ 0, & i \notin S. \end{cases}$$5

with $$q_i = \begin{cases} p_i / \sum_{j \in S} p_j, & i \in S,\ 0, & i \notin S. \end{cases}$$6 nats and $$q_i = \begin{cases} p_i / \sum_{j \in S} p_j, & i \in S,\ 0, & i \notin S. \end{cases}$$7, so $$q_i = \begin{cases} p_i / \sum_{j \in S} p_j, & i \in S,\ 0, & i \notin S. \end{cases}$$8. After the first token, $$q_i = \begin{cases} p_i / \sum_{j \in S} p_j, & i \in S,\ 0, & i \notin S. \end{cases}$$9, which is feasible; after the second token, the renormalized distribution becomes approximately $H(q) = -\sum_{i \in S} q_i \log q_i.$0 and $H(q) = -\sum_{i \in S} q_i \log q_i.$1, which exceeds the budget. The greedy Top-H support is therefore the singleton containing only the top token (Potraghloo et al., 2 Sep 2025).

4. Relation to other decoding strategies

Top-H decoding is best understood against the behavior of existing truncated sampling methods. The Top-H paper contrasts it with temperature scaling, top-$H(q) = -\sum_{i \in S} q_i \log q_i.$2, top-$H(q) = -\sum_{i \in S} q_i \log q_i.$3, and min-$H(q) = -\sum_{i \in S} q_i \log q_i.$4 (Potraghloo et al., 2 Sep 2025).

Method	Constraint or mechanism	Limitation highlighted in the paper
Temperature scaling	Multiplies logits by $H(q) = -\sum_{i \in S} q_i \log q_i.$5	At high $H(q) = -\sum_{i \in S} q_i \log q_i.$6 the distribution flattens indiscriminately
Top-$H(q) = -\sum_{i \in S} q_i \log q_i.$7	Keeps the $H(q) = -\sum_{i \in S} q_i \log q_i.$8 highest-probability tokens	Insensitive to context and model confidence
Top-$H(q) = -\sum_{i \in S} q_i \log q_i.$9	Chooses the smallest set whose cumulative mass exceeds $H(q) \le \alpha H(p),$0	May admit many very low-probability tokens when the distribution is flat
Min-$H(q) \le \alpha H(p),$1	Thresholds using a fraction of the top-1 probability	Uses only the maximum probability and ignores tail structure
Top-H	Constrains $H(q) \le \alpha H(p),$2 with budget $H(q) \le \alpha H(p),$3	Uses both $H(q) \le \alpha H(p),$4 and the entropy of the retained distribution

Temperature scaling changes the sharpness of $H(q) \le \alpha H(p),$5 before any truncation, but it imposes no direct bound on the randomness of the distribution actually sampled. Top-$H(q) \le \alpha H(p),$6 constrains support size only, irrespective of probability geometry. Top-$H(q) \le \alpha H(p),$7 constrains retained mass, not post-renormalization entropy. Min-$H(q) \le \alpha H(p),$8 uses the top probability as a proxy for confidence, which the Top-H paper argues ignores how the rest of the probability mass is distributed (Potraghloo et al., 2 Sep 2025).

Top-H differs in that it constrains the entropy of the renormalized distribution $H(q) \le \alpha H(p),$9 itself. The paper characterizes this as a direct, global control knob on randomness rather than a local per-token threshold or a cumulative-mass rule (Potraghloo et al., 2 Sep 2025).

The top-$\alpha \in (0,1)$0 foundations paper provides a complementary perspective. It derives top-$\alpha \in (0,1)$1 as the KL special case of a broader family of sparse Bregman decoders with $\alpha \in (0,1)$2 regularization, where optimal support selection is greedy and the restricted cost is discretely convex in $\alpha \in (0,1)$3 (Noarov et al., 25 May 2025). That framework does not define Top-H explicitly, but it does suggest entropy-thresholded variants as an extrapolation. A plausible implication is that Top-H can be viewed as part of a broader movement from fixed-form truncation heuristics toward decoding rules grounded in explicit optimization criteria (Noarov et al., 25 May 2025).

The Top-H paper also gives a simple quantitative intuition. For a perfectly uniform distribution over $\alpha \in (0,1)$4 tokens, $\alpha \in (0,1)$5. Under a Top-H budget $\alpha \in (0,1)$6, the allowed set size is the largest $\alpha \in (0,1)$7 such that $\alpha \in (0,1)$8, equivalently $\alpha \in (0,1)$9. For peaked distributions, the entropy grows more slowly with support size, so the selected support typically remains smaller (Potraghloo et al., 2 Sep 2025).

5. Empirical evaluation and practical deployment

The empirical study in the Top-H paper evaluates LLaMA3.1–8B–Instruct, Qwen2.5–3B, and Phi-3–Mini–4K–Instruct, with additional validation on LLaMA3.3–70B–Instruct. Baselines are top-$H(p)$0, top-$H(p)$1, min-$H(p)$2, and $H(p)$3-sampling, using temperatures $H(p)$4 with hyperparameters $H(p)$5, $H(p)$6, and $H(p)$7. Benchmarks include Alpaca-Eval, MT-Bench, GSM8K, and GPQA, with GPT-4o used as an LLM-as-judge and implementation via HuggingFace and lm-eval-harness (Potraghloo et al., 2 Sep 2025).

On creative writing and dialogue, the paper reports that Top-H outperforms min-$H(p)$8 and top-$H(p)$9 across models and temperatures on Alpaca-Eval length-controlled win rate. For LLaMA3.1–8B, it improves win rate by up to $H(p)$0 over min-$H(p)$1. On MT-Bench judge scores from $H(p)$2 to $H(p)$3, it gives higher or comparable scores, especially at higher temperatures, which the paper interprets as reflecting a stronger creativity–coherence balance (Potraghloo et al., 2 Sep 2025).

On QA and reasoning tasks, the paper emphasizes robustness as temperature increases. On GSM8K with LLaMA3.1–8B at $H(p)$4, Top-H improves accuracy by $H(p)$5 over min-$H(p)$6, reported as $H(p)$7 versus $H(p)$8. On GPQA, it is described as consistently competitive and often superior across temperatures (Potraghloo et al., 2 Sep 2025).

The abstract summarizes the empirical picture by stating that Top-H outperforms the state-of-the-art alternative of min-$H(p)$9 sampling by up to $\mathcal{V}$00 on creative writing benchmarks while maintaining robustness on GPQA, GSM8K, and MT-Bench, and that LLM-as-judge evaluation confirms coherent outputs even at higher temperatures (Potraghloo et al., 2 Sep 2025).

Practical guidance in the paper centers on the entropy coefficient $\mathcal{V}$01, which is the sole hyperparameter. The reported default is $\mathcal{V}$02, selected via a small development set using an LLM-as-judge for creativity and coherence. Larger $\mathcal{V}$03 increases allowable entropy in $\mathcal{V}$04 and tends to increase creativity at some risk to coherence; smaller $\mathcal{V}$05 tightens the bound and favors coherence at some cost to diversity (Potraghloo et al., 2 Sep 2025).

The recommended placement is after temperature scaling and other logit processing: compute post-softmax probabilities, evaluate $\mathcal{V}$06, run the greedy selection, renormalize on the retained support, and sample. The paper notes that stable log-sum-exp expressions may be used if needed, and that prefiltering to a top-$\mathcal{V}$07 list such as $\mathcal{V}$08 to $\mathcal{V}$09 can improve speed while keeping memory small (Potraghloo et al., 2 Sep 2025).

6. Limitations, edge cases, and broader significance

The Top-H paper is explicit that the exact optimization problem is NP-hard and that the greedy method does not come with a formal constant-factor approximation guarantee. Nevertheless, exhaustive search over top-15 candidates per step reportedly shows that the greedy solution closely matches the true optimal ECMM solution, with the ratio $\mathcal{V}$10 near $\mathcal{V}$11 on average (Potraghloo et al., 2 Sep 2025).

Several edge cases are identified. For extremely flat distributions, even a large $\mathcal{V}$12 may yield a modest entropy budget if $\mathcal{V}$13 is small, so Top-H may still include only a few tokens. For extremely peaked distributions, the selected support may collapse to a very small set, often only the top token. The paper treats this behavior as desirable for coherence, but it can reduce variety unless $\mathcal{V}$14 is increased (Potraghloo et al., 2 Sep 2025).

The method also inherits the calibration properties and biases of the base model because it depends only on the next-token distribution. It does not directly solve repetition loops or long-range discourse failures, and the paper recommends combining it with repetition penalties or related mechanisms if needed (Potraghloo et al., 2 Sep 2025).

A further implementation subtlety concerns prefiltering. If decoding is restricted in advance to a small top-$\mathcal{V}$15 candidate set for speed, then the optimization is only over that truncated candidate pool rather than over the full vocabulary. The paper notes that, empirically, restricting to top-100 or top-1000 tends to work well (Potraghloo et al., 2 Sep 2025).

In the broader decoding literature, Top-H occupies a distinct position. It is neither a support-cardinality rule like top-$\mathcal{V}$16 nor a cumulative-mass rule like top-$\mathcal{V}$17, and it differs from min-$\mathcal{V}$18 by using the full distributional shape rather than the maximum probability alone. A plausible implication is that its contribution is methodological as much as empirical: it reframes decoding as bounded-entropy truncation with an explicit optimization objective and a dynamic uncertainty budget (Potraghloo et al., 2 Sep 2025).

The relationship to the top-$\mathcal{V}$19 theory paper clarifies this further. "Foundations of Top-$\mathcal{V}$20 Decoding For LLMs" derives top-$\mathcal{V}$21 from Bregman geometry and $\mathcal{V}$22 regularization, and suggests entropy-thresholded variants as a natural extension, but does not present Top-H as a finished algorithm (Noarov et al., 25 May 2025). By contrast, Top-H decoding is presented as a fully specified method with an entropy-constrained objective, an NP-hardness result, a greedy solver, empirical evaluation across creative writing and reasoning benchmarks, and a reference implementation (Potraghloo et al., 2 Sep 2025).