---
title: Top-H Decoding for Language Models
url: https://www.emergentmind.com/topics/top-h-decoding
type: topic
---

# Top-H Decoding for Language Models

Top-H decoding is a truncated sampling method for large language models that controls the trade-off between creativity and coherence by bounding the Shannon entropy of the truncated-and-renormalized distribution used for sampling. It was introduced as a method that incorporates model confidence through the full next-token distribution rather than through a single statistic such as the top-1 probability, with the stated goal of preserving coherence when the model is confident and allowing more diversity when it is uncertain [2509.02510].

## 1. Motivation and defining idea

Open-ended text generation exposes a persistent tension between diversity and logical coherence. If decoding permits many low-probability tokens, outputs can become more varied and creative, but they can also become incoherent, off-topic, or logically inconsistent. If decoding is restricted too aggressively to the highest-probability tokens, outputs may become more consistent and factual but also dull, repetitive, or brittle [2509.02510].

Top-H decoding addresses this trade-off by constraining the entropy of the actual sampling distribution after truncation and renormalization. Let the model’s next-token distribution be $p = (p_i)_{i \in \mathcal{V}}$ over vocabulary $\mathcal{V}$, and let $S \subseteq \mathcal{V}$ be the selected support. The retained mass is
$$
\Gamma_S := \sum_{i \in S} p_i,
$$
and the truncated-and-renormalized distribution is
$$
q_i =
\begin{cases}
p_i / \sum_{j \in S} p_j, & i \in S,\\
0, & i \notin S.
\end{cases}
$$
Its Shannon entropy is
$$
H(q) = -\sum_{i \in S} q_i \log q_i.
$$
The central design choice is to require
$$
H(q) \le \alpha H(p),
$$
where $\alpha \in (0,1)$ is a tunable coefficient and $H(p)$ is the entropy of the model distribution itself [2509.02510].

This formulation makes the allowed randomness proportional to the model’s uncertainty. When $H(p)$ is low, the entropy budget is tight and the decoding step tends to admit few tokens. When $H(p)$ is high, the budget relaxes and more tokens can be retained. The paper presents this as a direct mechanism for adapting creativity to model confidence [2509.02510].

A related but distinct context appears in "Foundations of Top-$k$ Decoding For Language Models" [2505.19371]. That work does not define Top-H decoding explicitly, but it suggests entropy-guided truncations as an extrapolation from a sparse Bregman decoding framework. This contrast is useful because the 2025 Top-H paper gives an explicit entropy-bounded construction, whereas the top-$k$ foundations paper treats entropy-thresholded variants as a natural extension rather than the primary object of study [2505.19371].

## 2. Optimization formulation and theoretical basis

The Top-H paper formulates truncated decoding as an entropy-constrained minimum divergence problem. Using Jensen–Shannon divergence rather than KL divergence, it defines
$$
\min_S \operatorname{JSD}(q \parallel p)
\quad \text{subject to} \quad
H(q) \le \alpha H(p).
$$
Here the constraint limits the randomness of the truncated distribution $q$ as a fraction of the model’s current uncertainty $H(p)$ [2509.02510].

The central theoretical result is an equivalence between this entropy-constrained minimum divergence problem and an entropy-constrained mass maximization problem. Under truncation and renormalization, the paper proves that $\operatorname{JSD}(q \parallel p)$ depends only on the retained mass $\Gamma_S$ and is strictly decreasing in $\Gamma_S$. Consequently,
$$
\min_S \operatorname{JSD}(q \parallel p)
\quad \text{subject to} \quad
H(q) \le \alpha H(p)
$$
is equivalent to
$$
\max_S \Gamma_S
\quad \text{subject to} \quad
H(q) \le \alpha H(p).
$$
The intuition given in the paper is that, once truncation and renormalization are fixed, any support retaining more original probability mass yields a renormalized distribution closer to $p$ in Jensen–Shannon divergence [2509.02510].

The paper further gives the closed-form dependence
$$
\operatorname{JSD}(p \parallel q)
=
\log 2
+
\frac{1}{2}
\left[
\Gamma_S \log \Gamma_S
-
(1+\Gamma_S)\log(1+\Gamma_S)
\right],
$$
with derivative
$$
\frac{d}{d\Gamma_S}\operatorname{JSD}(p \parallel q)
=
\frac{1}{2}
\left[
\log \Gamma_S - \log(1+\Gamma_S)
\right]
< 0.
$$
This makes the monotonic relationship explicit [2509.02510].

The exact entropy-constrained mass maximization problem is NP-hard, and its decision version is NP-complete. The reduction is from Cardinality-Constrained Subset Sum. The construction uses “heavy” items and many tiny “booster” items, with the entropy budget set so that any feasible subset including boosters violates the entropy bound, forcing feasible solutions to satisfy the cardinality and target-sum conditions of the source problem [2509.02510].

This hardness result is significant because it rules out a simple exact polynomial-time optimizer in the general case. It also motivates the decoding algorithm actually used in practice: a greedy approximation designed to retain as much mass as possible while respecting the entropy budget [2509.02510].

## 3. Greedy Top-H algorithm

Because exact ECMM is NP-hard, Top-H decoding is implemented via a computationally efficient greedy procedure. At each decoding step, the method sorts tokens in descending order of $p_i$ and adds them one by one while maintaining the entropy constraint on the renormalized distribution over the running support [2509.02510].

The algorithm uses the identity
$$
H(q) = \log \Gamma - \frac{h}{\Gamma},
$$
where
$$
\Gamma = \sum_{i \in S} p_i,
\qquad
h = \sum_{i \in S} p_i \log p_i.
$$
This permits $O(1)$ incremental entropy updates when a candidate token is tentatively added [2509.02510].

At a single decoding step, the procedure is:

1. Compute $H(p)$ and set $H_{\max} = \alpha H(p)$.
2. Sort tokens by descending probability.
3. Initialize $S \leftarrow \varnothing$, $\Gamma \leftarrow 0$, $h \leftarrow 0$.
4. For each token in sorted order, compute
   $$
   \Gamma' = \Gamma + p_i,
   \qquad
   h' = h + p_i \log p_i,
   \qquad
   H(q') = \log \Gamma' - h'/\Gamma'.
   $$
5. If $H(q') \le H_{\max}$, accept the token and continue; otherwise stop.
6. Sample from the renormalized distribution on the final support $S$ [2509.02510].

The complexity is dominated by sorting, giving $O(n \log n)$ time per decoding step, with the incremental pass taking $O(n)$. The paper reports negligible per-token runtime overhead relative to min-$p$ and top-$p$ in tested setups, with less than $1\%$ difference [2509.02510].

The paper also gives a termination guarantee. Under greedy selection in descending $p_i$, adding a token strictly increases $H(q)$ for the renormalized truncated distribution. Since the budget satisfies $H(q) \le \alpha H(p)$ with $\alpha < 1$, the procedure must stop before including all tokens [2509.02510].

An illustrative example in the paper uses
$$
p = [0.40, 0.25, 0.12, 0.08, 0.06, 0.04, 0.03, 0.02],
$$
with $H(p) \approx 1.65$ nats and $\alpha = 0.4$, so $H_{\max} \approx 0.66$. After the first token, $H(q)=0$, which is feasible; after the second token, the renormalized distribution becomes approximately $[0.615, 0.385]$ and $H(q)\approx 0.667$, which exceeds the budget. The greedy Top-H support is therefore the singleton containing only the top token [2509.02510].

## 4. Relation to other decoding strategies

Top-H decoding is best understood against the behavior of existing truncated sampling methods. The Top-H paper contrasts it with temperature scaling, top-$k$, top-$p$, and min-$p$ [2509.02510].

| Method | Constraint or mechanism | Limitation highlighted in the paper |
|---|---|---|
| Temperature scaling | Multiplies logits by $1/T$ | At high $T$ the distribution flattens indiscriminately |
| Top-$k$ | Keeps the $k$ highest-probability tokens | Insensitive to context and model confidence |
| Top-$p$ | Chooses the smallest set whose cumulative mass exceeds $p$ | May admit many very low-probability tokens when the distribution is flat |
| Min-$p$ | Thresholds using a fraction of the top-1 probability | Uses only the maximum probability and ignores tail structure |
| Top-H | Constrains $H(q)$ with budget $\alpha H(p)$ | Uses both $H(p)$ and the entropy of the retained distribution |

Temperature scaling changes the sharpness of $p$ before any truncation, but it imposes no direct bound on the randomness of the distribution actually sampled. Top-$k$ constrains support size only, irrespective of probability geometry. Top-$p$ constrains retained mass, not post-renormalization entropy. Min-$p$ uses the top probability as a proxy for confidence, which the Top-H paper argues ignores how the rest of the probability mass is distributed [2509.02510].

Top-H differs in that it constrains the entropy of the renormalized distribution $q$ itself. The paper characterizes this as a direct, global control knob on randomness rather than a local per-token threshold or a cumulative-mass rule [2509.02510].

The top-$k$ foundations paper provides a complementary perspective. It derives top-$k$ as the KL special case of a broader family of sparse Bregman decoders with $\ell_0$ regularization, where optimal support selection is greedy and the restricted cost is discretely convex in $k$ [2505.19371]. That framework does not define Top-H explicitly, but it does suggest entropy-thresholded variants as an extrapolation. A plausible implication is that Top-H can be viewed as part of a broader movement from fixed-form truncation heuristics toward decoding rules grounded in explicit optimization criteria [2505.19371].

The Top-H paper also gives a simple quantitative intuition. For a perfectly uniform distribution over $m$ tokens, $H(q)=\log m$. Under a Top-H budget $H_{\max}$, the allowed set size is the largest $m$ such that $\log m \le H_{\max}$, equivalently $m \le \exp(H_{\max})$. For peaked distributions, the entropy grows more slowly with support size, so the selected support typically remains smaller [2509.02510].

## 5. Empirical evaluation and practical deployment

The empirical study in the Top-H paper evaluates LLaMA3.1–8B–Instruct, Qwen2.5–3B, and Phi-3–Mini–4K–Instruct, with additional validation on LLaMA3.3–70B–Instruct. Baselines are top-$k$, top-$p$, min-$p$, and $\eta$-sampling, using temperatures $T \in \{1.0, 1.5, 2.0\}$ with hyperparameters $\text{min\_p}=0.1$, $\text{top\_p}=0.9$, and $\eta=0.0002$. Benchmarks include Alpaca-Eval, MT-Bench, GSM8K, and GPQA, with GPT-4o used as an LLM-as-judge and implementation via HuggingFace and lm-eval-harness [2509.02510].

On creative writing and dialogue, the paper reports that Top-H outperforms min-$p$ and top-$p$ across models and temperatures on Alpaca-Eval length-controlled win rate. For LLaMA3.1–8B, it improves win rate by up to $17.11\%$ over min-$p$. On MT-Bench judge scores from $1$ to $10$, it gives higher or comparable scores, especially at higher temperatures, which the paper interprets as reflecting a stronger creativity–coherence balance [2509.02510].

On QA and reasoning tasks, the paper emphasizes robustness as temperature increases. On GSM8K with LLaMA3.1–8B at $T=2$, Top-H improves accuracy by $25.63\%$ over min-$p$, reported as $39.35\%$ versus $13.72\%$. On GPQA, it is described as consistently competitive and often superior across temperatures [2509.02510].

The abstract summarizes the empirical picture by stating that Top-H outperforms the state-of-the-art alternative of min-$p$ sampling by up to $25.63\%$ on creative writing benchmarks while maintaining robustness on GPQA, GSM8K, and MT-Bench, and that LLM-as-judge evaluation confirms coherent outputs even at higher temperatures [2509.02510].

Practical guidance in the paper centers on the entropy coefficient $\alpha$, which is the sole hyperparameter. The reported default is $\alpha=0.4$, selected via a small development set using an LLM-as-judge for creativity and coherence. Larger $\alpha$ increases allowable entropy in $q$ and tends to increase creativity at some risk to coherence; smaller $\alpha$ tightens the bound and favors coherence at some cost to diversity [2509.02510].

The recommended placement is after temperature scaling and other logit processing: compute post-softmax probabilities, evaluate $H(p)$, run the greedy selection, renormalize on the retained support, and sample. The paper notes that stable log-sum-exp expressions may be used if needed, and that prefiltering to a top-$N$ list such as $N=100$ to $1000$ can improve speed while keeping memory small [2509.02510].

## 6. Limitations, edge cases, and broader significance

The Top-H paper is explicit that the exact optimization problem is NP-hard and that the greedy method does not come with a formal constant-factor approximation guarantee. Nevertheless, exhaustive search over top-15 candidates per step reportedly shows that the greedy solution closely matches the true optimal ECMM solution, with the ratio $\Gamma_S^{\text{greedy}} / \Gamma_S^{\text{opt}}$ near $1.0$ on average [2509.02510].

Several edge cases are identified. For extremely flat distributions, even a large $H(p)$ may yield a modest entropy budget if $\alpha$ is small, so Top-H may still include only a few tokens. For extremely peaked distributions, the selected support may collapse to a very small set, often only the top token. The paper treats this behavior as desirable for coherence, but it can reduce variety unless $\alpha$ is increased [2509.02510].

The method also inherits the calibration properties and biases of the base model because it depends only on the next-token distribution. It does not directly solve repetition loops or long-range discourse failures, and the paper recommends combining it with repetition penalties or related mechanisms if needed [2509.02510].

A further implementation subtlety concerns prefiltering. If decoding is restricted in advance to a small top-$N$ candidate set for speed, then the optimization is only over that truncated candidate pool rather than over the full vocabulary. The paper notes that, empirically, restricting to top-100 or top-1000 tends to work well [2509.02510].

In the broader decoding literature, Top-H occupies a distinct position. It is neither a support-cardinality rule like top-$k$ nor a cumulative-mass rule like top-$p$, and it differs from min-$p$ by using the full distributional shape rather than the maximum probability alone. A plausible implication is that its contribution is methodological as much as empirical: it reframes decoding as bounded-entropy truncation with an explicit optimization objective and a dynamic uncertainty budget [2509.02510].

The relationship to the top-$k$ theory paper clarifies this further. "Foundations of Top-$k$ Decoding For Language Models" derives top-$k$ from Bregman geometry and $\ell_0$ regularization, and suggests entropy-thresholded variants as a natural extension, but does not present Top-H as a finished algorithm [2505.19371]. By contrast, Top-H decoding is presented as a fully specified method with an entropy-constrained objective, an NP-hardness result, a greedy solver, empirical evaluation across creative writing and reasoning benchmarks, and a reference implementation [2509.02510].

Source: https://www.emergentmind.com/topics/top-h-decoding