Cross-Entropy: Probabilistic Information Measure

Updated 24 October 2026

Cross-entropy is an information-theoretic measure that evaluates how one probability distribution assigns probabilities to outcomes of another target distribution. It is widely used in fields such as machine learning and statistics for probabilistic prediction, optimization, and rare-event simulation.
In machine learning, categorical cross-entropy for classification problems uses the log probabilities from a model to evaluate predicted class probabilities against actual target labels through a loss function. It assesses model accuracy by comparing predicted distributions to the true distributions.
Cross-entropy method (CEM) is used for finding the rare good solutions in the optimization domain to perform rare event simulation and optimization processes which includes mixtures and semiparametric extensions to handle complex problems in multi-asset outperformance and structured prediction scenarios.

Cross-entropy is an information-theoretic functional that measures the expected negative logarithm of probabilities assigned by a model distribution to outcomes generated by a target distribution. For discrete distributions $p$ and $q$ on a common support $\mathcal X$ ,

$H(p,q)=-\sum_{x\in\mathcal X}p(x)\log q(x).$

It decomposes as

$H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),$

where $H(p)$ is Shannon entropy and $D_{\mathrm{KL}}(p\|q)$ is Kullback–Leibler divergence. Because $H(p)$ is independent of $q$ , minimizing cross-entropy is equivalent to minimizing the forward KL divergence and, for empirical data, to maximizing likelihood. The term denotes several related but distinct constructions: a statistical loss for probabilistic prediction, a KL-projection principle in rare-event simulation and stochastic optimization, a directed predictive discrepancy between time-series segments, and extensions to structured, quantum, Rényi, persistent, and unlearning settings.

1. Mathematical foundations and statistical interpretation

For a finite distribution $p$ , Shannon entropy is

$q$ 0

whereas cross-entropy evaluates the coding or predictive performance of $q$ 1 under outcomes distributed according to $q$ 2:

$q$ 3

The decomposition

$q$ 4

shows that cross-entropy contains an irreducible target-entropy term and a model-mismatch term. Cross-entropy is asymmetric in its arguments because, in general,

$q$ 5

The first argument supplies the expectation; the second supplies the probabilities evaluated inside the logarithm. This orientation is consequential in applications such as active information gathering, where expected cross-entropy uses the current belief as the first argument and a possible posterior as the second. The resulting objective is equivalent to the reverse KL direction from the usual expected-entropy criterion (Kulick et al., 2014).

For independent observations $q$ 6 and a parametric model $q$ 7, the empirical distribution $q$ 8 gives

$q$ 9

Thus,

$\mathcal X$ 0

This establishes the standard equivalence between cross-entropy minimization and maximum-likelihood estimation (Shangnan et al., 2021).

The base of the logarithm determines the unit: base $\mathcal X$ 1 gives bits and the natural logarithm gives nats. Cross-entropy is finite only when the model assigns nonzero probability to every outcome having positive probability under the target distribution. In the continuous case, sums are replaced by integrals, with the same expected-negative-log-density interpretation.

Categorical classification

For a categorical target with one-hot label $\mathcal X$ 2 and model probabilities $\mathcal X$ 3, categorical cross-entropy is

$\mathcal X$ 4

If logits are $\mathcal X$ 5, the softmax model is

$\mathcal X$ 6

The corresponding loss is the multiclass logistic loss,

$\mathcal X$ 7

with gradient

$\mathcal X$ 8

Every class contributes through the normalization term, including incorrect classes with high predicted probability. Standard cross-entropy is a proper scoring rule: under the usual support and realizability conditions, expected loss is minimized by predicting the true conditional distribution (Lucena, 2022).

For autoregressive models, the sequence loss is the sum of token-level negative log-likelihoods,

$\mathcal X$ 9

usually reported as the average over positions. Minimizing it is equivalent to maximizing the likelihood of the observed sequence (Garcia-Valencia, 2020).

2. Cross-entropy in Bayesian updating and information acquisition

Cross-entropy minimization is often proposed for probabilistic information that constrains a distribution without specifying an ordinary event on which to condition. Given a prior $H(p,q)=-\sum_{x\in\mathcal X}p(x)\log q(x).$ 0 and candidate posterior $H(p,q)=-\sum_{x\in\mathcal X}p(x)\log q(x).$ 1, the update minimizes

$H(p,q)=-\sum_{x\in\mathcal X}p(x)\log q(x).$ 2

subject to normalization, nonnegativity, and the received probabilistic constraints. This generalizes ordinary conditioning and Jeffrey conditioning, but its appropriateness depends on the interpretation of the information (Grove et al., 2013).

The Judy Benjamin problem

In the Judy Benjamin problem, the message imposes a conditional constraint such as

$H(p,q)=-\sum_{x\in\mathcal X}p(x)\log q(x).$ 3

Starting from a uniform prior over four quadrants, minimum cross-entropy preserves symmetry within the Blue region but adjusts the total Red probability. If $H(p,q)=-\sum_{x\in\mathcal X}p(x)\log q(x).$ 4 is binary entropy,

$H(p,q)=-\sum_{x\in\mathcal X}p(x)\log q(x).$ 5

the cross-entropy solution has

$H(p,q)=-\sum_{x\in\mathcal X}p(x)\log q(x).$ 6

For $H(p,q)=-\sum_{x\in\mathcal X}p(x)\log q(x).$ 7, the supplied derivation gives approximately

$H(p,q)=-\sum_{x\in\mathcal X}p(x)\log q(x).$ 8

More generally, for every $H(p,q)=-\sum_{x\in\mathcal X}p(x)\log q(x).$ 9, cross-entropy increases the probability of Blue territory above $H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),$ 0. Grove and Halpern regard this as potentially unsatisfactory because the announced Red-territory odds appear to concern only the relative probabilities of $H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),$ 1 and $H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),$ 2.

Their alternative is to interpret the message as a factual report about Headquarters’ beliefs rather than as a constraint on Judy’s own beliefs. They enlarge the state space to include possible Headquarters distributions and then apply ordinary conditioning. Under a uniform prior over the chosen parameterization, a truthful communication protocol, and a trust assumption, the conditionalization solution preserves

$H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),$ 3

while imposing

$H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),$ 4

The example distinguishes minimum cross-entropy from ordinary conditioning and from maximum entropy. Minimum cross-entropy preserves an existing prior as far as possible subject to constraints; maximum entropy selects the highest-entropy distribution when no informative prior is retained. They coincide when the prior is uniform because

$H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),$ 5

for a uniform $H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),$ 6.

Information-gathering criteria

In sequential experimental design, expected posterior-entropy minimization is equivalent to maximizing

$H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),$ 7

The paper proposes MaxCE, which instead maximizes expected cross-entropy between the current belief and the updated posterior:

$H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),$ 8

The reverse orientation favors observations capable of substantially revising the current belief, including observations that temporarily increase posterior entropy. Expected entropy reduction can become trapped in a low-entropy but incorrect belief because queries that challenge the current hypothesis may initially increase uncertainty. MaxCE is therefore designed to challenge refutable beliefs rather than merely reinforce concentration (Kulick et al., 2014).

MaxCE is not universally optimal for prediction. It can select queries that discriminate among hypotheses without substantially reducing predictive error. The paper combines it with uncertainty sampling through a weighted mixture; in the reported experiments, the mixture performed best for predictive accuracy and regression error, while MaxCE performed best for hypothesis discrimination.

3. The Cross-Entropy Method for rare events and optimization

The Cross-Entropy Method, or CE method, is an adaptive importance-sampling and population-based optimization procedure. Its original purpose was rare-event simulation; optimization is obtained by treating unusually good solutions as a rare event [0609461].

For a random variable $H(p,q)=H(p)+D_{\mathrm{KL}}(p\|q),$ 9 with density $H(p)$ 0 and nonnegative payoff $H(p)$ 1, importance sampling uses an alternative density $H(p)$ 2:

$H(p)$ 3

The formal zero-variance density is

$H(p)$ 4

Because $H(p)$ 5 depends on the unknown $H(p)$ 6, CE selects a tractable $H(p)$ 7 by minimizing

$H(p)$ 8

equivalently maximizing

$H(p)$ 9

Replacing the expectation with a sample average yields a weighted maximum-likelihood update (Wang et al., 2013).

Optimization reformulation

For minimization of $D_{\mathrm{KL}}(p\|q)$ 0, the CE method considers the rare event

$D_{\mathrm{KL}}(p\|q)$ 1

where $D_{\mathrm{KL}}(p\|q)$ 2 approaches the unknown optimum from above. At each iteration it:

samples candidate solutions from a parameterized distribution;
evaluates the objective;
selects an elite fraction with the smallest objective values;
fits the sampling distribution by maximum likelihood on the elite set;
optionally smooths the parameter update;
retains the best solution found;
stops according to a distribution-degeneracy, iteration, improvement, or probability criterion.

For a Gaussian sampling model with elite set $D_{\mathrm{KL}}(p\|q)$ 3, the updates are

$D_{\mathrm{KL}}(p\|q)$ 4

and

$D_{\mathrm{KL}}(p\|q)$ 5

Parameter smoothing takes the form

$D_{\mathrm{KL}}(p\|q)$ 6

Smoothing reduces premature collapse, particularly for categorical probabilities that might otherwise reach exactly $D_{\mathrm{KL}}(p\|q)$ 7 or $1 (Benham et al., 2015).

CE supports continuous, discrete, mixed, and constrained optimization. Independent normal models, Bernoulli models, and categorical models use elite-sample maximum-likelihood estimates. Linear constraints can be handled through truncated-normal sampling using acceptance-rejection or Gibbs sampling; general constraints can be incorporated through penalty functions. The CEoptim package implements these procedures in R (Benham et al., 2015).

Distributional limitations and extensions

A single exponential tilt can be inadequate when the important region is nonconvex or consists of separated components. A single Gaussian or exponential family may concentrate on one rare-event region while neglecting another. Mixture CE addresses this by using

$D_{\mathrm{KL}}(p\|q)$8

The mixture update combines importance weighting with EM responsibilities. The weights and component parameters are fitted to elite samples, allowing distinct components to cover separated regions. The method was applied to rainbow-option pricing, where mixture components substantially reduced empirical variance in multi-asset outperformance and pyramid rainbow payoffs (Wang et al., 2013).

A semiparametric extension replaces a finite-dimensional family with all independent product densities,

$D_{\mathrm{KL}}(p\|q)$9

The optimal product approximation to the zero-variance density is the product of its marginal densities:

$H(p)$0

The marginals are estimated from MCMC samples of the rare-event conditional distribution, using Rao–Blackwellized conditional densities. Under idealized assumptions, the method is logarithmically efficient for broad light-tailed, subexponential, Weibull, and Pareto classes (Botev et al., 2013).

CE variants have also used Gaussian-process surrogates, Gaussian-mixture search distributions, and evaluation scheduling. Surrogate CE reuses every expensive objective evaluation to train a Gaussian process, generates additional inexpensive surrogate evaluations, and performs local surrogate searches around true elites. Gaussian-mixture CE preserves multiple modes through EM. In the reported multimodal sierra objective, surrogate CE obtained better objective values and distances than standard CE using the same number of true evaluations, although at substantially higher runtime (Moss, 2020).

4. Differentiable, structured, and specialized variants

Differentiable Cross-Entropy Method

Standard CE is a zeroth-order optimizer with a nondifferentiable elite-selection operation. For candidates $H(p)$1 and objective values $H(p)$2, hard elite indicators are

$H(p)$3

where $H(p)$4 is the $H(p)$5th-smallest value. The indicator and top-$H(p)$6 operation are piecewise constant and provide no useful ordinary gradient.

The Differentiable Cross-Entropy Method, DCEM, replaces hard indicators with differentiable weights $H(p)$7 constrained to the Limited Multi-Label polytope,

$H(p)$8

The weights are produced by an entropy-smoothed projection with temperature $H(p)$9:

$q$0

The weighted maximum-likelihood update is

$q$1

For a diagonal Gaussian,

$q$2

and

$q$3

As $q$4, the soft weights converge to hard top-$q$5 selection when objective values are distinct. Reparameterized sampling allows gradients to flow through the sampling distribution, objective evaluation, soft selection, distribution fitting, and the final output. DCEM therefore permits end-to-end optimization of objective parameters, energy models, dynamics, costs, and latent action spaces (Amos et al., 2019).

In structured prediction, DCEM was used to train energy landscapes whose low-energy regions are more consistent with genuine local minima than those produced by fixed-horizon unrolled gradient descent. In control, a decoder maps a low-dimensional latent variable to an entire action sequence, and DCEM searches the latent space. The reported cartpole, cheetah, and walker experiments used substantially fewer trajectory samples than full-space CEM while approximately recovering its performance. The resulting differentiable controller could subsequently be fine-tuned with PPO.

Structured cross-entropy

Ordinary cross-entropy treats labels as unrelated atomic outcomes. Structured cross-entropy introduces a random partition $q$6 of the label space and averages cross-entropies over coarsenings. If $q$7 are partitions with weights $q$8, then for an example $q$9,

$p$0

The singleton partition recovers ordinary cross-entropy. Coarser partitions give partial structural credit to probability assigned to related classes. The construction supports hierarchical, circular, geographical, graph-based, and competing nonnested structures (Lucena, 2022).

The method retains nonnegativity, conditioning-reduces-entropy, the chain rule, mutual-information symmetry, and a structured KL decomposition. In CIFAR-100 experiments, structured cross-entropy improved fine-class and coarser-level accuracy relative to standard cross-entropy, with the largest gains generally occurring under smaller training sets. Circular structured losses improved month prediction, and graph-based random partitions improved county prediction.

Change-point detection

In change-point detection, cross-entropy is used as a directed predictive discrepancy between adjacent segments. A model is fitted to the preceding segment and evaluated on the following segment:

$p$1

The segmentation objective is

$p$2

The Optimal Two Adjacent Windows Algorithm, OTAWA, solves this objective with dynamic programming. Its recurrence is

$p$3

The method uses i.i.d. Gaussian models for Human Activities data and vector autoregressive models for Industrial Equipment data. It achieved the best average performance on most reported metrics, although its worst-case complexity is $p$4 (Serre et al., 2020).

5. Extensions across learning, quantum information, privacy, and topology

Quantum cross-entropy

For density matrices $p$5 and $p$6, quantum cross-entropy is

$p$7

provided $p$8. It decomposes as

$p$9

where $q$00 is von Neumann entropy and $q$01 is quantum relative entropy. It is unitarily invariant, linear in its first argument, convex in its second argument, jointly convex, and additive on tensor-product states.

The paper proves the fidelity-related bound

$q$02

With undisturbed quantum data, minimizing quantum cross-entropy has the same optimizer as maximum likelihood. When the state is reconstructed from measurement outcomes, however, the relation becomes an inequality:

$q$03

The gap reflects information lost through measurement and supports the paper’s emphasis on deferred measurement (Shangnan et al., 2021).

Rényi cross-entropy and leakage

For order $q$04, Rényi cross-entropy is defined by

$q$05

for $q$06, with endpoint definitions at $q$07, $q$08, and $q$09. Its minimum over soft decisions is Rényi entropy:

$q$10

attained at the tilted distribution

$q$11

Using this optimization principle in a privacy model gives $q$12-leakage as the reduction from prior to posterior uncertainty. The resulting quantity is Arimoto mutual information:

$q$13

Maximizing over compatible sensitive attributes yields Sibson mutual information, while $q$14 connects to maximal leakage and pointwise maximal leakage (Ding et al., 2024).

Persistent cross-entropy

Persistence diagrams generally have different points, cardinalities, and event spaces, so ordinary cross-entropy cannot be applied directly. Persistent cross-entropy uses one diagram $q$15 as the reference event space and another diagram $q$16 to determine explanatory power.

Persistence probability is proportional to lifetime:

$q$17

A similarity function $q$18 and bounded persistence weight $q$19 define explanatory power

$q$20

The explanatory difference is

$q$21

with retained weight

$q$22

The retained reference mass is $q$23; the missing mass is assigned to an unexplained event $q$24. The resulting cross-entropy satisfies

$q$25

and therefore

$q$26

Persistent cross-entropy is directional, stable under Wasserstein perturbations, and distinguishes diagrams having the same persistent entropy. The reported studies used it for topological composition, causal-direction analysis, and teacher-to-student topology distillation (Yeom et al., 25 Aug 2026).

Cross-entropy unlearning

CE-U modifies the target distribution rather than maximizing the ordinary loss on a forgotten example. If $q$27 is the token to forget, its modified target logit is set to $q$28, giving

$q$29

The loss is

$q$30

where the target is detached. Its target-logit gradient is

$q$31

Thus the update is large when the forgotten token is highly probable and small when it is already unlikely, reversing the confidence dependence of gradient-ascent unlearning.

General CE-U interpolates between learning and unlearning by assigning the original target a normalized weight $q$32:

$q$33

The cases $q$34 and $q$35 recover ordinary learning and complete unlearning, respectively. On the TOFU benchmark, CE-U was evaluated with LLaMA2-7B and reported strong forgetting quality while retaining utility, without an additional oracle model or additional positive samples (Yang, 3 Mar 2025).

6. Theoretical guarantees, scaling behavior, and limitations

Classification consistency

Cross-entropy is a member of a broader comp-sum family that includes generalized cross-entropy and mean absolute error. For a hypothesis class $q$36, an $q$37-consistency bound relates excess zero-one risk to excess surrogate risk:

$q$38

The bound depends on minimizability gaps, which measure the mismatch between global optimization over $q$39 and pointwise optimization. For standard cross-entropy, the paper gives the square-root form

$q$40

When the gaps vanish, cross-entropy excess risk of order $q$41 yields zero-one excess risk of order $q$42. The paper proves tightness of the functional form for $q$43, including ordinary cross-entropy (Mao et al., 2023).

Cross-entropy is not uniformly superior to every surrogate. Generalized cross-entropy can have smaller minimizability gaps, while MAE has a linear calibration relation, but their bounds contain stronger dependence on the number of classes and their optimization can be more difficult. Surrogate selection therefore involves calibration, hypothesis-class restriction, class-count dependence, smoothness, and optimization.

Scaling laws for LLMs

For autoregressive LLMs, token cross-entropy can be decomposed exactly according to rank-based error. Let $q$44 be the empirical distribution of the number of vocabulary tokens scoring above the correct token, let $q$45 be the normalized geometric-mean probability distribution grouped by that rank error, and let $q$46 be the sum of the groupwise geometric means. Then

$q$47

Error-Entropy measures dispersion of ranking errors, Self-Alignment measures mismatch between ranking-error frequencies and score magnitudes, and Confidence measures the scale of grouped correct-token probabilities. Experiments across model families and datasets reported that Error-Entropy has the most robust power-law relationship with model size, while Self-Alignment and Confidence are more architecture- and dataset-dependent. The paper proposes that conventional cross-entropy scaling appears clean at smaller scales because Error-Entropy dominates, but becomes less predictable at very large scales as the non-scaling components constitute a larger share of the remaining loss (Yan et al., 5 Oct 2025).

Common limitations and controversies

Cross-entropy does not by itself guarantee the behavior often attributed to it.

Predictive likelihood is not identical to task quality: low token-level cross-entropy can coexist with poor long-range structure, limited creativity, or weak application-level performance. The music-generation study reported an association between lower loss and improved local and long-range melodic behavior, but used a single illustrative generation without formal musical-quality metrics (Garcia-Valencia, 2020).
The KL direction matters: minimum cross-entropy, expected entropy reduction, and MaxCE implement different update principles. Applying cross-entropy mechanically to probabilistic information can produce counterintuitive posteriors (Grove et al., 2013).
Parametric projection can be misspecified: in rare-event simulation and optimization, convergence depends on the flexibility of the model family. Partially discriminating families and stochastic objective functions can cause CE to fail or converge poorly [0609461].
Global optimization is not guaranteed: finite-sample elite updates, premature covariance collapse, local optima, initialization, and insufficient mixture coverage remain important failure modes (Moss, 2020).
Full-softmax cost can be substantial: in recommendation, full cross-entropy creates global competition among all items and is theoretically connected to rank metrics, whereas BCE, BPR, NCE, and negative sampling use local or sampled comparisons. Scaled Cross-Entropy approximates the full normalizer with a scaled sampled negative mass, reducing computational cost while retaining a rank-dependent bound (Xu et al., 2024).
Structured variants require valid structure: structured cross-entropy can improve performance when label partitions represent meaningful relationships, but incoherent partitions can harm performance (Lucena, 2022).
Measurement can change the objective: in quantum learning, the equality between cross-entropy minimization and maximum likelihood holds in the undisturbed-state setting; measurement-derived empirical density matrices produce a lower bound rather than equality (Shangnan et al., 2021).
Unlearning is not guaranteed erasure: CE-U suppresses the probability of designated tokens, but a zero probability for an original token does not establish complete semantic erasure, elimination of paraphrased knowledge, or resistance to relearning (Yang, 3 Mar 2025).

Cross-entropy is therefore best understood not as a single universally interchangeable objective, but as a family of expected log-loss and KL-projection principles whose behavior depends on the target distribution, model class, argument orientation, event space, sampling mechanism, and downstream task.