Cross-Entropy Condition: Theory & Applications

• Cross-Entropy Condition is a family of criteria defined via asymmetric cross-entropy measures and KL divergence to optimally choose distributions in diverse settings.
• It underpins methods in rare-event simulation, iterative Bayesian information gathering, and probabilistic updating, ensuring efficient selection and convergence.
• Extensions to quantum, Rényi, and predictive coding frameworks reveal its role in optimizing model fidelity, benchmarking protocols, and training dynamics.

The expression cross-entropy condition is used in several technically distinct senses across the literature. In rare-event simulation, it denotes a bounded-KL sufficient condition guaranteeing asymptotic optimality of a cross-entropy estimator. In iterative information gathering, it denotes a one-step sampling rule that maximizes the expected cross entropy between old and new beliefs. In quantum and Rényi settings, it denotes minimization principles connecting cross entropy to likelihood, entropy, and leakage measures. In probabilistic updating, it denotes posterior selection by minimizing Kullback–Leibler divergence under probabilistic constraints (Ridder, 2010, Kulick et al., 2014, Shangnan et al., 2021, Ding et al., 2024, Grove et al., 2013). A unifying feature is the use of an asymmetric cross-entropy functional to choose a distribution, query, control, or benchmark statistic.

1. Formal scope and mathematical forms

In the classical updating literature, the cross-entropy from a posterior $Q$ to a prior $P$ is identified with the Kullback–Leibler divergence

$$D_{KL}(Q\parallel P)\;=\;\sum_{i=1}^n Q(i)\,\log\!\frac{Q(i)}{P(i)},$$

and it is explicitly noted that it is not symmetric in $P$ and $Q$ (Grove et al., 2013). That asymmetry is central to later distinctions between entropy-minimizing and cross-entropy-maximizing criteria.

In the quantum setting, the quantum cross entropy of $\sigma$ relative to $\rho$ is

$$S(\rho,\sigma)\;=\;-\,\mathrm{Tr}\bigl[\rho\,\log \sigma\bigr],$$

with the identity

$S(\rho,\sigma)\;=\;S(\rho\|\sigma)+S(\rho),$

where $S(\rho\|\sigma)$ is quantum relative entropy and $P$0 is the von Neumann entropy (Shangnan et al., 2021).

In the Rényi setting, the paper on $P$1-leakage defines a family of Rényi cross entropies $P$2 over $P$3, with the $P$4 case reducing to

$P$5

and proves the minimization identity

$P$6

(Ding et al., 2024).

These definitions are not interchangeable. They share a common variational structure, but they operate on different objects: probability vectors in Bayesian updating, density operators in quantum learning, and order-$P$7 deformations in Rényi information theory. This suggests that “cross-entropy condition” is best read as a family of domain-specific selection principles rather than a single canonical theorem.

2. Bayesian information gathering and the MaxCE condition

For iterative information gathering, the expected cross-entropy objective is defined for a candidate query $P$8 by

$P$9

Equivalently,

$$D_{KL}(Q\parallel P)\;=\;\sum_{i=1}^n Q(i)\,\log\!\frac{Q(i)}{P(i)},$$0

(Kulick et al., 2014).

The paper contrasts this with the classical expected-entropy criterion

$$D_{KL}(Q\parallel P)\;=\;\sum_{i=1}^n Q(i)\,\log\!\frac{Q(i)}{P(i)},$$1

The only formal difference is the ordering of the arguments in the KL divergence, but because

$$D_{KL}(Q\parallel P)\;=\;\sum_{i=1}^n Q(i)\,\log\!\frac{Q(i)}{P(i)},$$2

the two criteria behave differently. The paper states that $$D_{KL}(Q\parallel P)\;=\;\sum_{i=1}^n Q(i)\,\log\!\frac{Q(i)}{P(i)},$$3 favors experiments that drive the posterior towards states of lower entropy, whereas $$D_{KL}(Q\parallel P)\;=\;\sum_{i=1}^n Q(i)\,\log\!\frac{Q(i)}{P(i)},$$4 favors experiments that maximally change the belief in either direction, even if entropy rises temporarily (Kulick et al., 2014).

A central claim is that greedy entropy minimization can become trapped when the current belief is already low-entropy but wrongly concentrated. By contrast, MaxCE is designed to challenge refutable beliefs. The paper further states that the expected conditional entropy $$D_{KL}(Q\parallel P)\;=\;\sum_{i=1}^n Q(i)\,\log\!\frac{Q(i)}{P(i)},$$5 is not submodular in the query set, so there is no bounded-regret guarantee for $$D_{KL}(Q\parallel P)\;=\;\sum_{i=1}^n Q(i)\,\log\!\frac{Q(i)}{P(i)},$$6. In the Gaussian-process toy example with $$D_{KL}(Q\parallel P)\;=\;\sum_{i=1}^n Q(i)\,\log\!\frac{Q(i)}{P(i)},$$7, MaxCE selects inputs where the two GPs disagree most, rather than sampling near existing observations merely to confirm a $$D_{KL}(Q\parallel P)\;=\;\sum_{i=1}^n Q(i)\,\log\!\frac{Q(i)}{P(i)},$$8 bias (Kulick et al., 2014).

The implementation assumptions are explicit: one must compute or approximate $$D_{KL}(Q\parallel P)\;=\;\sum_{i=1}^n Q(i)\,\log\!\frac{Q(i)}{P(i)},$$9, $P$0, and $P$1. For a discrete hypothesis space, the objective reduces to finite sums; in continuous settings, the paper allows Monte Carlo or analytic approximations. The stated complexity per candidate $P$2 is $P$3, often dominated by computing $P$4. The paper also proposes a convex mixture

$P$5

to balance parameter learning with predictive accuracy (Kulick et al., 2014).

3. Rare-event simulation, optimization, and asymptotic optimality

In the rare-event simulation literature for Markov chains, the cross-entropy method is linked to the zero-variance change of measure. For a rare event $P$6, the implementable importance-sampling estimator has the form

$P$7

while the zero-variance but non-implementable change of measure is

$P$8

The cross-entropy program yields

$P$9

and Proposition 2.1 identifies the zero-variance kernel as

$Q$0

The paper then states that $Q$1 (Ridder, 2010).

The paper introduces a specific Cross-Entropy Condition for a family of rare-event problems $Q$2: $Q$3 Under this condition, the estimator is asymptotically optimal in the logarithmic sense: $Q$4 The proof sketch given in the paper passes through the second-moment identity

$Q$5

and Jensen’s inequality, which bounds the multiplicative overhead by $Q$6 (Ridder, 2010).

A different paper on convergence issues for extended implementation emphasizes that CE “works rather good when dealing with deterministic function optimization,” but that “two conditions are needed for a good convergence of the method.” First, “it is necessary to have a family of models sufficiently flexible for discriminating the optimal events.” Second, “indirectly, it appears also that the function to be optimized should be deterministic.” The abstract further states that CE can fail for a “partially discriminating model family” and for “stochastic functions,” and that alternative improvements are investigated and compared on random examples [0609461].

Taken together, these results separate two levels of analysis. One level gives a sharp bounded-KL condition for logarithmic efficiency in Markov-chain rare-event simulation. Another level warns that, once model-family discrimination or determinism is relaxed, convergence can fail. A plausible implication is that the practical success of CE depends not only on the optimization criterion, but also on representational adequacy of the proposal family.

4. Probability update under probabilistic constraints

In probability updating, the baseline rule is ordinary conditioning. If an event $Q$7 is learned with certainty, the updated distribution is

$Q$8

Cross-entropy minimization is introduced for uncertain information expressible as a constraint on the new distribution $Q$9, for example

$\sigma$0

The posterior is then chosen to minimize $\sigma$1 subject to normalization and the probabilistic constraint, yielding an exponential-family solution

$\sigma$2

for suitable multipliers $\sigma$3 and $\sigma$4 (Grove et al., 2013).

The worked example is the Judy Benjamin problem. Judy’s prior assigns probability $\sigma$5 to each quadrant $\sigma$6, and the message is

$\sigma$7

Minimizing KL divergence subject to $\sigma$8 yields a posterior in which, in “many presentations (including van Fraassen’s),” one obtains

$\sigma$9

The paper describes this as counter to the widely shared intuition that the message bears on $\rho$0 versus $\rho$1, not on Blue versus Red (Grove et al., 2013).

The proposed alternative is not another cross-entropy rule but an enlarged-space conditioning construction. The message is interpreted as a statement about HQ’s own probability function, Judy conditions in the bigger space of possible HQ-distributions, and then projects back. The paper states that after this calculation Judy’s marginal on $\rho$2 remains exactly

$\rho$3

for all values of the odds ratio (Grove et al., 2013).

The broader lesson drawn there is explicitly philosophical: cross-entropy minimization is “a mechanically simple, axiomatizable rule” that “always produces a unique posterior whenever the new information can be cast as linear constraints on $\rho$4,” but there is “no purely syntactic guarantee” that KL-nearest updating is the correct interpretation of evidence. The paper recommends ordinary conditioning whenever the evidence can sensibly be represented as an event in an appropriately enlarged probability space (Grove et al., 2013).

5. Quantum and Rényi cross-entropy conditions

In quantum machine learning, the quantum cross entropy

$\rho$5

is lower-bounded by overlap and fidelity terms: $\rho$6 where

$\rho$7

The paper distinguishes two scenarios. When the quantum data are undisturbed by measurement, minimizing $\rho$8 is equivalent to maximizing likelihood, with the common optimizer $\rho$9. When cross entropy is built from empirical density matrices constructed from projective measurements, the paper proves

$$S(\rho,\sigma)\;=\;-\,\mathrm{Tr}\bigl[\rho\,\log \sigma\bigr],$$0

and states that the strict inequality reflects information lost by measurement collapse. The stated conclusion is that, for “full quantum machine learning,” the deferred measurement principle is crucial because it preserves the exact cross-entropy versus likelihood equivalence (Shangnan et al., 2021).

In the Rényi framework, the paper introduces the generator

$$S(\rho,\sigma)\;=\;-\,\mathrm{Tr}\bigl[\rho\,\log \sigma\bigr],$$1

and interprets Rényi entropy as a $$S(\rho,\sigma)\;=\;-\,\mathrm{Tr}\bigl[\rho\,\log \sigma\bigr],$$2-mean cross-entropy measure. The associated minimization theorem is

$$S(\rho,\sigma)\;=\;-\,\mathrm{Tr}\bigl[\rho\,\log \sigma\bigr],$$3

This is then used to define prior and posterior uncertainty measures and the corresponding $$S(\rho,\sigma)\;=\;-\,\mathrm{Tr}\bigl[\rho\,\log \sigma\bigr],$$4-leakage: $$S(\rho,\sigma)\;=\;-\,\mathrm{Tr}\bigl[\rho\,\log \sigma\bigr],$$5 The paper states that this extends $$S(\rho,\sigma)\;=\;-\,\mathrm{Tr}\bigl[\rho\,\log \sigma\bigr],$$6-leakage from $$S(\rho,\sigma)\;=\;-\,\mathrm{Tr}\bigl[\rho\,\log \sigma\bigr],$$7 to $$S(\rho,\sigma)\;=\;-\,\mathrm{Tr}\bigl[\rho\,\log \sigma\bigr],$$8, with $$S(\rho,\sigma)\;=\;-\,\mathrm{Tr}\bigl[\rho\,\log \sigma\bigr],$$9 referring to nonstochastic leakage. It further states that maximal leakage emerges as a $S(\rho,\sigma)\;=\;S(\rho\|\sigma)+S(\rho),$0-mean of an elementary $S(\rho,\sigma)\;=\;S(\rho\|\sigma)+S(\rho),$1-leakage and that the supremum over $S(\rho,\sigma)\;=\;S(\rho\|\sigma)+S(\rho),$2 yields Sibson mutual information (Ding et al., 2024).

These two lines of work share a common pattern: cross entropy is not merely a loss but a variational characterization. In the quantum case, the characterization governs when maximum-likelihood reasoning survives measurement. In the Rényi case, it generates the underlying entropy and leakage quantities themselves.

6. Cross-entropy in training dynamics and predictive-coding analyses

For Transformer-type residual layers, cross-entropy training is embedded into a continuous-depth mean field control problem. The token-averaged cross-entropy loss is

$S(\rho,\sigma)\;=\;S(\rho\|\sigma)+S(\rho),$3

with gradient

$S(\rho,\sigma)\;=\;S(\rho\|\sigma)+S(\rho),$4

The continuous-depth population objective is

$S(\rho,\sigma)\;=\;S(\rho\|\sigma)+S(\rho),$5

and the Pontryagin system has terminal condition

$S(\rho,\sigma)\;=\;S(\rho\|\sigma)+S(\rho),$6

The paper states that the terminal adjoint contains the softmax residual and that the discrete Euler scheme satisfies an $S(\rho,\sigma)\;=\;S(\rho\|\sigma)+S(\rho),$7 pathwise approximation, while the statistical sampling error is $S(\rho,\sigma)\;=\;S(\rho\|\sigma)+S(\rho),$8 (Huan et al., 22 Jun 2026).

A different empirical setting concerns predictive-coding networks. There, a reduction of the K-way energy probe to a monotone transform of the log-softmax margin is said to rest on five assumptions, including A1. CE at the output and A3. Effectively feedforward inference dynamics. The pre-registered study compares standard predictive coding with CE, standard predictive coding with MSE, and bidirectional predictive coding. The reported probe-softmax gaps are

$S(\rho,\sigma)\;=\;S(\rho\|\sigma)+S(\rho),$9

with the paper stating that removing CE alone without changing inference dynamics halves the gap, and removing CE entirely flips it slightly positive. It further reports that CE training produces output logit norms approximately $S(\rho\|\sigma)$0 larger than MSE or bPC training, and that a temperature-scaling ablation attributes approximately $S(\rho\|\sigma)$1 of the original gap to logit-scale effects and approximately $S(\rho\|\sigma)$2 to a scale-invariant ranking advantage of CE-trained representations (Cacioli, 23 Apr 2026).

In these training-oriented analyses, the cross-entropy condition is not a bounded-KL theorem. It is instead a structural feature of the optimization or inference pipeline: in mean field control it fixes the terminal adjoint source term, and in predictive coding it is described as a “load-bearing” assumption for a claimed decomposition.

7. Benchmarking, ergodicity, and cross-entropy diagnostics

In random-circuit benchmarking, cross-entropy appears through cross-entropy benchmarking (XEB) and an associated ergodicity condition. For a circuit ensemble $S(\rho\|\sigma)$3 and post-processing function $S(\rho\|\sigma)$4, the paper defines ergodicity by requiring the single-circuit average

$S(\rho\|\sigma)$5

to be close to the ensemble average

$S(\rho\|\sigma)$6

with deviation $S(\rho\|\sigma)$7 and high probability. The main theorem states that unitary $S(\rho\|\sigma)$8-designs are ergodic for positive-coefficient polynomials

$S(\rho\|\sigma)$9

and the proof uses the fact that off-diagonal covariances are nonpositive, giving $P$00 before applying Chebyshev’s inequality (Cheng et al., 13 Feb 2025).

The paper then defines the deviation of ergodicity

$P$01

as a benchmarking score. For the quadratic choice

$P$02

the framework recovers Google’s linear XEB fidelity: $P$03 Under global depolarizing noise,

$P$04

the estimator satisfies

$P$05

so that $P$06 (Cheng et al., 13 Feb 2025).

This benchmarking usage is again distinct from the MaxCE query rule and from KL-based posterior updating. The commonality lies in the role of cross entropy as a computable surrogate whose deviation from an ideal reference captures fidelity, noise, or lack of ergodicity.