Nonlinear Cross-Entropy: Concepts & Applications

• Nonlinear cross-entropy is a generalized form of Shannon’s metric that replaces linear expectation with nonlinear averaging, as seen in Rényi and escort-scaled measures.
• Key contributions include deriving closed-form expressions for exponential families and extending cross-entropy methods to multimodal and constrained optimization problems.
• The framework underpins applications in privacy leakage, reinforcement learning, and nonlinear representation learning, offering robust analytical and practical insights.

Nonlinear cross-entropy denotes a family of generalizations and extensions of classical cross-entropy in which the Shannon form

$H(p;q)=-\sum_{x}p(x)\ln q(x)$

or its differential analogue

$h(p;q)=-\int p(x)\ln q(x)\,dx$

is replaced by a nonlinear averaging rule, a nonlinear dependence on the model distribution, or an iterative cross-entropy optimization procedure. In the literature, the term is used in several distinct but related senses: as a Rényi-order information measure, as a decision-theoretic uncertainty functional for privacy leakage, as a KL-based optimization method for multimodal and constrained problems, and as a framework for analyzing cross-entropy training in nonlinear models (Thierrin et al., 2022, Ding et al., 2024, Benham et al., 2015, Reizinger et al., 2024).

1. Classical baseline and the meaning of nonlinearity

In Shannon theory, cross-entropy is linear in the data-generating distribution $p$ because it is the expectation of $-\ln q(x)$ under $p$. The main information-theoretic nonlinear generalizations replace this logarithm-of-expectation structure by a power mean, a Kolmogorov–Nagumo mean, or an escort-scaled decision rule. The resulting quantity is no longer a linear expectation of log-loss, but a nonlinear functional of $p$ and $q$ (Thierrin et al., 2022, Ding et al., 2024).

A central instance is the Rényi cross-entropy

$$H_\alpha(p;q)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}} p(x)q(x)^{\alpha-1}, \qquad h_\alpha(p;q)=\frac{1}{1-\alpha}\ln \int_{\mathbb S} p(x)q(x)^{\alpha-1}\,dx,$$

which places the cross-term $q(x)^{\alpha-1}$ inside the sum or integral and scales the outer logarithm by $1/(1-\alpha)$. A second construction, the Natural Rényi cross-entropy,

$h(p;q)=-\int p(x)\ln q(x)\,dx$0

preserves the Shannon decomposition $h(p;q)=-\int p(x)\ln q(x)\,dx$1 at Rényi order $h(p;q)=-\int p(x)\ln q(x)\,dx$2 (Thierrin et al., 2022).

Outside information theory, “nonlinear cross-entropy” also refers to optimization schemes in which KL or cross-entropy minimization is carried out over nonlinear or multimodal sampling families, and to reinforcement-learning methods that update policies by increasing the probability of elite high-value joint actions rather than by standard centralized gradients (Wang et al., 2013, Wang et al., 24 Nov 2025). This suggests that the common denominator is not a single formula but a shift from linear expectation-based matching to nonlinear weighting, selection, or projection.

2. Rényi-type cross-entropies

The Rényi cross-entropy developed in "On the Rényi Cross-Entropy" (Thierrin et al., 2022) is designed to satisfy two consistency conditions: it reduces to Rényi entropy when $h(p;q)=-\int p(x)\ln q(x)\,dx$3, and it converges to Shannon cross-entropy as $h(p;q)=-\int p(x)\ln q(x)\,dx$4. Both properties hold: $h(p;q)=-\int p(x)\ln q(x)\,dx$5 and L’Hôpital’s rule yields the Shannon limit as $h(p;q)=-\int p(x)\ln q(x)\,dx$6 (Thierrin et al., 2022).

The same work establishes several basic properties. Under the finiteness assumptions used there, the differential Rényi cross-entropy $h(p;q)=-\int p(x)\ln q(x)\,dx$7 is non-increasing in $h(p;q)=-\int p(x)\ln q(x)\,dx$8, and the discrete case follows similarly. The discrete Rényi cross-entropy is nonnegative, whereas the differential version can be negative, as with ordinary differential cross-entropy. In the high-order limit,

$h(p;q)=-\int p(x)\ln q(x)\,dx$9

so the measure becomes dominated by the highest-probability mass of $p$0, largely independent of $p$1 except for positivity constraints (Thierrin et al., 2022).

"Rényi Cross-Entropy Measures for Common Distributions and Processes with Memory" (Thierrin et al., 2022) places this definition beside the Natural Rényi cross-entropy. The two constructions coincide with Shannon cross-entropy in the limit $p$2, and both reduce to the corresponding Rényi entropy measures when $p$3 almost everywhere. The paper explicitly characterizes both as nonlinear because they replace the linear expectation under $p$4 by a logarithm of a power mean or generalized moment, with $p$5 controlling how strongly large or small values of $p$6 are emphasized (Thierrin et al., 2022).

3. Decision-theoretic interpretation and $p$7-leakage

"A Cross Entropy Interpretation of Rényi Entropy for $p$8-leakage" (Ding et al., 2024) reinterprets Rényi entropy as the minimum of a generalized cross-entropy over soft decisions. The paper introduces the Rényi-scaled distribution

$p$9

and uses the nonlinear averaging function

$-\ln q(x)$0

The resulting generalized cross-entropy is

$-\ln q(x)$1

with the special cases

$-\ln q(x)$2

$-\ln q(x)$3

$-\ln q(x)$4

Its core theorem states that for fixed $-\ln q(x)$5,

$-\ln q(x)$6

Thus Rényi entropy is recovered as the best achievable generalized cross-entropy, exactly parallel to Shannon entropy at $-\ln q(x)$7 (Ding et al., 2024).

The privacy application is formulated on the Markov chain

$-\ln q(x)$8

Prior uncertainty is $-\ln q(x)$9, posterior uncertainty is the corresponding $p$0-mean over $p$1, and the leakage is defined by

$p$2

The paper states explicitly that

$p$3

so the proposed $p$4-leakage is exactly the Arimoto mutual information. It extends the usual $p$5-leakage from $p$6 to the full range $p$7, with $p$8 corresponding to nonstochastic leakage and $p$9 yielding maximal leakage behavior (Ding et al., 2024).

The same framework clarifies how the order parameter changes the effective decision rule. For $p$0, the scaled distribution emphasizes high-probability events more strongly; for $p$1, it becomes flatter. At $p$2,

$p$3

whereas at $p$4 it concentrates on the most probable events. The paper also identifies the elementary leakage

$p$5

and notes that

$p$6

is pointwise maximal leakage (Ding et al., 2024).

4. Closed forms, exponential families, and sources with memory

For exponential-family distributions,

$p$7

"On the Rényi Cross-Entropy" (Thierrin et al., 2022) derives a general closed form. If $p$8 and $p$9 are of the same exponential-family type with natural parameters $q$0 and $q$1, and

$q$2

then

$q$3

where

$q$4

When $q$5 is constant, the $q$6 term vanishes. The paper tabulates explicit formulas for Beta, $q$7, exponential, Gamma, Gaussian, and Laplace families (Thierrin et al., 2022).

A particularly important special case is the zero-mean multivariate Gaussian. For invertible covariance matrices $q$8 and $q$9,

$$H_\alpha(p;q)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}} p(x)q(x)^{\alpha-1}, \qquad h_\alpha(p;q)=\frac{1}{1-\alpha}\ln \int_{\mathbb S} p(x)q(x)^{\alpha-1}\,dx,$$0

with

$$H_\alpha(p;q)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}} p(x)q(x)^{\alpha-1}, \qquad h_\alpha(p;q)=\frac{1}{1-\alpha}\ln \int_{\mathbb S} p(x)q(x)^{\alpha-1}\,dx,$$1

From this finite-dimensional expression, the paper derives the Rényi differential cross-entropy rate for stationary zero-mean Gaussian processes: $$H_\alpha(p;q)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}} p(x)q(x)^{\alpha-1}, \qquad h_\alpha(p;q)=\frac{1}{1-\alpha}\ln \int_{\mathbb S} p(x)q(x)^{\alpha-1}\,dx,$$2 under the standard Toeplitz-matrix assumptions and the condition that $$H_\alpha(p;q)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}} p(x)q(x)^{\alpha-1}, \qquad h_\alpha(p;q)=\frac{1}{1-\alpha}\ln \int_{\mathbb S} p(x)q(x)^{\alpha-1}\,dx,$$3 be Riemann integrable (Thierrin et al., 2022).

For finite-alphabet time-invariant Markov sources, the same paper defines

$$H_\alpha(p;q)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}} p(x)q(x)^{\alpha-1}, \qquad h_\alpha(p;q)=\frac{1}{1-\alpha}\ln \int_{\mathbb S} p(x)q(x)^{\alpha-1}\,dx,$$4

and proves that, under irreducibility of $$H_\alpha(p;q)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}} p(x)q(x)^{\alpha-1}, \qquad h_\alpha(p;q)=\frac{1}{1-\alpha}\ln \int_{\mathbb S} p(x)q(x)^{\alpha-1}\,dx,$$5,

$$H_\alpha(p;q)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}} p(x)q(x)^{\alpha-1}, \qquad h_\alpha(p;q)=\frac{1}{1-\alpha}\ln \int_{\mathbb S} p(x)q(x)^{\alpha-1}\,dx,$$6

where $$H_\alpha(p;q)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}} p(x)q(x)^{\alpha-1}, \qquad h_\alpha(p;q)=\frac{1}{1-\alpha}\ln \int_{\mathbb S} p(x)q(x)^{\alpha-1}\,dx,$$7 is the largest positive eigenvalue of $$H_\alpha(p;q)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}} p(x)q(x)^{\alpha-1}, \qquad h_\alpha(p;q)=\frac{1}{1-\alpha}\ln \int_{\mathbb S} p(x)q(x)^{\alpha-1}\,dx,$$8. The proof uses Perron–Frobenius theory, and the paper remarks that the result extends to reducible $$H_\alpha(p;q)=\frac{1}{1-\alpha}\ln\sum_{x\in\mathbb{S}} p(x)q(x)^{\alpha-1}, \qquad h_\alpha(p;q)=\frac{1}{1-\alpha}\ln \int_{\mathbb S} p(x)q(x)^{\alpha-1}\,dx,$$9 by decomposition into communicating classes (Thierrin et al., 2022).

"Rényi Cross-Entropy Measures for Common Distributions and Processes with Memory" (Thierrin et al., 2022) broadens this program by tabulating closed forms for 13 common continuous distributions from the exponential family and by presenting corresponding formulas for the Natural Rényi cross-entropy. It includes explicit expressions for Beta, Exponential, Gamma, univariate and multivariate Gaussian, Half-Normal, Rayleigh, scaled and non-scaled $q(x)^{\alpha-1}$0, $q(x)^{\alpha-1}$1, Gumbel, Laplace, Maxwell–Boltzmann, and Pareto distributions. The same paper also summarizes Shannon, Natural Rényi, and Rényi cross-entropy rates for stationary Gaussian processes and finite-state Markov sources (Thierrin et al., 2022).

5. Cross-entropy methods in optimization, rare-event simulation, and multi-agent RL

A different usage of nonlinear cross-entropy appears in optimization. "CEoptim: Cross-Entropy R Package for Optimization" (Benham et al., 2015) presents the classical cross-entropy method as KL minimization relative to the zero-variance importance-sampling density. Given an optimization problem

$q(x)^{\alpha-1}$2

the method samples $q(x)^{\alpha-1}$3, converts optimization into a rare-event problem, selects elite samples through the sample $q(x)^{\alpha-1}$4-quantile

$q(x)^{\alpha-1}$5

and updates the sampling distribution by

$q(x)^{\alpha-1}$6

The paper emphasizes that this makes CE suitable for nonlinear, multimodal, nonconvex optimization, as well as continuous, discrete, mixed, and constrained settings (Benham et al., 2015).

"An Explicit Cross Entropy Scheme for Mixtures" (Wang et al., 2013) extends this idea from a single exponential tilt to a mixture density

$q(x)^{\alpha-1}$7

The motivation is that nonconvex or multimodal target regions may contain several separated important regions, so a single tilt can be suboptimal and can even produce erroneous estimates. The paper resolves the coupled maximization of $q(x)^{\alpha-1}$8 by combining CE with EM, introducing latent component labels and obtaining explicit updates for the mixture weights and component parameters. In the Gaussian case $q(x)^{\alpha-1}$9, the component update is

$1/(1-\alpha)$0

The paper’s central claim is that mixtures avoid collapse onto a single mode and improve robustness for multimodal rare-event estimation (Wang et al., 2013).

The same KL-and-elite-sample logic is used for constrained nonlinear design in "Enhancing the performance of a bistable energy harvesting device via the cross-entropy method" (Jr, 2021). There, the objective is the mean output power of a nonlinear electromechanical system, while feasibility is enforced by a binary chaos classifier from the 0–1 test. The penalized score is

$1/(1-\alpha)$1

with truncated-Gaussian sampling and smoothed updates of means and standard deviations. For the two-dimensional case, the paper reports that CE with $1/(1-\alpha)$2 used 1,300 function evaluations versus 65,536 for grid search, corresponding to a speed-up of about $1/(1-\alpha)$3, and that $1/(1-\alpha)$4 gave about $1/(1-\alpha)$5 speed-up with no loss in accuracy (Jr, 2021).

In cooperative MARL, "Multi-Agent Cross-Entropy Method with Monotonic Nonlinear Critic Decomposition" (Wang et al., 24 Nov 2025) extends the cross-entropy method to decentralized policies over joint actions. MCEM samples joint actions, evaluates them with a joint critic $1/(1-\alpha)$6, keeps the top $1/(1-\alpha)$7 quantile, and updates each agent’s policy by increasing the log-probability of the elite joint actions. The critic is a monotonic nonlinear decomposition

$1/(1-\alpha)$8

which preserves alignment between global and local greedy choices. The paper reports higher median win rates, faster convergence, and lower variance across 9 discrete-action scenarios, and stronger performance than MADDPG and FACMAC on 3 continuous-action Predator-Prey scenarios (Wang et al., 24 Nov 2025).

6. Nonlinear dynamics, generalized divergence, and entropy production

In statistical-physics and stochastic-process settings, nonlinear cross-entropy appears as a divergence or entropic distance adapted to nonlinear dynamics rather than imposed a priori. "Entropic Distance for Nonlinear Master Equation" (Biró et al., 2017) considers the nonlinear master equation

$1/(1-\alpha)$9

with stationary distribution $h(p;q)=-\int p(x)\ln q(x)\,dx$00 defined by the corresponding balance relation. For the power nonlinearity

$h(p;q)=-\int p(x)\ln q(x)\,dx$01

the paper identifies the associated entropic distance

$h(p;q)=-\int p(x)\ln q(x)\,dx$02

and shows that comparison with the uniform distribution yields the Tsallis entropy

$h(p;q)=-\int p(x)\ln q(x)\,dx$03

The guiding principle is that the correct generalized divergence is the Lyapunov-type functional whose time derivative is nonpositive for the nonlinear dynamics (Biró et al., 2017).

A related entropy-production viewpoint appears in "Entropy production in nonlinear recombination models" (Caputo et al., 2016). There the dynamics on

$h(p;q)=-\int p(x)\ln q(x)\,dx$04

preserves one-site marginals, and the natural equilibrium is the product measure

$h(p;q)=-\int p(x)\ln q(x)\,dx$05

The paper proves the exact dissipation identity

$h(p;q)=-\int p(x)\ln q(x)\,dx$06

and seeks a nonlinear log-Sobolev-type inequality

$h(p;q)=-\int p(x)\ln q(x)\,dx$07

which implies exponential decay of relative entropy. For four canonical recombination laws, it computes sharp subadditivity constants

$h(p;q)=-\int p(x)\ln q(x)\,dx$08

The paper explicitly frames these results as a nonlinear analogue of entropy methods from kinetic theory (Caputo et al., 2016).

This suggests that, in dynamics-oriented literatures, nonlinear cross-entropy is often best understood as a state-dependent relative-entropy construction matched to the evolution law, rather than as a single closed-form modification of Shannon’s expectation.

7. Cross-entropy in nonlinear representation learning and loss geometry

In modern learning theory, cross-entropy is also analyzed as a mechanism operating on nonlinear generative structure. "Cross-Entropy Is All You Need To Invert the Data Generating Process" (Reizinger et al., 2024) studies observations of the form

$h(p;q)=-\int p(x)\ln q(x)\,dx$09

where $h(p;q)=-\int p(x)\ln q(x)\,dx$10 is an injective nonlinear generator and the latent variables satisfy a cluster-centric model. For parametric instance discrimination and supervised classification, the paper shows that cross-entropy minimization identifies the latent representation up to a simple ambiguity: under the strongest normalization assumptions, the recovered map $h(p;q)=-\int p(x)\ln q(x)\,dx$11 is orthogonal linear, and more generally it is linear. In the supervised theorem, a cross-entropy classifier trained on class labels yields

$h(p;q)=-\int p(x)\ln q(x)\,dx$12

as a linear map from the latent sphere to the learned representation space. The paper presents this as a nonlinear ICA-style recovery result for standard supervised classification (Reizinger et al., 2024).

The bridge from cross-entropy minimization to inversion is the condition that, at the global optimum, the learned softmax posterior matches the true latent posterior: $h(p;q)=-\int p(x)\ln q(x)\,dx$13 which then implies

$h(p;q)=-\int p(x)\ln q(x)\,dx$14

Empirically, the paper reports $h(p;q)=-\int p(x)\ln q(x)\,dx$15 typically near $h(p;q)=-\int p(x)\ln q(x)\,dx$16 on simulated data matching its assumptions, successful recovery of latent factors on DisLib, and linear decoding of proxy factors from ImageNet representations (Reizinger et al., 2024).

A complementary perspective is geometric. "Wolkowicz-Styan Upper Bound on the Hessian Eigenspectrum for Cross-Entropy Loss in Nonlinear Smooth Neural Networks" (Omae et al., 11 Apr 2026) studies a three-layer smooth nonlinear network for binary classification with Linear, Sigmoid, Tanh, SoftPlus / SmoothReLU, and GELU activations. For the binary cross-entropy loss

$h(p;q)=-\int p(x)\ln q(x)\,dx$17

the paper derives an upper bound on the maximum Hessian eigenvalue,

$h(p;q)=-\int p(x)\ln q(x)\,dx$18

where $h(p;q)=-\int p(x)\ln q(x)\,dx$19 and $h(p;q)=-\int p(x)\ln q(x)\,dx$20 are determined by the trace of the Hessian and the trace of its square. The bound is expressed as a function of the affine transformation parameters, hidden layer dimensions, and the degree of orthogonality among the training samples. The paper interprets $h(p;q)=-\int p(x)\ln q(x)\,dx$21 as a sharpness measure, notes that large $h(p;q)=-\int p(x)\ln q(x)\,dx$22 correlates with more distorted decision boundaries and lower test macro-F1, and records the overfitting caveat

$h(p;q)=-\int p(x)\ln q(x)\,dx$23

Here nonlinear cross-entropy does not mean a new information measure; it means the geometry of the standard cross-entropy loss on nonlinear smooth neural networks (Omae et al., 11 Apr 2026).

Across these literatures, nonlinear cross-entropy therefore names a family of constructions rather than a single doctrine: Rényi-order deformations of Shannon cross-entropy, decision-theoretic uncertainty minimization under nonlinear averaging, KL-based elite-sample optimization for multimodal search, Lyapunov divergences for nonlinear stochastic dynamics, and structural analyses of cross-entropy training in nonlinear neural models.