---
title: Forward Cross-Entropy in Machine Learning
url: https://www.emergentmind.com/topics/forward-cross-entropy-objective
type: topic
---

# Forward Cross-Entropy in Machine Learning

The forward cross-entropy objective is a dominant paradigm in machine learning for fitting probabilistic models, particularly within deep learning, language modeling, robust control, and sequential information gathering. It quantifies the expected negative log-likelihood of a predictive model with respect to a reference, typically empirical, data distribution, and has mathematically precise links to Kullback–Leibler (KL) divergence, maximum likelihood estimation (MLE), and robust policy optimization.

## 1. Formal Definition and Autoregressive Model Context

For a data-generating distribution $P$ over sequences $x=(x_1,...,x_T)$, and an autoregressive model $Q_\theta$ parameterized by $\theta$ such that $Q_\theta(x) = \prod_{t=1}^T Q_\theta(x_t | x_{<t})$, the forward cross-entropy is
$$
H(P, Q_\theta) = -\mathbb{E}_{x\sim P}\left[ \log Q_\theta(x) \right] = -\sum_x P(x)\log Q_\theta(x).
$$
This decomposes, by the chain rule, into the sum of next-token losses:
$$
H(P,Q_\theta) = - \mathbb{E}_{x\sim P} \left[\sum_{t=1}^T \log Q_\theta(x_t|x_{<t})\right].
$$
Empirically, over $N$ sequences of lengths $T_i$:
$$
L_{CE}(\theta) = - \frac{1}{\sum_i T_i} \sum_{i=1}^N \sum_{t=1}^{T_i} \log Q_\theta(x_t^{(i)}|x_{<t}^{(i)}).
$$
This formulation is fundamental to modern language models and sequential predictors [2305.16958].

## 2. Equivalence to Maximum Likelihood Estimation

Minimizing the forward cross-entropy loss with respect to the model parameters $\theta$ is mathematically equivalent to maximizing the expected log-likelihood of the observed data—i.e., MLE. Since $P$ is unknown, the expectation is replaced by the empirical mean over a training dataset:
- **MLE objective:** maximize $\sum_{i,t} \log Q_\theta(x_t^{(i)}|x_{<t}^{(i)})$
- **Forward cross-entropy:** minimize $-\sum_{i,t} \log Q_\theta(x_t^{(i)}|x_{<t}^{(i)})$
This equivalence underpins most supervised deep learning algorithms where the forward cross-entropy is interpreted as the training criterion for probabilistic classifiers and sequence models [2305.16958][2311.16357].

## 3. Probabilistic and Information-Theoretic Interpretations

Forward cross-entropy measures how well the model’s probability mass covers the data distribution. It admits a decomposition:
$$
H(P, Q_\theta) = H(P) + D_{KL}(P \parallel Q_\theta)
$$
where $H(P)$ is the entropy of the data distribution (constant w.r.t. $\theta$), and $D_{KL}(P \parallel Q_\theta)$ is the forward KL divergence. This structure yields a “zero-avoiding” property: modes that have nonzero $P(x)$ must also be assigned nonzero $Q_\theta(x)$, otherwise the loss diverges. Consequently, the learned model is penalized heavily for missing modes of $P$, but penalized only weakly for spreading probability mass onto areas where $P(x)=0$ [2305.16958][1409.7552].

In information-gathering and Bayesian active learning, maximizing expected cross-entropy (“MaxCE”) between prior and posterior beliefs acts as a one-step criterion to promote exploration, formally operationalized as maximizing $\mathbb{E}_y \left[ H[p(\theta|D), p(\theta|D,x,y)] \right]$, which is the expected KL from the existing to the updated posterior and is notably asymmetric [1409.7552].

## 4. Variations and Computational Implementations

Forward cross-entropy serves as a core loss function in classifiers with normalized output layers, such as softmax:
$$
\mathrm{SoftMax}(z)_i = \frac{\exp(z_i)}{\sum_j \exp(z_j)}
$$
with the categorical cross-entropy loss:
$$
L_{CE}(z, y^\circ) = -\sum_i y^\circ_i \log \sigma_i(z)
$$
where $y^\circ$ is typically one-hot or label-smoothed. It is empirically and formally substantiated that the gradient of this composite (softmax + cross-entropy) is $\sigma(z) - y^\circ$; thus, explicit cross-entropy computation is redundant for gradient-based updates, as exploited in the ISBE approach for computational efficiency without impacting learning dynamics or accuracy [2311.16357].

## 5. Limitations and Contrast with Reverse Cross-Entropy

In high-capacity, large-data settings, minimizing forward cross-entropy theoretically recovers the true data distribution. However, in practical regimes with finite data and model capacity, its zero-avoiding nature leads to several drawbacks:
- **Over-generalization:** $Q_\theta$ is encouraged to “spread” probability mass into regions of $x$ where $P(x)=0$, resulting in non-humanlike or nonsensical outputs in generative modeling [2305.16958].
- **Language degeneration:** Unbiased decoding often yields incoherent or repetitive outputs due to the weak penalty for $Q_\theta$ assigning positive mass outside the support of $P$.
- **Data noise sensitivity:** The objective enforces coverage of even very rare or noisy empirical examples [2305.16958].

Reverse cross-entropy $H(Q_\theta, P) = -\mathbb{E}_{x\sim Q_\theta}[\log P(x)]$, equivalent to reverse KL $D_{KL}(Q_\theta\|P)$ up to an additive entropy, exhibits “zero-forcing” behavior and avoids assigning probability mass to off-support $x$, but is intractable except in synthetic or fully-known $P$ cases. Mixtures of forward and reverse cross-entropy (e.g., the MixCE objective) are designed to mitigate the respective deficiencies, promoting both mode coverage and avoidance of impossible regions [2305.16958].

## 6. Generalizations in Risk-Aware Control and Information Gathering

In robust dynamic programming contexts, forward cross-entropy, combined with entropy regularization, calibrates the adversarial nature and stochasticity of policies. In minsoftmax DP, the regularization term
$$
-\gamma_H\,\mathrm{CE}(p_k \Vert r_k) + \gamma_E H(p_k)
$$
balances likelihood-driven minimization and the injection of randomness, controlling robustness and optimism. Interpolation between pure minimax, KL-regularized, risk-sensitive, and standard stochastic DP is achieved via independent tuning of forward cross-entropy and entropy weights, exceeding what single KL-regularization admits [2505.11068].

In iterative information-gathering and experimental design, the expected forward cross-entropy (MaxCE) between prior and posterior beliefs promotes challenge to current hypotheses and mitigates local-optima pathologies observed with pure expected-entropy (negative entropy) objectives. Empirical results, in domains ranging from Gaussian process model selection to robotic structure discovery, demonstrate faster model identification compared to entropy-minimization approaches, with additional benefit from mixture objectives combining MaxCE and predictive-uncertainty sampling [1409.7552].

## 7. Computational and Practical Considerations

Computing the forward cross-entropy objective and its gradient is efficient when combined with normalized output layers, enabling vectorized implementation as in ISBE, where the explicit loss layer is omitted in favor of direct error signal computation. In control-theoretic or active information acquisition settings, forward cross-entropy governs the exploration/exploitation trade-off and can be efficiently incorporated into dynamic programming updates via log-sum-exp operations or quadratic analytic forms in linear-Gaussian problems [2311.16357][2505.11068].

A summary of key roles and properties of the forward cross-entropy objective is provided in the table below:

| Context                      | Mathematical Role                             | Noted Outcomes/Limitations      |
|------------------------------|-----------------------------------------------|---------------------------------|
| Language modeling            | MLE via forward CE, $D_{KL}(P||Q_\theta)$     | Over-generalization, degeneration [2305.16958]|
| Classifier training (softmax)| Efficient gradient, identical w/ ISBE         | Explicit CE layer is redundant [2311.16357]   |
| Robust control (minsoftmax)  | Likelihood-driven adversarial regularization  | Tunable robustness/optimism [2505.11068]      |
| Bayesian information gathering | MaxCE: expected forward CE vs. prior         | Avoids entropic local optima [1409.7552]      |

The forward cross-entropy objective remains a central tool in statistical learning due to its analytical tractability, tight links to probabilistic inference, and versatility, but practical weaknesses in open-domain generation and active learning have driven research into hybrid or alternative objective functions.

Source: https://www.emergentmind.com/topics/forward-cross-entropy-objective