---
title: Collision Cross-Entropy Explained
url: https://www.emergentmind.com/topics/collision-cross-entropy
type: topic
---

# Collision Cross-Entropy Explained

Collision cross-entropy is a loss function designed for scenarios where class labels are soft (i.e., represented by categorical probability distributions), offering distinct advantages over the traditional Shannon cross-entropy, especially in contexts of ambiguous or uncertain targets. It is defined as the negative logarithm of the probability that two independent samples—one drawn from the predicted distribution and one from the target distribution—collide in class assignment. This formulation leads to robust learning dynamics in supervised classification with noisy labels and in deep clustering with self-labeling frameworks, providing improved stability and performance under label uncertainty [2303.07321].

## 1. Formal Definition and Mathematical Properties

Given two categorical distributions $p = (p_1, \dots, p_K)$ and $q = (q_1, \dots, q_K)$ over $K$ classes, the collision cross-entropy (of order 2) is defined as
\[
H_2(p, q) := -\ln\left( \Pr_{C \sim p,\,T \sim q}[C = T] \right) = -\ln\left( \sum_{k=1}^K p_k q_k \right).
\]
This expression computes the negative log of the total "collision probability"—the chance that a label drawn independently from $p$ and $q$ coincides. Minimizing the collision cross-entropy with respect to the model predictions $q$ is equivalent to maximizing this collision probability, driving alignment between the predicted and target distributions [2303.07321].

## 2. Relationship to Shannon’s Cross-Entropy and One-hot Labels

For one-hot target labels (i.e., $p_i=1$ for some $i$, $p_{k \ne i} = 0$), the collision cross-entropy reduces to the standard cross-entropy:
\[
H_2(p, q) = -\ln q_i,
\]
which exactly matches Shannon's cross-entropy $H(p,q) = -\ln q_i$ for such labels. Therefore, in traditional settings with hard labels, collision cross-entropy and Shannon's cross-entropy are equivalent [2303.07321].

However, the distinction becomes significant for soft targets. When $p$ is uniform ($p_k = 1/K$), $H_2(p, q) = \ln K$ is constant, and its gradient with respect to $q$ is zero. Thus, training gradients vanish for data points with maximally uncertain targets; these points do not influence parameter updates. In contrast, Shannon's cross-entropy retains dependence on $q$ for soft labels, causing the model to mimic or "copy" the uncertainty in soft targets, potentially amplifying noise or label ambiguity [2303.07321].

## 3. Symmetry and Behavior with Soft Labels

A distinctive property of collision cross-entropy is symmetry in its arguments:
\[
H_2(p, q) = H_2(q, p).
\]
This symmetry is not shared by Shannon's cross-entropy in general ($H(p, q) \neq H(q, p)$), and has practical importance in unsupervised and self-labeling clustering setups, where both pseudo-labels and predictions are variable distributions [2303.07321].

For soft categorical targets, the collision cross-entropy is selective: it disregards points where maximum uncertainty is present (i.e., uniform targets), effectively suppressing the contribution of ambiguous data to the training signal. This behavior is advantageous in settings where label uncertainty reflects lack of information rather than inherent ambiguity in the data [2303.07321].

## 4. Application in Deep Clustering and EM Optimization

Collision cross-entropy has particular utility in deep clustering, especially where pseudo-labels (latent targets) are estimated jointly with model parameters. A typical self-labeling loss uses
\[
L_{\mathrm{CCE}^+} = \frac{1}{M} \sum_{i=1}^M H_2(y_i, \sigma_i) + \lambda\, D_{\mathrm{KL}}(u \Vert \bar y),
\]
where $y_i$ are pseudo-labels, $\sigma_i$ are model predictions, $u$ is the uniform prior, $\bar y$ is the mean pseudo-label vector, and $D_{\mathrm{KL}}$ enforces balance across clusters.

An efficient EM (Expectation-Maximization) algorithm is derived for pseudo-label estimation with collision cross-entropy. The E-step introduces hidden cluster supports and applies Jensen's inequality for a tractable surrogate, while the M-step optimizes the pseudo-label assignments per sample via root-finding in a single variable. This approach achieves significant computational efficiency: each E- and M-step operates in $O(BK)$ time per minibatch (where $B$ is batch size and $K$ number of classes), and empirical convergence is reported within 10–20 EM iterations, which is substantially faster (by factors of 10–100$\times$) than projected-gradient descent on the simplex [2303.07321].

## 5. Empirical Performance and Robustness

Collision cross-entropy demonstrates enhanced robustness to label noise and soft targets across multiple experimental domains:

- **Supervised learning with label corruption:** On the Natural Scene dataset, tan accuracy of collision cross-entropy remains above 80% up to 30% synthetic label corruption rate ($\eta$), while Shannon’s cross-entropy degrades rapidly beyond $\eta \approx 10\%$ [2303.07321].
- **Clustering with fixed features:** Using ResNet-50 features, collision cross-entropy outperforms k-means and other losses on benchmarks such as STL10 (92.3% vs. 85.2%), CIFAR10 (73.5% vs. 67.8%), CIFAR100-20 (43.7% vs. 43.0%), and MNIST (58.4% vs. 47.6%) [2303.07321].
- **End-to-end deep clustering:** With a VGG-4 backbone, collision cross-entropy achieves 95.11% accuracy on MNIST, compared to IIC (82.5%) and MIADM (78.9%). On STL10, performance with self-augmentation is 25.98% (±1.1%), with substantial improvements over reference losses [2303.07321].
- **Transfer with pretext-trained features:** Improvements are observed for both CIFAR10 (83.27% vs. 81.8%) and STL10 (78.12% vs. 75.5%), with corresponding ARI/NMI gains [2303.07321].
- **Weakly-supervised classification:** The method yields gains in low-label regimes; for instance, on STL10, accuracy increases from 26.1% (seed labels only) to 27.2% using collision cross-entropy [2303.07321].

In every evaluated case, collision cross-entropy matches or outperforms Shannon CE and other information-theoretic objectives (IIC, IMSAT, MIADM), exhibiting higher stability under label uncertainty [2303.07321].

## 6. Comparison with Other Information-Theoretic Losses

Collision cross-entropy distinguishes itself from existing information-theoretic losses in several dimensions. Unlike Shannon's cross-entropy, it does not reinforce uncertainty if the target itself is uncertain, making it an attractive loss under weak supervision or noisy label conditions. In deep clustering, where both pseudo-labels and predictions may be ambiguous, its symmetry and selective disregard of maximally uncertain data are advantageous [2303.07321]. Empirical comparisons show superior or comparable performance to IIC, IMSAT, and MIADM losses across both classification and clustering benchmarks.

## 7. Significance and Practical Considerations

The design of collision cross-entropy addresses specific pathologies encountered when training with soft labels, especially the tendency of Shannon’s cross-entropy to propagate uncertainty from noisy or ambiguous supervision into the model. By coupling zero-gradient behavior on fully uncertain targets with symmetric properties suitable for self-labeling, collision cross-entropy offers a robust, computationally efficient, and theoretically grounded alternative for deep learning tasks involving uncertain or estimated label distributions [2303.07321]. The associated EM procedure further enhances practicality by accelerating pseudo-label optimization in clustering settings. A plausible implication is improved model generalization and resilience to annotation noise in weakly labeled or self-supervised machine learning pipelines.

Source: https://www.emergentmind.com/topics/collision-cross-entropy