Beyond I-Con: Divergence–Kernel Framework

• Beyond I-Con is a representation-learning framework that generalizes I-Con by replacing the KL divergence with alternative statistical divergences, offering a two-dimensional design space over divergences and kernels.
• It jointly optimizes divergence and kernel choices to improve performance in tasks such as clustering, supervised contrastive learning, and dimensionality reduction.
• Empirical results demonstrate that bounded divergences like TV, JSD, and Hellinger yield more stable gradients and superior performance compared to traditional KL-based methods.

Beyond I-Con is a representation-learning framework that extends the Information Contrastive (I-Con) view of learning as conditional neighborhood-distribution matching. In I-Con, many methods are written as minimizing an integrated Kullback–Leibler divergence between a data-induced or supervisory neighborhood distribution $p(\cdot\mid i)$ and a learned neighborhood distribution $q_\phi(\cdot\mid i)$. Beyond I-Con preserves that conditional-distribution perspective but replaces the KL divergence with alternative statistical divergences and treats the similarity kernel used to construct $q_\phi$ as a second, explicit design axis. In this formulation, clustering, supervised contrastive learning, and dimensionality reduction are no longer tied to a single KL-based geometry; they become instances of a broader two-dimensional design space over divergences and kernels (Shone et al., 5 Sep 2025, Alshammari et al., 23 Apr 2025).

1. Origins in the I-Con framework

I-Con introduced a unifying claim: a large collection of modern representation-learning objectives can be understood as minimizing an integrated KL divergence between two conditional distributions over neighbors, one supervisory and one learned (Alshammari et al., 23 Apr 2025). In that formulation, the central loss is

$$\mathcal{L}_{\text{I-Con}} = \mathbb{E}_{i \sim p(i)} \left[ \mathcal{D}_{\mathrm{KL}}\bigl(p(\cdot \mid i)\,\|\, q(\cdot \mid i)\bigr) \right].$$

The anchor index $i$ identifies a data point, $p(\cdot\mid i)$ specifies which other points should count as neighbors, and $q_\phi(\cdot\mid i)$ specifies which points the learned representation treats as neighbors.

Beyond I-Con begins from the observation that this unification is descriptive, not normative. The fact that many methods can be rewritten as KL minimization does not imply that KL is the uniquely appropriate divergence for representation learning. The framework therefore reopens a design choice that I-Con had largely standardized: how mismatch between $p(\cdot\mid i)$ and $q_\phi(\cdot\mid i)$ should be measured. It also emphasizes that divergence choice cannot be separated from the parameterization of learned similarities, because the same divergence can behave differently under different kernels (Shone et al., 5 Sep 2025).

This shift changes the interpretation of I-Con. Instead of a final common form, KL becomes one member of a larger family. A common misconception addressed by Beyond I-Con is that once a method is cast in the I-Con language, its optimization geometry is effectively fixed. The paper’s position is the opposite: once neighborhood matching is recognized as the underlying abstraction, divergence and kernel choice become the central design variables.

2. General formulation and mathematical structure

Beyond I-Con replaces the KL divergence in the original framework with a general positive definite divergence $\mathcal{D}$, yielding

$q_\phi(\cdot\mid i)$0

The meanings of the constituent objects remain unchanged: $q_\phi(\cdot\mid i)$1 is the anchor point, $q_\phi(\cdot\mid i)$2 is the distribution over anchors, $q_\phi(\cdot\mid i)$3 is the target neighborhood distribution, and $q_\phi(\cdot\mid i)$4 is the learned neighborhood distribution (Shone et al., 5 Sep 2025).

The learned conditional is induced by a softmax-normalized similarity kernel,

$q_\phi(\cdot\mid i)$5

The two emphasized choices are an angular-based kernel, $q_\phi(\cdot\mid i)$6, and a distance-based kernel, $q_\phi(\cdot\mid i)$7. The first is standard in contrastive learning; the second is more naturally aligned with clustering and neighborhood embedding.

The framework explicitly studies divergences “most directly comparable to KL”: KL divergence, total variation (TV), Jensen–Shannon divergence (JSD), and Hellinger distance. In the technical synthesis, these are given in their standard forms. For KL,

$q_\phi(\cdot\mid i)$8

For TV,

$q_\phi(\cdot\mid i)$9

For JSD,

$q_\phi$0

For squared Hellinger,

$q_\phi$1

The conceptual consequence is that representation learning becomes a problem of matching local geometries under a chosen discrepancy measure. The original I-Con program established the neighborhood-distribution viewpoint; Beyond I-Con generalizes the discrepancy geometry within that viewpoint.

3. Divergence properties and kernel interactions

The main criticism of KL as a default choice is threefold. First, KL is asymmetric: $q_\phi$2 Second, it is unbounded above and diverges when $q_\phi$3 while $q_\phi$4. Third, there is no reason it must be the best surrogate for downstream objectives such as clustering quality, classification accuracy, or useful low-dimensional geometry (Shone et al., 5 Sep 2025).

These properties have direct optimization implications. The paper argues that KL can generate large gradients and early training spikes, and can over-penalize cases where the learned model assigns very small mass to a true neighbor. By contrast, TV, JSD, and Hellinger are symmetric or bounded, less sensitive to support mismatch, and more stable when tiny learned probabilities arise. In the technical synthesis, the paper specifically connects this boundedness to more stable gradient behavior and less crowding in dimensionality reduction.

A second major point is that divergence and kernel must be analyzed jointly. The paper reports a particularly sharp example in supervised contrastive learning: KL with an angular kernel is strong and stable, but KL with a distance-based kernel collapses during training, whereas TV with that same distance-based kernel performs best. This suggests that the failure mode is not reducible to “bad divergence” or “bad kernel” separately; it arises from their interaction.

This interaction is central to the framework’s interpretive value. It implies that traditional pairings such as KL plus cosine-like similarity may reflect historical stability rather than theoretical optimality. Beyond I-Con treats those pairings as design conventions rather than fixed principles.

4. Task-specific instantiations

Beyond I-Con is instantiated in three principal settings: unsupervised clustering, supervised contrastive learning, and dimensionality reduction. In each case, the modification is largely at the loss level: the underlying encoder or feature extractor is retained, the supervisory distribution $q_\phi$5 is kept or only lightly modified, and the divergence and optionally the kernel are changed (Shone et al., 5 Sep 2025).

In unsupervised clustering, the paper modifies the Pointwise Mutual Information (PMI) clustering algorithm of Adaloglou et al. The baseline is framed as an I-Con-style KL objective over neighborhood distributions, while the proposed variant replaces KL with TV: $q_\phi$6 The setup uses pretrained DINO ViT embeddings and trains a linear classifier on top of those embeddings.

In supervised contrastive learning, the paper begins from the standard SupCon objective associated with Khosla et al., which is interpreted as a KL/I-Con-type objective with an angular similarity kernel. Beyond I-Con then varies both the divergence and the kernel. Its best-performing configuration uses TV with a distance-based kernel: $q_\phi$7 and the corresponding loss is

$q_\phi$8

When $q_\phi$9 is uniform over same-class positives and zero otherwise, this becomes an explicit TV discrepancy between a class-conditional target neighborhood and a Gaussian-neighborhood learned distribution.

In dimensionality reduction, the framework reinterprets Stochastic Neighbor Embedding (SNE) as an I-Con neighborhood-matching objective and replaces its KL divergence with bounded alternatives: $$\mathcal{L}_{\text{I-Con}} = \mathbb{E}_{i \sim p(i)} \left[ \mathcal{D}_{\mathrm{KL}}\bigl(p(\cdot \mid i)\,\|\, q(\cdot \mid i)\bigr) \right].$$0 The argument is that crowding in low-dimensional embeddings is not only a kernel issue, as in t-SNE, but also a divergence issue. Beyond I-Con therefore offers an alternative route to improved embedding geometry: retain the neighborhood-matching structure but alter the discrepancy measure.

5. Experimental results and reported performance

The reported empirical results are task-dependent and support the claim that there is no universal best divergence; rather, the optimum depends on the task and on the similarity kernel (Shone et al., 5 Sep 2025).

Setting	Best reported configuration	Reported result
ImageNet-1K clustering with DINO ViT embeddings	TV in PMI-style clustering	$$\mathcal{L}_{\text{I-Con}} = \mathbb{E}_{i \sim p(i)} \left[ \mathcal{D}_{\mathrm{KL}}\bigl(p(\cdot \mid i)\,\|\, q(\cdot \mid i)\bigr) \right].$$1 on ViT-B/14, $$\mathcal{L}_{\text{I-Con}} = \mathbb{E}_{i \sim p(i)} \left[ \mathcal{D}_{\mathrm{KL}}\bigl(p(\cdot \mid i)\,\|\, q(\cdot \mid i)\bigr) \right].$$2 on ViT-L/14
CIFAR-10 supervised contrastive learning with ResNet-50	TV + distance-based kernel	$$\mathcal{L}_{\text{I-Con}} = \mathbb{E}_{i \sim p(i)} \left[ \mathcal{D}_{\mathrm{KL}}\bigl(p(\cdot \mid i)\,\|\, q(\cdot \mid i)\bigr) \right].$$3 linear probe / $$\mathcal{L}_{\text{I-Con}} = \mathbb{E}_{i \sim p(i)} \left[ \mathcal{D}_{\mathrm{KL}}\bigl(p(\cdot \mid i)\,\|\, q(\cdot \mid i)\bigr) \right].$$4-NN
CIFAR-10 dimensionality reduction with SNE-style model	JSD replacing KL	$$\mathcal{L}_{\text{I-Con}} = \mathbb{E}_{i \sim p(i)} \left[ \mathcal{D}_{\mathrm{KL}}\bigl(p(\cdot \mid i)\,\|\, q(\cdot \mid i)\bigr) \right].$$5 linear probe / $$\mathcal{L}_{\text{I-Con}} = \mathbb{E}_{i \sim p(i)} \left[ \mathcal{D}_{\mathrm{KL}}\bigl(p(\cdot \mid i)\,\|\, q(\cdot \mid i)\bigr) \right].$$6-NN

In unsupervised clustering on ImageNet-1K, the reported metric is Hungarian Accuracy. The paper lists DINO ViT-S/14, ViT-B/14, and ViT-L/14 as backbones. TV achieves the best reported results on ViT-B/14 and ViT-L/14, with $$\mathcal{L}_{\text{I-Con}} = \mathbb{E}_{i \sim p(i)} \left[ \mathcal{D}_{\mathrm{KL}}\bigl(p(\cdot \mid i)\,\|\, q(\cdot \mid i)\bigr) \right].$$7 and $$\mathcal{L}_{\text{I-Con}} = \mathbb{E}_{i \sim p(i)} \left[ \mathcal{D}_{\mathrm{KL}}\bigl(p(\cdot \mid i)\,\|\, q(\cdot \mid i)\bigr) \right].$$8, respectively. On ViT-S/14, TV does not surpass the strongest baseline. The state-of-the-art claim is therefore specific to the larger DINO-ViT embeddings.

In supervised contrastive learning on CIFAR-10 with a ResNet-50 backbone, the evaluation is by linear probing and $$\mathcal{L}_{\text{I-Con}} = \mathbb{E}_{i \sim p(i)} \left[ \mathcal{D}_{\mathrm{KL}}\bigl(p(\cdot \mid i)\,\|\, q(\cdot \mid i)\bigr) \right].$$9-NN classification. KL with an angular-based kernel yields $i$0, but TV with a distance-based kernel improves this to $i$1. The gain over KL plus angular is $i$2 points for linear probing and $i$3 points for $i$4-NN. The most notable negative result is that KL with a distance-based kernel collapses: across five seeds, validation accuracy rises initially and then crashes to 10–20%.

In dimensionality reduction on CIFAR-10 with a CNN architecture, vanilla KL-SNE achieves $i$5 linear-probe accuracy and $i$6 $i$7-NN accuracy at $i$8. TV improves these to $i$9 and $p(\cdot\mid i)$0, Hellinger to $p(\cdot\mid i)$1 and $p(\cdot\mid i)$2, and JSD to $p(\cdot\mid i)$3 and $p(\cdot\mid i)$4. The qualitative figure description states that KL produces highly overlapping categories, whereas TV, Hellinger, and JSD produce visibly better separation. The paper also reports that KL exhibits large gradient-norm spikes near initialization, while bounded divergences show more stable gradient behavior across all network layers.

Taken together, these experiments support two factual conclusions advanced by the paper. First, divergence choice materially affects final representation quality. Second, the best divergence depends on the kernel and on the task class.

6. Interpretation, limitations, and research significance

Beyond I-Con is primarily empirical and conceptual rather than theorem-heavy. The paper does not present formal convergence theorems or an exhaustive theory of why particular divergence–kernel pairings succeed. Its theoretical argument is instead optimization-centered: when learned neighborhood distributions assign extremely small probabilities, bounded divergences such as TV, JSD, and Hellinger may provide smoother and more robust signals than KL (Shone et al., 5 Sep 2025).

This leaves an important open question. The paper clearly demonstrates that divergence and kernel choices interact nontrivially, but it also explicitly states that the theory explaining this interaction is incomplete. A rigorous treatment of gradient dynamics, loss-surface geometry, and optimization landscapes is identified as a limitation. The strongest failure case presently documented is KL with a distance-based kernel in supervised contrastive learning, which shows that arbitrary divergence–kernel combinations should not be assumed stable.

The practical significance of the framework lies in its reformulation of loss design. Historically, much work in representation learning emphasized the choice of positives and negatives, or the design of similarity kernels, while treating KL or cross-entropy-like discrepancies as fixed. Beyond I-Con argues that divergence selection is equally fundamental. In that sense, it converts the I-Con unification into a search space for new objectives.

A plausible implication is that future work will not treat neighborhood matching as synonymous with KL matching. Instead, the neighborhood-distribution viewpoint may persist while the underlying geometry of mismatch becomes task-specific. The paper’s broader claim is therefore not that TV, JSD, or Hellinger should replace KL everywhere, but that representation learning should be understood as the joint design of $p(\cdot\mid i)$5, the kernel that induces $p(\cdot\mid i)$6, and the divergence $p(\cdot\mid i)$7 that compares them. That is the specific sense in which Beyond I-Con goes beyond the original I-Con framework.