---
title: 'CLEAR: Contrastive & Anti-contrastive Regularization'
url: https://www.emergentmind.com/topics/contrastive-learning-with-anti-contrastive-regularization-clear
type: topic
---

# CLEAR: Contrastive & Anti-contrastive Regularization

Searching arXiv for the CLEAR paper and closely related contrastive/anti-contrastive work to ground the article in current literature.
arxiv_search(query="CLEAR anti-contrastive regularization style content associations", max_results=10)
arxiv_search(query="Contrastive LEarning with Anti-contrastive Regularization 2507.18794", max_results=10)
arxiv_search(query="AdCo adversarial contrast 2011.08435", max_results=5)
Contrastive Learning with Anti-contrastive Regularization (CLEAR) is a weakly supervised representation-learning framework designed to separate task-relevant **content** from task-irrelevant **style** under distribution shifts in superficial attributes. In CLEAR, a supervised contrastive objective is applied to content latents so that they retain label information, while an anti-contrastive penalty called **Pair-Switching (PS)** is applied to style latents so that they become uninformative about labels. The framework is instantiated in a Variational Auto-Encoder (VAE), yielding CLEAR-VAE, which supports content–style swapping and interpolation and improves downstream prediction when test data contain previously unseen combinations of content and style [2507.18794].

## 1. Problem setting and semantic decomposition

CLEAR assumes that an input \(x\) can be decomposed into **content** \(z^{(c)}\) and **style** \(z^{(s)}\). Content denotes information directly related to the downstream outcome label \(y\), whereas style denotes task-irrelevant superficial attributes that may nevertheless correlate with content in the training distribution [2507.18794]. The central failure mode is a **spurious style–content association**: models trained on observational data can use style as a shortcut for predicting \(y\), then fail when they encounter unseen style combinations at test time.

The framework explicitly targets **distribution shifts in style for fixed content**. Training data cover only a subset of content–style combinations, and the test set contains disjoint style subsets for each content class. Under this protocol, the desired representation should preserve label information in \(z^{(c)}\) while forcing \(z^{(s)}\) to be independent of \(y\) [2507.18794].

The paper illustrates this decomposition on several datasets.

| Dataset | Content label \(y\) | Style |
|---|---|---|
| Styled-MNIST | Digit identity | Corruption type |
| Colored-MNIST | Digit identity | Digit color |
| PACS | Object category | Domain |
| CelebA | Gender \(\times\) smiling | Hair color + other attributes |
| Camelyon17-WILDS | Gender \(\times\) tumor/normal | Staining and hospital/site-specific characteristics |

This formulation is narrower than generic invariance learning. CLEAR does not merely seek robustness to nuisance variation; it specifies a semantic partition of the latent space and uses the supervised label to control which information is encouraged or suppressed. A common misconception is that anti-contrastive regularization eliminates contrastive learning. In CLEAR, the opposite is true: supervised contrastive learning is retained for content, and the anti-contrastive component acts only on style [2507.18794].

## 2. Objective: supervised contrastive content learning and Pair-Switching

CLEAR combines a contrastive objective on \(z^{(c)}\) with a pair-switched anti-contrastive objective on \(z^{(s)}\). For a batch of size \(N\), content latents \(\{z_i^{(c)}\}\), labels \(\{y_i\}\), cosine similarity \(\mathrm{sim}(\cdot,\cdot)\), and temperature \(\tau\), the modified Soft Nearest Neighbor loss is

$$
L_{\text{SNN}}^{(c)} = \frac{1}{N} \sum_{i=1}^{N} -\log \frac{\mathrm{pos}_i^{(c)}}{\mathrm{neg}_i^{(c)} + \mathrm{pos}_i^{(c)}},
$$

with

$$
\mathrm{pos}_i^{(c)} \coloneqq \sum_{j\ne i} \mathbb{I}_{[y_i = y_j]} \exp\left( \frac{\mathrm{sim}(z_i^{(c)}, z_j^{(c)})}{\tau} \right),
$$

$$
\mathrm{neg}_i^{(c)} \coloneqq \sum_{j\ne i} \mathbb{I}_{[y_i \ne y_j]} \exp\left( \frac{\mathrm{sim}(z_i^{(c)}, z_j^{(c)})}{\tau} \right).
$$

Minimizing \(L_{\text{SNN}}^{(c)}\) pulls together same-label content embeddings and pushes apart different-label content embeddings [2507.18794].

A naïve attempt to minimize label information in style would reverse this objective by maximizing a standard style contrastive loss or minimizing its negative. The paper identifies two problems with that route: \(-l_i^{(s)}\) is unbounded below, making the total objective unstable, and minimizing a lower bound does not provide a clean guarantee on \(I(y; z^{(s)})\) [2507.18794].

CLEAR addresses this with **Pair-Switching**, which swaps the role of positive and negative pairs in a stable supervised contrastive form. The practical style loss is

$$
\mathcal{L}_{\text{PS-SNN}}^{(s)} = \frac{1}{N}\sum_{i=1}^{N} -\log\left(\frac{\mathrm{neg}_i^{(s)}}{\mathrm{pos}_i^{(s)} + \mathrm{neg}_i^{(s)}}\right),
$$

where \(\mathrm{pos}_i^{(s)}\) and \(\mathrm{neg}_i^{(s)}\) are defined from \(z^{(s)}\) using the same label partitions as above. This pair-switched form rewards style embeddings that are similar across different labels and therefore unstructured with respect to \(y\) [2507.18794].

The full training loss is

$$
L = L_{\text{VAE}(\beta)} + \alpha_1 L_{\text{SNN}}^{(c)} + \alpha_2 L_{\text{PS-SNN}}^{(s)},
$$

where \(L_{\text{VAE}(\beta)}\) is a \(\beta\)-VAE objective and \(\alpha_1,\alpha_2\) weight the contrastive and anti-contrastive terms. In experiments, the paper typically sets \(\alpha_1=\alpha_2\) [2507.18794].

## 3. Mutual-information interpretation

The theoretical core of CLEAR is an information-theoretic asymmetry: maximize label information in content and minimize label information in style. Under the CLEAR generative model with independent priors for \(z^{(c)}\) and \(z^{(s)}\), the paper derives an ELBO whose final term behaves like a mutual-information term involving \(y\) and \(z^{(c)}\). Using an InfoNCE-style approximation, the paper states that minimizing a content InfoNCE objective corresponds to maximizing \(I(y; z^{(c)})\), and then derives the practical SNN objective from that construction [2507.18794].

For style, the key theoretical claim is stronger. The pair-switched InfoNCE objective is

$$
L_{\text{PS-InfoNCE}}^{(s)} = -\mathbb{E}_{z^{(s)} \sim q_{\phi},~ y \sim p} \left[ \log \frac{\sum_{y' \in \mathcal{Y}\setminus\{y\}} h(y', z^{(s)})}{\sum_{y' \in \mathcal{Y}} h(y', z^{(s)})} \right],
$$

and the paper shows that

$$
L_{\text{PS-InfoNCE}}^{(s)} \ge I(y; z^{(s)}) - \log(N).
$$

It then states that the practical PS-SNN loss is a looser upper bound:

$$
L_{\text{PS-SNN}}^{(s)} > I(y; z^{(s)}) - \log(N).
$$

Accordingly, minimizing \(L_{\text{PS-SNN}}^{(s)}\) directly reduces \(I(y; z^{(s)})\) up to the additive \(-\log N\) constant [2507.18794].

This mutual-information view clarifies the meaning of “anti-contrastive” in CLEAR. The penalty is not an adversarial discriminator on style, nor a generic decorrelation penalty between latent groups. It is a label-conditioned regularizer that explicitly makes the style representation uninformative about \(y\). The paper contrasts this with alternatives such as TC, CLUB, and L1OutUB, which minimize \(I(z^{(c)}; z^{(s)})\) and only indirectly affect \(I(y; z^{(s)})\) through data processing inequality [2507.18794].

The paper also notes a theoretical caveat: the connection between SNN or multi-positive supervised-contrastive losses and mutual information is empirical rather than fully formalized, even though the InfoNCE bound itself is clear [2507.18794]. That limitation matters because CLEAR’s asymmetry is strongest at the PS-InfoNCE level and somewhat looser at the practical PS-SNN level.

## 4. CLEAR-VAE architecture and training procedure

CLEAR is instantiated as **CLEAR-VAE**. The encoder \(q_{\phi}(z\mid x)\) partitions the latent into

$$
z = (z^{(c)}, z^{(s)}),
$$

with independent Gaussian priors

$$
p(z^{(c)}) = \mathcal{N}(0,I), \qquad p(z^{(s)}) = \mathcal{N}(0,I),
$$

and the decoder reconstructs \(x\) from both parts via \(p_{\theta}(x \mid z^{(c)}, z^{(s)})\) [2507.18794]. The VAE component is

$$
L_{\text{VAE}(\beta)} =
- E_{q_{\phi}(z^{(c)}, z^{(s)} \mid x)} [\log p_{\theta}(x \mid z^{(c)}, z^{(s)})]
+ \beta\, D_{\text{KL}}(q_{\phi}(z^{(c)}|x) \| p(z^{(c)}))
+ \beta\, D_{\text{KL}}(q_{\phi}(z^{(s)}|x) \| p(z^{(s)})).
$$

The reconstruction term ensures that \(z^{(c)}\) and \(z^{(s)}\) jointly retain the information needed to reconstruct the input, while the contrastive and anti-contrastive terms determine which semantics populate which subspace [2507.18794].

Pair construction is label-driven rather than augmentation-driven. For content SNN, positive pairs are all pairs with \(y_i=y_j\), and negatives are all pairs with \(y_i\ne y_j\). For style PS-SNN, the same partitions are used, but the numerator favors cross-label similarity. The paper reports that cosine similarity between normalized latent means works better than Mahalanobis or Jeffrey divergences in this setting [2507.18794].

Architecturally, the encoders are CNN stacks of Conv2D–BatchNorm–ReLU blocks, with depth depending on image size; decoders are symmetric ConvTranspose2D–BatchNorm–ReLU stacks. The latent dimension is dataset-dependent, typically between 16 and 64, with a split between content and style dimensions. The paper reports that \(\beta < 1\) gives the best disentanglement behavior, because too large a \(\beta\) over-factorizes individual latent dimensions and harms semantic disentanglement. Values such as \(\tau=0.3\) worked well for SNN and PS-SNN, whereas large \(\tau\) such as 1 can cause loss of detail and numerical instability in adversarial variants [2507.18794].

Ablations on Styled-MNIST show a three-way distinction. A plain \(\beta\)-VAE yields group Mutual Information Gap near zero; an SNN-only model clusters content by labels but leaves content information in style; and the full CLEAR-VAE with SNN plus PS-SNN gives the best separation in t-SNE, with clear label-based clustering in \(z^{(c)}\) and style clusters independent of label in \(z^{(s)}\) [2507.18794]. The paper therefore treats both terms as necessary.

## 5. Representation control, evaluation protocol, and empirical behavior

Because CLEAR separates content and style latents, it enables direct latent manipulation. Given two samples \(x_i\) and \(x_j\), one may decode \((z_i^{(c)}, z_j^{(s)})\) to preserve the content of \(x_i\) while transferring the style of \(x_j\), or swap content by using \((z_j^{(c)}, z_i^{(s)})\) [2507.18794]. The same factorization supports interpolation in either subspace: interpolate \(z^{(c)}\) while fixing \(z^{(s)}\), or interpolate \(z^{(s)}\) while fixing \(z^{(c)}\). The paper reports that content interpolation changes digit identity or face expression and gender while keeping background or facial style constant, whereas style interpolation changes corruption, color, background, hair, or domain while preserving content.

For quantitative evaluation, the paper pre-trains each VAE variant, freezes the encoder, trains a small MLP on \(z^{(c)}\), and evaluates under unseen content–style combinations. For Styled-MNIST, Colored-MNIST, PACS, and CelebA, each content class is assigned a disjoint set of styles at training and testing. For Camelyon17-WILDS, the standard train–test split is used, where test hospitals are out-of-distribution [2507.18794].

On Camelyon17-WILDS, CLEAR-PS outperforms the plain CNN baseline and the LAM domain-generalization baseline on all reported metrics.

| Model | Acc (mean±sd) | AUC | AP |
|---|---:|---:|---:|
| Vanilla CNN | 0.638 ± 0.022 | 0.713 ± 0.030 | 0.667 ± 0.046 |
| LAM | 0.713 ± 0.012 | 0.804 ± 0.025 | 0.791 ± 0.012 |
| CLEAR-PS | **0.747 ± 0.019** | **0.832 ± 0.019** | **0.804 ± 0.027** |

The paper also reports that contrastive VAE encoders consistently outperform group-based VAEs such as MLVAE and GVAE in out-of-distribution settings, with larger gains when training sees fewer styles per content class or when style–content correlation is high [2507.18794].

To quantify semantic disentanglement, the paper introduces **group Mutual Information Gap (gMIG)**,

$$
\mathrm{gMIG}(y) = \frac{1}{H(y)} \left( \frac{1}{d_c} \sum_{j=1}^{d_c} I(z_j^{(c)};y) - \frac{1}{d_s} \sum_{j=1}^{d_s} I(z_j^{(s)};y) \right).
$$

High gMIG means the content group carries more label information than the style group. The paper reports that a plain \(\beta\)-VAE tends to have gMIG near zero, while CLEAR variants have significantly higher gMIG, indicating successful semantic-level disentanglement [2507.18794].

These results support a specific interpretation of robustness. CLEAR is not merely learning invariant features in the usual contrastive sense; it is reallocating predictive and nuisance information across two latent groups. A plausible implication is that the method is especially appropriate when “style” corresponds to demographic, institutional, or domain variables that are spuriously correlated with the label in training data.

## 6. Relation to adjacent contrastive methods, disambiguation, and limitations

CLEAR belongs to a broader family of methods that modify contrastive geometry, but its mechanism is distinct. AdCo learns a global set of trainable negative adversaries in a min–max game, thereby changing the **negative distribution** itself; CLEAR instead keeps a supervised contrastive objective on content and applies an anti-contrastive regularizer to the **style subspace**, with the explicit goal of minimizing \(I(y; z^{(s)})\) [2011.08435]. This difference is substantive: AdCo is feature-level adversarial negative learning, whereas CLEAR is semantic subspace allocation.

Other related methods illuminate adjacent design choices. “Contrastive Regularization for Semi-Supervised Learning” uses pseudo-label-aware attraction and repulsion to propagate label information through unlabeled clusters [2201.06247]. “Context-Enriched Contrastive Loss” introduces a self-based regularizer that counters label-driven distortion by insisting that two views of the same instance remain the closest pair [2512.02152]. “Your contrastive learning problem is secretly a distribution alignment problem” recasts CLEAR-like anti-contrastive regularization as modified target transport plans, modified costs, or unbalanced optimal transport that allow non-zero off-diagonal mass for selected pairs [2502.20141]. “Contrasting the landscape of contrastive and non-contrastive learning” argues that repulsive structure is important because non-contrastive objectives admit abundant non-collapsed bad minima [2203.15702]. “Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning” shows that contrastive regularization can also be useful at fine-tuning time, especially for reducing intra-class scattering inherited from instance-discriminative pretraining [2102.06605].

The term **CLEAR** is itself overloaded. The 2020 paper “CLEAR: Contrastive Learning for Sentence Representation” studies sentence-level contrastive pre-training with textual augmentations and does not introduce anti-contrastive regularization [2012.15466]. In current usage, “CLEAR” may therefore refer either to the sentence-representation method or to the 2025 style–content disentanglement framework; the latter is the subject here.

The 2025 CLEAR paper also states several limitations. The method is currently implemented only with VAEs on image data; its theoretical mutual-information account is strongest for InfoNCE and less formal for multi-positive SNN; and the framework assumes that content is well captured by the available supervised label and that style is ideally independent of that label in an unbiased population [2507.18794]. These assumptions delimit the scope of the method. When the content–style split is ill-defined, or when the provided label is itself a biased proxy, the semantics of anti-contrastive regularization become less clear.

Within those limits, CLEAR provides a concrete instantiation of a broader idea in contemporary contrastive learning: representation quality can be improved not only by strengthening attraction and repulsion, but also by specifying **where** label information should reside and **where it should not**.

Source: https://www.emergentmind.com/topics/contrastive-learning-with-anti-contrastive-regularization-clear