Clear-VAE: Weakly Supervised Disentanglement

• Clear-VAE is a method that integrates contrastive and anti-contrastive losses in a VAE to separate task-relevant content from task-irrelevant style.
• It uses supervised contrastive learning for content and a Pair-Switching loss for style to reduce bias and improve robustness to distribution shifts.
• Empirical studies on datasets like PACS and Camelyon17-WILDS demonstrate effective semantic manipulation and robust performance in out-of-distribution scenarios.

Searching arXiv for the target paper and closely related references to ground the article. Clear-VAE denotes the instantiation of CLEAR—Contrastive LEarning with Anti-contrastive Regularization—in the latent space of a Variational Auto-Encoder (VAE). It is designed for weakly supervised disentanglement of task-relevant content and task-irrelevant style, using only content or outcome labels and requiring neither style labels nor paired data. The framework is motivated by settings in which models learn spurious associations between content and superficial attributes, so that previously unseen combinations of content and style at test time degrade downstream performance; the paper presents CLEAR-VAE as a means to support both semantic manipulation of latent factors and more robust prediction under such shifts (Sun et al., 24 Jul 2025).

1. Problem setting and conceptual basis

Clear-VAE is formulated around a semantic partition of representations into content features and style features. In the terminology used by the source paper, content features are task-relevant and contain information needed for downstream prediction, whereas style features are task-irrelevant and capture superficial attributes such as race, sex, color, texture, background, or site artifacts. The central concern is that training data may contain associations between these two kinds of factors that do not generalize, even when they appear predictive in-distribution (Sun et al., 24 Jul 2025).

The motivating examples are especially explicit in healthcare. The paper states that one may wish to learn features that contain information about pathology yet are unaffected by race, sex, and other sources of physiologic variability, so that predictions remain equitable and generalizable across demographics. In this framing, Clear-VAE is not merely a latent-variable model for interpretability; it is a representation-learning approach targeted at distribution shift induced by changes in superficial characteristics.

A central clarification in the paper is that prior unsupervised disentanglement methods such as $\beta$-VAE and FactorVAE do not explicitly tie latent content to the supervised task. Consequently, content is not reliably identified in the representation, and disentanglement by itself does not necessarily improve downstream supervised performance if the learned factors fail to align with labels. The paper also contrasts its approach with doubly supervised methods that require labels for both content and style, arguing that such style labels are usually rare. Clear-VAE is therefore positioned as a practical form of weak supervision: labels are used only for the task-relevant factor, while style is discouraged from carrying label information.

2. CLEAR objective: contrastive attraction and anti-contrastive repulsion

The CLEAR framework combines two regularization objectives with opposite roles. The first is a contrastive learning term applied to the content latent variable $z^{(c)}$, and the second is an anti-contrastive term applied to the style latent variable $z^{(s)}$ (Sun et al., 24 Jul 2025).

For content, CLEAR seeks to maximize the mutual information between the label $y$ and the content representation $z^{(c)}$. The paper implements this with a Supervised Contrastive Loss, described as based on Soft Nearest Neighbor (SNN) or InfoNCE. Given a batch $\{(x_i,y_i)\}_{i=1}^N$, the model encodes each sample into $z_i^{(c)}$ and minimizes

$$\mathcal{L}_{\mathrm{SNN}^{(c)}}= \frac{1}{N}\sum_{i=1}^{N} -\log \frac{\mathrm{pos}_i^{(c)}}{\mathrm{pos}_i^{(c)}+\mathrm{neg}_i^{(c)}},$$

where

$$\mathrm{pos}_i^{(c)}= \sum_{j\neq i}\mathbb{I}[y_i=y_j] \exp\left(\frac{\mathrm{sim}(z_i^{(c)},z_j^{(c)})}{\tau}\right),$$

and

$$\mathrm{neg}_i^{(c)}= \sum_{j\neq i}\mathbb{I}[y_i\neq y_j] \exp\left(\frac{\mathrm{sim}(z_i^{(c)},z_j^{(c)})}{\tau}\right).$$

Here, $z^{(c)}$0 is cosine similarity and $z^{(c)}$1 is the temperature. Operationally, this draws samples with the same content label together and pushes samples with different labels apart.

For style, CLEAR introduces Pair-Switching (PS), an anti-contrastive penalty intended to minimize the mutual information between the label $z^{(c)}$2 and the style representation $z^{(c)}$3. The key construction is the Pair-Switching SNN Loss:

$z^{(c)}$4

The paper describes this as swapping the roles of positive and negative pairs relative to the content loss. Minimizing this term encourages style representations for samples with the same label not to cluster together, thereby removing label information from style. This makes CLEAR distinctive among weakly supervised disentanglement methods: instead of learning style indirectly through reconstruction alone, it explicitly regularizes style to be uninformative for prediction.

3. VAE instantiation and learning objective

Clear-VAE realizes CLEAR within a VAE latent space by factorizing the encoder output into content and style components. The encoder maps $z^{(c)}$5 and models

$z^{(c)}$6

with the two components treated as independent (Sun et al., 24 Jul 2025). The decoder reconstructs $z^{(c)}$7 from the concatenated latent vector $z^{(c)}$8.

The full training objective is

$z^{(c)}$9

where $z^{(s)}$0 is the usual $z^{(s)}$1-VAE loss:

$z^{(s)}$2

Within this objective, $z^{(s)}$3 encourages $z^{(s)}$4 to be predictive of $z^{(s)}$5, while $z^{(s)}$6 encourages $z^{(s)}$7 to be independent of $z^{(s)}$8. The coefficients $z^{(s)}$9 and $y$0 are weighting parameters and are described as typically set equal for simplicity.

Architecturally, the paper notes that the encoder and decoder use standard convolutional layers, with details deferred to supplementary material. It also emphasizes an implementation distinction: no additional branches or paired data are required, and no discriminator or auxiliary network is needed for PS-based mutual-information minimization. The stated implication is faster and simpler training relative to adversarial or variational mutual-information estimators. This suggests that Clear-VAE is intended as a comparatively direct alternative to methods that enforce invariance through extra estimation networks.

4. Theoretical characterization of Pair-Switching

A central theoretical claim of the paper is that the PS loss provides an upper bound on the mutual information between the style representation and the content label (Sun et al., 24 Jul 2025). Specifically, the appendix is said to prove that

$y$1

Under this result, minimizing the PS loss minimizes an upper bound on $y$2. In the paper’s interpretation, this makes the anti-contrastive regularizer a direct mechanism for suppressing spurious label information in style.

This theoretical perspective differentiates Clear-VAE from disentanglement approaches that rely on total-correlation penalties or from supervised invariance methods that approximate mutual information through adversarial training or auxiliary estimators. The paper further states that the anti-contrastive PS term is as effective as more complex mutual-information estimators, but simpler and more stable. Because no auxiliary estimator is introduced, the optimization target remains integrated into the primary latent geometry rather than delegated to a separate critic.

A related clarification concerns what the paper treats as disentanglement. The goal is not solely latent-factor independence in an unsupervised sense, but a supervised asymmetry: content should retain information about $y$3, whereas style should discard it. In that sense, Clear-VAE is organized around task-aligned disentanglement rather than generic factor decomposition. This suggests that its notion of disentanglement is operationally tied to downstream robustness, not only to factor interpretability.

5. Empirical behavior: disentanglement, manipulation, and OOD prediction

The paper evaluates Clear-VAE on Styled-MNIST, Colored-MNIST, PACS, CelebA, and Camelyon17-WILDS (Sun et al., 24 Jul 2025). The experiments are designed both to inspect the semantics of the learned latent variables and to test downstream prediction when content-style associations shift at evaluation time.

On the generative side, the paper reports that Clear-VAE allows swapping and interpolation of content and style between any pair of samples. In swapping experiments, combining the content code $y$4 of one sample with the style code $y$5 of another yields decoded images with content from the first and style from the second. In interpolation experiments, the model produces smooth transitions in either content or style while holding the other fixed. The qualitative outcome is described as semantically meaningful manipulation of factors such as digit identity, gender, background, color, and texture, with no leakage across factors.

For downstream classification, the training and evaluation protocol uses observed combinations of $y$6 during training and unseen style-content combinations at test time, i.e., an out-of-distribution (OOD) setting. The paper states that CNN and vanilla VAE baselines suffer large drops in accuracy, AUROC, and Average Precision on these OOD splits. By contrast, Clear-VAE achieves the highest performance or close to best across datasets and is described as particularly robust to style shift because content is not contaminated with style.

The paper also reports state-of-the-art OOD generalization on challenging pathology and domain generalization datasets such as Camelyon17-WILDS, where Clear-VAE is said to outperform other VAE variants and specifically tailored domain generalization methods. Since the article’s source does not reproduce the full comparison table, the exact margin is not specified here, but the stated finding situates Clear-VAE as a method for predictive robustness rather than only latent controllability.

6. Quantitative disentanglement, ablations, and significance

For quantitative evaluation of disentanglement, the paper uses the Group Mutual Information Gap (gMIG), defined as

$y$7

This metric measures how strongly the content variables encode the label relative to the style variables. The paper states that Clear-VAE achieves much higher gMIG than plain $y$8-VAE and is competitive with or better than other disentangling approaches. It further summarizes the observed values by stating that Clear-VAE achieves gMIG $y$9–$z^{(c)}$0 on various datasets (Sun et al., 24 Jul 2025).

Ablation studies are used to argue that both regularization components are necessary. Removing either the contrastive content term or the anti-contrastive style term reduces separation and clustering structure in latent space, as illustrated by t-SNE plots and decreases in gMIG. The paper’s conclusion is explicit: both terms are essential for full semantic disentanglement and robust generalization.

The broader significance claimed by the paper centers on fairness and robustness. Because the method seeks to make predictions invariant to style, it is presented as supporting equitable performance across subpopulations and as being practical in real-world settings where only outcome labels are available. A plausible implication is that Clear-VAE belongs to a line of methods that treat disentanglement not as an end in itself but as a mechanism for controlling spurious correlations under domain shift. At the same time, the paper’s claims remain specific to the reported image benchmarks and to the supervised signal provided by content labels; Clear-VAE is not presented as a fully unsupervised disentanglement method, and its guarantees are tied to the assumptions encoded in the CLEAR objective.