---
title: Contrastive Variational Autoencoder (cVAE)
url: https://www.emergentmind.com/topics/contrastive-variational-autoencoder-cvae
type: topic
---

# Contrastive Variational Autoencoder (cVAE)

A Contrastive Variational Autoencoder (cVAE) is a generative modeling framework that extends variational autoencoding by incorporating explicit contrastive mechanisms to isolate, disentangle, or enrich latent factors that are uniquely salient in a "target" dataset relative to a "background" or to a set of negative examples. cVAE models provide a principled and scalable approach for contrastive representation learning within the probabilistic modeling paradigm, and are especially applied in settings ranging from biomedicine to sequential recommendation and speech processing. This paradigm encompasses multiple architectural and training variants, united by the common goal of separating shared from salient sources of variation via contrastive objectives or explicit dataset splitting.

## 1. Motivations and Contrastive Principle

The cVAE paradigm is motivated by scientific and practical tasks where one wishes to extract variation present in a "target" dataset that is absent or diminished in a "background" or reference dataset, or simply disentangle distinct latent factors within observed data. Standard VAEs, while effective at reconstructing data and learning compressed representations, tend to model the dominant axes of variation in the marginal data distribution, potentially ignoring rare, task-salient, or group-specific factors. Contrastive learning reframes this challenge by leveraging a reference distribution (or an implicit notion of negative examples) to "subtract" away nuisance variation, forcing the model to allocate representational capacity to the target-relevant signals [1902.04601, 2202.10560].

## 2. Formal cVAE Generative Models

At the core of classical cVAE formulations is an explicit separation of latent variables into components that are "shared" across both datasets and "salient" to the target dataset. The canonical generative model is as follows [1902.04601]:

- For target sample $x_i$: 
  $$
  s_i \sim \mathcal{N}(0,I),\quad z_i \sim \mathcal{N}(0,I), \quad x_i \sim p_\theta(x \mid s_i, z_i)
  $$
- For background sample $b_j$: 
  $$
  z_j' \sim \mathcal{N}(0,I),\quad b_j \sim p_\theta(b \mid 0, z_j')
  $$

Here, $s$ denotes salient (target-specific) latent factors, and $z$ denotes shared latent factors. The decoder receives both $s$ and $z$ for target samples, but only $z$ (with $s$ fixed to zero) for background data, enforcing that background samples cannot utilize the salient subspace for reconstruction [1902.04601, 2202.10560].

Generalizations include structured Gaussian mixture priors, split-branch architectures for other modalities, and sequence settings with static/dynamic partitions in sequential data [2110.12091, 2005.12963].

## 3. Learning Objectives and Contrastive Regularization

cVAE models optimize an augmented ELBO that includes components for both target and background samples:

- For target:
  $$
  \mathcal{L}_{\mathrm{target}}(x_i) = \mathbb{E}_{q_{\phi_s}(s \mid x_i) q_{\phi_z}(z \mid x_i)} [\log p_\theta(x_i \mid s, z)] - \mathrm{KL}[q_{\phi_s}(s \mid x_i) \| p(s)] - \mathrm{KL}[q_{\phi_z}(z \mid x_i) \| p(z)]
  $$
- For background:
  $$
  \mathcal{L}_{\mathrm{bg}}(b_j) = \mathbb{E}_{q_{\phi_z}(z \mid b_j)} [\log p_\theta(b_j \mid 0, z)] - \mathrm{KL}[q_{\phi_z}(z \mid b_j) \| p(z)]
  $$

The ELBO is then summed over both datasets. Explicit independence between $s$ and $z$ is enforced via total correlation penalties (estimated by the density ratio trick or MMD) [2202.10560, 2307.06206]. More advanced variants

Source: https://www.emergentmind.com/topics/contrastive-variational-autoencoder-cvae