---
title: Contrastive Log-Ratio Upper Bound (CLUB)
url: https://www.emergentmind.com/topics/contrastive-log-ratio-upper-bound-club
type: topic
---

# Contrastive Log-Ratio Upper Bound (CLUB)

The Contrastive Log-ratio Upper Bound (CLUB) is a framework for estimating and minimizing mutual information (MI) in high-dimensional scenarios where only samples from the relevant distributions, rather than their explicit forms, are available. CLUB constructs a tight, variationally-approximable upper bound on MI that enables stable and scalable MI minimization—tasks where lower bound estimators are inapplicable. The CLUB methodology addresses the estimation bias, computational scalability, and numerical instability that afflict earlier MI upper bound strategies, thereby facilitating its use in representation learning, domain adaptation, and information bottleneck contexts [2006.12013].

## 1. Formal Definition of CLUB

Let $(X, Y)$ be random variables with joint distribution $p(x, y)$. Mutual information is defined as
\[
I(X; Y) = \mathbb{E}_{p(x, y)} \left[ \log \frac{p(x, y)}{p(x)p(y)} \right] = \mathbb{E}_{p(x, y)} \left[ \log \frac{p(x \mid y)}{p(x)} \right].
\]
The CLUB estimator presumes access to the conditional density $p(x \mid y)$ and defines
\[
I_{\mathrm{CLUB}}(X; Y) = \mathbb{E}_{p(x, y)} [\log p(x \mid y)] - \mathbb{E}_{p(x)p(y)} [\log p(x \mid y)].
\]
Given $N$ i.i.d. samples $\{(x_i, y_i)\}_{i=1}^N$, the empirical estimate is
\[
\hat I_{\mathrm{CLUB}} = \frac{1}{N} \sum_{i=1}^N \log p(x_i | y_i) - \frac{1}{N^2} \sum_{i=1}^N \sum_{j=1}^N \log p(x_j | y_i) = \frac{1}{N^2} \sum_{i, j} \left[\log p(x_i | y_i) - \log p(x_j | y_i)\right].
\]
This estimator relies on the explicit evaluation or modeling of $p(x|y)$, and operates by contrasting log-likelihoods across matched and mismatched pairs.

## 2. Upper Bound Derivation and Tightness

Define the gap $\Delta = I_{\mathrm{CLUB}}(X; Y) - I(X; Y)$. The derivation proceeds as follows:
\[
\Delta = \mathbb{E}_{p(x,y)}\log p(x) - \mathbb{E}_{p(x)p(y)}\log p(x|y) 
= \mathbb{E}_{p(y)}\left[\log p(x) - \mathbb{E}_{p(x)}\log p(x|y)\right].
\]
By concavity of $\log$ and Jensen’s inequality, $\log p(x) = \log \mathbb{E}_{p(y)}[p(x|y)] \geq \mathbb{E}_{p(y)}[\log p(x|y)]$, guaranteeing that $\Delta \geq 0$. Thus,
\[
I(X; Y) \leq I_{\mathrm{CLUB}}(X; Y),
\]
with equality if and only if $p(x|y)$ is independent of $y$, i.e., $X \perp Y$. The bound is tight for independent variables and grows otherwise; the magnitude reflects the deviation from independence.

## 3. Variational Approximation: vCLUB

When $p(x|y)$ is unknown or intractable, CLUB is made practical via a variational approximation. A conditional model $q_\theta(x|y)$ (typically parameterized by neural networks) is introduced:
\[
I_{\mathrm{vCLUB}}(X; Y) = \mathbb{E}_{p(x, y)}[\log q_\theta(x|y)] - \mathbb{E}_{p(x)p(y)}[\log q_\theta(x|y)],
\]
with the sample estimator
\[
\hat I_{\mathrm{vCLUB}} = \frac{1}{N^2} \sum_{i, j} [\log q_\theta(x_i | y_i) - \log q_\theta(x_j | y_i)].
\]
For parametric exponential family forms, such as Gaussians with mean $\mu_\theta(y)$ and diagonal covariance, this estimator admits closed-form computations based on Mahalanobis distances.

## 4. Theoretical Properties and Bias Analysis

The theoretical guarantees of CLUB are as follows:
- **Exactness**: $I(X; Y) \leq I_{\mathrm{CLUB}}(X; Y)$, with equality if and only if $X \perp Y$.
- **vCLUB as Upper Bound**: Let $q_\theta(x, y) = p(x)q_\theta(x|y)$. If $\mathrm{KL}(p(x, y)\|q_\theta(x, y)) \leq \mathrm{KL}(p(x)p(y)\|q_\theta(x, y))$, then $I(X;Y) \leq I_{\mathrm{vCLUB}}(X;Y)$. 
- **Approximation Error**: If, in addition, $\mathrm{KL}(p(x|y)\|q_\theta(x|y)) \leq \epsilon$ and $\mathrm{KL}(p(x, y)\|q_\theta(x, y)) > \mathrm{KL}(p(x)p(y)\|q_\theta(x, y))$, then $|I(X;Y) - I_{\mathrm{vCLUB}}(X;Y)| < \epsilon$.

A plausible implication is that strong variational modeling of $p(x|y)$ ensures not only the validity but the tightness of the CLUB upper bound relative to the true mutual information.

## 5. Scalable MI Minimization Training: Negative Sampling Scheme

The original estimator is quadratic in $N$. CLUB–S and vCLUB–S accelerate computation via negative sampling, lowering the complexity to $O(N)$. Training involves two main phases:
1. **Conditional Model Update**: Fit $q_\theta(x|y)$ to maximize conditional log-likelihood over data batches.
2. **MI Minimization via Negative Sampling**: For each positive $(x_i, y_i)$ pair, one negative is sampled, constructing $U_i = \log q_\theta(x_i|y_i) - \log q_\theta(x_{k_i}|y_i)$. Then,
\[
\hat I = \frac{1}{N} \sum_{i=1}^N U_i
\]
is minimized w.r.t. the generative model parameters $\sigma$.

```python
Initialize model p_σ(x,y) and encoder q_θ(x|y).
repeat
  Sample a batch {(x_i, y_i)}_{i=1}^N from p_σ.
  // (1) Update q_θ by maximizing conditional log‐likelihood
  L(θ) ← (1/N) ∑_{i=1}^N log q_θ(x_i|y_i).
  θ ← θ + η ∇_θ L(θ).
  // (2) Compute one‒negative vCLUB estimate
  for i=1…N do
    draw k_i ∼ Uniform({1,…,N});
    U_i ← log q_θ(x_i|y_i) - log q_θ(x_{k_i}|y_i).
  end for
  Ĩ ← (1/N) ∑_{i=1}^N U_i.
  // (3) Update σ by minimizing Ĩ
  σ ← σ − η′ ∇_σ Ĩ   (backprop through samples via reparametrization).
until convergence
```

This sampled variant is unbiased and substantially more scalable. The negative sampling strategy improves both statistical and computational characteristics of the bound.

## 6. Empirical Evaluation and Practical Performance

CLUB and its variational extensions have undergone validation on both synthetic and real-world benchmarks:
- **Synthetic Estimation**: On Gaussian/Cubic data ($d=20$, true $I \in \{2,4,6,8,10\}$), CLUB attains the lowest bias and minimum squared error compared to competing lower and upper bound MI estimators. CLUB–S incurs slightly higher variance but maintains unbiasedness.
- **Information Bottleneck (MNIST, latent dim=256)**: CLUB and vCLUB achieve lower test classification error (approaching $1.06\%$) than DVB (VUB), MINE, NWJ, InfoNCE, and Leave-One-Out estimators. Negative sampling further enhances generalization.
- **Unsupervised Domain Adaptation**: In MNIST$\rightarrow$MNIST-M and USPS$\rightarrow$MNIST settings, within a disentangled representation objective minimizing $I(z_c; z_d)$, CLUB–S provides the highest or near-highest target domain accuracy ($94.6\%$–$98.9\%$), surpassing lower-bound and prior upper-bound estimators that suffer from numerical instability.

The aggregate findings demonstrate that CLUB delivers a tight, stable, and computationally efficient upper bound for MI estimation and minimization in high-dimensional deep learning tasks [2006.12013].

Source: https://www.emergentmind.com/topics/contrastive-log-ratio-upper-bound-club